Organization pixel tag acquisition method based on generative adversarial network and related equipment

Abstract

The invention discloses a generative adversarial network-based tissue pixel tag acquisition method and related equipment. The method comprises the steps of generating a plurality of CK dyeing image blocks and acquiring a plurality of DAB channel images based on a full-slide image of a tissue section in a CK dyeing style; based on an Otsu method and morphological operation, acquiring a coarse annotation image of each DAB channel image; correcting the coarse annotation image of each DAB channel image to obtain a tissue pixel label of each CK dyeing image block; and converting the CK dyeing image block into a target dyeing style image block by using a generative adversarial network model, and setting the tissue pixel tag of the CK dyeing image block as the tissue pixel tag of the target dyeing style image block. According to the method, the excessive dependence on pathological professional knowledge is avoided, the labeling time and energy of pathologists are reduced, and the tissue pixel labels of the tissue slices under different dyeing styles can be efficiently obtained.

Description

Technical Field

[0001] The present application relates to the technical field of image segmentation pixel labels, and more specifically, to a method for obtaining tissue pixel labels based on a generative adversarial network and related equipment. Background Art

[0002] In the field of pathology and medical image processing, tissue semantic segmentation is a basic and critical task. It plays a vital role in many clinical studies and applications, including immune infiltration analysis and histological grading analysis of patients. At present, the deep learning model used for tissue semantic segmentation is still dominated by fully supervised models, which require a large amount of accurately labeled training data, while the annotation of pathological images is still mainly based on manual delineation to obtain tissue pixel labels. This method requires pathologists with professional knowledge to annotate the tissues in pathological images at the pixel level. This process is not only time-consuming, but also increases the workload of pathologists and affects their work efficiency. In addition, the quality of manual delineation annotation may vary depending on the experience and subjective judgment of the annotator, making it difficult to ensure the accuracy and consistency of the annotation results. In addition, the same tissue section can be stained with CK staining, H&E staining or CD3 staining to obtain stained images with different staining styles. Each staining style highlights different cells or tissue components in the tissue section, and each has its own unique advantages and application scenarios. How to efficiently obtain tissue pixel labels of tissue sections in different staining styles has become an urgent problem to be solved. Summary of the invention

[0003] In view of this, the present application provides a method and related equipment for acquiring tissue pixel labels based on a generative adversarial network to achieve efficient acquisition of tissue pixel labels of tissue sections under different staining styles.

[0004] To achieve the above objectives, the first aspect of the present application provides a method for obtaining tissue pixel labels based on a generative adversarial network, comprising:

[0005] Generate multiple CK staining image blocks based on the whole slide image of the tissue section in the CK staining style, and obtain multiple DAB channel images based on each CK staining image block, wherein each CK staining image block has the same preset size;

[0006] Based on the Otsu method and morphological operations, a coarsely annotated image of each DAB channel image is obtained, wherein the coarsely annotated image is annotated with an epithelial area and a mesenchymal area;

[0007] Correcting the coarsely labeled image of each DAB channel image to obtain the tissue pixel label of each CK-stained image block, wherein the tissue pixel label is labeled with a normal gland area, a normal gland interstitial area, a tumor area, and a tumor interstitial area;

[0008] A generative adversarial network model is used to convert a CK-stained image block into a target-stained style image block, and the tissue pixel label of the CK-stained image block is set as the tissue pixel label of the target-stained style image block. The generative adversarial network model is configured to have the ability to convert an image in a CK-stained style into an image in a target-stained style, and the target-stained style is an H&E-stained style or a CD3-stained style.

[0009] Preferably, the process of generating a plurality of CK-stained image blocks based on the whole slide image of the tissue section in the CK-stained style, and acquiring a plurality of DAB channel images based on each CK-stained image block, comprises:

[0010] Cropping the whole slide image to a preset size at a preset first multiple to obtain a cropped whole slide image;

[0011] Downsampling the cropped whole slide image to a preset second multiple to obtain a plurality of CK staining image blocks of preset sizes;

[0012] The color deconvolution algorithm is used to extract the DAB channel information of each CK stained image block to obtain multiple DAB channel images.

[0013] Preferably, the process of obtaining a coarsely annotated image of each DAB channel image based on the Otsu method and morphological operations includes:

[0014] Based on Otsu's method and morphological opening operation, the epithelial area mask of each DAB channel image was obtained;

[0015] Based on Otsu's method and morphological opening operation, the interstitial mask of each DAB channel image is obtained;

[0016] Based on each DAB channel image, the epithelial region mask and the stroma mask of the DAB channel image, a coarse annotation image of the DAB channel image is generated.

[0017] Preferably, the process of obtaining the epithelial area mask of each DAB channel image based on the Otsu method and morphological opening operation includes:

[0018] Using a Gaussian filter with a preset first coefficient value to smooth each DAB channel image, to obtain each first DAB channel image;

[0019] The epithelial area and other areas of each first DAB channel image are segmented based on the threshold determined by the Otsu method to obtain a first mask image;

[0020] Using a morphological opening operation to remove the first invalid area in each first mask image, to obtain each second mask image;

[0021] Performing a negation operation on each second mask image to obtain an epithelial region mask of each DAB channel image;

[0022] The first invalid area includes an area where the gland mask area is smaller than a preset first pixel number and an area where the background area is smaller than a preset second pixel number.

[0023] Preferably, the process of obtaining the interstitial mask of each DAB channel image based on the Otsu method and morphological opening operation includes:

[0024] Setting the pixels corresponding to the median value of 1 in the tumor region mask in each DAB channel image to a preset third pixel value to obtain each third mask image;

[0025] Using a Gaussian filter with a preset second coefficient value to smooth each third mask image, to obtain each fourth mask image;

[0026] Segmenting the background and foreground of each fourth mask image based on a threshold value determined by the Otsu method;

[0027] Using a morphological opening operation to remove the second invalid area in each fourth mask image, to obtain each fifth mask image;

[0028] Performing a negation operation on each fifth mask image to obtain an interstitial mask of each DAB channel image;

[0029] The second invalid area includes an area where the interstitial mask area is smaller than a preset third number of pixels and an area where the background area is smaller than a preset fourth number of pixels.

[0030] Preferably, the process of generating a coarsely annotated image of the DAB channel image based on each DAB channel image, the epithelial region mask and the stroma mask of the DAB channel image comprises:

[0031] For each DAB channel image:

[0032] Creating a three-channel image with the same size as the DAB channel image, and initializing each pixel in the three-channel image to a preset fourth pixel value;

[0033] Setting the pixel points in the three-channel image corresponding to the median value of the epithelial area mask of 1 to a preset first pixel value, and setting the pixel points in the three-channel image corresponding to the median value of the stroma mask of 1 to a preset second pixel value;

[0034] The three-channel image is set as a coarsely annotated image of the DAB channel image.

[0035] Preferably, the process of correcting the coarsely labeled image of each DAB channel image to obtain the tissue pixel label of each CK staining image block includes:

[0036] The apoptotic bodies and cell debris are removed from the epithelial region of the coarsely annotated image of each DAB channel image to obtain a first annotated image of each DAB channel image;

[0037] The normal glandular region and the tumor region in the epithelial region of the first annotated image of each DAB channel image are identified and annotated, and the normal glandular interstitial region and the tumor interstitial region in the interstitial region of the first annotated image of each DAB channel image are identified and annotated to obtain the tissue pixel labels of the tissue sections in the CK staining style.

[0038] Preferably, the generative adversarial network model is a contrastive learning unpaired image conversion network.

[0039] The second aspect of the present application provides a device for acquiring a tissue pixel label based on a generative adversarial network, comprising:

[0040] A channel extraction unit, used to generate a plurality of CK staining image blocks based on the whole slide image of the tissue section in the CK staining style, and to obtain a plurality of DAB channel images based on each CK staining image block, wherein each CK staining image block has the same preset size;

[0041] A rough annotation unit, based on the Otsu method and morphological operations, obtains a rough annotation image of each DAB channel image, wherein the rough annotation image is annotated with an epithelial area and a mesenchymal area;

[0042] A label correction unit, used for correcting the coarsely labeled image of each DAB channel image to obtain a tissue pixel label of each CK stained image block, wherein the tissue pixel label is labeled with a normal gland area, a normal gland interstitial area, a tumor area, and a tumor interstitial area;

[0043] A staining conversion unit is used to convert a CK staining image block into a target staining style image block by using a generative adversarial network model, and set the tissue pixel label of the CK staining image block to the tissue pixel label of the target staining style image block. The generative adversarial network model is configured to have the ability to convert an image in a CK staining style into an image in a target staining style, and the target staining style is an H&E staining style or a CD3 staining style.

[0044] A third aspect of the present application provides a device for acquiring tissue pixel labels based on a generative adversarial network, comprising: a memory and a processor;

[0045] The memory is used to store programs;

[0046] The processor is used to execute the program to implement the various steps of the above-mentioned method for acquiring tissue pixel labels.

[0047] A fourth aspect of the present application provides a storage medium having a computer program stored thereon. When the computer program is executed by a processor, the steps of the method for acquiring a tissue pixel label as described above are implemented.

[0048] It can be known from the above-mentioned technical scheme that the present application first generates multiple CK staining image blocks based on the full slide image of tissue section under CK staining style, and based on each CK staining image block, obtains multiple DAB channel images, wherein, each CK staining image block and each DAB channel image have the same preset size.Due to the DAB dye can be selectively combined with the target molecule and the complex, specifically, the color development of the DAB channel image for the epithelial cell keratin can be more obvious, so that the contrast of the target and the background in the DAB channel image is stronger, which is more conducive to the segmentation of the target.Then, based on the Otsu method and morphological operation, the coarse annotation image of each DAB channel image is obtained, wherein, the coarse annotation image is annotated with epithelial area and interstitial area.It can be understood that the threshold method is used to segment the foreground and background in the DAB channel image, and the morphological operation is used to remove the specific area in the DAB channel image, by using different colors to fill the epithelial area and the interstitial area to realize the annotation of the epithelial area and the interstitial area. Then, the coarsely annotated images of each DAB channel image are corrected to obtain the tissue pixel labels of each CK staining image block, and the tissue pixel labels are annotated with normal glandular area, normal glandular interstitial area, tumor area and tumor interstitial area. It can be understood that in addition to specific staining of epithelium, CK staining also stains apoptotic bodies and cell fragments. These structures should not be misidentified as tumors, so further corrections are required to ensure the accuracy of annotation. Finally, the CK staining image block is converted into a target staining style image block using a generative adversarial network model, and the tissue pixel labels of the CK staining image block are set to the tissue pixel labels of the target staining style image block. The generative adversarial network model is configured to have the ability to convert images under the CK staining style into images under the target staining style, and the target staining style is H&E staining style or CD3 staining style. The present application avoids the transitional reliance on pathological expertise during the annotation process, greatly reduces the annotation time and energy of pathologists, and can directly convert the tissue pixel labels under the CK staining style into tissue pixel labels under the target staining style, and can efficiently obtain tissue pixel labels under different staining styles of tissue sections. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying any creative work.

[0050] Figure 1 A schematic diagram of a method for acquiring tissue pixel labels based on a generative adversarial network disclosed in an embodiment of the present application;

[0051] Figure 2 The CK stained image block disclosed in the embodiment of the present application is illustrated;

[0052] Figure 3 The DAB channel image disclosed in the embodiment of the present application is illustrated;

[0053] Figure 4 The epithelial region disclosed in the examples of the present application is illustrated;

[0054] Figure 5 The interstitial region disclosed in the embodiments of the present application is illustrated;

[0055] Figure 6 The coarsely annotated image disclosed in the embodiment of the present application is illustrated;

[0056] Figure 7 The tissue pixel label disclosed in the embodiment of the present application is illustrated;

[0057] Figure 8 The CD3 stained image block generated by the conversion disclosed in the embodiment of the present application is illustrated;

[0058] Fig. 9 An example of an H&E stained image block generated by conversion disclosed in an embodiment of the present application is provided;

[0059] Fig.10 A schematic diagram of a device for acquiring tissue pixel labels based on a generative adversarial network disclosed in an embodiment of the present application;

[0060] Fig.11 A schematic diagram of a device for acquiring tissue pixel labels based on a generative adversarial network disclosed in an embodiment of the present application. DETAILED DESCRIPTION

[0061] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0062] The following describes a method for obtaining tissue pixel labels based on a generative adversarial network provided in an embodiment of the present application. Figure 1 The method for obtaining tissue pixel labels based on a generative adversarial network provided in an embodiment of the present application may include the following steps:

[0063] Step S101 : generating a plurality of CK-stained image blocks based on a whole slide image of a tissue section in a CK-stained style, and acquiring a plurality of DAB channel images based on each CK-stained image block.

[0064] CK staining is a staining technique used for tumor sections. It targets antibodies to epithelial cytokeratin and is often used to help identify and analyze tumors. In CK-stained images, normal glands and tumor epithelial cells are usually displayed in brown, while non-target tissues are displayed in light blue. This strong color contrast facilitates subsequent image processing and analysis, and it is relatively easy to obtain accurate pixel-level labels for epithelial areas from CK-stained images.

[0065] Whole-Slide Images (WSIs) in CK-stained style usually contain hundreds of millions of pixels and cannot be directly input into the model for calculation. Therefore, the whole-slide images are divided into multiple CK-stained image patches as input. The common deep learning patch sizes are 256×256 pixels, 512×512 pixels, and 1024×1024 pixels.

[0066] In order to achieve more accurate segmentation, the color deconvolution algorithm can be used to perform staining separation on each CK staining image block, retaining only the DAB (3,3'-Diaminobenzidine) channel image. The operating principle of this step is that the two dyes H and DAB will selectively bind to the target molecules and complexes, and the color deconvolution algorithm can convert the RGB (Red-Green-Blue) color space into H and DAB color spaces in order to separate different staining components. Specifically, the input of the color deconvolution operation is a CK staining image block in RGB format. According to the absorbance of different dyes, the contribution of the dye is calculated to separate different staining channels. In an embodiment of the present application, each CK staining image block and each DAB channel image have the same preset size. Exemplarily, the preset size can be 1024×1024 pixels. Figure 2 , Figure 3 A CK-stained image block and its DAB channel image in an embodiment of the present application are illustrated respectively.

[0067] Step S102: Based on the Otsu method and morphological operations, a coarsely labeled image of each DAB channel image is obtained.

[0068] Otsu's Method is an algorithm for determining the threshold value of image binary segmentation. It is based on the entropy principle in information theory and determines the optimal threshold value by calculating the maximum entropy in the image histogram. The core idea of ​​this method is to find a threshold value so that the gray value distribution inside the two parts of the image divided into foreground and background is the most uniform, while the gray value distribution between the two parts is as different as possible, so as to achieve the best segmentation effect.

[0069] Morphological operations are a technology widely used in image processing and machine vision, especially in the image feature extraction stage, which can be used to extract component information in an image, which is usually important for expressing and describing the shape of the image. Morphological operations are based on concepts such as dilation and erosion in set theory, and more complex operations such as opening and closing are derived from the combination of these two basic operations.

[0070] The embodiment of the present application determines the foreground and background of each DAB channel image based on the Otsu method, and uses morphological operations to determine the epithelial area and the mesenchymal area therefrom. Figure 4 , Figure 5 The corresponding examples are given respectively Figure 3 The epithelial and mesenchymal regions of the DAB channel image. Figure 6 The example corresponds to Figure 3 Coarse annotation image of the DAB channel image.

[0071] Step S103, correcting the coarsely labeled image of each DAB channel image to obtain the tissue pixel label of each CK staining image block.

[0072] Exemplarily, the correction may include correction on content and correction on color. Specifically, in addition to specifically staining epithelium, CK also stains apoptotic bodies and cell fragments. These structures should not be misidentified as tumors, so they need to be corrected to ensure the accuracy of the labeling. The epithelial area includes the tumor area and the normal glandular area, and the normal glandular area is identified and marked as another color. The interstitial area is divided into tumor interstitium and normal glandular interstitium, and the normal glandular interstitium is identified and marked as another color. After a series of corrections, the tissue pixel label can include the following four annotations: normal glandular area, normal glandular interstitial area, tumor area, and tumor interstitial area. Figure 7 The example corresponds to Figure 2It should be noted that since the CK-stained image block is an abnormal area in the tissue (ie, a tumor), the tissue pixel label does not include a normal glandular area or a normal glandular interstitial area.

[0073] Step S104: using a generative adversarial network model to convert the CK stained image block into a target stained style image block, and setting the tissue pixel label of the CK stained image block as the tissue pixel label of the target stained style image block.

[0074] It can be understood that the generative adversarial network model is configured to have the ability to convert an image in a CK staining style into an image in a target staining style, and the target staining style is an H&E staining style or a CD3 staining style. CK staining, H&E staining, and CD3 staining are all common staining techniques for tumor sections. Although each staining method highlights different cell or tissue components, they have a common structural basis and morphological characteristics. This commonality allows the stained image to be migrated from one staining style to another staining style image through a generative adversarial network model, while keeping the structural information of the original image unchanged. Figure 8 , Fig. 9 The corresponding examples are given respectively Figure 2 The conversion of the CK staining image block generates the CD3 staining image block and the H&E staining image block.

[0075] In this step, the CK-stained image blocks under the CK-stained style are used to generate the target-stained style image blocks under the target-stained style. The two images correspond one to one. Although the styles are different, the contents are the same. Therefore, the corresponding tissue pixel labels are also the same. That is, the image blocks after style transfer share the same tissue pixel labels as the original image blocks, so that the labeled data is utilized to the greatest extent, while ensuring the accuracy and consistency of the annotations, providing high-quality data support for subsequent tissue segmentation tasks.

[0076] The present application first generates multiple CK staining image blocks based on the full slide image of tissue section under CK staining style, and based on each CK staining image block, obtains multiple DAB channel images, wherein each CK staining image block and each DAB channel image have the same preset size.Due to the DAB dye can be selectively combined with the target molecule and the complex, specifically, the color development of the DAB channel image for the epithelial cell keratin can be more obvious, so that the contrast between the target and the background in the DAB channel image is stronger, which is more conducive to the segmentation of the target.Then, based on the Otsu method and morphological operation, the coarse annotation image of each DAB channel image is obtained, wherein the coarse annotation image is annotated with epithelial area and interstitial area.It can be understood that the threshold method is used to segment the foreground and background in the DAB channel image, and the morphological operation is used to remove the specific area in the DAB channel image, by using different colors to fill the epithelial area and the interstitial area to realize the annotation of the epithelial area and the interstitial area. Then, the coarsely annotated images of each DAB channel image are corrected to obtain the tissue pixel labels of each CK staining image block, and the tissue pixel labels are annotated with normal glandular area, normal glandular interstitial area, tumor area and tumor interstitial area. It can be understood that in addition to specific staining of epithelium, CK staining also stains apoptotic bodies and cell fragments. These structures should not be misidentified as tumors, so further corrections are required to ensure the accuracy of annotation. Finally, the CK staining image block is converted into a target staining style image block using a generative adversarial network model, and the tissue pixel labels of the CK staining image block are set to the tissue pixel labels of the target staining style image block. The generative adversarial network model is configured to have the ability to convert images under the CK staining style into images under the target staining style, and the target staining style is H&E staining style or CD3 staining style. The present application avoids the transitional reliance on pathological expertise during the annotation process, greatly reduces the annotation time and energy of pathologists, and can directly convert the tissue pixel labels under the CK staining style into tissue pixel labels under the target staining style, and can efficiently obtain tissue pixel labels under different staining styles of tissue sections.

[0077] In some embodiments of the present application, the process of generating multiple CK-stained image blocks based on the whole slide image of the tissue section in the CK-stained style and acquiring multiple DAB channel images based on each CK-stained image block in step S101 may include:

[0078] S1, cropping the whole slide image to a preset size at a preset first magnification to obtain a cropped whole slide image.

[0079] Exemplarily, the first multiple may be 40 times, and the preset size may be 4096×4096 pixels.

[0080] S2, down-sampling the cropped whole slide image to a preset second multiple to obtain a plurality of CK staining image blocks of preset sizes.

[0081] Exemplarily, the second multiple may be 10 times, and the preset size may be 1024×1024 pixels.

[0082] S3, using a color deconvolution algorithm to extract DAB channel information of each CK stained image block to obtain multiple DAB channel images.

[0083] In some embodiments of the present application, the process of obtaining a coarsely annotated image of each DAB channel image based on the Otsu method and morphological operation in step S102 may include:

[0084] S1, based on Otsu's method and morphological opening operation, the epithelial area mask of each DAB channel image is obtained.

[0085] S2, based on Otsu’s method and morphological opening operation, obtain the interstitial mask of each DAB channel image.

[0086] S3 , generating a coarsely annotated image of the DAB channel image based on each DAB channel image, the epithelial region mask and the stroma mask of the DAB channel image.

[0087] In some embodiments of the present application, the process of obtaining the epithelial area mask of each DAB channel image in S1 based on the Otsu method and morphological opening operation may include:

[0088] S11 , performing smoothing processing on each DAB channel image by using a Gaussian filter with a preset first coefficient value to obtain each first DAB channel image.

[0089] Exemplarily, the first coefficient value may be 2.

[0090] S12, segmenting the epithelial area and other areas of each first DAB channel image based on the threshold determined by the Otsu method to obtain a first mask image.

[0091] S13, using a morphological opening operation to remove the first invalid area in each first mask image to obtain each second mask image.

[0092] The first invalid area includes an area where the gland mask area is smaller than a preset first pixel number and an area where the background area is smaller than a preset second pixel number.

[0093] Exemplarily, the first number of pixels may be 200 pixels, and the second number of pixels may be 60 pixels.

[0094] S14, performing a negation operation on each second mask image to obtain an epithelial region mask of each DAB channel image.

[0095] Specifically, the image mask is usually a binary or Boolean image of the same size as the original image, in which the selected area is marked as 1 (or True) and the remaining area is marked as 0 (or False). In the embodiment of the present application, the selected area of ​​the mask is marked as 1, and the remaining area is marked as 0. Therefore, in the negation operation of S14, the non-empty area in each second mask image is marked as 0, and the empty area is marked as 1.

[0096] In some embodiments of the present application, the process of obtaining the interstitial mask of each DAB channel image in S2 based on the Otsu method and morphological opening operation may include:

[0097] S21, setting the pixels corresponding to the tumor region mask median value of 1 in each DAB channel image to a preset third pixel value to obtain each third mask image.

[0098] Exemplarily, the third pixel value may be 255, ie, the tumor portion is changed to a white background in the DAB channel image.

[0099] S22, performing smoothing processing on each third mask image by using a Gaussian filter with a preset second coefficient value to obtain each fourth mask image.

[0100] Exemplarily, the second coefficient value may be 2.

[0101] S23, segmenting the background and foreground of each fourth mask image based on the threshold value determined by the Otsu method.

[0102] S24, using a morphological opening operation to remove the second invalid area in each fourth mask image to obtain each fifth mask image.

[0103] The second invalid area includes an area where the tumor stroma mask area is smaller than a preset third number of pixels and an area where the background part area is smaller than a preset fourth number of pixels.

[0104] Exemplarily, the third number of pixels and the fourth number of pixels are both 200 pixels.

[0105] S25 , performing a negation operation on each fifth mask image to obtain an interstitial mask of each DAB channel image.

[0106] In some embodiments of the present application, the process of generating a coarsely annotated image of the DAB channel image in S3 based on each DAB channel image, the epithelial region mask and the stroma mask of the DAB channel image may include:

[0107] For each DAB channel image:

[0108] S31, creating a three-channel image with the same size as the DAB channel image, and initializing each pixel in the three-channel image to a preset fourth pixel value.

[0109] Exemplarily, the fourth pixel value may be white (R=255, G=0255, B=255).

[0110] S32, setting the pixel points corresponding to the median value of 1 in the epithelial area mask in the three-channel image to a preset first pixel value.

[0111] Exemplarily, the first pixel value may be red (R=230, G=0, B=18).

[0112] S33, setting the pixel points corresponding to the median value of the interstitial mask of the three-channel image to a preset second pixel value.

[0113] Exemplarily, the second pixel value may be yellow (R=255, G=241, B=0).

[0114] In some embodiments of the present application, the process of step S103 of correcting the coarsely labeled image of each DAB channel image to obtain the tissue pixel label of each CK stained image block may include:

[0115] S1, removing apoptotic bodies and cell debris from the epithelial region of the coarsely annotated image of each DAB channel image to obtain a first annotated image of each DAB channel image.

[0116] S2, identifying and labeling the normal glandular region and the tumor region in the epithelial region of the first labeled image of each DAB channel image, and identifying and labeling the normal glandular interstitial region and the tumor interstitial region in the tumor interstitial region of the first labeled image of each DAB channel image, and obtaining the tissue pixel label of the tissue section under the CK staining style.

[0117] The glands are divided into tumors and normal glands. Therefore, the tumor area marked in the coarsely marked image may contain normal gland parts. It is necessary to identify the normal gland parts and mark the normal gland areas with other colors (such as blue: R=0, G=160, B=233).

[0118] In some embodiments of the present application, the generative adversarial network model in step S104 is a contrastive learning for unpaired image translation network (CUT).

[0119] The generative adversarial network model is mainly composed of a generator and a discriminator. The generator consists of an encoder and a decoder, which is used to generate pseudo-dyed images in the target style, and the discriminator distinguishes true from false. The generator consists of an encoder and a decoder, and the discriminator uses PatchGAN, which divides the input image into 70×70 overlapping image blocks, judges the probability of each image block being real or false, and then obtains the overall judgment based on the judgment of the image blocks. In order to keep the generated image as consistent as possible with the original image content, contrastive learning is introduced. After the input image x passes through the encoder, a feature map is generated in the form of a stack. If a feature layer is taken, a certain spatial position can be represented as an image block of the input image. The deeper the layer, the larger the image block. If the number of layers is selected as L, then the feature map H is obtained through MLP l , the feature space obtained after the operation is expressed as:

[0120]

[0121] represents the output of the encoder in the lth (l∈{1,2,…,L}) layer generator.

[0122] Assume that the number of spatial locations in layer l is S l , the spatial position is represented by s(s∈{1,…,S l}). Then the feature representation of the lth layer and the sth spatial position is in Indicates that the feature is a l Vector in C-dimensional space l is the number of channels in layer l, except Other features are represented as in Similarly, the image output by the generator is represented as

[0123]

[0124] G(x) represents the generator output. Here we use the InfoNCE loss function in contrastive learning, and the formula is:

[0125]

[0126] Among them, ν, ν + , ν - represents query, positive sample and negative sample, v - represents the (N-1)th negative number, where query refers to the image patch selected from the output image of the generator, the positive sample is the image patch at the corresponding position of the input image, and the negative sample is the image patch at the non-corresponding position. τ is the temperature hyperparameter.

[0127] If a Patch is selected in the generator output, the Patch is regarded as the query, the Patch with the matching position in the corresponding generator input is the positive sample, and the Patch with the mismatched position is the negative sample, thus we can get:

[0128]

[0129] The embodiment of the present application uses an adversarial loss function to constrain the encoder and encourage the generator to generate images with a visually similar style to the Y domain. The adversarial loss function is expressed as follows:

[0130]

[0131] Where x is the input X domain image, and y is the target Y domain image. D(G(x)) represents the discriminator's judgment on the authenticity of the generated X domain image. Therefore, for In this part, the goal of the generator is to minimize this part value to deceive the discriminator, while the goal of the discriminator is to maximize this loss.

[0132] In order to maximize the input and output information of the generator, the idea of ​​contrastive learning is introduced, and the contrastive loss function InfoNCE function is used for constraint. The total loss function is expressed as follows:

[0133]

[0134] The embodiment of the present application converts tissue pixel labels under CK staining into tissue pixel labels under multiple staining styles (such as H&E staining and CD3 staining) through a generative adversarial network model, which can efficiently obtain tissue pixel labels of tissue sections under different staining styles.

[0135] The training phase of the generative adversarial network model requires the preparation of sufficient training data and test data. In the embodiment of the present application, the WSIs of CK staining and target domain style staining (H&E, CD3) are cut into image blocks of 512*512 pixels. The CK-stained image blocks are used as the training set A of the X domain, and the target style staining Patch is used as the training set B of the Y domain. The test data is the CK-stained image blocks of 1024*1024 pixels used to generate coarse annotations.

[0136] The following is a description of the device for acquiring tissue pixel labels based on a generative adversarial network provided in an embodiment of the present application. The device for acquiring tissue pixel labels based on a generative adversarial network described below and the method for acquiring tissue pixel labels based on a generative adversarial network described above can refer to each other.

[0137] See also Fig.10 , the device for acquiring tissue pixel labels based on a generative adversarial network provided in an embodiment of the present application may include:

[0138] A channel extraction unit 21 is used to generate a plurality of CK staining image blocks based on the whole slide image of the tissue section in the CK staining style, and obtain a plurality of DAB channel images based on each CK staining image block, wherein each CK staining image block has the same preset size;

[0139] A rough annotation unit 22, based on the Otsu method and morphological operations, obtains a rough annotation image of each DAB channel image, wherein the rough annotation image is annotated with an epithelial area and a stroma area;

[0140] The annotation correction unit 23 is used to correct the coarse annotation image of each DAB channel image to obtain the tissue pixel label of each CK staining image block, wherein the tissue pixel label is annotated with a normal gland area, a normal gland interstitial area, a tumor area, and a tumor interstitial area;

[0141] The dye conversion unit 24 is used to convert the CK dyed image block into a target dyed style image block by using a generative adversarial network model, and set the tissue pixel label of the CK dyed image block to the tissue pixel label of the target dyed style image block. The generative adversarial network model is configured to have the ability to convert an image under the CK dyeing style into an image under the target dyeing style, and the target dyeing style is the H&E dyeing style or the CD3 dyeing style.

[0142] The apparatus for acquiring tissue pixel labels based on a generative adversarial network provided in the embodiment of the present application can be applied to a device for acquiring tissue pixel labels based on a generative adversarial network, such as a computer. Fig.11 The hardware structure block diagram of the device for acquiring the tissue pixel label based on the generative adversarial network is shown. Figure 3 The hardware structure of the device for acquiring organizational pixel labels based on a generative adversarial network may include: at least one processor 31, at least one communication interface 32, at least one memory 33 and at least one communication bus 34.

[0143] In the embodiment of the present application, the number of the processor 31, the communication interface 32, the memory 33, and the communication bus 34 is at least one, and the processor 31, the communication interface 32, and the memory 33 communicate with each other through the communication bus 34;

[0144] The processor 31 may be a central processing unit CPU, or an application-specific integrated circuit ASIC (Application Specific Integrated Circuit), or one or more integrated circuits configured to implement the embodiments of the present application, etc.;

[0145] The memory 33 may include a high-speed RAM memory, and may also include a non-volatile memory, such as at least one disk memory;

[0146] The memory 33 stores a program, and the processor 31 can call the program stored in the memory 33, and the program is used to:

[0147] Generate multiple CK staining image blocks based on the whole slide image of the tissue section in the CK staining style, and obtain multiple DAB channel images based on each CK staining image block, wherein each CK staining image block has the same preset size;

[0148] Based on the Otsu method and morphological operations, a coarsely annotated image of each DAB channel image is obtained, wherein the coarsely annotated image is annotated with an epithelial area and a mesenchymal area;

[0149] Correcting the coarsely labeled image of each DAB channel image to obtain the tissue pixel label of each CK-stained image block, wherein the tissue pixel label is labeled with a normal gland area, a normal gland interstitial area, a tumor area, and a tumor interstitial area;

[0150] A generative adversarial network model is used to convert a CK-stained image block into a target-stained style image block, and the tissue pixel label of the CK-stained image block is set as the tissue pixel label of the target-stained style image block. The generative adversarial network model is configured to have the ability to convert an image in a CK-stained style into an image in a target-stained style, and the target-stained style is an H&E-stained style or a CD3-stained style.

[0151] Optionally, the detailed functions and extended functions of the program may refer to the above description.

[0152] The embodiment of the present application further provides a storage medium, which may store a program suitable for execution by a processor, wherein the program is used to:

[0153] Generate multiple CK staining image blocks based on the whole slide image of the tissue section in the CK staining style, and obtain multiple DAB channel images based on each CK staining image block, wherein each CK staining image block has the same preset size;

[0154] Based on the Otsu method and morphological operations, a coarsely annotated image of each DAB channel image is obtained, wherein the coarsely annotated image is annotated with an epithelial area and a mesenchymal area;

[0155] Correcting the coarsely labeled image of each DAB channel image to obtain the tissue pixel label of each CK-stained image block, wherein the tissue pixel label is labeled with a normal gland area, a normal gland interstitial area, a tumor area, and a tumor interstitial area;

[0156] The CK stained image block is input into the trained style transfer model to obtain a target stained style image block corresponding to the CK stained image block, and the tissue pixel label of the CK stained image block is set as the tissue pixel label of the target stained style image block. The generative adversarial network model is configured to have the ability to convert an image under the CK staining style into an image under the target staining style, and the target staining style is the H&E staining style or the CD3 staining style.

[0157] Optionally, the detailed functions and extended functions of the program may refer to the above description.

[0158] In summary:

[0159] The present application first generates multiple CK staining image blocks based on the full slide image of tissue section under CK staining style, and based on each CK staining image block, obtains multiple DAB channel images, wherein each CK staining image block and each DAB channel image have the same preset size.Due to the DAB dye can be selectively combined with the target molecule and the complex, specifically, the color development of the DAB channel image for the epithelial cell keratin can be more obvious, so that the contrast between the target and the background in the DAB channel image is stronger, which is more conducive to the segmentation of the target.Then, based on the Otsu method and morphological operation, the coarse annotation image of each DAB channel image is obtained, wherein the coarse annotation image is annotated with epithelial area and interstitial area.It can be understood that the threshold method is used to segment the foreground and background in the DAB channel image, and the morphological operation is used to remove the specific area in the DAB channel image, by using different colors to fill the epithelial area and the interstitial area to realize the annotation of the epithelial area and the interstitial area. Then, the coarsely annotated images of each DAB channel image are corrected to obtain the tissue pixel labels of each CK staining image block, and the tissue pixel labels are annotated with normal glandular area, normal glandular interstitial area, tumor area and tumor interstitial area. It can be understood that in addition to specific staining of epithelium, CK staining also stains apoptotic bodies and cell fragments. These structures should not be misidentified as tumors, so further corrections are required to ensure the accuracy of annotation. Finally, the CK staining image block is converted into a target staining style image block using a generative adversarial network model, and the tissue pixel labels of the CK staining image block are set to the tissue pixel labels of the target staining style image block. The generative adversarial network model is configured to have the ability to convert images under the CK staining style into images under the target staining style, and the target staining style is H&E staining style or CD3 staining style. The present application avoids the transitional reliance on pathological expertise during the annotation process, greatly reduces the annotation time and energy of pathologists, and can directly convert the tissue pixel labels under the CK staining style into tissue pixel labels under the target staining style, and can efficiently obtain tissue pixel labels under different staining styles of tissue sections.

[0160] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.

[0161] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The various embodiments can be combined as needed, and the same or similar parts can refer to each other.

[0162] The above description of the disclosed embodiments enables those skilled in the art to implement or use the present application. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to the embodiments shown herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for obtaining tissue pixel labels based on a generative adversarial network, characterized in that: include: Generate multiple CK staining image blocks based on the whole slide image of the tissue section in the CK staining style, and obtain multiple DAB channel images based on each CK staining image block, wherein each CK staining image block and each DAB channel image have the same preset size; Based on the Otsu method and morphological operations, a coarsely annotated image of each DAB channel image is obtained, wherein the coarsely annotated image is annotated with an epithelial area and a mesenchymal area; Correcting the coarsely labeled image of each DAB channel image to obtain the tissue pixel label of each CK-stained image block, wherein the tissue pixel label is labeled with a normal gland area, a normal gland interstitial area, a tumor area, and a tumor interstitial area; A generative adversarial network model is used to convert a CK-stained image block into a target-stained style image block, and the tissue pixel label of the CK-stained image block is set as the tissue pixel label of the target-stained style image block. The generative adversarial network model is configured to have the ability to convert an image in a CK-stained style into an image in a target-stained style, and the target-stained style is an H&E-stained style or a CD3-stained style.

2. The method according to claim 1, characterized in that: The process of generating a plurality of CK staining image blocks based on the whole slide image of the tissue section in the CK staining style, and acquiring a plurality of DAB channel images based on each CK staining image block, comprises: Cropping the whole slide image to a preset size at a preset first multiple to obtain a cropped whole slide image; Downsampling the cropped whole slide image to a preset second multiple to obtain a plurality of CK staining image blocks of preset sizes; The color deconvolution algorithm is used to extract the DAB channel information of each CK stained image block to obtain multiple DAB channel images.

3. The method according to claim 1, characterized in that The process of obtaining the coarsely labeled image of each DAB channel image based on the Otsu method and morphological operation includes: Based on Otsu's method and morphological opening operation, the epithelial area mask of each DAB channel image was obtained; Based on Otsu's method and morphological opening operation, the interstitial mask of each DAB channel image is obtained; Based on each DAB channel image, the epithelial region mask and the stroma mask of the DAB channel image, a coarse annotation image of the DAB channel image is generated.

4. The method according to claim 3, characterized in that The process of obtaining the epithelial region mask of each DAB channel image based on the Otsu method and morphological opening operation includes: Using a Gaussian filter with a preset first coefficient value to smooth each DAB channel image, to obtain each first DAB channel image; The epithelial area and other areas of each first DAB channel image are segmented based on the threshold determined by the Otsu method to obtain a first mask image; Using a morphological opening operation to remove the first invalid area in each first mask image, to obtain each second mask image; Performing a negation operation on each second mask image to obtain an epithelial region mask of each DAB channel image; The first invalid area includes an area where the gland mask area is smaller than a preset first pixel number and an area where the background area is smaller than a preset second pixel number.

5. The method according to claim 3, characterized in that: The process of obtaining the interstitial mask of each DAB channel image based on the Otsu method and morphological opening operation includes: Setting the pixels corresponding to the median value of 1 in the tumor region mask in each DAB channel image to a preset third pixel value to obtain each third mask image; Using a Gaussian filter with a preset second coefficient value to smooth each third mask image, to obtain each fourth mask image; Segmenting the background and foreground of each fourth mask image based on a threshold value determined by the Otsu method; Using a morphological opening operation to remove the second invalid area in each fourth mask image, to obtain each fifth mask image; Performing a negation operation on each fifth mask image to obtain an interstitial mask of each DAB channel image; The second invalid area includes an area where the interstitial mask area is smaller than a preset third number of pixels and an area where the background area is smaller than a preset fourth number of pixels.

6. The method according to claim 3, characterized in that The process of generating a coarsely annotated image of the DAB channel image based on each DAB channel image, an epithelial region mask and a stroma mask of the DAB channel image comprises: For each DAB channel image: Creating a three-channel image with the same size as the DAB channel image, and initializing each pixel in the three-channel image to a preset fourth pixel value; Setting the pixel points in the three-channel image corresponding to the median value of the epithelial area mask of 1 to a preset first pixel value, and setting the pixel points in the three-channel image corresponding to the median value of the stroma mask of 1 to a preset second pixel value; The three-channel image is set as a coarsely annotated image of the DAB channel image.

7. The method according to claim 1, characterized in that The process of correcting the coarsely labeled image of each DAB channel image to obtain the tissue pixel label of each CK staining image block includes: The apoptotic bodies and cell debris are removed from the epithelial region of the coarsely annotated image of each DAB channel image to obtain a first annotated image of each DAB channel image; The normal glandular region and the tumor region in the epithelial region of the first annotated image of each DAB channel image are identified and annotated, and the normal glandular interstitial region and the tumor interstitial region in the interstitial region of the first annotated image of each DAB channel image are identified and annotated to obtain the tissue pixel labels of the tissue sections in the CK staining style.

8. A device for acquiring tissue pixel labels based on a generative adversarial network, characterized in that: include: A channel extraction unit, used to generate a plurality of CK staining image blocks based on the whole slide image of the tissue section in the CK staining style, and to obtain a plurality of DAB channel images based on each CK staining image block, wherein each CK staining image block has the same preset size; A rough annotation unit, based on the Otsu method and morphological operations, obtains a rough annotation image of each DAB channel image, wherein the rough annotation image is annotated with an epithelial area and a mesenchymal area; A label correction unit, used for correcting the coarsely labeled image of each DAB channel image to obtain a tissue pixel label of each CK stained image block, wherein the tissue pixel label is labeled with a normal gland area, a normal gland interstitial area, a tumor area, and a tumor interstitial area; A staining conversion unit is used to convert a CK staining image block into a target staining style image block by using a generative adversarial network model, and set the tissue pixel label of the CK staining image block to the tissue pixel label of the target staining style image block. The generative adversarial network model is configured to have the ability to convert an image in a CK staining style into an image in a target staining style, and the target staining style is an H&E staining style or a CD3 staining style.

9. A device for acquiring tissue pixel labels based on a generative adversarial network, characterized in that: include: Memory and processor; The memory is used to store programs; The processor is used to execute the program to implement each step of the method for acquiring a tissue pixel label according to any one of claims 1 to 7.

10. A storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, each step of the method for acquiring a tissue pixel label according to any one of claims 1 to 7 is implemented.

Images