Tissue pixel label acquisition method based on generative adversarial network and related equipment

Abstract

The application discloses a kind of based on the generation of antinomy network's tissue pixel label acquisition method and related equipment, comprising: based on the whole slide image of tissue slice under the CK dyeing style generates multiple CK dyeing image blocks and obtains multiple DAB channel images;Based on Otsu method and morphological operation, the rough annotation image of each DAB channel image is obtained;The rough annotation image of each DAB channel image is corrected, and the tissue pixel label of each CK dyeing image block is obtained;CK dyeing image block is converted into target dyeing style image block using generative adversarial network model, and the tissue pixel label of the CK dyeing image block is set as the tissue pixel label of the target dyeing style image block.The application avoids excessive dependence on pathological professional knowledge, reduces the labeling time and effort of pathologists, and can efficiently obtain the tissue pixel label of tissue slice under different dyeing styles.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image segmentation pixel label, and more particularly to a tissue pixel label acquisition method based on a generative adversarial network and related equipment. BACKGROUND

[0002] In the field of pathology and medical image processing, tissue semantic segmentation is a fundamental and key task. It plays a crucial role in many clinical researches and applications, including immune infiltration analysis and histological grading analysis of patients. At present, the deep learning model used for tissue semantic segmentation is still mainly supervised model. The supervised model needs a large amount of labeled training data, and the labeling of pathological images is still mainly based on manual delineation to obtain tissue pixel labels. This method requires pathologists with professional knowledge to annotate the tissues in the pathological images at the pixel level. This process not only consumes time, but also increases the workload of pathologists, affecting their work efficiency. In addition, the quality of manual delineation may vary due to the experience and subjective judgment of the annotators, making it difficult to ensure the accuracy and consistency of the annotation results. Furthermore, the same tissue section can be applied to CK staining, H&E staining or CD3 staining for staining processing, thereby obtaining stained images under different staining styles. Each staining style highlights different cells or tissue components in the tissue section, each with its unique advantages and application scenarios. How to efficiently obtain tissue pixel labels of tissue sections under different staining styles has become a problem to be solved. SUMMARY

[0003] In view of the above, the present application provides a tissue pixel label acquisition method based on a generative adversarial network and related equipment to efficiently obtain tissue pixel labels of tissue sections under different staining styles.

[0004] To achieve the above-mentioned purpose, the first aspect of the present application provides a tissue pixel label acquisition method based on a generative adversarial network, comprising:

[0005] Based on the whole slide image of the tissue section under the CK staining style, a plurality of CK stained image blocks are generated, and based on each CK stained image block, a plurality of DAB channel images are obtained, wherein each CK stained image block has the same preset size;

[0006] Based on the Otsu method and morphological operation, a rough annotation image of each DAB channel image is obtained, and the rough annotation image is annotated with epithelial region and interstitial region;

[0007] The rough annotation image of each DAB channel image is corrected to obtain the tissue pixel label of each CK stained image block, and the tissue pixel label is annotated with normal gland region, normal gland interstitial region, tumor region and tumor interstitial region;

[0008] The CK staining image block is converted into a target staining style image block by using a generative adversarial network model configured to have the ability to convert an image in a CK staining style into an image in a target staining style, and the tissue pixel label of the CK staining image block is set as the tissue pixel label of the target staining style image block, and the target staining style is an H&E staining style or a CD3 staining style.

[0009] Preferably, the process of generating a plurality of CK staining image blocks based on the whole slide image of the tissue slice in the CK staining style, and obtaining a plurality of DAB channel images based on each CK staining image block, comprises:

[0010] The whole slide image is cropped to a preset size at a preset first multiple to obtain a cropped whole slide image;

[0011] The cropped whole slide image is down-sampled to a preset second multiple to obtain a plurality of CK staining image blocks of a preset size;

[0012] The DAB channel information of each CK staining image block is extracted by using a color deconvolution algorithm to obtain a plurality of DAB channel images.

[0013] Preferably, the process of obtaining a coarse annotation image of each DAB channel image based on the Otsu method and morphological operation, comprises:

[0014] The epithelial region mask of each DAB channel image is obtained based on the Otsu method and morphological opening operation;

[0015] The interstitial mask of each DAB channel image is obtained based on the Otsu method and morphological opening operation;

[0016] The coarse annotation image of each DAB channel image is generated based on each DAB channel image, the epithelial region mask and the interstitial mask of the DAB channel image.

[0017] Preferably, the process of obtaining the epithelial region mask of each DAB channel image based on the Otsu method and morphological opening operation, comprises:

[0018] Each DAB channel image is smoothed by using a Gaussian filter with a preset first coefficient value to obtain a first DAB channel image;

[0019] The epithelial region and other regions of each first DAB channel image are segmented based on the threshold value determined by the Otsu method to obtain a first mask image;

[0020] The first invalid region in each first mask image is removed by using morphological opening operation to obtain a second mask image;

[0021] performing an inversion operation on each second mask image to obtain an epithelial region mask of each DAB channel image;

[0022] The first invalid region includes a region with an adenoid mask area less than a preset first pixel number and a region with a background part area less than a preset second pixel number.

[0023] Preferably, the process of obtaining the interstitial mask of each DAB channel image based on the Otsu method and the morphological opening operation includes:

[0024] setting pixels corresponding to the tumor region mask with a value of 1 in each DAB channel image to a preset third pixel value to obtain each third mask image;

[0025] performing smoothing processing on each third mask image using a Gaussian filter with a preset second coefficient value to obtain each fourth mask image;

[0026] segmenting the background and the foreground of each fourth mask image based on the threshold value determined by the Otsu method;

[0027] removing the second invalid region in each fourth mask image using the morphological opening operation to obtain each fifth mask image;

[0028] performing an inversion operation on each fifth mask image to obtain an interstitial mask of each DAB channel image;

[0029] The second invalid region includes a region with an interstitial mask area less than a preset third pixel number and a region with a background part area less than a preset fourth pixel number.

[0030] Preferably, the process of generating the coarse annotation image of each DAB channel image based on each DAB channel image, the epithelial region mask and the interstitial mask of the DAB channel image includes:

[0031] for each DAB channel image:

[0032] newly creating a three-channel image with the same size as the DAB channel image and initializing each pixel point in the three-channel image to a preset fourth pixel value;

[0033] setting pixel points corresponding to the epithelial region mask with a value of 1 in the three-channel image to a preset first pixel value and setting pixel points corresponding to the interstitial mask with a value of 1 in the three-channel image to a preset second pixel value;

[0034] setting the three-channel image as the coarse annotation image of the DAB channel image.

[0035] Preferably, the process of correcting the coarse-labeled image of each DAB channel image to obtain the tissue pixel label of each CK-stained image block comprises:

[0036] eliminating the apoptotic bodies and cell debris from the epithelial region of the coarse-labeled image of each DAB channel image to obtain a first-labeled image of each DAB channel image;

[0037] identifying and labeling the normal gland region and the tumor region in the epithelial region of the first-labeled image of each DAB channel image, and identifying and labeling the normal gland stroma region and the tumor stroma region in the stromal region of the first-labeled image of each DAB channel image to obtain the tissue pixel label of the tissue slice under the CK staining style.

[0038] Preferably, the generative adversarial network model is a contrastive learning unpaired image translation network.

[0039] The second aspect of the present application provides an apparatus for obtaining tissue pixel labels based on a generative adversarial network, comprising:

[0040] a channel extraction unit configured to generate a plurality of CK-stained image blocks based on a whole slide image of a tissue slice under a CK staining style, and obtain a plurality of DAB channel images based on each CK-stained image block, wherein each CK-stained image block has the same predetermined size;

[0041] a coarse labeling unit configured to obtain a coarse-labeled image of each DAB channel image based on the Otsu method and morphological operations, wherein the coarse-labeled image is labeled with an epithelial region and a stromal region;

[0042] a labeling correction unit configured to correct the coarse-labeled image of each DAB channel image to obtain the tissue pixel label of each CK-stained image block, wherein the tissue pixel label is labeled with a normal gland region, a normal gland stroma region, a tumor region and a tumor stroma region;

[0043] a staining conversion unit configured to convert a CK-stained image block into a target staining style image block using a generative adversarial network model, and set the tissue pixel label of the CK-stained image block as the tissue pixel label of the target staining style image block, wherein the generative adversarial network model is configured to have the ability to convert an image under a CK staining style into an image under a target staining style, and the target staining style is an H&E staining style or a CD3 staining style.

[0044] The third aspect of the present application provides an apparatus for obtaining tissue pixel labels based on a generative adversarial network, comprising a memory and a processor;

[0045] the memory is configured to store a program;

[0046] The processor is configured to execute the program to implement each step of the method for obtaining the tissue pixel label.

[0047] The fourth aspect of the present application provides a storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements each step of the method for obtaining the tissue pixel label.

[0048] According to the technical solution described above, the present application first generates a plurality of CK staining image blocks based on the whole slide image of the tissue section under the CK staining style, and obtains a plurality of DAB channel images based on each CK staining image block, wherein each CK staining image block and each DAB channel image have the same preset size. Since DAB dye can selectively bind to target molecules and complexes, specifically, the color development of DAB channel images for epithelial keratin can be more obvious, making the contrast between the target and the background in the DAB channel image stronger, which is more conducive to the segmentation of the target. Then, based on the Otsu method and morphological operation, a rough annotation image of each DAB channel image is obtained, wherein the rough annotation image is annotated with epithelial and interstitial regions. It can be understood that the threshold method is used to segment the foreground and background in the DAB channel image, and the morphological operation is used to remove specific regions in the DAB channel image, and the epithelial and interstitial regions are annotated by filling the epithelial and interstitial regions with different colors. Next, the rough annotation image of each DAB channel image is corrected to obtain the tissue pixel label of each CK staining image block, wherein the tissue pixel label is annotated with normal gland region, normal gland interstitial region, tumor region and tumor interstitial region. It can be understood that CK staining not only specifically stains epithelium, but also stains apoptotic bodies and cell fragments, which should not be misidentified as tumors, so further correction is needed to ensure the accuracy of the annotation. Finally, a generative adversarial network model is used to convert the CK staining image block to a target staining style image block, and the tissue pixel label of the CK staining image block is set as the tissue pixel label of the target staining style image block, wherein the generative adversarial network model is configured to have the ability to convert images under the CK staining style to images under the target staining style, and the target staining style is H&E staining style or CD3 staining style. The present application avoids excessive dependence on pathological professional knowledge in the annotation process, greatly reduces the annotation time and effort of pathologists, and can directly convert the tissue pixel label under the CK staining style to the tissue pixel label under the target staining style, which can efficiently obtain the tissue pixel label of the tissue section under different staining styles. BRIEF DESCRIPTION OF DRAWINGS

[0049] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiments or prior art description. Obviously, the drawings described below only illustrate some of the embodiments of the present application, and not all the embodiments. Based on the embodiments of the present application, all other embodiments obtained by a person of ordinary skill in the art without creative labor should fall within the scope of the present application.

[0050] Figure 1 A schematic diagram of the method for acquiring tissue pixel labels based on a generative adversarial network disclosed in the embodiments of the present application is shown.

[0051] Figure 2 A CK-stained image block disclosed in the embodiments of the present application is shown.

[0052] Figure 3 A DAB channel image disclosed in the embodiments of the present application is shown.

[0053] Figure 4 An epithelial region disclosed in the embodiments of the present application is shown.

[0054] Figure 5 An interstitial region disclosed in the embodiments of the present application is shown.

[0055] Figure 6 A coarsely labeled image disclosed in the embodiments of the present application is shown.

[0056] Figure 7 A tissue pixel label disclosed in the embodiments of the present application is shown.

[0057] Figure 8 A CD3-stained image block generated by conversion disclosed in the embodiments of the present application is shown.

[0058] Figure 9 An H&E-stained image block generated by conversion disclosed in the embodiments of the present application is shown.

[0059] Figure 10 A schematic diagram of the device for acquiring tissue pixel labels based on a generative adversarial network disclosed in the embodiments of the present application is shown.

[0060] Figure 11 A schematic diagram of the equipment for acquiring tissue pixel labels based on a generative adversarial network disclosed in the embodiments of the present application is shown. DETAILED DESCRIPTION

[0061] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only some of the embodiments of the present application, and not all the embodiments. Based on the embodiments of the present application, all other embodiments obtained by a person of ordinary skill in the art without creative labor should fall within the scope of the present application.

[0062] The method for obtaining tissue pixel labels based on a generative adversarial network provided in the embodiments of the present application is described below. Referring to Figure 1 The method for obtaining tissue pixel labels based on a generative adversarial network provided in the embodiments of the present application can include the following steps:

[0063] In step S101, a plurality of CK-stained image patches are generated based on a whole slide image of a tissue slice in a CK-stained style, and a plurality of DAB channel images are obtained based on each CK-stained image patch.

[0064] CK staining is a staining technique for tumor sections, which labels antibodies against epithelial keratin, and is usually used to help identify and analyze tumors. In a CK-stained image, normal glands and tumor epithelial cells are usually displayed in brown, while non-target tissues are displayed in light blue. This strong color contrast facilitates subsequent image processing and analysis, and the precise pixel-level label of the epithelial region can be relatively easily obtained from the CK-stained image.

[0065] The whole slide image (WSI) in the CK-stained style usually contains hundreds of millions of pixels, which cannot be directly input into the model for calculation. Therefore, the whole slide image is divided into a plurality of CK-stained image patches (Patch) as input. The common size of the Patch in deep learning is 256x256 pixels, 512x512 pixels and 1024x1024 pixels.

[0066] In order to achieve more accurate segmentation, a color deconvolution algorithm can be used to perform staining separation on each CK-stained image patch, and only the DAB (3,3'-Diaminobenzidine, 3,3'-diaminobenzidine) channel image is retained. The principle of this operation is that H and DAB dyes will selectively bind to target molecules and complexes, and the color deconvolution algorithm can convert the RGB (Red-Green-Blue) color space into the H and DAB color space to separate different staining components. Specifically, the input of the color deconvolution operation is a CK-stained image patch in RGB format. According to the light absorption rate of different dyes, the contribution of the dye is calculated to separate different staining channels. In the embodiments of the present application, each CK-stained image patch and each DAB channel image has the same preset size. For example, the preset size can be 1024x1024 pixels. Figure 2 、 Figure 3 An example of a CK-stained image patch and its DAB channel image in an embodiment of the present application is shown.

[0067] In step S102, a coarse annotation image of each DAB channel image is obtained based on the Otsu method and morphological operations.

[0068] Otsu's Method is an algorithm for determining the threshold of image binarization segmentation, which is based on the entropy principle in information theory, and determines the optimal threshold by calculating the maximum entropy in the image histogram. The core idea of this method is to find a threshold that makes the gray value distribution within the two parts of the image divided into foreground and background as uniform as possible, while the gray value distribution between the two parts is as different as possible, so as to achieve the best segmentation effect.

[0069] Morphological Operations is a technique widely used in the field of image processing and machine vision, especially in the image feature extraction stage, which can be used to extract component information in the image, which is usually important for expressing and depicting the shape of the image. Morphological operations are based on concepts such as dilation and erosion in set theory, and more complex operations such as opening and closing are derived through the combination of these two basic operations.

[0070] The embodiments of the present application determine the foreground and background of each DAB channel image based on Otsu's method, and determine the epithelial region and interstitial region from it using morphological operations. Figure 4 、 Figure 5 The epithelial region and interstitial region of the DAB channel image corresponding to Figure 3 are exemplified, Figure 6 a coarse labeling image corresponding to the DAB channel image of Figure 3 is exemplified.

[0071] In step S103, the coarse labeling image of each DAB channel image is corrected to obtain the tissue pixel label of each CK staining image block.

[0072] Exemplarily, the correction can include content correction and color correction. Specifically, CK will also stain apoptotic bodies and cell fragments in addition to specifically staining epithelium, and these structures should not be misidentified as tumors, so they need to be corrected to ensure the accuracy of the labeling. The epithelial region includes tumor regions and normal gland regions, and the normal gland regions are identified and labeled as another color. The interstitial region is divided into tumor interstitium and normal gland interstitium, and the normal gland interstitium is identified and labeled as another color. After a series of corrections, the tissue pixel label can include the following four labels: normal gland region, normal gland interstitial region, tumor region, and tumor interstitial region. Figure 7 a coarse labeling image corresponding to the DAB channel image of Figure 2the CK staining image block is an abnormal area (i.e., tumor) in the tissue, so the tissue pixel label does not contain normal glandular area and normal glandular stroma area.

[0073] In step S104, the CK staining image block is converted into a target staining style image block by using a generative adversarial network model, and the tissue pixel label of the CK staining image block is set as the tissue pixel label of the target staining style image block.

[0074] It can be understood that the generative adversarial network model is configured to have the ability to convert images in the CK staining style into images in the target staining style, and the target staining style is the H&E staining style or the CD3 staining style. CK staining, H&E staining and CD3 staining are common staining techniques for tumor sections, although each staining method highlights different cell or tissue components, but they have common structural basis and morphological characteristics, and this commonality makes it possible for the generative adversarial network model to migrate the staining style from one staining style to another staining style image while keeping the structural information of the original image unchanged. Figure 8 、 Figure 9 The conversion of the CK staining image block into the CD3 staining image block and the H&E staining image block is illustrated respectively. Figure 2

[0075] In this step, the CK staining image block in the CK staining style is used to generate the target staining style image block in the target staining style, and the two are one-to-one corresponding. Although the styles are different, the contents are consistent, so the corresponding tissue pixel labels are also the same, that is, the image block after style migration and the original image block share the same tissue pixel label, so that the labeled data is used to the greatest extent, and the accuracy and consistency of the labeling are guaranteed, providing high-quality data support for the subsequent tissue segmentation task.

[0076] ​The present application first generates a plurality of CK staining image blocks based on the whole slide image of the tissue section in the CK staining style, and obtains a plurality of DAB channel images based on each CK staining image block, wherein each CK staining image block and each DAB channel image have the same preset size. Since DAB dye can selectively bind to target molecules and complexes, specifically, the color development of DAB channel images for epithelial keratin can be more obvious, making the contrast between the target and the background in the DAB channel image stronger, which is more conducive to the segmentation of the target. Then, based on the Otsu method and morphological operation, a rough annotation image of each DAB channel image is obtained, wherein the rough annotation image is annotated with epithelial and interstitial regions. It can be understood that the threshold method is used to segment the foreground and background in the DAB channel image, and the morphological operation is used to remove specific regions in the DAB channel image, and the epithelial and interstitial regions are annotated by filling them with different colors. Next, the rough annotation image of each DAB channel image is corrected to obtain a tissue pixel label of each CK staining image block, which is annotated with normal gland region, normal gland interstitial region, tumor region and tumor interstitial region. It can be understood that CK staining not only specifically stains epithelium, but also stains apoptotic bodies and cell fragments, which should not be misidentified as tumors, so further correction is needed to ensure the accuracy of the annotation. Finally, a generative adversarial network model is used to convert the CK staining image block into a target staining style image block, and the tissue pixel label of the CK staining image block is set as the tissue pixel label of the target staining style image block. The generative adversarial network model is configured to have the ability to convert images in the CK staining style to images in the target staining style, and the target staining style is H&E staining style or CD3 staining style. The present application avoids excessive dependence on pathological professional knowledge in the annotation process, greatly reduces the annotation time and effort of pathologists, and can directly convert the tissue pixel label in the CK staining style to the tissue pixel label in the target staining style, which can efficiently obtain the tissue pixel label of the tissue section in different staining styles.

[0077] In some embodiments of the present application, the process of step S101 of generating a plurality of CK staining image blocks based on the whole slide image of the tissue section in the CK staining style, and obtaining a plurality of DAB channel images based on each CK staining image block can include:

[0078] S1, the whole slide image is cropped to a preset size at a preset first magnification to obtain a cropped whole slide image.

[0079] Exemplarily, the first magnification can be 40 times, and the preset size can be 4096x4096 pixels.

[0080] S2, down-sampling the cropped whole slide image to a preset second multiple to obtain a plurality of CK-stained image blocks of a preset size.

[0081] Exemplarily, the second multiple can be 10 times, and the preset size can be 1024*1024 pixels.

[0082] S3, extracting DAB channel information of each CK-stained image block by using a color deconvolution algorithm to obtain a plurality of DAB channel images.

[0083] In some embodiments of the present application, the process of obtaining the coarse annotation image of each DAB channel image based on the Otsu method and morphological operation in step S102 can include:

[0084] S1, obtaining an epithelial region mask of each DAB channel image based on the Otsu method and morphological opening operation.

[0085] S2, obtaining a stroma mask of each DAB channel image based on the Otsu method and morphological opening operation.

[0086] S3, generating a coarse annotation image of each DAB channel image based on each DAB channel image, the epithelial region mask and the stroma mask of the DAB channel image.

[0087] In some embodiments of the present application, the process of obtaining the epithelial region mask of each DAB channel image based on the Otsu method and morphological opening operation in S1 can include:

[0088] S11, performing smoothing processing on each DAB channel image by using a Gaussian filter with a preset first coefficient value to obtain a first DAB channel image.

[0089] Exemplarily, the first coefficient value can be 2.

[0090] S12, segmenting the epithelial region and other regions of each first DAB channel image based on a threshold value determined by the Otsu method to obtain a first mask image.

[0091] S13, removing a first invalid region in each first mask image by using a morphological opening operation to obtain a second mask image.

[0092] The first invalid region includes a region with an adenoid mask area less than a preset first pixel number and a region with a background part area less than a preset second pixel number.

[0093] Exemplarily, the first pixel number can be 200 pixels, and the second pixel number can be 60 pixels.

[0094] S14, performing an inversion operation on each second mask image to obtain an epithelial region mask of each DAB channel image.

[0095] Specifically, the image mask is usually a binary or Boolean image with the same size as the original image, wherein the selected region is marked as 1 (or True), and the remaining region is marked as 0 (or False). In the embodiments of the present application, the selected region of the mask is marked as 1, and the remaining region is marked as 0. Therefore, in the inversion operation of S14, the non-empty region in each second mask image is marked as 0, and the empty region is marked as 1.

[0096] In some embodiments of the present application, the process of obtaining the interstitial mask of each DAB channel image based on the Otsu method and the morphological opening operation in S2 can include:

[0097] S21, setting the pixels corresponding to the tumor region mask with a value of 1 in each DAB channel image to a preset third pixel value to obtain each third mask image.

[0098] Exemplarily, the third pixel value can be 255, that is, the tumor part in the DAB channel image is changed to a white background.

[0099] S22, performing smoothing processing on each third mask image by using a Gaussian filter with a preset second coefficient value to obtain each fourth mask image.

[0100] Exemplarily, the second coefficient value can be 2.

[0101] S23, segmenting the background and foreground of each fourth mask image based on the threshold value determined by the Otsu method.

[0102] S24, removing the second invalid region in each fourth mask image by using the morphological opening operation to obtain each fifth mask image.

[0103] The second invalid region includes a region with an interstitial mask area less than a preset third pixel number and a background part area less than a preset fourth pixel number.

[0104] Exemplarily, the third pixel number and the fourth pixel number are both 200 pixels.

[0105] S25, performing an inversion operation on each fifth mask image to obtain an interstitial mask of each DAB channel image.

[0106] In some embodiments of the present application, the process of generating the coarse annotation image of each DAB channel image based on each DAB channel image, the epithelial region mask and the interstitial mask of the DAB channel image in S3 can include:

[0107] For each DAB channel image:

[0108] S31, a three-channel image with the same size as the DAB channel image is created, and each pixel point in the three-channel image is initialized as a preset fourth pixel value.

[0109] Exemplarily, the fourth pixel value can be white (R=255, G=0255, B=255).

[0110] S32, the pixel points corresponding to the value 1 in the epithelial region mask in the three-channel image are set as a preset first pixel value.

[0111] Exemplarily, the first pixel value can be red (R=230, G=0, B=18).

[0112] S33, the pixel points corresponding to the value 1 in the interstitial mask in the three-channel image are set as a preset second pixel value.

[0113] Exemplarily, the second pixel value can be yellow (R=255, G=241, B=0).

[0114] In some embodiments of the present application, the process of correcting the coarse labeling image of each DAB channel image to obtain the tissue pixel label of each CK staining image block in step S103 can include:

[0115] S1, the apoptotic bodies and cell debris in the epithelial region of the coarse labeling image of each DAB channel image are removed to obtain a first labeling image of each DAB channel image.

[0116] S2, the normal gland region and tumor region in the epithelial region of the first labeling image of each DAB channel image are identified and labeled, and the normal gland stroma region and tumor stroma region in the tumor stroma region of the first labeling image of each DAB channel image are identified and labeled, to obtain the tissue pixel label of the tissue section under the CK staining style.

[0117] The gland is divided into tumor and normal gland, so the tumor region labeled in the coarse labeling image can contain a normal gland part, which needs to be identified from the normal gland part and labeled as other colors (such as blue: R=0, G=160, B=233).

[0118] In some embodiments of the present application, the generative adversarial network model in step S104 is a contrastive learning unpaired image-to-image translation network (Contrastive Learning for Unpaired Image-to-Image Translation, CUT).

[0119] The generative adversarial network model mainly consists of a generator and a discriminator. The generator consists of an encoder and a decoder and is used to generate target style pseudo-color images. The discriminator discriminates between true and false. The generator consists of an encoder and a decoder, and the discriminator uses PatchGAN, that is, the input image is divided into 70x70 overlapping image blocks, and the probability of each image block being true or false is determined, and then the overall discrimination is obtained according to the image block discrimination. In order to keep the generated image as consistent as possible with the original image content, contrastive learning is introduced. After the input image x is input into the encoder, a feature map stack is generated. If a feature map layer is taken, a certain spatial position can be represented as an image block of the input image, and a deeper layer corresponds to a larger image block. Suppose the number of layers is L, and then the feature map H l is obtained through an MLP, and the feature space obtained after operation is represented as:

[0120]

[0121] represents the output of the encoder in the generator at the l-th (l e {1, 2, …, L}) layer.

[0122] Suppose the number of spatial positions in the l-th layer is S l , and the s-th (s e {1, …, S l}) spatial position is represented. Then the feature of the s-th spatial position in the l-th layer is represented as wherein represents that the feature is a vector in a C l -dimensional space, C l is the number of channels in the l-th layer, and other features are represented as wherein By analogy, the image output by the generator is represented as

[0123]

[0124] G(x) represents the output of the generator. The InfoNCE loss function in the contrastive learning is used here, and the formula is:

[0125]

[0126] wherein, v, v + , v - represent query, positive sample and negative sample, v - represents the (N-1)th negative sample, wherein the query refers to the image block selected from the output image of the generator, the positive sample refers to the image block at the corresponding position of the input image, and the negative sample refers to the image block at the non-corresponding position. τ is a temperature hyperparameter.

[0127] ​If a Patch is selected in the generator output, the Patch is considered as query, the Patch with the same position in the generator input is the positive sample, and the Patch with different position is the negative sample, thus the following can be obtained:

[0128]

[0129] The embodiment of the present application adopts an adversarial loss function to constrain the encoder and encourage the generator to generate images of a visual style similar to the Y domain, and the adversarial loss function is represented as follows:

[0130]

[0131] Wherein, x is the input X domain image, and y is the target Y domain image. D(G(x)) represents that the discriminator discriminates whether the generated X domain image is true or false, and thus for In this part, the goal of the generator is to minimize the value of this part to deceive the discriminator, and the goal of the discriminator is to maximize the loss.

[0132] In order to realize the maximization of the input and output information of the generator, the idea of contrastive learning is introduced, and a contrastive loss function InfoNCE function is used for constraint, and the total loss function is represented as follows:

[0133]

[0134] The embodiment of the present application converts the pixel labels of the tissue under CK staining into the pixel labels of the tissue under multiple staining styles (such as H&E staining and CD3 staining) through the generative adversarial network model, and can efficiently obtain the pixel labels of the tissue under different staining styles.

[0135] The training phase of the generative adversarial network model needs to prepare sufficient training data and test data, and the embodiment of the present application cuts the WSI of CK staining and the WSI of target domain style staining (H&E, CD3) into 512*512 pixel image blocks, the image block of CK staining is used as the training set A of the X domain, the image block of target style staining is used as the training set B of the Y domain, and the test data is the 1024*1024 pixel image block used for generating the coarse annotation of CK staining.

[0136] The device for obtaining the pixel label of the tissue based on the generative adversarial network provided by the embodiment of the present application is described below, and the device for obtaining the pixel label of the tissue based on the generative adversarial network described below can be correspondingly referred to the method for obtaining the pixel label of the tissue based on the generative adversarial network described above.

[0137] Please refer to Figure 10 The device for obtaining the pixel label of the tissue based on the generative adversarial network provided by the embodiment of the present application can include:

[0138] The channel extraction unit 21 is configured to generate a plurality of CK staining image blocks based on the whole slide image of the tissue slice in the CK staining style, and obtain a plurality of DAB channel images based on each CK staining image block, wherein each CK staining image block has a same preset size;

[0139] The rough labeling unit 22 is configured to obtain a rough labeling image of each DAB channel image based on the Otsu method and morphological operation, wherein the rough labeling image is labeled with an epithelial region and an interstitial region;

[0140] The labeling correction unit 23 is configured to correct the rough labeling image of each DAB channel image to obtain a tissue pixel label of each CK staining image block, wherein the tissue pixel label is labeled with a normal gland region, a normal gland interstitial region, a tumor region and a tumor interstitial region;

[0141] The staining conversion unit 24 is configured to convert the CK staining image block into a target staining style image block by using a generative adversarial network model, and set the tissue pixel label of the CK staining image block as the tissue pixel label of the target staining style image block, wherein the generative adversarial network model is configured to have the ability to convert an image in the CK staining style into an image in a target staining style, and the target staining style is an H&E staining style or a CD3 staining style.

[0142] The device for obtaining tissue pixel labels based on a generative adversarial network provided in the embodiments of the present application can be applied to an apparatus for obtaining tissue pixel labels based on a generative adversarial network, such as a computer and the like. Optionally, Figure 11 The hardware structure block diagram of the apparatus for obtaining tissue pixel labels based on a generative adversarial network is shown, and Figure 3 The hardware structure of the apparatus for obtaining tissue pixel labels based on a generative adversarial network can include at least one processor 31, at least one communication interface 32, at least one memory 33 and at least one communication bus 34.

[0143] In the embodiments of the present application, the number of the processor 31, the communication interface 32, the memory 33 and the communication bus 34 is at least one, and the processor 31, the communication interface 32 and the memory 33 complete the communication among each other through the communication bus 34;

[0144] The processor 31 can be a central processing unit CPU, or a specific integrated circuit ASIC (Application Specific Integrated Circuit), or one or more integrated circuits configured to implement the embodiments of the present application, etc.

[0145] The memory 33 can include a high-speed RAM memory and can also include a non-volatile memory, such as at least one disk memory;

[0146] The memory 33 stores a program, and the processor 31 can call the program stored in the memory 33, and the program is used for:

[0147] Based on the whole slide image of the tissue slice in the CK staining style, a plurality of CK staining image blocks are generated, and based on each CK staining image block, a plurality of DAB channel images are obtained, wherein each CK staining image block has the same preset size;

[0148] Based on the Otsu method and morphological operation, a coarse annotation image of each DAB channel image is obtained, and the coarse annotation image is annotated with epithelial region and interstitial region;

[0149] The coarse annotation image of each DAB channel image is corrected to obtain a tissue pixel label of each CK staining image block, and the tissue pixel label is annotated with normal gland region, normal gland interstitial region, tumor region and tumor interstitial region;

[0150] The CK staining image block is converted into a target staining style image block by using a generative adversarial network model, and the tissue pixel label of the CK staining image block is set as the tissue pixel label of the target staining style image block, and the generative adversarial network model is configured to have the ability to convert images in the CK staining style into images in the target staining style, and the target staining style is H&E staining style or CD3 staining style.

[0151] Optionally, the refinement function and the extension function of the program can refer to the description above.

[0152] The embodiments of the present application also provide a storage medium which can store a program suitable for processor execution, and the program is used for:

[0153] Based on the whole slide image of the tissue slice in the CK staining style, a plurality of CK staining image blocks are generated, and based on each CK staining image block, a plurality of DAB channel images are obtained, wherein each CK staining image block has the same preset size;

[0154] Based on the Otsu method and morphological operation, a coarse annotation image of each DAB channel image is obtained, and the coarse annotation image is annotated with epithelial region and interstitial region;

[0155] The coarse annotation image of each DAB channel image is corrected to obtain a tissue pixel label of each CK staining image block, and the tissue pixel label is annotated with normal gland region, normal gland interstitial region, tumor region and tumor interstitial region;

[0156] inputting the CK staining image block into the trained style transfer model to obtain a target staining style image block corresponding to the CK staining image block, and setting a tissue pixel label of the CK staining image block as a tissue pixel label of the target staining style image block, the generative adversarial network model being configured to have the ability to convert an image in a CK staining style into an image in a target staining style, the target staining style being an H&E staining style or a CD3 staining style.

[0157] Optionally, the refinement function and the expansion function of the program can refer to the description above.

[0158] In summary:

[0159] The present application first generates a plurality of CK staining image blocks based on the whole slide image of the tissue section in the CK staining style, and obtains a plurality of DAB channel images based on each CK staining image block, wherein each CK staining image block and each DAB channel image have the same preset size. Since DAB dye can selectively bind to target molecules and complexes, specifically, the color development of the DAB channel image for epithelial keratin is more obvious, making the contrast between the target and the background in the DAB channel image stronger, which is more conducive to the segmentation of the target. Then, based on the Otsu method and morphological operation, a rough annotation image of each DAB channel image is obtained, wherein the rough annotation image is annotated with epithelial and interstitial regions. It can be understood that thresholding is used to segment the foreground and background in the DAB channel image, and morphological operation is used to remove specific regions in the DAB channel image, and the epithelial and interstitial regions are annotated by filling them with different colors. Then, the rough annotation image of each DAB channel image is corrected to obtain a tissue pixel label of each CK staining image block, wherein the tissue pixel label is annotated with normal gland region, normal gland stroma region, tumor region and tumor stroma region. It can be understood that CK staining not only specifically stains epithelium, but also stains apoptotic bodies and cell fragments, which should not be misidentified as tumors, so further correction is needed to ensure the accuracy of the annotation. Finally, a generative adversarial network model is used to convert the CK staining image block into a target staining style image block, and the tissue pixel label of the CK staining image block is set as the tissue pixel label of the target staining style image block, the generative adversarial network model being configured to have the ability to convert an image in a CK staining style into an image in a target staining style, the target staining style being an H&E staining style or a CD3 staining style. The present application avoids excessive dependence on pathological professional knowledge in the annotation process, greatly reduces the annotation time and effort of pathologists, and can directly convert the tissue pixel label in the CK staining style into the tissue pixel label in the target staining style, which can efficiently obtain the tissue pixel label of the tissue section in different staining styles.

[0160] Finally, it should be noted that the terminology used herein, such as first and second, is merely used for the convenience of the reader and is not intended to, and should not, be taken to imply or provide any actual limitation on the scope of the application. Moreover, the term "comprising" or "comprises" as used in this specification is taken to mean one or more of the stated features or steps are present in the process, method, article, or apparatus, but not excluding others. The term "comprising" therefore does not exclude that there are yet other features or steps involved in addition to those enumerated. Furthermore, the term "comprising" as used in this specification means "including" or "comprising" one or more stated features or steps but not excluding others.

[0161] The various embodiments described in this specification are presented by way of example, and are not intended to limit the scope of the application. Each embodiment is presented in a way that emphasizes the practical application of the embodiments, and the embodiments can be combined according to the needs of the designer. The same reference numerals in different embodiments designate the same or similar elements.

[0162] The above description of disclosed embodiments provides enabling disclosure sufficient for one of ordinary skill in the art to practice the application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and generic principles defined herein can be applied to other embodiments without departing from the spirit or scope of the application. Thus, the present application is not to be limited to the embodiments shown herein but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for obtaining organizational pixel labels based on generative adversarial networks, characterized in that, include: Multiple CK-stained image blocks are generated based on the whole slide image of tissue sections in CK staining style, and multiple DAB channel images are obtained based on each CK-stained image block. Each CK-stained image block and each DAB channel image has the same preset size. Based on Otsu's method and morphological operations, coarsely annotated images of each DAB channel are obtained, wherein the coarsely annotated images are labeled with epithelial and stromal regions. The coarsely annotated images of each DAB channel are corrected to obtain tissue pixel labels for each CK-stained image block. The tissue pixel labels are labeled with normal glandular regions, normal glandular stroma regions, tumor regions, and tumor stroma regions. A generative adversarial network (GAN) model is used to convert CK-stained image patches into target-staining style image patches, and the tissue pixel labels of the CK-stained image patches are set as the tissue pixel labels of the target-staining style image patches. The GAN model is configured to have the ability to convert images under the CK staining style into images under the target staining style, wherein the target staining style is H&E staining style or CD3 staining style. The process of obtaining coarsely annotated images for each DAB channel based on Otsu's method and morphological operations includes: Based on Otsu's method and morphological opening operation, the epithelial region mask of each DAB channel image is obtained; Based on Otsu's method and morphological opening operation, the interstitial mask of each DAB channel image is obtained; Based on each DAB channel image, the epithelial region mask and the mesenchymal mask of the DAB channel image, a coarsely annotated image of the DAB channel image is generated. The process of obtaining the epithelial region mask for each DAB channel image based on Otsu's method and morphological opening operation includes: The images of each DAB channel are smoothed by using a Gaussian filter with a preset first coefficient value to obtain the images of each first DAB channel. The epithelial region and other regions of each first DAB channel image are segmented based on the threshold determined by the Otsu method to obtain the first mask image. The first invalid region in each first mask image is removed by morphological opening operation to obtain each second mask image; Invert each second mask image to obtain the epithelial region mask for each DAB channel image; The first invalid region includes a region where the area of ​​the gland mask is less than a preset number of first pixels and a region where the area of ​​the background portion is less than a preset number of second pixels. The process of obtaining the interstitial mask for each DAB channel image based on Otsu's method and morphological opening operation includes: The pixels in each DAB channel image that correspond to the tumor region mask with a value of 1 are set to the preset third pixel value to obtain each third mask image; Each third mask image is smoothed using a Gaussian filter with a preset second coefficient value to obtain each fourth mask image; The background and foreground of each fourth mask image are segmented based on the threshold determined by Otsu's method; The second invalid region in each fourth mask image is removed by morphological opening operation to obtain each fifth mask image; Invert each fifth mask image to obtain the interstitial mask for each DAB channel image; The second invalid region includes regions where the area of ​​the interstitial mask is less than a preset number of third pixels and regions where the area of ​​the background portion is less than a preset number of fourth pixels.

2. The method according to claim 1, characterized in that, The process of generating multiple CK-stained image blocks from a whole slide image of a tissue section in CK staining style, and obtaining multiple DAB channel images based on each CK-stained image block, includes: The whole glass slide image is cropped to a preset size at a preset first multiple to obtain the cropped whole glass slide image; The cropped full-slide image is downsampled to a preset second multiple to obtain multiple CK staining image blocks of preset size; The DAB channel information of each CK-colored image block is extracted using the color deconvolution algorithm to obtain multiple DAB channel images.

3. The method according to claim 2, characterized in that, The process of generating a coarsely annotated image of the DAB channel image based on each DAB channel image, the epithelial region mask and the mesenchymal mask of the DAB channel image includes: For each DAB channel image: Create a new three-channel image with the same size as the DAB channel image, and initialize each pixel in the three-channel image to a preset fourth pixel value; The pixels in the three-channel image that correspond to a value of 1 in the epithelial region mask are set to a preset first pixel value, and the pixels in the three-channel image that correspond to a value of 1 in the mesenchymal mask are set to a preset second pixel value. Set the three-channel image as the coarse-annotated image of the DAB channel image.

4. The method according to claim 1, characterized in that, The process of correcting the coarsely annotated images of each DAB channel to obtain the tissue pixel labels of each CK-stained image block includes: Apoptotic bodies and cell debris were removed from the epithelial region of the coarsely annotated image of each DAB channel to obtain the first annotated image of each DAB channel. The normal glandular region and tumor region in the epithelial region of the first labeled image of each DAB channel image are identified and labeled, as are the normal glandular stroma region and tumor stroma region in the stroma region of the first labeled image of each DAB channel image, to obtain the tissue pixel label of the tissue section under CK staining style.

5. An apparatus for implementing the method for acquiring organizational pixel tags based on generative adversarial networks as described in any one of claims 1 to 4, characterized in that, include: The channel extraction unit is used to generate multiple CK-stained image blocks based on the whole slide image of the tissue section in CK staining style, and to obtain multiple DAB channel images based on each CK-stained image block, wherein each CK-stained image block has the same preset size; The coarse annotation unit, based on Otsu's method and morphological operations, obtains coarsely annotated images of each DAB channel image, wherein the coarsely annotated images are annotated with epithelial regions and mesenchymal regions; The annotation correction unit is used to correct the coarse annotation image of each DAB channel image to obtain the tissue pixel label of each CK stained image block. The tissue pixel label is labeled with normal glandular region, normal glandular stroma region, tumor region and tumor stroma region. A coloring conversion unit is used to convert CK-colored image patches into target-colored style image patches using a generative adversarial network model, and to set the tissue pixel labels of the CK-colored image patches as the tissue pixel labels of the target-colored style image patches. The generative adversarial network model is configured to have the ability to convert images under the CK-colored style into images under the target-colored style, wherein the target-colored style is the H&E-colored style or the CD3-colored style.

6. A device for acquiring organizational pixel tags based on generative adversarial networks, characterized in that, include: Memory and processor; The memory is used to store programs; The processor is configured to execute the program to implement each step of the method for obtaining organization pixel tags as described in any one of claims 1 to 4.

7. A storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements each step of the method for obtaining tissue pixel tags as described in any one of claims 1 to 4.

Images