Pathological image defect repair method, device, electronic device and storage medium

Abstract

The present invention provides a method, device, electronic device, and storage medium for repairing defects in pathological images. The method comprises: inputting a pathological image into a defect detector, which divides the defect area of ​​the pathological image into sections to obtain a slice image of the defect area; inputting the slice image of the defect area into a defect classifier, which classifies each defect in the slice image of the defect area to obtain the type of each defect in the slice image of the defect area; and, if any defect is a repairable defect, inputting the slice image of any defect and its corresponding defect saliency map into a first image generator, which performs defect repair on the slice image of any defect based on the defect location information contained in the defect saliency map to generate a corresponding normal image. The present invention can accurately detect and repair different types of defects in pathological images, and ensure the stability of microstructures after defect repair.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to a method, device, electronic device and storage medium for repairing pathological image defects. Background Art

[0002] Whole Slide Imaging (WSI) refers to images acquired by medical imaging technology that depict tissue and organ pathology. Pathology images can be represented digitally and have been widely used in various scientific research fields in recent years.

[0003] Before digitization, WSIs need to go through processing steps such as sample cutting and staining. If not handled properly, defects may appear in the WSI, such as squeezed, wrinkled, or torn tissue, or dark spots, linear objects, or uneven staining. Currently, pathologists rely on their personal experience to identify defects in pathology images and discard defective images. However, this method relies on the professional level of pathologists and is highly subjective, which may lead to misjudgments. Moreover, the discarded defective images may contain critical information, making it impossible to fully obtain the information of the pathology images. Summary of the Invention

[0004] The present invention provides a pathological image defect repair method, device, electronic device and storage medium to solve the defects in the prior art that pathological image defects cannot be accurately detected and pathological image information cannot be fully obtained.

[0005] The present invention provides a method for repairing pathological image defects, comprising:

[0006] Determine pathological images;

[0007] Inputting the pathological image into a defect detector, the defect detector divides the defect area of ​​the pathological image to obtain a defect area slice image; the defect detector is trained based on a sample pathological image and a corresponding sample defect area slice image;

[0008] Inputting the defect area slice image into a defect classifier, the defect classifier classifies each defect in the defect area slice image to obtain the type of each defect in the defect area slice image; the defect classifier is trained based on the sample defect area slice image and the corresponding sample defect type;

[0009] If any defect is a repairable defect, the slice image of the defect and the corresponding defect saliency map are input into a first image generator, and the first image generator performs defect repair on the slice image of the defect based on defect position information contained in the defect saliency map to generate a corresponding normal image;

[0010] The first image generator is obtained by combining the second image generator, the first image discriminator and the second image discriminator through cyclic adversarial training.

[0011] According to a pathological image defect repair method provided by the present invention, the loss function of the cyclic adversarial training includes a staining consistency loss function and a cyclic loss function;

[0012] The coloring consistency loss function is used to characterize the loss value of coloring style consistency, wherein the coloring style consistency includes the coloring style consistency between the sample defect image and the sample defect repair image, the coloring style consistency between the sample defect enhanced image and the first sample enhanced repair image, the coloring style consistency between the sample defect image and the sample defect enhanced image, and the coloring style consistency between the sample normal image and the sample normal enhanced image;

[0013] The cyclic loss function is used to characterize the loss value of the cyclic training of the first image generator, the second image generator, the first image discriminator, and the second image discriminator.

[0014] According to a pathological image defect repair method provided by the present invention, the first image generator is trained based on the following steps:

[0015] Inputting the sample defect image and the corresponding sample saliency map into a first initial generator of the first image generator, and having the first initial generator perform defect repair on the sample defect image to obtain the sample defect repaired image;

[0016] Inputting the sample defect repair image into the second initial generator of the second image generator, and having the second initial generator generate defects on the sample defect repair image to obtain a first fake sample defect image;

[0017] Inputting the sample defect-enhanced image and the corresponding sample saliency map into the first initial generator, and having the first initial generator perform defect repair on the sample defect-enhanced image to obtain a first sample enhanced and repaired image;

[0018] Inputting the first sample enhanced restoration image into the second initial generator, and having the second initial generator generate defects on the sample enhanced restoration image to obtain a first fake sample defect enhanced image;

[0019] Inputting the sample normal image and the corresponding sample saliency map into the second initial generator, and having the second initial generator generate defects on the sample normal image to obtain a second fake sample defect image;

[0020] Inputting the second fake sample defect image into the first initial generator, and having the first initial generator perform defect repair on the second fake sample defect image to obtain a fake sample normal image;

[0021] Inputting the sample normal enhanced image and the corresponding sample saliency map into the second initial generator, and having the second initial generator generate defects on the sample normal enhanced image to obtain a second fake sample defect enhanced image;

[0022] Inputting the second fake sample defect-enhanced image into the first initial generator, and having the first initial generator perform defect repair on the second fake sample defect-enhanced image to obtain a second sample enhanced and repaired image;

[0023] Determining a loss value of the dyeing style consistency based on the sample defect image, the sample defect repaired image, the sample defect enhanced image, the first sample enhanced repaired image, the sample normal image, and the sample normal enhanced image;

[0024] Determining a loss value of the cyclic training based on the sample defect image, the first fake sample defect image, the sample defect-enhanced image, the first fake sample defect-enhanced image, the sample normal image, the fake sample normal image, the sample normal-enhanced image, and the second sample enhanced repaired image;

[0025] Training is performed based on the loss value of the dyeing style consistency and the loss value of the cyclic training to obtain the first image generator.

[0026] According to a pathological image defect restoration method provided by the present invention, determining the loss value of the staining style consistency based on the sample defect image, the sample defect restoration image, the sample defect enhanced image, the first sample enhanced restoration image, the sample normal image, and the sample normal enhanced image, includes:

[0027] determining a first staining loss value based on a pixel-level distance between a non-defective area in the sample defect image and a non-defective area in the sample defect-repaired image, and a pixel-level distance between a non-defective area in the sample defect-enhanced image and a non-defective area in the first sample enhanced-repaired image;

[0028] determining a second staining loss value based on a staining difference between a non-defect area in the sample defect image and a non-defect area in the sample defect-enhanced image, and a staining difference between a non-defect area in the sample normal image and a non-defect area in the sample normal-enhanced image;

[0029] Based on the first coloring loss value and the second coloring loss value, a loss value of the coloring style consistency is determined.

[0030] According to a pathological image defect repair method provided by the present invention, the loss value of the dyeing style consistency is determined based on the following formula:

[0031]

[0032]

[0033]

[0034] in, represents the loss value of the consistency of the dyeing style, represents the first staining loss value, represents the second staining loss value, x s ′ represents the non-defective area in the sample defect image, represents the non-defective area in the sample defect repair image, represents the non-defect area in the sample defect-enhanced image, represents the non-defective area in the first sample enhanced restoration image, x′ t represents the non-defective area in the normal image of the sample, represents the non-defective area in the normal enhanced image of the sample, γ sc1 and γ sc2 is the weight parameter.

[0035] According to a pathological image defect repair method provided by the present invention, determining the loss value of the cyclic training based on the sample defect image, the first false sample defect image, the sample defect enhanced image, the first false sample defect enhanced image, the sample normal image, the false sample normal image, the sample normal enhanced image, and the second sample enhanced repaired image includes:

[0036] Determining a first cycle loss value of the cycle training based on the sample defect image, the first fake sample defect image, the sample defect-enhanced image, and the first fake sample defect-enhanced image;

[0037] A second cycle loss value of the cycle training is determined based on the sample normal image, the false sample normal image, the sample normal enhanced image, and the second sample enhanced restoration image.

[0038] According to a pathological image defect repair method provided by the present invention, the first cycle loss value is determined based on the following formula:

[0039]

[0040] in, represents the first cycle loss value, x s represents the sample defect image, represents the first false sample defect image, represents the sample defect enhanced image, represents the first false sample defect enhanced image;

[0041] The second cycle loss value is determined based on the following formula:

[0042]

[0043] in, represents the second cycle loss value, x t represents the sample normal image, represents the normal image of the false sample, represents the sample normal enhanced image, represents the second sample enhanced restoration image.

[0044] The present invention also provides a pathological image defect repair device, comprising:

[0045] a determining unit, configured to determine a pathological image;

[0046] a detection unit, configured to input the pathological image into a defect detector, and have the defect detector divide the defect area of ​​the pathological image to obtain a defect area slice image; the defect detector is trained based on a sample pathological image and a corresponding sample defect area slice image;

[0047] a classification unit, configured to input the defect region slice image into a defect classifier, and have the defect classifier classify each defect in the defect region slice image to obtain a type of each defect in the defect region slice image; the defect classifier is trained based on the sample defect region slice image and the corresponding sample defect type;

[0048] a repairing unit configured to, when any defect is a repairable defect, input a slice image of the defect and a corresponding defect saliency map to a first image generator, and have the first image generator perform defect repair on the slice image of the defect based on defect position information contained in the defect saliency map to generate a corresponding normal image;

[0049] The first image generator is obtained by combining the second image generator, the first image discriminator and the second image discriminator through cyclic adversarial training.

[0050] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, any of the above-described pathological image defect repair methods is implemented.

[0051] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements any of the above-mentioned pathological image defect repair methods.

[0052] The present invention also provides a computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the computer program implements any of the above-mentioned pathological image defect repair methods.

[0053] The pathological image defect repair method, apparatus, electronic device, and storage medium provided by the present invention utilize a defect detector, a defect classifier, and a first image generator to accurately detect and repair different types of defects in pathological images, ensuring that the repaired normal image retains the original information of the pathological image. Furthermore, when repairing slice images corresponding to repairable defects, a defect saliency map containing defect location information is introduced to guide defect repair and ensure the stability of microstructures after defect repair. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] In order to more clearly illustrate the technical solutions in the present invention or the prior art, a brief introduction is given below to the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0055] Figure 1 Schematic diagram of the process of the pathological image defect repair method provided by the present invention;

[0056] Figure 2 Schematic diagram of a process for repairing pathological image defects provided by the present invention;

[0057] Figure 3 1 is a schematic diagram of the training process of the first image generator provided by the present invention;

[0058] Figure 4 Schematic diagram of the comparison of pathological images before and after restoration provided by the present invention;

[0059] Figure 5 Schematic diagram of the structure of the pathological image defect repair device provided by the present invention;

[0060] Figure 6It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0061] To make the objectives, technical solutions, and advantages of the present invention more clear, the technical solutions of the present invention will be clearly and completely described below in conjunction with the accompanying drawings. Obviously, the embodiments described are only some of the embodiments of the present invention, not all of them. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts shall fall within the scope of protection of the present invention.

[0062] The present invention provides a method for repairing pathological image defects. Figure 1 FIG. 1 is a flow chart of the pathological image defect repair method provided by the present invention, such as Figure 1 As shown, the method includes the following steps:

[0063] Step 110: Determine the pathological image.

[0064] Here, a pathological image is an image that requires defect detection and, if defects are present, defect repair. It is understood that a pathological image may or may not contain defects. If defects are present in a pathological image, the defect type may include repairable defects and unrepairable defects. Repairable defects can be understood as minor defects, while unrepairable defects can be understood as severe defects.

[0065] Step 120: Input the pathological image into the defect detector, which divides the defect area of ​​the pathological image to obtain a defect area slice image; the defect detector is trained based on the sample pathological image and the corresponding sample defect area slice image.

[0066] Specifically, the defect detector is used to detect whether there are defects in the pathological image, and if there are defects, it divides the defect area and generates a bounding box containing the defect. Considering that pathological images usually have a high resolution, and high-resolution pathological images are usually relatively large, the pathological image can be divided into multiple slice images of a first size, and each slice image of the first size is input into the defect detector in batches. The defect detector divides the defect area in each slice image and generates a bounding box containing the defect. The position information of the slice image is used to integrate the defect area across slices, thereby obtaining the defect distribution of the entire pathological image. Among them, the defect detector can be constructed based on YOLO5 (You Only Look Once Version 5).

[0067] Step 130: Input the defect area slice image into a defect classifier, and the defect classifier classifies each defect in the defect area slice image to obtain the type of each defect in the defect area slice image; the defect classifier is trained based on sample defect area slice images and corresponding sample defect types.

[0068] Specifically, the defect classifier is used to classify each defect in the defect region slice image to obtain the type of each defect in the defect region slice image.

[0069] Optionally, the types of defects may include normal defects (ie, no defects), repairable defects, and non-repairable defects, depending on the severity, transparency, and reliability of the recovery of the concealed information of the defects.

[0070] To further accurately classify each defect, an embodiment of the present invention can divide the defect area slice image into multiple slice images of a second size, and input each of the second-sized slice images into a defect classifier in batches, so that the defect classifier classifies the defects in each slice image. The second size is smaller than the first size.

[0071] Step 140: If any defect is a repairable defect, the slice image of the defect and the corresponding defect saliency map are input into a first image generator. The first image generator performs defect repair on the slice image of the defect based on the defect location information contained in the defect saliency map to generate a corresponding normal image.

[0072] The first image generator is obtained by combining the second image generator, the first image discriminator, and the second image discriminator through cyclic adversarial training.

[0073] Specifically, after determining the type of each defect in step 130, since normal defects do not require repair and irreparable defects cannot be repaired, this embodiment of the present invention repairs repairable defects. That is, if any defect is a repairable defect, defect repair is performed on the slice image of the defect.

[0074] In addition, considering that the defective slice image contains tiny pathological structures, in order to ensure the stability of the image after defect repair, the embodiment of the present invention inputs the defective slice image into the first image generator at the same time as the corresponding defect saliency map is input into the first image generator. The defect position information contained in the defect saliency map guides the first image generator to repair the defective slice image, so that the obtained normal image can retain the tiny structure information in the original image, thereby ensuring the stability of the tiny structure after defect repair.

[0075] Among them, the first image generator, combined with the second image generator, the first image discriminator and the second image discriminator, is obtained through cyclic adversarial training, that is, it can be understood that the first image generator, the second image generator, the first image discriminator and the second image discriminator constitute a generative adversarial network (Cycle-Consistent Generative Adversarial Networks, CycleGAN). The first image generator is used to generate a flawless normal image based on the defective image mapping, the second image generator is used to generate a realistic defective image based on the flawless normal image, the first image discriminator is used to determine whether the input image is a real flawless image, and the second image discriminator is used to determine whether the input image is a real flawless normal image. The generative adversarial network composed of the first image generator, the second image generator, the first image discriminator and the second image discriminator in the embodiment of the present invention can be simply referred to as AR-CycleGAN.

[0076] like Figure 2 As shown, the embodiment of the present invention realizes a complete process of defect detection, defect classification and defect repair for different types of defects in pathological images through a defect detector, a defect classifier and a defect repairer. Specifically, the WSI is input into the defect detector (AR-Detector), and the AR-Detector divides the defect area of ​​the pathological image to obtain a slice image of the defect area; then, the slice image of the defect area is input into the defect classifier (AR-Classifier), and the defect classifier (AR-Classifier) ​​classifies each defect in the slice image of the defect area to obtain the type of each defect in the slice image of the defect area. When any defect type is an unrepairable defect, the corresponding slice image (i.e., rejected patches) is discarded; when any defect type is a repairable defect, the corresponding slice image and the corresponding defect saliency map are input into AR-CycleGAN, and the first image generator in AR-CycleGAN repairs the slice image based on the defect position information contained in the defect saliency map to generate the corresponding normal image (i.e., restored patches).

[0077] The pathological image defect repair method provided by the present invention utilizes a defect detector, a defect classifier, and a first image generator to accurately detect and repair different types of defects in pathological images. This ensures that the repaired normal image retains the original information of the pathological image. Furthermore, when repairing slice images corresponding to repairable defects, a defect saliency map containing defect location information is introduced to guide the repair process and ensure the stability of microstructures after defect repair.

[0078] Based on the above embodiment, the loss function of the cyclic adversarial training includes a color consistency loss function and a cyclic loss function;

[0079] The coloring consistency loss function is used to characterize the loss value of coloring style consistency. The coloring style consistency includes the coloring style consistency between the sample defect image and the sample defect repair image, the coloring style consistency between the sample defect enhanced image and the first sample enhanced repair image, the coloring style consistency between the sample defect image and the sample defect enhanced image, and the coloring style consistency between the sample normal image and the sample normal enhanced image.

[0080] The cyclic loss function is used to characterize the loss value of the cyclic training of the first image generator, the second image generator, the first image discriminator, and the second image discriminator.

[0081] Specifically, the coloring consistency loss function is used to characterize the loss value of coloring style consistency, that is, to characterize the similarity of coloring styles between the sample defect image and the sample defect repair image, the similarity of coloring styles between the sample defect enhanced image and the first sample enhanced repair image, the similarity of coloring styles between the sample defect image and the sample defect enhanced image, and the similarity of coloring styles between the sample normal image and the sample normal enhanced image.

[0082] The dyeing consistency loss function constructed in this way is used to minimize the dyeing style difference between the sample defect image and the sample defect repair image, minimize the dyeing style difference between the sample defect enhanced image and the first sample enhanced repair image, and minimize the difference between the dyeing style difference between the sample defect image and the sample defect enhanced image and the dyeing style difference between the sample normal image and the sample normal enhanced image, so that the trained first image generator can not only remove defects in the slice image, but also make the generated normal image consistent with the dyeing style of the slice image.

[0083] In addition, a cyclic loss function is used to represent the loss value of the cyclic training of the first image generator, the second image generator, the first image discriminator, and the second image discriminator. The loss value of the cyclic training is used to represent the difference between the image after defect repair by the first image generator and the corresponding original image without defect, and the difference between the defect image generated by the second image generator and the corresponding original image with defect. The smaller the difference between the image after defect repair by the first image generator and the corresponding original image without defect, and the smaller the difference between the defect image generated by the second image generator and the corresponding original image with defect, the smaller the corresponding loss value of the cyclic training.

[0084] The loss function of the cyclic adversarial training is constructed based on the color consistency loss function and the cycle loss function. It can be to directly add the loss value of the color style consistency and the loss value of the cyclic training, or to perform a weighted sum of the two, etc. The embodiment of the present invention does not specifically limit this. The loss function of the cyclic adversarial training thus obtained not only covers the cyclic adversarial training itself, which can use defective images and flawless images to train to obtain a better first image generator, but also takes into account the loss value of the color style consistency, so that the color style of the normal image output by the trained first image generator is consistent with the color style of the corresponding slice image.

[0085] It can be understood that the loss function of cyclic adversarial training can also include an adversarial loss function, whose training goal is to enable the image generated by the image generator to be able to mislead the image discriminator into being the original image to the greatest extent, and to enable the image discriminator to recognize to the greatest extent whether the input image is the original image or the image generated by the image generator.

[0086] Based on any of the above embodiments, the first image generator is trained based on the following steps:

[0087] Inputting the sample defect image and the corresponding sample saliency map into the first initial generator of the first image generator, and having the first initial generator perform defect repair on the sample defect image to obtain a sample defect repaired image;

[0088] Inputting the sample defect repair image into the second initial generator of the second image generator, and generating defects on the sample defect repair image by the second initial generator to obtain a first fake sample defect image;

[0089] Inputting the sample defect-enhanced image and the corresponding sample saliency map into a first initial generator, and having the first initial generator perform defect repair on the sample defect-enhanced image to obtain a first sample enhanced and repaired image;

[0090] Inputting the first sample enhanced restoration image into the second initial generator, the second initial generator generates defects on the sample enhanced restoration image to obtain a first false sample defect enhanced image;

[0091] Inputting the sample normal image and the corresponding sample saliency map into the second initial generator, and generating defects on the sample normal image by the second initial generator to obtain a second fake sample defect image;

[0092] Inputting the second fake sample defect image into the first initial generator, and having the first initial generator perform defect repair on the second fake sample defect image to obtain a fake sample normal image;

[0093] Inputting the sample normal enhanced image and the corresponding sample saliency map into the second initial generator, the second initial generator generates defects on the sample normal enhanced image to obtain a second fake sample defect enhanced image;

[0094] Inputting the second fake sample defect-enhanced image into the first initial generator, and having the first initial generator perform defect repair on the second fake sample defect-enhanced image to obtain a second sample enhanced repaired image;

[0095] Determining a loss value of dyeing style consistency based on the sample defect image, the sample defect repaired image, the sample defect enhanced image, the first sample enhanced repaired image, the sample normal image, and the sample normal enhanced image;

[0096] Determining a loss value of a loop training based on the sample defect image, the first fake sample defect image, the sample defect-enhanced image, the first fake sample defect-enhanced image, the sample normal image, the fake sample normal image, the sample normal-enhanced image, and the second sample enhanced repaired image;

[0097] The first image generator is obtained by training based on the loss value of coloring style consistency and the loss value of cyclic training.

[0098] Specifically, the first image generator, combined with the second image generator, the first image discriminator and the second image discriminator are subjected to cyclic adversarial training. The training process is as follows: Figure 3 As shown, the sample defect image (x s ), sample defect enhanced image Sample normal image (x t ) and sample normal enhanced image The first initial generator (GeneratorAB), the second initial generator (Generator BA), the first image discriminator, and the second image discriminator are subjected to cyclic adversarial training. The cyclic adversarial training specifically includes:

[0099] The sample defect image (x s ) and the corresponding sample saliency map are input to the first initial generator to obtain a sample defect repair image And the sample defect repair image Input to the second initial generator to obtain the first fake sample defect image

[0100] Enhance the image with sample defects And the corresponding sample saliency map is input into the first initial generator to obtain the first sample enhanced repair image And enhance the first sample to repair the image Input to the second initial generator to obtain the first fake sample defect enhanced image Among them, the sample defect enhanced image The sample defect image (x s ) was obtained by staining enhancement.

[0101] The sample normal image (x t ) and the corresponding sample saliency map are input to the second initial generator to obtain the second false sample defect image And the second false sample defect image Input to the first initial generator to obtain a fake sample normal image

[0102] Normally enhance the sample image And the corresponding sample saliency map is input into the second initial generator to obtain the second false sample defect enhanced image And the second false sample defect enhanced image Input to the first initial generator to obtain the second sample enhanced repair image Among them, the sample normal enhanced image The sample normal image (x t ) was obtained by staining enhancement.

[0103] According to the sample defect image (x s ), sample defect repair image Sample defect enhanced image First sample enhanced repaired image Sample normal image (x t ) and sample normal enhanced image The loss value of dyeing style consistency can be determined According to the sample defect image (x s ), the first false sample defect image Sample defect enhanced image The first fake sample defect enhanced image Sample normal image (x t ), fake sample normal image Sample normal enhanced image And the second sample enhanced repair image The loss value of the training cycle can be determined

[0104] Finally, the loss value based on the consistency of the coloring style And the loss value of the cycle training Perform training to obtain the first image generator.

[0105] It should be noted that the above sample defect image (x s), sample defect enhanced image Sample normal image (x t ) and sample normal enhanced image The corresponding sample saliency maps are introduced. s ) introduces the sample saliency map SaliencyMap(x s ) as an example to illustrate:

[0106] In the process of defect removal, the sample saliency map SaliencyMap(x s ) and the sample defect image (x s ) are input together, and the sample saliency map SaliencyMap(x s ) contains defect location information to guide the sample defect image (x s ) defect repair, thereby improving the stability of tiny structures in pathological images. This process is expressed as:

[0107]

[0108] Among them, the concat operation is to merge the channels of the image in the channel dimension. During the defect repair process, the sample saliency map of each image only needs to be generated once in the offline mode, which ensures the training efficiency. t ) and sample normal enhanced image There is no defect location information, so a random generation method can be used to generate a corresponding sample saliency map to guide the second initial generator to generate defects according to the defect location information of the sample saliency map.

[0109] Based on any of the foregoing embodiments, determining a loss value of coloring style consistency based on the sample defect image, the sample defect repaired image, the sample defect enhanced image, the first sample enhanced repaired image, the sample normal image, and the sample normal enhanced image includes:

[0110] Determining a first staining loss value based on a pixel-level distance between a non-defective area in the sample defect image and a non-defective area in the sample defect-repaired image, and a pixel-level distance between a non-defective area in the sample defect-enhanced image and a non-defective area in the first sample enhanced-repaired image;

[0111] determining a second staining loss value based on a staining difference between a non-defect area in the sample defect image and a non-defect area in the sample defect-enhanced image, and a staining difference between a non-defect area in the sample normal image and a non-defect area in the sample normal-enhanced image;

[0112] Based on the first coloring loss value and the second coloring loss value, a coloring style consistency loss value is determined.

[0113] Specifically, the pixel-level distance refers to the distance between pixels. The pixel-level distance between two images is used to characterize the similarity between the two images, that is, the pixel-level distance between the non-defect area in the sample defect image and the non-defect area in the sample defect repair image is used to characterize the similarity between the non-defect area in the sample defect image and the non-defect area in the sample defect repair image, and the pixel-level distance between the non-defect area in the sample defect-enhanced image and the non-defect area in the first sample enhanced repair image is used to characterize the similarity between the non-defect area in the sample defect-enhanced image and the non-defect area in the first sample enhanced repair image. Considering that the pixels of the non-defect area in the sample defect image should be consistent with those of the non-defect area in the sample defect repair image, and the pixels of the non-defect area in the sample defect enhanced image should be consistent with those of the non-defect area in the first sample enhanced repair image, the training objectives are to minimize the pixel-level distance between the non-defect area in the sample defect image and the non-defect area in the sample defect repair image, and to minimize the pixel-level distance between the non-defect area in the sample defect enhanced image and the non-defect area in the first sample enhanced repair image, that is, to minimize the first staining loss value.

[0114] The staining difference is used to characterize the staining change difference between the source image domain and the target image domain, that is, the degree of staining consistency between the source image domain and the target image domain. The larger the staining difference, the greater the staining change difference between the source image domain and the target image domain, that is, the lower the degree of staining consistency. In order to ensure staining consistency, the embodiment of the present invention uses the difference between minimizing the staining difference between the non-defect area in the sample defect image and the non-defect area in the sample defect-enhanced image, and the difference between the staining difference between the non-defect area in the sample normal image and the non-defect area in the sample normal-enhanced image as the training target, so that the staining enhancement effects of the defect image domain (source image domain) and the normal image domain (that is, the target image domain) are consistent.

[0115] Optionally, after obtaining the first coloring loss value and the second coloring loss value, the first coloring loss value and the second coloring loss value may be weightedly added to obtain a coloring style consistency loss value.

[0116] Based on any of the above embodiments, considering that a major obstacle to defect restoration is the change in staining and tissue structure of pathological images, such as incorrect color normalization results, the staining consistency loss function ensures the consistency of the staining style of the image before and after restoration.

[0117] Take the input sample defect image (x s ) and the corresponding sample saliency map as an example to illustrate the method of determining the loss value of the coloring consistency loss function corresponding to the coloring style consistency:

[0118] The sample defect image (xs ) is enhanced to another staining style to obtain a sample defect enhanced image Input the first initial generator, that is And determine the non-defective area in the target domain image based on the defect location information described by the sample saliency map:

[0119]

[0120] Where ⊙ represents pixel-by-pixel multiplication. Since the output of SaliencyMap(x) is always 0 or 1, x′ represents the non-defective region of the target domain image. The non-defective region of the image should remain consistent before and after being processed by the first or second initial generator.

[0121] On this basis, the loss value of dyeing style consistency is determined based on the following formula:

[0122]

[0123]

[0124]

[0125] in, Represents the loss value of dyeing style consistency, represents the first coloring loss value, represents the second coloring loss value, x′ s represents the non-defect area in the sample defect image, Indicates the non-defective area in the sample defect repair image, represents the non-defect area in the sample defect enhanced image, Indicates the non-defective area in the first sample enhanced repair image, x′ t Indicates the non-defective area in the sample normal image, Indicates the non-defective area in the sample normal enhanced image, γ sc1 and γ sc2 is the weight parameter.

[0126] It can be seen from the above formula that The mean square error (MSE) loss is used to represent the pixel-wise distance between the non-defective region of an image and its corresponding repaired image. In addition, in order to minimize the difference in the dyeing style of the image before and after defect repair, RandStainNA dye enhancement should play an equal role in the training of the first and second initial generators. The difference in color changes between the image and its color-enhanced image in the source image domain and the target image domain is considered, and the structural similarity index (SSIM) is used to measure the color enhancement change of the image. The color enhancement difference between the two domains is then compared using the L1 norm of the difference between the two.

[0127] Based on any of the foregoing embodiments, determining a loss value of cyclic training based on the sample defect image, the first fake sample defect image, the sample defect-enhanced image, the first fake sample defect-enhanced image, the sample normal image, the fake sample normal image, the sample normal-enhanced image, and the second sample enhanced repaired image includes:

[0128] Determining a first cycle loss value of the cyclic training based on the sample defect image, the first false sample defect image, the sample defect enhanced image, and the first false sample defect enhanced image;

[0129] A second cycle loss value of the cycle training is determined based on the sample normal image, the fake sample normal image, the sample normal enhanced image, and the second sample enhanced restoration image.

[0130] Specifically, the first cycle loss value is used to characterize the difference between the defect image generated by the second initial generator and the corresponding original image with defects, that is, the difference between the sample defect image and the first false sample defect image, and the difference between the sample defect enhanced image and the first false sample defect enhanced image.

[0131] The second cycle loss value is used to characterize the difference between the image after the defect is repaired by the first initial generator and the corresponding defect-free original image, that is, the difference between the sample normal image and the fake sample normal image, and the difference between the sample normal enhanced image and the second sample enhanced repaired image.

[0132] Based on any of the above embodiments, the first cycle loss value is determined based on the following formula:

[0133]

[0134] in, represents the first cycle loss value, x s represents the sample defect image, represents the first false sample defect image, represents the sample defect enhanced image, represents the first false sample defect enhanced image;

[0135] The second cycle loss value is determined based on the following formula:

[0136]

[0137] in, represents the second cycle loss value, x t represents a sample normal image, represents the fake sample normal image, represents the sample normal enhanced image, Represents the second sample enhanced repaired image. represents the mathematical expectation, and ‖·‖2 represents the L2 norm.

[0138] The embodiment of the present invention proposes a complete workflow for pathological image defect detection, defect classification, and defect repair. Based on the deep learning method, the characteristics of pathological images are fully considered, and accurate detection and reasonable repair of defects of various types and manifestations in the entire pathological image WSI are achieved. The defect detector divides the defect area slice image at the macro level, and the defect area slice image is further subdivided into small blocks. For images with minor defects and inferred to be flawless, a deep repair network is used to repair them while maintaining staining characteristics and structural consistency. For images with larger defects, they can be marked in red to remind pathologists that the image is less credible. The idea of ​​pathological image defect repair in the embodiment of the present invention imitates the clinical process of pathologists in detecting defects, exploring the information hidden under minor defects, and ignoring or pointing out serious defects, so as to achieve a comprehensive exploration of pathological image information.

[0139] In summary, the embodiments of the present invention provide highly reliable and effective pathology images for both manual review and machine processing. The restoration results are superior to those achieved by current mainstream generative networks in terms of RAM, MS-SSIM, and SRE metrics, and are on par with mainstream methods in terms of PSNR and SSIM. This process also maintains staining consistency before and after restoration, making the generated images realistic and effective. The embodiments of the present invention also effectively reduce the negative impact of defects on deep learning models in downstream cell nucleus segmentation experiments.

[0140] The embodiment of the present invention determines the position of defects in a pathological image by using a defect detector, classifies the pathological image slices by using a defect classifier, and finally restores the defect image by using a defect restorer. Figure 4 Schematic diagram comparing before (Before) and after (After) repair using the method of an embodiment of the present invention.

[0141] The pathological image defect repairing device provided by the present invention is described below. The pathological image defect repairing device described below and the pathological image defect repairing method described above can be referenced to each other.

[0142] Based on any of the above embodiments, the present invention also provides a pathological image defect repair device, such as Figure 5 As shown, the device includes:

[0143] a determination unit 510, configured to determine a pathological image;

[0144] The detection unit 520 is configured to input the pathological image into a defect detector, which divides the defect area of ​​the pathological image into sections to obtain defect area slice images; the defect detector is trained based on sample pathological images and corresponding sample defect area slice images;

[0145] The classification unit 530 is configured to input the defect region slice image into a defect classifier, and the defect classifier classifies each defect in the defect region slice image to obtain a type of each defect in the defect region slice image; the defect classifier is trained based on the sample defect region slice image and the corresponding sample defect type;

[0146] The repairing unit 540 is configured to, if any defect is a repairable defect, input the slice image of the defect and the corresponding defect saliency map to the first image generator, so that the first image generator performs defect repair on the slice image of the defect based on the defect position information contained in the defect saliency map to generate a corresponding normal image;

[0147] The first image generator is obtained by combining the second image generator, the first image discriminator and the second image discriminator through cyclic adversarial training.

[0148] Figure 6 Schematic diagram of the structure of the electronic device provided by the present invention, such as Figure 6As shown, the electronic device may include: a processor (processor) 610, a memory (memory) 620, a communication interface (Communications Interface) 630 and a communication bus 640, wherein the processor 610, the memory 620 and the communication interface 630 communicate with each other via the communication bus 640. The processor 610 may call the logic instructions in the memory 620 to execute a pathological image defect repair method, the method comprising: determining a pathological image; inputting the pathological image into a defect detector, wherein the defect detector divides the defect area of ​​the pathological image to obtain a defect area slice image; the defect detector is trained based on a sample pathological image and a corresponding sample defect area slice image; inputting the defect area slice image into a defect classifier, wherein the defect classifier classifies each defect in the defect area slice image to obtain each defect in the defect area slice image. The defect type; the defect classifier is trained based on the sample defect area slice image and the corresponding sample defect type; in the case that any defect type is a repairable defect, the slice image of any defect and the corresponding defect saliency map are input into the first image generator, and the first image generator performs defect repair on the slice image of any defect based on the defect position information contained in the defect saliency map to generate a corresponding normal image; the first image generator is obtained by cyclic adversarial training in combination with the second image generator, the first image discriminator and the second image discriminator.

[0149] In addition, the logic instructions in the above-mentioned memory 620 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when sold or used as an independent product. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0150] On the other hand, the present invention also provides a computer program product, which includes a computer program stored on a non-transitory computer-readable storage medium, and the computer program includes program instructions. When the program instructions are executed by a computer, the computer can execute the pathological image defect repair method provided by the above methods, the method including: determining a pathological image; inputting the pathological image into a defect detector, and having the defect detector divide the defect area of ​​the pathological image to obtain a defect area slice image; the defect detector is trained based on a sample pathological image and a corresponding sample defect area slice image; inputting the defect area slice image into a defect classifier, and having the defect classifier The device classifies each defect in the defect area slice image to obtain the type of each defect in the defect area slice image; the defect classifier is trained based on the sample defect area slice image and the corresponding sample defect type; when any defect type is a repairable defect, the slice image of any defect and the corresponding defect saliency map are input into the first image generator, and the first image generator performs defect repair on the slice image of any defect based on the defect position information contained in the defect saliency map to generate a corresponding normal image; the first image generator is obtained by cyclic adversarial training in combination with the second image generator, the first image discriminator and the second image discriminator.

[0151] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which is implemented when the computer program is executed by a processor to perform the above-mentioned pathological image defect repair method, the method comprising: determining a pathological image; inputting the pathological image into a defect detector, and having the defect detector divide the defect area of ​​the pathological image to obtain a defect area slice image; the defect detector is trained based on a sample pathological image and a corresponding sample defect area slice image; inputting the defect area slice image into a defect classifier, and having the defect classifier classify each defect in the defect area slice image into a defect classifier. class, and obtain the type of each defect in the defect area slice image; the defect classifier is trained based on the sample defect area slice image and the corresponding sample defect type; in the case that the type of any defect is a repairable defect, the slice image of any defect and the corresponding defect saliency map are input into the first image generator, and the first image generator performs defect repair on the slice image of any defect based on the defect position information contained in the defect saliency map to generate a corresponding normal image; the first image generator is obtained by cyclic adversarial training in combination with the second image generator, the first image discriminator and the second image discriminator.

[0152] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, i.e., they may be located in one location or distributed across multiple network units. Some or all of the modules may be selected based on actual needs to achieve the objectives of the present embodiment. Persons of ordinary skill in the art will be able to understand and implement the present invention without inventive effort.

[0153] Through the description of the above embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus a necessary general hardware platform, or of course, by hardware. Based on this understanding, the essence of the above technical solution or the part that contributes to the existing technology can be embodied in the form of a software product. The computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a magnetic disk, an optical disk, etc., and includes a number of instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or certain parts of the embodiments.

[0154] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present invention.

Claims

1. A method for repairing pathological image defects, characterized in that: include: Determine pathological images; Inputting the pathological image into a defect detector, and dividing the defect area of ​​the pathological image by the defect detector to obtain a slice image of the defect area; The defect detector is trained based on the sample pathological image and the corresponding sample defect area slice image; Inputting the defect area slice image into a defect classifier, and having the defect classifier classify each defect in the defect area slice image to obtain the type of each defect in the defect area slice image; The defect classifier is trained based on the sample defect area slice image and the corresponding sample defect type; If any defect is a repairable defect, the slice image of the defect and the corresponding defect saliency map are input into a first image generator, and the first image generator performs defect repair on the slice image of the defect based on defect position information contained in the defect saliency map to generate a corresponding normal image; The first image generator is obtained by combining the second image generator, the first image discriminator and the second image discriminator through cyclic adversarial training; The second image generator is used to generate a realistic defect image based on a flawless normal image, the first image discriminator is used to determine whether the input image is a real defective image, and the second image discriminator is used to determine whether the input image is a real flawless normal image; The loss function of the cyclic adversarial training includes a color consistency loss function and a cyclic loss function; The coloring consistency loss function is used to characterize the loss value of coloring style consistency, wherein the coloring style consistency includes the coloring style consistency between the sample defect image and the sample defect repair image, the coloring style consistency between the sample defect enhanced image and the first sample enhanced repair image, the coloring style consistency between the sample defect image and the sample defect enhanced image, and the coloring style consistency between the sample normal image and the sample normal enhanced image; The cyclic loss function is used to characterize the loss value of the cyclic training of the first image generator, the second image generator, the first image discriminator, and the second image discriminator.

2. The pathological image defect repair method according to claim 1, characterized in that: The first image generator is trained based on the following steps: Inputting the sample defect image and the corresponding sample saliency map into a first initial generator of the first image generator, and having the first initial generator perform defect repair on the sample defect image to obtain the sample defect repaired image; Inputting the sample defect repair image into the second initial generator of the second image generator, and having the second initial generator generate defects on the sample defect repair image to obtain a first fake sample defect image; Inputting the sample defect-enhanced image and the corresponding sample saliency map into the first initial generator, and having the first initial generator perform defect repair on the sample defect-enhanced image to obtain a first sample enhanced and repaired image; Inputting the first sample enhanced restoration image into the second initial generator, and having the second initial generator generate defects on the sample enhanced restoration image to obtain a first fake sample defect enhanced image; Inputting the sample normal image and the corresponding sample saliency map into the second initial generator, and having the second initial generator generate defects on the sample normal image to obtain a second fake sample defect image; Inputting the second fake sample defect image into the first initial generator, and having the first initial generator perform defect repair on the second fake sample defect image to obtain a fake sample normal image; Inputting the sample normal enhanced image and the corresponding sample saliency map into the second initial generator, and having the second initial generator generate defects on the sample normal enhanced image to obtain a second fake sample defect enhanced image; Inputting the second fake sample defect-enhanced image into the first initial generator, and having the first initial generator perform defect repair on the second fake sample defect-enhanced image to obtain a second sample enhanced and repaired image; Determining a loss value of the dyeing style consistency based on the sample defect image, the sample defect repaired image, the sample defect enhanced image, the first sample enhanced repaired image, the sample normal image, and the sample normal enhanced image; Determining a loss value of the cyclic training based on the sample defect image, the first fake sample defect image, the sample defect-enhanced image, the first fake sample defect-enhanced image, the sample normal image, the fake sample normal image, the sample normal-enhanced image, and the second sample enhanced repaired image; Training is performed based on the loss value of the dyeing style consistency and the loss value of the cyclic training to obtain the first image generator.

3. The method for repairing pathological image defects according to claim 2, characterized in that: The determining of the loss value of the consistency of the dyeing style based on the sample defect image, the sample defect repaired image, the sample defect enhanced image, the first sample enhanced repaired image, the sample normal image, and the sample normal enhanced image includes: determining a first staining loss value based on a pixel-level distance between a non-defective area in the sample defect image and a non-defective area in the sample defect-repaired image, and a pixel-level distance between a non-defective area in the sample defect-enhanced image and a non-defective area in the first sample enhanced-repaired image; determining a second staining loss value based on a staining difference between a non-defect area in the sample defect image and a non-defect area in the sample defect-enhanced image, and a staining difference between a non-defect area in the sample normal image and a non-defect area in the sample normal-enhanced image; Based on the first coloring loss value and the second coloring loss value, a loss value of the coloring style consistency is determined.

4. The method for repairing pathological image defects according to claim 3, characterized in that: The loss value of the dyeing style consistency is determined based on the following formula: in, represents the loss value of the consistency of the dyeing style, represents the first staining loss value, represents the second staining loss value, x s ′ represents the non-defective area in the sample defect image, represents the non-defective area in the sample defect repair image, represents the non-defect area in the sample defect-enhanced image, represents the non-defective area in the image enhanced and repaired by the first sample, x t ′ represents the non-defective area in the normal image of the sample, represents the non-defective area in the normal enhanced image of the sample, γ sc1 and γ sc2 is the weight parameter.

5. The method for repairing pathological image defects according to claim 2, characterized in that: The determining of the loss value of the cyclic training based on the sample defect image, the first fake sample defect image, the sample defect-enhanced image, the first fake sample defect-enhanced image, the sample normal image, the fake sample normal image, the sample normal-enhanced image, and the second sample enhanced restoration image includes: Determining a first cycle loss value of the cycle training based on the sample defect image, the first fake sample defect image, the sample defect-enhanced image, and the first fake sample defect-enhanced image; A second cycle loss value of the cycle training is determined based on the sample normal image, the false sample normal image, the sample normal enhanced image, and the second sample enhanced restoration image.

6. The method for repairing pathological image defects according to claim 5, characterized in that: The first circulation loss value is determined based on the following formula: in, represents the first cycle loss value, x s represents the sample defect image, represents the first false sample defect image, represents the sample defect enhanced image, represents the first false sample defect enhanced image; The second cycle loss value is determined based on the following formula: in, represents the second cycle loss value, x t represents the sample normal image, represents the normal image of the false sample, represents the sample normal enhanced image, represents the second sample enhanced restoration image.

7. A pathological image defect repair device, characterized in that: include: a determining unit, configured to determine a pathological image; a detection unit, configured to input the pathological image into a defect detector, and have the defect detector divide the defect area of ​​the pathological image to obtain a defect area slice image; the defect detector is trained based on a sample pathological image and a corresponding sample defect area slice image; a classification unit, configured to input the defect area slice image into a defect classifier, and have the defect classifier classify each defect in the defect area slice image to obtain a type of each defect in the defect area slice image; The defect classifier is trained based on the sample defect area slice image and the corresponding sample defect type; a repairing unit configured to, when any defect is a repairable defect, input a slice image of the defect and a corresponding defect saliency map to a first image generator, and have the first image generator perform defect repair on the slice image of the defect based on defect position information contained in the defect saliency map to generate a corresponding normal image; The first image generator is obtained by combining the second image generator, the first image discriminator and the second image discriminator through cyclic adversarial training; The second image generator is used to generate a realistic defect image based on a flawless normal image, the first image discriminator is used to determine whether the input image is a real defective image, and the second image discriminator is used to determine whether the input image is a real flawless normal image; The first image generator is obtained by combining the second image generator, the first image discriminator and the second image discriminator through cyclic adversarial training; The loss function of the cyclic adversarial training includes a color consistency loss function and a cyclic loss function; The coloring consistency loss function is used to characterize the loss value of coloring style consistency, wherein the coloring style consistency includes the coloring style consistency between the sample defect image and the sample defect repair image, the coloring style consistency between the sample defect enhanced image and the first sample enhanced repair image, the coloring style consistency between the sample defect image and the sample defect enhanced image, and the coloring style consistency between the sample normal image and the sample normal enhanced image; The cyclic loss function is used to characterize the loss value of the cyclic training of the first image generator, the second image generator, the first image discriminator, and the second image discriminator.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the pathological image defect repair method according to any one of claims 1 to 6 is implemented.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the pathological image defect repair method according to any one of claims 1 to 6 is implemented.

Images