Image virtual dyeing method and device based on deep learning and related components

Abstract

The embodiment of the invention provides an image virtual dyeing method and device based on deep learning and a related assembly. The image virtual dyeing method comprises the following steps: utilizing a preprocessed photoacoustic microscopic image and a Hamp; e, performing iterative training on a pre-constructed virtual dyeing model by the dyeing image to obtain a trained virtual dyeing model, and performing virtual dyeing on the new photoacoustic microscopic image by using the trained virtual dyeing model to obtain Hamp; e, dyeing the image of the style (i.e., pseudo Hamp; e, dyeing the image). According to the embodiment of the invention, the image segmentation module is embedded in the generator to help the generator to distinguish cytoplasm and cell nucleus, so that the virtual staining model is constrained, and the generator can better generate a Hamp conforming result; e, dyeing the morphological characteristics of the image to ensure the generated pseudo Hamp; the E dyeing image retains more useful cell nucleus information, and the pre-constructed virtual dyeing model reduces noise in the learning process and effectively prevents the problem of overfitting due to the fact that an image segmentation module is embedded in a generator.

Description

Technical Field

[0001] The present invention relates to the technical field of image staining, and particularly to an image virtual staining method, device and related components based on deep learning. Background Art

[0002] In the medical field, histopathological examination is a key technology for disease diagnosis and treatment, which provides a diagnostic basis for doctors by observing the morphological changes of tissue sections. Pathologists mainly achieve accurate judgment of diseases such as tumors, infections, and inflammations through the morphological changes of tissues and the microscopic structural changes of cells in pathological sections. However, the existing histopathological examination technologies have many limitations. Specifically, conventional pathological examinations are extremely complex and time-consuming in tissue sample preparation, often taking several days to complete, which far fails to meet the timeliness requirements for intraoperative pathological diagnosis; the frozen section technology can provide rapid histological analysis within dozens of minutes and is commonly used for intraoperative pathological diagnosis, but the frozen treatment process used will affect the structures of edematous tissues and tissues with high fat content, easily generating artifacts and resulting in poor quality of pathological images, which is not conducive to judgment.

[0003] In view of the limitations of traditional pathological detection technologies, new imaging means have emerged, such as photoacoustic imaging technology. This technology is based on the optical absorption characteristics of biological tissues and includes two processes: optical excitation and acoustic detection. It does not require the use of exogenous labels and has advantages such as high contrast, large depth, and high spatial resolution, showing broad prospects in the fields of biomedical research and clinical applications. Since the cell nucleus is rich in DNA / RNA and has strong optical absorption characteristics in the ultraviolet light band, after being irradiated by a short-pulse laser, broadband ultrasonic waves (i.e., photoacoustic signals) generated due to the transient thermoelastic effect can be detected. The detected photoacoustic signals can reflect the specific tissue morphology after reconstruction. Therefore, ultraviolet photoacoustic technology can be used to avoid the cumbersome processes of conventional pathological examinations, without the need for sectioning and exogenous labels, and has a sub-micron lateral resolution, showing great application potential in intraoperative pathological detection.

[0004] However, the endogenous information such as cell nuclei in tissue samples is reflected in the intensity changes of photoacoustic signals. For example, the ultrasonic wave signal amplitude excited by the cell nucleus due to its strong optical absorption is strong, and after data reconstruction, it appears bright in the grayscale image; the ultrasonic wave signal amplitude excited by the cytoplasm due to its weak optical absorption is weak, and after data reconstruction, it appears dark in the grayscale image. The contrast between the two is strong, jointly showing the microscopic structure of the tissue sample.

[0005] At present, photoacoustic microscopy has been able to quickly provide high-resolution tissue section images. However, photoacoustic imaging technology generally provides tissue morphological features in the form of grayscale images. Grayscale images do not conform to the habits of existing pathologists, and the sample size of grayscale images is scarce and there is too much background information in the images. Because the pixel ratio of background information in grayscale images is about 90%, while the pixel ratio of the information of the cell nucleus useful for pathologists is only 10%. During the learning process, the model is likely to learn irrelevant information such as noise, resulting in the problem of model overfitting. Summary of the Invention

[0006] An embodiment of the present invention provides an image virtual staining method, device and related components based on deep learning, aiming to solve the problem that there is too much background information and too little useful cell nucleus information in grayscale images, and the model is likely to learn irrelevant information such as noise, resulting in overfitting.

[0007] In a first aspect, an embodiment of the present invention provides an image virtual staining method based on deep learning, including:

[0008] Obtain a photoacoustic microscopy image and an H&E staining image as a data set, and preprocess the data set to obtain image data;

[0009] Pre-construct a virtual staining model, and use the image data to iteratively train the virtual staining model, and adjust the parameters of the virtual staining model to obtain a trained virtual staining model; wherein, the virtual staining model includes two generators and two discriminators, both of the two discriminators adopt the PatchGAN network, both of the two generators adopt an architecture with an encoder and a decoder, and an image segmentation module based on the yolo v8 network is embedded in the two generators;

[0010] Use the trained virtual staining model to perform virtual staining on a newly obtained photoacoustic microscopy image to obtain a pseudo H&E staining image.

[0011] In a second aspect, an embodiment of the present invention provides an image virtual staining device based on deep learning, including:

[0012] A preprocessing unit, configured to obtain a photoacoustic microscopy image and an H&E staining image as a data set, and preprocess the data set to obtain image data;

[0013] A training unit for pre - constructing a virtual staining model, iteratively training the virtual staining model using the image data, adjusting the parameters of the virtual staining model, and obtaining a trained virtual staining model; wherein the virtual staining model includes two generators and two discriminators, both of the two discriminators adopt a PatchGAN network, both of the two generators adopt an architecture with an encoder and a decoder, and an image segmentation module constructed based on the yolov8 network is embedded in the two generators;

[0014] A staining unit for virtually staining newly acquired photoacoustic microscopy images using the virtual staining model to obtain pseudo H&E stained images.

[0015] In a third aspect, an embodiment of the present invention provides a computer device, including a memory, a processor, and a computer program stored on the memory and executable on the processor, where when the processor executes the computer program, it implements the image virtual staining method as described above.

[0016] In a fourth aspect, an embodiment of the present invention provides a computer - readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, it implements the image virtual staining method as described above.

[0017] An embodiment of the present invention provides a method, device, and related components for image virtual staining based on deep learning. This image virtual staining method iteratively trains a pre - constructed virtual staining model by using pre - processed photoacoustic microscopy images and H&E stained images, obtains a trained virtual staining model, and then uses the trained virtual staining model to virtually stain new photoacoustic microscopy images to obtain images in the H&E staining style (pseudo H&E stained images). In the embodiment of the present invention, an image segmentation module is embedded in the generator to help the generator distinguish between cytoplasm and nucleus, thereby constraining the virtual staining model, enabling the generator to better generate morphological features consistent with H&E stained images, ensuring that the generated pseudo H&E stained images have less background information, and since the pre - constructed virtual staining model embeds an image segmentation module in the generator, it reduces noise during the learning process and effectively prevents the problem of overfitting. Description of the Drawings

[0018] To more clearly illustrate the technical solutions of the embodiments of the present invention, the following will briefly introduce the drawings required for the description of the embodiments. Obviously, the following - described drawings are some embodiments of the present invention. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0019] Figure 1Schematic flowchart of an image virtual staining method based on deep learning provided by an embodiment of the present invention;

[0020] Figure 2 Schematic diagram of the principle of ultraviolet photoacoustic imaging;

[0021] Figure 3 Schematic sub - process of an image virtual staining method based on deep learning provided by an embodiment of the present invention Figure 1 ;

[0022] Figure 4 Schematic sub - process of an image virtual staining method based on deep learning provided by an embodiment of the present invention Figure 2 ;

[0023] Figure 5 Schematic sub - process of an image virtual staining method based on deep learning provided by an embodiment of the present invention Figure 3 ;

[0024] Figure 6 Schematic sub - process of an image virtual staining method based on deep learning provided by an embodiment of the present invention Figure 4 ;

[0025] Figure 7 Schematic diagram of the training of the virtual staining model;

[0026] Figure 8 Schematic block diagram of an image virtual staining device based on deep learning provided by an embodiment of the present invention. Detailed implementation manners

[0027] Next, the technical solutions in the embodiments of the present invention will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, rather than all of the embodiments. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts shall fall within the protection scope of the present invention.

[0028] It should be understood that when used in this specification and the appended claims, the terms "include" and "comprise" indicate the presence of the described features, wholes, steps, operations, elements, and / or components, but do not exclude the presence or addition of one or more other features, wholes, steps, operations, elements, components, and / or their combinations.

[0029] It should also be understood that the terms used in this specification of the present invention are only for the purpose of describing specific embodiments and are not intended to limit the present invention. As used in this specification of the present invention and the appended claims, unless the context clearly indicates otherwise, the singular forms "a", "an", and "the" are intended to include the plural forms.

[0030] It should be further understood that the term "and / or" used in the specification and appended claims of the present invention refers to any combination and all possible combinations of one or more of the associated listed items, and includes these combinations.

[0031] Photoacoustic imaging technology generally provides morphological features of tissues in the form of grayscale images. However, different from the "gold standard" for pathologists' judgment (i.e., the H&E (hematoxylin-eosin) staining method), during the H&E staining process, hematoxylin dye binds to the DNA of the cell nucleus in the form of a nucleophilic dye, making the cell nucleus dark blue or purple; eosin dye stains the cytoplasm and extracellular matrix in the form of an acidic dye, making them pink. The above differences make it difficult for pathologists to make accurate judgments from photoacoustic grayscale images. Therefore, new technical means are needed to convert photoacoustic grayscale images into H&E-like images or pseudo-stained H&E images that are easier for pathologists to understand. Specifically, please refer to Figure 1 , an embodiment of the present invention provides an image virtual staining method based on deep learning, including S10-S30:

[0032] S10. Obtain a photoacoustic microscopy image and an H&E staining image as a data set, and preprocess the data set to obtain image data;

[0033] In this step, photoacoustic microscopy images and H&E staining images of adjacent sections of a mouse brain are mainly selected. Photoacoustic imaging technology is based on the optical absorption characteristics of biological tissues to obtain photoacoustic microscopy images, which include two processes: optical excitation and acoustic detection. Without exogenous labeling, it has advantages such as high contrast, large depth, and high spatial resolution; the H&E staining image is a commonly used staining image in tissue pathological examinations (i.e., the "gold standard"), which can specifically present structures such as cell nuclei, cytoplasm, and extracellular matrix in different colors, providing an important basis for pathological diagnosis. For the imaging tissue samples of mice, the brain is obtained from euthanized mice, and the tissue processor is used to dehydrate, clarify, and infiltrate the formalin-fixed tissue to obtain paraffin-embedded blocks, and then they are cut into sections with a thickness of 5-7.0 μm. To consider almost the same anatomical structure, two adjacent sections are divided into a group. One of them is stained by the H&E staining method (i.e., the hematoxylin-eosin staining method), and the H&E staining image of the section is captured under a bright-field optical microscope, and the process takes about 3-5 days; the other one is not stained, and the microscopic grayscale image (i.e., the photoacoustic microscopy image) of the tissue sample is reconstructed using ultraviolet photoacoustic imaging (UV_PAM) technology, and the process takes about ten minutes.

[0034] Among them, the ultraviolet photoacoustic imaging steps of the tissue sample are as follows: The unlabeled mouse brain tissue section sample gently placed on a transparent quartz coverslip is immersed in an internal water tank filled with deionized water (i.e.,Figure 2 in a water tank) to ensure acoustic coupling and maintain photoacoustic confocal. The water tank is installed on a 2D scanning stage (i.e., Figure 2 the displacement stage). An interconnected card programmed with LabVIEW (i.e., Figure 2 the server and acquisition card) controls and synchronizes the laser pulse, motor movement, and data acquisition processes. For UV PAM image reconstruction, by recording the time course of the photoacoustic output, each depth-resolved signal (A-line) is generated from a single laser pulse, and 3D raw data is obtained from the 2D raster scan of the specimen. Finally, a photoacoustic image of the mouse brain tissue sample is obtained by performing a maximum amplitude projection (MAP) along the depth direction for each A-line. The photoacoustic imaging system uses an ultraviolet objective lens with a numerical aperture of 0.13 (i.e., Figure 2 the objective lens) and a pulsed laser with a wavelength of 266 nm (i.e., Figure 2 the laser). The numerical aperture of the ultraviolet objective lens and the laser wavelength determine the lateral resolution of the photoacoustic image. The lateral resolution of the system is approximately 1.1 μm. The mouse brain slice tissue is photoacoustically imaged using an illumination laser with a pulse repetition frequency (PRF) of 20 kHz, and the scanning step size is approximately 1.0 μm. The photoacoustic imaging technology shows that it can obtain an image with a size of 5000 pixels (x) × 5000 pixels (y) and a field of view (FOV) of 5 × 5 square millimeters in approximately 25 minutes.

[0035] In one embodiment, as Figure 3 shown, S10 includes:

[0036] S11. Cut each image in the dataset into a predetermined size to obtain a set of cut images;

[0037] S12. Judge the information entropy of each cut image in the set of cut images. If the information entropy is lower than a predetermined entropy value, remove the corresponding cut image, and perform data augmentation processing on the remaining cut images to obtain image data, where the data augmentation processing includes flipping, mirroring, or random cropping.

[0038] In S11, each image in the dataset can be cut into a size of 256×256 to unify the image size. In the model training of deep learning, the size of the input image can also be ensured to be consistent by magnifying and randomly cropping it into a fixed-size image, which is beneficial to improving the training efficiency and stability.

[0039] In S12, the information entropy is used to measure the richness of image information. An image with a low information entropy contains less effective information. Therefore, when the information entropy of any segmented image is lower than a predetermined entropy value, the corresponding segmented image is removed, thus avoiding the interference of invalid or redundant data on model training and enabling the model to focus on learning valuable image features, improving the training effect and model performance. Then, data augmentation is performed on the remaining segmented images using methods such as flipping, mirroring, and random cropping. Among them, flipping and mirroring operations can increase the diversity of images, enabling the model to learn image features in different directions and enhancing the model's adaptability to image direction changes; random cropping can enable the model to access the features of different parts of the image, enhancing the model's ability to recognize local features and improving the model's generalization ability, enabling it to perform image conversion more accurately in the face of various situations.

[0040] After the above series of processes, the obtained image data is unlabeled and not paired one by one. Unlabeled means that the samples in the dataset do not carry specific annotation information or labels; not paired one by one means that there is no strict one-to-one correspondence between photoacoustic microscopy images and H&E staining images. This way can enable the model to learn more extensive image conversion rules and enhance the generality of the model.

[0041] S20. A virtual staining model is pre-constructed, and the image data is used to perform iterative training on the virtual staining model, and the parameters of the virtual staining model are adjusted to obtain a trained virtual staining model; wherein, the virtual staining model includes two generators and two discriminators. Both of the two discriminators adopt the PatchGAN network, both of the two generators adopt an architecture with an encoder and a decoder, and an image segmentation module constructed based on the yolo v8 network is embedded in the two generators;

[0042] In this step, the virtual staining model includes two generators and two discriminators, and an image segmentation module based on the YOLO v8 network is embedded in the two generators. Among them, the architectures of the two generators are both designed as encoder-decoder structures. Each generator includes a downsampling block, a residual connection block, and an upsampling block. Specifically, the encoder is composed of multiple downsampling blocks, and each downsampling block contains a reflection padding layer, a convolutional layer, a batch normalization layer, and an activation layer. The reflection padding layer can perform axisymmetric mirror padding on the input image with the outermost pixels as the axis of symmetry to prevent information loss at the image edges and avoid obvious breaks at the image edges. The convolutional layer is the core component of the convolutional neural network. Its main task is to perform convolutional operations on the input data through the local receptive field to extract low-level features and high-level features. By gradually reducing the size of the image and increasing the number of channels, the convolutional neural network can learn the high-level features of the input image. For example, the first downsampling block can use a 3×3 convolutional kernel with a stride of 2 to halve the size of the input image and double the number of channels. The batch normalization layer calculates the mean and variance along the second dimension of the input (i.e., the channel dimension) to accelerate network training and alleviate the problems of gradient disappearance and gradient explosion. The activation layer is located between the output of each layer of the network and the input of the next layer. Its main function is to introduce a non-linear transformation so that the neural network can represent and learn complex non-linear relationships. The decoder corresponds to the encoder and is composed of multiple upsampling blocks. The upsampling block also contains a reflection padding layer, a convolutional layer, a batch normalization layer, and an activation layer, and its function is similar to that of the encoder. Among them, the convolutional layer used in the upsampling block is a transposed convolutional layer, which can restore the low-resolution feature map to a high-resolution one, thereby helping the generator generate an image with rich details and a high resolution (i.e., fusing features at different levels and restoring the image resolution). The encoder and the decoder are connected by a residual connection block, enabling them to fuse feature information at different levels and generate an image with rich details.

[0043] Both discriminators adopt the PatchGAN network structure. The discriminator is composed of three downsampling blocks, and each downsampling block also contains a reflection padding layer, a convolutional layer, a batch normalization layer, and an activation layer, and its function is similar to that of the encoder. The PatchGAN discriminator discriminates in units of small image patches. For example, the input image is divided into small patches of 70×70 pixels. For each small patch, it judges whether the area of each small patch conforms to the cell structure distribution law of the real image, thereby prompting the generator to generate a more realistic image.

[0044] The image segmentation module is constructed using the YOLO v8 network because the YOLO v8 network has powerful object detection and segmentation capabilities. Then, the constructed virtual staining model is iteratively trained using image data to obtain a trained virtual staining model.

[0045] In one embodiment, as Figure 4 shown, S20 includes:

[0046] S21. Draw a mask on the nucleus region in a small amount of image data to obtain a mask image;

[0047] S22. Use the YOLO v8 network to perform image instance segmentation training on the original image and the mask image to obtain a trained YOLO v8 model, and embed the YOLO v8 model into the two generators to obtain an image segmentation module;

[0048] S23. After confirming that the image instance segmentation training is completed, alternately train the generator and the discriminator until the loss function value in the training process reaches a stable convergence state, and use the virtual staining model in the current state as the trained virtual staining model.

[0049] In S21, the pre-training process is to draw a mask on the nucleus region in some image data in the dataset to obtain a mask image. Since the workload of manually annotating a large number of nucleus regions is huge, only a small number of images are annotated to form a micro mask drawing dataset (i.e., a mask image set). Specifically, professionals can draw masks on the nucleus regions in the selected images. The annotation method can be to outline the nucleus contour with a polygon to generate the corresponding mask image.

[0050] In S22, the original image is the image in the image data that has not been masked. The image instance segmentation training process is to input the original image and the corresponding mask image into the YOLO v8 network. By continuously adjusting the network parameters, a trained YOLO v8 model is obtained. The YOLO v8 model can accurately predict the category, bounding box, and mask of each image. Embed the YOLO v8 model into the two generators, and at this time, the YOLO v8 model serves as the image segmentation module.

[0051] In S23, after confirming that the image instance segmentation training is completed, alternately train the generator and the discriminator. Since the image segmentation module is embedded in the generator, this module can enable the virtual staining model to focus on the physical location information of the cells of interest, enabling the generator and the discriminator to achieve better performance levels in overall image generation and discrimination, and being able to generate relatively clear virtual staining images (i.e., the loss function value in the training process reaches a stable convergence state). At this time, the virtual staining model has been trained, and the virtual staining model in this state is used as the trained virtual staining model.

[0052] In one embodiment, as Figure 5 shown, S23 includes:

[0053] S231. Input the image data into the two generators respectively to generate virtual sample data;

[0054] S232. Use the two discriminators to discriminate the input image data and virtual sample data, obtain the discrimination results, calculate the discrimination loss, and update the discriminator parameters through the backpropagation algorithm;

[0055] S233. Calculate the generation loss and discrimination loss according to the discrimination results and update the generator parameters through the backpropagation algorithm;

[0056] S234. Repeat and alternately update the discriminator parameters and generator parameters.

[0057] In this embodiment, taking the generator for converting photoacoustic microscopy images into H&E staining style images as an example, the training process of the virtual staining model is to randomly select photoacoustic microscopy images from the image data and input them into the generator. The generator generates virtual H&E staining images (i.e., virtual sample data). The generated virtual H&E staining images and real H&E staining images are respectively input into the discriminator for discrimination to obtain the discrimination results. The discrimination loss can be calculated by the mean square error loss to measure the difference between the discriminator output and the real label (the real image label is 1, and the generated image label is 0). According to the discrimination loss, the parameters of the discriminator are updated through the backpropagation algorithm. Input the photoacoustic microscopy images into the generator again, calculate the generation loss of the generator and the discrimination loss. Among them, the generation loss includes the adversarial loss and the L1 loss. The adversarial loss is based on the output of the discriminator (i.e., the discrimination result) and measures the ability of the generator to deceive the discriminator; the L1 loss calculates the pixel-level difference between the generated virtual H&E staining image and the real H&E staining image. According to the generation loss and the discrimination loss, the parameters of the generator are updated through the backpropagation algorithm. Continuously repeat the above process for multiple rounds of iterative training. Every 10 rounds of training, evaluate the performance of the generator and the discriminator on the validation set, calculate the peak signal-to-noise ratio (PSNR), structural similarity index (SSIM), and other indicators between the generated virtual sample data and the real images, and observe the change of the loss function value. When the change of the loss function value is less than 0.01 for 5 consecutive rounds on the validation set, it is considered that the loss function value reaches stability, and at this time, the alternating training stage of the generator and the discriminator ends.

[0058] In one embodiment, as Figure 6 shown, S23 further includes:

[0059] S235. Preset the segmentation threshold of the image segmentation module in advance, and then input the image data into the image segmentation module for nuclear instance segmentation to obtain a segmentation mask;

[0060] S236. Calculate the segmentation loss using the binary cross-entropy loss to obtain the region of interest compensation loss value;

[0061] S237. Multiply the compensation loss value of the region of interest by a predetermined coefficient and then apply it to the two generators respectively to adjust the loss function of the generators. Through the backpropagation algorithm, calculate the gradient according to the obtained generation loss to adjust the parameters of the virtual staining model.

[0062] In this embodiment, the segmentation threshold of the image segmentation module is set to 0.6. The segmentation network (i.e., YOLO v8) performs nuclear instance segmentation on the photoacoustic microscopy image and the generated H&E staining image respectively to obtain a segmentation mask (i.e., Figure 7 the H&E mask in Figure 7 ). The segmentation mask calculates the loss value through binary cross-entropy loss, and this loss value is called the compensation loss value of the region of interest. As Figure 7 shown (

[0063] in, PAM is the photoacoustic microscopy image, H&E is the H&E staining image, and the region of interest compensation loss function is the compensation loss value of the region of interest multiplied by a predetermined coefficient). The compensation loss value on the right acts on the gradient update of generator 1 by multiplying by a coefficient (according to the adjusted loss function, through the backpropagation algorithm, calculate the generation loss, and then calculate the gradient according to the generation loss, that is, the gradient update can be realized). Similarly, the compensation loss value on the left acts on the gradient update of generator 2, thereby adjusting the corresponding generator parameters.

[0064] The compensation loss value of the region of interest can solve the problem that the number of pixels in the cytoplasm image of the cell image is too large and the number of pixels in the nucleus image is too small, resulting in the generator tending to generate too much cytoplasm information or noise information, and effectively improves the richness of the cell morphological structure of the generated pseudo-H&E staining image.

[0065] S30. Use the trained virtual staining model to perform virtual staining on the newly acquired photoacoustic microscopy image to obtain a pseudo-H&E staining image.

[0066] In this step, during the training process, the virtual staining model can master the mapping relationship and feature conversion law between the two by learning a large number of photoacoustic microscopy images and H&E staining images. When virtual staining new acquired photoacoustic microscopy images, the virtual staining model can apply the learned knowledge to different photoacoustic microscopy images, thereby generating relatively accurate images in the style of H&E staining (i.e., pseudo-H&E staining images).

[0067] The image virtual staining method further includes: during the training process, using the Adam optimizer to update the discriminator parameters and generator parameters, where the learning rate of the Adam optimizer is 0.001, the first momentum parameter is 0.5, and the second momentum parameter is 0.999.

[0068] In this step, the Adam optimizer is used during training. The number of photoacoustic microscopy images input in each training round is 1, the initial learning rate is 0.001, the first momentum parameter β1 = 0.5, and the second momentum parameter β2 = 0.999. Among them, the value of the initial learning rate can update the parameters of the discriminator and generator with a relatively large step size at the initial stage of training. The relatively large step size allows the model to quickly explore the parameter space, accelerating the convergence speed in the initial stage and avoiding the problem of slow convergence due to too small a learning rate at the initial stage of training. As the training progresses, the learning rate will be adjusted in a Step shape according to the adaptive mechanism of the optimizer (i.e., by introducing two momentum parameters and an adaptive learning rate).

[0069] An embodiment of the present invention further provides an image virtual staining device based on deep learning. This device is used to execute any embodiment of the foregoing image virtual staining method. Specifically, please refer to Figure 8 , Figure 8 which is a schematic block diagram of an image virtual staining device based on deep learning provided by an embodiment of the present invention. The image virtual staining device 400 includes:

[0070] A preprocessing unit 410, configured to obtain photoacoustic microscopy images and H&E staining images as a data set, and preprocess the data set to obtain image data;

[0071] A training unit 420, configured to pre-construct a virtual staining model, and iteratively train the virtual staining model using the image data, and adjust the parameters of the virtual staining model to obtain a trained virtual staining model; wherein, the virtual staining model includes two generators and two discriminators, both of the two discriminators adopt the PatchGAN network, both of the two generators adopt an architecture with an encoder and a decoder, and an image segmentation module constructed based on the yolo v8 network is embedded in the two generators;

[0072] A staining unit 430 for virtually staining a newly acquired photoacoustic microscopy image using the trained virtual staining model to obtain a pseudo H&E stained image.

[0073] In one embodiment, the preprocessing unit 410 is configured to:

[0074] Cut each image in the dataset into a predetermined size to obtain a set of cut images;

[0075] Judge the information entropy of each cut image in the set of cut images. If the information entropy is lower than a predetermined entropy value, remove the corresponding cut image and perform data augmentation on the remaining cut images to obtain image data, where the data augmentation includes flipping, mirroring, or random cropping.

[0076] In one embodiment, each generator includes a downsampling block, a residual connection block, and an upsampling block, and each discriminator includes three downsampling blocks, where the downsampling block is used to extract image features, the residual connection block is used to prevent overfitting, and the upsampling block is used to help the generator generate an image that fuses different levels of features and restores the image resolution.

[0077] In one embodiment, the training unit 420 further includes:

[0078] A mask drawing unit for drawing a mask on the nucleus region in a small amount of image data to obtain a mask image;

[0079] An instance segmentation unit for performing image instance segmentation training on the original image and the mask image using the yolo v8 network to obtain a trained yolo v8 model, and embedding the yolo v8 model into the two generators to obtain an image segmentation module;

[0080] An alternating training unit for, after confirming that the image instance segmentation training is completed, alternately training the generator and the discriminator until the loss function value in the training process reaches a stable convergence state, and using the virtual staining model in the current state as the trained virtual staining model.

[0081] In one embodiment, the alternating training unit is configured to:

[0082] Input the image data into the two generators respectively to generate virtual sample data;

[0083] Use the two discriminators to discriminate the input image data and virtual sample data to obtain a discrimination result, calculate the discrimination loss, and update the discriminator parameters through the backpropagation algorithm;

[0084] Calculate the generation loss and the discrimination loss according to the discrimination result and update the generator parameters through the backpropagation algorithm;

[0085] Repeatedly and alternately update the discriminator parameters and the generator parameters.

[0086] In one embodiment, the alternating training unit is further configured to:

[0087] Preset a segmentation threshold of the image segmentation module, and then input the image data into the image segmentation module for nuclear instance segmentation to obtain a segmentation mask;

[0088] Calculate a segmentation loss using binary cross-entropy loss to obtain an interest region compensation loss value;

[0089] Multiply the interest region compensation loss value by a predetermined coefficient and then apply it to the two generators respectively to adjust the loss function of the generators. Through the backpropagation algorithm, calculate the gradient according to the obtained generation loss to adjust the parameters of the virtual staining model.

[0090] The image virtual staining device is further configured to:

[0091] During training, use the Adam optimizer to update the discriminator parameters and the generator parameters, where the learning rate of the Adam optimizer is 0.001, the first momentum parameter is 0.5, and the second momentum parameter is 0.999.

[0092] An embodiment of the present invention provides a computer device, including a memory, a processor, and a computer program stored on the memory and executable on the processor. When the processor executes the computer program, it implements the image virtual staining method as described in the foregoing embodiments.

[0093] An embodiment of the present invention provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, it implements the image virtual staining method as described in the foregoing embodiments.

[0094] Those skilled in the art can clearly understand that for the convenience and conciseness of description, the specific working processes of the above-described devices, apparatuses, and units can refer to the corresponding processes in the foregoing method embodiments and will not be repeated here. Those of ordinary skill in the art can realize that the units and algorithm steps of each example described in combination with the embodiments disclosed herein can be implemented by electronic hardware, computer software, or a combination of the two. To clearly illustrate the interchangeability of hardware and software, the components and steps of each example have been generally described according to their functions in the above description. Whether these functions are executed in a hardware or software manner depends on the specific application and design constraints of the technical solution. Professional technicians can use different methods for each specific application to implement the described functions, but such implementation should not be considered to exceed the scope of the present invention.

[0095] In several embodiments provided by the present invention, it should be understood that the disclosed devices, apparatuses and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For example, the division of the units is only a logical function division. In actual implementation, there may be other division methods, or units with the same function can be aggregated into one unit. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the displayed or discussed coupling or direct coupling or communication connection between each other can be an indirect coupling or communication connection through some interfaces, devices or units, or can also be an electrical, mechanical or other form of connection.

[0096] The units described as separate components may or may not be physically separated. The components displayed as units may or may not be physical units, that is, they can be located in one place, or can also be distributed to multiple network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of the embodiments of the present invention.

[0097] In addition, each functional unit in various embodiments of the present invention can be integrated in a processing unit, or each unit can exist physically alone, or two or more units can be integrated in one unit. The above-mentioned integrated units can be implemented in the form of hardware or in the form of software functional units.

[0098] If the above-mentioned integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in various embodiments of the present invention. And the aforementioned storage medium includes: USB flash drives, mobile hard disks, read-only memories (ROMs), magnetic disks or optical discs and other various media that can store program codes.

[0099] As described above, the above are only specific embodiments of the present invention, but the protection scope of the present invention is not limited thereto. Any person skilled in the art within the technical scope disclosed by the present invention can easily think of various equivalent modifications or substitutions, and these modifications or substitutions should all be covered within the protection scope of the present invention. Therefore, the protection scope of the present invention should be subject to the protection scope of the claims.

Claims

1. A method for virtual coloring of images based on deep learning, characterized in that: include: Acquiring a photoacoustic microscopic image and an H&E staining image as a data set, and preprocessing the data set to obtain image data; Pre-constructing a virtual dyeing model, and iteratively training the virtual dyeing model using the image data, and adjusting the parameters of the virtual dyeing model to obtain a trained virtual dyeing model; wherein the virtual dyeing model includes two generators and two discriminators, both of the two discriminators use a PatchGAN network, both of the two generators use an architecture with an encoder and a decoder, and an image segmentation module based on a yolo v8 network is embedded in the two generators; The trained virtual staining model is used to virtually stain the newly acquired photoacoustic microscopy images to obtain pseudo H&E staining images.

2. The image virtual coloring method according to claim 1, characterized in that: Preprocessing the data set to obtain image data includes: Cutting each image in the data set according to a predetermined size to obtain a cut image set; The information entropy of each cut image in the cut image set is determined. If the information entropy is lower than a predetermined entropy value, the corresponding cut image is removed, and data enhancement processing is performed on the remaining cut images to obtain image data, wherein the data enhancement processing includes flipping, mirroring or random cropping.

3. The image virtual coloring method according to claim 1, characterized in that: Each generator includes a downsampling block, a residual connection block and an upsampling block, and each discriminator includes three downsampling blocks, wherein the downsampling block is used to extract image features, the residual connection block is used to prevent overfitting, and the upsampling block is used to help the generator generate an image that integrates features at different levels and restores image resolution.

4. The image virtual coloring method according to claim 1, characterized in that: Iteratively training the virtual staining model using the image data to obtain the virtual staining model includes: Masking the cell nucleus region in a small amount of image data to obtain a mask image; Perform image instance segmentation training on the original image and the mask image using the yolo v8 network to obtain a trained yolo v8 model, and embed the yolo v8 model into the two generators to obtain an image segmentation module; When it is confirmed that the image instance segmentation training is completed, the generator and the discriminator are alternately trained until the loss function value in the training process reaches a stable convergence state, and the virtual coloring model in the current state is used as the trained virtual coloring model.

5. The image virtual coloring method according to claim 4, characterized in that: The generator and the discriminator are alternately trained, including: Inputting the image data into the two generators respectively to generate virtual sample data; Using the two discriminators to discriminate the input image data and virtual sample data, obtain a discrimination result, calculate the discrimination loss, and update the discriminator parameters through a back propagation algorithm; Calculate the generation loss and the discrimination loss according to the discrimination result, and update the generator parameters through the back propagation algorithm; The discriminator parameters and the generator parameters are updated alternately and repeatedly.

6. The image virtual coloring method according to claim 5, characterized in that: The generator and the discriminator are alternately trained, further comprising: Presetting a segmentation threshold of the image segmentation module, and then inputting the image data into the image segmentation module to perform cell nucleus instance segmentation to obtain a segmentation mask; The segmentation loss is calculated using the binary cross entropy loss to obtain the compensation loss value of the region of interest; The interest region compensation loss value is multiplied by a predetermined coefficient and then applied to the two generators respectively to adjust the loss function of the generator. Through the back propagation algorithm, the gradient is calculated according to the obtained generation loss to adjust the parameters of the virtual staining model.

7. The image virtual coloring method according to claim 5, characterized in that: Also includes: During the training process, the Adam optimizer is used to update the discriminator parameters and the generator parameters, wherein the learning rate of the Adam optimizer is 0.001, the first momentum parameter is 0.5, and the second momentum parameter is 0.

999.

8. A deep learning-based image virtual coloring device, used to implement the image virtual coloring method according to any one of claims 1 to 7, characterized in that: include: A preprocessing unit, used to obtain a photoacoustic microscopic image and an H&E staining image as a data set, and preprocess the data set to obtain image data; A training unit is used to pre-build a virtual staining model, iteratively train the virtual staining model using the image data, and adjust the parameters of the virtual staining model to obtain a trained virtual staining model; wherein the virtual staining model includes two generators and two discriminators, both of the two discriminators use a PatchGAN network, both of the two generators use an architecture with an encoder and a decoder, and an image segmentation module based on a yolo v8 network is embedded in the two generators; The staining unit is used to use the trained virtual staining model to virtually stain the newly acquired photoacoustic microscopy image to obtain a pseudo H&E staining image.

9. A computer device, characterized in that: The method comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the processor implements the image virtual coloring method according to any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the image virtual coloring method according to any one of claims 1 to 7 is implemented.

Images