UNET supervised network slice HE virtual staining method and system for pathological analysis

Abstract

The invention discloses a UNET supervised network slice HE virtual staining method and system for pathological analysis, and belongs to the technical field of artificial intelligence digital pathological analysis. Global scanning imaging is performed on a pathological slice to obtain an HE scanning image and a fluorescence confocal scanning image, and a UNET supervised network data set is established; a UNET supervised virtual dyeing network model is constructed; training and verifying the UNET supervised virtual dyeing network model by using the UNET supervised network data set; and carrying out HE virtual staining on the fluorescence imaging pathological section image by using the trained UNET supervised virtual staining network model. The method has the remarkable advantages in the aspects of dyeing precision, dyeing efficiency and robustness, and automatic and rapid virtual dyeing of different types of biological tissue sections can be achieved without accurate image registration.

Description

Background Art

[0002] Digital Pathology, as a part of modern medical imaging technology, is rapidly changing the way of traditional pathology diagnosis. Through digital pathological sections, pathologists can perform remote diagnosis and data management with higher efficiency. However, there are problems in the preparation and staining processes of pathological sections, such as large time consumption, high technical requirements, and poor staining consistency. Especially in large-scale clinical applications, the traditional Hematoxylin and Eosin (HE) staining method is not only time-consuming but also prone to batch differences.

[0003] In recent years, the rapid development of deep learning has brought new opportunities to pathological image analysis, especially in the field of Virtual Staining technology. Through machine learning methods, Virtual Staining converts simple fluorescence images (such as DAPI imaging) into complex pathological images (such as HE staining), thereby improving the efficiency of pathological analysis and reducing the processing time and resource consumption of samples. The application of Virtual Staining technology is gradually increasing in multiple fields, mainly including: Cancer Pathology: Through Virtual Staining, researchers can perform more accurate analysis on cancer sections, such as the identification of pathological features of breast cancer, prostate cancer, etc. These virtual staining images can be comparable to traditional methods and even outperform traditional staining in some aspects. Neuropathology: In neuroscience research, Virtual Staining technology is used for multiple stainings of brain tissue sections to help researchers study the distribution and function of neurons to reveal the mechanisms of nervous system diseases. Cardiovascular Pathology: In the study of cardiomyopathy, Virtual Staining technology is applied to analyze myocardial section images to help identify lesions of cardiomyocytes and promote the understanding of cardiovascular diseases. In summary, Virtual Staining technology has important research significance and has obvious advantages over traditional methods in the following aspects.

[0004] • Improve efficiency and accuracy: Virtual Staining can quickly generate high-quality images for analysis, significantly shortening the traditional staining process. Research shows that image analysis using Virtual Staining technology can maintain high accuracy and reduce the risk of human error.

[0005] • Sample protection and resource conservation: During the traditional staining process, tissue samples may be damaged due to chemical treatment, while Virtual Staining can effectively avoid this problem, thus protecting precious biological samples. In addition, the use of chemical reagents is reduced, achieving more sustainable research and clinical practice.

[0006] • Facilitate remote diagnosis and collaboration: The implementation of Virtual Staining technology helps to carry out remote pathological diagnosis in resource-scarce areas. Through the storage and transmission of digital images, pathologists can achieve cross-regional collaboration and discussion, improving the accessibility of diagnosis.

[0007] • Multi-modal data fusion and analysis: Virtual staining enables the generation of multiple staining results from the same tissue sample, supporting more complex multi-modal data fusion and in-depth analysis. This feature gives it broad application prospects in the research of cancer, cardiovascular diseases, and other complex pathological conditions.

[0008] In the research of virtual staining technology, various imaging techniques and software processing techniques are involved. Deep learning models have become the main tools, especially models based on convolutional neural networks (CNNs) and generative adversarial networks (GANs). In addition, using manually extracted image features (such as texture and morphological features), combined with traditional machine learning classifiers, such as support vector machines (SVMs) and random forests, can also classify and predict the staining of unstained tissue images. The CNN architecture is widely used due to its efficient performance in medical image segmentation. Through the downsampling and upsampling processes of feature extraction and reconstruction, this model can effectively preserve details. In the field of virtual staining, this model can generate results similar to traditional HE staining from simple fluorescence staining images. CycleGAN, as an unsupervised generative adversarial network, is suitable for converting one type of image into another, especially in situations lacking paired data. The advantage of this model lies in its flexible conversion of different staining patterns.

[0009] Methods based on optical and physical imaging techniques can be mainly divided into: Autofluorescence Imaging: Autofluorescence imaging utilizes the fluorescence characteristics of tissues themselves to obtain tissue feature information without staining. By detecting the emission spectra of endogenous fluorescent molecules in tissues, images similar to those after staining can be generated. Multispectral Imaging: Multispectral imaging captures the optical characteristics of samples at different wavelengths, providing richer spectral information than traditional RGB imaging. Through the analysis of multispectral data, virtual staining can be achieved. Raman Spectroscopy Microscopy: Raman spectroscopy microscopy provides information on the chemical composition of tissues by detecting the Raman scattering signals of molecular vibrations. Since different tissue components have unique Raman spectra, it can be used to generate virtual staining images. Photoacoustic Microscopy: Photoacoustic microscopy combines optical and ultrasonic technologies to generate ultrasonic signals through laser excitation and obtain the optical absorption characteristics of tissues. It can be used to generate high-contrast tissue structure images to replace traditional staining. Summary of the Invention

[0010] The technical problem to be solved by the present invention is to provide a UNET supervised network slice HE virtual staining method and system for pathological analysis, aiming at the deficiencies in the above-mentioned existing technologies, and to solve the technical problems of long cycle, high cost, limited key samples, and poor technical consistency existing in the HE staining process during the preparation of pathological sections.

[0011] The present invention adopts the following technical solutions: A UNET supervised network slice HE virtual staining method for pathological analysis, comprising the following steps: S1. Globally scan and image the pathological section to obtain an HE scan image and a fluorescence confocal scan image, and establish a UNET supervised network dataset; S2. Construct a UNET supervised virtual staining network model; S3. Use the UNET supervised network dataset to train and validate the UNET supervised virtual staining network model; S4. Use the trained UNET supervised virtual staining network model to perform HE virtual staining on the fluorescence imaging pathological section image.

[0012] Preferably, step S1 is specifically as follows: Obtain the real HE pathological section scan image and the fluorescence scan image of the original tissue section without HE staining, and perform image preprocessing to establish a UNET supervised network dataset; The image preprocessing includes multi-channel image synthesis; image segmentation; image cropping; and gray distribution correction.

[0013] Preferably, the process for obtaining the real HE pathological section image is as follows: Sampling; fixing; dehydrating and clearing; infiltrating with wax and embedding; sectioning; pasting and staining; then globally scan the finished HE section, and finally obtain a high-resolution tissue HE global imaging; The process for fluorescence scan imaging of the original tissue section is as follows: First, take a local sample of the biological tissue, then make the original tissue section, and then use a laser confocal fluorescence scanning system to perform fluorescence imaging on the original tissue section; respectively use light sources with wavelengths of 488nm and 408nm to perform fluorescence microscopic scanning imaging on the tissue surface to obtain only the tissue background image I 1 and the nucleus image I 2 The global scan images of two channels.

[0014] Preferably, the training set in the UNET supervised network dataset is the grayscale tissue HE slice image X_train, and the validation set is X_test. 10% of the samples in the training set X_train and the validation set X_test are randomly sampled for reverse optimization processing, which is specifically as follows: First, perform 7×7 large-scale Gaussian filtering GaussianFilter(src, dst, [7,7]) to blur the image; extract the image histograms of X_train and X_test, and then compress their gray-scale distributions to cause mixing of cell nuclei and background tissues; finally, mix them with normal samples to complete the establishment of the UNET supervised network dataset.

[0015] Preferably, in step S2, the UNET supervised virtual staining network model includes: An input layer, which is a two-dimensional convolutional layer used to match the data stream dimension of the input training set data; An encoder, including 4 levels of downsampling feature extraction modules, and each of the 4 levels of downsampling feature extraction modules contains a two-dimensional convolutional layer, a max pooling layer, regularization, and a neuron dropout unit A decoder, including 4 levels of upsampling feature extraction modules, and each of the 4 levels of upsampling feature extraction modules contains a two-dimensional transposed convolutional layer, an upsampling layer, regularization, and a neuron dropout An output layer, which is a 1x1 convolutional layer. At the last layer of the decoder, use 1x1 convolution to convert the number of channels into the number of target classes to generate the final virtual staining image.

[0016] Preferably, in step S3, the loss function for training the UNET supervised virtual staining network model is as follows:

[0017] Among them, is the grayscale slice grayscale image and the HE staining image, UNET() is the supervised network model, m is the total number of pixels, SSIM() is the image similarity function, is the weight coefficient, and its range is (0, 1).

[0018] Preferably, the verification steps are as follows: The validation set data with dimensions (W = 256, H = 256, C = 3, M = 500) is input into the trained UNET supervised virtual staining network model at a rate of Batch_Size = 8, and the loss error value Loss and the global average error value Mean[Loss(Y_Fitting, Y_Test)] of the virtual staining Y_Fitting result of each sample X_Test in the validation set relative to the benchmark Y_Test are calculated. The HE virtual staining accuracy and processing efficiency of the UNET supervised virtual staining network model are evaluated using the results.

[0019] Preferably, the Vahadane method is used to normalize the results of HE virtual staining in step S4. By decomposing the color space of the image into a staining matrix and a concentration matrix, and then standardizing to achieve staining normalization.

[0020] Preferably, the specific method of using the Vahadane method to normalize the results of HE virtual staining in step S4 is as follows: Let a pathological image I be represented as a pixel color matrix, and the color of each pixel is jointly determined by a set of staining agents and corresponding concentrations; non-negative matrix factorization is applied to the color matrix to decompose it into a staining basis matrix W and a concentration matrix H : ; The staining basis matrix W represents that each column represents the spectral characteristics of a staining agent, and together they constitute the color basis of the entire staining agent. The concentration matrix H represents that each row represents the concentration information of different staining agents in each pixel point; The image is decomposed into a staining basis and a concentration matrix through non-negative matrix factorization; Color decomposition is performed on a high-quality reference image to obtain its staining basis matrix W ref and a concentration matrix H ref ; This reference staining basis W ref will be used as the target staining standard; Perform NMF decomposition on the target image to obtain its staining basis matrix W target and a concentration matrix H target . Use the staining basis W ref of the reference image to replace the staining basis matrix W target of the target image; Through the new staining basis W ref and the concentration matrix Htarget Reconstruct the target image to achieve color consistency with the reference image.

[0021] In a second aspect, an embodiment of the present invention provides a UNET supervised network slice HE virtual staining system for pathological analysis, including: A data module that globally scans and images the pathological section to obtain a HE scan image and a fluorescence confocal scan image, and establishes a UNET supervised network dataset; A network module that constructs a UNET supervised virtual staining network model; A training module that trains and validates the UNET supervised virtual staining network model using the UNET supervised network dataset; An output module that performs HE virtual staining on the fluorescence imaging pathological section image using the trained UNET supervised virtual staining network model.

[0022] In a third aspect, a computer device includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the steps of the above-mentioned UNET supervised network slice HE virtual staining method for pathological analysis are implemented.

[0023] In a fourth aspect, an embodiment of the present invention provides a computer-readable storage medium including a computer program. When the computer program is executed by a processor, the steps of the above-mentioned UNET supervised network slice HE virtual staining method for pathological analysis are implemented.

[0024] In a fifth aspect, a chip includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the steps of the above-mentioned UNET supervised network slice HE virtual staining method for pathological analysis are implemented.

[0025] In a sixth aspect, an embodiment of the present invention provides an electronic device including a computer program. When the computer program is executed by the electronic device, the steps of the above-mentioned UNET supervised network slice HE virtual staining method for pathological analysis are implemented.

[0026] Compared with the prior art, the present invention has at least the following beneficial effects: A UNET supervised network slicing HE virtual staining method for pathological analysis. In the method distribution of claim 1, image information necessary for the UNET supervised virtual staining network model of the present invention can be acquired, including fluorescence images and chemical HE staining images. Then, based on a large number of experimental samples, a dataset can be established to train and optimize the UNET model, obtaining the best-performing staining network. Finally, after the training and verification of the UNET model are completed, the technology of this method can be put into practical application. Only by acquiring the fluorescence image of the pathological section can a virtual staining image be output.

[0027] Furthermore, a large number of sample images of the tissue to be virtually stained are acquired through image acquisition, so that the initially established neural network model can be trained and verified to evaluate its performance.

[0028] Furthermore, this method uses the UNET model to perform image processing virtual staining on the fluorescence image and then outputs a HE staining image. Therefore, when training the neural network model, paired real HE pathological section images and fluorescence images at corresponding positions need to be acquired. Then, they are used as the target staining output effect y and the input fluorescence image x for training and verification respectively.

[0029] Furthermore, the UNET supervised virtual staining network model has the advantages of a simple model, low computing power requirements, strong color space transformation ability and image style transfer ability, and can be applied to a variety of virtual staining application scenarios in virtual staining.

[0030] Furthermore, the setting of the loss function is used to quantitatively evaluate the quantitative deviation between the virtual HE staining image output by the UNET supervised virtual staining network model and the target HE image effect in the real image. This index can be used for the evaluation of the UNET network training and verification links.

[0031] Furthermore, the initially acquired data is divided to establish neural network training, verification and test datasets. Optimization, verification and testing of the neural network model can be realized, and the three datasets do not overlap with each other.

[0032] Furthermore, please supplement and explain the purpose or benefits of using the Vahadane method to perform uniform setting on the results of HE virtual staining in step S4 according to the content of claim 8, and give a principle analysis explanation.

[0033] It can be understood that the beneficial effects of the second to sixth aspects above can be referred to the relevant descriptions in the first aspect above, and will not be elaborated here.

[0034] In summary, the method of the present invention does not require chemical reagent treatment. Through computer image processing methods, virtual staining of tissue section characteristics in fluorescence imaging is performed. This method has low cost, high efficiency, and good consistency, and can be applied to various application scenarios of pathological section staining, and can stain the initial section in multiple styles, providing high convenience for remote medical diagnosis services.

[0035] The technical solution of the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the accompanying drawings required for use in the embodiments of the present application will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present application. For those of ordinary skill in the art, other accompanying drawings can be obtained based on these drawings without creative efforts.

[0037] Figure 1 It is a schematic flow chart of the present invention; Figure 2 It is a flow chart for establishing a data set; Figure 3 It is a HE virtual staining model of the present invention based on the UNET supervised neural network; FIG. 4 is a virtual staining result diagram of the present invention. Among them, (a) is the gray-scale mixing phenomenon of HE virtual staining, and (b) is the result of staining homogenization; Figure 5 It is a processing flow of the Vahadane method of the present invention; Figure 6 It is a schematic diagram of virtual staining of a fluorescence image of an unstained pathological section of the present invention. Among them, (a) is a fluorescence image of a pathological section, and (b) is a sample image of a virtual HE-stained pathological section; Figure 7 It is a global scanned image obtained by scanning a HE pathological section sample with a whole-slide scanner for pathological sections of the present invention. Among them, (a) is a sample image of a real HE-stained pathological section, and (b) is a sample image of a virtual HE-stained pathological section; Figure 8 It is a schematic diagram of a computer device provided by an embodiment of the present invention.

[0038] Figure 9 It is a block diagram of an electronic device provided according to an embodiment of the present invention.

[0039] Among them, 60. Computer device; 61. Processor; 62. Memory; 63. Computer program; 600. Electronic device; 610. Processing unit; 620. Storage unit; 6201. Random access storage unit; 6202. Cache storage unit; 6203. Read-only storage unit; 6204. Program / utility; 6205. Program module; 630. Bus; 640. Display unit; 650. Input / output interface; 660. Network adapter; 700. External device. Detailed implementation

[0040] Next, the technical solutions in the embodiments of the present invention will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, rather than all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts fall within the protection scope of the present invention.

[0041] In the description of the present invention, it should be understood that the terms "include" and "comprise" indicate the presence of the described features, wholes, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more other features, wholes, steps, operations, elements, components and / or their combinations.

[0042] It should also be understood that the terms used in the specification of the present invention are only for the purpose of describing specific embodiments and are not intended to limit the present invention. As used in the specification of the present invention and the appended claims, unless the context clearly indicates otherwise, the singular forms "a", "an" and "the" are intended to include the plural forms.

[0043] It should be further understood that the term " / and" used in the specification of the present invention and the appended claims refers to any combination and all possible combinations of one or more of the related listed items, and includes these combinations. For example, A and / or B can represent: A exists alone, A and B exist simultaneously, and B exists alone. In addition, the character " / " in the present invention generally represents an "or" relationship between the preceding and following related objects.

[0044] It should be understood that although the terms first, second, third, etc. may be used in the embodiments of the present invention to describe preset ranges, etc., these preset ranges should not be limited to these terms. These terms are only used to distinguish the preset ranges from each other. For example, without departing from the scope of the embodiments of the present invention, the first preset range can also be referred to as the second preset range, and similarly, the second preset range can also be referred to as the first preset range.

[0045] Depending on the context, as used herein, the word "if" can be interpreted as "when" or "while" or "in response to determining" or "in response to detecting". Similarly, depending on the context, the phrase "if determined" or "if detecting (stated condition or event)" can be interpreted as "when determined" or "in response to determining" or "when detecting (stated condition or event)" or "in response to detecting (stated condition or event)".

[0046] Various structural schematic diagrams according to the disclosed embodiments of the present invention are shown in the accompanying drawings. These figures are not drawn to scale, where for the purpose of clear expression, some details are enlarged and some details may be omitted. The shapes of various regions and layers shown in the figures and their relative sizes and positional relationships are merely exemplary, and in practice, there may be deviations due to manufacturing tolerances or technical limitations, and those skilled in the art can design regions / layers with different shapes, sizes, and relative positions according to actual needs.

[0047] The present invention provides a UNET supervised network slice HE virtual staining method for pathological analysis, which virtually converts the autofluorescence image of the original sample slice of biological tissue into a HE stained image by using deep learning technology. To achieve virtual staining, first, slice imaging and dataset establishment are performed; secondly, a UNET supervised network is established to perform virtual staining; then, the Vahadane method is used for image normalization processing to improve the consistency and contrast between the virtual staining result and the real HE effect; finally, virtual staining quality evaluation is performed, including image quantization evaluation methods and subjective evaluation methods. Through image quality quantization evaluation and pathological expert review, the HE virtual staining technology proposed by the present invention has significant advantages in terms of staining accuracy, staining efficiency, and robustness, and can automatically and quickly perform virtual staining on different types of biological tissue slices without accurate image registration.

[0048] Example 1 Please refer to Figure 1 , a UNET supervised network slice HE virtual staining method for pathological analysis of the present invention, comprises the following steps: S1. Globally scan and image the pathological slice to obtain a HE scan image and a fluorescence confocal scan image for establishing a UNET supervised network dataset; The present invention realizes HE virtual staining based on the UNET supervised network model. To construct a virtual staining dataset, it is necessary to simultaneously obtain the scanned image of the real HE pathological slice and the fluorescence scan image of the original tissue slice without HE staining. The supervised network model is trained with a large number of samples, and finally, with the fluorescence image of the original tissue slice as the input, HE virtual staining is performed through the model, and the effect approximating the scanned image of the real HE pathological slice is output to assist digital pathological analysis and diagnosis.

[0049] S101, Section imaging; For tissue section imaging, it is necessary to simultaneously obtain the scanned image of the real HE pathological section and the fluorescence scanned imaging map of the original tissue section without HE staining.

[0050] Obtaining the real HE pathological section image: The real HE pathological section samples can be obtained from the hospital pathology department. Its production process includes the following steps: 1) Sampling; 2) Fixing; 3) Dehydration and clearing; 4) Wax infiltration and embedding; 5) Sectioning; 6) Mounting and staining. Generally, this whole manufacturing process takes 2 - 3 days. Then, use a pathological section global scanner to globally scan the finished HE section with an optical magnification of X40, and finally obtain a high-resolution tissue HE global imaging (111711 x 100267 pixels) with a resolution of 0.174 um x 0.174 um. Scanner imaging parameters: Color level bit depth: 14 Bit, Exposure time: 0.2ms.

[0051] Fluorescence scanned imaging of the original tissue section: First, take a local sample of the biological tissue, and then make the original tissue section. Its production process includes the following steps: 1) Sampling; 2) Fixing; 3) Dehydration and clearing; 4) Drop DAPI fluorescence developer on the tissue surface; Then use a laser confocal fluorescence scanning system to perform fluorescence imaging on the original tissue section; Use light sources with wavelengths of 488nm and 408nm respectively to perform fluorescence microscopic scanning imaging on the tissue surface, and obtain only the tissue background image I 1 (488nm) and the cell nucleus image I 2 Global scanned images of two channels.

[0052] Here, the tissues sampled for the real HE pathological section can be sampled at different locations from the tissues used for the fluorescence scanned imaging.

[0053] S102, Image preprocessing; After obtaining the fluorescence scanned imaging map of the biological tissue section, preprocess the global image, mainly completing four tasks, namely 1) Multi-channel image synthesis; 2) Image segmentation; 3) Image cropping; 4) Gray distribution correction.

[0054] First, for the two channels of fluorescence imaging with wavelengths of 488nm and 408nm, the tissue background (488nm) image I 1Figure with cell nucleus (408nm) I 2 synthesize with the global scan figure, and use the same image coordinate system to I 1 and I 2 perform a summation operation to obtain I f = I 1 + I 2 .

[0055] Secondly, use the OTSU algorithm to perform image segmentation on the synthesized global fluorescence image, and only retain the tissue background and cell nucleus regions in the synthesized figure I t , and discard other redundant information in the global image.

[0056] Subsequently, perform image cropping on the segmented image, and divide the high-resolution global scan figure into M images of 256×256 size local images I t , at this time the data dimension is (W = 256, H = 256, C = 3, M), which is used to establish the virtual staining dataset. Where W and H are the pixel width and height of the local image respectively, C is the number of color channels, and M is the number of local images.

[0057] Finally, perform gray-level distribution correction on the segmented sub-images. The main purpose of this processing is to correct the real HE pathological section scan figure after gray-scale conversion and the fluorescence figure after gray-scale conversion to the same range, improve the similarity of the two in gray-scale features and shape features, and prepare for the establishment of the dataset.

[0058] 1) Perform image gray-scale conversion on the fluorescent local image I t to obtain I tg , at this time the data dimension is (W = 256, H = 256, C = 1, M), and its gray-level distribution is D tg .

[0059] 2) Perform gray-scale conversion on the pink real HE pathological section image I h to obtain I hg , and its gray-level distribution is D hg .

[0060] 3) Take D hg as the gray-level distribution target, and use the gray-level stretching method to process the gray-scale fluorescent imageI tg Perform gray-scale distribution correction. After correction, I tg ’the gray-scale distribution error is less than 1.0 as shown in Equation (1), making its distribution effect close to the true HE-stained image after gray-scale conversion.

[0061] (1) Among them, is the gray-scale value of the original image D tg , is the target gray-scale value after gray-scale stretching D hg , are the minimum and maximum gray-scale values of the original image I tg respectively, are the minimum and maximum gray-scale values of the target image I hg after stretching respectively.

[0062] At this time, the fluorescence local map I tg ’after gray-scale correction and the gray-scale HE-stained local image I hg will be used as the input X for virtual staining and enter the UNET supervised network model for virtual staining.

[0063] S103. Establish a data set.

[0064] Using the collected fluorescence images of biological tissue sections as input and the HE-stained section images as the target for virtual staining. Through image preprocessing, the section fluorescence images after gray-scale correction are obtained, which are almost the same as the gray-scale HE pathological images. At this time, the collected fluorescence images and the HE-stained section images are scanned globally, and then image preprocessing is performed. Finally, a data set for training and validating the UNET supervised model is established. Among them, the training set is the gray-scale tissue HE section map X_train, and the validation set is X_test, with the same sample form as X_train, but the section map areas do not overlap. The experimental set consists of the fluorescence local maps I tg ’after gray-scale correction.

[0065] And the HE-stained section image I hAs virtual staining targets, the training set HE images Y_train and the validation set HE images Y_test are constructed, and there is no overlapping area between the samples of the two. Y_train (Y_test) and X_train (X_test) form a virtual staining data group, which only changes in the color channel, and the position and field of view remain the same. The biological samples in the dataset are sampled from no less than 10 experimental mice or human organ tissues. The information of the finally established dataset is shown in Table 1.

[0066] Table 1 Information of the virtual staining dataset

[0067] In addition, in order to improve the processing accuracy and robustness of the virtual staining model, it is necessary to enhance the dataset. 10% of the samples in the training set X_train and the validation set X_test are randomly sampled and reverse optimization processing is carried out as follows: 1) First, perform a large-scale Gaussian filter of 7×7 GaussianFilter(src, dst, [7,7]) to blur the image and reduce its edge quality.

[0068] 2) Extract the image histograms of X_train and X_test, then compress their gray-scale distributions to reduce the contrast, causing the mixing of the cell nuclei and the background tissues. Finally, mix them with the normal samples to complete the establishment of the dataset.

[0069] S2. Construct a UNET supervised virtual staining network model; Please refer to Figure 3 , based on the UNET supervised deep neural network model, perform HE virtual staining on the fluorescence images of pathological tissue sections, and regard this staining task as a process of processing two-dimensional image RGB three-channel color space transformation information, which is specifically classified as a data regression problem. The UNET supervised deep neural network mainly consists of an input layer, an encoder, a decoder, and an output layer. The model structure diagram is as Figure 3 shown.

[0070] In the model, the input layer is a two-dimensional convolutional layer, and its main task is to match the data stream dimension of the input training set data. In the present invention, the data dimensions of the training set and the validation set input to the model are (W, H, C, M) and (W, H, C, N) respectively, where W and H are the width and height of the training image respectively, C is the number of channels of the sample image, and M and N are the numbers of samples in the training set and the validation set.

[0071] The encoder (downsampling path) consists of 4 levels of downsampling feature extraction modules, and each module contains a two-dimensional convolutional layer (2DConv), a max-pooling layer (Max-pooling), regularization (Batch Norm), and a neuron dropout (Drop) unit.

[0072] Two-dimensional convolutional layer: Each encoding module contains two 3x3 convolutional layers, each followed by a ReLU activation function. The convolutional operation can extract local features and, at the same time, increase the non-linear representation ability by stacking.

[0073] Max pooling layer: Downsampling is performed through a 2x2 max pooling layer to reduce the spatial dimension of the feature map (e.g., from 256x256 to 128x128), while preserving the important information in the feature map. Downsampling enables the network to learn higher-level features.

[0074] The decoder (upsampling path) consists of 4 levels of upsampling feature extraction modules, each of which contains a two-dimensional transposed convolutional layer (2DConvTrans), an upsampling layer (Up-sampling), regularization (Batch Norm), and neuron dropout (Drop).

[0075] Transposed convolutional layer: The transposed convolution (or upsampling layer) is used to upsample the feature map. This process can restore the spatial resolution of the feature map and help reconstruct the output image.

[0076] Skip connection: At each stage of the decoder, the feature map of the corresponding encoder stage is concatenated with the feature map of the current decoder. This skip connection mechanism allows the network to utilize low-level spatial information (such as edges and textures), enhancing the segmentation accuracy.

[0077] The output layer is a 1x1 convolutional layer. At the last layer of the decoder, a 1x1 convolution is used to convert the number of channels to the number of target classes, generating the final virtual staining image. The output feature map has the same spatial size as the input image.

[0078] The symmetric design of UNET enables the network to effectively learn multi-scale features of the image. Through the combination of downsampling and upsampling, the network can integrate information at different levels. In terms of detail retention, skip connections help the network retain detail information during the upsampling process, which is particularly important for processing pathological tissue images of complex structures (such as brain tissue, liver and kidney tissues). With its superior fitting performance and flexible architecture design, UNET has become a key technology for color space transformation in HE virtual staining tasks. It effectively combines context information and detail information, can effectively balance the global and local detail information in the virtual staining image, and ensures the accuracy of the gray-scale features and shape features in the staining result.

[0079] S3. Use the dataset to train and validate the network model; To achieve HE virtual staining of pathological sections, based on the established dataset of fluorescence images of pathological tissues - HE staining images, a UNET supervised deep neural network model is trained and verified.

[0080] Neural network training section Before training the UNET supervised network model, the present invention defines an image loss function Loss to characterize the error amount between the HE virtual staining image reconstructed by the model and the real HE image. The loss function Loss is shown in Equation 2: (2) Where, is the grayscale image of the grayscale section and the HE staining image, UNET() is the supervised network model, m is the total number of pixels, SSIM() is the image similarity function, is the weight coefficient, and its range is (0, 1). The optimization algorithm of the network model adopts the Adam algorithm, and the weight parameters of the network are updated through the backpropagation algorithm. To prevent overfitting during the training process, the invention adopts mechanisms such as EarlyStopping, ModelCheckpoint, and dynamic learning rate decreasing to control the update speed of the network model parameters.

[0081] Finally, the training set data with dimensions (W = 256, H = 256, C = 3, M = 3000) is input into the initial network model at a rate of Batch_Size = 8, and trained with the above strategy. The training iteration test Epochs is set to 200. The training termination condition is Loss < 0.01.

[0082] Neural network verification section The validation set data with dimensions (W = 256, H = 256, C = 3, M = 500) is input into the trained network model at a rate of Batch_Size = 8, and the loss error value Loss of the virtual staining Y_Fitting result of each sample X_Test in the validation set relative to the benchmark Y_Test and the global average error value Mean[Loss(Y_Fitting, Y_Test)] are calculated to evaluate the HE virtual staining accuracy and processing efficiency of the UNET supervised network model.

[0083] S4. Use the trained UNET supervised virtual staining network to perform HE virtual staining on the fluorescence imaging pathological section images; S5. Perform normalization processing on the initial results of virtual staining based on the Vahadane method; Due to the differences in the process quality of tissue sections, there is a certain range of frequency mixing in the gray-scale distribution of cell nuclei, background tissues, and tissue cavity regions in the initial images after scanning and imaging. After virtual HE staining by the UNET supervised network, this error will be transmitted in severely mixed regions, reflecting problems such as blurred outlines and overlapping colors of various elements, which affects the global virtual staining accuracy (as shown in Figure 4) and the accuracy of pathological diagnosis, and reduces the accuracy of the computer-aided diagnosis system. To address this issue, the invention uses the Vahadane method for normalization. By decomposing the color space of the image into a staining matrix and a concentration matrix, and then standardizing these components to achieve staining normalization.

[0084] For reference Figure 5 In order to make the virtual staining image further approximate the real HE image, stain normalization processing is selected in the post-processing part. The Vahadane method aims to make the images between different samples more consistent visually and analytically by eliminating the illumination variations and color inconsistencies between images. It adopts a processing strategy based on the color space and combines statistical models to achieve efficient normalization. The implementation process is as follows: S501. Image color decomposition The core of the Vahadane method lies in decomposing the color information in the image into the spectral information of the stain (staining substrate) and the staining concentration of each pixel: 1) Assume a pathological image I represented as a pixel color matrix, where the color of each pixel is jointly determined by a set of stains and the corresponding concentrations.

[0085] 2) Non-negative matrix factorization (NMF) is applied to the color matrix to decompose it into a staining substrate matrix W and a concentration matrix H : .

[0086] Among them, the staining substrate matrix W : Each column represents the spectral characteristics of a stain, and together they constitute the color substrate of the entire stain. The concentration matrix H : Each row represents the concentration information of different stains in each pixel point.

[0087] S502. Application of non-negative matrix factorization (NMF) NMF is a matrix factorization technique that requires all matrix elements to be non-negative. By decomposing the image color information in this way, the physical interpretability of the staining substrate and concentration distribution is ensured (that is, the stains in the image have actual biological meanings). By decomposing the image into a staining substrate and a concentration matrix, the Vahadane method can change the color information of the image while maintaining the image structure.

[0088] S503. Obtaining a reference staining substrate Select a high-quality reference image (usually selected from representative images under the same experimental conditions), perform color decomposition on it, and obtain its staining substrate matrix W ref and concentration matrix H ref . This reference staining substrate W ref will be used as the target staining standard for the standardization process of other images.

[0089] S504. Color standardization of the target image Perform NMF decomposition on the target image to obtain its staining substrate matrix W target and concentration matrix H target . Use the staining substrate of the reference image W ref to replace the staining substrate matrix of the target image W target . Reconstruct the target image through the new staining substrate W ref and the concentration matrix of the target image H target to achieve color consistency with the reference image. The standardized image I ’ is expressed as:

[0090] The specific implementation of the Vahadane method is as follows: 1) Non-negative matrix factorization: For a pathological image I , non-negative matrix factorization is performed into a staining substrate W and concentration matrix H : (3) 2) Substrate replacement and reconstruction: Use the reference substrate W ref to replace the target substrate W target , and reconstruct the image: (4) 3) Optimization algorithm: NMF decomposition is solved by the gradient descent method algorithm. Since the matrix dimension is relatively high, to improve efficiency, the Vahadane method often uses the Alternating Least Squares (ALS) method to solve the non-negative matrix factorization problem to ensure the physical meaning of the staining substrate and concentration matrix.

[0091] S6. Perform accuracy analysis and evaluation.

[0092] Image quality quantitative evaluation index MSE (Mean Squared Error): Measures the average pixel difference between the generated image and the real image. The smaller the MSE, the closer the generated image is to the real image.

[0093] (5) Among them, UNET() is the supervised staining model, is the grayscale image sample of the slice in the training set, is the HE staining image sample, n is the total number of samples in the dataset.

[0094] PSNR (Peak Signal-to-Noise Ratio): PSNR is used to measure the difference between the generated image and the real image. The higher the value, the better the quality of the generated image. PSNR is a classic index for image quality evaluation.

[0095] (6) Among them, MAX is the maximum pixel value of the image, and MSE is the mean squared error between the generated image and the real image.

[0096] SSIM (Structural Similarity Index): SSIM measures the similarity between two images in terms of brightness, contrast, structure, etc. The range is [0, 1], and the value closer to 1 represents a higher image similarity. SSIM is suitable for evaluating the preservation of structural or edge information in images.

[0097] (7) Among them, and are the average values of the two images, and are the standard deviations, is the covariance, C 1 and C 2 are small constants used to stabilize the results.

[0098] Medical image specific evaluation Expert review: Subjectively evaluate the virtual staining images through the review of experts in the field of pathology. Such evaluations usually use scales, for example: Likert scale: Evaluate the image quality and staining effect from level 1 to 5.

[0099] MOS (Mean Opinion Score): Pathologists give an overall score to the images.

[0100] Quantitative analysis of tissues or cells: Detect the edge contours of tissue regions, quantify indicators such as the area of tissue regions and the number of cell nuclei using image features, and compare with real HE-stained images to evaluate the accuracy of virtual staining.

[0101] Those skilled in the art can understand that various aspects of the present invention can be implemented as a system, a method, or a program product. Therefore, various aspects of the present invention can be specifically implemented in the following forms, namely: a complete hardware implementation, a complete software implementation (including firmware, microcode, etc.), or an implementation combining hardware and software aspects, which can be collectively referred to as "circuit", "module", or "platform" here.

[0102] Example 2 The present invention provides a UNET supervised network slice HE virtual staining system for pathological analysis, which can be used to implement the above-mentioned UNET supervised network slice HE virtual staining method for pathological analysis. Specifically, the UNET supervised network slice HE virtual staining system for pathological analysis includes a data module, a network module, a training module, and an output module.

[0103] Among them, the data module globally scans and images the pathological sections to obtain HE scan images and fluorescence confocal scan images, and establishes a UNET supervised network dataset; The network module constructs a UNET supervised virtual staining network model; The training module trains and validates the UNET supervised virtual staining network model using the UNET supervised network dataset; The output module performs HE virtual staining on the fluorescence imaging pathological section images using the trained UNET supervised virtual staining network model.

[0104] Example 3 The present invention provides a terminal device, which includes a processor and a memory. The memory is used to store a computer program, and the computer program includes program instructions. The processor is used to execute the program instructions stored in the computer storage medium. The processor may be a Central Processing Unit (CPU), or may also be other general-purpose processors, Graphics Processing Units (GPU), Tensor Processing Units (TPU), Digital Signal Processors (DSP), Application Specific Integrated Circuits (ASIC), Field-Programmable Gate Arrays (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. It is the computing core and control core of the terminal, and is suitable for implementing one or more instructions. Specifically, it is suitable for loading and executing one or more instructions to implement the corresponding method flow or corresponding function. The processor described in the embodiments of the present invention can be used for the operation of the UNET supervised network slice HE virtual staining method for pathological analysis, including: Globally scan and image the pathological section to obtain an HE scan image and a fluorescence confocal scan image, and establish a UNET supervised network dataset; construct a UNET supervised virtual staining network model; use the UNET supervised network dataset to train and verify the UNET supervised virtual staining network model; use the trained UNET supervised virtual staining network model to perform HE virtual staining on the fluorescence imaging pathological section image.

[0105] Please refer to Figure 8 , the terminal device is a computer device. The computer device 60 in this embodiment includes: a processor 61, a memory 62, and a computer program 63 stored in the memory 62 and executable on the processor 61. When the computer program 63 is executed by the processor 61, it implements the UNET supervised network slice HE virtual staining method for pathological analysis in the embodiment. To avoid repetition, it will not be elaborated here one by one. Alternatively, when the computer program 63 is executed by the processor 61, it implements the functions of each model / unit in the UNET supervised network slice HE virtual staining system for pathological analysis in the embodiment. To avoid repetition, it will not be elaborated here one by one.

[0106] The computer device 60 may be a computing device such as a desktop computer, a notebook, a palm computer, and a cloud server. The computer device 60 may include, but is not limited to, a processor 61 and a memory 62. Those skilled in the art can understand,Figure 8 This is only an example of the computer device 60, which does not constitute a limitation on the computer device 60. It may include more or fewer components than those shown in the figure, or combine some components, or different components. For example, the computer device may also include input / output devices, network access devices, buses, etc.

[0107] The so-called processor 61 may be a central processing unit (CPU), or other general-purpose processors, graphics processing units (GPUs), tensor processing units (TPUs), digital signal processors (DSPs), application specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0108] The memory 62 may be an internal storage unit of the computer device 60, such as the hard disk or memory of the computer device 60. The memory 62 may also be an external storage device of the computer device 60, such as a plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, etc. equipped on the computer device 60.

[0109] Furthermore, the memory 62 may also include both an internal storage unit and an external storage device of the computer device 60. The memory 62 is used to store computer programs and other programs and data required by the computer device. The memory 62 may also be used to temporarily store data that has been output or will be output.

[0110] Please refer to Figure 9 , the terminal device is the electronic device 600, and the electronic device 600 is presented in the form of a general computing device. The components of the electronic device may include, but are not limited to: at least one processing unit 610, at least one storage unit 620, a bus 630 connecting different platform components (including the storage unit 620 and the processing unit 610), a display unit 640, etc.

[0111] Among them, the storage unit stores program code, which can be executed by the processing unit 610, enabling the processing unit 610 to execute the steps according to various exemplary embodiments of the present invention described in the method section of this specification above. For example, the processing unit 610 can execute the steps as shown in Figure 1 as shown.

[0112] The storage unit 620 may include a readable medium in the form of a volatile storage unit, such as a random access storage unit (RAM) 6201 and / or a cache storage unit 6202, and may further include a read-only storage unit (ROM) 6203.

[0113] The storage unit 620 may also include a program / utility 6204 having a set (at least one) of program modules 6205. Such program modules 6205 include, but are not limited to: an operating system, one or more application programs, other program modules, and program data. Each or some combination of these examples may include the implementation of a network environment.

[0114] The bus 630 may represent one or more of several types of bus structures, including a storage unit bus or a storage unit controller, a peripheral bus, a graphics acceleration port, a processing unit, or a local area bus using any of the multiple bus structures.

[0115] The electronic device 600 can also communicate with one or more external devices 700 (such as a keyboard, a pointing device, a Bluetooth device, etc.), can also communicate with one or more devices that enable a user to interact with the electronic device 600, and / or can communicate with any device that enables the electronic device 600 to communicate with one or more other computing devices (such as a router, a modem). Such communication can be carried out through the input / output interface 650. Moreover, the electronic device 600 can also communicate with one or more networks (such as a local area network, a wide area network, and / or a public network, such as the Internet) through the network adapter 660. The network adapter 660 can communicate with other modules of the electronic device 600 through the bus 630. It should be understood that although not shown in the figure, other hardware and / or software modules can be used in combination with the electronic device 600, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage platforms, etc.

[0116] Example 4 The present invention also provides a storage medium, specifically a computer-readable storage medium, which is a memory device in a terminal device and is used to store programs and data. It can be understood that the computer-readable storage medium here can include both the built-in storage medium in the terminal device and, of course, the extended storage medium supported by the terminal device. It can be any tangible medium that contains or stores a program, and this program can be used by or in combination with an instruction execution system, device, or component. The computer-readable storage medium provides a storage space, and the operating system of the terminal is stored in this storage space. Moreover, one or more instructions suitable for being loaded and executed by the processor are also stored in this storage space, and these instructions can be one or more computer programs (including program codes). It should be noted that more specific examples of the computer-readable storage medium here include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory, a read-only memory, an erasable programmable read-only memory, an optical fiber, a portable compact disk read-only memory, an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0117] The computer-readable storage medium also includes data signals propagated in the baseband or as part of a carrier wave, in which the readable program code is carried. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. The readable storage medium can also be any readable medium other than the readable storage medium, and this readable medium can send, propagate, or transmit a program for use by or in combination with an instruction execution system, device, or component. The program code contained on the readable storage medium can be transmitted using any appropriate medium, including but not limited to wireless, wired, optical cable, radio frequency, etc., or any suitable combination of the above.

[0118] The program code for performing the operations of the present invention can be written in any combination of one or more programming languages. The programming languages include object-oriented programming languages - such as Java, C++, etc., and also include conventional procedural programming languages - such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computing device, partially on the user's device, executed as an independent software package, partially on the user's computing device and partially on a remote computing device, or entirely on a remote computing device or server. In the case of a remote computing device, the remote computing device can be connected to the user's computing device through any type of network, including a local area network or a wide area network, or can be connected to an external computing device (for example, by using an Internet service provider to connect through the Internet).

[0119] One or more instructions stored in a computer-readable storage medium can be loaded and executed by a processor to implement the corresponding steps of the UNET supervised network slice HE virtual staining method for pathological analysis in the above embodiments; one or more instructions in the computer-readable storage medium are loaded and executed by the processor to perform the following steps: Globally scan and image the pathological section to obtain an HE scan image and a fluorescence confocal scan image, and establish a UNET supervised network dataset; construct a UNET supervised virtual staining network model; use the UNET supervised network dataset to train and validate the UNET supervised virtual staining network model; use the trained UNET supervised virtual staining network model to perform HE virtual staining on the fluorescence imaging pathological section image.

[0120] The databases involved in the embodiments provided in the present application may include at least one of a relational database and a non-relational database. The non-relational database may include a distributed database based on blockchain, etc., and is not limited thereto. The processors involved in the embodiments provided in the present application may be a general-purpose processor, a central processing unit, a graphics processing unit, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., and is not limited thereto.

[0121] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are some, but not all, of the embodiments of the present invention. Usually, the components described and shown in the accompanying drawings here can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present invention provided in the drawings is not intended to limit the scope of the claimed invention, but merely represents selected embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present invention.

[0122] Use the technology proposed by the present invention to perform virtual staining on the fluorescence image of an unstained pathological section to verify the effectiveness of the method.

[0123] 1) First, sample the human kidney tissue to make a paraffin pathological section; 2) Subsequently, drop the DAPI fluorescence developer on the sample, and obtain the global fluorescence image of the pathological section by a laser confocal scanner; 3) Use the image preprocessing algorithm designed by the present invention to process the global image and then establish a dataset for virtual staining. The local sample fluorescence images in the dataset are as shown in Figure 6 a; 4) Use the trained UNET supervised network to perform virtual staining on the dataset to obtain a virtual HE staining image, as shown in Figure 6 b; 5) Further perform real HE staining on the paraffin section sample to obtain a real HE pathological section; 6) Use a whole-slide scanner for pathological sections to scan the HE pathological section sample to obtain a global scanned image, and then perform image cropping and registration, as shown in Figure 7 shown; 7) Compare the collected actual HE section staining image with the virtual staining image to evaluate the staining accuracy. The evaluation data is shown in Table 2: Table 2 Virtual Staining Accuracy Evaluation Data

[0124] In summary, the UNET supervised network slice HE virtual staining method and system for pathological analysis of the present invention do not require chemical reagent treatment, and perform image processing virtual staining on the characteristics of tissue sections with fluorescence imaging through computer image processing methods. This method has low cost, high efficiency, and good consistency, can be applied to various application scenarios of pathological section staining, and can stain the initial section in multiple styles, providing high convenience for remote medical diagnosis services.

[0125] Those skilled in the art can clearly understand that for the convenience and conciseness of description, only the above-mentioned division of each functional unit and module is used as an example. In actual applications, the above functions can be allocated to different functional units and modules according to needs, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. Each functional unit and module in the embodiment can be integrated into a processing unit, or each unit can exist physically alone, or two or more units can be integrated into one unit. The above integrated unit can be implemented in the form of hardware or in the form of a software functional unit. In addition, the specific names of each functional unit and module are only for the convenience of mutual distinction and do not limit the protection scope of the present application. The specific working processes of the units and modules in the above system can refer to the corresponding processes in the foregoing method embodiments and will not be elaborated here.

[0126] In the above embodiments, the descriptions of each embodiment have their own emphases. For the parts not detailed or recorded in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0127] Those of ordinary skill in the art will realize that the units and algorithm steps of each example described in combination with the embodiments disclosed in the present invention can be implemented by electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are executed in a hardware or software manner depends on the specific application and design constraints of the technical solution. A professional technician can use different methods for each specific application to implement the described functions, but such implementation should not be considered to exceed the scope of the present invention.

[0128] In the embodiments provided by the present invention, it should be understood that the disclosed device / terminal and method can be implemented in other ways. For example, the device / terminal embodiments described above are merely illustrative. For example, the division of the modules or units is only a logical function division. In actual implementation, there may be other division methods. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the displayed or discussed coupling or direct coupling or communication connection to each other can be through some interfaces. The indirect coupling or communication connection of the device or unit can be in an electrical, mechanical or other form.

[0129] The units described as separate components may or may not be physically separated. The components displayed as units may or may not be physical units, that is, they can be located in one place, or they can be distributed to multiple network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0130] In addition, the functional units in each embodiment of the present invention can be integrated in a processing unit, or each unit can exist physically alone, or two or more units can be integrated in one unit. The above integrated units can be implemented in the form of hardware or in the form of software functional units.

[0131] When the integrated module / unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, to implement all or part of the processes in the above-described embodiment methods of the present invention, it can also be completed by a computer program instructing relevant hardware. The computer program can be stored in a computer-readable storage medium. When the computer program is executed by a processor, the steps of the above-described various method embodiments can be implemented. Among them, the computer program includes computer program code, and the computer program code can be in the form of source code, object code, executable file, or some intermediate form, etc. The computer-readable medium can include: any entity or device capable of carrying the computer program code, recording medium, USB flash drive, mobile hard disk, magnetic disk, optical disc, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signal, telecommunication signal, and software distribution medium, etc. It should be noted that the content included in the computer-readable medium can be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, the computer-readable medium does not include electrical carrier signals and telecommunication signals.

[0132] This application is described with reference to the flowcharts and / or block diagrams of methods, apparatuses, and computer program products according to embodiments of the present application. It should be understood that each process and / or block in the flowchart and / or block diagram, and the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to the processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing device to generate a machine, such that the instructions executed by the processor of the computer or other programmable data processing device generate a device for implementing the functions specified in Figure 1 one process or multiple processes and / or blocks Figure 1 one block or multiple blocks.

[0133] These computer program instructions can also be stored in a computer-readable memory that can direct a computer or other programmable data processing device to work in a specific manner, such that the instructions stored in the computer-readable memory generate a manufactured article including an instruction device that implements the functions specified in Figure 1 one process or multiple processes and / or blocks Figure 1 one block or multiple blocks.

[0134] These computer program instructions can also be loaded onto a computer or other programmable data processing apparatus, so that a series of operation steps are performed on the computer or other programmable apparatus to generate a computer-implemented process, thereby providing instructions for implementing the functions specified in one process or a plurality of processes and / or boxes Figure 1 in one or more processes and / or boxes Figure 1 steps for the functions specified in one or more boxes.

[0135] The above is only to illustrate the technical idea of the present invention and should not be used to limit the protection scope of the present invention. Any modification made on the basis of the technical solution according to the technical idea proposed by the present invention shall fall within the protection scope of the claims of the present invention.

Claims

1. A UNET-supervised network slice HE virtual staining method for pathological analysis, characterized in that: The following steps are involved: S1. Perform global scanning imaging on the pathological sections to obtain HE scanning images and fluorescence confocal scanning images, and establish a UNET supervised network dataset; S2, build UNET supervised virtual dyeing network model; S3, using the UNET supervised network dataset to train and verify the UNET supervised virtual dyeing network model; S4. Use the trained UNET supervised virtual staining network model to perform HE virtual staining on fluorescent imaging pathological section images.

2. The UNET-supervised network slice HE virtual staining method for pathological analysis according to claim 1 is characterized in that: Step S1 is specifically as follows: Obtain real HE pathological slice scanning images and original tissue slice fluorescence scanning images without HE staining, perform image preprocessing, and establish a UNET supervised network dataset; Image preprocessing includes multi-channel image synthesis; image segmentation; image cropping; and grayscale distribution correction.

3. The UNET-supervised network slice HE virtual staining method for pathological analysis according to claim 2 is characterized in that: The process of acquiring real HE pathological slice images is as follows: Sampling; fixation; dehydration and transparency; wax embedding; sectioning; mounting and staining; and then global scanning of the HE section product to finally obtain high-resolution tissue HE global imaging; The original tissue section fluorescence scanning imaging process is as follows: First, local sampling of biological tissue is performed, and then original tissue slices are made. Then, the laser confocal fluorescence scanning system is used to perform fluorescence imaging of the original tissue slices. The tissue surface is scanned by fluorescence microscopy using light sources with wavelengths of 488nm and 408nm, respectively, to obtain an image containing only the tissue background. I 1 and cell nucleus I 2Global scan of two channels.

4. The method for HE virtual staining of UNET-supervised network slices for pathological analysis according to claim 2, characterized in that: The training set in the UNET supervised network data set is the grayscale HE slice image X_train, and the validation set is X_test. 10% of the samples in the training set X_train and the validation set X_test are randomly sampled and reversely optimized as follows: First, a 7×7 large-scale Gaussian filter (src, dst, [7, 7]) is performed to blur the image; the image histograms in X_train and X_test are extracted, and then their grayscale distribution is compressed to cause mixing of cell nuclei and background tissues; finally, it is mixed with normal samples to complete the establishment of the UNET supervised network dataset.

5. The UNET-supervised network slice HE virtual staining method for pathological analysis according to claim 1, characterized in that: In step S2, the UNET supervised virtual coloring network model includes: Input layer: The input layer is a two-dimensional convolutional layer used to match the data flow dimension of the input training set data; The encoder includes a 4-level downsampling feature extraction module, which includes a 2D convolution layer, a maximum pooling layer and a regularization and neuron discarding unit. The decoder includes a 4-level upsampling feature extraction module, which includes a 2D transposed convolution layer, an upsampling layer and regularization, and a neuron discarding layer. The output layer is a 1x1 convolution layer. In the last layer of the decoder, a 1x1 convolution is used to convert the number of channels to the number of target categories to generate the final virtual stained image.

6. The UNET-supervised network slice HE virtual staining method for pathological analysis according to claim 1, characterized in that: In step S3, the loss function for training the UNET supervised virtual coloring network model is as follows: in, is the grayscale image of the grayscale slice and the HE stained image, UNET() is the supervised network model, m is the total number of pixels, SSIM() is the image similarity function, is the weight coefficient and its range is (0, 1).

7. The method for HE virtual staining of UNET-supervised network slices for pathological analysis according to claim 6, characterized in that: The verification steps are as follows: The validation set data with dimensions (W=256, H=256, C=3, M=500) is input into the trained UNET supervised virtual coloring network model at a rate of Batch_Size=8. The loss error value Loss and the global average error value Mean[Loss(Y_Fitting, Y_Test)] of the virtual coloring Y_Fitting result of each sample X_Test in the validation set relative to the benchmark Y_Test are calculated. The results are used to evaluate the HE virtual coloring accuracy and processing efficiency of the UNET supervised virtual coloring network model.

8. The UNET-supervised network slice HE virtual staining method for pathological analysis according to claim 1, characterized in that: The Vahadane method is used to homogenize the results of the virtual staining in step S4HE by decomposing the color space of the image into a staining matrix and a concentration matrix, and then normalizing them to achieve staining normalization.

9. The method for HE virtual staining of UNET-supervised network slices for pathological analysis according to claim 8, characterized in that: The Vahadane method is used to homogenize the results of step S4HE virtual staining as follows: Suppose a pathological image I Represented as a pixel color matrix, the color of each pixel is determined by a set of dyes and their corresponding concentrations; non-negative matrix factorization is applied to the color matrix to decompose it into a dye basis matrix W and concentration matrix H : ; Staining basement matrix W Indicates that each column represents the spectral characteristics of a dye, which together constitute the color base of the entire dye, the concentration matrix H Indicates that each row represents the concentration information of different dyes in each pixel; Decompose the image into staining basis and concentration matrices via non-negative matrix factorization; Perform color decomposition on the high-quality reference image to obtain its stained basis matrix W ref and concentration matrix H ref ; The reference dye substrate W ref will be used as target staining standard; Perform NMF decomposition on the target image to obtain its stained basis matrix W target and concentration matrix H target . Use the reference image for the dyed substrate W ref Replace the stained basis matrix of the target image W target ; Through the new dyeing base W ref and the density matrix of the target image H target Reconstruct the target image to achieve color consistency with the reference image.

10. A UNET-supervised network slice HE virtual staining system for pathological analysis, characterized in that: include: The data module performs global scanning and imaging of the pathological sections to obtain HE scanning images and fluorescence confocal scanning images, and establishes the UNET supervised network dataset; Network module, building UNET supervised virtual dyeing network model; The training module uses the UNET supervised network dataset to train and verify the UNET supervised virtual dyeing network model; The output module uses the trained UNET supervised virtual staining network model to perform HE virtual staining on the fluorescent imaging pathological section images.

Images