Virtual dyeing method and device, computer equipment, storage medium and program product

Abstract

The invention relates to the technical field of image processing, and discloses a virtual dyeing method and device, computer equipment, a storage medium and a program product. The method comprises the following steps: acquiring a hyperspectral microscopic image; performing spectral feature extraction on the hyperspectral microscopic image through a spectral attention module in the virtual dyeing model to obtain a first spectral feature map; wherein the spectrum attention module comprises an intermediate module which is used for segmenting a hyperspectral microscopic image and respectively extracting spectrum information and space information; processing the first spectral feature map through an encoder and a decoder in a virtual dyeing model in sequence to obtain a virtual dyeing intermediate map; processing the hyperspectral microscopic image through a dimensionality reduction convolution layer in the virtual dyeing model to obtain a second spectral feature map; and carrying out feature fusion on the second spectral feature map and the virtual dyeing intermediate map to obtain a virtual dyeing map. According to the scheme, when the virtual dyeing function is achieved, the accuracy is good, and the efficiency is high.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to a virtual coloring method, device, computer equipment, storage medium and program product. Background Art

[0002] The principle of histological staining of tissue specimen sections is to use specific markers (such as isotopes, enzymes, metal ions, chemical dyes, etc.) to mark different biological elements according to the biochemical characteristics of various tissue cells, so as to make the visualization of tissue and cell structures clearer.

[0003] In the related art, experts slice tissue specimens in designated laboratories, mark and stain the slices based on experience, mount the stained tissue specimen slices on glass slides, and image them through a microscope to obtain the final microscopic image of the stained tissue specimen slices.

[0004] However, the above scheme requires experts to operate, with high labor costs and high technical barriers. The entire process is greatly affected by the subjective operation of experts and the process is cumbersome, with low accuracy and efficiency. Once a staining is completed, additional staining and further molecular analysis cannot be performed, and the flexibility is poor. Summary of the invention

[0005] In view of this, the purpose of the present invention is to provide a virtual staining method, apparatus, computer equipment, storage medium and program product to solve the problems of low accuracy, low efficiency and poor flexibility when staining tissue specimen sections.

[0006] In a first aspect, the present invention provides a virtual coloring method, the method comprising: Acquire hyperspectral microscopy images; The spectral feature extraction of the hyperspectral microscopy image is performed through the spectral attention module in the virtual staining model to obtain a first spectral feature map; wherein the spectral attention module includes an intermediate module for segmenting the hyperspectral microscopy image and extracting spectral information and spatial information respectively; The first spectral feature map is processed sequentially by an encoder and a decoder in a virtual dyeing model to obtain a virtual dyeing intermediate map; Processing the hyperspectral microscopic image through a dimension reduction convolution layer in a virtual staining model to obtain a second spectral feature map; The second spectral feature map is feature-fused with the virtual staining intermediate map to obtain a virtual staining map.

[0007] In an optional embodiment, the intermediate module includes a segmentation module and an instance normalization module; the spectral feature extraction of the hyperspectral microscopy image by the spectral attention module in the virtual staining model to obtain a first spectral feature map includes: Projecting the hyperspectral microscopic image through a first convolutional layer to obtain a spectral projection image; The spectral projection image is divided into a first intermediate spectral characteristic image and a second intermediate spectral characteristic image by a division module; Performing instance normalization processing on the first intermediate spectrum feature map by an instance normalization module to obtain a third intermediate spectrum feature map; The second intermediate spectrum characteristic graph and the third intermediate spectrum characteristic graph are concatenated and subjected to convolution integral processing to obtain a fourth intermediate spectrum characteristic graph; The fourth intermediate spectrum characteristic graph and the spectrum projection graph are feature fused to obtain a first spectrum characteristic graph.

[0008] In an optional implementation, the step of concatenating and convolution-integrating the second intermediate spectral characteristic graph and the third intermediate spectral characteristic graph to obtain a fourth intermediate spectral characteristic graph includes: Splicing the second intermediate spectrum characteristic graph and the third intermediate spectrum characteristic graph to obtain an intermediate spectrum characteristic splicing graph; Performing a first convolution integral process on the intermediate spectral feature splicing map through a second convolution layer to obtain a first intermediate spectral feature splicing convolution map; Processing the first intermediate spectral feature concatenated convolution map through an activation function to obtain a processed first intermediate spectral feature concatenated convolution map; The processed first intermediate spectral feature splicing convolution map is subjected to a second convolution integral process through a third convolution layer to obtain a fourth intermediate spectral feature map.

[0009] In an optional embodiment, the virtual staining model further includes a local enhancement window Transformer block; After the first spectral feature map is processed by the encoder in the virtual staining model, the method further includes: The first spectral feature map processed by the encoder is processed by self-attention through a local enhancement window Transformer block.

[0010] In an optional embodiment, the method further includes: Acquire a hyperspectral microscopic sample image and a corresponding stained sample image; The virtual staining model to be trained is iteratively trained using the hyperspectral microscopic sample image and the corresponding stained sample image until the loss function of the virtual staining model to be trained converges to obtain the virtual staining model.

[0011] In an optional embodiment, after acquiring the hyperspectral microscopic sample image and the corresponding stained sample image, the method further includes: Performing black-and-white correction, cropping, and band selection on the hyperspectral microscopic sample image and the stained sample image, respectively, to obtain a processed hyperspectral microscopic sample image and a processed stained sample image; The processed hyperspectral microscopic sample image and the processed stained sample image are subjected to image registration processing to obtain a registered hyperspectral microscopic sample image and a registered stained sample image.

[0012] In a second aspect, the present invention provides a virtual dyeing device, the device comprising: An image acquisition unit, used for acquiring a hyperspectral microscopic image; A spectral attention unit, used for processing the hyperspectral microscopic image through a spectral attention module in a virtual staining model to obtain a first spectral feature map; wherein the spectral attention module includes an intermediate module for segmenting the hyperspectral microscopic image and extracting spectral information and spatial information respectively; An encoding and decoding unit, used for sequentially processing the first spectral feature map through an encoder and a decoder in a virtual staining model to obtain a virtual staining intermediate map; A dimensionality reduction convolution unit, used for processing the hyperspectral microscopic image through a dimensionality reduction convolution layer in a virtual staining model to obtain a second spectral feature map; A feature fusion unit is used to perform feature fusion on the second spectral feature map and the virtual coloring intermediate map to obtain a virtual coloring map.

[0013] In a third aspect, the present invention provides a computer device, comprising: a memory and a processor, the memory and the processor being communicatively connected to each other, the memory storing computer instructions, and the processor executing the virtual coloring method of the first aspect or any corresponding embodiment thereof by executing the computer instructions.

[0014] In a fourth aspect, the present invention provides a computer-readable storage medium having computer instructions stored thereon, the computer instructions being used to enable a computer to execute the virtual coloring method of the first aspect or any corresponding embodiment thereof.

[0015] In a fifth aspect, the present invention provides a computer program product, comprising computer instructions for causing a computer to execute the virtual coloring method of the first aspect or any corresponding embodiment thereof.

[0016] The technical solution provided by the present invention may include the following beneficial effects: The virtual staining method provided by the present invention first obtains a hyperspectral microscopic image, then extracts spectral features of the hyperspectral microscopic image through a spectral attention module in a virtual staining model to obtain a first spectral feature map, wherein the spectral attention module includes an intermediate module for segmenting the hyperspectral microscopic image and extracting spectral information and spatial information respectively, and then processes the first spectral feature map through an encoder and a decoder in the virtual staining model in turn to obtain a virtual staining intermediate map, and then processes the hyperspectral microscopic image through a dimensionality reduction convolutional layer in the virtual staining model to obtain a second spectral feature map, and finally performs feature fusion of the second spectral feature map and the virtual staining intermediate map to obtain a virtual staining map. The above scheme does not need to rely on the judgment and operation of experts. By setting the spectral attention module, the hyperspectral microscopic image can be divided into two parts to extract the spectral information and spatial information respectively. The two extraction processes do not interfere with each other, ensuring the integrity and accuracy of the spectral information and spatial information of the hyperspectral microscopic image. By fusing the features of the second spectral feature map that has not been virtually dyed and the virtual dyeing intermediate map that has been virtually dyed, the obtained virtual dyeing map can be made more realistic, ensuring the accuracy of virtual dyeing. The operation process is simple and efficient, without any actual impact on the hyperspectral microscopic image, and does not affect the repeated use of the hyperspectral microscopic image. The scheme has good flexibility. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the specific implementation methods of the present invention or the technical solutions in the prior art, the drawings required for use in the specific implementation methods or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some implementation methods of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0018] Figure 1 is a schematic flow chart of a virtual dyeing method according to an embodiment of the present invention; Figure 2 is a schematic flow chart of another virtual dyeing method according to an embodiment of the present invention; Figure 3 is a schematic diagram of the structure of a virtual dyeing model according to an embodiment of the present invention; Figure 4 is a schematic diagram of the structure of a spectral attention module according to an embodiment of the present invention; Figure 5is a schematic diagram of a process of training a virtual dyeing model to be trained according to an embodiment of the present invention; Figure 6 is a schematic diagram of a process of performing virtual dyeing by using a virtual dyeing model according to an embodiment of the present invention; Figure 7 is a structural block diagram of a virtual dyeing device according to an embodiment of the present invention; Figure 8 It is a schematic diagram of the hardware structure of a computer device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0019] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.

[0020] The principle of histological staining of tissue specimen sections is to use specific markers (such as isotopes, enzymes, metal ions, chemical dyes, etc.) to mark different biological elements according to the biochemical characteristics of various tissue cells, so as to make the visualization of tissue and cell structures clearer.

[0021] In the related technology, experts in designated laboratories fix tissue specimens in formalin and embed them in paraffin, then cut the tissue specimens into specimens (usually about 2-10 μm thick), mark and stain the sections according to experience, and then mount the stained tissue specimen sections on slides and image them through a microscope to obtain the final microscopic image of the stained tissue specimen sections.

[0022] However, the above scheme requires experts to operate and supervise, and the materials and equipment required are expensive, with high costs and high technical barriers. The entire process is greatly affected by the subjective operation of experts and the process is cumbersome, with low accuracy and efficiency. Once a staining is completed, additional staining and further molecular analysis cannot be performed, and the flexibility is poor.

[0023] According to an embodiment of the present invention, an embodiment of a virtual coloring method is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although the logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0024] In this embodiment, a virtual coloring method is provided, which can be used for desktop computers, notebook computers, servers, etc. Figure 1is a flow chart of a virtual coloring method according to an embodiment of the present invention. Figure 1 As shown, the process includes the following steps: Step S101, acquiring a hyperspectral microscopic image.

[0025] Optionally, the tissue specimen is sliced, and the multi-band spectral information of the tissue specimen slice is collected through a microscope equipped with a hyperspectral camera, and the information is segmented in detail in the spectral dimension into multiple channels (e.g., dozens to hundreds of channels), each channel corresponding to a specific spectral band, to obtain a hyperspectral microscopic image containing rich spatial and spectral information, thereby improving the accuracy of subsequent virtual staining. The hyperspectral camera can be installed above the microscope. The multi-band spectral information can contain spectral information of hundreds of bands.

[0026] Step S102: extract spectral features from the hyperspectral microscopy image through a spectral attention module in a virtual staining model to obtain a first spectral feature map.

[0027] Among them, the spectral attention module includes an intermediate module for segmenting the hyperspectral microscopic image and extracting spectral information and spatial information respectively. By setting the intermediate module, the hyperspectral microscopic image can be segmented and the spectral information and spatial information can be extracted respectively, ensuring that the process of extracting spectral information and the process of extracting spatial information do not interfere with each other, ensuring the integrity of the spectral information and spatial information of the hyperspectral microscopic image, improving the accuracy of subsequent virtual staining, and making the spectral information and spatial morphology of the subsequently generated virtual staining image more realistic and accurate.

[0028] Step S103, the first spectral feature map is processed in sequence by the encoder and the decoder in the virtual coloring model to obtain a virtual coloring intermediate map.

[0029] The encoder is used to convert the first spectral feature map into a series of intermediate feature representations to capture the feature information contained in the first spectral feature map, such as texture, shape, color, etc. The decoder receives the output of the encoder and converts it into a target output. The encoder and decoder can be implemented using modules with corresponding functions in related technologies.

[0030] Step S104, processing the hyperspectral microscopic image through the dimension reduction convolution layer in the virtual staining model to obtain a second spectral feature map.

[0031] It should be noted that step S102 and step S103 are performed sequentially, step S102 and step S104 are performed in parallel, and the order in which step S102 and step S104 are performed is not limited.

[0032] Step S105, performing feature fusion on the second spectral feature map and the virtual coloring intermediate map to obtain a virtual coloring map.

[0033] By performing feature fusion on the second spectral characteristic map and the virtual dyeing intermediate map, the second spectral characteristic map that has not been virtually dyed and the virtual dyeing intermediate map that has been virtually dyed are further feature fused, retaining the original features to the greatest extent, and obtaining a more realistic virtual dyeing map, thereby completing the virtual dyeing process of the hyperspectral microscopic image.

[0034] The virtual staining method provided in this embodiment first obtains a hyperspectral microscopic image, then extracts spectral features of the hyperspectral microscopic image through the spectral attention module in the virtual staining model to obtain a first spectral feature map, then processes the first spectral feature map through the encoder and decoder in the virtual staining model in turn to obtain a virtual staining intermediate map, then processes the hyperspectral microscopic image through the dimension reduction convolution layer in the virtual staining model to obtain a second spectral feature map, and finally fuses the second spectral feature map with the virtual staining intermediate map to obtain a virtual staining map. The above scheme does not need to rely on the judgment and operation of experts. By setting the spectral attention module, the hyperspectral microscopic image is divided into two parts to extract spectral information and spatial information respectively. The two extraction processes do not interfere with each other, ensuring the integrity and accuracy of the spectral information and spatial information of the hyperspectral microscopic image. By fusing the features of the second spectral feature map that has not been virtually stained and the virtual staining intermediate map that has been virtually stained, the obtained virtual staining map can be made more realistic, ensuring the accuracy of virtual staining, and the operation process is simple and efficient, without having a practical impact on the hyperspectral microscopic image, and does not affect the repeated use of the hyperspectral microscopic image, and the scheme has good flexibility.

[0035] In this embodiment, a virtual coloring method is provided, which can be used for desktop computers, notebook computers, servers, etc. Figure 2 is a flow chart of a virtual coloring method according to an embodiment of the present invention. Figure 2 As shown, the process includes the following steps: Step S201, obtaining a hyperspectral microscopic sample image and a corresponding stained sample image.

[0036] The hyperspectral microscopic sample image is an image obtained after microscopic imaging of an unstained tissue specimen slice. The stained sample image corresponds to the hyperspectral microscopic sample image one by one, and is an image after the cell nucleus and matrix of the unstained hyperspectral microscopic sample image are truly stained. The hyperspectral microscopic sample image and the stained sample image constitute a training set and a validation set. Among them, the hyperspectral microscopic sample image can be stained by an expert using a staining method in the relevant technology to obtain a corresponding stained sample image. The staining method can be hematoxylin and eosin (HE) staining, Masson's trichrome staining (MT), periodic acid Schiff staining (PAS), immunohistochemistry (IHC) staining, etc., which will not be repeated here.

[0037] Optionally, after obtaining the hyperspectral microscopic sample image and the corresponding stained sample image, the hyperspectral microscopic sample image and the stained sample image can be preprocessed. Specifically, the hyperspectral microscopic sample image and the stained sample image are respectively subjected to black and white correction, cropping and band selection, retaining the band with high signal-to-noise ratio, and then filtering and noise reduction processing are performed to further improve the signal-to-noise ratio, so as to obtain the processed hyperspectral microscopic sample image and the processed stained sample image. Black and white correction corrects the chromatic aberration of the hyperspectral microscopic sample image and the stained sample image through a standard whiteboard and a standard blackboard, which can eliminate the influence of light intensity and baseline drift on the hyperspectral microscopic sample image and the stained sample image, and improve the accuracy and consistency of the hyperspectral microscopic sample image and the stained sample image. Cropping can select the region of the hyperspectral microscopic sample image and the stained sample image, extract the effective information in the region of interest, reduce the interference of irrelevant data, and improve the efficiency and accuracy of subsequent image processing. Band selection can reduce the information redundancy of the hyperspectral microscopic sample image and the stained sample image while retaining useful information, reduce the amount of data, save resources, and improve the accuracy and efficiency of subsequent image processing. Next, the processed hyperspectral microscopic sample image and the processed stained sample image are subjected to image registration processing, so that the processed hyperspectral microscopic sample image and the corresponding processed stained sample image are registered at the pixel level, thereby obtaining a registered hyperspectral microscopic sample image and a registered stained sample image.

[0038] Step S202 , iteratively training the virtual staining model to be trained using the hyperspectral microscopic sample image and the corresponding stained sample image until the loss function of the virtual staining model to be trained converges to obtain the virtual staining model.

[0039] The purpose of training the virtual staining model to be trained is to make the virtual staining sample image obtained by the virtual staining model to be trained to virtually stain the hyperspectral microscopic sample image as close as possible to the staining sample image corresponding to the hyperspectral microscopic sample image. In other words, the model structure of the virtual staining model to be trained is consistent with that of the virtual staining model, but the specific parameters of the model are different. The virtual staining model is a virtual staining model to be trained whose performance meets the actual requirements after multiple trainings.

[0040] Optionally, a loss function based on pixel difference is set, and the pixel difference between the virtually stained image (virtual stained sample image) output after the unstained image (hyperspectral microscopy sample image) is input into the virtual staining model to be trained for processing and the actual chemically stained image (stained sample image) input into the virtual staining model to be trained is used as the loss value to determine whether the virtual staining model to be trained has converged, and the converged virtual staining model to be trained is used as the virtual staining model for subsequently realizing the virtual staining function.

[0041] Exemplarily, the loss function formula based on pixel difference is as follows:

[0042] in, is the stained sample image, represents a virtual dyeing sample image, is a small constant, usually 10 -3 .

[0043] Specifically, when performing model training, firstly, the spectral feature extraction of the stained sample image corresponding to the hyperspectral microscopy sample image is performed through the spectral attention module in the virtual staining model to be trained to obtain a first sample spectral feature map; wherein, the spectral attention module includes an intermediate module for segmenting the hyperspectral microscopy sample image and the corresponding stained sample image and extracting spectral information and spatial information respectively; then, the first sample spectral feature map is processed in turn through the encoder and decoder in the virtual staining model to obtain a virtual staining intermediate map; then, the hyperspectral microscopy sample image is processed through the dimensionality reduction convolution layer in the virtual staining model to obtain a second sample spectral feature map; finally, the second sample spectral feature map is feature fused with the sample virtual staining intermediate map to obtain a virtual staining sample map.

[0044] Optionally, when extracting spectral features of the stained sample image, the stained sample image is projected through the first convolutional layer so that the format of the stained sample image is aligned with that of the hyperspectral microscopy sample image, especially the number of channels. That is to say, the number of channels of the three-channel stained sample image is expanded to be consistent with the number of channels of the hyperspectral microscopy sample image, so as to facilitate subsequent encoding processing in the input encoder.

[0045] It should be noted that the stained sample image is an RGB (Red, Green, Blue) image, which contains three channels, while the hyperspectral microscopy sample image contains spectral information of multiple channels (for example, dozens to hundreds of channels). In other words, the stained sample image contains very little spectral information. Although it also participates in the spectral feature extraction and encoding and decoding process, the processing effect is limited. The stained sample image is mainly used to compare the pixel difference with the corresponding hyperspectral microscopy sample image as the loss value to determine whether the virtual staining model to be trained has converged.

[0046] Step S203, obtaining a hyperspectral microscopic image.

[0047] This hyperspectral microscopic image is an image obtained by microscopic imaging of an unstained tissue specimen section that actually needs to be virtually stained. Figure 1 Step S101 of the illustrated embodiment will not be described in detail here.

[0048] Step S204: extract spectral features from the hyperspectral microscopy image using a spectral attention module in the virtual staining model to obtain a first spectral feature map.

[0049] Specifically, the above step S204 includes: Step S2041, projecting the hyperspectral microscopic image through the first convolutional layer to obtain a spectral projection image.

[0050] Figure 3 is a schematic diagram of the structure of a virtual dyeing model according to an embodiment of the present invention, Figure 4 is a schematic diagram of the structure of the spectral attention module according to an embodiment of the present invention. After acquiring the hyperspectral microscopic image, the spectral feature extraction of the hyperspectral microscopic image is performed through the spectral attention module (SA). Specifically, the hyperspectral microscopic image is projected to the target size through the first convolution layer (Conv.) to obtain a spectral projection map, which can be expressed by the following formula: F eHSI = Conv(F HSI ) Among them, F HSI Represents a hyperspectral microscopy image, Conv(F HSI ) represents the projection of the hyperspectral microscopy image through the first convolutional layer, F eHSI Represents a spectral projection map. Through projection, it is possible to ensure that the input hyperspectral microscopy image is aligned with the format of the required virtual staining map.

[0051] Exemplarily, the process of projecting the hyperspectral microscopy image through the first convolutional layer is a coupled affine transformation of a reversible neural operator. A coupled affine transformation is a linear transformation that can translate, rotate, scale, and other operations on the input data, so that the input data can be processed at different perspectives and scales. In the feature extraction stage of the neural network, the affine coupled transformation can help extract the features of the input data, so that the neural network can better understand the meaning of the input data. Here, the coupled affine transformation of the reversible neural operator extracts the spectral features of the hyperspectral microscopy image. Each neural operator can effectively extract the spectral features of the hyperspectral microscopy image without losing the spatial features, thereby ensuring the details and authenticity of the generated spectral projection image.

[0052] Exemplarily, the first convolution layer can be a 1×1 convolution layer. The 1×1 convolution layer is mainly used to reduce the dimension of the number of channels of the hyperspectral microscopy image. Compared with the ordinary 3×3 convolution layer, the 1×1 convolution layer can significantly increase the nonlinear characteristics by using the subsequent nonlinear activation function while keeping the feature scale unchanged (i.e., without losing resolution).

[0053] Step S2042: segmenting the spectrum projection image into a first intermediate spectrum characteristic image and a second intermediate spectrum characteristic image through a segmentation module.

[0054] Among them, the intermediate module includes a segmentation module and an instance normalization module.

[0055] Specifically, the process of segmenting the spectral projection image through the segmentation module can be expressed by the following formula: F 1eHSI ,F 2eHI =split(F eHSI ) Among them, split(F eHSI ) represents the spectrum projection graph F eHSI Segmentation, F 1eHSI is the first intermediate spectral characteristic diagram, F 2eHI It is the second intermediate spectrum characteristic diagram.

[0056] Dividing the spectral projection image into two different paths (the first intermediate spectral feature image and the second intermediate spectral feature image) to extract spectral information and spatial information respectively can ensure that the data of the two paths will not interfere with each other during the feature extraction process, ensure the integrity of the spectral information and spatial information, and avoid unnecessary information loss.

[0057] Optionally, the spectrum projection image is evenly split (Spliting) into a first intermediate spectrum feature image and a second intermediate spectrum feature image through a splitting module.

[0058] Step S2043: performing instance normalization processing on the first intermediate spectrum feature graph through an instance normalization module to obtain a third intermediate spectrum feature graph.

[0059] Instance Normalization (IN) processing is performed on the first intermediate spectral feature map, which can calculate the mean and variance of a single image on a single channel, stabilize the training process, alleviate the problems of gradient vanishing and gradient exploding, accelerate the convergence speed, and realize style transfer without changing the content structure of the first intermediate spectral feature map, making the style features more coordinated with the content features, thereby ensuring the authenticity of the subsequently generated virtual coloring map.

[0060] Step S2044: concatenate and perform convolution integration processing on the second intermediate spectral characteristic graph and the third intermediate spectral characteristic graph to obtain a fourth intermediate spectral characteristic graph.

[0061] Specifically, the second intermediate spectrum characteristic graph and the third intermediate spectrum characteristic graph are firstly spliced ​​to obtain an intermediate spectrum characteristic spliced ​​graph.

[0062] Next, the intermediate spectral feature splicing map is subjected to the first convolution integral processing through the second convolution layer to obtain the first intermediate spectral feature splicing convolution map. Exemplarily, the second convolution layer (Conv.) is a 3×3 convolution layer. Using a 3×3 convolution layer to extract feature information can retain more spatial information, making the subsequently generated image more realistic in spatial morphology.

[0063] Then, the first intermediate spectrum feature concatenated convolution map is processed by an activation function to obtain a processed first intermediate spectrum feature concatenated convolution map. Exemplarily, the activation function is LeakeyRelu.

[0064] Finally, the processed first intermediate spectrum feature concatenated convolution map is subjected to a second convolution integral process through a third convolution layer, and the extracted features are integrated to obtain a fourth intermediate spectrum feature map. Exemplarily, the third convolution layer (Conv.) is a 3×3 convolution layer.

[0065] Optionally, the concatenation and convolution integration process is expressed by the following formula: F SIHSI =Conv{Cat[IN(F 1eHSI ),F 2eHSI ]} Among them, F SIHSI Represents the fourth intermediate spectrum characteristic diagram, IN(F 1eHSI ) represents the first intermediate spectral characteristic graph F 1eHSI The third intermediate spectral feature map obtained after the instance normalization process, Cat[IN(F 1eHSI),F 2eHSI ] represents the second intermediate spectral characteristic graph F 2eHSI and the third intermediate spectral characteristic graph IN(F 1eHSI ) after splicing to obtain the intermediate spectral feature splicing image, Conv{Cat[IN(F 1eHSI ),F 2eHSI ]} means to stitch the intermediate spectral features Cat[IN(F 1eHSI ),F 2eHSI ] to perform convolution integral processing.

[0066] Step S2045, performing feature fusion on the fourth intermediate spectrum feature graph and the spectrum projection graph to obtain a first spectrum feature graph.

[0067] The spectral projection image retains more original spatial information and detail features. By fusing the fourth intermediate spectral feature image after spectral feature extraction and the spectral projection image, features at different levels can be fused, information loss can be reduced, the robustness of the virtual coloring model can be improved, and the accuracy of subsequent virtual coloring can be improved.

[0068] Specifically, the fourth intermediate spectrum characteristic graph and the spectrum projection graph are added to obtain the first spectrum characteristic graph.

[0069] Step S205, the first spectral feature map is processed in sequence by the encoder and the decoder in the virtual coloring model to obtain a virtual coloring intermediate map.

[0070] Optionally, an input projection module (Input Projection) may be provided before the encoder to map the first spectral feature map to a required feature space, which is generally a feature space of lower dimension than the original data, so as to reduce the amount of calculation and improve efficiency.

[0071] The encoder (Encoder Layer) and decoder (Decoder Layer) each include multiple convolutional layers. The encoder is used to convert the first spectral feature map into a series of intermediate feature representations to capture the feature information contained in the first spectral feature map, such as texture, shape, color, etc. The decoder receives the output of the encoder and converts it into a target output. The encoder and decoder can be implemented using modules with corresponding functions in related technologies. Exemplarily, the convolutional layers in the encoder and decoder are 3×3 convolutional layers.

[0072] Optionally, the encoder also includes a feedforward enhancement network and a multi-scale modem, and the feedforward enhancement network (Feed-Forward Neural Network, FFNN) is used to perform nonlinear transformation, feature enhancement and dimensionality change, capture more complex features and representations, increase the expressiveness of the model, and handle nonlinear relationships. Since the feedforward network in the related art has limitations in capturing local context information, in this embodiment, a deep convolution is added to the feedforward network in the related art to form a feedforward enhancement network, which improves the continuity and effectiveness when extracting local information. The multi-scale modem (Multi-Scale Modulation and Demodulation Modem) can improve coding efficiency and generalization ability, enhance the processing ability of complex and non-explicit mapping functions, and thus improve the accuracy and efficiency of data assimilation. Since relying only on local feature information of a certain scale is not comprehensive enough, and the computational complexity of extracting feature information globally is too high and it is easy to cause problems such as imperfect information extraction, a multi-scale modem is used so that the network can extract image feature information from different scales for learning during the learning process, so that the generated image is more realistic.

[0073] Optionally, an output projection module (Output Projection) may be provided after the decoder to map the virtual stained intermediate image back to the original data space or the target data space.

[0074] Optionally, a normalization layer can be set after the output projection module to adjust the distribution of the virtual coloring intermediate image, speed up the training speed and improve the model stability.

[0075] Optionally, the virtual coloring model also includes a local enhancement window Transformer block, which is set between the encoder and the decoder. After the first spectral feature map is processed by the encoder in the virtual coloring model, the first spectral feature map processed by the encoder is processed by the local enhancement window Transformer block for non-overlapping window self-attention processing. Compared with the Transformer global window in the related art, the local enhancement window Transformer block can effectively extract feature information while taking into account the contextual relationship, while significantly reducing the computational complexity of the model, making the model more lightweight.

[0076] Specifically, the first spectral feature map is processed in sequence through the input projection module, encoder, local enhancement window Transformer block, decoder, output projection module and normalization layer in the virtual coloring model to obtain a virtual coloring intermediate map.

[0077] Step S206, processing the hyperspectral microscopic image through the dimension reduction convolution layer in the virtual staining model to obtain a second spectral feature map.

[0078] By processing the hyperspectral microscopy image through the dimensionality reduction convolution layer, the dimensions of the generated second spectral feature map and the virtual staining intermediate map can be guaranteed to be unified.

[0079] It should be noted that step S204 and step S205 are performed sequentially, step S204 and step S206 are performed in parallel, and the order in which step S204 and step S206 are performed is not limited.

[0080] Step S207, performing feature fusion on the second spectral feature map and the virtual coloring intermediate map to obtain a virtual coloring map.

[0081] Specifically, the second spectral characteristic map and the virtual dyeing intermediate map are added (Addition), so that the second spectral characteristic map that has not been virtually dyed and the virtual dyeing intermediate map that has been virtually dyed are further feature-fused, retaining the original features to the greatest extent, and obtaining a more realistic virtual dyeing map, so as to complete the virtual dyeing process of the hyperspectral microscopic image.

[0082] The virtual staining method provided in this embodiment first obtains a hyperspectral microscopic image, then extracts spectral features of the hyperspectral microscopic image through the spectral attention module in the virtual staining model to obtain a first spectral feature map, then processes the first spectral feature map through the encoder and decoder in the virtual staining model in turn to obtain a virtual staining intermediate map, then processes the hyperspectral microscopic image through the dimension reduction convolution layer in the virtual staining model to obtain a second spectral feature map, and finally fuses the second spectral feature map with the virtual staining intermediate map to obtain a virtual staining map. The above scheme does not need to rely on the judgment and operation of experts. By setting the spectral attention module, the hyperspectral microscopic image can be divided into two parts to extract spectral information and spatial information respectively. The two extraction processes do not interfere with each other, ensuring the integrity and accuracy of the spectral information and spatial information of the hyperspectral microscopic image. By fusing the features of the second spectral feature map that has not been virtually stained and the virtual staining intermediate map that has been virtually stained, the obtained virtual staining map can be made more realistic, ensuring the accuracy of virtual staining, and the operation process is simple and efficient, without having a practical impact on the hyperspectral microscopic image, and does not affect the repeated use of the hyperspectral microscopic image, and the scheme has good flexibility.

[0083] As one or more specific application embodiments of the embodiments of the present invention, the optimal implementation scheme or the scheme that the inventor most wants to embody is described below in combination with specific application scenarios.

[0084] Figure 5FIG. 1 is a flow chart of training a virtual dyeing model to be trained according to an embodiment of the present invention. Figure 5 As shown, firstly, training data collection is performed. In this embodiment, the staining method used is hematoxylin and eosin (HE) staining. The relevant technicians prepare the hyperspectral microscopic sample image, and the hyperspectral microscopic sample image is stained with hematoxylin and eosin (HE) to obtain the stained sample image. Then, the hyperspectral microscopic sample image and the stained sample image are preprocessed to train the virtual staining model (U-shaped Spectral Transformer, UST model) to be trained as a training set. Among them, the virtual staining model to be trained is constructed using the PyTorch framework, the training graphics card uses NVIDA TeslaA100 40G, the optimizer uses Adam, the optimizer parameters are set to 0.5 and 0.999, the learning rate is set to 2e-4, the learning rate is kept unchanged in the first 100 rounds of training, and the learning rate gradually decreases to 0 in the last 100 rounds, and a total of 200 rounds of training are performed. Then, the generalization ability of the virtual staining model to be trained can be verified through additional clinical data to determine whether the virtual staining model to be trained has converged. If the virtual staining model to be trained has not converged, the data set is expanded and the model parameters are adjusted, and the training is performed again until the virtual staining model to be trained converges, completing the model training process and obtaining the virtual staining model.

[0085] Figure 6 FIG. 1 is a flow chart of virtual dyeing by using a virtual dyeing model according to an embodiment of the present invention. Figure 6 As shown, when it is necessary to virtually stain clinical unstained slice data (hyperspectral microscopic images), the clinical unstained slice data are first preprocessed, and then the preprocessed clinical unstained slice data are input into the virtual staining model to obtain the slice data after virtual staining.

[0086] In this embodiment, a virtual dyeing device is also provided, which is used to implement the above-mentioned embodiments and preferred implementation modes, and the descriptions that have been made will not be repeated. As used below, the term "module" can implement a combination of software and / or hardware of a predetermined function. Although the device described in the following embodiments is preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and conceivable.

[0087] This embodiment provides a virtual dyeing device, such as Figure 7 As shown, including: An image acquisition unit 701 is used to acquire a hyperspectral microscopic image; A spectral attention unit 702 is used to process the hyperspectral microscopic image through a spectral attention module in a virtual staining model to obtain a first spectral feature map; wherein the spectral attention module includes an intermediate module for segmenting the hyperspectral microscopic image and extracting spectral information and spatial information respectively; The encoding and decoding unit 703 is used to process the first spectral feature map through the encoder and the decoder in the virtual coloring model in sequence to obtain a virtual coloring intermediate map; A dimensionality reduction convolution unit 704 is used to process the hyperspectral microscopic image through a dimensionality reduction convolution layer in a virtual staining model to obtain a second spectral feature map; The feature fusion unit 705 is used to perform feature fusion on the second spectral feature map and the virtual coloring intermediate map to obtain a virtual coloring map.

[0088] In an optional embodiment, the intermediate module includes a segmentation module and an instance normalization module; the spectral attention unit includes: A projection unit, used for projecting the hyperspectral microscopic image through the first convolutional layer to obtain a spectral projection image; A segmentation unit, configured to segment the spectrum projection image into a first intermediate spectrum characteristic image and a second intermediate spectrum characteristic image through a segmentation module; an instance normalization unit, configured to perform instance normalization processing on the first intermediate spectrum feature map through an instance normalization module to obtain a third intermediate spectrum feature map; a concatenation and convolution unit, configured to concatenate and perform convolution integration processing on the second intermediate spectral characteristic graph and the third intermediate spectral characteristic graph to obtain a fourth intermediate spectral characteristic graph; The fusion unit is used to perform feature fusion on the fourth intermediate spectrum feature map and the spectrum projection map to obtain a first spectrum feature map.

[0089] In an optional embodiment, the concatenation and convolution unit includes: A splicing unit, used for splicing the second intermediate spectrum characteristic graph and the third intermediate spectrum characteristic graph to obtain an intermediate spectrum characteristic splicing graph; A first convolution unit is used to perform a first convolution integral process on the intermediate spectrum feature splicing map through a second convolution layer to obtain a first intermediate spectrum feature splicing convolution map; an activation unit, configured to process the first intermediate spectral feature concatenated convolution map through an activation function to obtain a processed first intermediate spectral feature concatenated convolution map; The second convolution unit is used to perform a second convolution integration process on the processed first intermediate spectrum feature splicing convolution map through a third convolution layer to obtain a fourth intermediate spectrum feature map.

[0090] In an optional embodiment, the virtual staining model further includes a local enhancement window Transformer block; The encoding and decoding unit is also used for: The first spectral feature map processed by the encoder is processed by self-attention through a local enhancement window Transformer block.

[0091] In an optional embodiment, the device further includes a training unit, and the training unit includes: A sample acquisition unit, used to acquire a hyperspectral microscopic sample image and a corresponding stained sample image; The iterative training unit is used to iteratively train the virtual staining model to be trained through the hyperspectral microscopic sample image and the corresponding stained sample image until the loss function of the virtual staining model to be trained converges to obtain the virtual staining model.

[0092] In an optional implementation, the training unit further includes a preprocessing unit, and the preprocessing unit includes: An image processing unit is used to perform black and white correction, cropping and band selection on the hyperspectral microscopic sample image and the stained sample image respectively to obtain a processed hyperspectral microscopic sample image and a processed stained sample image; The image registration unit is used to perform image registration processing on the processed hyperspectral microscopic sample image and the processed stained sample image to obtain a registered hyperspectral microscopic sample image and a registered stained sample image.

[0093] The further functional description of each of the above modules and units is the same as that of the above corresponding embodiments and will not be repeated here.

[0094] The virtual dyeing device in this embodiment is presented in the form of a functional unit, where the unit refers to an ASIC (Application Specific Integrated Circuit) circuit, a processor and memory that executes one or more software or fixed programs, and / or other devices that can provide the above functions.

[0095] The embodiment of the present invention also provides a computer device having the above Figure 7 Virtual staining apparatus shown.

[0096] See also Figure 8 , Figure 8 is a schematic diagram of the structure of a computer device provided by an optional embodiment of the present invention, such as Figure 8As shown, the computer device comprises: one or more processors 10, memory 20, and interfaces for connecting various components, including high-speed interfaces and low-speed interfaces. Various components utilize different buses to communicate with each other, and can be installed on a common mainboard or installed in other ways as required. The processor can process the instructions executed in the computer device, including instructions stored in the memory or on the memory to display the graphical information of the GUI on an external input / output device (such as, a display device coupled to the interface). In some optional embodiments, if necessary, multiple processors and / or multiple buses can be used together with multiple memories and multiple memories. Equally, multiple computer devices can be connected, and each device provides part of the necessary operation (for example, as a server array, a group of blade servers, or a multi-processor system). Figure 8 A processor 10 is taken as an example.

[0097] The processor 10 may be a central processing unit, a network processor or a combination thereof. The processor 10 may further include a hardware chip. The hardware chip may be a dedicated integrated circuit, a programmable logic device or a combination thereof. The programmable logic device may be a complex programmable logic device, a field programmable gate array, a general purpose array logic or any combination thereof.

[0098] The memory 20 stores instructions executable by at least one processor 10, so that the at least one processor 10 executes the method shown in the above embodiment.

[0099] The memory 20 may include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function; the data storage area may store data created according to the use of the computer device, etc. In addition, the memory 20 may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some optional embodiments, the memory 20 may optionally include a memory remotely arranged relative to the processor 10, and these remote memories may be connected to the computer device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0100] The memory 20 may include a volatile memory, such as a random access memory; the memory may also include a non-volatile memory, such as a flash memory, a hard disk or a solid state drive; the memory 20 may also include a combination of the above types of memory.

[0101] The computer device also includes an input device 30 and an output device 40. The processor 10, the memory 20, the input device 30 and the output device 40 may be connected via a bus or other means. Figure 8 The example of connecting through bus is taken in the following.

[0102] The input device 30 can receive input digital or character information, and generate key signal input related to the user settings and function control of the computer device, such as a touch screen, a keypad, a mouse, a track pad, a touch pad, an indicator bar, one or more mouse buttons, a trackball, a joystick, etc. The output device 40 may include a display device, an auxiliary lighting device (e.g., an LED) and a tactile feedback device (e.g., a vibration motor), etc. The above-mentioned display device includes but is not limited to a liquid crystal display, a light emitting diode, a display and a plasma display. In some optional embodiments, the display device may be a touch screen.

[0103] The embodiment of the present invention also provides a computer-readable storage medium. The method according to the embodiment of the present invention can be implemented in hardware, firmware, or can be implemented as a computer code that can be recorded in a storage medium, or can be implemented as a computer code that is originally stored in a remote storage medium or a non-temporary machine-readable storage medium and will be stored in a local storage medium through a network download, so that the method described herein can be stored in such software processing on a storage medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware. Among them, the storage medium can be a magnetic disk, an optical disk, a read-only storage memory, a random access memory, a flash memory, a hard disk or a solid-state hard disk, etc.; further, the storage medium can also include a combination of the above types of memories. It can be understood that a computer, a processor, a microprocessor controller, or programmable hardware includes a storage component that can store or receive software or computer code. When the software or computer code is accessed and executed by a computer, a processor, or hardware, the method shown in the above embodiment is implemented.

[0104] A part of the present invention may be applied as a computer program product, such as a computer program instruction, which, when executed by a computer, can call or provide the method and / or technical solution according to the present invention through the operation of the computer. Those skilled in the art should understand that the existence of the computer program instruction in a computer-readable medium includes, but is not limited to, a source file, an executable file, an installation package file, etc., and accordingly, the way in which the computer program instruction is executed by the computer includes, but is not limited to: the computer directly executes the instruction, or the computer compiles the instruction and then executes the corresponding compiled program, or the computer reads and executes the instruction, or the computer reads and installs the instruction and then executes the corresponding installed program. Here, the computer-readable medium may be any available computer-readable storage medium or communication medium accessible to the computer.

[0105] Although the embodiments of the present invention have been described in conjunction with the accompanying drawings, those skilled in the art may make various modifications and variations without departing from the spirit and scope of the present invention, and such modifications and variations shall all fall within the scope of protection of the present invention.

Claims

1. A virtual coloring method, characterized in that: The method comprises: Acquire hyperspectral microscopy images; The spectral feature extraction of the hyperspectral microscopy image is performed through the spectral attention module in the virtual staining model to obtain a first spectral feature map; wherein the spectral attention module includes an intermediate module for segmenting the hyperspectral microscopy image and extracting spectral information and spatial information respectively; The first spectral feature map is processed sequentially by an encoder and a decoder in a virtual dyeing model to obtain a virtual dyeing intermediate map; Processing the hyperspectral microscopic image through a dimension reduction convolution layer in a virtual staining model to obtain a second spectral feature map; The second spectral feature map is feature-fused with the virtual staining intermediate map to obtain a virtual staining map.

2. The method according to claim 1, characterized in that The intermediate module includes a segmentation module and an instance normalization module; the spectral feature extraction of the hyperspectral microscopy image by the spectral attention module in the virtual staining model to obtain a first spectral feature map includes: Projecting the hyperspectral microscopic image through a first convolutional layer to obtain a spectral projection image; The spectral projection image is divided into a first intermediate spectral characteristic image and a second intermediate spectral characteristic image by a division module; Performing instance normalization processing on the first intermediate spectrum feature map by an instance normalization module to obtain a third intermediate spectrum feature map; The second intermediate spectrum characteristic graph and the third intermediate spectrum characteristic graph are concatenated and subjected to convolution integral processing to obtain a fourth intermediate spectrum characteristic graph; The fourth intermediate spectrum characteristic graph and the spectrum projection graph are feature fused to obtain a first spectrum characteristic graph.

3. The method according to claim 2, characterized in that The step of concatenating and convolutionally integrating the second intermediate spectral characteristic graph and the third intermediate spectral characteristic graph to obtain a fourth intermediate spectral characteristic graph comprises: Splicing the second intermediate spectrum characteristic graph and the third intermediate spectrum characteristic graph to obtain an intermediate spectrum characteristic splicing graph; Performing a first convolution integral process on the intermediate spectral feature splicing map through a second convolution layer to obtain a first intermediate spectral feature splicing convolution map; Processing the first intermediate spectral feature concatenated convolution map through an activation function to obtain a processed first intermediate spectral feature concatenated convolution map; The processed first intermediate spectral feature splicing convolution map is subjected to a second convolution integral process through a third convolution layer to obtain a fourth intermediate spectral feature map.

4. The method according to any one of claims 1 to 3, characterized in that: The virtual dyeing model also includes a local enhancement window Transformer block; After the first spectral feature map is processed by the encoder in the virtual staining model, the method further includes: The first spectral feature map processed by the encoder is processed by self-attention through a local enhancement window Transformer block.

5. The method according to claim 1, characterized in that The method further comprises: Acquire a hyperspectral microscopic sample image and a corresponding stained sample image; The virtual staining model to be trained is iteratively trained using the hyperspectral microscopic sample image and the corresponding stained sample image until the loss function of the virtual staining model to be trained converges to obtain the virtual staining model.

6. The method according to claim 5, characterized in that After acquiring the hyperspectral microscopic sample image and the corresponding stained sample image, the method further includes: Performing black-and-white correction, cropping, and band selection on the hyperspectral microscopic sample image and the stained sample image, respectively, to obtain a processed hyperspectral microscopic sample image and a processed stained sample image; The processed hyperspectral microscopic sample image and the processed stained sample image are subjected to image registration processing to obtain a registered hyperspectral microscopic sample image and a registered stained sample image.

7. A virtual dyeing device, characterized in that: The device comprises: An image acquisition unit, used for acquiring a hyperspectral microscopic image; A spectral attention unit, used for processing the hyperspectral microscopic image through a spectral attention module in a virtual staining model to obtain a first spectral feature map; wherein the spectral attention module includes an intermediate module for segmenting the hyperspectral microscopic image and extracting spectral information and spatial information respectively; An encoding and decoding unit, used for sequentially processing the first spectral feature map through an encoder and a decoder in a virtual staining model to obtain a virtual staining intermediate map; A dimensionality reduction convolution unit, used for processing the hyperspectral microscopic image through a dimensionality reduction convolution layer in a virtual staining model to obtain a second spectral feature map; A feature fusion unit is used to perform feature fusion on the second spectral feature map and the virtual coloring intermediate map to obtain a virtual coloring map.

8. A computer device, characterized in that: include: A memory and a processor, wherein the memory and the processor are communicatively connected to each other, the memory stores computer instructions, and the processor executes the virtual coloring method according to any one of claims 1 to 6 by executing the computer instructions.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a computer to execute the virtual coloring method according to any one of claims 1 to 6.

10. A computer program product, characterized in that The method comprises computer instructions for causing a computer to execute the virtual coloring method according to any one of claims 1 to 6.

Images