Virtual staining method, device, computer device, storage medium and program product

Abstract

The present invention relates to the technical field of image processing, and discloses a virtual staining method, apparatus, computer device, storage medium and program product. The method includes: obtaining a hyperspectral microscopic image; extracting spectral features of the hyperspectral microscopic image through a spectral attention module in a virtual staining model to obtain a first spectral feature map; wherein, the spectral attention module includes an intermediate module for segmenting the hyperspectral microscopic image and separately extracting spectral information and spatial information; sequentially processing the first spectral feature map through an encoder and a decoder in the virtual staining model to obtain a virtual staining intermediate map; processing the hyperspectral microscopic image through a dimensionality reduction convolutional layer in the virtual staining model to obtain a second spectral feature map; fusing the features of the second spectral feature map and the virtual staining intermediate map to obtain a virtual staining map. In the above solution, when implementing the virtual staining function, the accuracy is good and the efficiency is high.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and particularly relates to a virtual staining method, device, computer device, storage medium and program product. Background Art

[0002] The principle of histological staining for tissue specimen sections is to label different biological elements using specific markers (such as isotopes, enzymes, metal ions, chemical dyes, etc.) according to the biochemical characteristics of various tissue cells, making the visualization of tissue and cell structures clearer.

[0003] In the related art, experts slice tissue specimens in a designated laboratory, then label and stain the slices according to experience, mount the stained tissue specimen slices on glass slides, and image them through a microscope to obtain the final microscopic image of the stained tissue specimen slice.

[0004] However, the above solution requires experts to operate, with high labor costs and high technical thresholds. The whole process is greatly affected by the subjective operation of experts, the process is cumbersome, the accuracy is low, the efficiency is low, and once a staining is completed, additional staining and further molecular analysis cannot be carried out, resulting in poor flexibility. Summary of the Invention

[0005] In view of this, the purpose of the present invention is to provide a virtual staining method, device, computer device, storage medium and program product to solve the problems of low accuracy, low efficiency and poor flexibility in staining tissue specimen slices.

[0006] In a first aspect, the present invention provides a virtual staining method, which includes:

[0007] Obtain a hyperspectral microscopic image;

[0008] Extract spectral features from the hyperspectral microscopic image through a spectral attention module in the virtual staining model to obtain a first spectral feature map; wherein, the spectral attention module includes an intermediate module for segmenting the hyperspectral microscopic image and separately extracting spectral information and spatial information;

[0009] Process the first spectral feature map through an encoder and a decoder in the virtual staining model in sequence to obtain an intermediate virtual staining map;

[0010] Process the hyperspectral microscopic image through a dimensionality reduction convolutional layer in the virtual staining model to obtain a second spectral feature map;

[0011] Fuse the features of the second spectral feature map and the intermediate virtual staining map to obtain a virtual staining map.

[0012] In an alternative embodiment, the intermediate module includes a segmentation module and an instance normalization module; the spectral feature extraction of the hyperspectral microscopic image by the spectral attention module in the virtual staining model to obtain a first spectral feature map includes:

[0013] Project the hyperspectral microscopic image through a first convolutional layer to obtain a spectral projection map;

[0014] Segment the spectral projection map into a first intermediate spectral feature map and a second intermediate spectral feature map through the segmentation module;

[0015] Perform instance normalization processing on the first intermediate spectral feature map through the instance normalization module to obtain a third intermediate spectral feature map;

[0016] Stitch and perform convolutional integration processing on the second intermediate spectral feature map and the third intermediate spectral feature map to obtain a fourth intermediate spectral feature map;

[0017] Fuse the features of the fourth intermediate spectral feature map and the spectral projection map to obtain a first spectral feature map.

[0018] In an alternative embodiment, the stitching and convolutional integration processing of the second intermediate spectral feature map and the third intermediate spectral feature map to obtain a fourth intermediate spectral feature map includes:

[0019] Stitch the second intermediate spectral feature map and the third intermediate spectral feature map to obtain an intermediate spectral feature stitching map;

[0020] Perform first convolutional integration processing on the intermediate spectral feature stitching map through a second convolutional layer to obtain a first intermediate spectral feature stitching convolutional map;

[0021] Process the first intermediate spectral feature stitching convolutional map through an activation function to obtain a processed first intermediate spectral feature stitching convolutional map;

[0022] Perform second convolutional integration processing on the processed first intermediate spectral feature stitching convolutional map through a third convolutional layer to obtain a fourth intermediate spectral feature map.

[0023] In an alternative embodiment, the virtual staining model further includes a local enhancement window Transformer block;

[0024] After processing the first spectral feature map by the encoder in the virtual staining model, the method further includes:

[0025] Perform self-attention processing on the first spectral feature map processed by the encoder through the local enhancement window Transformer block.

[0026] In an alternative embodiment, the method further includes:

[0027] Obtaining a hyperspectral microscopic sample image and a corresponding stained sample image;

[0028] Iteratively training the virtual staining model to be trained through the hyperspectral microscopic sample image and the corresponding stained sample image until the loss function of the virtual staining model to be trained converges, so as to obtain a virtual staining model.

[0029] In an alternative embodiment, after obtaining the hyperspectral microscopic sample image and the corresponding stained sample image, the method further includes:

[0030] Performing black and white correction, cropping, and band selection on the hyperspectral microscopic sample image and the stained sample image respectively to obtain a processed hyperspectral microscopic sample image and a processed stained sample image;

[0031] Performing image registration processing on the processed hyperspectral microscopic sample image and the processed stained sample image to obtain a registered hyperspectral microscopic sample image and a registered stained sample image.

[0032] In a second aspect, the present invention provides a virtual staining device, and the device includes:

[0033] An image acquisition unit for acquiring a hyperspectral microscopic image;

[0034] A spectral attention unit for processing the hyperspectral microscopic image through a spectral attention module in the virtual staining model to obtain a first spectral feature map; wherein, the spectral attention module includes an intermediate module for segmenting the hyperspectral microscopic image and respectively extracting spectral information and spatial information;

[0035] An encoding and decoding unit for sequentially processing the first spectral feature map through an encoder and a decoder in the virtual staining model to obtain an intermediate virtual staining map;

[0036] A dimensionality reduction convolution unit for processing the hyperspectral microscopic image through a dimensionality reduction convolution layer in the virtual staining model to obtain a second spectral feature map;

[0037] A feature fusion unit for fusing the second spectral feature map with the intermediate virtual staining map to obtain a virtual staining map.

[0038] In a third aspect, the present invention provides a computer device, comprising: a memory and a processor, which are communicatively connected to each other. The memory stores computer instructions, and the processor executes the computer instructions to execute the virtual staining method according to the first aspect or any corresponding embodiment thereof.

[0039] In a fourth aspect, the present invention provides a computer-readable storage medium, on which computer instructions are stored, and the computer instructions are used to cause a computer to execute the virtual staining method according to the first aspect or any corresponding embodiment thereof.

[0040] In a fifth aspect, the present invention provides a computer program product, comprising computer instructions, and the computer instructions are used to cause a computer to execute the virtual staining method according to the first aspect or any corresponding embodiment thereof.

[0041] The technical solution provided by the present invention may include the following beneficial effects:

[0042] For the virtual staining method provided by the present invention, first, a hyperspectral microscopic image is obtained, and then the spectral attention module in the virtual staining model is used to extract spectral features from the hyperspectral microscopic image to obtain a first spectral feature map. The spectral attention module includes an intermediate module for segmenting the hyperspectral microscopic image and separately extracting spectral information and spatial information. Then, the first spectral feature map is processed by the encoder and decoder in the virtual staining model in sequence to obtain an intermediate virtual staining map. Then, the hyperspectral microscopic image is processed by the dimensionality reduction convolutional layer in the virtual staining model to obtain a second spectral feature map. Finally, the second spectral feature map and the intermediate virtual staining map are subjected to feature fusion to obtain a virtual staining map. In the above solution, without relying on the judgment and operation of experts, by setting the spectral attention module, the hyperspectral microscopic image can be segmented into two parts to separately extract spectral information and spatial information, and the two extraction processes do not interfere with each other, ensuring the integrity and accuracy of the spectral information and spatial information of the hyperspectral microscopic image. By performing feature fusion on the second spectral feature map without virtual staining and the intermediate virtual staining map with virtual staining, the obtained virtual staining map can be made more realistic, ensuring the accuracy of virtual staining. Moreover, the operation process is simple and efficient, and it does not have an actual impact on the hyperspectral microscopic image and does not affect the repeated utilization of the hyperspectral microscopic image, and the solution has good flexibility. Description of the Drawings

[0043] To more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the following will briefly introduce the drawings required for the description of the specific embodiments or the prior art. Obviously, the drawings in the following description are some embodiments of the present invention. For those of ordinary skill in the art, without creative efforts, other drawings can also be obtained based on these drawings.

[0044] Figure 1 is a schematic flowchart of a virtual staining method according to an embodiment of the present invention;

[0045] Figure 2 is a schematic flowchart of another virtual staining method according to an embodiment of the present invention;

[0046] Figure 3 is a schematic structural diagram of a virtual staining model according to an embodiment of the present invention;

[0047] Figure 4 is a schematic structural diagram of a spectral attention module according to an embodiment of the present invention;

[0048] Figure 5 is a schematic flowchart of training a virtual staining model to be trained according to an embodiment of the present invention;

[0049] Figure 6 is a schematic flowchart of virtual staining through a virtual staining model according to an embodiment of the present invention;

[0050] Figure 7 is a schematic block diagram of a virtual staining device according to an embodiment of the present invention;

[0051] Figure 8 is a schematic hardware structure diagram of a computer device according to an embodiment of the present invention. Specific Embodiments

[0052] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the following will clearly and completely describe the technical solutions in the embodiments of the present invention with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are some, but not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts belong to the scope of protection of the present invention.

[0053] The principle of histological staining for tissue specimen sections is to label different biological elements using specific markers (such as isotopes, enzymes, metal ions, chemical dyes, etc.) according to the biochemical characteristics of various tissue cells, making the visualization of tissue and cell structures clearer.

[0054] In the related art, experts fix and embed tissue specimens in paraffin in a designated laboratory, then cut the tissue specimens into sections (usually about 2-10 μm thick), and then label and stain the sections according to experience. Then, the stained tissue specimen sections are mounted on glass slides and imaged through a microscope to obtain the final microscopic image of the stained tissue specimen sections.

[0055] However, the above-mentioned solution requires experts to operate and supervise, and the materials and equipment required are costly, with high costs and high technical thresholds. The entire process is greatly affected by the subjective operation of experts and is cumbersome, with low accuracy and low efficiency. Moreover, once a staining is completed, additional staining and further molecular analysis cannot be carried out, resulting in poor flexibility.

[0056] According to an embodiment of the present invention, an embodiment of a virtual staining method is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. And although the logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in a different order than here.

[0057] In this embodiment, a virtual staining method is provided, which can be used in desktop computers, laptop computers, servers, etc. Figure 1 It is a flowchart of the virtual staining method according to an embodiment of the present invention, as Figure 1 shown, and this process includes the following steps:

[0058] Step S101, obtain a hyperspectral microscopic image.

[0059] Optionally, section the tissue specimen, and collect the multi-band spectral information of the tissue specimen section through a microscope equipped with a hyperspectral camera. Perform fine segmentation in the spectral dimension, dividing it into multiple channels (for example, dozens to hundreds of channels), and each channel corresponds to a specific spectral band, to obtain a hyperspectral microscopic image containing rich spatial information and spectral information, thereby improving the accuracy of subsequent virtual staining. The hyperspectral camera can be installed above the microscope. The multi-band spectral information can include spectral information of hundreds of bands.

[0060] Step S102, extract spectral features from the hyperspectral microscopic image through the spectral attention module in the virtual staining model to obtain a first spectral feature map.

[0061] Among them, the spectral attention module includes an intermediate module for segmenting the hyperspectral microscopic image and separately extracting spectral information and spatial information. By setting the intermediate module, the hyperspectral microscopic image can be segmented and spectral information and spatial information can be separately extracted, ensuring that the process of extracting spectral information and the process of extracting spatial information do not interfere with each other, ensuring the integrity of the spectral information and spatial information of the hyperspectral microscopic image, improving the accuracy of subsequent virtual staining, and making the spectral information and spatial morphology of the subsequent generated virtual staining map more real and accurate.

[0062] Step S103, sequentially process the first spectral feature map through the encoder and decoder in the virtual staining model to obtain an intermediate virtual staining map.

[0063] The encoder is used to convert the first spectral feature map into a series of intermediate feature representations to capture the feature information contained in the first spectral feature map, such as texture, shape, color, etc. The decoder receives the output of the encoder and converts it into the target output. The encoder and decoder can be implemented using modules with corresponding functions in related technologies.

[0064] Step S104, process the hyperspectral microscopic image through the dimensionality reduction convolutional layer in the virtual staining model to obtain a second spectral feature map.

[0065] It should be noted that step S102 and step S103 are carried out sequentially, step S102 and step S104 are carried out in parallel, and the order of execution of step S102 and step S104 is not limited.

[0066] Step S105, perform feature fusion on the second spectral feature map and the intermediate virtual staining map to obtain a virtual staining map.

[0067] By performing feature fusion on the second spectral feature map and the intermediate virtual staining map, the second spectral feature map without virtual staining and the intermediate virtual staining map after virtual staining are further feature fused, retaining the original features to the greatest extent, and obtaining a more realistic virtual staining map to complete the virtual staining process of the hyperspectral microscopic image.

[0068] The virtual staining method provided in this embodiment first obtains a hyperspectral microscopic image, then extracts spectral features from the hyperspectral microscopic image through the spectral attention module in the virtual staining model to obtain a first spectral feature map, and then sequentially processes the first spectral feature map through the encoder and decoder in the virtual staining model to obtain an intermediate virtual staining map. Then, the hyperspectral microscopic image is processed through the dimensionality reduction convolutional layer in the virtual staining model to obtain a second spectral feature map. Finally, the second spectral feature map and the intermediate virtual staining map are fused in features to obtain a virtual staining map. In the above solution, it is not necessary to rely on the judgment and operation of experts. By setting the spectral attention module, the hyperspectral microscopic image is segmented into two parts to extract spectral information and spatial information respectively, and the two extraction processes do not interfere with each other, ensuring the integrity and accuracy of the spectral information and spatial information of the hyperspectral microscopic image. By fusing the second spectral feature map without virtual staining and the intermediate virtual staining map after virtual staining, the obtained virtual staining map can be made more realistic, ensuring the accuracy of virtual staining. Moreover, the operation process is simple and efficient, without having an actual impact on the hyperspectral microscopic image and not affecting the repeated utilization of the hyperspectral microscopic image. The solution has good flexibility.

[0069] In this embodiment, a virtual staining method is provided, which can be used in desktop computers, laptop computers, servers, etc. Figure 2 It is a flowchart of the virtual staining method according to an embodiment of the present invention, as Figure 2 shown, and this process includes the following steps:

[0070] Step S201, obtain a hyperspectral microscopic sample image and a corresponding stained sample image.

[0071] The hyperspectral microscopic sample image is an image obtained after microscopic imaging of an unstained tissue specimen section. The stained sample image corresponds to the hyperspectral microscopic sample image one by one, and is an image obtained after real staining of the cell nucleus and matrix of the unstained hyperspectral microscopic sample image. The hyperspectral microscopic sample image and the stained sample image form a training set and a validation set. Among them, an expert can stain the hyperspectral microscopic sample image through a staining method in related technologies to obtain a corresponding stained sample image. The staining method can be hematoxylin and eosin (HE) staining, Masson's trichrome staining (MT), periodic acid Schiff staining (PAS), immunohistochemistry (IHC) staining, etc., which will not be elaborated here.

[0072] Optionally, after obtaining the hyperspectral microscopic sample image and the corresponding stained sample image, the hyperspectral microscopic sample image and the stained sample image can be preprocessed. Specifically, the hyperspectral microscopic sample image and the stained sample image are respectively subjected to black and white correction, cropping, and band selection. Bands with high signal-to-noise ratio are retained, and then filtering and noise reduction processing is performed to further improve the signal-to-noise ratio, obtaining the processed hyperspectral microscopic sample image and the processed stained sample image. Black and white correction corrects the color difference of the hyperspectral microscopic sample image and the stained sample image through a standard white board and a standard black board, which can eliminate the influence of light intensity and baseline drift on the hyperspectral microscopic sample image and the stained sample image, and improve the accuracy and consistency of the hyperspectral microscopic sample image and the stained sample image. Cropping can perform region selection on the hyperspectral microscopic sample image and the stained sample image, extract the effective information within the region of interest, reduce the interference of irrelevant data, and improve the efficiency and accuracy of subsequent image processing. Band selection can retain useful information while reducing the information redundancy of the hyperspectral microscopic sample image and the stained sample image, reduce the data volume, save resources, and improve the accuracy and efficiency of subsequent image processing. Then, the processed hyperspectral microscopic sample image and the processed stained sample image are subjected to image registration processing, so that the processed hyperspectral microscopic sample image and the corresponding processed stained sample image achieve pixel-level registration, obtaining the registered hyperspectral microscopic sample image and the registered stained sample image.

[0073] Step S202: Iteratively train the virtual staining model to be trained through the hyperspectral microscopic sample image and the corresponding stained sample image until the loss function of the virtual staining model to be trained converges, so as to obtain the virtual staining model.

[0074] The purpose of training the virtual staining model to be trained is to make the virtual staining sample image obtained after the virtual staining model to be trained performs virtual staining processing on the hyperspectral microscopic sample image as close as possible to the stained sample image corresponding to the hyperspectral microscopic sample image. That is to say, the model structure of the virtual staining model to be trained is the same as that of the virtual staining model, and there are differences in the specific parameters of the model. The virtual staining model is the virtual staining model to be trained whose performance reaches the actual requirements after multiple trainings.

[0075] Optionally, a loss function based on pixel difference is set, and the pixel difference between the virtual stained image (virtual staining sample image) output after inputting the unstained image (hyperspectral microscopic sample image) into the virtual staining model to be trained and the image (stained sample image) that has been chemically stained and input into the virtual staining model to be trained is used as the loss value to determine whether the virtual staining model to be trained converges, and the converged virtual staining model to be trained is used as the virtual staining model for subsequent realization of the virtual staining function.

[0076] Exemplarily, the loss function formula based on pixel difference is as follows:

[0077]

[0078] Wherein, is the stained sample image, represents the virtual stained sample image, is a very small constant, usually 10 -3 .

[0079] Specifically, during model training, first, the spectral attention module in the virtual staining model to be trained extracts spectral features from the stained sample image corresponding to the hyperspectral microscopic sample image to obtain the first sample spectral feature map; wherein, this spectral attention module includes an intermediate module for segmenting the hyperspectral microscopic sample image and the corresponding stained sample image and respectively extracting spectral information and spatial information; then, the first sample spectral feature map is processed sequentially through the encoder and decoder in the virtual staining model to obtain the virtual staining intermediate map; then, the hyperspectral microscopic sample image is processed through the dimensionality reduction convolutional layer in the virtual staining model to obtain the second sample spectral feature map; finally, the second sample spectral feature map and the sample virtual staining intermediate map are feature fused to obtain the virtual stained sample image.

[0080] Optionally, when extracting spectral features from the stained sample image, the stained sample image is projected through the first convolutional layer to align the format of the stained sample image with that of the hyperspectral microscopic sample image, especially the number of channels, that is, the number of channels of the three-channel stained sample image is expanded to be the same as that of the hyperspectral microscopic sample image, facilitating subsequent encoding processing in the encoder.

[0081] It should be noted that the stained sample image is an RGB (Red, Green, Blue) image, containing three channels, while the hyperspectral microscopic sample image contains spectral information of multiple channels (for example, dozens to hundreds of channels). That is to say, the spectral information contained in the stained sample image is very little. Although it also participates in the spectral feature extraction and encoding and decoding processes, the processing effect is limited. The stained sample image is mainly used to compare the pixel difference with the corresponding hyperspectral microscopic sample image as the loss value to judge whether the virtual staining model to be trained converges.

[0082] Step S203, obtain the hyperspectral microscopic image.

[0083] This hyperspectral microscopic image is an image obtained after microscopic imaging of an unstained tissue specimen section that actually needs to be virtually stained. For details, please refer to Figure 1 Step S101 of the illustrated embodiment, which will not be elaborated here.

[0084] Step S204: Extract spectral features from the hyperspectral microscopic image through the spectral attention module in the virtual staining model to obtain a first spectral feature map.

[0085] Specifically, the above step S204 includes:

[0086] Step S2041: Project the hyperspectral microscopic image through a first convolutional layer to obtain a spectral projection map.

[0087] Figure 3 It is a schematic structural diagram of the virtual staining model according to an embodiment of the present invention. Figure 4 It is a schematic structural diagram of the spectral attention module according to an embodiment of the present invention. After obtaining the hyperspectral microscopic image, spectral features of the hyperspectral microscopic image are extracted through the spectral attention module (SA). Specifically, the hyperspectral microscopic image is projected to a target size through a first convolutional layer (Conv.) to obtain a spectral projection map, which can be expressed by the following formula:

[0088] F eHSI = Conv(F HSI )

[0089] where F HSI represents the hyperspectral microscopic image, and Conv(F HSI ) represents projecting the hyperspectral microscopic image through the first convolutional layer, and F eHSI represents the spectral projection map. Through projection, it can ensure that the input hyperspectral microscopic image is aligned with the format of the required virtual staining map.

[0090] Exemplarily, the process of projecting the hyperspectral microscopic image through the first convolutional layer is a coupled affine transformation of a reversible neural operator. The coupled affine transformation is a linear transformation that can perform operations such as translation, rotation, and scaling on the input data, enabling the input data to be processed from different perspectives and scales. In the feature extraction stage of the neural network, the affine coupling transformation can help extract the features of the input data, enabling the neural network to better understand the meaning of the input data. Here, the coupled affine transformation of the reversible neural operator extracts the spectral features of the hyperspectral microscopic image, and each neural operator can effectively extract the spectral features of the hyperspectral microscopic image without losing spatial features, thus ensuring the details and authenticity of the generated spectral projection map.

[0091] Exemplarily, the first convolutional layer can be a 1×1 convolutional layer. The 1×1 convolutional layer is mainly used to reduce the number of channels of the hyperspectral microscopic image. Compared with the ordinary 3×3 convolutional layer, the 1×1 convolutional layer can greatly increase the non-linear characteristics by using the subsequent non-linear activation function on the premise of keeping the feature scale unchanged (i.e., without losing resolution).

[0092] Step S2042: The spectral projection map is segmented into a first intermediate spectral feature map and a second intermediate spectral feature map by the segmentation module.

[0093] Among them, the intermediate module includes a segmentation module and an instance normalization module.

[0094] Specifically, the process of segmenting the spectral projection map by the segmentation module can be expressed by the following formula:

[0095] F 1eHSI ,F 2eHI =split(F eHSI )

[0096] Among them, split(F eHSI ) means to segment the spectral projection map F eHSI , F 1eHSI is the first intermediate spectral feature map, and F 2eHI is the second intermediate spectral feature map.

[0097] Dividing the spectral projection map into two different paths (the first intermediate spectral feature map and the second intermediate spectral feature map) to extract spectral information and spatial information respectively can ensure that the data of the two paths do not interfere with each other during the feature extraction process, guarantee the integrity of the spectral information and spatial information, and avoid unnecessary information loss.

[0098] Optionally, the spectral projection map is evenly segmented (Spliting) into a first intermediate spectral feature map and a second intermediate spectral feature map by the segmentation module.

[0099] Step S2043: The first intermediate spectral feature map is subjected to instance normalization processing by the instance normalization module to obtain a third intermediate spectral feature map.

[0100] Performing instance normalization (IN) processing on the first intermediate spectral feature map can calculate the mean and variance of a single image on a single channel, stabilize the training process, alleviate the problems of gradient disappearance and gradient explosion, accelerate the convergence speed, and can also achieve style transfer without changing the content structure of the first intermediate spectral feature map, making the style features and content features more coordinated, thereby ensuring the authenticity of the subsequent generated virtual staining map.

[0101] Step S2044: Concatenate and perform convolution integral processing on the second intermediate spectral feature map and the third intermediate spectral feature map to obtain a fourth intermediate spectral feature map.

[0102] Specifically, first concatenate the second intermediate spectral feature map and the third intermediate spectral feature map to obtain an intermediate spectral feature concatenated map.

[0103] Next, perform first convolution integral processing on the intermediate spectral feature concatenated map through a second convolutional layer to obtain a first intermediate spectral feature concatenated convolution map. Exemplarily, the second convolutional layer (Conv.) is a 3×3 convolutional layer. Using a 3×3 convolutional layer to extract feature information can retain more spatial information, making the subsequently generated image more realistic in terms of spatial morphology.

[0104] Then, process the first intermediate spectral feature concatenated convolution map through an activation function to obtain a processed first intermediate spectral feature concatenated convolution map. Exemplarily, the activation function is LeakeyRelu.

[0105] Finally, perform second convolution integral processing on the processed first intermediate spectral feature concatenated convolution map through a third convolutional layer to integrate the extracted features and obtain a fourth intermediate spectral feature map. Exemplarily, the third convolutional layer (Conv.) is a 3×3 convolutional layer.

[0106] Optionally, the concatenation and convolution integral processing process is represented by the following formula:

[0107] F SIHSI =Conv{Cat[IN(F 1eHSI ),F 2eHSI}

[0108] where F SIHSI represents the fourth intermediate spectral feature map, IN(F 1eHSI ) represents the third intermediate spectral feature map obtained after instance normalization processing of the first intermediate spectral feature map F 1eHSI , Cat[IN(F 1eHSI ),F 2eHSI represents the intermediate spectral feature concatenated map obtained by concatenating the second intermediate spectral feature map F 2eHSI and the third intermediate spectral feature map IN(F 1eHSI ), and Conv{Cat[IN(F 1eHSI ),F 2eHSI} represents performing convolution integral processing on the intermediate spectral feature concatenated map Cat[IN(F 1eHSI ),F 2eHSI .

[0109] Step S2045: Perform feature fusion on the fourth intermediate spectral feature map and the spectral projection map to obtain the first spectral feature map.

[0110] More original spatial information and detailed features are retained in the spectral projection map. By performing feature fusion on the fourth intermediate spectral feature map after spectral feature extraction and the spectral projection map, features at different levels can be fused, information loss can be reduced, the robustness of the virtual staining model can be improved, and the accuracy of subsequent virtual staining can be enhanced.

[0111] Specifically, perform an addition operation (Addition) on the fourth intermediate spectral feature map and the spectral projection map to obtain the first spectral feature map.

[0112] Step S205: Process the first spectral feature map through the encoder and decoder in the virtual staining model in sequence to obtain an intermediate virtual staining map.

[0113] Optionally, an input projection module (Input Projection) can be set before the encoder to map the first spectral feature map to the required feature space, generally a feature space with a lower dimension than the original data, so as to reduce the computational amount and improve the efficiency.

[0114] Both the encoder (Encoder Layer) and decoder (Decoder Layer) contain multiple convolutional layers. The encoder is used to convert the first spectral feature map into a series of intermediate feature representations to capture the feature information contained in the first spectral feature map, such as texture, shape, color, etc. The decoder receives the output of the encoder and converts it into the target output. The encoder and decoder can be implemented using modules with corresponding functions in related technologies. Exemplarily, the convolutional layers in the encoder and decoder are 3×3 convolutional layers.

[0115] Optionally, the encoder further includes a feed-forward enhancement network and a multi-scale modem. The feed-forward enhancement network (Feed-Forward Neural Network, FFNN) is used for non-linear transformation, feature enhancement, and dimensional change, capturing more complex features and representations, increasing the expressive power of the model, and dealing with non-linear relationships. Since the feed-forward network in the related art has limitations in capturing local context information, in this embodiment, a depth convolution is added to the feed-forward network in the related art to form a feed-forward enhancement network, improving the continuity and effectiveness in extracting local information. The multi-scale modem (Multi-Scale Modulation and Demodulation Modem) can improve the coding efficiency and generalization ability, enhance the processing ability for complex and non-explicit mapping functions, thereby improving the accuracy and efficiency of data assimilation. Since relying solely on the local feature information of a certain scale is not comprehensive enough, and the computational complexity of global feature extraction is too high and prone to problems such as imperfect information extraction, a multi-scale modem is adopted, enabling the network to extract image feature information from different scales for learning during the learning process, making the generated image more realistic.

[0116] Optionally, an output projection module (Output Projection) can also be set after the decoder, which is used to map the virtual staining intermediate map back to the original data space or the target data space.

[0117] Optionally, a normalization layer (Normalization Layer) can also be set after the output projection module, which is used to adjust the distribution of the virtual staining intermediate map, accelerating the training speed and enhancing the model stability.

[0118] Optionally, the virtual staining model further includes a local enhancement window Transformer block, which is set between the encoder and the decoder. After the first spectral feature map is processed by the encoder in the virtual staining model, the first spectral feature map processed by the encoder is subjected to self-attention processing of non-overlapping windows through the local enhancement window Transformer block. Compared with the Transformer global window in the related art, the local enhancement window Transformer block can effectively extract feature information while taking into account the context relationship, and at the same time significantly reduce the computational complexity of the model, making the model more lightweight.

[0119] Specifically, the first spectral feature map is processed sequentially through the input projection module, encoder, local enhancement window Transformer block, decoder, output projection module, and normalization layer in the virtual staining model to obtain the virtual staining intermediate map.

[0120] Step S206: Process the hyperspectral microscopic image through the dimensionality reduction convolutional layer in the virtual staining model to obtain a second spectral feature map.

[0121] Processing the hyperspectral microscopic image through the dimensionality reduction convolutional layer can ensure that the dimensions of the generated second spectral feature map and the virtual staining intermediate map are unified.

[0122] It should be noted that Step S204 and Step S205 are carried out sequentially, Step S204 and Step S206 are carried out in parallel, and the order of execution of Step S204 and Step S206 is not limited.

[0123] Step S207: Perform feature fusion on the second spectral feature map and the virtual staining intermediate map to obtain a virtual staining map.

[0124] Specifically, perform addition processing (Addition) on the second spectral feature map and the virtual staining intermediate map, so that the second spectral feature map without virtual staining and the virtual staining intermediate map after virtual staining are further feature-fused, and the original features are retained to the greatest extent, obtaining a more realistic virtual staining map to complete the virtual staining process of the hyperspectral microscopic image.

[0125] The virtual staining method provided in this embodiment first obtains a hyperspectral microscopic image, then extracts spectral features of the hyperspectral microscopic image through the spectral attention module in the virtual staining model to obtain a first spectral feature map, and then sequentially processes the first spectral feature map through the encoder and decoder in the virtual staining model to obtain a virtual staining intermediate map. Then, process the hyperspectral microscopic image through the dimensionality reduction convolutional layer in the virtual staining model to obtain a second spectral feature map. Finally, perform feature fusion on the second spectral feature map and the virtual staining intermediate map to obtain a virtual staining map. In the above solution, it is not necessary to rely on the judgment and operation of experts. By setting the spectral attention module, the hyperspectral microscopic image can be segmented into two parts to extract spectral information and spatial information respectively, and the two extraction processes do not interfere with each other, ensuring the integrity and accuracy of the spectral information and spatial information of the hyperspectral microscopic image. By fusing the second spectral feature map without virtual staining and the virtual staining intermediate map after virtual staining, the obtained virtual staining map can be made more realistic, ensuring the accuracy of virtual staining, and the operation process is simple and efficient, without having an actual impact on the hyperspectral microscopic image and not affecting the repeated use of the hyperspectral microscopic image. The solution has good flexibility.

[0126] As one or more specific application embodiments of the present invention, the optimal implementation scheme or the scheme that the inventor most wants to embody will be described below in combination with specific application scenarios.

[0127] Figure 5It is a schematic flowchart for training a virtual staining model to be trained according to an embodiment of the present invention. As Figure 5 shown, first, training data is collected. In this embodiment, the staining method used is hematoxylin and eosin (HE) staining. Relevant technicians prepare hyperspectral microscopic sample images and perform hematoxylin and eosin (HE) staining on the hyperspectral microscopic sample images to obtain stained sample images. Then, the hyperspectral microscopic sample images and the stained sample images are preprocessed to be used as a training set to train the virtual staining model to be trained (U-shaped Spectral Transformer, UST model). Among them, the virtual staining model to be trained is constructed using the PyTorch framework, the training graphics card uses NVIDA Tesla A100 40G, the optimizer uses Adam, the optimizer parameters are set to 0.5 and 0.999, the learning rate is set to 2e-4, and the learning rate remains unchanged in the first 100 rounds of training and gradually decreases to 0 in the subsequent 100 rounds, with a total of 200 training rounds. Then, the generalization ability of the virtual staining model to be trained can be verified through additional clinical data collected, and it is judged whether the virtual staining model to be trained converges. If the virtual staining model to be trained does not converge, the data set is expanded and the model parameters are adjusted, and training is performed again until the virtual staining model to be trained converges, completing the model training process and obtaining the virtual staining model.

[0128] Figure 6 It is a schematic flowchart for virtual staining through a virtual staining model according to an embodiment of the present invention. As Figure 6 shown, when virtual staining of clinically unstained section data (hyperspectral microscopic images) is required, first, the clinically unstained section data is preprocessed, and then the preprocessed clinically unstained section data is input into the virtual staining model, and the section data after virtual staining can be obtained.

[0129] In this embodiment, a virtual staining device is also provided. This device is used to implement the above embodiments and preferred implementation manners, and those that have been described will not be repeated. As used below, the term "module" can be a combination of software and / or hardware that can achieve a predetermined function. Although the devices described in the following embodiments are preferably implemented in software, implementation in hardware, or a combination of software and hardware is also possible and contemplated.

[0130] This embodiment provides a virtual staining device, as Figure 7 shown, including:

[0131] An image acquisition unit 701, configured to acquire hyperspectral microscopic images;

[0132] The spectral attention unit 702 is used to process the hyperspectral microscopic image through the spectral attention module in the virtual staining model to obtain the first spectral feature map; wherein, the spectral attention module includes an intermediate module for segmenting the hyperspectral microscopic image and separately extracting spectral information and spatial information;

[0133] The encoding and decoding unit 703 is used to sequentially process the first spectral feature map through the encoder and decoder in the virtual staining model to obtain the virtual staining intermediate map;

[0134] The dimensionality reduction convolution unit 704 is used to process the hyperspectral microscopic image through the dimensionality reduction convolution layer in the virtual staining model to obtain the second spectral feature map;

[0135] The feature fusion unit 705 is used to fuse the features of the second spectral feature map and the virtual staining intermediate map to obtain the virtual staining map.

[0136] In an alternative embodiment, the intermediate module includes a segmentation module and an instance normalization module; the spectral attention unit includes:

[0137] The projection unit is used to project the hyperspectral microscopic image through the first convolutional layer to obtain the spectral projection map;

[0138] The segmentation unit is used to segment the spectral projection map into a first intermediate spectral feature map and a second intermediate spectral feature map through the segmentation module;

[0139] The instance normalization unit is used to perform instance normalization processing on the first intermediate spectral feature map through the instance normalization module to obtain the third intermediate spectral feature map;

[0140] The splicing and convolution unit is used to splice and convolve the second intermediate spectral feature map and the third intermediate spectral feature map to obtain the fourth intermediate spectral feature map;

[0141] The fusion unit is used to fuse the features of the fourth intermediate spectral feature map and the spectral projection map to obtain the first spectral feature map.

[0142] In an alternative embodiment, the splicing and convolution unit includes:

[0143] The splicing unit is used to splice the second intermediate spectral feature map and the third intermediate spectral feature map to obtain the intermediate spectral feature splicing map;

[0144] The first convolution unit is used to perform the first convolution integration processing on the intermediate spectral feature splicing map through the second convolutional layer to obtain the first intermediate spectral feature splicing convolution map;

[0145] An activation unit, configured to process the first intermediate spectral feature concatenated convolution map through an activation function to obtain a processed first intermediate spectral feature concatenated convolution map;

[0146] A second convolution unit, configured to perform a second convolution integration process on the processed first intermediate spectral feature concatenated convolution map through a third convolution layer to obtain a fourth intermediate spectral feature map.

[0147] In an optional implementation manner, the virtual staining model further includes a local enhancement window Transformer block;

[0148] The encoding and decoding unit is further configured to:

[0149] Perform self-attention processing on the first spectral feature map processed by the encoder through the local enhancement window Transformer block.

[0150] In an optional implementation manner, the apparatus further includes a training unit, and the training unit includes:

[0151] A sample acquisition unit, configured to acquire a hyperspectral microscopic sample image and a corresponding stained sample image;

[0152] An iterative training unit, configured to iteratively train the virtual staining model to be trained through the hyperspectral microscopic sample image and the corresponding stained sample image until the loss function of the virtual staining model to be trained converges, so as to obtain a virtual staining model.

[0153] In an optional implementation manner, the training unit further includes a preprocessing unit, and the preprocessing unit includes:

[0154] An image processing unit, configured to perform black and white correction, cropping, and band selection on the hyperspectral microscopic sample image and the stained sample image respectively to obtain a processed hyperspectral microscopic sample image and a processed stained sample image;

[0155] An image registration unit, configured to perform image registration processing on the processed hyperspectral microscopic sample image and the processed stained sample image to obtain a registered hyperspectral microscopic sample image and a registered stained sample image.

[0156] The further function descriptions of the above-mentioned various modules and units are the same as those in the corresponding above-mentioned embodiments, and will not be elaborated here.

[0157] The virtual staining apparatus in this embodiment is presented in the form of functional units. Here, the unit refers to an ASIC (Application Specific Integrated Circuit) circuit, a processor and a memory that execute one or more software or fixed programs, and / or other devices that can provide the above functions.

[0158] An embodiment of the present invention further provides a computer device having the above-mentioned Figure 7 virtual staining device shown.

[0159] Please refer to Figure 8 , Figure 8 which is a schematic structural diagram of a computer device provided by an optional embodiment of the present invention. As Figure 8 shown, the computer device includes: one or more processors 10, a memory 20, and an interface for connecting each component, including a high-speed interface and a low-speed interface. Each component communicates with each other using different buses and can be installed on a common motherboard or installed in other ways as needed. The processor can process instructions executed within the computer device, including instructions stored in the memory or on the memory to display graphical information of the GUI on an external input / output device (such as a display device coupled to the interface). In some optional embodiments, if necessary, multiple processors and / or multiple buses can be used together with multiple memories and multiple memories. Similarly, multiple computer devices can be connected, and each device provides some necessary operations (for example, as a server array, a set of blade servers, or a multi-processor system). Figure 8 In

[0160] FIG., one processor 10 is taken as an example.

[0161] The processor 10 can be a central processing unit, a network processor, or a combination thereof. Among them, the processor 10 can further include a hardware chip. The above-mentioned hardware chip can be an application-specific integrated circuit, a programmable logic device, or a combination thereof. The above-mentioned programmable logic device can be a complex programmable logic device, a field programmable gate array, a general array logic, or any combination thereof.

[0162] The memory 20 stores instructions executable by at least one processor 10, so that the at least one processor 10 executes the method shown in the above embodiment.

[0163] The memory 20 may include volatile memory, such as random access memory; the memory may also include non-volatile memory, such as flash memory, a hard disk, or a solid state drive; the memory 20 may further include a combination of the above types of memory.

[0164] The computer device further includes an input device 30 and an output device 40. The processor 10, the memory 20, the input device 30, and the output device 40 may be connected by a bus or other means. Figure 8 Taking connection by bus as an example.

[0165] The input device 30 can receive input digital or character information, and generate key signal inputs related to the user settings and function controls of the computer device, such as a touch screen, a keypad, a mouse, a trackpad, a touchpad, a pointing stick, one or more mouse buttons, a trackball, a joystick, etc. The output device 40 may include a display device, an auxiliary lighting device (e.g., an LED), and a haptic feedback device (e.g., a vibration motor), etc. The above display device includes, but is not limited to, a liquid crystal display, a light emitting diode, a display, and a plasma display. In some alternative embodiments, the display device may be a touch screen.

[0166] The embodiments of the present invention also provide a computer-readable storage medium. The methods according to the embodiments of the present invention can be implemented in hardware, firmware, or be implemented as computer code that can be recorded on a storage medium, or be implemented by downloading through a network the original computer code stored in a remote storage medium or a non-transitory machine-readable storage medium and to be stored in a local storage medium, so that the methods described herein can be stored as such software processing on a storage medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware. Among them, the storage medium may be a magnetic disk, an optical disk, a read-only memory, a random access memory, a flash memory, a hard disk, or a solid state drive, etc.; further, the storage medium may further include a combination of the above types of memory. It can be understood that a computer, a processor, a microprocessor controller, or programmable hardware includes a storage component that can store or receive software or computer code, and when the software or computer code is accessed and executed by the computer, the processor, or the hardware, the methods shown in the above embodiments are implemented.

[0167] A part of the present invention can be applied as a computer program product, such as computer program instructions, which, when executed by a computer, can invoke or provide the methods and / or technical solutions according to the present invention through the operations of the computer. Those skilled in the art should understand that the forms of existence of computer program instructions in a computer-readable medium include but are not limited to source files, executable files, installation package files, etc. Correspondingly, the ways for computer program instructions to be executed by a computer include but are not limited to: the computer directly executes the instructions, or the computer compiles the instructions and then executes the corresponding compiled program, or the computer reads and executes the instructions, or the computer reads and installs the instructions and then executes the corresponding installed program. Herein, the computer-readable medium can be any available computer-readable storage medium or communication medium accessible by the computer.

[0168] Although the embodiments of the present invention are described in conjunction with the accompanying drawings, those skilled in the art can make various modifications and variations without departing from the spirit and scope of the present invention, and such modifications and variations all fall within the protection scope of the present invention.

Claims

1. A virtual coloring method, characterized in that: The method comprises: Acquire hyperspectral microscopy images; The spectral feature extraction of the hyperspectral microscopy image is performed through the spectral attention module in the virtual staining model to obtain a first spectral feature map; wherein the spectral attention module includes an intermediate module for segmenting the hyperspectral microscopy image and extracting spectral information and spatial information respectively; The first spectral feature map is processed sequentially by an encoder and a decoder in a virtual dyeing model to obtain a virtual dyeing intermediate map; Processing the hyperspectral microscopic image through a dimension reduction convolution layer in a virtual staining model to obtain a second spectral feature map; Performing feature fusion on the second spectral feature map and the virtual dyeing intermediate map to obtain a virtual dyeing map; The intermediate module includes a segmentation module and an instance normalization module; the spectral feature extraction of the hyperspectral microscopy image by the spectral attention module in the virtual staining model to obtain a first spectral feature map includes: Projecting the hyperspectral microscopic image through a first convolutional layer to obtain a spectral projection image; The spectral projection image is divided into a first intermediate spectral characteristic image and a second intermediate spectral characteristic image by a division module; Performing instance normalization processing on the first intermediate spectrum feature map by an instance normalization module to obtain a third intermediate spectrum feature map; The second intermediate spectrum characteristic graph and the third intermediate spectrum characteristic graph are concatenated and subjected to convolution integral processing to obtain a fourth intermediate spectrum characteristic graph; The fourth intermediate spectrum characteristic graph and the spectrum projection graph are feature fused to obtain a first spectrum characteristic graph.

2. The method according to claim 1, characterized in that The step of concatenating and convolutionally integrating the second intermediate spectral characteristic graph and the third intermediate spectral characteristic graph to obtain a fourth intermediate spectral characteristic graph comprises: Splicing the second intermediate spectrum characteristic graph and the third intermediate spectrum characteristic graph to obtain an intermediate spectrum characteristic splicing graph; Performing a first convolution integral process on the intermediate spectral feature splicing map through a second convolution layer to obtain a first intermediate spectral feature splicing convolution map; Processing the first intermediate spectral feature concatenated convolution map through an activation function to obtain a processed first intermediate spectral feature concatenated convolution map; The processed first intermediate spectral feature splicing convolution map is subjected to a second convolution integral process through a third convolution layer to obtain a fourth intermediate spectral feature map.

3. The method according to claim 1 or 2, characterized in that: The virtual dyeing model also includes a local enhancement window Transformer block; After the first spectral feature map is processed by the encoder in the virtual staining model, the method further includes: The first spectral feature map processed by the encoder is processed by self-attention through a local enhancement window Transformer block.

4. The method according to claim 1, characterized in that: The method further comprises: Acquire a hyperspectral microscopic sample image and a corresponding stained sample image; The virtual staining model to be trained is iteratively trained using the hyperspectral microscopic sample image and the corresponding stained sample image until the loss function of the virtual staining model to be trained converges to obtain the virtual staining model.

5. The method according to claim 4, characterized in that After acquiring the hyperspectral microscopic sample image and the corresponding stained sample image, the method further includes: Performing black-and-white correction, cropping, and band selection on the hyperspectral microscopic sample image and the stained sample image, respectively, to obtain a processed hyperspectral microscopic sample image and a processed stained sample image; The processed hyperspectral microscopic sample image and the processed stained sample image are subjected to image registration processing to obtain a registered hyperspectral microscopic sample image and a registered stained sample image.

6. A virtual dyeing device, characterized in that: The device comprises: An image acquisition unit, used for acquiring a hyperspectral microscopic image; A spectral attention unit, used for processing the hyperspectral microscopic image through a spectral attention module in a virtual staining model to obtain a first spectral feature map; wherein the spectral attention module includes an intermediate module for segmenting the hyperspectral microscopic image and extracting spectral information and spatial information respectively; An encoding and decoding unit, used for sequentially processing the first spectral feature map through an encoder and a decoder in a virtual staining model to obtain a virtual staining intermediate map; A dimensionality reduction convolution unit, used for processing the hyperspectral microscopic image through a dimensionality reduction convolution layer in a virtual staining model to obtain a second spectral feature map; A feature fusion unit, used for performing feature fusion on the second spectral feature map and the virtual dyeing intermediate map to obtain a virtual dyeing map; The intermediate module includes a segmentation module and an instance normalization module; the spectral attention unit is also used for: Projecting the hyperspectral microscopic image through a first convolutional layer to obtain a spectral projection image; The spectral projection image is divided into a first intermediate spectral characteristic image and a second intermediate spectral characteristic image by a division module; Performing instance normalization processing on the first intermediate spectrum feature map by an instance normalization module to obtain a third intermediate spectrum feature map; The second intermediate spectrum characteristic graph and the third intermediate spectrum characteristic graph are concatenated and subjected to convolution integral processing to obtain a fourth intermediate spectrum characteristic graph; The fourth intermediate spectrum characteristic graph and the spectrum projection graph are feature fused to obtain a first spectrum characteristic graph.

7. A computer device, characterized in that: include: A memory and a processor, wherein the memory and the processor are communicatively connected to each other, computer instructions are stored in the memory, and the processor executes the virtual dyeing method according to any one of claims 1 to 5 by executing the computer instructions.

8. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a computer to execute the virtual coloring method according to any one of claims 1 to 5.

9. A computer program product, characterized in that The method comprises computer instructions for causing a computer to execute the virtual coloring method according to any one of claims 1 to 5.

Images