A digital pathological staining method based on a multi-modal learning double attention neural network

Abstract

The application discloses a digital pathological staining method based on a multimodal learning double-attention neural network, and comprises the following steps: inputting a hyperspectral image of an unstained pathological section into a staining submodel, and outputting a digital stained hyperspectral image of the same size by using the staining submodel; inputting a color card linear RGB image into a pre-trained imaging submodel to decouple a camera response curve, and fusing a microsystem light source spectral image collected in advance and the digital stained hyperspectral image to generate a digital stained linear RGB image; and inputting the digital stained linear RGB image into a pathological image enhancement submodel to adjust color balance of the image, so that the image is close to traditional chemical staining. The application has the advantages of simple process, low cost and easy integration, and an end-to-end linkage optimization strategy enables deep coupling between staining and imaging, and high-precision color reconstruction can be realized. In addition, based on parallel design and a deep learning algorithm, the application ensures fast staining speed and high efficiency.

Description

Technical Field

[0001] This invention relates to the fields of optical imaging and deep learning, and in particular to a digital pathological staining method based on a multimodal learning dual-attention neural network. Background Technology

[0002] In pathological examination, because it is difficult to distinguish the structure and morphology of tissue sections, chemical staining is necessary. Chemical reagents are used to induce different degrees of staining in the cell nuclei, cytoplasm, and intercellular substances within the tissue sections, thereby distinguishing different types of cells and tissue structures. Hematoxylin-eosin (H&E) staining is the most commonly used method for differentiating cell nuclei and cytoplasm.

[0003] Traditional pathological staining mainly includes chemical staining and digital staining based on machine learning-based unsupervised clustering. Due to different reagent formulations, scanners, section thicknesses, and staining times, traditional chemical staining can lead to significant differences in the appearance of similar tissues after staining. Furthermore, chemical staining requires a long time for staining and post-processing, and the reagents are toxic. On the other hand, digital staining based on machine learning-based unsupervised clustering has low clustering accuracy and is not suitable for complex tissue images. Summary of the Invention

[0004] To address the aforementioned shortcomings in existing technologies, this invention provides a digital pathological staining method based on a multimodal learning dual-attention neural network. This method solves the problems of traditional chemical staining leading to changes in tissue appearance, long staining time, and toxic reagents, as well as the low clustering accuracy of digital staining based on machine learning-based unsupervised clustering, which makes it unsuitable for complex tissue images.

[0005] To achieve the aforementioned objectives, the present invention employs the following technical solution: a digital pathological staining method based on a multimodal learning dual-attention neural network, comprising the following steps:

[0006] S1: Input the hyperspectral image of the unstained pathological section into the chromatograph. The chromatograph performs matrix transformation on the transmission spectrum of the input hyperspectral image within the channel and outputs a digitally stained hyperspectral image of the same size.

[0007] S2: Input the linear RGB image of the color chart into the pre-trained imaging sub-model to decouple the camera response curve, and fuse the pre-acquired light source spectrum image of the microscopic system and the digital staining hyperspectral image output by the staining sub-model. The digital staining linear RGB image is generated inside the network through the imaging integral model.

[0008] S3: Input the digitally stained linear RGB image into the pathological image enhancement sub-model, adjust the color balance of the image to make the image effect of digital pathological staining close to that of traditional chemical staining, and complete the digital pathological staining based on multimodal learning dual attention neural network.

[0009] The beneficial effects of the above scheme are as follows: Through the above technical scheme, a multi-scale convolutional neural network is established to calibrate the camera response curve, realize the linkage optimization of staining and imaging, and solve the problems of traditional chemical staining causing changes in tissue appearance, long staining time and toxic reagents, as well as the low accuracy of digital staining clustering based on machine learning unsupervised clustering, which is not suitable for complex tissue images.

[0010] Furthermore, the formula for the chromosome model in S1 is:

[0011] The hyperspectral image of the pathological section before staining is an M×M×N matrix Q;

[0012] The hyperspectral image of the stained pathological section is an M×M×N matrix Q';

[0013] Each pixel of a pre-stained pathological section hyperspectral image is a 1×1×N matrix Q. ij ;

[0014] Each pixel of a hyperspectral image of a stained pathological section is a 1×1×N matrix Q. i ' j ;

[0015] The formula for the chromosome model is:

[0016] Q' ij =H(Diag(Q) ij ))

[0017] Where H is a 1×1×N coloring mapping operator, Diag(·) is a diagonal matrix, and M and N are different parameters.

[0018] The beneficial effects of the above-mentioned further scheme are: the chromatograph reveals the changes in the transmission spectrum of pathological sections before and after staining through the mathematical means of matrix transformation, and the calculation formula of the chromatograph is obtained based on the hyperspectral images of pathological sections before and after staining.

[0019] Furthermore, in S1, a feedforward network for the chromatid model is established based on the Unet network architecture to learn the mapping relationship of transmission spectra between the same tissues before and after staining.

[0020] The beneficial effect of the above-mentioned further scheme is that, based on the chromatid model, a feedforward network of the chromatid model is established based on the Unet network architecture, which is mainly used to learn the mapping relationship of transmission spectra between the same tissues before and after staining.

[0021] Furthermore, at the skip connections in the Unet network architecture, an attention module combining spatial and channel-based attention is designed. This attention module employs patch-level spatial attention, discretizing the input features into several patches to analyze the correlation between different patches, as shown in the formula:

[0022]

[0023] Among them, s ij Let L represent the effect of the i-th patch on the j-th patch, L be the total number of patches, exp be an exponential function with base e, and Q be the resultant of the i-th patch. i For the i-th query statement, K j For the j-th query key value;

[0024] The attention module includes the following steps:

[0025] S1-1: Input features A∈R H×W×C The input is fed into a convolutional layer to generate feature map Q, feature map K, and feature map V, where Q ∈ R. H×W , K∈R H×W , V∈R H×W×C R represents the feature map, H is the feature map spatial length, W is the spatial width, and C is the channel dimension;

[0026] S1-2: Reshape feature maps Q and K to R using the reshape operation. (X×Y)×(P×P) Where X×Y is the number of patches and P×P is the patch size;

[0027] S1-3: Multiply the resized feature map Q and the transpose of feature map K, then obtain the spatial attention map S∈R through a softmax operation. (X×Y)×(X×Y) ;

[0028] S1-4: Reshape the feature map V to R using the reshape operation. N×(P×P×C) The feature map V and the spatial attention map S∈R (X×Y)×(X×Y) Multiply, then perform reshape and summation operations, and sum the results with the input features A∈R. H×W×C The final output B∈R is obtained by adding each point one by one. H×W×C .

[0029] The beneficial effects of the above-mentioned further solutions are: This invention designs an attention module at the skip connections of the existing Unet, which facilitates the analysis of correlations between different patches. Because the attention module focuses more on global information, it can accurately learn the intensity and spectral distribution of the original hyperspectral image. Furthermore, the input and output features have the same shape and size, making it plug-and-play.

[0030] Furthermore, the multi-scale convolutional neural network model established in S2 is used as an imaging sub-model to calibrate the camera response curve. The calibration method includes the following steps:

[0031] S2-1: Establish a multi-scale convolutional neural network model and output the reconstructed 381×3 spectral response curve matrix through splicing operations;

[0032] S2-2: Obtain subsets of light source data, object reflectivity, and camera response curves, and construct a linear RGB image dataset using an imaging integral model;

[0033] S2-3: Input the constructed linear RGB image dataset into the multi-scale convolutional neural network model and output the camera response curve;

[0034] S2-4: Based on the camera response curve, train a multi-scale convolutional neural network by minimizing the loss function using the backpropagation gradient descent algorithm;

[0035] S2-5: Extract the linear RGB image from the RAW image captured by any camera, input it into the trained multi-scale convolutional neural network, reconstruct the camera response curve, and complete the calibration of the camera response curve.

[0036] The beneficial effect of the above-mentioned further solution is that the present invention uses the established multi-scale convolutional neural network model as an imaging sub-model to provide key response curve information within the network.

[0037] Furthermore, the multi-scale convolutional neural network model in S2-1 includes continuous convolutional layers, multi-scale convolutional layers, and an output layer. The continuous convolutional layers include convolutional layers, max pooling layers, and Gamma layers. The multi-scale convolutional layers include three sets of scale convolutional layers, and each set of scale convolutional layers includes two parallel multi-branch architectures.

[0038] The beneficial effect of the above-mentioned further solutions is that the above technical solutions constitute the specific structure of multi-size convolutional neural network models, which facilitates the implementation of the overall solution.

[0039] Furthermore, the construction of the linear RGB image dataset in S2-2 includes the following formula:

[0040] Q k,xy =pER xy S k

[0041] Among them, Q k,xy For a linear RGB image, p represents the transmittance of the optical system and color filter array, E represents the spectrum of the illumination source, and R... xy S is the reflectivity of the object. k This is the camera response curve.

[0042] The beneficial effect of the above-mentioned further scheme is that, by using the above formula, a linear RGB image is obtained using the imaging integral model, which is then used to input the model into a multi-scale convolutional neural network model to obtain the camera response curve.

[0043] Furthermore, the formula for minimizing the loss function L in S2-4 is:

[0044] L = L MSE +λL reg

[0045] Among them, L MSE Let L be the mean square error between the true value and the reconstruction result, and λ be the coefficient of the L2 penalty term. reg This is the regularization loss.

[0046] The beneficial effect of the above further scheme is that it can train a multi-scale convolutional neural network by minimizing the loss function using the above formula.

[0047] Furthermore, the staining sub-model, imaging sub-model, and pathological image enhancement sub-model constitute a digital pathological staining network. The loss function of the pathological staining network is the perceptual loss, which is calculated through the convolutional outputs of the original image and the generated image in different layers of the VGG network. The formula is as follows:

[0048]

[0049] Among them, w i For the sub-loss weights, l i y represents the output of the last convolutional layer of each module in the VGG16 network, y and y' are ground truth images, and ||·||2 is the L2 loss.

[0050] The beneficial effect of the above-mentioned further solutions is that, through the above technical solutions, a digital pathological staining network is realized, and the loss function of the network is provided. Attached Figure Description

[0051] Figure 1 This is a flowchart of a digital pathological staining method based on a multimodal learning dual-attention neural network.

[0052] Figure 2 A coloring diagram of a dual-attention network for multimodal learning.

[0053] Figure 3 This is a diagram of the staining process in a digital pathology staining network. Detailed Implementation

[0054] The present invention will be further described below with reference to the accompanying drawings and specific embodiments.

[0055] like Figure 1As shown, a digital pathological staining method based on a multimodal learning dual-attention neural network includes the following steps:

[0056] S1: Input the hyperspectral image of the unstained pathological section into the chromatograph. The chromatograph performs matrix transformation on the transmission spectrum of the input hyperspectral image within the channel and outputs a digitally stained hyperspectral image of the same size.

[0057] S2: Input the linear RGB image of the color chart into the pre-trained imaging sub-model to decouple the camera response curve, and fuse the pre-acquired light source spectrum image of the microscopic system and the digital staining hyperspectral image output by the staining sub-model. The digital staining linear RGB image is generated inside the network through the imaging integral model.

[0058] S3: Input the digitally stained linear RGB image into the pathological image enhancement sub-model, adjust the color balance of the image to make the image effect of digital pathological staining close to that of traditional chemical staining, and complete the digital pathological staining based on multimodal learning dual attention neural network.

[0059] The formula for the chromosome model in S1 is:

[0060] The hyperspectral image of the pathological section before staining is an M×M×N matrix Q;

[0061] The hyperspectral image of the stained pathological section is an M×M×N matrix Q';

[0062] Each pixel of a pre-stained pathological section hyperspectral image is a 1×1×N matrix Q. ij ;

[0063] Each pixel of a hyperspectral image of a stained pathological section is a 1×1×N matrix Q. i ' j ;

[0064] The formula for the chromosome model is:

[0065] Q' ij =H(Diag(Q) ij ))

[0066] Where H is a 1×1×N coloring mapping operator, Diag(·) is a diagonal matrix, and M and N are different parameters.

[0067] S1 uses a forward network based on the Unet network architecture to build a chromogenic sub-model, which is used to learn the mapping relationship of transmission spectra between the same tissues before and after staining.

[0068] At the skip connections in the Unet network architecture, an attention module combining spatial and channel-based attention is designed. This attention module employs patch-level spatial attention, discretizing the input features into several patches to analyze the correlation between different patches, as shown in the formula:

[0069]

[0070] Among them, s ij Let L represent the effect of the i-th patch on the j-th patch, L be the total number of patches, exp be an exponential function with base e, and Q be the resultant of the i-th patch. i For the i-th query statement, K j For the j-th query key value;

[0071] The attention module includes the following steps:

[0072] S1-1: Input features A∈R H×W×C The input is fed into a convolutional layer to generate feature map Q, feature map K, and feature map V, where Q ∈ R. H×W , K∈R H×W , V∈R H×W×C R represents the feature map, H is the feature map spatial length, W is the spatial width, and C is the channel dimension;

[0073] S1-2: Reshape feature maps Q and K to R using the reshape operation. (X×Y)×(P×P) Where X×Y is the number of patches and P×P is the patch size;

[0074] S1-3: Multiply the resized feature map Q and the transpose of feature map K, then obtain the spatial attention map S∈R through a softmax operation. (X×Y)×(X×Y) ;

[0075] S1-4: Reshape the feature map V to R using the reshape operation. N×(P×P×C) The feature map V and the spatial attention map S∈R (X×Y)×(X×Y) Multiply, then perform reshape and summation operations, and sum the results with the input features A∈R. H×W×C The final output B∈R is obtained by adding each point one by one. H×W×C .

[0076] Because the attention module focuses more on global information, it can accurately learn the intensity and spectral distribution of the original hyperspectral image. Furthermore, the input and output features have the same shape and size, making it plug-and-play.

[0077] Spatial attention helps the network better understand the label of each pixel, resulting in more refined and accurate segmentation. Channel attention helps the network select the most relevant feature channels, thereby better achieving transmission spectrum transformation.

[0078] In S2, the multi-scale convolutional neural network model is established as an imaging sub-model for calibrating the camera response curve. The calibration method includes the following steps:

[0079] S2-1: Establish a multi-scale convolutional neural network model and output the reconstructed 381×3 spectral response curve matrix through splicing operations;

[0080] S2-2: Obtain subsets of light source data, object reflectivity, and camera response curves, and construct a linear RGB image dataset using an imaging integral model;

[0081] S2-3: Input the constructed linear RGB image dataset into the multi-scale convolutional neural network model and output the camera response curve;

[0082] S2-4: Based on the camera response curve, train a multi-scale convolutional neural network by minimizing the loss function using the backpropagation gradient descent algorithm;

[0083] S2-5: Extract the linear RGB image from the RAW image captured by any camera, input it into the trained multi-scale convolutional neural network, reconstruct the camera response curve, and complete the calibration of the camera response curve.

[0084] The multi-scale convolutional neural network model in S2-1 includes continuous convolutional layers, multi-scale convolutional layers, and an output layer. The continuous convolutional layers include convolutional layers, max pooling layers, and Gamma layers. The multi-scale convolutional layers include three sets of scale convolutional layers, and each set of scale convolutional layers includes two parallel multi-branch architectures.

[0085] Continuous convolutional layers are used to extract basic features of the input image; multi-scale convolutional layers are used to extract enough information from the feature maps output by the continuous convolutional layers to decouple the camera spectral response curve; the output layer is used to organize the extracted features and generate the R, G, and B channels of the spectral response curve respectively.

[0086] The following formula is used to construct the linear RGB image dataset in S2-2:

[0087] Q k,xy =pER xy S k

[0088] Among them, Q k,xy For a linear RGB image, p represents the transmittance of the optical system and color filter array, E represents the spectrum of the illumination source, and R... xyS is the reflectivity of the object. k This is the camera response curve.

[0089] The formula for minimizing the loss function L in S2-4 is:

[0090] L = L MSE +λL reg

[0091] Among them, L MSE Let L be the mean square error between the true value and the reconstruction result, and λ be the coefficient of the L2 penalty term. reg This is the regularization loss.

[0092] The staining sub-model, imaging sub-model, and pathological image enhancement sub-model constitute the digital pathological staining network. The loss function of the pathological staining network is the perceptual loss, which is calculated by the convolutional outputs of the original image and the generated image in different layers of the VGG network. The formula is as follows:

[0093]

[0094] Among them, w i For the sub-loss weights, l i y represents the output of the last convolutional layer of each module in the VGG16 network, y and y' are ground truth images, and ||·||2 is the L2 loss.

[0095] like Figure 2 As shown, in one embodiment of the present invention, a 128×128×92 hyperspectral image of an unstained pathological slide is input into a staining sub-model. Without altering the spatial relationships of the input image, the staining sub-model performs a matrix transformation on its transmission spectrum in the channel dimension, outputting a digitally stained hyperspectral image of the same size. Simultaneously, a 512×512×3 linear RGB image of a 140-color chart is input into a pre-trained imaging sub-model. The camera response curve is decoupled, and the pre-acquired spectrum of the microscopic system's light source and the digitally stained hyperspectral image output by the staining sub-model are fused. An imaging integral model generates a 128×128×3 digitally stained linear RGB image within the network, which is then input into the pathological image enhancement sub-model to adjust the image's color balance, making the digital pathological staining image effect closer to traditional chemical staining.

[0096] like Figure 3 As shown, this is the staining process of an input unstained hyperspectral image by a digital pathological staining network. Since there are many channels in the hyperspectral image, this scheme extracts three channels to generate pseudo-RGB images for visualization. From left to right, they are the unstained hyperspectral image, the digitally stained hyperspectral image generated by the staining sub-model, the digitally stained linear RGB image simulated by combining the response curve and the light source spectrum integral, and the color-corrected digitally stained RGB image. The staining time for a single image is 134ms.

[0097] This invention addresses the problems of traditional chemical staining, such as significant staining variations, long processing times, and toxicity. It proposes a deep learning-based digital pathological staining technique. Since the input is a hyperspectral image of an unstained pathological slide, the cumbersome chemical staining process is avoided, and the method is non-toxic. To address the low accuracy of clustering algorithms, a multi-scale convolutional neural network is established to calibrate the camera response curve and applied to a multimodal learning framework. This enables coordinated optimization of staining and imaging, resulting in deep coupling between the two processes. This invention is simple, low-cost, and easy to integrate. The end-to-end coordinated optimization strategy achieves deep coupling between staining and imaging, enabling high-precision color reconstruction compared to the ground truth. Furthermore, the parallel design and deep learning algorithm ensure fast staining speed and high efficiency.

[0098] Those skilled in the art will recognize that the embodiments described herein are intended to help the reader understand the principles of the invention, and should be understood that the scope of protection of the invention is not limited to such specific statements and embodiments. Those skilled in the art can make various other specific modifications and combinations based on the technical teachings disclosed in this invention without departing from the spirit of the invention, and these modifications and combinations are still within the scope of protection of the invention.

Claims

1. A digital pathological staining method based on a multimodal learning dual-attention neural network, characterized in that, Includes the following steps: S1: Input the hyperspectral image of the unstained pathological section into the chromatograph. The chromatograph performs matrix transformation on the transmission spectrum of the input hyperspectral image within the channel and outputs a digitally stained hyperspectral image of the same size. In S1, a feedforward network for the staining sub-model is established based on the Unet network architecture to learn the mapping relationship of transmission spectra between the same tissues before and after staining. At the skip connections in the Unet network architecture, an attention module combining spatial and channel-based attention is designed. This attention module employs patch-level spatial attention, discretizing the input features into several patches to analyze the correlation between different patches, as shown in the formula: in, For the first The patch is for the first The impact of each patch Total number of patches For An exponential function with base 0. For the first A query statement, For the first Each query key value; The attention module includes the following steps: S1-1: Input features The data are fed into a convolutional layer to generate feature maps Q, K, and V, where... , , , Representing feature maps, The length of the feature map space. For space width, The channel dimension; S1-2: The reshape operation is used to change the size of feature maps Q and K. ,in, For the number of patches, This refers to the patch size. S1-3: Multiply the resized feature map Q and the transpose of feature map K, then obtain the spatial attention map through a softmax operation. ; S1-4: The size of the feature map V is changed by the reshape operation. The feature map V and the spatial attention map Multiply, then perform reshape and summation operations, and add the input features. The final output is obtained by adding each point one by one. ; S2: Input the linear RGB image of the color chart into the pre-trained imaging sub-model to decouple the camera response curve, and fuse the pre-acquired light source spectrum image of the microscopic system and the digital staining hyperspectral image output by the staining sub-model. The digital staining linear RGB image is generated inside the network through the imaging integral model. In step S2, the multi-scale convolutional neural network model is established as an imaging sub-model for calibrating the camera response curve. The calibration method includes the following steps: S2-1: Establish a multi-scale convolutional neural network model and output the reconstructed 381×3 spectral response curve matrix through splicing operations; S2-2: Obtain subsets of light source data, object reflectivity, and camera response curves, and construct a linear RGB image dataset using an imaging integral model; S2-3: Input the constructed linear RGB image dataset into the multi-scale convolutional neural network model and output the camera response curve; S2-4: Based on the camera response curve, train a multi-scale convolutional neural network by minimizing the loss function using the backpropagation gradient descent algorithm; S2-5: Extract the linear RGB image from the RAW image captured by any camera, input it into the trained multi-scale convolutional neural network, reconstruct the camera response curve, and complete the calibration of the camera response curve. S3: Input the digitally stained linear RGB image into the pathological image enhancement sub-model, adjust the color balance of the image to make the image effect of digital pathological staining close to that of traditional chemical staining, and complete the digital pathological staining based on multimodal learning dual attention neural network.

2. The digital pathological staining method based on a multimodal learning dual-attention neural network according to claim 1, characterized in that, The formula for the chromosome model in S1 is: Hyperspectral images of pathological sections before staining are matrix ; Hyperspectral images of stained pathological sections are matrix ; A single pixel in a hyperspectral image of a pre-stained pathological section is matrix ; A single pixel in a hyperspectral image of a stained pathological section is matrix ; The formula for the chromosome model is: in, for The coloring mapping operator, It is a diagonal matrix. and For different parameters.

3. The digital pathological staining method based on a multimodal learning dual-attention neural network according to claim 2, characterized in that, The multi-scale convolutional neural network model in S2-1 includes continuous convolutional layers, multi-scale convolutional layers, and an output layer. The continuous convolutional layers include convolutional layers, max pooling layers, and Gamma layers. The multi-scale convolutional layers include three sets of scale convolutional layers, and each set of scale convolutional layers includes two parallel multi-branch architectures.

4. The digital pathological staining method based on a multimodal learning dual-attention neural network according to claim 3, characterized in that, The construction of the linear RGB image dataset in S2-2 includes the following formula: in, For linear RGB images, For the transmittance of optical systems and color filter arrays, For the spectrum of the lighting source, The reflectivity of an object This is the camera response curve.

5. The digital pathological staining method based on a multimodal learning dual-attention neural network according to claim 4, characterized in that, Minimize the loss function in S2-4 The formula is in, The mean square error between the true value and the reconstruction result. The coefficient of the L2 penalty term, This is the regularization loss.

6. The digital pathological staining method based on a multimodal learning dual-attention neural network according to claim 5, characterized in that, The staining sub-model, imaging sub-model, and pathological image enhancement sub-model constitute a digital pathological staining network, and the loss function of the pathological staining network is perceptual loss. The calculation is performed by using the convolutional outputs of the original image and the generated image in different layers of the VGG network, as shown in the formula: in, For the sub-loss weight, This refers to the output of the last convolutional layer in each module of the VGG16 network. and For the true value image, This is the L2 loss.

Images