Medical hyperspectral image spectrum reconstruction method and device based on deep learning

Abstract

The invention provides a medical hyperspectral image spectrum reconstruction method and device based on deep learning, and relates to the field of image spectrum reconstruction, and the method comprises the steps: constructing a medical hyperspectral image spectrum reconstruction network; dividing the medical RGB-hyperspectral image pair set into a training image set and a test image set; training the medical hyperspectral image spectrum reconstruction network through the training image set to obtain a trained medical hyperspectral image spectrum reconstruction network; and obtaining a medical RGB image to be reconstructed from the test image set, and inputting the medical RGB image to be reconstructed into the trained medical hyperspectral image spectrum reconstruction network to obtain a reconstructed medical hyperspectral image. According to the invention, a brand new medical hyperspectral image spectrum reconstruction network is provided, and high-precision spectrum reconstruction of the medical hyperspectral image is realized by extracting local space information and global spectrum information at the same time.

Description

Technical Field

[0001] The present invention relates to the field of image spectrum reconstruction, and in particular to a method and device for spectrum reconstruction of medical hyperspectral images based on deep learning. Background Art

[0002] Medical hyperspectral images (MHSI) are a series of images that are accurately registered in continuous and subdivided spectral bands and acquired through hyperspectral imaging technology in medical research and clinical applications. These images cover adjacent and narrow wavelength ranges and together constitute a multidimensional data set containing rich spectral and spatial information.

[0003] Although MHSI has significant advantages in the field of medical imaging, its acquisition process is relatively complex and costly. Compared with standard RGB images, MHSI requires specialized custom equipment, and the imaging time is relatively long due to the need to collect data from multiple spectral bands. This has limited the widespread application of MHSI in clinical practice to a certain extent. On the contrary, RGB images, as the most common color image format, can be easily captured by any standard digital color camera, with mature technology and low cost. Given the close connection between RGB images and MHSI in color description, application scenarios, and potential conversion relationships, studying methods to reconstruct high-quality medical hyperspectral images from medical RGB images can significantly reduce the threshold for obtaining MHSI without sacrificing the richness of spectral information, and promote its wider application in the medical field.

[0004] In the field of medical image reconstruction, a variety of spatial super-resolution reconstruction methods have been developed. These methods aim to address the low signal-to-noise ratio (SNR) and low contrast-to-noise ratio (CNR) problems in medical image datasets generated by imaging techniques such as X-ray imaging, computed tomography (CT), magnetic resonance imaging (MRI), and ultrasound imaging. However, these methods do not address the high cost of acquiring medical hyperspectral images. Therefore, the current medical imaging field lacks spectral reconstruction methods for complex and larger datasets to overcome the limitations faced in acquiring medical hyperspectral images.

[0005] Spectral reconstruction is currently mainly used in the field of natural images. Since there may be a large number of solution spaces in the process of mapping from three spectral bands to more spectral bands, this is an unavoidable pathological conversion process. To solve this problem, methods based on prior knowledge and data-driven have been proposed. In the early stages of the development of spectral reconstruction technology, methods mainly rely on prior knowledge and use prior information such as sparsity, low rank and scene information to roughly model shallow feature representations. However, due to the simple modeling design and excessive reliance on prior knowledge, this method can easily have a negative impact on the accuracy and generation effect of the model. In order to further improve the accuracy and feature representation ability of reconstructed hyperspectral images (HSI), researchers have gradually developed data-driven methods based on deep learning. These methods mainly include convolutional neural networks (CNN), generative adversarial networks (GAN) and Transformer, among which the Transformer-based method has shown excellent performance in the reconstruction process.

[0006] Compared with natural images, biomedical cell images show unique foreground and background distinction characteristics. In biomedical cell images, the foreground area is mainly composed of cells, which have significant differences from the background area in grayscale value, morphological structure and texture characteristics. However, methods that directly use natural images face many challenges. On the one hand, due to spectral confusion, the edge contours of cells are easily misclassified as background during reconstruction, resulting in unclear cell contour information; at the same time, these methods often find it difficult to take into account the fidelity of the foreground area (cells) and the background area in the reconstruction, so that the reconstruction results may have blurred cell details or distorted background information. On the other hand, the fineness of the cell structure and the complexity of the texture characteristics also make these key information easily lost or damaged during the reconstruction process, which in turn affects the accuracy of subsequent analysis and diagnosis. Summary of the invention

[0007] In view of this, the purpose of the present invention is to provide a medical hyperspectral image spectral reconstruction method and equipment based on deep learning, which is used to solve the technical problems that the existing image reconstruction results may have blurred cell details or distorted background information, and the fineness of cell structure and the complexity of texture features also make these key information easily lost or damaged during the reconstruction process, thereby affecting the accuracy of subsequent analysis and diagnosis.

[0008] The present invention provides a method for spectral reconstruction of medical hyperspectral images based on deep learning, comprising the steps of:

[0009] S1: Construct a spectral reconstruction network for medical hyperspectral images based on CNN-Transformer;

[0010] S2: obtaining a medical RGB-hyperspectral image pair set, and dividing the medical RGB-hyperspectral image pair set into a training image set and a test image set;

[0011] S3: training the medical hyperspectral image spectral reconstruction network through the training image set to obtain a trained medical hyperspectral image spectral reconstruction network;

[0012] S4: Obtain a medical RGB image to be reconstructed from the test image set, input the medical RGB image to be reconstructed into a trained medical hyperspectral image spectral reconstruction network, and obtain a reconstructed medical hyperspectral image.

[0013] Preferred:

[0014] The spectral reconstruction network of medical hyperspectral images includes a shallow feature extraction module, a local space multi-scale adaptive feature fusion module, a position encoding-based Transformer module, and a channel integration module connected in sequence;

[0015] The shallow feature extraction module is connected with the positional encoding based Transformer module.

[0016] Preferably, the workflow of the medical hyperspectral image spectral reconstruction network is as follows:

[0017] Input the three-channel medical RGB image X into the shallow feature extraction module to obtain the feature map Y SFE ;

[0018] The feature map Y SFE Input the local space multi-scale adaptive feature fusion module to obtain the feature map Y' LMFE , the feature map Y' LMFE and feature map Y SFE Add together to get the feature map Y LMFE ;

[0019] The feature map Y LMFE Input the Transformer module based on position encoding to obtain the feature map Y PET ;

[0020] The feature map Y PET The input channel integration module obtains the reconstructed medical hyperspectral image Y.

[0021] Preferred:

[0022] The local space multi-scale adaptive feature fusion module includes: a first convolution module, a second convolution module, a third convolution module, a fourth convolution module, a fifth convolution module, a first 3×3conv module and a second 3×3conv module;

[0023] The shallow feature extraction module is connected to the first convolution module, the second convolution module, the third convolution module, the fourth convolution module and the fifth convolution module;

[0024] The first convolution module, the second convolution module, the third convolution module, the fourth convolution module and the fifth convolution module are connected to the first 3×3conv module and the second 3×3conv module.

[0025] Preferably, the workflow of the local space multi-scale adaptive feature fusion module is as follows:

[0026] Get the feature map Y of the input local space multi-scale adaptive feature fusion module SFE ;

[0027] The feature map Y SFE After 2x upsampling, the first convolution module and downsampling, the feature map X1 is obtained;

[0028] The feature map Y SFE After the second convolution module, the feature map X2 is obtained;

[0029] The feature map Y SFE After 2x downsampling, the third convolution module and upsampling, the feature map X3 is obtained;

[0030] The feature map Y SFE After 4 times downsampling, the fourth convolution module and upsampling, the feature map X4 is obtained;

[0031] The feature map Y SFE After 8 times downsampling, the fifth convolution module and upsampling, the feature map X5 is obtained;

[0032] The feature maps X1, X2, X3, X4 and X5 are concatenated in the channel dimension and input into the first 3×3conv module. After being processed by the Sigmoid function, the weight matrix w is obtained. The expression of the weight matrix w is:

[0033] w=Sigmoid(conv(concat(X1,X2,X3,X4,X5)))

[0034] Among them, concat() represents the concatenation operation of the channel dimension, conv() represents the convolution operation of the 3×3conv module, and Sigmoid() represents the sigmoid operation;

[0035] Multiply the feature map X1, feature map X2, feature map X3, feature map X4 and feature map X5 by the weight matrix w respectively to obtain feature map X'1, feature map X'2, feature map X'3, feature map X'4 and feature map X'5;

[0036] Add the feature map X'1, the feature map X'2, the feature map X'3, the feature map X'4 and the feature map X'5 to obtain the feature map X';

[0037] Concatenate the feature maps X', X1, X2, X3, X4 and X5 in the channel dimension and input them into the second 3×3conv module to obtain the feature map Y' LMFE , feature map Y' LMFE The expression is:

[0038] Y' LMFE =Conv(concat(X',X1,X2,X3,X4,X5)).

[0039] Preferred:

[0040] The Transformer module based on position coding includes a first PCSA module, a first LayerNorm module, a first FFN module, a downsampling module, an upsampling module, a second PCSA module, a second LayerNorm module and a second FFN module connected in sequence;

[0041] The local spatial multi-scale adaptive feature fusion module is connected to the first PCSA module and the first Layer Norm module;

[0042] The first PCSA module is connected to the downsampling module;

[0043] The first PCSA module is connected to the second PCSA module;

[0044] The up-sampling module is connected to the second Layer Norm module;

[0045] The second PCSA module and the second FFN module are connected to the channel integration module.

[0046] Preferred:

[0047] The first PCSA module and the second PCSA module have the same structure;

[0048] The first PCSA module includes: a first 1×1conv module, a first 3×3DW module, a second 1×1conv module, a second 3×3DW module, a third 1×1conv module and a third 3×3DW module;

[0049] The input end of the first PCSA module is connected to the first 1×1conv module, the second 1×1conv module and the third 1×1conv module;

[0050] The first 1×1conv module is connected to the first 3×3DW module, the second 1×1conv module is connected to the second 3×3DW module, and the third 1×1conv module is connected to the third 3×3DW module;

[0051] The first 3×3 DW module, the second 3×3 DW module and the third 3×3 DW module are connected to the output terminal of the first PCSA module.

[0052] Preferably, the workflow of the first PCSA module is specifically as follows:

[0053] Get the feature map F of the first PCSA module in ;

[0054] The feature map F in After the first 1×1conv module and the first 3×3DW module, the projection value Q is obtained;

[0055] The feature map F in After the second 1×1conv module and the second 3×3DW module, the projection value K is obtained;

[0056] The feature map F in After the third 1×1conv module and the third 3×3DW module, the projection value V is obtained;

[0057] The feature map F output by the first PCSA module is obtained by calculating the projection values ​​Q, K and V. out , feature map F out The calculation formula is:

[0058] Attention=Softmax((PE(K)+K)·(PE(Q)+Q) T ))

[0059] F out =Attention·(PE(V)+V) T

[0060] Among them, PE() represents the position embedding mechanism function, and Softmax() represents the normalized exponential function.

[0061] Preferably, step S3 specifically comprises:

[0062] S31: obtaining a training medical RGB image and a real medical hyperspectral image from a training image set, and adjusting parameters of a medical hyperspectral image spectral reconstruction network;

[0063] S32: input the training medical RGB image into a medical hyperspectral image spectral reconstruction network to obtain a predicted medical hyperspectral image;

[0064] S33: Calculate the RMSE loss between the predicted medical hyperspectral image and the real medical hyperspectral image;

[0065] S34: Repeat steps S31-S33 until the RMSE loss is within a preset range, and obtain a trained medical hyperspectral image spectral reconstruction network.

[0066] A medical hyperspectral image spectrum reconstruction device based on deep learning, comprising: a processor and a storage medium; the processor loads and executes instructions and data in the storage medium to implement the medical hyperspectral image spectrum reconstruction method based on deep learning.

[0067] The present invention has the following beneficial effects:

[0068] 1. A new spectral reconstruction network for medical hyperspectral images is proposed, including: shallow feature extraction module, local spatial multi-scale adaptive feature fusion module, position encoding-based Transformer module and channel integration module. By extracting local spatial information and global spectral information simultaneously, high-precision spectral reconstruction of medical hyperspectral images is achieved;

[0069] 2. The local spatial multi-scale adaptive feature fusion module adds an upsampling branch to the network structure to retain the fine structure of the cell. It extracts rich texture information, edge information and high-level abstract semantic information from the feature map through a multi-scale adaptive weighted fusion strategy, thereby improving the robustness of the feature map.

[0070] 3. The Transformer module based on position encoding improves the way of generating Q, K, and V, and embeds the position encoding into Q, K, and V, so that the generated channel self-attention map has the ability to pay attention to the relative position relationship within the spectrum, which helps to better integrate the interdependence between spectra, thereby further improving the reconstruction accuracy of the network;

[0071] 4. RMSE loss is used as the loss function in the training process of the spectral reconstruction network of medical hyperspectral images, so that the reconstruction network can balance the reconstruction importance of the foreground area and the background area, ensuring the reconstruction accuracy and reconstruction effect of the foreground area. BRIEF DESCRIPTION OF THE DRAWINGS

[0072] Figure 1 This is a flow chart of a method according to an embodiment of the present invention;

[0073] Figure 2 Structural diagram of the spectral reconstruction network for medical hyperspectral images;

[0074] Figure 3 It is the structural diagram of the local space multi-scale adaptive feature fusion module;

[0075] Figure 4This is the structural diagram of the Transformer module based on position encoding;

[0076] Figure 5 This is a system structure diagram of an embodiment of the present invention;

[0077] The realization of the purpose, functional features and advantages of the present invention will be further explained in conjunction with embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0078] It should be understood that the specific embodiments described herein are only used to explain the present invention, and are not used to limit the present invention.

[0079] Reference Figure 1 The present invention provides a method for spectral reconstruction of medical hyperspectral images based on deep learning, comprising the steps of:

[0080] S1: Construct a spectral reconstruction network for medical hyperspectral images based on CNN-Transformer;

[0081] As an example, refer to Figure 2 :

[0082] The spectral reconstruction network of medical hyperspectral images includes a shallow feature extraction module, a local space multi-scale adaptive feature fusion module, a position encoding-based Transformer module, and a channel integration module connected in sequence;

[0083] The shallow feature extraction module is connected with the positional encoding based Transformer module.

[0084] Specifically, the shallow feature extraction module and the channel integration module are both convolution combination modules consisting of two 3×3 convolutions and a PReLU activation function.

[0085] As an example:

[0086] The specific workflow of the medical hyperspectral image spectral reconstruction network is as follows:

[0087] Input the three-channel medical RGB image X into the shallow feature extraction module to obtain the feature map Y SFE ;

[0088] The feature map Y SFE Input the local space multi-scale adaptive feature fusion module to obtain the feature map Y' LMFE , the feature map Y' LMFE and feature map Y SFE Add together to get the feature map Y LMFE ;

[0089] The feature map Y LMFEInput the Transformer module based on position encoding to obtain the feature map Y PET ;

[0090] The feature map Y PET The input channel integration module obtains the reconstructed medical hyperspectral image Y.

[0091] Specifically, the input of the network is a three-channel RGB image The RGB image will be extracted through the shallow feature extraction module to obtain a feature map with shallow feature information of the image. Subsequently, the feature map is used as the input of the next module to enter the local space multi-scale adaptive feature fusion module proposed by the present invention, in which the present invention can obtain a new feature map that fuses multiple scale feature information. The feature map contains both shallow texture and edge features and abstract semantic features obtained with the deepening of feature extraction, which preserves the fine structure of cells and suppresses background noise from the spatial dimension. In order to capture the long-distance dependencies between spectra, the present invention inputs the feature map into the Transformer module based on position encoding. By incorporating position information, the generated attention map is endowed with the ability to pay attention to the relative position relationship within the spectrum, which helps to better integrate the interdependence between spectra and output a new feature map. Finally, in order to obtain the target number of channels, the feature map is modified into the number of channels of the target hyperspectral image by the channel integration module to obtain the final output.

[0092] The whole process can be summarized by the following formulas:

[0093] Y SFE =M SFE (X)

[0094] Y LMFE =M LMSE (Y SFE )+Y SFE

[0095] Y PET =M PET (Y LMSE )

[0096] Y=M CI (Y PET )

[0097] Among them, M SFE (·) represents the shallow feature extraction module, M LMSE (·) represents the local spatial multi-scale feature fusion module, M PET(·) represents the Transformer module based on positional encoding, M CI (·) represents the channel integration module.

[0098] As an example, refer to Figure 3 :

[0099] The local space multi-scale adaptive feature fusion module includes: a first convolution module, a second convolution module, a third convolution module, a fourth convolution module, a fifth convolution module, a first 3×3conv module and a second 3×3conv module;

[0100] The shallow feature extraction module is connected to the first convolution module, the second convolution module, the third convolution module, the fourth convolution module and the fifth convolution module;

[0101] The first convolution module, the second convolution module, the third convolution module, the fourth convolution module and the fifth convolution module are connected to the first 3×3conv module and the second 3×3conv module.

[0102] Specifically, the local space multi-scale adaptive feature fusion module is mainly composed of a multi-scale feature extraction module and an adaptive weighted fusion module. Among them, the multi-scale feature extraction module is composed of a convolution module with five branches.

[0103] As an example:

[0104] The specific workflow of the local space multi-scale adaptive feature fusion module is as follows:

[0105] Get the feature map Y of the input local space multi-scale adaptive feature fusion module SFE ;

[0106] The feature map Y SFE After 2x upsampling, the first convolution module and downsampling, the feature map X1 is obtained;

[0107] The feature map Y SFE After the second convolution module, the feature map X2 is obtained;

[0108] The feature map Y SFE After 2 times downsampling, the third convolution module and upsampling, the feature map X3 is obtained;

[0109] The feature map Y SFE After 4 times downsampling, the fourth convolution module and upsampling, the feature map X4 is obtained;

[0110] The feature map Y SFE After 8 times downsampling, the fifth convolution module and upsampling, the feature map X5 is obtained;

[0111] Specifically, for the first branch, 2x upsampling is used to increase the resolution of the input feature map to obtain X1. The higher resolution feature map enables the network to better capture local spatial detail features and retain the texture and edge information of the feature map. The second branch maintains the same resolution as the input feature map and is named X2. The third, fourth, and fifth branches are downsampled by 2x, 4x, and 8x, respectively, to obtain X3, X4, and X5. The downsampling strategy expands the receptive field of the feature map, thereby extracting higher-level and more abstract semantic information. Finally, X1 is downsampled, while X3, X4, and X5 are upsampled to ensure that the feature maps of each branch have the same dimension. Subsequently, each branch is spliced ​​in the channel dimension to obtain multi-scale features. These features combine shallow texture information and abstract high-level visual information, which helps with subsequent reconstruction and category distinction in spectrally confused areas.

[0112] Furthermore, traditional feature fusion usually generates a fused feature map by element-by-element addition or splicing along the channel dimension, but this method fails to weigh the importance of multi-scale features. The present invention proposes an adaptive weighted fusion method, which assigns different weights to the features of each channel in the feature map, and can effectively fuse and integrate feature maps from different branches along the channel dimension. The new feature map integrates the feature information from each branch and exhibits richer feature representation capabilities. The feature map generated by the previous multi-scale feature extraction and fusion module undergoes a 3×3 convolution operation and a sigmoid operation to generate a weight matrix Its dimension is the same as that of the feature maps from the five branches. The specific process is:

[0113] The feature maps X1, X2, X3, X4 and X5 are concatenated in the channel dimension and input into the first 3×3conv module. After being processed by the Sigmoid function, the weight matrix w is obtained. The expression of the weight matrix w is:

[0114] w=Sigmoid(conv(concat(X1,X2,X3,X4,X5)))

[0115] Among them, concat() represents the concatenation operation of the channel dimension, conv() represents the convolution operation of the 3×3conv module, and Sigmoid() represents the sigmoid operation;

[0116] Specifically, the elements in the weight matrix fuse multi-scale features and represent the importance of features in each channel.

[0117] Furthermore, the weight matrix is ​​element-wise multiplied with the original feature maps X1, X2, X3, X4, and X5 to obtain the transformed feature maps, which are recorded as X'1, X'2, X'3, X'4, and X'5 respectively. Although the newly generated feature maps still belong to the original five branches, the feature maps of each branch now obtain the important features of other branches to a certain extent through the weight matrix. Subsequently, these transformed feature maps are combined by element-wise addition and further subjected to another 3×3 convolution operation to obtain X', which integrates the feature information from different branches and enhances the richness and expression of the features. Finally, X' is concatenated with the original five feature maps X1, X2, X3, X4, and X5 along the channel dimension, and the obtained feature maps are subjected to the final 3×3 convolution operation to produce the output feature map. This method of combining original features with weighted features helps the model to more comprehensively capture and utilize the information in the input data, thereby improving its performance. The specific process is as follows:

[0118] Multiply the feature map X1, feature map X2, feature map X3, feature map X4 and feature map X5 by the weight matrix w respectively to obtain feature map X'1, feature map X'2, feature map X'3, feature map X'4 and feature map X'5;

[0119] Add the feature map X'1, the feature map X'2, the feature map X'3, the feature map X'4 and the feature map X'5 to obtain the feature map X';

[0120] Concatenate the feature maps X', X1, X2, X3, X4 and X5 in the channel dimension and input them into the second 3×3conv module to obtain the feature map Y' LMFE , feature map Y' LMFE The expression is:

[0121] Y' LMFE =Conv(concat(X',X1,X2,X3,X4,X5)).

[0122] As an example, refer to Figure 4 :

[0123] The Transformer module based on position coding includes a first PCSA module, a first LayerNorm module, a first FFN module, a downsampling module, an upsampling module, a second PCSA module, a second LayerNorm module and a second FFN module connected in sequence;

[0124] The local spatial multi-scale adaptive feature fusion module is connected to the first PCSA module and the first Layer Norm module;

[0125] The first PCSA module is connected to the downsampling module;

[0126] The first PCSA module is connected to the second PCSA module;

[0127] The up-sampling module is connected to the second Layer Norm module;

[0128] The second PCSA module and the second FFN module are connected to the channel integration module.

[0129] Specifically, Figure 4 (1) is the overall structure diagram of Transformer. Figure 4 (2) is the structure diagram of the feedforward network FFN. Figure 4 (3) is the position coding based channel self-attention module proposed by the present invention. In the module of Figure (1), the feature map passes through layer normalization (Layer Norm), position coding channel self-attention (PCSA) module, another layer normalization, and finally the feed-forward neural network (FFN) module.

[0130] As an example:

[0131] The first PCSA module and the second PCSA module have the same structure;

[0132] The first PCSA module includes: a first 1×1conv module, a first 3×3DW module, a second 1×1conv module, a second 3×3DW module, a third 1×1conv module and a third 3×3DW module;

[0133] The input end of the first PCSA module is connected to the first 1×1conv module, the second 1×1conv module and the third 1×1conv module;

[0134] The first 1×1conv module is connected to the first 3×3DW module, the second 1×1conv module is connected to the second 3×3DW module, and the third 1×1conv module is connected to the third 3×3DW module;

[0135] The first 3×3 DW module, the second 3×3 DW module and the third 3×3 DW module are connected to the output terminal of the first PCSA module.

[0136] As an example, Transformer has been applied in the field of spectral reconstruction. Unlike the traditional visual Transformer (ViT) method, since hyperspectral images are sparse in space but have high self-similarity in the spectral domain, modeling the correlation between spectra is more cost-effective than capturing spatial interdependencies. Influenced by these methods, the present invention proposes a position encoding channel self-attention module. Figure 4 (3), the method of the present invention incorporates position encoding in the process of deriving Q (query), K (key) and V (value), and the specific process is as follows:

[0137] The workflow of the first PCSA module is as follows:

[0138] Get the feature map F of the first PCSA module in ;

[0139] The feature map F in After the first 1×1conv module and the first 3×3DW module, the projection value Q is obtained;

[0140] The feature map F in After the second 1×1conv module and the second 3×3DW module, the projection value K is obtained;

[0141] The feature map F in After the third 1×1conv module and the third 3×3DW module, the projection value V is obtained;

[0142] The feature map F output by the first PCSA module is obtained by calculating the projection values ​​Q, K and V. out , feature map F out The calculation formula is:

[0143] Attention=Softmax((PE(K)+K)·(PE(Q)+Q) T ))

[0144] F out =Attention·(PE(V)+V) T

[0145] Among them, PE() represents the position embedding mechanism function, and Softmax() represents the normalized exponential function.

[0146] Specifically, Q, K, and V represent the projection values ​​obtained after the input is transformed by 1×1 convolution and 3×3 deep convolution, respectively. They play the roles of query, key, and value, respectively. The PE(·) function represents the position embedding mechanism, which includes a 3×3 convolution, followed by a Gaussian error linear unit (GELU) activation function, and then another 3×3 convolution. Once processed by PE(·), Q, K, and V can all be considered to contain position information. By incorporating position information, the generated attention map has the ability to pay attention to the relative position relationship within the spectrum, which helps to better integrate the interdependence between spectra and ultimately help the network achieve accurate reconstruction.

[0147] S2: obtaining a medical RGB-hyperspectral image pair set, and dividing the medical RGB-hyperspectral image pair set into a training image set and a test image set;

[0148] Specifically, in the medical RGB-hyperspectral image pair set, a medical RGB image and its corresponding medical hyperspectral image are taken as an image pair.

[0149] S3: training the medical hyperspectral image spectral reconstruction network through the training image set to obtain a trained medical hyperspectral image spectral reconstruction network;

[0150] As an example:

[0151] Step S3 is specifically as follows:

[0152] S31: obtaining a training medical RGB image and a real medical hyperspectral image from a training image set, and adjusting parameters of a medical hyperspectral image spectral reconstruction network;

[0153] S32: input the training medical RGB image into a medical hyperspectral image spectral reconstruction network to obtain a predicted medical hyperspectral image;

[0154] S33: Calculate the RMSE loss between the predicted medical hyperspectral image and the real medical hyperspectral image;

[0155] Specifically, unlike the commonly used loss functions in the current field, the present invention uses RMSE as the error between the reconstructed hyperspectral image and the real hyperspectral image. This loss calculation method enables the reconstruction network to balance the reconstruction of the foreground and background areas of the image at the same time, rather than making the reconstruction biased towards the background area and ignoring the foreground area like other methods. The calculation formula of the loss function is as follows:

[0156]

[0157] in, represents the hyperspectral image (HSI) cube obtained after the reconstruction network, represents the true hyperspectral image cube, and N = H × W × C represents the total number of all values ​​in the image.

[0158] S34: Repeat steps S31-S33 until the RMSE loss is within a preset range, and obtain a trained medical hyperspectral image spectral reconstruction network.

[0159] S4: Obtain a medical RGB image to be reconstructed from the test image set, input the medical RGB image to be reconstructed into a trained medical hyperspectral image spectral reconstruction network, and obtain a reconstructed medical hyperspectral image.

[0160] Specifically, the present invention uses the commonly used evaluation indicators RMSE, PSNR, SAM and SSIM in the field of spectral reconstruction to compare with other methods, wherein the larger the values ​​of PSNR and SSIM, the better the reconstruction effect, and the smaller the values ​​of RMSE and SAM, the higher the reconstruction accuracy. The two methods listed in the table are the two best spectral reconstruction methods in the field of natural images at present, and can show the credibility of the results of the present invention compared with these two methods. It can be found that our invention shows better results in all four evaluation indicators, which illustrates the innovation of the method and the superiority of performance of the present invention.

[0161] Table 1 Comparison of reconstruction accuracy of different algorithms on medical image datasets

[0162]

[0163] See also Figure 5 , Figure 5 4 is a schematic diagram of the working of the hardware device of an embodiment of the present invention, and the hardware device specifically includes: a medical hyperspectral image spectral reconstruction device 401 based on deep learning, a processor 402 and a storage medium 403.

[0164] A medical hyperspectral image spectrum reconstruction device 401 based on deep learning: The medical hyperspectral image spectrum reconstruction device 401 based on deep learning implements the medical hyperspectral image spectrum reconstruction method based on deep learning.

[0165] Processor 402: The processor 402 loads and executes the instructions and data in the storage medium 403 to implement the medical hyperspectral image spectral reconstruction method based on deep learning.

[0166] Storage medium 403: The storage medium 403 stores instructions and data; the storage medium 403 is used to implement the medical hyperspectral image spectral reconstruction method based on deep learning.

[0167] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or system including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or system. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or system including the element.

[0168] The serial numbers of the embodiments of the present invention are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments. In a unit claim that lists several means, several of these means may be embodied by the same hardware item. The use of the words first, second, and third, etc. does not indicate any order and these words may be interpreted as identifiers.

[0169] The above are only preferred embodiments of the present invention, and are not intended to limit the patent scope of the present invention. Any equivalent structure or equivalent process transformation made using the contents of the present invention specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present invention.

Claims

1. A method for spectral reconstruction of medical hyperspectral images based on deep learning, characterized in that: Includes steps: S1: Construct a spectral reconstruction network for medical hyperspectral images based on CNN-Transformer; S2: obtaining a medical RGB-hyperspectral image pair set, and dividing the medical RGB-hyperspectral image pair set into a training image set and a test image set; S3: training the medical hyperspectral image spectral reconstruction network through the training image set to obtain a trained medical hyperspectral image spectral reconstruction network; S4: Obtain a medical RGB image to be reconstructed from the test image set, input the medical RGB image to be reconstructed into a trained medical hyperspectral image spectral reconstruction network, and obtain a reconstructed medical hyperspectral image.

2. The method for spectral reconstruction of medical hyperspectral images based on deep learning according to claim 1, characterized in that: The spectral reconstruction network of medical hyperspectral images includes a shallow feature extraction module, a local space multi-scale adaptive feature fusion module, a position encoding-based Transformer module, and a channel integration module connected in sequence; The shallow feature extraction module is connected with the positional encoding based Transformer module.

3. The method for spectral reconstruction of medical hyperspectral images based on deep learning according to claim 2, characterized in that: The specific workflow of the medical hyperspectral image spectral reconstruction network is as follows: Input the three-channel medical RGB image X into the shallow feature extraction module to obtain the feature map Y SFE ; The feature map Y SFE Input the local space multi-scale adaptive feature fusion module to obtain the feature map Y' LMFE , the feature map Y' LMFE and feature map Y SFE Add together to get the feature map Y LMFE ; The feature map Y LMFE Input the Transformer module based on position encoding to obtain the feature map Y PET ; The feature map Y PET The input channel integration module obtains the reconstructed medical hyperspectral image Y.

4. The method for spectral reconstruction of medical hyperspectral images based on deep learning according to claim 2, characterized in that: The local space multi-scale adaptive feature fusion module includes: a first convolution module, a second convolution module, a third convolution module, a fourth convolution module, a fifth convolution module, a first 3×3conv module and a second 3×3conv module; The shallow feature extraction module is connected to the first convolution module, the second convolution module, the third convolution module, the fourth convolution module and the fifth convolution module; The first convolution module, the second convolution module, the third convolution module, the fourth convolution module and the fifth convolution module are connected to the first 3×3conv module and the second 3×3conv module.

5. The method for spectral reconstruction of medical hyperspectral images based on deep learning according to claim 4, characterized in that: The specific workflow of the local space multi-scale adaptive feature fusion module is as follows: Get the feature map Y of the input local space multi-scale adaptive feature fusion module SFE ; The feature map Y SFE After 2x upsampling, the first convolution module and downsampling, the feature map X1 is obtained; The feature map Y SFE After the second convolution module, the feature map X2 is obtained; The feature map Y SFE After 2x downsampling, the third convolution module and upsampling, the feature map X3 is obtained; The feature map Y SFE After 4 times downsampling, the fourth convolution module and upsampling, the feature map X4 is obtained; The feature map Y SFE After 8 times downsampling, the fifth convolution module and upsampling, the feature map X5 is obtained; The feature maps X1, X2, X3, X4 and X5 are concatenated in the channel dimension and input into the first 3×3conv module. After being processed by the Sigmoid function, the weight matrix w is obtained. The expression of the weight matrix w is: w=Sigmoid(conv(concat(X1,X2,X3,X4,X5))) Among them, concat() represents the concatenation operation of the channel dimension, conv() represents the convolution operation of the 3×3conv module, and Sigmoid() represents the sigmoid operation; Multiply the feature map X1, feature map X2, feature map X3, feature map X4 and feature map X5 by the weight matrix w respectively to obtain feature map X'1, feature map X'2, feature map X'3, feature map X'4 and feature map X'5; Add the feature map X'1, the feature map X'2, the feature map X'3, the feature map X'4 and the feature map X'5 to obtain the feature map X'; Concatenate the feature maps X', X1, X2, X3, X4 and X5 in the channel dimension and input them into the second 3×3conv module to obtain the feature map Y' LMFE , feature map Y' LMFE The expression is: Y' LMFE =Conv(concat(X',X1,X2,X3,X4,X5))。 6. The method for spectral reconstruction of medical hyperspectral images based on deep learning according to claim 2, characterized in that: The Transformer module based on position coding includes a first PCSA module, a first Layer Norm module, a first FFN module, a downsampling module, an upsampling module, a second PCSA module, a second Layer Norm module and a second FFN module connected in sequence; The local spatial multi-scale adaptive feature fusion module is connected to the first PCSA module and the first Layer Norm module; The first PCSA module is connected to the downsampling module; The first PCSA module is connected to the second PCSA module; The up-sampling module is connected to the second Layer Norm module; The second PCSA module and the second FFN module are connected to the channel integration module.

7. The method for spectral reconstruction of medical hyperspectral images based on deep learning according to claim 6, characterized in that: The first PCSA module and the second PCSA module have the same structure; The first PCSA module includes: a first 1×1conv module, a first 3×3DW module, a second 1×1conv module, a second 3×3DW module, a third 1×1conv module and a third 3×3DW module; The input end of the first PCSA module is connected to the first 1×1conv module, the second 1×1conv module and the third 1×1conv module; The first 1×1conv module is connected to the first 3×3DW module, the second 1×1conv module is connected to the second 3×3DW module, and the third 1×1conv module is connected to the third 3×3DW module; The first 3×3 DW module, the second 3×3 DW module and the third 3×3 DW module are connected to the output terminal of the first PCSA module.

8. The method for spectral reconstruction of medical hyperspectral images based on deep learning according to claim 7, characterized in that: The workflow of the first PCSA module is as follows: Get the feature map F of the first PCSA module in ; The feature map F in After the first 1×1conv module and the first 3×3DW module, the projection value Q is obtained; The feature map F in After the second 1×1conv module and the second 3×3DW module, the projection value K is obtained; The feature map F in After the third 1×1conv module and the third 3×3DW module, the projection value V is obtained; The feature map F output by the first PCSA module is obtained by calculating the projection values ​​Q, K and V. out , feature map F out The calculation formula is: Attention=Softmax((PE(K)+K)·(PE(Q)+Q) T )) F out =Attention·(FOR(V)+V) T Among them, PE() represents the position embedding mechanism function, and Softmax() represents the normalized exponential function.

9. The method for spectral reconstruction of medical hyperspectral images based on deep learning according to claim 1, characterized in that: Step S3 is specifically as follows: S31: obtaining a training medical RGB image and a real medical hyperspectral image from a training image set, and adjusting parameters of a medical hyperspectral image spectral reconstruction network; S32: input the training medical RGB image into a medical hyperspectral image spectral reconstruction network to obtain a predicted medical hyperspectral image; S33: Calculate the RMSE loss between the predicted medical hyperspectral image and the real medical hyperspectral image; S34: Repeat steps S31-S33 until the RMSE loss is within a preset range, and obtain a trained medical hyperspectral image spectral reconstruction network.

10. A medical hyperspectral image spectrum reconstruction device based on deep learning, characterized in that: include: Processor and storage medium; the processor loads and executes instructions and data in the storage medium to implement the deep learning-based medical hyperspectral image spectral reconstruction method described in any one of claims 1 to 9.

Images