Microscopic hyperspectral image segmentation method based on multi-scale attention fusion

Abstract

The invention discloses a microscopic hyperspectral image segmentation method based on multi-scale attention fusion, and aims to improve the precision and robustness of medical image segmentation. According to the method, advantages of Transform and CNN in extraction of global and local features are combined, an innovative fusion module MSA (multi-scale attention) is designed, and global features of the Transform are efficiently fused into local features of the CNN through multi-scale, channel attention and space attention mechanisms. Meanwhile, residual connection is introduced in the up-sampling stage of the Transform branch, so that the gradient disappearance problem is relieved, and the network expression capability is enhanced; an SE module is added behind each CNN module, and feature channel weights are dynamically adjusted, so that useful features are enhanced, and redundant information is suppressed. Experiments show that the method shows excellent segmentation precision, robustness and efficiency in a microscopic hyperspectral image segmentation task, and particularly shows significant advantages for challenges such as boundary blurring and tissue mixing of a cancerous region and image differences of different patients.

Description

Technical Field

[0001] The present invention relates to the fields of medical image processing and deep learning, and in particular to a deep learning model of a multi-scale attention fusion microscopic hyperspectral image segmentation method and its application in microscopic hyperspectral image segmentation. Background Art

[0002] At present, the diagnosis of cancer mainly relies on laboratory tests, imaging tests, tumor marker detection and combined with the patient's clinical manifestations. Among them, imaging tests include ultrasound, computer tomography (CT), magnetic resonance imaging (MRI), etc. However, the above diagnostic methods still have a certain degree of non-specificity, and histopathological examination is still the gold standard for cancer diagnosis. After the pathological sections are made, the doctor will observe the pathological sections through a microscope to analyze tumor cells and tumor tissues. This process is often time-consuming and inefficient. If a sufficiently high accuracy rate is to be achieved, the professional skills of the doctor are required to be high. In particular, in the face of some small lesions with high uncertainty, doctors need to repeatedly consider and discuss during the diagnosis process. In this process, doctors may need to spend a lot of energy due to fatigue or differences in personal diagnostic standards, and may also cause misdiagnosis and missed diagnosis. Therefore, there is an urgent need for an auxiliary means that can improve efficiency to reduce the labor intensity of doctors and improve the consistency and accuracy of diagnosis. Computer-Aided Diagnosis (CAD) is increasingly trusted by doctors due to its fast and efficient characteristics and not affected by subjective factors.

[0003] Generally speaking, the diagnosis of cancer using CAD mainly relies on traditional digital pathology slice images, but the amount of information contained in traditional digital images is limited and has gradually approached the ceiling of intelligent cancer pathology diagnosis. Hyperspectral imaging technology is a technology that obtains the optical properties of objects through spectral analysis and image processing. It was first applied in the field of remote sensing. Since hyperspectral imaging technology combines spectral technology and traditional imaging technology, it can obtain two-dimensional spatial information and one-dimensional spectral information of the photographed object, so it has the characteristic of "graph-spectrum integration". With the development of science and technology and the cross-integration of disciplines, hyperspectral imaging technology has achieved fruitful research results in agricultural monitoring, food testing, military reconnaissance and other fields. Due to the complexity of pathological images themselves and the demand for medical data information in medical image processing, hyperspectral imaging technology has also developed rapidly in the field of medical image processing. Hyperspectral imaging of tissue pathology slices, through spectral analysis of light of different wavelengths, obtains the difference in optical properties between diseased tissue and normal tissue, which cannot be presented by traditional digital images.

[0004] At present, deep learning methods have made continuous breakthroughs in the field of medical image processing, providing new ideas for cancer diagnosis. As an end-to-end learning model, convolutional neural network (CNN) can learn complex image features and structural information by learning a large amount of medical image data, thereby achieving accurate segmentation of medical images. Although CNN networks have achieved great success in the field of medical image segmentation, their shortcomings make it difficult to make greater breakthroughs. CNN is a convolution operation based on local receptive field. Although the receptive field can be gradually expanded by stacking multiple convolution and pooling layers, it may still be insufficient to capture global information and it is difficult to establish long-distance dependencies, especially for large-scale associative segmentation tasks. To solve this problem, many researchers have introduced the Transformer structure used for natural language processing (NLP) into computer vision. The Transformer architecture is distinguished from CNN by replacing the convolution operation with a self-attention mechanism. This modification helps to understand the global information more comprehensively in an image. Although the Transformer performs well in extracting global information, it has certain limitations in extracting local information. This is because the Transformer's self-attention mechanism ignores the spatial position relationship of pixels in the image, which may lead to the problem of local information loss when processing the image. In order to effectively integrate local and global information, many researchers have proposed using CNN and Transformer for feature extraction at the same time and fusing their output features. Researchers usually use convolution and other methods to combine the two feature extraction methods to fully utilize their respective advantages. However, these methods have limitations in effectively combining local and global features. Summary of the invention

[0005] Purpose of the invention: In order to effectively combine local and global features and improve the segmentation effect of challenges such as blurred boundaries of cancerous areas, tissue mixing, and image differences between different patients, the present invention discloses a microscopic hyperspectral image segmentation method with multi-scale attention fusion. In order to effectively extract features, we adopted a dual encoder framework, in which one branch uses CNN to extract local features, and the other branch uses Transformer to capture global contextual information. This dual encoder structure can extract more complementary features, making up for the limitations of using CNN or Transformer alone. In order to fuse the features extracted by the encoder branches, a new multi-scale attention fusion module MSA is proposed, which can effectively fuse features from different branches.

[0006] Technical solution: To achieve the above purpose, the present invention discloses a multi-scale attention fusion microscopic hyperspectral image segmentation method, comprising the following steps:

[0007] Step 1: Perform dimensionality reduction processing on the input microscopic hyperspectral image using principal component analysis, and perform data set preprocessing, dividing the preprocessed data set into a training set, a validation set, and a test set;

[0008] Step 2: Extract features from the image using a dual encoder architecture, where the dual encoder includes: a CNN encoder branch for extracting local features of the image, and a Transformer encoder branch for capturing global features and long-distance dependencies of the image;

[0009] Step 3: Design a multi-scale attention fusion module (MSA), including a multi-scale feature fusion module (MSF) and a dual attention module (DAM), to perform multi-scale fusion of the features extracted by CNN and Transformer;

[0010] Step 4: The decoder upsamples and convolves the fused features to restore the image size and generate the segmentation result.

[0011] As a preferred embodiment of the present invention, the specific content of step 2 is:

[0012] 1.1: CNN encoder branch: The ResNet-34 model is adopted, and its first four stages are used as the CNN branch structure. The SE module (Squeeze-and-Excitation module) is introduced after the output of each stage of ResNet34 to dynamically adjust the weights of the feature map channels to enhance the feature expression ability, and fuse the output of each stage of the CNN branch with the corresponding features of the Transformer branch;

[0013] 1.2: Transformer encoder branch: The Transformer branch is a model consisting of 6 layers of Transformer encoder block reshape and upsampling operations. First, the input image is divided into image blocks of size P×P, and each image block is converted into a one-dimensional vector, which is mapped to an embedding vector of fixed dimension through a linear layer. Then, a multi-layer Transformer encoder is used to capture global features. Each layer of Transformer encoder includes a multi-head self-attention mechanism (MSA), a feedforward neural network (MLP), residual connections, and layer normalization. Finally, in the upsampling process, a bilinear interpolation method is used, and a residual connection is introduced to compensate for the loss of low-level semantic information.

[0014] As a preferred embodiment of the present invention, the specific content of step 3 is:

[0015] 2.1: Multi-scale feature fusion module (MSF): The features of the CNN branch and the Transformer branch are concatenated in the channel dimension. Multi-scale convolution (with kernel sizes of 1×1, 3×3, 5×5, and 7×7) is then used to extract features under different receptive fields. The features after the four groups of convolution are then concatenated again in the channel dimension to form a feature representation containing multi-scale information. Finally, a 1×1 convolution is used to map the concatenated multi-scale features back to the original number of channels.

[0016] 2.2: The Dual Attention Module (DAM) consists of a spatial attention module and a channel attention module, and aims to enhance the feature expression capability by integrating spatial position and channel information. The spatial attention module (SAM) first performs a 3×3 convolution operation, global average pooling, and another 3×3 convolution operation on the input features to extract important spatial information. Then, the output feature map of the above operations is element-wise multiplied (Hadamard product) and feature summed with the original input features. This module focuses on the key spatial positions in the feature map, enhances the feature expression of the target area, and suppresses irrelevant areas. The channel attention module (CAM) first performs global average pooling and global maximum pooling on the input features to extract the global statistical information of the channel. Then, the outputs of the two pooling operations are element-wise multiplied to obtain the importance score of each channel, and normalized to weights through the Sigmoid function. Finally, the channel weights are multiplied with the output features of the spatial attention module to adjust the channel response strength of the spatial features.

[0017] As a preferred embodiment of the present invention, the specific content of step 4 is:

[0018] The fused features are upsampled by the bilinear interpolation method, and the upsampled features are convolved by two 3×3 convolutions and one 1×1 convolution. A residual connection is introduced after the convolution operation to enhance the feature expression ability, and the decoder output is aligned with the resolution of the input image to generate the final segmentation result.

[0019]

[0020] G i represents the features obtained after a decoder block, up i The features after upsampling, The output of the fusion module.

[0021] Beneficial effects: Compared with the prior art, the present invention has the following beneficial technical effects:

[0022] 1: A novel dual encoder architecture is proposed, which combines the advantages of Transformer and Convolutional Neural Network (CNN). Transformer encoder is good at capturing long-distance dependencies, while CNN encoder excels in capturing local features. By fusing these two encoders, the model can handle complex image segmentation tasks more effectively.

[0023] 2: An MSA module consisting of a multi-scale feature fusion module and a dual attention module is proposed. The scale fusion module is combined with the dual attention module to significantly improve the model's ability to capture complex structures and details through the adaptive fusion of multi-scale features and the joint application of the attention mechanism.

[0024] 3: The effectiveness of the proposed model was verified in the microscopic hyperspectral image segmentation task. The results showed that the model has significant advantages in dealing with complex structures and image differences between different patients, demonstrating its potential in clinical applications. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Figure 1 Overall structure diagram of the model.

[0026] Figure 2 .Schematic diagram of the Transformer encoder module.

[0027] Figure 3 Schematic diagram of the multi-scale attention module (MSA).

[0028] Figure 4 Schematic diagram of the multi-scale feature fusion module (MSF).

[0029] Figure 5 Schematic diagram of the dual attention module (DAM). DETAILED DESCRIPTION

[0030] The exemplary embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although the exemplary embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be limited by the embodiments set forth herein. On the contrary, these embodiments are provided to enable a more thorough understanding of the present disclosure and to fully convey the scope of the present disclosure to those skilled in the art.

[0031] like Figure 1 As shown, the embodiment of the present invention discloses a multi-scale attention fusion microscopic hyperspectral image segmentation method, which specifically includes the following steps:

[0032] A: Dataset preprocessing: Gastric cancer and bile duct cancer hyperspectral image datasets were selected, and principal component analysis (PCA) was used to reduce the data dimension. The data was then preprocessed and standardized to ensure the consistency and effectiveness of the model input.

[0033] B: Segmentation network model construction: The network model includes an encoder, a feature fusion module and a decoder. The structure is as follows Figure 2 shown.

[0034] C: Dataset division and model training: The dataset is divided into training set, validation set and test set in a ratio of 7:1:2. The training set is used to learn model parameters; the validation set is used to adjust hyperparameters to prevent overfitting; and the test set is used to evaluate the final model performance.

[0035] B.1: Encoder part: The encoder is a dual-branch structure, including CNN encoder and Transformer encoder:

[0036] B.1.1: CNN encoder:

[0037] Based on the first four stages of ResNet-34, multi-scale features are extracted layer by layer. SE modules are added after each stage output to dynamically adjust channel weights to enhance important features while reducing computational costs. The input size is R H×W×C After multi-stage processing of ResNet-34, the output feature dimension of each stage is And fuse the output of each layer with the corresponding features of the Transformer encoder to combine local and global information.

[0038] B.1.2: Transformer encoder:

[0039] The input image size is R H×W×C , divided into P×P image blocks (default P = 16), generating patches, flattened into a one-dimensional vector and embedded through a linear layer as The Transformer module consists of 6 layers of encoder blocks. The features output by the Transformer module are reshaped and upsampled to obtain features with the same resolution as the CNN encoder. and

[0040] B.2: Feature fusion module: Simple feature concatenation, weighting or averaging methods may not fully exploit the advantages of local and global features. We designed a fusion module MSA (e.g. Figure 3 As shown in Figure 2), to more effectively fuse local and global features

[0041] B.2.1: Multi-scale feature fusion module:

[0042] Multi-scale feature fusion module (such as Figure 4 As shown in the figure, the features extracted by the CNN and Transformer encoders are concatenated in the channel dimension and then input into four groups of convolutions with different convolution kernel sizes (1, 3, 5, and 7) to extract multi-scale features. After the multi-scale features are concatenated again, they are mapped back to the original number of channels through 1×1 convolution to maintain input consistency and enhance feature expression capabilities.

[0043] B.2.2: Dual Attention Module:

[0044] Dual attention modules (such as Figure 5 As shown in Figure 2, it includes spatial attention and channel attention branches. The spatial attention module (SAM) first performs a 3×3 convolution operation and global average pooling on the input features, and then another 3×3 convolution operation to extract important spatial information. Then, the output feature map of the above operation is element-wise multiplied (Hadamard product) and feature added with the original input features. This module focuses on the key spatial positions in the feature map, enhances the feature expression of the target area, and suppresses irrelevant areas. The channel attention module (CAM) first performs global average pooling and global maximum pooling on the input features to extract the global statistical information of the channel. Then the outputs of the two pooling operations are element-wise multiplied to obtain the importance score of each channel, and normalized to weights through the Sigmoid function. Finally, the channel weights are multiplied by the output features of the spatial attention module to adjust the channel response strength of the spatial features.

[0045] B.3: Decoder part:

[0046] The decoder uses bilinear interpolation upsampling, combined with multi-layer convolution and residual connection. The specific design is as follows: the upsampling part uses bilinear interpolation to increase the resolution, the convolution part includes 3×3 convolution kernel to extract features, 1×1 convolution to adjust the number of channels, and combined with residual connection to reduce information loss. The decoder output formula is:

[0047]

[0048] G i represents the features obtained after a decoder block, up i The features after upsampling, The output of the fusion module.

[0049] C.1: MSA-Net is built with the PyTorch framework and trained on a single NVIDIA RTX 3090 GPU with epochs = 50 and batch size = 2. The Adam optimizer with a learning rate of 0.00006 and a cross entropy loss function is used. Multi-step decay is used during training, and the learning rate is reduced to half of the original at the 30th and 40th epochs.

[0050] C.2: Input the training set into the network of B for training. After the training, use the validation set for evaluation and save the model parameters with the best training results. Use the model with the best training results to perform image segmentation on the test set to obtain the segmentation results.

[0051] C3: Use global accuracy (Accuracy), average accuracy (MeanAcc), average intersection over union (Mean Iou), and average Dice coefficient (Mean Dice) as evaluation indicators to evaluate the segmentation effect. The formulas of these indicators are as follows:

[0052]

[0053]

[0054]

[0055]

[0056]

[0057]

[0058] TP (True Positive): The number of samples predicted as positive and actually positive. TN (True Negative): The number of samples predicted as negative but actually negative. FP (False Positive): The number of samples predicted as positive but actually negative. FN (False Negative): The number of samples predicted as negative but actually positive. K represents the number of segmentation classes.

Claims

1. This paper introduces a multi-scale attention fusion microscopic hyperspectral image segmentation method, which is characterized by: The following steps are involved: Step 1: Perform dimensionality reduction processing on the input microscopic hyperspectral image using principal component analysis, and perform data set preprocessing, dividing the preprocessed data set into a training set, a validation set, and a test set; Step 2: Extract features from the image using a dual encoder architecture, where the dual encoder includes: a CNN encoder branch for extracting local features of the image, and a Transformer encoder branch for capturing global features and long-distance dependencies of the image; Step 3: Design a multi-scale attention fusion module (MSA), including a multi-scale feature fusion module (MSF) and a dual attention module (DAM), to perform multi-scale fusion of features extracted by CNN and Transformer; Step 4: The decoder upsamples and convolves the fused features to restore the image size and generate the segmentation result.

2. According to the multi-scale attention fusion microscopic hyperspectral image segmentation method of claim 1, it is characterized in that: The specific content of step 2 is: 1.1: CNN encoder branch: The ResNet-34 model is adopted, and its first four stages are used as the CNN branch structure. The SE module (Squeeze-and-Excitation module) is introduced after the output of each stage of ResNet-34 to dynamically adjust the weights of the feature map channels to enhance the feature expression ability, and fuse the output of each stage of the CNN branch with the corresponding features of the Transformer branch; 1.2: Transformer encoder branch: The Transformer branch is a model consisting of 6 layers of Transformer encoder blocks, Reshape and upsampling operations. First, the input image is divided into image blocks of size P×P, and each image block is converted into a one-dimensional vector, which is mapped to an embedding vector of fixed dimension through a linear layer. Then, a multi-layer Transformer encoder is used to capture global features. Each layer of Transformer encoder includes a multi-head self-attention mechanism (MSA), a feedforward neural network (MLP), residual connections and layer normalization. Finally, in the upsampling process, a bilinear interpolation method is used, and a residual connection is introduced to compensate for the loss of low-level semantic information.

3. According to the multi-scale attention fusion microscopic hyperspectral image segmentation method of claim 1, it is characterized in that: The specific process of step 3 is as follows: 2.1: Multi-scale feature fusion module (MSF): The features of the CNN branch and the Transformer branch are concatenated in the channel dimension. Multi-scale convolution (with kernel sizes of 1×1, 3×3, 5×5, and 7×7) is then used to extract features under different receptive fields. The features after the four groups of convolution are then concatenated again in the channel dimension to form a feature representation containing multi-scale information. Finally, a 1×1 convolution is used to map the concatenated multi-scale features back to the original number of channels. 2.2: The Dual Attention Module (DAM) consists of a spatial attention module and a channel attention module, and aims to enhance the feature expression capability by integrating spatial position and channel information. The spatial attention module (SAM) first performs a 3×3 convolution operation, global average pooling, and another 3×3 convolution operation on the input features to extract important spatial information. Then, the output feature map of the above operations is element-wise multiplied (Hadamard product) and feature summed with the original input features. This module focuses on the key spatial positions in the feature map, enhances the feature expression of the target area, and suppresses irrelevant areas. The channel attention module (CAM) first performs global average pooling and global maximum pooling on the input features to extract the global statistical information of the channel. Then, the outputs of the two pooling operations are element-wise multiplied to obtain the importance score of each channel, and normalized to weights through the Sigmoid function. Finally, the channel weights are multiplied with the output features of the spatial attention module to adjust the channel response strength of the spatial features.

4. According to the multi-scale attention fusion microscopic hyperspectral image segmentation method of claim 1, it is characterized in that: The specific process of step 4 is as follows: The fused features are upsampled by bilinear interpolation, and the upsampled features are convolved by two 3×3 convolutions and one 1×1 convolution. After the convolution operation, residual connections are introduced to enhance the feature expression capability, and the decoder output is aligned with the resolution of the input image to generate the final segmentation result. The detailed steps are shown in Formula 1: G i represents the features obtained after a decoder block, up i The features after upsampling, The output of the fusion module.

Images