An image segmentation method based on microscopic hyperspectral imaging technology

Abstract

The application discloses an image segmentation method based on a microscopic hyperspectral imaging technology, which comprises the following steps: collecting images of a sample by a microscopic hyperspectral imaging system, constructing a microscopic hyperspectral image database, and marking the images by a doctor; pre-processing original hyperspectral data collected; inputting the pre-processed data into a feature extraction backbone network to obtain multi-scale feature images, wherein the backbone network is composed of four stages, each stage comprises an image merging layer, a spectral fusion feature extraction module and a spatial fusion feature extraction module; inputting the multi-scale feature images obtained by the backbone network into an FPN decoding network to output a segmentation prediction result image. The application effectively utilizes rich spectral spatial information of hyperspectral data, makes up for the deficiency of a convolutional network in global context information feature extraction, and improves the segmentation accuracy.

Description

Technical Field

[0001] This invention relates to the field of medical microscopic hyperspectral image segmentation, and in particular to an image segmentation method based on microscopic hyperspectral imaging technology. Background Technology

[0002] Hyperspectral imaging (HSI) is an extension of multispectral imaging (MSI), a technique that integrates spectral and imaging technologies to obtain two-dimensional spatial images and one-dimensional spectral information of a target object. Initially applied in remote sensing, HSI was primarily used in agriculture, vegetation monitoring, and geological exploration. In recent years, with the continuous development of the optical structures of hyperspectral imaging systems, its application in medical imaging has yielded significant research results, particularly in the field of microscopic imaging. In the diagnosis of pathological sections, some early pathological changes manifest as variations in chemical composition, but these changes are not obvious in tissue structure and cell morphology, making it difficult for pathologists to distinguish the differences with the naked eye. Microscopic hyperspectral imaging can simultaneously acquire rich spectral and spatial information of biological tissues at the microscopic level, making the differences in chemical composition invisible to the naked eye clearly visible, providing a new approach for early pathological section diagnosis.

[0003] While microscopic hyperspectral imaging technology offers new and effective auxiliary diagnostic methods for pathological research, it also presents challenges such as massive data volume and information redundancy. Therefore, an efficient method for processing hyperspectral data is needed. Traditional image recognition and classification methods primarily utilize machine learning algorithms, where researchers manually design feature extraction algorithms to analyze and process image information. However, the large volume and high dimensionality of hyperspectral data undoubtedly make feature extraction difficult for traditional methods. In contrast, deep learning algorithms, with their numerous parameters and deep structure, are capable of learning richer and deeper hidden features from hyperspectral data, while also exhibiting stronger generalization performance.

[0004] Deep learning, especially methods based on convolutional neural networks (CNNs), has been successfully applied to the field of medical hyperspectral image segmentation with good results. However, existing CNN-based hyperspectral image segmentation methods lack the ability to model global information and pay attention to spectral information, thus failing to effectively utilize the rich spectral spatial information of hyperspectral images. Summary of the Invention

[0005] The purpose of this invention is to address the shortcomings of existing technologies by proposing an image segmentation method based on microscopic hyperspectral imaging technology.

[0006] The purpose of this invention is achieved through the following technical solution: In order to solve the problems of insufficient utilization of spectral spatial information of hyperspectral images in the case of small samples, lack of global information extraction capability of convolutional neural networks, and huge computational load caused by global attention in the transformer module, the purpose of this invention is to provide an image segmentation method that can effectively extract spectral spatial features of hyperspectral images, integrates convolution operation and transformer module based on window attention mechanism, and achieves efficient feature extraction of global and local information while reducing the number of parameters and computational load of network model.

[0007] To achieve the above-mentioned objectives of this invention, this invention provides an image segmentation method based on microscopic hyperspectral imaging technology, the method comprising the following steps:

[0008] Dataset processing: Hyperspectral and visible light images of gastric cancer pathological sections were acquired, and the visible light images were manually labeled; the hyperspectral images were preprocessed and the dataset was divided to obtain a gastric cancer microscopic hyperspectral image dataset.

[0009] Constructing a network model for segmenting hyperspectral images of microscopic pathology: This model includes a feature extraction backbone network and a decoding head network;

[0010] The feature extraction backbone network includes four stages, each stage including an image embedding module and several hybrid feature extraction modules. The hybrid feature extraction module includes a spectral hybrid feature extraction submodule and a spatial hybrid feature extraction submodule, which extract features of different dimensions through spectral attention windows and spatial attention windows, respectively.

[0011] The decoder network is used to fuse feature outputs of different resolutions obtained from each stage of the backbone feature extraction network;

[0012] The network model is trained to obtain a well-trained microscopic pathological hyperspectral image segmentation network model;

[0013] The hyperspectral image of the gastric cancer pathological section to be tested is input into the trained network model to obtain the segmentation prediction result of the microscopic hyperspectral image of the gastric cancer pathological section.

[0014] Furthermore, the manual annotation of the visible light image specifically involves: selecting the cancerous area in the visible light image and setting it as a corresponding positive label; similarly annotating the background and some non-cancer areas inside the cancerous area in the visible light image and setting them as negative labels.

[0015] Furthermore, the data preprocessing includes spectral correction and spectral dimensionality reduction. The spectral correction specifically involves dividing the acquired hyperspectral data of the pathological slides by the corresponding blank slide data position by position, converting the light intensity values ​​in the original data into transmittance. The spectral dimensionality reduction specifically involves selecting clear bands in the spectral bands and using the PCA principal component analysis method to further reduce the number of spectral bands in the image.

[0016] Furthermore, the image embedding module in the feature extraction backbone network specifically comprises:

[0017] Input at stage i First, the input is processed through an image embedding layer to obtain feature representations of the hyperspectral image at different spatial resolutions. The first-stage image embedding module uses 2D convolution with a kernel size of 7×7, a stride of 4×4, a padding size of 3×3, and an input channel of C. i-1 And the output channel is 2C i-1 The image is divided into overlapping 4×4 image blocks, resulting in (H) i / 4)×(W i / 4)×C i ) image patches;

[0018] The remaining embedding layers use 2D convolution with a kernel size of 3×3, a stride of 2×2, padding size of 1×1, and an input channel of C. i-1 And the output channel is 2C i-1 The image is divided into overlapping 2×2 image blocks, resulting in (H) i / 2)×(W i / 2)×C i Image blocks;

[0019] After completing the image patch embedding, layer normalization is added to obtain the feature representation X of the embedded image. i .

[0020] Furthermore, the spectral mixture feature extraction module in the feature extraction backbone network specifically comprises two branches: a point convolutional path and a spectral window attention path. The outputs of the two branches are input into the first Add&Norm layer and then into the pre-convolution feedback layer, and the result is input into the second Add&Norm layer. The spatial mixture feature extraction module specifically comprises two branches: a depthwise convolutional path and a spatial window attention path. The outputs of the two branches are input into the first Add&Norm layer and then into the pre-convolution feedback layer, and the result is input into the second Add&Norm layer.

[0021] Furthermore, the point convolutional pathway consists of three point convolutional layers, one normalization layer, and one activation layer, and the spectral window attention SeWA pathway uses a size of 1×1×M. c A window is used to segment the input feature maps into H non-overlapping segments. i ×W i ×[C i / M c Each has a size of 1×1×M c The spectral attention window, where M c The spectral dimension of the window is no larger than the number of feature channels; swapping the dimensions makes it 1×1×M. c A length of M c sequence of tokens Then, after calculating the attention for each word sequence in each window, the windows are merged to restore the original feature representation size.

[0022] The attention calculation specifically involves summing the output of the point convolutional path with the value V (Value) in SeWA before using it for attention calculation. The specific formula is as follows:

[0023]

[0024] Where Q, K, and V represent the query, key, and value in the attention mechanism, respectively, V_conv is the output of the point convolutional path, and d k This is the scaling factor.

[0025] Furthermore, the first Add&Norm layer consists of two parts: Add and Norm, and its calculation formula is as follows:

[0026] LayerNorm(X+(PW(X)+SeWA(X)))

[0027] Where Add is a residual connection, Norm refers to layer normalization, PW(X) represents the output of the point convolution path, and SeWA(X) represents the output of the spectral window attention path.

[0028] The calculation of the second Add&Norm layer is as follows:

[0029] LayerNorm(X+CFFN(X))

[0030] Where CFFN(X) represents the output of the pre-convolutional feedback layer.

[0031] Furthermore, the spatial hybrid feature extraction module includes two branches: a depthwise convolutional path and a spatial window attention path.

[0032] The deep convolutional channel branch consists of three deep convolutional layers, one normalization layer, and one activation layer; the spatial window attention path uses a size of M. h ×M w A ×1 window is used to segment the input feature map into [H] non-overlapping segments. i / M h ]×[W i / M w ]×C i Each size is M h ×M w A spatial attention window of ×1 is swapped to become [H]. i / M h [W] i / M w C i A length of M h ×M w ×1 word sequence Then, attention is calculated for each word sequence within each window; M h M w These represent the spatial height and width of the window, respectively. When calculating the attention, the output of the depthwise convolution path is summed with the value V in the spatial window attention.

[0033] Furthermore, the decoding head network includes a pyramid pooling module (PMM) and an FPN network. The PMM uses pooling layers of different lengths to pool the input features, convolves the pooled features, restores the features back to their original resolution using bilinear interpolation, concatenates them with the input features, and finally fuses the multi-scale features through convolution.

[0034] Then, FPN is used to fuse feature images of different resolutions. FPN uses a set of convolutional layers with the same number of kernels to convolve the input feature map. Next, double bilinear interpolation is performed on the feature map from top to bottom, and it is summed with the high-resolution feature map. Finally, convolution is used to extract features from the fused feature map of different resolutions.

[0035] Furthermore, the cross-entropy loss function is used in training the network model, and its calculation formula is as follows:

[0036]

[0037] in, y represents the probability that sample i is predicted to be of the positive class. i The label represents sample i, with 1 for positive and 0 for negative.

[0038] The beneficial effects of this invention are as follows: First, the spectral and spatial window self-attention computation mechanism in this invention has lower computational overhead compared to the global self-attention mechanism, with its computational complexity increasing linearly with image size. Second, the dual-branch hybrid feature extraction module in this invention integrates the advantages of convolution computation in fine-grained local feature extraction, while simultaneously introducing the convolution result into the window self-attention computation, bringing the ability for cross-window interaction between windows to the window self-attention mechanism, thus overcoming the problem of limited receptive field in window self-attention and achieving the goal of global information modeling. Third, the spectral hybrid feature extraction module MixSE and the spatial hybrid feature extraction module MixSA in this invention focus on the extraction of spectral and spatial information features from hyperspectral images, respectively, giving them better feature extraction capabilities compared to joint spectral and spatial feature extraction. Attached Figure Description

[0039] Figure 1 The images and labels of gastric cancer pathological tissue sections acquired in this invention are shown.

[0040] Figure 2 This is a diagram of the backbone network MixDT structure of the present invention;

[0041] Figure 3 This is the spectral and spatial mixing feature extraction module of the present invention;

[0042] Figure 4 This invention relates to the window attention and point convolution interaction structure of the spectral mixing module.

[0043] Figure 5 This invention relates to the spatial hybrid module window attention and depth convolution interaction structure;

[0044] Figure 6 Here is a diagram of the UPerNet decoding header structure;

[0045] Figure 7 This is the segmentation result of the present invention on a gastric cancer microscopic hyperspectral dataset;

[0046] Figure 8 The results show the segmentation of six segmentation methods on a gastric cancer microscopic hyperspectral dataset. Detailed Implementation

[0047] The specific embodiments of the present invention will be further described in detail below with reference to the accompanying drawings.

[0048] The first step is to construct a hyperspectral microscopic dataset of gastric cancer pathological tissue sections. The specific steps are as follows:

[0049] Data collection

[0050] Data acquisition consisted of two parts: obtaining pathological slides and performing hyperspectral imaging on the slides. The pathological slides were provided by the hospital, and each slide included both normal tissue areas and cancerous areas. After obtaining the slide samples, they were placed in a hyperspectral microscopy imaging system. The microscope light source, objective lens setting, imager exposure parameters, and spectral band number parameters were adjusted. The stage was moved and the microscope focus was adjusted. A suitable area with a clear and bright field of view was selected for image acquisition, resulting in a hyperspectral microscopy image. Simultaneously, the corresponding visible light image was saved for subsequent image annotation. After sample acquisition, blank areas of the slides were also imaged for spectral correction.

[0051] Image annotation

[0052] Doctors used annotation software to annotate the acquired visible light images. They selected cancerous areas by clicking on markers, forming closed polygons, and assigned them corresponding positive labels. The background and some non-cancerous areas within the cancerous regions were also annotated by the doctors and assigned negative labels. The acquired images and labels are shown below. Figure 1 As shown, the left side is the acquired visible light image, and the right side is the labeled image, where the red part represents the cancerous area and the gray part represents the non-cancer area.

[0053] Data preprocessing

[0054] First, spectral correction is performed on the original microscopic hyperspectral images: the hyperspectral data of the acquired pathological slides are divided position by position with the corresponding blank slide data, and the light intensity values ​​in the original data are converted into transmittance.

[0055] Secondly, spectral dimensionality reduction is required: the hyperspectral images acquired by the microscopic visible light hyperspectral imaging system have a spectral range of 400-1000 nm and 300 channels. Due to data redundancy and the large number of spectral bands, some bands have low signal-to-noise ratios. Therefore, 150 clear bands (0-150 nm) are selected and retained. Then, principal component analysis (PCA) is used to reduce the number of spectral bands to 60 channels, which contains most of the image information.

[0056] After completing the data collection and preprocessing, the data needs to be further divided. The dataset includes 260 microscopic hyperspectral images. Considering the small size of the dataset, to avoid random errors caused by its small size, a five-fold cross-validation method is adopted for the experiment. All hyperspectral images are divided into five piles, each containing 52 hyperspectral images. Each fold uses one pile as the test set, and the remaining four piles as the training set. Each training set in each fold contains 208 hyperspectral images.

[0057] After completing the dataset construction, we will build a deep learning algorithm model to implement the image segmentation task. The specific steps are as follows:

[0058] Data Augmentation

[0059] Due to the small number of samples in the dataset, data augmentation is necessary to achieve better training results. Considering the massive amount of hyperspectral data, it cannot be directly input into the model for training. Therefore, it needs to be scaled first, randomly cropped to a size of 224*224, and then randomly flipped.

[0060] Model building

[0061] The deep learning algorithm model consists of two parts: the feature extraction backbone network MixDT (Mixing Dual Transformer) and the decoding head network UPerNet. The specific steps for building the deep learning algorithm model are as follows:

[0062] After data augmentation, the input hyperspectral image data X∈R H×W×C , where H, W, and C represent the length, width, and number of spectral bands of the input image, respectively.

[0063] The feature extraction backbone network consists of four stages, each stage comprising an image embedding module and several hybrid feature extraction modules, such as... Figure 2 As shown. The hybrid feature extraction module is based on the window attention mechanism, and the hybrid convolution calculation constitutes the fusion feature extraction module, such as... Figure 3 As shown, the number of hybrid feature extraction modules in each stage are [3, 4, 6, 3].

[0064] The image embedding module specifically refers to the input of the i-th stage. First, the input is processed through an image embedding layer to obtain feature representations of the hyperspectral image at different spatial resolutions. The embedding layer uses 2D convolution with a kernel size of 3×3 (7×7 in the first stage), a stride of 2×2 (4×4 in the first stage), a padding size of 1×1 (3×3 in the first stage), and C input channels. i-1 And the output channel is 2C i-1 The image is divided into overlapping image blocks of size 2×2 (4×4 in the first stage), resulting in a total of (H) i / 2)×(W i / 2)×C i (The first stage is (H) i / 4)×(W i / 4)×C i( ) image patches. After image patch embedding is completed, layer normalization (LN) is added to facilitate network convergence, resulting in the feature representation X after image embedding. i .

[0065] Then the obtained feature representation X i The input is fed into the hybrid feature extraction module, which includes a spectral hybrid feature extraction module and a spatial hybrid feature extraction module, specifically including:

[0066] The obtained feature representation X i The input is fed into the Transformer-based spectral blending feature extraction module MixSE, which contains two branches: a point convolution path and a spectral window attention path, such as... Figure 4 As shown.

[0067] The point convolutional pathway consists of three point convolutional layers, one normalization layer, and one activation layer, enabling cross-channel interaction of spectral channels.

[0068] The spectral window attention SeWA pathway uses a size of 1×1×M. c A window is used to segment the input feature maps into H non-overlapping segments. i ×W i ×[C i / M c Each has a size of 1×1×M c The spectral attention window, where M c The spectral dimension of the window is such that its size does not exceed the number of feature channels. In this embodiment, M c =64. Swap the dimensions to make it 1×1×M c A length of M c sequence of tokens Then, after calculating the attention for each word sequence within each window, the windows are merged to restore the original feature representation size.

[0069] The specific steps for calculating attention include: Since window attention calculation lacks communication between windows, limiting the receptive field to within the window fails to achieve global modeling. To address this issue, the output of the point convolutional path is summed with the value V(Value) in SeWA before being used in the attention calculation. The specific formula is as follows:

[0070]

[0071] Where Q, K, and V represent the query, key, and value in the attention mechanism, respectively, V_conv is the output of the point convolutional path, and d k This is the scaling factor.

[0072] The outputs from both branches are fed into the Add&Norm layer, which consists of two parts: Add and Norm. The calculation formula is as follows:

[0073] LayerNorm(X+(PW(X)+SeWA(X)))

[0074] Where Add is a residual connection, Norm refers to Layer Normalization, PW represents a point convolutional path, PW(x) = V_conv, and SeWA represents a spectral window attention path.

[0075] Finally, the convolutional feed-forward layer is reached, which consists of fully connected layers, depthwise convolutional layers, GELU layers, and another fully connected layer. The feature amplification factors of the fully connected layers at each stage are [8, 8, 4, 4]. The depthwise convolutional layers with zero padding can replace fixed-size positional encoding to achieve positional encoding of words. Similarly, the result is fed into an Add&Norm layer, where the calculation is as follows:

[0076] LayerNorm(X+CFFN(X))

[0077] Where CFFN(X) represents the output of the pre-convolutional feedback layer.

[0078] The results are then fed into MixSA, a Transformer-based spatial blending feature extraction module. This module contains two branches: a deep convolutional path and a spatial window attention path. Figure 5 As shown, the structures of the remaining residual connections, layer normalization, and pre-convolution feedback layers are the same as those in the MixSE spectral mixing module and will not be listed one by one.

[0079] The deep convolutional channel branch consists of three deep convolutional layers, one normalization layer, and one activation layer. This pathway enables the extraction of local spatial features in each channel.

[0080] Spatial window attention SaWA pathway uses size M h ×M w A ×1 window is used to segment the input feature map into [H] non-overlapping segments. i / M h ]×[W i / M w ]×C i Each size is M h ×M w A spatial attention window of ×1 is swapped to become [H]. i / Mh [W] i / M w C i A length of M h ×M w ×1 word sequence Then, attention is calculated separately for each sequence of words within each window. M h M w These are the dimensions of the window's height and width, respectively. In this embodiment, M... h =M w =7.

[0081] Similarly, to address the issue that the limited receptive field of window attention prevents global modeling, the output of the deep convolutional path is summed with V in SaWA to achieve cross-window information interaction and fusion between windows in the spatial dimension.

[0082] Spatial window self-attention SaWA uses a 7×7×1 window to segment the input feature map non-overlappingly into [H... i / 7]×[W i / 7]×C i A spatial attention window of size 7×7×1 is swapped to transform it into a sequence of 49 tokens of length 1. Then, attention is calculated for each sequence of words within each window.

[0083] Additionally, a Dropout layer needs to be added after each module, with the random depth ratio parameter set to 0.1 and dropout set to 0.1, to prevent overfitting.

[0084] After repeating the above hybrid feature extraction module several times in each stage, the output feature representation of each stage is extracted and fed into the UPerNet segmentation network, such as... Figure 6 Its structure is as follows. UperNet uses the pyramid pooling module PMM proposed in PSPNet (Pyramid Scene Parsing) for the top-level input feature image. PMM uses pooling layers with strides of [1,2,3,6] to pool the input features, convolves the pooled features, and uses bilinear interpolation to restore the features back to the original resolution. These features are then concatenated with the input features, and finally, multi-scale features are fused through convolution.

[0085] The feature outputs at different resolutions obtained from each stage of the backbone feature extraction network are fused using FPN. The specific structure includes:

[0086] For feature maps of the same resolution, FPN uses a set of convolutional layers with the same number of kernels to convolve the input feature map. Then, it performs double bilinear interpolation on the feature map from top to bottom, simultaneously restoring the output of each layer to its original resolution using bilinear interpolation. Finally, it concatenates the feature maps of all layers according to their feature dimensions, and uses convolution to extract features from the feature maps containing information from different resolutions. The final output is the segmentation prediction image.

[0087] Model training

[0088] Five-fold cross-validation experiments were conducted using the obtained gastric cancer microscopic hyperspectral image dataset. The CrossEntropy Loss function was used, and its calculation formula is as follows:

[0089]

[0090] in, y represents the probability that sample i is predicted to be of the positive class. i The label represents sample i, with 1 for positive and 0 for negative.

[0091] The Adam optimizer was used to optimize the model parameters through backpropagation; finally, a trained microscopic pathology hyperspectral image segmentation network model was obtained and tested.

[0092] The hyperspectral image of the gastric cancer pathological section to be tested is input into the trained network model to obtain the segmentation prediction result of the microscopic hyperspectral image of the gastric cancer pathological section.

[0093] Model training and testing results

[0094] Using this dataset, a microscopic pathological hyperspectral image segmentation network model was trained and tested to obtain segmentation prediction results for microscopic hyperspectral images of gastric cancer pathological sections.

[0095] The training hardware and software environment for the model is shown in Table 1:

[0096]

[0097]

[0098] The training hyperparameters are shown in Table 2:

[0099] hyperparameters numerical values Patch_size 224*224 Batch_size 4 Iterations 30000 Optimizer AdmaW Base LR 0.0005

[0100] Table 3 shows the results of the five-fold cross-validation of the model:

[0101] Evaluation indicators mIoU% mDice% mAcc% Experimental results 75.93±2.34 86.29±1.78 86.43±1.30

[0102] like Figure 7The images shown represent the segmentation experiment results. From left to right, the three columns represent the pseudo-color image, the ground truth label, and the predicted label, with the white areas indicating predicted or normal cancerous regions. The first hyperspectral image shows relatively clear features and excellent segmentation results; the second hyperspectral image presents some challenges, with only average segmentation results; the third hyperspectral image lacks clear features, making it difficult to distinguish between cancerous and normal regions, resulting in poor segmentation results.

[0103] Table 4 shows the results of the ablation experiment:

[0104]

[0105] The ablation experiment results show that spatial features have a significant impact on the segmentation results, but spectral features can also effectively improve the network segmentation performance. This also proves that hyperspectral imaging technology has the advantage of rich spectral information compared to traditional images, and can achieve better performance in image segmentation, classification and other fields.

[0106] Table 5 shows the experimental results of this invention compared with other segmentation networks:

[0107]

[0108]

[0109] like Figure 8 As shown, the images from top to bottom are the segmentation results of the corresponding models in Table 5, where the first row is the pseudo-color image of the input data and the second row is the corresponding label. Experimental results show that the MixDT segmentation backbone network model proposed in this invention combines the advantages of convolution in local feature extraction and the advantages of Transformer in global feature extraction. It performs excellently on the image segmentation task of microscopic hyperspectral gastric cancer pathological tissue slice images. Its comprehensive performance has a significant advantage over the current cutting-edge algorithms in the field of image segmentation, which also shows that hyperspectral imaging technology has broad application prospects in the field of medical imaging.

[0110] The above embodiments are used to explain and illustrate the present invention, but not to limit the present invention. Any modifications and changes made to the present invention within the spirit and scope of the claims shall fall within the protection scope of the present invention.

Claims

1. An image segmentation method based on microscopic hyperspectral imaging technology, characterized in that, The method includes the following steps: Dataset processing: Hyperspectral and visible light images of gastric cancer pathological sections were acquired, and the visible light images were manually labeled; the hyperspectral images were preprocessed and the dataset was divided to obtain a gastric cancer microscopic hyperspectral image dataset. Constructing a network model for segmenting hyperspectral images of microscopic pathology: This model includes a feature extraction backbone network and a decoding head network; The feature extraction backbone network includes four stages, each stage including an image embedding module and several hybrid feature extraction modules. The hybrid feature extraction module includes a spectral hybrid feature extraction submodule and a spatial hybrid feature extraction submodule, which extract features of different dimensions through spectral attention windows and spatial attention windows, respectively. The image embedding module in the feature extraction backbone network is specifically: No. Phase input First, the input is processed through an image embedding module to obtain feature representations of the hyperspectral image at different spatial resolutions. The first-stage image embedding module uses two-dimensional convolution with the kernel size set to [value missing]. Step size is Fill size is Input channel is and the output channel is The image is segmented into overlapping parts of size . The image blocks obtained a total of ) image blocks; The embedding modules in the remaining stages use two-dimensional convolution with the kernel size set to 2D. Step size is Fill size is Input channel is and the output channel is The image is segmented into overlapping parts of size . The image blocks obtained a total of Image blocks; After completing the image patch embedding, layer normalization is added to obtain the feature representation of the embedded image. ; The spectral blending feature extraction module in the feature extraction backbone network specifically includes two branches: a point convolutional path and a spectral window attention path. The outputs of the two branches are input into the first Add & Norm layer and then into the pre-convolution feedback layer. The result is then input into the second Add & Norm layer. The spatial blending feature extraction module specifically includes two branches: a depthwise convolutional path and a spatial window attention path. The outputs of the two branches are input into the first Add & Norm layer and then into the pre-convolution feedback layer. The result is then input into the second Add & Norm layer. The decoding head network is used to fuse feature outputs of different resolutions obtained from each stage of the feature extraction backbone network; The network model is trained to obtain a well-trained microscopic pathological hyperspectral image segmentation network model; The hyperspectral image of the gastric cancer pathological section to be tested is input into the trained network model to obtain the segmentation prediction result of the microscopic hyperspectral image of the gastric cancer pathological section.

2. The image segmentation method based on microscopic hyperspectral imaging technology according to claim 1, characterized in that, The manual annotation of the visible light image specifically involves: selecting the cancerous area in the visible light image and setting it as a corresponding positive label; similarly annotating the background and some non-cancer areas inside the cancerous area in the visible light image and setting them as negative labels.

3. The image segmentation method based on microscopic hyperspectral imaging technology according to claim 1, characterized in that, The data preprocessing includes spectral correction and spectral dimensionality reduction. The spectral correction specifically involves dividing the acquired hyperspectral data of the pathological slides with the corresponding blank slide data position by position to convert the light intensity values ​​in the original data into transmittance. The spectral dimensionality reduction specifically involves selecting clear bands in the spectral bands and using the PCA principal component analysis method to further reduce the number of spectral bands in the image.

4. The image segmentation method based on microscopic hyperspectral imaging technology according to claim 1, characterized in that, The point convolutional pathway consists of three point convolutional layers, one normalization layer, and one activation layer. The spectral window attention SeWA pathway uses a size of [size missing]. A window is used to segment the input feature maps into non-overlapping segments. Each size is The spectral attention window, in which The spectral dimension of the window is the size, which does not exceed the number of feature channels; Exchange dimensions to make it become A length of sequence of tokens Then, attention is calculated for each word sequence within each window, and the windows are merged to restore the original feature representation size. The attention calculation specifically involves summing the output of the point convolutional path with the value in SeWA before using it in the attention calculation. The specific formula is as follows: Where Q, K, and V represent the query, key, and value in the attention mechanism, respectively, and V_conv is the output of the point convolution path. This is the scaling factor.

5. The image segmentation method based on microscopic hyperspectral imaging technology according to claim 1, characterized in that, The first Add & Norm layer consists of two parts: Add and Norm. The calculation formula is as follows: Here, Add represents a residual connection, and Norm refers to layer normalization. Represents the point convolutional path, Represents the attention pathway of the spectral window; The calculation of the second Add & Norm layer is as follows: Where CFFN(X) represents the output of the pre-convolutional feedback layer.

6. The image segmentation method based on microscopic hyperspectral imaging technology according to claim 1, characterized in that, The spatial hybrid feature extraction module includes two branches: a depthwise convolutional path and a spatial window attention path. The deep convolutional channel branch consists of three deep convolutional layers, one normalization layer, and one activation layer; the spatial window attention path uses a size of [size missing]. The window segments the input feature map into non-overlapping segments. Each size is Spatial attention window, swapping dimensions to make it into A length of sequence of tokens Then, attention is calculated for each word sequence within each window; , These represent the spatial height and width dimensions of the window, respectively. When calculating the attention, the output of the depthwise convolutional path is summed with the value in the spatial window attention.

7. The image segmentation method based on microscopic hyperspectral imaging technology according to claim 1, characterized in that, The decoding head network includes a pyramid pooling module (PMM) and an FPN network. The PMM uses pooling layers of different lengths to pool the input features, convolves the pooled features, and uses bilinear interpolation to restore the features back to their original resolution. These features are then concatenated with the input features, and finally, multi-scale features are fused through convolution. Then, FPN is used to fuse feature images of different resolutions. FPN uses a set of convolutional layers with the same number of kernels to convolve the input feature map. Next, double bilinear interpolation is performed on the feature map from top to bottom, and it is summed with the high-resolution feature map. Finally, convolution is used to extract features from the fused feature map of different resolutions.

8. The image segmentation method based on microscopic hyperspectral imaging technology according to claim 1, characterized in that, The cross-entropy loss function is used in training the network model, and its calculation formula is as follows: in, Indicates sample The probability of predicting a positive class. Indicates sample The label is 1 for positive and 0 for negative.

Images