Hyperspectral image semantic segmentation method and device based on spatial-spectral dual-stream network

Abstract

The present invention discloses a method and apparatus for semantic segmentation of hyperspectral images based on a spatial-spectral dual-stream network, comprising the following steps: Step S1: Acquiring a standard dataset for semantic segmentation of hyperspectral images; Step S2: Inputting the standard dataset into the spatial-spectral dual-stream network for semantic mask prediction training to obtain a prediction network; Step S3: Inputting the hyperspectral image to be processed into the prediction network for target semantic mask prediction. The technical solution of the present invention overcomes the shortcomings of existing solutions based on 2D networks, such as insufficient spatial-spectral information extraction, and low computational efficiency in 3D network solutions.

Description

Technical Field

[0001] The present invention belongs to the technical field of medical image processing, and in particular relates to a method and device for semantic segmentation of hyperspectral images based on a spatial-spectral dual-stream network. Background Art

[0002] Medical hyperspectral imaging is an emerging imaging modality that can provide rich spectral information to color medical images. Typically, a medical hyperspectral image is represented as a hypercube, with hundreds of continuous spectral bands in the spectral dimension and thousands of pixels in the spatial dimension. However, the high spatial and spectral dimensionality makes it difficult to effectively extract spectral and spatial information features, posing a challenge to hyperspectral image segmentation.

[0003] In the past few years, a lot of research has been done to develop efficient and accurate hyperspectral image feature extraction and segmentation methods. Figure 2 (a) The method of feature extraction and segmentation of HSI based on two-dimensional (2D) deep convolutional neural network is to treat the hyperspectral image as two-dimensional data and input it into the 2D network, and its spectral dimension is regarded as the input channel dimension; Figure 2 (b) Combining principal component analysis (PCA) to reduce the dimensionality of the hyperspectral image and then input it into the 2D network. However, this type of 2D network-based method will mix the spectral and spatial information in the first convolution layer, so that the spectral correlation information of the hyperspectral image is not fully mined. Figure 2 (c) The hyperspectral image feature extraction method based on 3D networks can better construct spatial spectral features at the same time, but the computational complexity of 3D networks is usually much higher than that of 2D networks, which affects the practical application of the model in clinical practice.

[0004] Since the amount of information in spatial and spectral features is different, there is no need to process both spatial and spectral information simultaneously, as is done in 3D networks. Figure 2 (d) A two-stream strategy can be designed to process spatial and spectral features separately. Some hyperspectral image classification networks attempt to employ a two-stream architecture to process spatial structure and spectral features separately. However, these networks simply add or concatenate the two-stream features in the final layer. This feature fusion algorithm is immature. Effectively exploring the interaction of spatial and spectral features at high spatial and spectral dimensions is not yet established, and migrating these classification network architectures to segmentation networks is not easy.

[0005] The high spatial and spectral dimensions make it difficult to conduct a comprehensive analysis of hyperspectral images. In HSI, there are two types of correlations, one is the spectral correlation among adjacent pixels. Figure 3As shown in Figure 2, the amplitudes of local positive (cancer) and negative (normal) regions are highly correlated with the spectral bands. Another is the spatial correlation between adjacent bands. Figure 3 The spatial similarity between all bands is plotted, showing that the cosine similarity between nearby bands is large, while the cosine similarity between distant bands is small. This correlation has spectral redundancy in characterizing spatial features and spatial redundancy in learning spectral features. Summary of the Invention

[0006] The present invention aims to address the deficiencies of the prior art and proposes a method and device for semantic segmentation of hyperspectral images based on a spatial-spectral dual-stream network with better segmentation effect and reduced computational consumption, thereby overcoming the deficiencies of insufficient spatial-spectral information extraction in existing 2D network-based solutions and low computational efficiency in 3D network solutions.

[0007] To achieve the above object, the present invention provides the following solutions:

[0008] A method for semantic segmentation of hyperspectral images based on a spatial-spectral dual-stream network includes the following steps:

[0009] Step S1, obtaining a standard dataset for semantic segmentation of hyperspectral images;

[0010] Step S2: inputting the hyperspectral image semantic segmentation standard dataset into the space-spectrum dual-stream network for mask prediction training to obtain a prediction network;

[0011] Step S3: inputting the hyperspectral image to be processed into the prediction network to perform target semantic mask prediction.

[0012] Preferably, the hyperspectral image semantic segmentation standard dataset includes: a pathological microscopic hyperspectral cholangiocarcinoma dataset and a spectral gastric cancer dataset.

[0013] Preferably, step S2 includes:

[0014] The hyperspectral images in the training set of the hyperspectral image semantic segmentation standard dataset are input into the spatial-spectral two-stream network to obtain their predicted masks;

[0015] The gap between the output mask of the spatial-spectral dual-stream network and the labeled standard mask in the training set is calculated to obtain the loss and the gradient is returned to update the network parameters. This process is repeated until the number of training times reaches a preset number to complete the mask prediction training and obtain the prediction network.

[0016] Preferably, in step S2, the spatial-spectral dual-stream network adopts a dual-stream parallel network structure, including an encoder that decouples space and a spectral domain and a decoder that fuses spatial-spectral features, and there are four horizontally connected cross-scale features between the encoder and the decoder; wherein, the semantic mask prediction training is specifically as follows: the hyperspectral image semantic segmentation standard dataset is first decoupled from the spatial and spectral domains and input into the spatial-spectral dual-stream encoder respectively, and the spatial and spectral features of the encoder are fused at four different spatial sizes and then input into four multi-scale decoders, and the features obtained by the decoders of different scales are decoded step by step to the same scale as the original image to obtain a predicted mask.

[0017] Preferably, the encoder is composed of two branches, namely a spatial encoder and a spectral encoder. The spatial encoder is composed of multiple layers of convolution modules; the spectral encoder is composed of multiple layers of nonlinear layers.

[0018] Preferably, during the spatial-spectral dual-stream network training process, when calculating the loss, the hyperspectral images in the training set are decoupled from the spatial and spectral domains respectively, and then input into the dual-stream encoder and decoder. The decoder outputs the predicted mask and calculates the loss together with the annotated real mask.

[0019] The present invention also provides a hyperspectral image semantic segmentation device based on a spatial-spectral dual-stream network, comprising:

[0020] Acquisition module, used to obtain the standard dataset of hyperspectral image semantic segmentation;

[0021] A training module is used to input the hyperspectral image semantic segmentation standard dataset into the spatial-spectral dual-stream network for mask prediction training to obtain a prediction network;

[0022] The segmentation module is used to input the hyperspectral image to be processed into the prediction network to perform target semantic mask prediction.

[0023] Preferably, the hyperspectral image semantic segmentation standard dataset includes: a pathological microscopic hyperspectral cholangiocarcinoma dataset and a spectral gastric cancer dataset.

[0024] Preferably, the spatial-spectral dual-stream network adopts a dual-stream parallel network structure, which includes an encoder for decoupling space and a spectral domain and a decoder for fusing spatial-spectral features. There are four horizontally connected cross-scale features between the encoder and the decoder.

[0025] Compared with the prior art, the present invention has the following beneficial effects:

[0026] This paper proposes a dual-stream hyperspectral semantic segmentation network, decoupling hyperspectral spatial and spectral information. This network achieves efficient utilization of spatial and spectral information by separately encoding information in the spatial and spectral domains. Furthermore, by combining a 2D-U-shaped network structure with an attention mechanism, it effectively fuses and decodes spatial and spectral information, while improving both accuracy and computational efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] In order to more clearly illustrate the technical solution of the present invention, the following briefly introduces the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0028] Figure 1 Schematic diagram of the process of a fast hyperspectral image semantic segmentation method based on a spatial-spectral dual-stream network according to an embodiment of the present invention;

[0029] Figure 2 Schematic diagram of traditional hyperspectral image processing method;

[0030] Figure 3 Schematic diagram for visualizing spatial redundancy and spectral sparsity of hyperspectral images;

[0031] Figure 4 Schematic diagram of the space-spectrum dual-stream network model of the present invention;

[0032] Figure 5 Schematic diagram of an example of empty spectrum dual-stream network test data. DETAILED DESCRIPTION

[0033] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0034] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0035] Example 1:

[0036] The embodiment of the present invention provides a hyperspectral image semantic segmentation method based on a spatial-spectral dual-stream network, comprising the following steps:

[0037] Step S1, obtaining a standard dataset for semantic segmentation of hyperspectral images;

[0038] Step S2: inputting the hyperspectral image semantic segmentation standard dataset into the space-spectrum dual-stream network for mask prediction training to obtain a prediction network;

[0039] Step S3: inputting the hyperspectral image to be processed into the prediction network to perform target semantic mask prediction.

[0040] As an implementation example of the present invention, the hyperspectral image semantic segmentation standard dataset includes: a pathological microscopic hyperspectral cholangiocarcinoma dataset and a spectral gastric cancer dataset.

[0041] As an embodiment of the present invention, step S2 includes:

[0042] The hyperspectral images in the training set of the hyperspectral image semantic segmentation standard dataset are input into the spatial-spectral two-stream network to obtain their predicted masks;

[0043] The gap between the output mask of the spatial-spectral dual-stream network and the labeled standard mask in the training set is calculated to obtain the loss and the gradient is returned to update the network parameters. This process is repeated until the number of training times reaches a preset number to complete the mask prediction training and obtain the prediction network.

[0044] As an implementation mode of the present invention, in step S2, the spatial-spectral dual-stream network adopts a dual-stream parallel network structure, including an encoder for decoupling space and a spectral domain and a decoder for fusing spatial-spectral features, and there are four horizontally connected cross-scale features between the encoder and the decoder; wherein, the semantic mask prediction training is specifically as follows: the hyperspectral image semantic segmentation standard dataset is first decoupled from the spatial and spectral domains and input into the spatial-spectral dual-stream encoder respectively, and the spatial and spectral features of the encoder are fused at four different spatial sizes and then input into four multi-scale decoders, and the features obtained by the decoders of different scales are decoded step by step to the same scale as the original image to obtain a predicted mask.

[0045] As an implementation mode of the present invention, the encoder is composed of two branches, namely spatial and spectral encoders. The spatial encoder is composed of multiple layers of convolution modules; the spectral encoder is composed of multiple layers of nonlinear layers.

[0046] As an implementation method of the present invention, during the spatial-spectral dual-stream network training process, when calculating the loss, the hyperspectral images in the training set will be decoupled from the spatial and spectral domains respectively, and then input into the dual-stream encoder and decoder. The decoder outputs the predicted mask and calculates the loss together with the annotated real mask.

[0047] The present invention combines the spectral information and spatial information of hyperspectral images, and can effectively reduce the computational complexity of the model and improve the model prediction efficiency by decoupling the spectral and spatial information.

[0048] The innovation of the present invention lies in the design of a spatial-spectral dual-stream network. The semantic segmentation task of hyperspectral images requires the simultaneous extraction of spatial and spectral features during image processing. However, current mainstream 3D convolutional network methods for simultaneously acquiring spatial and spectral features, such as HyperNet and 3D-UNet, suffer from problems such as long computation time and high video memory consumption. Furthermore, current mainstream deep neural networks still focus on the spatial information of hyperspectral images, while the spectral information of hyperspectral images is currently neglected in medical hyperspectral images. The spatial-spectral dual-stream network successfully integrates the spatial and spectral information of features into the convolutional neural network, increasing the model's ability to acquire and process spectral information, improving model performance, and reducing computational costs.

[0049] Example 2:

[0050] like Figure 1 As shown, the embodiment of the present invention provides a hyperspectral image semantic segmentation method based on a spatial-spectral dual-stream network, which includes: a network training stage and a network prediction stage; wherein,

[0051] The training phase of the network consists of the following steps:

[0052] 1. Obtain a hyperspectral image semantic segmentation dataset, which includes the pathological microscopic hyperspectral cholangiocarcinoma dataset (MDC) and the hyperspectral gastric cancer dataset (PLGC).

[0053] 2. Preprocess the hyperspectral image semantic segmentation dataset. Before inputting the hyperspectral image semantic segmentation dataset into the spatial-spectral two-stream network training, the data is spatially downsampled by a factor of 4. The image size of the MDC dataset is 256×320, and the image size of the PLGC dataset is 512×640. The data is randomly flipped horizontally.

[0054] 3. The hyperspectral images in the training set of the hyperspectral image semantic segmentation dataset are input into the spatial-spectral dual-stream network to obtain the estimated possible target semantic mask spatial-spectral dual-stream.

[0055] 4. Calculate the gap between the output image of the spatial-spectral dual-stream network and the clear image in the training set to obtain the loss and return the gradient to update the network parameters, and repeat this process until the number of training times reaches a preset number. The training configuration adopted by the present invention on the training set of each data set is inherited from, the optimizer adopts Adam, the gradient descent strategy is the cosine annealing strategy (Cosine Annealing Strategy), the number of samples trained at one time is 8, the initial learning rate is 1e-4, the final learning rate is 1e-6, and a total of 100 rounds of training. When calculating the loss, the hyperspectral images in the training set will be decoupled into the spatial and spectral domains to the dual-stream encoder and decoder respectively, and the decoder outputs the predicted mask and calculates the loss with the annotated real mask. The loss function of network training is the joint loss of Dice loss and Cross-entropy loss:

[0056] 1))Dice Loss:

[0057]

[0058] 2) Cross-Entropy Loss:

[0059]

[0060] The final loss function is as follows:

[0061] L=L Dice +aL CE

[0062] Among them, α and ∈ are hyperparameters, which are set to 1.0 and 0.001 respectively. i and y i They represent the prediction results of the model and the true mask annotation respectively, and L is the joint loss of Dice loss and Cross-entropy loss.

[0063] Prediction phase of the network: The hyperspectral image to be processed is input into the prediction network to perform target semantic mask prediction.

[0064] Further, refer to Figure 4, the spatial-spectral dual-stream network (Dual-Stream) adopts a dual-stream parallel network structure, which includes an encoder that decouples the spatial and spectral domains and a decoder that fuses the spatial-spectral features. There are four horizontal connections and four cross-scale connections between the encoder and the decoder. The hyperspectral image semantic segmentation dataset first decouples the spatial and spectral domains and inputs them into the spatial-spectral dual-stream encoder respectively. The spatial and spectral features of the encoder are fused at four different spatial sizes and then input into four multi-scale decoders. The features obtained by the decoders at different scales are decoded step by step to the same scale as the original image, and finally the target prediction mask is obtained. The present invention mainly designs the spatial and spectral encoders and the spatial-spectral attention fusion module. These three modules are plug-and-play and can be applied to different segmentation architectures and spatial backbone encoders. Reference Figure 4 , we use a 2D convolutional neural network with UNet as the backbone. The 2D convolutional neural network in the figure is an example that can represent all widely used two-dimensional architectures. It represents a spatial encoder that focuses on extracting spatial features from the hyperspectral spectrum. For the spectral encoder, in order to produce a multi-layer representation that can effectively contain space, we propose a patch fusion layer to reduce the spatial redundancy of spectral information. In order to integrate information from the spatial spectral dimension, we propose an S 2 -Fusion module. Figure 4 , after the second and fourth encoder layers, S 2 The -Fusion module is applied twice, and then the SE module is applied to model the interdependence between the channels of the fused features to fuse the modular spatial and spectral features.

[0065] Through the above operations, the present invention completes the modeling of all spatial and spectral information and correlations. The final output of the multi-layer cascade is then calculated through a convolutional network to obtain a segmentation prediction mask.

[0066] Reference Figure 5 , are the prediction results for two examples of hyperspectral images using different hyperspectral segmentation models: ResNet34-UNet, PCA-ResNet34-UNet, 3D-UNet, nnUNet, HyperNet, and UNETR. The corresponding number for the prediction result of the spatial-spectral dual-stream network (Dual-Stream) is the quantitative indicator DSC (dice index). Comparing the results of Dual-Stream with other similar methods demonstrates that this method can effectively improve the ability of convolutional neural networks to extract spatial and spectral features of hyperspectral images. Compared to currently mainstream 3D convolutional neural networks, Dual-Stream has a greater advantage in computational speed.

[0067] In addition, the embodiment of the present invention also provides a specific example to assist in proving that:

[0068] ResNet34-UNet, PCA-ResNet34-UNet, 3D-UNet, nnUNet, HyperNet, UNETR, and a spatial-spectral dual-stream network (Dual-Stream) were trained using the MDC and PLGC training sets and validated on the test set. Table 1 shows the DSC (Dice Index) and throughput performance of each network. On the MDC dataset, Dual-Stream achieved a 0.95% DSC improvement over nnUNet, while increasing inference speed by 24 times. On the PLGC dataset, Dual-Stream achieved a 2.36% DSC improvement over HyperNet, while increasing inference speed by 9.53 times. Therefore, the Dual-Stream model surpassed previous models in both speed and accuracy.

[0069] Table 1

[0070]

[0071] Example 3:

[0072] The embodiment of the present invention further provides a hyperspectral image semantic segmentation device based on a spatial-spectral dual-stream network, comprising:

[0073] Acquisition module, used to obtain the standard dataset of hyperspectral image semantic segmentation;

[0074] A training module is used to input the hyperspectral image semantic segmentation standard dataset into the spatial-spectral dual-stream network for mask prediction training to obtain a prediction network;

[0075] The segmentation module is used to input the hyperspectral image to be processed into the prediction network to perform target semantic mask prediction.

[0076] As an implementation example of the present invention, the hyperspectral image semantic segmentation standard dataset includes: a pathological microscopic hyperspectral cholangiocarcinoma dataset and a spectral gastric cancer dataset.

[0077] As an implementation example of the present invention, the spatial-spectral dual-stream network adopts a dual-stream parallel network structure, including an encoder for decoupling space and a spectral domain and a decoder for fusing spatial-spectral features. There are four horizontally connected cross-scale features between the encoder and the decoder.

[0078] The embodiments described above are merely descriptions of preferred embodiments of the present invention and are not intended to limit the scope of the present invention. Without departing from the spirit of the present invention, various modifications and improvements made to the technical solutions of the present invention by persons skilled in the art should fall within the scope of protection defined by the claims of the present invention.

Claims

1. A hyperspectral image semantic segmentation method based on a spatial-spectral dual-stream network, characterized in that: The following steps are involved: Step S1, obtaining a standard dataset for semantic segmentation of hyperspectral images; Step S2: inputting the hyperspectral image semantic segmentation standard dataset into the space-spectrum dual-stream network for mask prediction training to obtain a prediction network; Step S3: inputting the hyperspectral image to be processed into the prediction network to perform target semantic mask prediction; In step S2, the spatial-spectral dual-stream network adopts a dual-stream parallel network structure, including an encoder that decouples the spatial and spectral domains and a decoder that fuses the spatial-spectral features. There are four horizontally connected cross-scale features between the encoder and the decoder. The semantic mask prediction training is specifically as follows: the hyperspectral image semantic segmentation standard dataset is first decoupled from the spatial and spectral domains and input into the spatial-spectral dual-stream encoder respectively. The spatial and spectral features of the encoder are fused at four different spatial scales and then input into four multi-scale decoders. The features obtained by the decoders at different scales are decoded step by step to the same scale as the original image to obtain a predicted mask. The encoder consists of two branches, namely spatial and spectral encoders. The spatial encoder is composed of multiple layers of convolution modules; the spectral encoder is composed of multiple layers of nonlinear layers.

2. The method for semantic segmentation of hyperspectral images based on a spatial-spectral dual-stream network according to claim 1, wherein: The hyperspectral image semantic segmentation standard dataset includes: a pathological microscopic hyperspectral bile duct cancer dataset and a spectral gastric cancer dataset.

3. The method for semantic segmentation of hyperspectral images based on a spatial-spectral dual-stream network according to claim 2, wherein: Step S2 includes: The hyperspectral images in the training set of the hyperspectral image semantic segmentation standard dataset are input into the spatial-spectral two-stream network to obtain their predicted masks; The gap between the output mask of the spatial-spectral dual-stream network and the labeled standard mask in the training set is calculated to obtain the loss and the gradient is returned to update the network parameters. This process is repeated until the number of training times reaches a preset number to complete the mask prediction training and obtain the prediction network.

4. The method for semantic segmentation of hyperspectral images based on a spatial-spectral dual-stream network according to claim 3, wherein: During the spatial-spectral dual-stream network training process, when calculating the loss, the hyperspectral images in the training set are decoupled from the spatial and spectral domains respectively, and then input into the dual-stream encoder and decoder. The decoder outputs the predicted mask and calculates the loss together with the annotated real mask.

5. A hyperspectral image semantic segmentation device based on a spatial-spectral dual-stream network that implements the hyperspectral image semantic segmentation method based on a spatial-spectral dual-stream network according to claim 1, characterized in that: include: Acquisition module, used to obtain the standard dataset of hyperspectral image semantic segmentation; A training module is used to input the hyperspectral image semantic segmentation standard dataset into the spatial-spectral dual-stream network for mask prediction training to obtain a prediction network; The segmentation module is used to input the hyperspectral image to be processed into the prediction network to perform target semantic mask prediction.

6. The hyperspectral image semantic segmentation device based on a spatial-spectral dual-stream network according to claim 5, characterized in that: The hyperspectral image semantic segmentation standard dataset includes: a pathological microscopic hyperspectral bile duct cancer dataset and a spectral gastric cancer dataset.

Images