A pathological hyperspectral image classification method based on multi-scale space spectrum features

Abstract

The application discloses a pathological hyperspectral image classification method based on multi-scale space-spectrum features, acquires a pathological section hyperspectral image and corresponding pixel-level labels, pre-processes the hyperspectral image to obtain a hyperspectral patch image and corresponding classification labels, builds a multi-scale space-spectrum feature classification network and presets a loss function of the network, inputs a preset training set into the built multi-scale space-spectrum feature classification network for training, calculates a loss value of the network according to the preset loss function, and reversely propagates to update network parameters of the network to obtain an updated multi-scale space-spectrum feature classification network, inputs the hyperspectral patch image into the updated multi-scale space-spectrum feature classification network, combines the classification labels, and outputs a classification result of the hyperspectral patch image. The classification method fully utilizes space neighborhood features and hyperspectral features, removes irrelevant input features, and adds regularization to relieve overfitting problems when a small-scale data set is used.

Description

Technical Field

[0001] This invention belongs to the field of hyperspectral image processing, and in particular relates to a pathological hyperspectral image classification method based on multi-scale spatial spectral features. Background Technology

[0002] Hyperspectral images (HSI) typically contain hundreds of spectral channels, and each pixel in the image has a corresponding spectral curve. Hyperspectral image classification involves categorizing pixels within the image based on the characteristics of their spectral curves. Hyperspectral image classification has been widely applied in land cover mapping, mineral exploration, and water pollution detection, and also holds broad promise for applications in medical pathology diagnosis.

[0003] A typical medical pathology diagnostic process includes tissue sample preparation, slide analysis, and disease diagnosis; the procedure is complex and time-consuming. Computer-aided image analysis algorithms can significantly assist pathologists in viewing digitized images, greatly aiding doctors in assessing patient conditions; however, these algorithms require large-scale datasets. In reality, collecting sufficiently large amounts of medical image data is both difficult and expensive. Algorithms combining some transfer learning methods and unsupervised learning with clustering are often difficult for medical professionals and developers to understand and implement, thus requiring further work to improve the interpretability of the research. Hyperspectral imaging technology, as a novel imaging method, can simultaneously acquire spectral and spatial information from pathological slides, and the measurement data contains meaningful and rich information, showing great potential for applications in medical diagnosis.

[0004] Current classification methods for hyperspectral images can be divided into supervised and unsupervised approaches. Unsupervised methods require only a small amount of labeled data, but their classification accuracy is relatively lower than that of supervised methods. Therefore, improving the classification accuracy of networks when labeled data is very limited is an urgent problem to be solved. This method constructs a novel hyperspectral image classification framework that integrates effective spectral spatial features from hyperspectral data. By removing irrelevant input features and adding regularization, it alleviates the overfitting problem when using small datasets, demonstrating superior classification performance even with limited labeled data. Summary of the Invention

[0005] This invention provides a pathological hyperspectral image classification method based on multi-scale spatial spectral features, which can exhibit superior classification performance even when training samples are significantly insufficient.

[0006] The technical solution adopted by this invention to solve its technical problem is:

[0007] A pathological hyperspectral image classification method based on multi-scale spatial spectral features includes the following steps:

[0008] S1. Obtain hyperspectral images of pathological sections and corresponding pixel-level labels. Preprocess the hyperspectral images to obtain hyperspectral patch images. Obtain classification labels for the hyperspectral patch images based on the pixel-level labels and the hyperspectral patch images.

[0009] S2. Construct a multi-scale spatial spectral feature classification network. The network includes an adaptive spectral feature weighting network, a multi-scale spatial spectral fusion network, and a reconstruction network connected in sequence. Preset the loss functions of the adaptive spectral feature weighting network, the multi-scale spatial spectral fusion network, and the reconstruction network respectively, and add the loss functions to obtain the total loss function of the multi-scale spatial spectral feature classification network.

[0010] S3. Input the preset training set into the constructed multi-scale spatial spectrum feature classification network for training, calculate the total loss of the multi-scale spatial spectrum feature classification network according to the total loss function of the multi-scale spatial spectrum feature classification network, and backpropagate to update the network parameters of the multi-scale spatial spectrum feature classification network to obtain the updated multi-scale spatial spectrum feature classification network.

[0011] S4. Input the hyperspectral patch image into the updated multi-scale spatial spectral feature classification network, combine it with the classification label of the hyperspectral patch image, and output the classification result of the hyperspectral patch image.

[0012] In one embodiment, step S1 involves preprocessing the hyperspectral image to obtain a hyperspectral patch image, specifically as follows:

[0013] S11. Compress the hyperspectral image to obtain a compressed hyperspectral image of a preset size;

[0014] S12. Perform Gaussian smoothing on the compressed hyperspectral image and crop it to obtain a hyperspectral patch image.

[0015] In one embodiment, the adaptive spectral feature weighting network in S2 includes a first feature extraction module and a fully connected layer. The first feature extraction module is used to extract hyperspectral feature vectors from the hyperspectral patch image. The fully connected layer outputs weights of the hyperspectral features based on the extracted hyperspectral feature vectors, and then combines these weights with the weights of the hyperspectral patch image x. i Multiplication yields a hyperspectral patch image with reduced hyperspectral features.

[0016] In one embodiment, the loss function of the adaptive spectral feature weighting network in S2 is:

[0017]

[0018] In the formula, L1 is the loss of the adaptive spectral feature weighting network, ω F As the weight of hyperspectral features, y iLet ||ω be the classification label for the i-th hyperspectral patch image. F ||1 represents the weight ω of the hyperspectral features F The L1 norm, The weight ω of the hyperspectral features F The classification label y of the i-th hyperspectral patch image i The maximum correlation between them, where N is the number of training samples.

[0019] In one embodiment, the multi-scale spatial spectral fusion network in S2 includes a spectral feature extraction network, a spatial neighborhood feature extraction network, and a feature fusion module. The spectral feature extraction network is used to extract hyperspectral features from the hyperspectral patch image with reduced hyperspectral features to obtain a hyperspectral feature vector of a preset size. The spatial neighborhood feature extraction network is used to extract spatial neighborhood features from the hyperspectral patch image with reduced hyperspectral features to obtain a spatial neighborhood feature vector of a preset size. The feature fusion module is used to fuse the hyperspectral feature vector and the spatial neighborhood feature vector, and combine them with the classification label to predict the classification result of the hyperspectral patch image.

[0020] In one embodiment, the spectral feature extraction network in the multi-scale spatial spectral fusion network in S2 includes a second feature extraction module and a third feature extraction module. The second feature extraction module is used to extract spectral features from the hyperspectral patch image with reduced hyperspectral features by performing conventional sampling, and obtain a conventionally sampled hyperspectral feature vector of a preset size. The third feature extraction module is used to extract spectral features from the hyperspectral patch image with reduced hyperspectral features by performing downsampling, and obtain a downsampled hyperspectral feature vector of a preset size.

[0021] In one embodiment, the spatial neighborhood feature extraction network in the multi-scale spatial spectral fusion network in S2 includes a fourth feature extraction module and a fifth feature extraction module. The fourth feature extraction module is used to extract spatial neighborhood features from the hyperspectral patch image with reduced hyperspectral features by performing conventional sampling, and obtain a conventionally sampled spatial neighborhood feature vector of a preset size. The fifth feature extraction module is used to extract spatial neighborhood features from the hyperspectral patch image with reduced hyperspectral features by performing downsampling, and obtain a downsampled spatial neighborhood feature vector of a preset size.

[0022] In one embodiment, the loss function of the multi-scale spatial spectral fusion network in S2 is:

[0023]

[0024] In the formula, L2 is the loss of the multi-scale spatial spectral fusion network, N is the number of training samples, and y i Let i be the classification label for the i-th hyperspectral patch image. Let be the classification label for the predicted i-th hyperspectral patch image.

[0025] In one embodiment, the reconstruction network in S2 includes a decoder module and a 3D-CNN module. The decoder module is used to process the conventionally sampled hyperspectral feature vector, the downsampled hyperspectral feature vector, the conventionally sampled spatial neighborhood feature vector, and the downsampled spatial neighborhood feature vector to obtain a feature map of a preset size. The 3D-CNN module is used to generate a reconstructed hyperspectral image from the feature map.

[0026] In one embodiment, the loss function for reconstructing the network in S2 is:

[0027]

[0028] Where L3 is the loss of the reconstructed network, x i Given the i-th hyperspectral patch image as input, Let N be the reconstructed i-th hyperspectral patch image, and N be the number of training samples.

[0029] The aforementioned pathological hyperspectral image classification method based on multi-scale spatial-spectral features first acquires hyperspectral images of pathological sections and their corresponding pixel-level labels. The hyperspectral images are preprocessed, and combined with the pixel-level labels to obtain hyperspectral patch images and corresponding classification labels. Next, a multi-scale spatial-spectral feature classification network is constructed with a pre-defined loss function. Then, a pre-defined training set is input into the constructed multi-scale spatial-spectral feature classification network for training. The loss value of the multi-scale spatial-spectral feature classification network is calculated based on its loss function, and backpropagation is used to update the network parameters, resulting in an updated multi-scale spatial-spectral feature classification network. Finally, the hyperspectral patch images are input into the updated multi-scale spatial-spectral feature classification network, and combined with the classification labels, the classification results of the hyperspectral patch images are output. This classification method fully utilizes spatial neighborhood features and hyperspectral features, and alleviates overfitting problems when using small datasets by removing irrelevant input features and adding regularization. Attached Figure Description

[0030] Figure 1 This is a flowchart of a pathological hyperspectral image classification method based on multi-scale spatial spectral features in one embodiment of the present invention;

[0031] Figure 2 This is a framework diagram of a multi-scale spatial spectrum feature classification network in one embodiment of the present invention;

[0032] Figure 3 This is a framework diagram of a multi-scale spatial-spectral feature fusion network in one embodiment of the present invention. Detailed Implementation

[0033] To enable those skilled in the art to better understand the technical solution of the present invention, the present invention will be further described in detail below with reference to the accompanying drawings.

[0034] A pathological hyperspectral image classification method based on multi-scale spatial spectral features includes the following steps:

[0035] S1. Obtain hyperspectral images of pathological sections and corresponding pixel-level labels. Preprocess the hyperspectral images to obtain hyperspectral patch images. Obtain classification labels for the hyperspectral patch images based on the pixel-level labels and the hyperspectral patch images.

[0036] S2. Construct a multi-scale spatial spectral feature classification network. The network includes an adaptive spectral feature weighting network, a multi-scale spatial spectral fusion network, and a reconstruction network connected in sequence. Preset the loss functions of the adaptive spectral feature weighting network, the multi-scale spatial spectral fusion network, and the reconstruction network respectively, and add the loss functions to obtain the total loss function of the multi-scale spatial spectral feature classification network.

[0037] S3. Input the preset training set into the constructed multi-scale spatial spectrum feature classification network for training, calculate the total loss of the multi-scale spatial spectrum feature classification network according to the total loss function of the multi-scale spatial spectrum feature classification network, and backpropagate to update the network parameters of the multi-scale spatial spectrum feature classification network to obtain the updated multi-scale spatial spectrum feature classification network.

[0038] S4. Input the hyperspectral patch image into the updated multi-scale spatial spectral feature classification network, combine it with the classification label of the hyperspectral patch image, and output the classification result of the hyperspectral patch image.

[0039] See Figure 1 and Figure 2 , Figure 1 This is a flowchart of a pathological hyperspectral image classification method based on multi-scale spatial spectral features in one embodiment of the present invention. Figure 2 This is a framework diagram of a multi-scale spatial spectral feature classification network in one embodiment of the present invention.

[0040] A pathological hyperspectral image classification method based on multi-scale spatial-spectral features first captures pathological sections through a microscope using a 20x objective lens. Then, it filters along the spectral dimension using an acousto-optic tunable filter (AOTF). The hyperspectral image corresponding to each band is imaged using a grayscale CMOS detector, thus obtaining the hyperspectral image of the pathological section. The hyperspectral image is annotated at the pixel level, with normal pixel regions labeled "0" and abnormal pixel regions labeled "1," resulting in a pixel-level label y, where y = 0 or 1, where y = 0 indicates a normal pixel region and y = 1 indicates an abnormal pixel region. The hyperspectral image is then processed to obtain a hyperspectral patch image x. i Take the hyperspectral patch image x iThe pixel-level label corresponding to the center pixel is used as the hyperspectral patch image x i Category tags y i , where y i =0 or 1, y i =0 indicates that the hyperspectral patch image x i Normal, y i =1 indicates that the hyperspectral patch image x i Anomalies were identified; then, a multi-scale spatial spectral feature classification network was constructed, consisting of an adaptive spectral feature weighting network, a multi-scale spatial spectral fusion network, and a reconstruction network connected sequentially. Loss functions for these three branches were designed, and the sum of the loss functions of the three branches was used as the total loss function of the multi-scale spatial spectral feature classification network. Next, the multi-scale spatial spectral feature classification network was trained and its loss value was calculated based on the obtained hyperspectral patch image, the corresponding classification label, and the total loss function of the multi-scale spatial spectral feature classification network. Backpropagation was then used to update the network parameters of the multi-scale spatial spectral feature classification network, resulting in the updated multi-scale spatial spectral feature classification network. Finally, the hyperspectral patch image was input into the updated multi-scale spatial spectral feature classification network, and the classification result of the hyperspectral patch image was output by combining the classification label of the hyperspectral patch image.

[0041] In one embodiment, step S1 involves preprocessing the hyperspectral image to obtain a hyperspectral patch image, specifically as follows:

[0042] S11. Compress the hyperspectral image to obtain a compressed hyperspectral image of a preset size;

[0043] S12. Perform Gaussian smoothing on the compressed hyperspectral image and crop it to obtain a hyperspectral patch image.

[0044] Specifically, each hyperspectral image obtained through a microscope and a grayscale CMOS detector has a shape of H×W×B=1280×1024×60, where H represents the image height, W represents the image width, and B represents the number of bands. To reduce computational costs, the hyperspectral images are compressed, specifically by adjusting the shape of each hyperspectral image from H×W×B=1280×1024×60 to H×W×B=320×256×60. Next, Gaussian smoothing is performed to avoid aliasing and ghosting. This involves dividing the spectrum of each input pixel of each hyperspectral image by the spectral geometric mean, and then by the spatial geometric mean to generate a new hyperspectral image. A low-pass Gaussian filter with a kernel size of 5×5 is then used to convolve the new hyperspectral image. Finally, the new hyperspectral image after Gaussian smoothing is cropped to obtain a hyperspectral patch image with a shape of H×W×B=9×9×60. Cropping the hyperspectral image to obtain a hyperspectral patch image allows the network to process less data at a time, resulting in faster processing speed.

[0045] In one embodiment, the adaptive spectral feature weighting network in S2 includes a first feature extraction module and a fully connected layer. The first feature extraction module is used to extract hyperspectral feature vectors from the hyperspectral patch image. The fully connected layer outputs weights of the hyperspectral features based on the extracted hyperspectral feature vectors, and then combines these weights with the weights of the hyperspectral patch image x. i Multiplication yields a hyperspectral patch image with reduced hyperspectral features.

[0046] Specifically, due to the large amount of redundant hyperspectral image information, the adaptive spectral feature weighting network adaptively identifies the importance of each band in the hyperspectral image to reduce the number of hyperspectral features and avoid network overfitting. The first feature extraction module extracts the hyperspectral feature vector of the hyperspectral patch image. The fully connected layer first randomly initializes the weights of the hyperspectral features corresponding to each band in the hyperspectral patch image based on the hyperspectral feature vector. Then, it multiplies the weights of the hyperspectral features with the weights of the hyperspectral features to obtain the hyperspectral patch image with reduced hyperspectral features: the weights ω of the hyperspectral features. F It can be represented by a 60×1 vector, or it can be represented as... Where 60 corresponds to each band, the weight ω of the hyperspectral features is... F With input hyperspectral patch image x i Multiplication yields a hyperspectral patch image with reduced hyperspectral features.

[0047] In one embodiment, the loss function of the adaptive spectral feature weighting network in S2 is:

[0048]

[0049] In the formula, L1 is the loss of the adaptive spectral feature weighting network, ω F As the weight of hyperspectral features, y i Let ||ω be the classification label for the i-th hyperspectral patch image. F ||1 represents the weight ω of the hyperspectral features F The corresponding L1 norm, HSIC(ω) F ,y i ) represents the weight ω of the hyperspectral features. F The classification label y of the i-th hyperspectral patch image i The maximum correlation between them, where N is the number of training samples.

[0050] Specifically, the weights of the hyperspectral features are expressed as follows: 60 corresponds to each band.

[0051] Introducing the L1 norm to optimize the weight ω of hyperspectral features FSpecifically, this is achieved by weighting the hyperspectral features ω using the L1 norm. F Regularization is performed to make the weights ω of the hyperspectral features... F As sparse as possible:

[0052]

[0053] In the formula, ||ω F ||1 represents the weight ω of the hyperspectral features F The L1 norm of the band, B represents the number of bands (here B = 60), k = 1, 2, ..., B.

[0054] Introducing the Hilbert-Schmidt Independence Criterion (HSIC) to maximize the weight ω of hyperspectral features F and category tags y i Correlation between them:

[0055]

[0056] In the formula, HSIC(ω) F ,y i ) represents the weight ω of the hyperspectral features. F and category tags y i The maximum correlation between them, MMD represents the maximum mean difference. The weight ω of the hyperspectral features F The classification label y of the i-th hyperspectral patch image i The joint probability distribution between them The weight ω of the hyperspectral features F The marginal probability distribution, The classification label y for the i-th hyperspectral patch image i The marginal probability distribution.

[0057] If we simply minimize the weight ω of the hyperspectral features F The L1 norm might lose meaningful hyperspectral features, therefore the Hilbert-Schmidt Independence Criterion (HSIC) is introduced to measure the weight ω of the hyperspectral features of the two variables. F The classification label y of the i-th hyperspectral patch image i The correlation between them, 0≤HSIC(ω) F ,y i )≤1, when ω F and y i When the features are independent, HSIC is 0. Therefore, the loss function of the adaptive spectral feature weighting network is designed as follows:

[0058]

[0059] In the formula, L1 is the loss of the adaptive spectral feature weighting network, ωF As the weight of hyperspectral features, y i Let y be the classification label corresponding to the i-th hyperspectral patch image. i =0 or 1, ||ω F ||1 represents the weight ω of the hyperspectral features F The L1 norm, HSIC(ω) F ,y i ) represents the weight ω of the hyperspectral features. F The classification label y of the i-th hyperspectral patch image i The maximum correlation between them.

[0060] By optimizing the loss function L1, the weights ω of the hyperspectral features are made more efficient. F It is both sparse and retains as much information as possible from the category tags. i Relevant weights.

[0061] In one embodiment, the multi-scale spatial spectral fusion network in S2 includes a spectral feature extraction network, a spatial neighborhood feature extraction network, and a feature fusion module. The spectral feature extraction network is used to extract hyperspectral features from the hyperspectral patch image with reduced hyperspectral features to obtain a hyperspectral feature vector of a preset size. The spatial neighborhood feature extraction network is used to extract spatial neighborhood features from the hyperspectral patch image with reduced hyperspectral features to obtain a spatial neighborhood feature vector of a preset size. The feature fusion module is used to fuse the hyperspectral feature vector and the spatial neighborhood feature vector, and combine them with the classification label to predict the classification result of the hyperspectral patch image.

[0062] Specifically, see Figure 3 , Figure 3 This is a framework diagram of a multi-scale spatial-spectral feature fusion network in one embodiment of the present invention.

[0063] Figure 3 In this context, a spectral feature extraction network is used to extract hyperspectral patch images with reduced hyperspectral features. Hyperspectral features are extracted to obtain hyperspectral feature vectors c1 and c2 of preset size. A spatial neighborhood feature extraction network is used to patch the hyperspectral image by reducing the hyperspectral features. Spatial neighborhood features are extracted to obtain spatial neighborhood feature vectors c3 and c4 of preset size. The feature fusion module fuses the hyperspectral feature vectors c1 and c2 with the spatial neighborhood feature vectors c3 and c4, and combines them with the classification label y. i Predict the corresponding hyperspectral patch image x i The label category to which the center pixel belongs in, This yields the classification results for the hyperspectral patch image. For example, If the center pixel of the predicted hyperspectral patch image is classified as normal, then the predicted hyperspectral patch image is classified as normal; otherwise, ... This indicates that the classification result of the center pixel of the predicted hyperspectral patch image is abnormal, thus concluding that the classification result of the predicted hyperspectral patch image is abnormal and should be taken seriously.

[0064] In one embodiment, the spectral feature extraction network in the multi-scale spatial spectral fusion network in S2 includes a second feature extraction module and a third feature extraction module. The second feature extraction module is used to extract spectral features from the hyperspectral patch image with reduced hyperspectral features by performing conventional sampling, and obtain a conventionally sampled hyperspectral feature vector of a preset size. The third feature extraction module is used to extract spectral features from the hyperspectral patch image with reduced hyperspectral features by performing downsampling, and obtain a downsampled hyperspectral feature vector of a preset size.

[0065] Specifically, see Figure 3 , Figure 3 This is a framework diagram of a multi-scale spatial-spectral feature fusion network in one embodiment of the present invention.

[0066] The second feature extraction module includes a first 3D convolutional layer, a first convolutional neural network (DB) block, a second 3D convolutional layer, a channel attention module (CAM), and a global average pooling layer. The first 3D convolutional layer consists of a 3D-CNN layer with a kernel size of 1×1×7 and a stride of (1,1,2), and 24 channels. The first DB block consists of three 3D-CNN layers, each with a kernel size of 1×1×7 and a stride of (1,1,1), padding of (0,0,3), and 12 channels. The second 3D convolutional layer consists of a 3D-CNN layer with a kernel size of 1×1×27 and a stride of (1,1,1), and 60 channels. An activation function and a batch normalization layer are attached to each 3D-CNN layer, using Mish as the activation function. The specific formula is as follows:

[0067] Mish(x) = x × tanh(ln(1+e)) x ))

[0068] In the formula, x is the input of the activation function, that is, the feature map of the output of the previous convolutional layer, and tanh(·) represents the hyperbolic tangent function.

[0069] The second feature extraction module is used to extract spectral features from the hyperspectral patch image with reduced hyperspectral features using conventional sampling, resulting in a conventionally sampled hyperspectral feature vector of a preset size. The specific process is as follows:

[0070] 1) Hyperspectral patch image with reduced features Input the first 3D convolutional layer and obtain a hyperspectral feature map a1 with a shape of 24×9×9×27 through conventional sampling processing;

[0071] 2) Input the hyperspectral feature map a1 with a shape of 24×9×9×27 into the first convolutional neural network DB block, and output the hyperspectral feature map a2 with a shape of 60×9×9×27 after feature extraction;

[0072] 3) Input the hyperspectral feature map a2 with a shape of 60×9×9×27 into the second three-dimensional convolutional layer, where the 27 spectral band information is captured by the 27 corresponding channels to generate a hyperspectral feature map a3 with a size of 60×9×9×1;

[0073] 4) The Channel Attention Module (CAM) is used to refine the spectral information, and the 60×9×9×1 feature map a3 is further adjusted;

[0074] 5) Use a global average pooling layer to shrink the adjusted hyperspectral feature map a3 to a 60×1 shape to generate a regular sampled hyperspectral feature vector c1.

[0075] The first feature extraction module in the adaptive spectral feature weighting network has the same network structure as the second feature extraction module in the multi-scale spatial spectral feature fusion network. The only difference is that the weight parameters of each 3D-CNN layer are different, resulting in different hyperspectral feature vectors.

[0076] The third feature extraction module includes a third 3D convolutional layer, a second convolutional neural network (DB) block, a fourth 3D convolutional layer, a channel attention module (CAM), and a global average pooling layer. The third 3D convolutional layer consists of a single 3D-CNN layer with a kernel size of 3×3×7 and a stride of (2,2,2), and 24 channels. The second DB block consists of three 3D-CNN layers, each with a kernel size of 1×1×7 and a stride of (1,1,1), padding of (0,0,3), and 12 channels. The fourth 3D convolutional layer consists of a single 3D-CNN layer with a kernel size of 1×1×27, a stride of (1,1,1), and 60 channels. Activation functions and batch normalization layers are applied to each 3D-CNN layer, using Mish as the activation function.

[0077] The third feature module is used to extract downsampled spectral features from the hyperspectral patch image with reduced hyperspectral features, resulting in a downsampled hyperspectral feature vector of a preset size. The specific process is as follows:

[0078] 1) Hyperspectral patch image with reduced features The input is processed by the third 3D convolutional layer, and a hyperspectral feature map a4 with a shape of 24×4×4×27 is obtained through downsampling.

[0079] 2) Input the hyperspectral feature map a4 with a shape of 24×4×4×27 into the second convolutional neural network DB block. After feature extraction, the output hyperspectral feature map a5 with a shape of 60×4×4×27 is generated.

[0080] 3) Input the hyperspectral feature map a5 with a shape of 60×4×4×27 into the fourth three-dimensional convolutional layer, where 27 spectral band information is captured by 27 corresponding channels, generating a hyperspectral feature map a6 with a size of 60×4×4×1;

[0081] 4) The Channel Attention Module (CAM) is used to refine the spectral information, and the 60×4×4×1 feature map a6 is further adjusted;

[0082] 5) Use a global average pooling layer to shrink the adjusted hyperspectral feature map a6 to a 60×1 shape to generate a downsampled hyperspectral feature vector c2.

[0083] In one embodiment, the spatial neighborhood feature extraction network in the multi-scale spatial spectral fusion network in S2 includes a fourth feature extraction module and a fifth feature extraction module. The fourth feature extraction module is used to extract spatial neighborhood features from the hyperspectral patch image with reduced hyperspectral features by performing conventional sampling, and obtain a conventionally sampled spatial neighborhood feature vector of a preset size. The fifth feature extraction module is used to extract spatial neighborhood features from the hyperspectral patch image with reduced hyperspectral features by performing downsampling, and obtain a downsampled spatial neighborhood feature vector of a preset size.

[0084] The fourth feature extraction module includes a fifth 3D convolutional layer, a third convolutional neural network (DB) block, a sixth 3D convolutional layer, a positional attention (PAM) module, and a global average pooling layer. The fifth 3D convolutional layer consists of a 3D-CNN layer with a kernel size of 1×1×60 and a stride of (1,1,1), and 24 channels. The third convolutional neural network (DB) block consists of three 3D-CNN layers, each with a kernel size of 3×3×1 and a stride of (1,1,1), and 12 channels. Padding with (1,1,0) is added to maintain the shape of the feature map. The sixth 3D convolutional layer consists of a 3D-CNN layer with a kernel size of 3×3×1, a stride of (1,1,1), and 60 channels. Activation functions and batch normalization layers are attached to each 3D-CNN layer, using Mish as the activation function.

[0085] The fourth feature extraction module is used to extract spatial neighborhood features from the hyperspectral patch image with reduced hyperspectral features using conventional sampling, resulting in a conventionally sampled spatial neighborhood feature vector of a preset size. The specific process is as follows:

[0086] 1) Hyperspectral patch image with reduced features Input the fifth 3D convolutional layer and, through conventional sampling processing, obtain a spatial neighborhood feature map b1 with a shape of 24×9×9×1;

[0087] 2) Input the spatial neighborhood feature map b1 with a shape of 24×9×9×1 into the third convolutional neural network DB block. After feature extraction, the output spatial neighborhood feature map b2 with a shape of 60×9×9×1 is generated.

[0088] 3) Input the spatial neighborhood feature map b2 with a shape of 60×9×9×1 into the sixth three-dimensional convolutional layer to generate a spatial neighborhood feature map b3 with a size of 60×9×9×1;

[0089] 4) The location attention module (CAM) is used to refine the channel information, and the 60×9×9×1 spatial neighborhood feature map b3 is further adjusted;

[0090] 5) Use a global average pooling layer to shrink the adjusted spatial neighborhood feature map b3 to a 60×1 shape to generate a regular sampled spatial neighborhood feature vector c3.

[0091] The fifth feature extraction module includes a seventh 3D convolutional layer, a fourth convolutional neural network (DB) block, an eighth 3D convolutional layer, a positional attention (PAM) module, and a global average pooling layer. The seventh 3D convolutional layer consists of a 3D-CNN layer with a kernel size of 3×3×60 and a stride of (2,2,1), and 24 channels. The fourth DB block consists of three 3D-CNN layers, each with a kernel size of 3×3×1 and a stride of (1,1,1), and 12 channels. Padding (1,1,0) is added to maintain the feature map shape. The eighth 3D convolutional layer (3D-CNN) has a kernel size of 3×3×1, a stride of (1,1,1), and 60 channels. Activation functions and batch normalization layers are attached to each 3D-CNN layer, using Mish as the activation function.

[0092] The fifth feature extraction module is used to extract downsampled spatial neighborhood features from the hyperspectral patch image with reduced hyperspectral features, resulting in a downsampled spatial neighborhood feature vector of a preset size. The specific process is as follows:

[0093] 1) Hyperspectral patch image with reduced features Input the seventh 3D convolutional layer and, through downsampling, obtain a spatial neighborhood feature map b4 with a shape of 24×4×4×1;

[0094] 2) Input the spatial neighborhood feature map b4 with a shape of 24×4×4×1 into the fourth convolutional neural network DB block. After feature extraction, the output spatial neighborhood feature map b5 with a shape of 60×4×4×1 is generated.

[0095] 3) Input the spatial neighborhood feature map b5 with a shape of 60×4×4×1 into the eighth three-dimensional convolutional layer to generate a spatial neighborhood feature map b6 with a size of 60×4×4×1;

[0096] 4) The location attention module (CAM) is used to refine the channel information, and the 60×4×4×1 spatial neighborhood feature map b6 is further adjusted;

[0097] 5) Use a global average pooling layer to shrink the adjusted spatial neighborhood feature map b6 to a 60×1 shape to generate a downsampled spatial neighborhood feature vector c4.

[0098] In one embodiment, the loss function of the multi-scale spatial spectral fusion network in S2 is:

[0099]

[0100] In the formula, L2 is the loss of the multi-scale spatial spectral fusion network, N is the number of training samples, and y i Let i be the classification label for the i-th hyperspectral patch image. Let be the classification label for the predicted i-th hyperspectral patch image.

[0101] Specifically, using binary cross-entropy loss as the loss function of the multi-scale spatial spectral fusion network can constrain the network to extract effective features and output correct classification results.

[0102] In one embodiment, the reconstruction network in S2 includes a decoder module and a 3D-CNN module. The decoder module is used to process the conventionally sampled hyperspectral feature vector, the downsampled hyperspectral feature vector, the conventionally sampled spatial neighborhood feature vector, and the downsampled spatial neighborhood feature vector to obtain a feature map of a preset size. The 3D-CNN module is used to generate a reconstructed hyperspectral image from the feature map.

[0103] Specifically, a reconstruction network is designed to provide additional constraints to the multi-scale spatial spectral fusion network. Given the feature vector extracted by the multi-scale spatial spectral fusion network, the hyperspectral patch image x is reconstructed by the reconstruction network. i The reconstructed hyperspectral patch image was obtained.

[0104] The reconstruction network consists of a decoder module and a 3D-CNN module, with the following specific settings:

[0105] The decoder module consists of three 3D-CNN layers and a DB module. The kernel size and stride of the first 3D-CNN layer are set to 4×4×21 and (1,1,1), respectively, with 24 channels. The kernel size and stride of the second and third 3D-CNN layers are set to 3×3×21 and (1,1,1), respectively, with 12 and 6 channels. The Mish activation function and batch normalization layers are appended to each 3D-CNN layer. The DB module consists of three 3D-CNN layers, each with a kernel size of 1×1×7, a stride of (1,1,1), padding size of (0,0,3), and 6 output channels.

[0106] The 3D-CNN module includes a 3D-CNN layer with a kernel size of 1×1×1, a stride of (1,1,1), and 1 channel. It generates a reconstructed hyperspectral patch image using the Sigmoid function.

[0107] The reconstruction network processes the conventionally sampled hyperspectral feature vector c1, the downsampled hyperspectral feature vector c2, the conventionally sampled spatial neighborhood feature vector c3, and the downsampled spatial neighborhood feature vector c4 to obtain the reconstructed hyperspectral patch image. The reconstruction process includes:

[0108] 1) Concatenate the feature vectors c1, c2, c3, and c4 into a feature map of shape 60×2×2×1;

[0109] 2) After processing the feature map with a shape of 60×2×2×1 through three 3D-CNN layers, a feature map with a shape of 6×9×9×60 is generated;

[0110] 3) Input the feature map with a shape of 6×9×9×60 into the DB module for processing, and generate a feature map with a shape of 24×9×9×60;

[0111] 4) Input the feature map with a shape of 24×9×9×60 into the 3D-CNN module to generate the reconstructed hyperspectral patch image.

[0112] In one embodiment, the objective function for reconstructing the network in S2 is:

[0113]

[0114] Where L3 is the loss of the reconstructed network, x i Given the i-th hyperspectral patch image as input, Let N be the reconstructed i-th hyperspectral patch image, and N be the number of training samples.

[0115] Using mean squared error as the objective function for the reconstruction task can make the reconstructed hyperspectral patch image as similar as possible to the input hyperspectral patch image.

[0116] The total loss L of the multi-scale spatial spectral feature classification network is obtained by summing the loss L1 of the adaptive spectral feature weighting network, the loss L2 of the multi-scale spatial spectral fusion network, and the loss L3 of the reconstruction network, specifically as follows:

[0117] L = L1 + L2 + L3

[0118] In the formula, L is the total loss of the multi-scale spatial spectral feature classification network, L1 is the loss of the adaptive spectral feature weighting network, L2 is the loss of the multi-scale spatial spectral fusion network, and L3 is the loss of the reconstruction network.

[0119] To minimize the total loss of the multi-scale spatial spectral feature classification network by optimizing each branch network, it is necessary to adjust the network parameters of each branch network during the optimization process.

[0120] Furthermore, the classification performance of the pathological hyperspectral image classification method based on multi-scale spatial spectral features was experimentally verified.

[0121] Hyperspectral images of pathological slides and their corresponding pixel-level labels were acquired. The hyperspectral images were then compressed from a size of 1280×1024×60 to 320×256×60, resulting in the compressed hyperspectral image. Two percent of this compressed hyperspectral image was used as the training dataset to train a multi-scale spatial spectral feature classification network. The remaining images in the compressed hyperspectral image were then subjected to Gaussian smoothing and cropping to obtain the hyperspectral patch image x. i x i ∈R 9×9×60 The hyperspectral image data used as input to the multi-scale spatial spectral feature classification network was tested with only 2% of the total hyperspectral image data being trained. Two metrics were used to validate the classification performance of different methods: AUC and F1. ROC is a metric for evaluating classification performance, where AUC is the area under the ROC curve and the coordinate axes, measuring classification performance under different threshold settings; a higher AUC indicates better network classification performance. F1 (F1 score) is the average of recall and precision; a higher F1 score indicates better model performance. Existing classification methods include: DBDA (Optimal Spatial Spectral Approach), SSRN (Spatial Spectral Based Algorithm), and NaiveBayes (Spectral Based Method, also known as Naive Bayes Classifier).

[0122] Table 1 compares the classification performance of the classification method in this application with existing classification methods when using the same 2% training samples for image classification.

[0123] To ensure unbiased input data, the classification method in this application uses the same data preprocessing procedure as other classification methods. The dataset used comes from the multidimensional Choledoch database, and the entire dataset includes 689 hyperspectral images with partial cancer regions. We selected four representative scenarios for experiments. Scenario 1 is sample “032236c20x-roi3-L”, where 032236c represents the tissue specimen number, 20x is the objective magnification, roi3 is the field of view, and L indicates a sample with partial cancer regions. Scenario 2 is sample “034080-3-20x-roi3-L”; Scenario 3 is sample “041602-20x-roi1-L”; and Scenario 4 is sample “041602-20x-roi2-L”.

[0124] Table 1

[0125]

[0126] As shown in Table 1, the classification method in this application achieves an average AUC of 95.2% and an average F1 score of 89.3%. The classification method in this application outperforms DBDA by 2.4% and 0.8% in terms of average AUC and average F1 score, respectively. Compared to DBDA in four individual scenarios, the classification method in this application is superior to or comparable to DBDA. For example, the classification method in this application improves the AUC by 3.4% in scenario 2. Compared to SSRN, the classification method in this application has an average AUC and average F1 score that are 4.8% and 7.9% higher, respectively. In particular, the F1 score in scenario 3 increased by 10.3%. Furthermore, the classification method in this application significantly outperforms all three existing classification methods. Compared to NaiveBayes, the classification method in this application improves the average AUC and F1 score by 20.1% and 43%, respectively. The results demonstrate that the method proposed in this invention achieves better cancer detection accuracy.

[0127] The aforementioned pathological hyperspectral image classification method based on multi-scale spatial-spectral features first acquires hyperspectral images of pathological sections and their corresponding pixel-level labels. The hyperspectral images are preprocessed, and combined with the pixel-level labels to obtain hyperspectral patch images and corresponding classification labels. Then, a multi-scale spatial-spectral feature classification network is constructed with a pre-defined loss function. Next, a pre-defined training set is input into the constructed multi-scale spatial-spectral feature classification network for training. The loss value of the multi-scale spatial-spectral feature classification network is calculated based on its loss function, and backpropagation is used to update the network parameters, resulting in an updated multi-scale spatial-spectral feature classification network. Finally, the hyperspectral patch images are input into the multi-scale spatial-spectral feature classification network, and combined with the classification labels, the classification results of the hyperspectral patch images are output. This classification method fully utilizes spatial neighborhood features and spectral features, and alleviates overfitting problems when using small datasets by removing irrelevant input features and adding regularization.

[0128] The above provides a detailed description of a pathological hyperspectral image classification method based on multi-scale spatial spectral features provided by this invention. Specific examples have been used to illustrate the principles and implementation methods of this invention. The descriptions of the embodiments above are merely for the purpose of helping to understand the core ideas of this invention. It should be noted that those skilled in the art can make various improvements and modifications to this invention without departing from its principles, and these improvements and modifications also fall within the protection scope of the claims of this invention.

Claims

1. A pathological hyperspectral image classification method based on multi-scale spatial-spectral features, characterized in that, The method includes: S1. Obtain hyperspectral images of pathological slides and corresponding pixel-level labels, preprocess the hyperspectral images to obtain hyperspectral patch images, and obtain classification labels for the hyperspectral patch images based on the pixel-level labels and the hyperspectral patch images. S2. Construct a multi-scale spatial spectral feature classification network, which includes an adaptive spectral feature weighting network, a multi-scale spatial spectral fusion network and a reconstruction network connected in sequence. Preset loss functions for the adaptive spectral feature weighting network, the multi-scale spatial spectral fusion network and the reconstruction network respectively, and add the loss functions to obtain the total loss function of the multi-scale spatial spectral feature classification network. S3. Input the preset training set into the constructed multi-scale spatial spectrum feature classification network for training, calculate the total loss of the multi-scale spatial spectrum feature classification network according to the total loss function of the multi-scale spatial spectrum feature classification network, and backpropagate to update the network parameters of the multi-scale spatial spectrum feature classification network to obtain the updated multi-scale spatial spectrum feature classification network. S4. Input the hyperspectral patch image into the updated multi-scale spatial spectral feature classification network, and combine it with the classification label of the hyperspectral patch image to output the classification result of the hyperspectral patch image; The adaptive spectral feature weighting network in S2 includes a first feature extraction module and a fully connected layer. The first feature extraction module is used to extract the hyperspectral feature vector of the hyperspectral patch image. The fully connected layer outputs the weights of the hyperspectral features based on the extracted hyperspectral feature vector, and then modifies the weights of the hyperspectral features with the weights of the hyperspectral patch image x. i Multiplication yields a hyperspectral patch image with reduced hyperspectral features. The multi-scale spatial spectral fusion network in S2 includes a spectral feature extraction network, a spatial neighborhood feature extraction network, and a feature fusion module. The spectral feature extraction network is used to extract hyperspectral features from the hyperspectral patch image with reduced hyperspectral features to obtain a hyperspectral feature vector of a preset size. The spatial neighborhood feature extraction network is used to extract spatial neighborhood features from the hyperspectral patch image with reduced hyperspectral features to obtain a spatial neighborhood feature vector of a preset size. The feature fusion module is used to fuse the hyperspectral feature vector and the spatial neighborhood feature vector, and combine them with the classification label to predict the classification result of the hyperspectral patch image. The reconstruction network in S2 includes a decoder module and a 3D-CNN module. The decoder module is used to process the conventionally sampled hyperspectral feature vector, the downsampled hyperspectral feature vector, the conventionally sampled spatial neighborhood feature vector, and the downsampled spatial neighborhood feature vector to obtain a feature map of a preset size. The 3D-CNN module is used to generate a reconstructed hyperspectral image from the feature map.

2. The pathological hyperspectral image classification method based on multi-scale spatial-spectral features as described in claim 1, characterized in that, In step S1, the hyperspectral image is preprocessed to obtain a hyperspectral patch image, specifically as follows: S11. Compress the hyperspectral image to obtain a compressed hyperspectral image of a preset size; S12. Perform Gaussian smoothing on the compressed hyperspectral image and crop it to obtain a hyperspectral patch image.

3. The pathological hyperspectral image classification method based on multi-scale spatial-spectral features as described in claim 1, characterized in that, The loss function of the adaptive spectral feature weighting network in S2 is: In the formula, L1 is the loss of the adaptive spectral feature weighting network, ω F As the weight of hyperspectral features, y i Let ||ω be the classification label for the i-th hyperspectral patch image. F ||1 represents the weight ω of the hyperspectral features F The L1 norm, HSIC(ω) Fi ,y i ) represents the weight ω of the hyperspectral features. F The classification label y of the i-th hyperspectral patch image i The maximum correlation between them, where N is the number of training samples.

4. The pathological hyperspectral image classification method based on multi-scale spatial-spectral features as described in claim 1, characterized in that, The spectral feature extraction network in the multi-scale spatial spectral fusion network of S2 includes a second feature extraction module and a third feature extraction module. The second feature extraction module is used to extract spectral features from the hyperspectral patch image with reduced hyperspectral features by performing conventional sampling to obtain a conventionally sampled hyperspectral feature vector of a preset size. The third feature extraction module is used to extract spectral features from the hyperspectral patch image with reduced hyperspectral features by performing downsampling to obtain a downsampled hyperspectral feature vector of a preset size.

5. The pathological hyperspectral image classification method based on multi-scale spatial-spectral features as described in claim 4, characterized in that, The spatial neighborhood feature extraction network in the multi-scale spatial spectral fusion network of S2 includes a fourth feature extraction module and a fifth feature extraction module. The fourth feature extraction module is used to extract spatial neighborhood features from the hyperspectral patch image with reduced hyperspectral features by performing conventional sampling, and obtain a conventionally sampled spatial neighborhood feature vector of a preset size. The fifth feature extraction module is used to extract spatial neighborhood features from the hyperspectral patch image with reduced hyperspectral features by performing downsampling, and obtain a downsampling spatial neighborhood feature vector of a preset size.

6. The pathological hyperspectral image classification method based on multi-scale spatial spectral features as described in claim 5, characterized in that, The loss function of the multi-scale spatial spectral fusion network in S2 is: In the formula, L2 is the loss of the multi-scale spatial spectral fusion network, N is the number of training samples, and y i Let i be the classification label for the i-th hyperspectral patch image. Let be the classification label for the predicted i-th hyperspectral patch image.

7. The pathological hyperspectral image classification method based on multi-scale spatial-spectral features as described in claim 6, characterized in that, The loss function for reconstructing the network in S2 is: Where L3 is the loss of the reconstructed network, x i Given the i-th hyperspectral patch image as input, Let N be the reconstructed i-th hyperspectral patch image, and N be the number of training samples.

Images