Pathological hyperspectral image detection method based on cross-scale spectral feature fusion network

Abstract

The application discloses a pathological hyperspectral image detection method based on a cross-scale space spectrum feature fusion network, which proposes a feature extraction network based on cross-scale space spectrum feature fusion. First, in the encoder stage, multi-scale information is extracted by using dilated convolution, so that the network can better obtain feature information of multiple receptive fields. Then, in the jump connection stage of the encoder and the decoder, cross-scale feature extraction is performed by using the deformation of the transformer, so that the global context information and layout information of the input image sample can be better preserved. Finally, in order to alleviate the difficulty of space spectrum feature extraction, a feature attention mechanism is proposed in the decoder part to process the output of different layers of the decoder, obtain richer cross-layer features, extract more important feature information between different layers, thereby improve the segmentation performance of the network, make the neural network better extract the space spectrum feature fusion information, and improve the detection precision of the pathological hyperspectral image.

Description

Technical Field

[0001] The present invention belongs to the technical field of image processing, and in particular relates to a pathological hyperspectral image detection method based on a cross-scale spatial-spectral feature fusion network. Background Art

[0002] Hyperspectral images generally refer to spectral images of the same scene across hundreds of consecutive wavelengths, typically with a spectral resolution on the order of 10^-2λ. Compared to the human eye, which can only see visible light between approximately 380 and 780 nm, hyperspectral images cover a much wider wavelength range, presenting "colors beyond colors" that are inaccessible to the human eye. These images are acquired using imaging spectrometers, or hyperspectral sensors. These images contain not only the image information of the target but also its spectral information, achieving a fusion of spectrum and image. Because different components absorb light differently, hyperspectral images can significantly reveal certain characteristics of a sample, such as size, shape, and defects, at specific wavelengths.

[0003] With advances in hyperspectral imaging technology, hyperspectral images have been widely used in fields such as agricultural monitoring, materials testing, and medical diagnosis. In the clinical diagnosis of cancer pathology images, microscopic analysis based on pathological tissue sections can be used to observe the development of cancerous tissue and thus determine changes and metastasis. Therefore, pathology images are considered the gold standard in clinical cancer diagnosis. Most early diagnoses require detailed observation and analysis of pathology images by professional pathologists. These results depend on the physician's experience and subjective analysis, are biased, and consume significant time and labor costs. In pathology diagnosis, hyperspectral imaging technology can provide information on the tissue composition and spatial structure of the imaged object, enabling rapid and reliable pathology diagnosis.

[0004] However, compared to traditional remote sensing hyperspectral imagery, pathology hyperspectral imagery presents several challenges: Existing hyperspectral data samples for pathology slides are relatively scarce, and patient ethics and privacy concerns hinder the direct sharing of patient medical hyperspectral images between hospitals and research institutions. Furthermore, standards for pathology slides vary across institutions, making hyperspectral image annotation difficult. Furthermore, the diverse cell types and morphologies within pathology slides require the extraction of rich feature information. Consequently, feature extraction from pathology hyperspectral images is extremely challenging.

[0005] Therefore, a method is needed to effectively extract pathological hyperspectral features, adopt appropriate technologies to solve the problem of inter-scale feature fusion, and use appropriate methods to fully fuse spatial spectral features, so as to improve the detection accuracy of pathological hyperspectral features. Summary of the Invention

[0006] In response to the above technical problems, the present invention provides a pathological hyperspectral image detection method based on a cross-scale spatial-spectral feature fusion network.

[0007] The technical scheme adopted by the present application to solve its technical problems is:

[0008] The pathological hyperspectral image detection method based on a cross-scale spectral feature fusion network comprises the following steps:

[0009] S100: Obtain a hyperspectral picture constituting dataset, pre-process the dataset, and obtain a pre-processed dataset;

[0010] S200: Construct a hyperspectral image detection model, including an encoder based on multi-scale feature extraction, a cross-scale feature extraction network based on a transformer, and a feature attention decoder;

[0011] S300: Input the pre-processed dataset into the encoder based on multi-scale feature extraction, extract image features, and obtain a multi-scale encoding feature map output;

[0012] S400: Input the multi-scale encoding features into the cross-scale feature extraction network based on the transformer, perform cross-scale feature extraction on the encoding features, and obtain a final cross-scale feature map;

[0013] S500: Input the cross-scale feature map into the feature attention decoder for feature attention mechanism decoding processing, and obtain a segmentation result;

[0014] S600: Train the hyperspectral image detection model using a cross-validation method combined with a pre-set loss function, obtain a trained detection model when a pre-set training end condition is reached, and complete real-time pathological hyperspectral image segmentation according to the trained detection model.

[0015] Preferably, S100 comprises:

[0016] S110: Perform a scaling operation on the dataset to scale it to a size that is one-fourth of the original size;

[0017] S120: Perform data enhancement processing operations using an albumentation library to obtain a pre-training dataset; the data enhancement processing operations include rotation, scaling, inversion, color transformation, and blurring;

[0018] S130: Divide the pre-training dataset according to a certain proportion to obtain a corresponding training set and test set, and crop the training set to a pre-set size.

[0019] Preferably, the encoder based on multi-scale feature extraction comprises a 3D adaptive pooling layer, a depth separable convolution, and a multi-scale convolution, and S300 comprises:

[0020] S310: input the preprocessed data set into an encoder based on multi-scale feature extraction, pool the images in the preprocessed data set using a 3D adaptive pooling layer to obtain an initial feature image;

[0021] S320: process the initial feature image using two deep separable convolutions to change the dimensions of the initial feature image;

[0022] S330: use multi-scale dilated convolution to extract features from the feature map with changed dimensions to obtain multi-scale encoding features of the first stage,

[0023] S340: input the multi-scale encoding features of the first stage into a 3D adaptive pooling layer, repeat the steps of S310 to S330 to obtain multi-scale encoding features of the second stage, use the output of the previous stage as the input of the next stage, and repeat this process four times to obtain four different size encoding feature maps.

[0024] Preferably, S310 is specifically:

[0025] The 3D adaptive pooling layer is a pooling operation that automatically adjusts the pooling region according to the size of the input feature map, so that each dimension of the output feature map can reach the preset size, and the initial feature image after pooling is obtained;

[0026] The two deep separable convolutions include deep convolution and pointwise convolution, and S320 is specifically: assuming that the dimensions of the initial feature image are HxWxC, the deep convolution applies a KxK convolution kernel to each channel, performs spatial convolution to produce HxWxC output, and then the pointwise convolution uses a 1x1xC convolution kernel to combine the output of the deep convolution into a new feature map, and converts the output to HxWxD, where D is the number of output channels;

[0027] S330 is specifically: applying three dilated convolutions with different dilation rates to extract features from the feature map with changed dimensions, extracting feature maps of different scales respectively, combining the feature maps of different scales through weighted average method to obtain a comprehensive multi-scale encoding feature map;

[0028] Preferably, the transformer-based cross-scale feature extraction network includes a multi-head attention module, a cross-scale feature extraction module, and a multi-layer perceptron module, and S400 includes:

[0029] S410: use a multi-head self-attention module to perform self-attention mechanism processing on multi-scale encoding feature maps other than the first encoding feature map;

[0030] S420: uniformize the different sizes of feature maps obtained in S410 by interpolation, and perform cross-scale feature extraction on the uniform size feature maps based on transformer cross attention to obtain cross-scale feature fusion output;

[0031] S430: perform layer-by-layer processing and transformation on the obtained cross-scale feature fusion output by using a multilayer perceptron, enhance the representation of local information, and obtain final cross-scale feature output.

[0032] Preferably, S410 specifically comprises: inputting the multi-scale encoded feature maps other than the first encoded feature map into different multi-head self-attention mechanisms respectively, performing feature segmentation processing, dividing them into several spaces according to the number of attention heads, performing parallel self-attention mechanism processing, and finally splicing and merging the outputs;

[0033] S420 specifically comprises: for each encoded feature map, adjusting other encoded feature maps to a uniform size by bilinear interpolation, bilinear interpolation calculates the value of a target pixel by using the values of four surrounding pixels; then performing cross-scale feature extraction on the feature maps output by S410 by using transformer-based cross attention, regarding the encoded feature map being processed as a value vector, regarding the other two encoded feature maps as a query vector and a key vector respectively, using three learnable projection matrices W q , W k and W v for mapping processing, then using the query vector Q and the key vector K for dot product operation, and obtaining attention weights by using a Relu function, automatically ensuring the sparsity of the attention weights; finally, multiplying the attention weights with the value vector V, and summing the results to obtain cross-scale feature fusion output corresponding to the number of encoded feature maps;

[0034] S430: input the cross-scale feature fusion output corresponding to the number of encoded feature maps into the corresponding multilayer perceptron module, perform layer-by-layer processing and transformation on the input features, and obtain final cross-scale feature output corresponding to the number of cross-scale feature fusion outputs.

[0035] Preferably, in S420, for each encoded feature map, adjusting other encoded feature maps to a uniform size by bilinear interpolation specifically comprises:

[0036] I'(x',y')=(1-Δx)(1-Δy)I(x1,y1)+Δx(1-Δy)I(x2,y1)+Δy(1-Δx)I(x1,y ( )+ΔxΔyI(x2,y2)

[0037] Wherein, (x', y') is the pixel coordinate in the target image, (x1, y1) and (x2, y2) are the coordinates of the four pixels adjacent to the target pixel in the input feature map, and Δx and Δy are the offset of the target coordinate relative to the input coordinate.

[0038] In S420, the currently processed encoded feature map is regarded as a value vector, and the other two encoded feature maps are regarded as a query vector and a key vector respectively, and the three learnable projection matrices W q , W k and W v are used for mapping processing, specifically:

[0039] Q = W q x q, K = W k x k, V = W v x v

[0040] Wherein, q, k, v respectively represent the value vector, the query vector and the key vector of the feature map before processing, and Q, K, V respectively represent the value vector, the query vector and the key vector of the feature map after mapping processing.

[0041] Preferably, S500 comprises:

[0042] S510: up-sampling the cross-scale feature map, and splicing the up-sampled cross-scale feature map with the corresponding size of the encoder feature map, and using interpolation to unify the size of all spliced feature maps to obtain a decoding feature map;

[0043] S520: splicing all the decoding feature maps, obtaining a pooling feature map through a pooling layer, and performing attention operation on the original spliced feature map to obtain a corresponding feature attention map;

[0044] S530: finally, the feature attention map is subjected to 3D convolution, sigmoid and spectral average pooling operation to obtain a segmentation result.

[0045] Preferably, S510 specifically comprises:

[0046] Receiving the last multi-scale encoded feature map from the encoder and the corresponding cross-scale feature output map, the decoder first up-samples the multi-scale encoded feature map to expand the spatial resolution of the feature map by one time, and then splices the up-sampled feature map of the decoder with the encoded feature map of the corresponding layer in the encoder, and repeats the above process until the size of the original spliced feature map restores to the size of the input image, and uses the bilinear interpolation method to transform the four decoding feature maps to a uniform size.

[0047] S520 is specifically: the decoding feature map after uniform size is spliced along the spectral direction to obtain a spliced feature map, which is sent into a pooling layer for a pooling operation to obtain a corresponding pooled feature map, and then the pooled feature map is sent into a linear layer respectively to obtain a corresponding Q decoder ,K decoder Therefore, the feature attention weight matrix can be calculated by the following formula:

[0048] W decoder =conv2D(Q decoder K decoder T )

[0049] Wherein, conv2D(·) represents a 2D convolution with a kernel of 3*3 and a padding of 1, which is used to merge the related information between adjacent pixels, and linearly operates the original spliced feature map to obtain a corresponding V decoder , thereby obtaining a corresponding feature attention map Out decoder , which is calculated by the following formula:

[0050]

[0051] Preferably, S600 comprises:

[0052] The data set is randomly and evenly divided into four non-intersecting subsets, then four training and verification are performed, each time using one subset as a verification set and the remaining three subsets as training sets, and recording the result of each verification; finally, the average value of the four verification results is calculated as the final evaluation index of the model, and the above steps are repeatedly repeated, and the loss function obtained from the training set is iterated until the loss is less than a certain threshold or a predetermined number of iterations is completed, to obtain the final model weight for recording, and the trained detection model is used to complete the segmentation of real-time pathological hyperspectral images.

[0053] The pathological hyperspectral image detection method based on the cross-scale space-spectrum feature fusion network, proposes a feature extraction network based on cross-scale space-spectrum feature fusion, first, in the encoder stage, multi-scale information is extracted by using dilated convolution; then, in the jump connection stage of the encoder and the decoder, cross-scale feature extraction is performed by using the deformation of the transformer; finally, in order to alleviate the difficulty of space-spectrum feature extraction, a feature attention mechanism is proposed in the decoder part, so that the neural network can better extract the space-spectrum feature fusion information and improve the detection accuracy of the pathological hyperspectral image. BRIEF DESCRIPTION OF DRAWINGS

[0054] Figure 1 The flowchart of the pathological hyperspectral image detection method based on the cross-scale space-spectrum feature fusion network in an embodiment of the present application;

[0055] Figure 2A schematic diagram of an encoder network based on multi-scale feature extraction in an embodiment of the present application;

[0056] Figure 3 A schematic diagram of a cross-scale feature extraction network based on transformer in an embodiment of the present application;

[0057] Figure 4 A schematic diagram of a feature attention decoder network in an embodiment of the present application. DETAILED DESCRIPTION

[0058] In order to enable those skilled in the art to better understand the technical solutions of the present application, the present application will be further described in detail below with reference to the accompanying drawings.

[0059] In one embodiment, as shown in Figure 1 The pathological hyperspectral image detection method based on the cross-scale spectral feature fusion network comprises the following steps:

[0060] S100: Obtain a hyperspectral image data set, pre-process the data set, and obtain a pre-processed data set;

[0061] S200: Construct a hyperspectral image detection model, including an encoder based on multi-scale feature extraction, a cross-scale feature extraction network based on transformer, and a feature attention decoder;

[0062] S300: Input the pre-processed data set into the encoder based on multi-scale feature extraction, extract image features, and obtain a multi-scale encoding feature map output;

[0063] S400: Input the multi-scale encoding features into the cross-scale feature extraction network based on transformer, perform cross-scale feature extraction on the encoding features, and obtain a final cross-scale feature map;

[0064] S500: Input the cross-scale feature map into the feature attention decoder for feature attention mechanism decoding processing, and obtain a segmentation result;

[0065] S600: Train the hyperspectral image detection model using a cross-validation method combined with a pre-set loss function, obtain a trained detection model when a pre-set training end condition is reached, and complete real-time pathological hyperspectral image segmentation according to the trained detection model.

[0066] The pathological hyperspectral image detection method based on the cross-scale space spectrum feature fusion network, proposes a feature extraction network based on cross-scale space spectrum feature fusion, first, in the encoder stage, multi-scale information is extracted by using dilated convolution; then, in the jump connection stage of the encoder and the decoder, cross-scale feature extraction is performed by using the deformation of the transformer; finally, in order to alleviate the difficulty of space spectrum feature extraction, a feature attention mechanism is proposed in the decoder part, so that the neural network can better extract the spectral feature fusion information and improve the detection accuracy of the pathological hyperspectral image.

[0067] In one embodiment, S100 includes:

[0068] S110: performing a scaling operation on the data set to scale it to a size of one quarter of the original size;

[0069] S120: performing a data enhancement processing operation on the data set using an albumentation library to obtain a pre-training data set; the data enhancement processing operation includes rotation, scaling, inversion, color transformation and blurring;

[0070] S130: dividing the pre-training data set according to a certain proportion to obtain a corresponding training set and test set, and cropping the training set to a preset size.

[0071] Specifically, the original size of the hyperspectral data is 1280x1024, and the scaling operation is performed to reduce the length and width to one quarter of the original size, becoming 320x256; then, the albumentations library is used to perform rotation, scaling, flipping, deformation, color transformation, blurring and other operations to realize data enhancement, and obtain a pre-training data set D all ; finally, the data set is evenly divided into four parts, and one of them is randomly selected as a validation set D val , and the remaining data is used as a training set D train , and the training set data is cropped to 192x192. The number of pictures in the training set is Nx75%, and the number of pictures in the validation set is Nx25%.

[0072] In one embodiment, the encoder based on multi-scale feature extraction includes a 3D adaptive pooling layer, a depth separable convolution and a multi-scale convolution, and S300 includes:

[0073] S310: inputting the preprocessed data set into the encoder based on multi-scale feature extraction, using the 3D adaptive pooling layer to perform pooling processing on the images in the preprocessed data set to obtain initial feature images;

[0074] S320: using two depth separable convolutions to process the initial feature images to change the dimensions of the initial feature images;

[0075] S330: performing feature extraction on the feature map with changed dimensions by using a multi-scale dilated convolution, to obtain multi-scale encoding features of a first stage,

[0076] S340: inputting the multi-scale encoding features of the first stage into a 3D adaptive pooling layer, repeating the steps of S310 to S330, to obtain multi-scale encoding features of a second stage, taking the output of the previous stage as the input of the next stage, and repeating the above steps four times to obtain four encoding feature maps with different sizes.

[0077] In one embodiment, S310 specifically comprises:

[0078] The 3D adaptive pooling layer is a pooling operation that automatically adjusts the pooling region according to the size of the input feature map, so that each dimension of the output feature map can reach the preset size, to obtain the initial feature map after pooling;

[0079] The two depth separable convolutions include a depth convolution and a point-wise convolution, and S320 specifically comprises: assuming that the dimensions of the initial feature map are HxWxC, the depth convolution applies a convolution kernel with a size of KxK to each channel, performs spatial convolution, and generates an output with dimensions of HxWxC, then the point-wise convolution uses a convolution kernel with a size of 1x1xC to combine the output of the depth convolution into a new feature map, and converts the output into HxWxD, where D is the number of output channels;

[0080] S330 specifically comprises: applying three dilated convolutions with different dilated rates to perform feature extraction on the feature map with changed dimensions, to extract feature maps of different scales respectively, combining the feature maps of different scales by a weighted average method, and obtaining a comprehensive multi-scale encoding feature map;

[0081] Specifically, as shown in Figure 2 the input whole hyperspectral image is input, and is defined as XER H×W×C where H, W and C represent the length, width and number of bands of the hyperspectral image respectively, the encoder based on multi-scale feature extraction mainly includes three parts: a 3D adaptive pooling layer, a depth separable convolution and a multi-scale convolution.

[0082] Firstly, the input hyperspectral image is processed by a 3D adaptive pooling layer. The 3D adaptive pooling layer is a kind of pooling operation, which automatically adjusts the pooling region according to the size of the input feature map to generate a fixed-size output feature map. Unlike the traditional fixed-size pooling window, the adaptive pooling adjusts the size and stride of the pooling window, so that the input feature map is divided into a target number of regions, and each region gets an output value through maximum pooling or average pooling, so that each dimension of the output feature map can reach the preset size, and thus the initial feature image after pooling is obtained.

[0083] Then two depthwise separable convolutions are used to extract features from the initial feature image. Depthwise separable convolution is a kind of efficient convolution operation with low parameter quantity and operation cost, which mainly includes two steps: depthwise convolution and pointwise convolution. Assuming that the dimension of the input feature map is HxWxC, the depthwise convolution applies a KxK convolution kernel to each channel for spatial convolution, producing an HxWxC output. Then, the pointwise convolution uses a 1x1xC convolution kernel to combine the output of the depthwise convolution into a new feature map, and converts the output to HxWxD, where D is the number of output channels.

[0084] Finally, multi-scale convolution is used for feature extraction to obtain the final multi-scale encoding feature. The multi-scale convolution mechanism based on dilated convolution is introduced to effectively extract features. Dilated convolution expands the size of the convolution kernel by using a dilation factor, thereby increasing the receptive field without sacrificing resolution. The operation of dilated convolution realizes wider feature coverage by introducing intervals in the convolution process. For dilated convolution with a dilation factor of S, the receptive field calculation formula is: K s =Sx(K0-1)+1, where K s is the convolution kernel size designed by dilating the ordinary convolution kernel, and K0 is the ordinary convolution kernel size. In order to avoid the "lattice effect", three different dilution rates (1, 2 and 3) are selected to ensure that there is no common divisor greater than 1 between these dilution rates. The specific process is as follows: first, apply dilated convolutions with three different dilution rates (1, 2 and 3) to extract feature maps of different scales. Although these convolutions share the same convolution kernel weights, they can capture features of different scales due to different dilution rates. Then, these feature maps of different scales are combined by weighted average method to obtain a comprehensive multi-scale encoding feature map.

[0085] This process is repeated four times to obtain four feature maps of different sizes, effectively capturing and utilizing multi-scale features to provide more rich feature information for subsequent steps.

[0086] In one embodiment, the transformer-based cross-scale feature extraction network comprises a multi-head attention module, a cross-scale feature extraction module, and a multi-layer perceptron module, S400 comprises:

[0087] S410: using a multi-head self-attention module to perform self-attention mechanism processing on the multi-scale encoded feature maps other than the first encoded feature map;

[0088] S420: using interpolation to unify the sizes of the feature maps obtained in S410, using transformer-based cross-attention to perform cross-scale feature extraction on the unified size feature maps, and obtaining cross-scale feature fusion output;

[0089] S430: using a multi-layer perceptron to perform layer-by-layer processing and transformation on the obtained cross-scale feature fusion output, enhancing the representation of local information, and obtaining final cross-scale feature output.

[0090] In one embodiment, S410 specifically comprises: inputting the multi-scale encoded feature maps other than the first encoded feature map into different multi-head self-attention mechanisms respectively, performing feature segmentation processing, dividing them into several spaces according to the number of attention heads, performing parallel self-attention mechanism processing, and finally splicing and merging the outputs;

[0091] S420 specifically comprises: for each encoded feature map, adjusting the other encoded feature maps to a unified size using bilinear interpolation, bilinear interpolation calculates the value of the target pixel by using the values of the surrounding four pixels; then using transformer-based cross-attention to perform cross-scale feature extraction on the feature maps output by S410, regarding the currently processed encoded feature map as a value vector, and regarding the other two encoded feature maps as a query vector and a key vector respectively, using three learnable projection matrices W q , W k and W v for mapping processing, then using the query vector Q and the key vector K for dot product operation, and obtaining attention weights through a Relu function, automatically ensuring the sparsity of the attention weights; finally, multiplying the attention weights by the value vector V, and summing the results to obtain cross-scale feature fusion output corresponding to the number of encoded feature maps;

[0092] S430: inputting the cross-scale feature fusion output corresponding to the number of encoded feature maps into the corresponding multi-layer perceptron module, performing layer-by-layer processing and transformation on the input features, and obtaining final cross-scale feature output corresponding to the number of cross-scale feature fusion outputs.

[0093] In one embodiment, for each encoded feature map in S420, the other encoded feature maps are adjusted to a uniform size using bilinear interpolation, which calculates the value of a target pixel by using the values of the surrounding four pixels, specifically:

[0094] I'(x',y')=(1-Δx)(1-Δy)I(x1,y1)+Δx(1-Δy)I(x2,y1)+Δy(1-Δx)I(x1,y2)+ΔxΔyI(x2,y2)

[0095] where (x',y') is the pixel coordinate in the target map, (x1,y1) and (x2,y2) are the coordinates of the four pixels adjacent to the target pixel in the input feature map, and Δx and Δy are the offsets of the target coordinate relative to the input coordinate.

[0096] In S420, the currently processed encoded feature map is regarded as a value vector, and the other two encoded feature maps are regarded as a query vector and a key vector respectively, and mapping processing is performed using three learnable projection matrices W q , W k and W v respectively, specifically:

[0097] Q=W q x q, K=W k x k, V=W v x v

[0098] where q, k, v represent the value vector, the query vector and the key vector of the feature map before processing respectively, and Q, K, V represent the value vector, the query vector and the key vector of the feature map after mapping processing respectively.

[0099] Specifically, as shown in Figure 3 , S300 obtains four multi-scale encoded feature maps of different sizes, and processes the last three layers of feature maps using a transformer-based cross-scale feature extraction network, which mainly includes three steps: multi-head attention module, cross-scale feature extraction and multi-layer perceptron module. Multi-head attention mechanism and multi-layer perceptron module are key technologies in the Transformer model.

[0100] Firstly, different encoded feature maps are input into different multi-head self-attention mechanisms for feature segmentation processing, which is divided into several spaces according to the number of attention heads, and parallel self-attention mechanism processing is performed, and finally the outputs are spliced and combined to capture the relationship and difference in the sequence more carefully;

[0101] Then, for each feature map, the following operations are performed: for each encoded feature map, the other feature maps are adjusted to a uniform size using bilinear interpolation, which calculates the value of a target pixel by using the values of the surrounding four pixels, with the formula:

[0102] I'(x', y') = (1 - Δx)(1 - Δy)I(x1, y1) + Δx(1 - Δy)I(x2, y1) + Δy(1 - Δx)I(x1, y2)

[0103] + ΔxΔyI(x2, y2)

[0104] where (x', y') is the pixel coordinate in the target map, (x1, y1) and (x2, y2) are the coordinates of the four neighboring pixels in the input feature map, and Δx and Δy are the offsets of the target coordinates relative to the input coordinates. Then, cross-attention based on the transformer is used for cross-scale feature extraction. For which encoded feature map is processed, which feature map is regarded as the value vector, and the other two are regarded as the query vector and key vector, respectively. Then, three learnable projection matrices W q , W k , and W v are used for mapping, i.e.: Q = W q ×q, K = W k ×k, V = W v ×v. Then, the dot product operation is performed using the query vector Q and the key vector K, and the attention weight is obtained through the Relu function. Using the Relu function can aggregate the most useful attention features and automatically ensure the sparsity of the attention weight. Finally, the attention weight is multiplied by the value vector V, and the result is summed to obtain the cross-scale feature fusion output.

[0105] Finally, the cross-scale feature fusion output is input into the multi-layer perceptron module to obtain the final cross-scale feature output. The multi-layer perceptron module can process and transform the input features layer by layer, extracting more complex and high-order feature representations. This module plays an important role in feature extraction and classification tasks, effectively improving the performance and accuracy of the model.

[0106] In one embodiment, S500 includes:

[0107] S510: upsample the cross-scale feature map and concatenate it with the corresponding size of the encoder feature map, and use interpolation to unify the size of all concatenated feature maps to obtain the decoding feature map;

[0108] S520: concatenate all decoding feature maps, pass through a pooling layer to obtain a pooling feature map, and perform attention operation on the original concatenated feature map to obtain the corresponding feature attention map;

[0109] S530: Finally, the feature attention map is passed through 3D convolution, sigmoid and spectral average pooling operations to obtain the segmentation result.

[0110] In one embodiment, S510 is specifically:

[0111] The last multi-scale encoded feature map and the corresponding cross-scale feature output map from the encoder are received, the decoder first upsamples the multi-scale encoded feature map to expand the spatial resolution of the feature map by one time, then splices the decoder- upsampl ed feature map with the encoded feature map of the corresponding layer in the encoder, and repeats the above process until the size of the original spliced feature map recovers to the size of the input image, and four decoded feature maps are transformed to a uniform size using a bilinear interpolation method.

[0112] S520 is specifically: the decoded feature map of the uniform size is spliced along the spectral direction to obtain a spliced feature map, which is sent into a pooling layer for pooling operation to obtain a corresponding pooled feature map, which is then sent into a linear layer to obtain a corresponding Q decoder ,K decoder Therefore, the feature attention weight matrix can be calculated by the following formula:

[0113] W decoder =conv2D(Q decoder K decoder T (

[0114] where conv2D(·) represents a 2D convolution with a kernel of 3x3 and padding set to 1, which is used to merge the related information between adjacent pixels, and linearly operates the original spliced feature map to obtain a corresponding V decoder , thereby obtaining a corresponding feature attention map Out decoder , which is calculated by the following formula:

[0115]

[0116] Specifically, the main task of the decoder is to gradually recover the spatial resolution of the feature map to generate an output image with the same size as the input image, while using the features of the encoder for detail recovery.

[0117] First, the decoder receives the last multi-scale encoded feature map and the corresponding cross-scale feature output map from the encoder, the decoder first upsamples the multi-scale encoded feature map to expand the spatial resolution of the feature map by one time, then splices the decoder- upsampl ed feature map with the feature map of the corresponding layer in the encoder, and repeats the above process until the size of the feature map recovers to the size of the input image, and four decoded feature maps are transformed to a uniform size using a bilinear interpolation method.

[0118] As shown in Figure 4 , the uniform-sized feature map is then spliced along the spectral direction to obtain a spliced feature map, which is sent to a pooling layer for a pooling operation to obtain a corresponding pooled feature map, which is then sent to a linear layer to obtain a corresponding Q decoder ,K decoder Therefore, the feature attention weight matrix can be calculated as follows: W decoder =conv2D(Q decoder K decoder T ), where conv2D(·) represents a 2D convolution with a kernel of 3x3 and padding set to 1. This operation is used to merge the relevant information between adjacent pixels. The original spliced feature map is subjected to a linear operation to obtain a corresponding V decoder , thereby obtaining a corresponding feature attention map Out decoder , which can be calculated as follows: Finally, the feature attention map is subjected to 3D convolution, sigmoid, and spectral average pooling operations to obtain a final 2D output result.

[0119] In one embodiment, S600 includes:

[0120] The data set is randomly and evenly divided into four mutually disjoint subsets, then four training and verification are performed, each time using one subset as the verification set and the remaining three subsets as the training set, recording the result of each verification; finally, the average value of the four verification results is calculated as the final evaluation index of the model. The above steps are repeatedly repeated, and the loss function obtained from the training set is iterated, until the loss is less than a certain threshold or a predetermined number of iterations is completed, to obtain the final model weight for recording. The trained detection model is used to complete the segmentation of real-time pathological hyperspectral images.

[0121] Specifically, in order to reduce overfitting and improve the generalization ability of the model, four-fold cross-validation is used to process the data set, and the specific steps are as follows: first, the data set is randomly and evenly divided into four mutually disjoint subsets, then four training and verification are performed, each time using one subset as the verification set and the remaining three subsets as the training set, recording the result of each verification; finally, the average value of the four verification results is calculated as the final evaluation index of the model.

[0122] The above steps are repeatedly repeated, and the loss function obtained from the training set is iterated, until the loss is less than a certain threshold or a predetermined number of iterations is completed, to obtain the final model weight for recording.

[0123] During the testing process, the final model weights are loaded; then, all data, whether training set or validation set, are used for testing. All data is fed into the final trained model, and the predicted segmentation results are compared with the true labels to obtain the final segmentation accuracy.

[0124] The above-mentioned pathological hyperspectral image detection method based on the cross-scale spatial-spectral feature fusion network has the following beneficial effects compared with the existing technology:

[0125] 1. The pathological hyperspectral image detection method based on a cross-scale spatial-spectral feature fusion network uses dilated convolution in the encoder part to extract multi-scale spatial-spectral features, so that the network can better obtain feature information of multiple receptive fields;

[0126] 2. In the jump connection stage between the encoder and decoder, the transformer’s deformation is used to extract cross-scale features, which can better preserve the global context and layout information of the input image samples;

[0127] 3. To address the problem of unequal information importance between different layers, a feature attention mechanism is proposed in the decoder stage to process the outputs of different layers of the decoder, obtain richer cross-layer features, and extract more important feature information between different layers, thereby improving the segmentation performance of the network.

[0128] The above is a detailed introduction to the pathological hyperspectral image detection method based on the cross-scale spatial-spectral feature fusion network provided by the present invention. This article uses specific examples to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the core idea of ​​the present invention. It should be pointed out that for ordinary technicians in this technical field, without departing from the principles of the present invention, the present invention can also be improved and modified in several ways, and these improvements and modifications also fall within the scope of protection of the claims of the present invention.

Claims

1. A pathological hyperspectral image detection method based on a cross-scale spatial-spectral feature fusion network, characterized by: The method comprises the following steps: S100: Acquire a hyperspectral image to form a data set, preprocess the data set, and obtain a preprocessed data set; S200: Build a hyperspectral image detection model, including an encoder based on multi-scale feature extraction, a transformer-based cross-scale feature extraction network, and a feature attention decoder; S300: Input the preprocessed data set into an encoder based on multi-scale feature extraction to extract image features and obtain a multi-scale encoding feature map output; S400: Input the multi-scale coding features into the transformer-based cross-scale feature extraction network, perform cross-scale feature extraction on the coding features, and obtain the final cross-scale feature map; the transformer-based cross-scale feature extraction network includes a multi-head attention module, a cross-scale feature extraction module, and a multi-layer perceptron module. S400 includes: S410: using a multi-head self-attention module to perform self-attention mechanism processing on the multi-scale encoding feature maps other than the first encoding feature map; S420: Using interpolation to convert the feature maps of different sizes obtained in S410 into a uniform size, using the transformer-based cross attention to perform cross-scale feature extraction on the feature maps of the uniform size, and obtain a cross-scale feature fusion output; S430: Using a multi-layer perceptron to process and transform the obtained cross-scale feature fusion output layer by layer, enhance the representation of local information, and obtain the final cross-scale feature output; S410 specifically includes: inputting the multi-scale encoded feature maps other than the first encoded feature map into different multi-head self-attention mechanisms, performing feature segmentation processing, evenly dividing them into several spaces according to the number of attention heads, performing parallel self-attention mechanism processing, and finally splicing and merging their outputs; S420 is specifically as follows: for each encoding feature map, the other encoding feature maps are adjusted to a uniform size using bilinear interpolation, and the bilinear interpolation calculates the value of the target pixel by using the values ​​of the four surrounding pixels; then the cross-attention based on the transformer is used to perform cross-scale feature extraction on the feature map output by S410, and the current encoding feature map is regarded as the value vector, and the other two encoding feature maps are regarded as the query vector and the key vector respectively, using three learnable projection matrices W q 、W k and W v Mapping is performed separately, and then the query vector Q and the key vector K are used for dot product operation. The attention weight is obtained through the ReLU function, and the most useful attention is aggregated to automatically ensure the sparsity of the attention weight. Finally, the attention weight is multiplied by the value vector V and the results are summed to obtain the cross-scale feature fusion output corresponding to the number of encoded feature maps. S430: Inputting the cross-scale feature fusion output corresponding to the number of encoded feature maps into the corresponding multi-layer perceptron module, processing and transforming the input features layer by layer, and obtaining final cross-scale feature output corresponding to the number of cross-scale feature fusion outputs; S500: Input the cross-scale feature map into the feature attention decoder for feature attention mechanism decoding processing to obtain the segmentation result; S500 includes: S510: upsampling the cross-scale feature map and concatenating it with the encoder feature map of the corresponding size, and using interpolation to unify the size of all concatenated feature maps to obtain a decoded feature map; S520: All decoded feature maps are spliced ​​together, and a pooled feature map is obtained through a pooling layer. The pooled feature map is then subjected to an attention operation on the original spliced ​​feature map to obtain a corresponding feature attention map. S530: Finally, the feature attention map is subjected to 3D convolution, sigmoid and spectral average pooling operations to obtain the segmentation result; S510 specifically: The decoder receives the last multi-scale encoded feature map and the corresponding cross-scale feature output map from the encoder. The decoder first upsamples the cross-scale feature map to double the spatial resolution of the feature map. Then, the decoder concatenates the upsampled feature map with the encoded feature map of the corresponding layer in the encoder. The above process is repeated until the size of the original concatenated feature map is restored to the size of the input image. The four decoded feature maps are transformed to a uniform size using the bilinear interpolation method. S520 is specifically as follows: the decoded feature maps after uniform size are spliced ​​along the spectral direction to obtain a spliced ​​feature map, which is sent to the pooling layer for pooling operation to obtain the corresponding pooled feature map, and then sent to the linear layer respectively to obtain the corresponding Q decoder ,K decoder , therefore, the feature attention weight matrix can be calculated as follows: W decoder =conv2D(Q decoder K decoder T ) Where conv2D(·) represents a 2D convolution with a kernel of 3×3 and a padding of 1. This operation is used to merge the relevant information between adjacent pixels and perform a linear operation on the original concatenated feature map to obtain the corresponding V decoder , thus obtaining the corresponding feature attention map Out decoder , calculated by the following formula: S600: The hyperspectral image detection model is trained using a cross-validation method combined with a preset loss function. When a preset training end condition is reached, a trained detection model is obtained, and the real-time pathological hyperspectral image segmentation is completed according to the trained detection model.

2. The method according to claim 1, characterized in that S100 includes: S110: performing a scaling operation on the data set to a quarter of its original size; S120: Perform data augmentation processing operations using the albumenation library to obtain a pre-training dataset; the data augmentation processing operations include rotation, scaling, inversion, color conversion, and blurring; S130: Divide the pre-training data set according to a certain ratio to obtain corresponding training sets and test sets, and cut the training set into a preset size.

3. The method according to claim 2, characterized in that The encoder based on multi-scale feature extraction includes 3D adaptive pooling layers, depthwise separable convolution, and multi-scale convolution. The S300 includes: S310: Input the preprocessed data set into an encoder based on multi-scale feature extraction, and use a 3D adaptive pooling layer to perform pooling processing on the images in the preprocessed data set to obtain an initial feature image; S320: Processing the initial feature image using two depthwise separable convolutions to change the dimension of the initial feature image; S330: Use multi-scale dilated convolution to extract features from the feature map after the dimension is changed to obtain the multi-scale coding features of the first stage. S340: Input the multi-scale coding features of the first stage into the 3D adaptive pooling layer, repeat steps S310 to S330 to obtain the multi-scale coding features of the second stage, use the output of the previous stage as the input of the next stage, repeat this four times, and obtain four coding feature map outputs of different sizes.

4. The method according to claim 3, characterized in that S310 is specifically: The images in the input preprocessed dataset are pooled using a 3D adaptive pooling layer. The 3D adaptive pooling layer is a pooling operation that automatically adjusts the pooling area according to the size of the input feature map so that each dimension of the output feature map reaches a preset size, obtaining the initial feature image after pooling. The two depth-wise separable convolutions include depth-wise convolution and point-wise convolution. Specifically, S320 is as follows: assuming that the dimension of the original feature image is H×W×C, the depth-wise convolution applies a convolution kernel of size K×K to each channel, performs spatial convolution, and produces an H×W×C output. Then, the point-wise convolution uses a convolution kernel of size 1×1×C to combine the output of the depth-wise convolution into a new feature map, and converts the output into H×W×D, where D is the number of output channels. S330 specifically includes: applying three dilated convolutions with different dilation rates to perform feature extraction on the feature map after the dimension is changed, extracting feature maps of different scales respectively, combining the feature maps of different scales by weighted averaging method, and obtaining a comprehensive multi-scale coding feature map.

5. The method according to claim 4, characterized in that In S420, for each coding feature map, other coding feature maps are adjusted to a uniform size using bilinear interpolation. The bilinear interpolation calculates the value of the target pixel by using the values ​​of the four surrounding pixels as follows: I′(x′,y′)=(1-Δx)(1-Δy)I(x1,y1)+Δx(1-Δy)I(x2,y1)+Δy(1-Δx)I(x1,y2)+ΔxΔyI(x2,y2) Where (x', y') is the pixel coordinate in the target image, (x1, y1) and (x2, y2) are the coordinates of the four pixels adjacent to the target pixel in the input feature map, and Δx and Δy are the offsets of the target coordinates relative to the input coordinates. In S420, the currently processed encoding feature map is regarded as the value vector, and the other two encoding feature maps are regarded as the query vector and the key vector respectively. The three learnable projection matrices W are used. q 、W k and W v Mapping is performed separately, specifically: Q=W q ×q,K=W k ×k,V=W v ×v Among them, q, k, and v represent the value vector, query vector, and key vector of the feature graph before processing, respectively, and Q, K, and V represent the value vector, query vector, and key vector of the feature graph after mapping processing, respectively.

6. The method according to claim 5, characterized in that S600 includes: The dataset is randomly divided into four non-overlapping subsets. Then, four training and validation cycles are performed, with one subset used as the validation set each time and the remaining three subsets used as the training set. The results of each validation are recorded. Finally, the average of the four validation results is calculated as the final evaluation indicator of the model. The above steps are repeated continuously, and the loss function obtained from the training set is continuously iterated until the loss is less than a certain threshold or the predetermined number of iterations is completed. The final model weights are recorded, and the real-time pathological hyperspectral image segmentation is completed according to the trained detection model.

Images