Pathological hyperspectral image detection method based on cross-scale spatial-spectral feature fusion network

Abstract

The invention discloses a pathological hyperspectral image detection method based on a cross-scale spatial-spectral feature fusion network, and the method proposes a feature extraction network based on cross-scale spatial-spectral feature fusion, and comprises the steps: firstly, extracting multi-scale information through employing expansion convolution at an encoder stage, so as to enable the network to better obtain the feature information of a plurality of receptive fields; then, in a jump connection stage of an encoder and a decoder, cross-scale feature extraction is carried out by utilizing deformation of transformer, so that global context information and layout information of an input image sample can be better reserved; and finally, in order to relieve the difficulty in spatial spectrum feature extraction, a feature attention mechanism is put forward at a decoder part, outputs of different layers of the decoder are processed, richer cross-layer features are obtained, and more important feature information between different layers is extracted, so that the segmentation performance of the network is improved, the neural network can better extract spectral feature fusion information, and the method is suitable for the spatial spectrum feature fusion. And the detection precision of the pathological hyperspectrum is improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of image processing, and in particular relates to a pathological hyperspectral image detection method based on a cross-scale spatial-spectral feature fusion network. Background Art

[0002] Hyperspectral images generally refer to spectral images of the same scene in hundreds of continuous bands, and their spectral resolution is usually on the order of 10^-2λ. Compared with the visible light of about 380-780nm that the human eye can only see, hyperspectral images can cover a wider band, thus presenting "Colors beyond colors" that the human eye cannot perceive. This kind of image is obtained through an imaging spectrometer, namely a hyperspectral sensor. These images contain not only the image information of the target, but also its spectral information, realizing the combination of spectrum and image. Due to the different absorption of the spectrum by different components, hyperspectral images can significantly reflect certain characteristics of the sample at a specific wavelength, such as size, shape and defects.

[0003] With the advancement of hyperspectral imaging technology, hyperspectral images have been widely used in agricultural monitoring, material testing, medical diagnosis and other fields. In the clinical diagnosis of cancer pathology images, microscopic analysis based on pathological tissue sections can be used to observe the development status of cancer tissues and thus judge the changes and metastasis of cancer. Therefore, pathological images are also called the gold standard in clinical diagnosis of cancer. Most early diagnoses require professional pathologists to observe and analyze pathological images in detail. The test results depend on the doctor's experience level and subjective analysis, which is one-sided and consumes a lot of time and labor costs. In pathological diagnosis, hyperspectral imaging technology can provide information on the tissue composition and spatial structure of the imaging object, which makes rapid and reliable pathological diagnosis possible.

[0004] However, compared with traditional remote sensing hyperspectral images, pathology hyperspectral images have the following problems: existing pathology slice hyperspectral data samples are relatively scarce, and factors such as patient ethics and privacy make it impossible for hospitals and research institutions to directly share patients' medical hyperspectral images. At the same time, the standards of pathology slices in different institutions are different, and pathology slice hyperspectral images are difficult to annotate. In addition, there are many types of cells in pathology slices and their morphologies vary, requiring the extraction of rich feature information. Therefore, it is extremely difficult to extract features from pathology hyperspectral images.

[0005] Therefore, a method is needed to effectively extract pathological hyperspectral features, adopt appropriate technology to solve the problem of inter-scale feature fusion, and use appropriate methods to fully fuse spatial spectral features, so as to improve the detection accuracy of pathological hyperspectral features. Summary of the invention

[0006] In view of the above technical problems, the present invention provides a pathological hyperspectral image detection method based on a cross-scale spatial-spectral feature fusion network.

[0007] The technical solution adopted by the present invention to solve the technical problem is:

[0008] A pathological hyperspectral image detection method based on a cross-scale spatial-spectral feature fusion network comprises the following steps:

[0009] S100: Acquire a hyperspectral image to form a data set, preprocess the data set, and obtain a preprocessed data set;

[0010] S200: Build a hyperspectral image detection model, including an encoder based on multi-scale feature extraction, a transformer-based cross-scale feature extraction network, and a feature attention decoder;

[0011] S300: inputting the preprocessed data set into an encoder based on multi-scale feature extraction, extracting image features, and obtaining a multi-scale encoding feature map output;

[0012] S400: inputting the multi-scale coding features into a transformer-based cross-scale feature extraction network, performing cross-scale feature extraction on the coding features, and obtaining a final cross-scale feature map;

[0013] S500: inputting the cross-scale feature map into the feature attention decoder for feature attention mechanism decoding processing to obtain the segmentation result;

[0014] S600: The hyperspectral image detection model is trained using a cross-validation method combined with a preset loss function. When a preset training end condition is reached, a trained detection model is obtained, and the real-time pathological hyperspectral image segmentation is completed according to the trained detection model.

[0015] Preferably, S100 includes:

[0016] S110: performing a scaling operation on the data set to scale it to a quarter of its original size;

[0017] S120: Perform data enhancement processing operations using the albumenation library to obtain a pre-training data set; the data enhancement processing operations include rotation, scaling, inversion, color conversion, and blurring;

[0018] S130: Divide the pre-training data set according to a certain ratio to obtain corresponding training sets and test sets, and cut the training set into a preset size.

[0019] Preferably, the encoder based on multi-scale feature extraction includes a 3D adaptive pooling layer, a depth-separable convolution and a multi-scale convolution, and S300 includes:

[0020] S310: inputting the preprocessed data set into an encoder based on multi-scale feature extraction, and performing pooling processing on the images in the preprocessed data set using a 3D adaptive pooling layer to obtain an initial feature image;

[0021] S320: Processing the initial feature image using two depth-wise separable convolutions to change the dimension of the initial feature image;

[0022] S330: Use multi-scale dilated convolution to extract features from the feature map after the dimension is changed to obtain the multi-scale coding features of the first stage.

[0023] S340: Input the multi-scale coding features of the first stage into the 3D adaptive pooling layer, repeat steps S310 to S330 to obtain the multi-scale coding features of the second stage, use the output of the previous stage as the input of the next stage, repeat this four times, and obtain four coding feature map outputs of different sizes.

[0024] Preferably, S310 specifically includes:

[0025] The images in the input preprocessed data set are pooled using a 3D adaptive pooling layer. The 3D adaptive pooling layer is a pooling operation that automatically adjusts the pooling area according to the size of the input feature map so that each dimension of the output feature map can reach a preset size, thereby obtaining the initial feature image after pooling.

[0026] The two depth-wise separable convolutions include depth-wise convolution and point-wise convolution. Specifically, S320 is as follows: assuming that the dimension of the original feature image is H×W×C, the depth-wise convolution applies a convolution kernel of size K×K to each channel, performs spatial convolution, and generates an output of H×W×C. Then, the point-wise convolution uses a convolution kernel of size 1×1×C to combine the output of the depth-wise convolution into a new feature map, and converts the output into H×W×D, where D is the number of output channels.

[0027] S330 specifically includes: applying three dilated convolutions with different dilation rates to perform feature extraction on the feature map after the dimension is changed, respectively extracting feature maps of different scales, combining the feature maps of different scales by a weighted average method, and obtaining a comprehensive multi-scale coding feature map;

[0028] Preferably, the transformer-based cross-scale feature extraction network includes a multi-head attention module, a cross-scale feature extraction module and a multi-layer perceptron module, and S400 includes:

[0029] S410: using a multi-head self-attention module to perform self-attention mechanism processing on the multi-scale encoding feature maps other than the first encoding feature map;

[0030] S420: Using interpolation to convert the feature maps of different sizes obtained in S410 into a uniform size, and using transformer-based cross attention to perform cross-scale feature extraction on the feature maps of the uniform size to obtain a cross-scale feature fusion output;

[0031] S430: Using a multi-layer perceptron to process and transform the obtained cross-scale feature fusion output layer by layer, enhance the representation of local information, and obtain the final cross-scale feature output.

[0032] Preferably, S410 specifically includes: inputting the multi-scale encoded feature maps other than the first encoded feature map into different multi-head self-attention mechanisms respectively, performing feature segmentation processing, evenly dividing them into a plurality of spaces according to the number of attention heads, performing parallel self-attention mechanism processing, and finally splicing and merging their outputs;

[0033] S420 is as follows: for each encoded feature map, other encoded feature maps are adjusted to a uniform size using bilinear interpolation, and the bilinear interpolation calculates the value of the target pixel by using the values ​​of the four surrounding pixels; then, the feature map output by S410 is subjected to cross-scale feature extraction using the transformer-based cross attention, and the currently processed encoded feature map is regarded as the value vector, and the other two encoded feature maps are regarded as the query vector and the key vector respectively, and three learnable projection matrices W are used. q , W k and W v Mapping is performed separately, and then the query vector Q and the key vector K are used for dot product operation, and the attention weight is obtained through the Relu function. The most useful attention is aggregated and the sparsity of the attention weight is automatically guaranteed. Finally, the attention weight is multiplied by the value vector V and the results are summed to obtain the cross-scale feature fusion output corresponding to the number of encoded feature maps.

[0034] S430: The multi-layer perceptron module corresponding to the cross-scale feature fusion output input corresponding to the number of encoded feature maps processes and transforms the input features layer by layer to obtain the final cross-scale feature output corresponding to the number of cross-scale feature fusion outputs.

[0035] Preferably, in S420, for each coding feature map, other coding feature maps are adjusted to a uniform size using bilinear interpolation, and the bilinear interpolation calculates the value of the target pixel by using the values ​​of the surrounding four pixels:

[0036] I′(x′,y′)=(1-Δx)(1-Δy)I(x1,y1)+Δx(1-Δy)I(x2,y1)+Δy(1-Δx)I(x1,y ( )+ΔxΔyI(x2,y2)

[0037] Among them, (x′, y′) is the pixel coordinate in the target image, (x1, y1) and (x2, y2) are the coordinates of the four pixels adjacent to the target pixel in the input feature map, and Δx and Δy are the offsets of the target coordinates relative to the input coordinates.

[0038] In S420, the currently processed encoding feature map is regarded as the value vector, and the other two encoding feature maps are regarded as the query vector and the key vector respectively. Three learnable projection matrices W are used. q , W k and W v Mapping is performed separately, specifically:

[0039] Q=W q ×q,K=W k ×k,V=W v ×v

[0040] Among them, q, k, and v represent the value vector, query vector, and key vector of the feature graph before processing, respectively, and Q, K, and V represent the value vector, query vector, and key vector of the feature graph after mapping processing, respectively.

[0041] Preferably, S500 includes:

[0042] S510: upsampling the cross-scale feature map, and concatenating it with the encoder feature map of the corresponding size, and using interpolation to unify the size of all the concatenated feature maps to obtain a decoded feature map;

[0043] S520: concatenate all decoded feature maps, pass through a pooling layer, obtain a pooled feature map, and perform an attention operation on the original concatenated feature map to obtain a corresponding feature attention map;

[0044] S530: Finally, the feature attention map is subjected to 3D convolution, sigmoid and spectral average pooling operations to obtain the segmentation result.

[0045] Preferably, S510 specifically includes:

[0046] Receive the last multi-scale encoded feature map and the corresponding cross-scale feature output map from the encoder. The decoder first upsamples the multi-scale encoded feature map to double the spatial resolution of the feature map. Then, the decoder concatenates the upsampled feature map with the encoded feature map of the corresponding layer in the encoder. Repeat the above process until the size of the original concatenated feature map is restored to the size of the input image. The four decoded feature maps are transformed to a uniform size using a bilinear interpolation method.

[0047] S520 is specifically as follows: the decoded feature maps after the uniform size are spliced ​​along the spectral direction to obtain the spliced ​​feature map, which is sent to the pooling layer for pooling operation to obtain the corresponding pooling feature map, and then sent to the linear layer respectively to obtain the corresponding Q decoder ,K decoder , therefore, the feature attention weight matrix can be calculated as follows:

[0048] W decoder =conv2D(Q decoder K decoder T )

[0049] Where conv2D(·) represents a 2D convolution with a kernel of 3×3 and padding set to 1. This operation is used to merge the relevant information between adjacent pixels. The original concatenated feature map is linearly operated to obtain the corresponding V decoder , thus obtaining the corresponding feature attention map Out decoder , calculated by the following formula:

[0050]

[0051] Preferably, S600 includes:

[0052] The data set is randomly divided into four non-overlapping subsets. Then, four training and validation are performed. Each time, one subset is used as the validation set, and the remaining three subsets are used as the training set. The result of each validation is recorded. Finally, the average of the four validation results is calculated as the final evaluation index of the model. The above steps are repeated continuously, and the loss function obtained from the training set is continuously iterated until the loss is less than a certain threshold or the predetermined number of iterations is completed. The final model weights are recorded, and the real-time pathological hyperspectral image segmentation is completed according to the trained detection model.

[0053] The above-mentioned pathological hyperspectral image detection method based on a cross-scale spatial-spectral feature fusion network proposes a feature extraction network based on cross-scale spatial-spectral feature fusion. First, in the encoder stage, dilated convolution is used to extract multi-scale information; then, in the jump connection stage between the encoder and the decoder, the deformation of the transformer is used to extract cross-scale features; finally, in order to alleviate the difficulty of spatial-spectral feature extraction, a feature attention mechanism is proposed in the decoder part, so that the neural network can better extract spectral feature fusion information and improve the detection accuracy of pathological hyperspectral images. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] Figure 1 It is a flow chart of a pathological hyperspectral image detection method based on a cross-scale spatial-spectral feature fusion network in one embodiment of the present invention;

[0055] Figure 2Schematic diagram of an encoder network based on multi-scale feature extraction in one embodiment of the present invention;

[0056] Figure 3 Schematic diagram of a transformer-based cross-scale feature extraction network in one embodiment of the present invention;

[0057] Figure 4 Schematic diagram of a feature attention decoder network in one embodiment of the present invention. DETAILED DESCRIPTION

[0058] In order to enable those skilled in the art to better understand the technical solution of the present invention, the present invention is further described in detail below in conjunction with the accompanying drawings.

[0059] In one embodiment, Figure 1 As shown, a pathological hyperspectral image detection method based on a cross-scale spatial-spectral feature fusion network comprises the following steps:

[0060] S100: Acquire a hyperspectral image to form a data set, preprocess the data set, and obtain a preprocessed data set;

[0061] S200: Build a hyperspectral image detection model, including an encoder based on multi-scale feature extraction, a transformer-based cross-scale feature extraction network, and a feature attention decoder;

[0062] S300: inputting the preprocessed data set into an encoder based on multi-scale feature extraction, extracting image features, and obtaining a multi-scale encoding feature map output;

[0063] S400: inputting the multi-scale coding features into a transformer-based cross-scale feature extraction network, performing cross-scale feature extraction on the coding features, and obtaining a final cross-scale feature map;

[0064] S500: inputting the cross-scale feature map into the feature attention decoder for feature attention mechanism decoding processing to obtain the segmentation result;

[0065] S600: The hyperspectral image detection model is trained by using a cross-validation method combined with a preset loss function. When a preset training end condition is reached, a trained detection model is obtained, and the real-time pathological hyperspectral image segmentation is completed according to the trained detection model.

[0066] The above-mentioned pathological hyperspectral image detection method based on a cross-scale spatial-spectral feature fusion network proposes a feature extraction network based on cross-scale spatial-spectral feature fusion. First, in the encoder stage, dilated convolution is used to extract multi-scale information; then, in the jump connection stage between the encoder and the decoder, the deformation of the transformer is used to extract cross-scale features; finally, in order to alleviate the difficulty of spatial-spectral feature extraction, a feature attention mechanism is proposed in the decoder part, so that the neural network can better extract spectral feature fusion information and improve the detection accuracy of pathological hyperspectral images.

[0067] In one embodiment, S100 includes:

[0068] S110: performing a scaling operation on the data set to scale it to a quarter of its original size;

[0069] S120: Perform data enhancement processing operations using the albumenation library to obtain a pre-training data set; the data enhancement processing operations include rotation, scaling, inversion, color conversion, and blurring;

[0070] S130: Divide the pre-training data set according to a certain ratio to obtain corresponding training sets and test sets, and cut the training set into a preset size.

[0071] Specifically, the hyperspectral data with an original size of 1280×1024 is scaled to reduce its length and width to a quarter of the original size, becoming 320×256. Then, the albumentations library is used to perform operations such as rotation, scaling, flipping, deformation, color change, blurring, etc. to achieve data enhancement and obtain the pre-training dataset D all ; Finally, the data set is divided into four parts, and one of them is randomly selected as the validation set D val , the remaining data is used as the training set D train , and the training set data is cropped to 192 × 192. The number of images in the training set is N × 75%, and the number of images in the validation set is N × 25%.

[0072] In one embodiment, the encoder based on multi-scale feature extraction includes a 3D adaptive pooling layer, a depth-separable convolution and a multi-scale convolution, and S300 includes:

[0073] S310: inputting the preprocessed data set into an encoder based on multi-scale feature extraction, and performing pooling processing on the images in the preprocessed data set using a 3D adaptive pooling layer to obtain an initial feature image;

[0074] S320: Processing the initial feature image using two depth-wise separable convolutions to change the dimension of the initial feature image;

[0075] S330: Use multi-scale dilated convolution to extract features from the feature map after the dimension is changed to obtain the multi-scale coding features of the first stage.

[0076] S340: Input the multi-scale coding features of the first stage into the 3D adaptive pooling layer, repeat steps S310 to S330 to obtain the multi-scale coding features of the second stage, use the output of the previous stage as the input of the next stage, repeat this four times, and obtain four coding feature map outputs of different sizes.

[0077] In one embodiment, S310 specifically includes:

[0078] The images in the input preprocessed data set are pooled using a 3D adaptive pooling layer. The 3D adaptive pooling layer is a pooling operation that automatically adjusts the pooling area according to the size of the input feature map so that each dimension of the output feature map can reach a preset size, thereby obtaining the initial feature image after pooling.

[0079] The two depth-wise separable convolutions include depth-wise convolution and point-wise convolution. Specifically, S320 is as follows: assuming that the dimension of the original feature image is H×W×C, the depth-wise convolution applies a convolution kernel of size K×K to each channel, performs spatial convolution, and generates an output of H×W×C. Then, the point-wise convolution uses a convolution kernel of size 1×1×C to combine the output of the depth-wise convolution into a new feature map, and converts the output into H×W×D, where D is the number of output channels.

[0080] S330 specifically includes: applying three dilated convolutions with different dilation rates to perform feature extraction on the feature map after the dimension is changed, respectively extracting feature maps of different scales, combining the feature maps of different scales by a weighted average method, and obtaining a comprehensive multi-scale coding feature map;

[0081] Specifically, Figure 2 As shown, the entire hyperspectral image is input and defined as X∈R H×W×C , where H, W, and C represent the length, width, and number of bands of the hyperspectral image, respectively. The encoder based on multi-scale feature extraction mainly consists of three parts: 3D adaptive pooling layer, depth-wise separable convolution, and multi-scale convolution.

[0082] First, the input hyperspectral image is pooled using a 3D adaptive pooling layer. The 3D adaptive pooling layer is a pooling operation that automatically adjusts the pooling area according to the size of the input feature map to generate a fixed-size output feature map. Different from the traditional fixed-size pooling window, adaptive pooling adjusts the size and stride of the pooling window so that the input feature map is equally divided into a target number of regions. Each region obtains an output value through maximum pooling or average pooling, so that each dimension of the output feature map can reach a preset size, thus obtaining the initial feature image after pooling;

[0083] Then, two depthwise separable convolutions are used to extract features from the initial feature image. Depthwise separable convolution is an efficient convolution operation with a relatively low number of parameters and computational cost. It mainly includes two steps: depthwise convolution and pointwise convolution. Assuming that the dimension of the input feature map is H×W×C, the depthwise convolution applies a convolution kernel of size K×K to each channel, performs spatial convolution, and produces an output of H×W×C. Then, the pointwise convolution uses a convolution kernel of size 1×1×C to combine the output of the depthwise convolution into a new feature map, converting the output to H×W×D, where D is the number of output channels;

[0084] Finally, multi-scale convolution is used for feature extraction to obtain the final multi-scale encoding features, and a multi-scale convolution mechanism based on dilated convolution is introduced to effectively extract features. Dilated convolution increases the receptive field without sacrificing resolution by enlarging the size of the convolution kernel using a dilation factor. The dilated convolution operation achieves a wider feature coverage by introducing intervals in the convolution process. For a dilated convolution with a dilation factor of S, the receptive field calculation formula is: K s =S×(K0-1)+1, where K s K0 is the size of the convolution kernel after the ordinary convolution kernel is expanded. In order to avoid the "grid effect", different expansion rates (1, 2 and 3) are selected to ensure that there is no common divisor greater than 1 between these expansion rates. The specific process is as follows: First, three dilated convolutions with different expansion rates (1, 2 and 3) are applied to extract feature maps of different scales respectively. Although these convolutions share the same convolution kernel weights, they can capture features of different scales due to different expansion rates. Then, these feature maps of different scales are combined by weighted averaging to obtain a comprehensive multi-scale encoding feature map.

[0085] This is repeated four times to obtain four feature map outputs of different sizes, which effectively captures and utilizes multi-scale features and provides richer feature information for subsequent steps.

[0086] In one embodiment, the transformer-based cross-scale feature extraction network includes a multi-head attention module, a cross-scale feature extraction module and a multi-layer perceptron module, and S400 includes:

[0087] S410: using a multi-head self-attention module to perform self-attention mechanism processing on the multi-scale encoding feature maps other than the first encoding feature map;

[0088] S420: Using interpolation to convert the feature maps of different sizes obtained in S410 into a uniform size, and using transformer-based cross attention to perform cross-scale feature extraction on the feature maps of the uniform size to obtain a cross-scale feature fusion output;

[0089] S430: Using a multi-layer perceptron to process and transform the obtained cross-scale feature fusion output layer by layer, enhance the representation of local information, and obtain the final cross-scale feature output.

[0090] In one embodiment, S410 specifically includes: inputting the multi-scale encoded feature maps other than the first encoded feature map into different multi-head self-attention mechanisms respectively, performing feature segmentation processing, evenly dividing them into a plurality of spaces according to the number of attention heads, performing parallel self-attention mechanism processing, and finally splicing and merging their outputs;

[0091] S420 is as follows: for each encoded feature map, other encoded feature maps are adjusted to a uniform size using bilinear interpolation, and the bilinear interpolation calculates the value of the target pixel by using the values ​​of the four surrounding pixels; then, the feature map output by S410 is subjected to cross-scale feature extraction using the transformer-based cross attention, and the currently processed encoded feature map is regarded as the value vector, and the other two encoded feature maps are regarded as the query vector and the key vector respectively, and three learnable projection matrices W are used. q , W k and W v Mapping is performed separately, and then the query vector Q and the key vector K are used for dot product operation, and the attention weight is obtained through the Relu function. The most useful attention is aggregated and the sparsity of the attention weight is automatically guaranteed. Finally, the attention weight is multiplied by the value vector V and the results are summed to obtain the cross-scale feature fusion output corresponding to the number of encoded feature maps.

[0092] S430: The multi-layer perceptron module corresponding to the cross-scale feature fusion output input corresponding to the number of encoded feature maps processes and transforms the input features layer by layer to obtain the final cross-scale feature output corresponding to the number of cross-scale feature fusion outputs.

[0093] In one embodiment, in S420, for each coding feature map, other coding feature maps are adjusted to a uniform size using bilinear interpolation, and the bilinear interpolation calculates the value of the target pixel by using the values ​​of the surrounding four pixels:

[0094] I′(x′,y′)=(1-Δx)(1-Δy)I(x1,y1)+Δx(1-Δy)I(x2,y1)+Δy(1-Δx)I(x1,y2)+ΔxΔyI(x2,y2)

[0095] Among them, (x′, y′) is the pixel coordinate in the target image, (x1, y1) and (x2, y2) are the coordinates of the four pixels adjacent to the target pixel in the input feature map, and Δx and Δy are the offsets of the target coordinates relative to the input coordinates.

[0096] In S420, the currently processed encoding feature map is regarded as the value vector, and the other two encoding feature maps are regarded as the query vector and the key vector respectively. Three learnable projection matrices W are used. q , W k and W v Mapping is performed separately, specifically:

[0097] Q=W q ×q,K=W k ×k,V=W v ×v

[0098] Among them, q, k, and v represent the value vector, query vector, and key vector of the feature graph before processing, respectively, and Q, K, and V represent the value vector, query vector, and key vector of the feature graph after mapping processing, respectively.

[0099] Specifically, Figure 3 As shown in the figure, S300 obtains four multi-scale encoded feature maps of different sizes, and uses a transformer-based cross-scale feature extraction network to process the feature maps of the following three layers, which mainly includes three steps: multi-head attention module, cross-scale feature extraction and multi-layer perceptron module. The multi-head attention mechanism and multi-layer perceptron module are key technologies in the Transformer model.

[0100] First, different encoding feature maps are input into different multi-head self-attention mechanisms for feature segmentation. They are evenly divided into several spaces according to the number of attention heads and processed by parallel self-attention mechanisms. Finally, their outputs are spliced ​​and merged to capture the connections and differences in the sequence in more detail.

[0101] Then, for each feature map, the following operations are performed: for each encoded feature map, other feature maps are adjusted to a uniform size using bilinear interpolation. Bilinear interpolation calculates the value of the target pixel by using the values ​​of the four surrounding pixels. The formula is:

[0102] I′(x′,y′)=(1-Δx)(1-Δy)I(x1,y1)+Δx(1-Δy)I(x2,y1)+Δy(1-Δx)I(x1,y2)

[0103] +ΔxΔyI(x2,y2)

[0104] Among them, (x′, y′) is the pixel coordinate in the target image, (x1, y1) and (x2, y2) are the coordinates of the four pixels adjacent to the target pixel in the input feature map, Δx and Δy are the offsets of the target coordinates relative to the input coordinates. Then, the transformer-based cross-attention is used to extract cross-scale features from the feature map. The feature map to be processed is regarded as the value vector, and the other two are regarded as the query vector and the key vector respectively. Then, three learnable projection matrices W are used. q , W k and W v Mapping is performed separately, namely: Q = W q ×q,K=W k ×k,V=W v ×v, and then perform a dot product operation on the query vector Q and the key vector K, and obtain the attention weight through the Relu function. The Relu function can aggregate the most useful attention features and automatically ensure the sparsity of the attention weight; finally, multiply the attention weight with the value vector V, and sum the results to obtain the cross-scale feature fusion output.

[0105] Finally, the cross-scale feature fusion output is input into the multi-layer perceptron module to obtain the final cross-scale feature output. The multi-layer perceptron module can process and transform the input features layer by layer to extract more complex and high-order feature representations. This module plays an important role in feature extraction and classification tasks and can effectively improve the performance and accuracy of the model.

[0106] In one embodiment, S500 includes:

[0107] S510: upsampling the cross-scale feature map, and concatenating it with the encoder feature map of the corresponding size, and using interpolation to unify the size of all the concatenated feature maps to obtain a decoded feature map;

[0108] S520: concatenate all decoded feature maps, pass through a pooling layer, obtain a pooled feature map, and perform an attention operation on the original concatenated feature map to obtain a corresponding feature attention map;

[0109] S530: Finally, the feature attention map is subjected to 3D convolution, sigmoid and spectral average pooling operations to obtain the segmentation result.

[0110] In one embodiment, S510 specifically includes:

[0111] Receive the last multi-scale encoded feature map and the corresponding cross-scale feature output map from the encoder. The decoder first upsamples the multi-scale encoded feature map to double the spatial resolution of the feature map. Then, the decoder concatenates the upsampled feature map with the encoded feature map of the corresponding layer in the encoder. Repeat the above process until the size of the original concatenated feature map is restored to the size of the input image. The four decoded feature maps are transformed to a uniform size using a bilinear interpolation method.

[0112] S520 is specifically as follows: the decoded feature maps after the uniform size are spliced ​​along the spectral direction to obtain the spliced ​​feature map, which is sent to the pooling layer for pooling operation to obtain the corresponding pooling feature map, and then sent to the linear layer respectively to obtain the corresponding Q decoder ,K decoder , therefore, the feature attention weight matrix can be calculated as follows:

[0113] W decoder =conv2D(Q decoder K decoder T (

[0114] Where conv2D(·) represents a 2D convolution with a kernel of 3×3 and padding set to 1. This operation is used to merge the relevant information between adjacent pixels. The original concatenated feature map is linearly operated to obtain the corresponding V decoder , thus obtaining the corresponding feature attention map Out decoder , calculated by the following formula:

[0115]

[0116] Specifically, the main task of the decoder is to gradually restore the spatial resolution of the feature map to generate an output image with the same size as the input image, while exploiting the features of the encoder for detail recovery.

[0117] First, the decoder receives the last multi-scale encoded feature map and the corresponding cross-scale feature output map from the encoder. The decoder first upsamples the multi-scale encoded feature map to double the spatial resolution of the feature map, and then concatenates the upsampled feature map of the decoder with the feature map of the corresponding layer in the encoder. The above process is repeated until the size of the feature map is restored to the size of the input image, and the four decoded feature maps are transformed to a uniform size using the bilinear interpolation method.

[0118] like Figure 4 As shown in Figure 1, the feature maps of uniform size are then spliced ​​along the spectral direction to obtain the spliced ​​feature map, which is sent to the pooling layer for pooling operation to obtain the corresponding pooled feature map, which is then sent to the linear layer to obtain the corresponding Q decoder ,K decoder , therefore, the feature attention weight matrix can be calculated as follows: W decoder =conv2D(Q decoder K decoder T ), where conv2D(·) represents a 2D convolution with a kernel of 3×3 and padding set to 1. This operation is used to merge the relevant information between adjacent pixels. The original concatenated feature map is linearly operated to obtain the corresponding V decoder , thus obtaining the corresponding feature attention map Out decoder , can be calculated by the following formula: Finally, the feature attention map is subjected to 3D convolution, sigmoid and spectral average pooling operations to obtain the final 2D output result.

[0119] In one embodiment, S600 includes:

[0120] The data set is randomly divided into four non-overlapping subsets. Then, four training and validation are performed. Each time, one subset is used as the validation set, and the remaining three subsets are used as the training set. The result of each validation is recorded. Finally, the average of the four validation results is calculated as the final evaluation index of the model. The above steps are repeated continuously, and the loss function obtained from the training set is continuously iterated until the loss is less than a certain threshold or the predetermined number of iterations is completed. The final model weights are recorded, and the real-time pathological hyperspectral image segmentation is completed according to the trained detection model.

[0121] Specifically, in order to reduce overfitting and improve the generalization ability of the model, the four-fold cross validation is used to process the data set. The specific steps are: first, the data set is randomly divided into four non-overlapping subsets, and then four training and validation are performed, each time one subset is used as the validation set, and the remaining three subsets are used as training sets, and the results of each validation are recorded; finally, the average of the four validation results is calculated as the final evaluation indicator of the model.

[0122] Repeat the above steps continuously, and iterate continuously according to the loss function obtained from the training set until the loss is less than a certain threshold or the predetermined number of iterations is completed, and the final model weights are obtained and recorded.

[0123] During the test, the final model weights are loaded; then, all data is used for testing, whether training set or validation set, all data is sent to the final trained model, and the predicted segmentation results are compared with the true labels to obtain the final segmentation accuracy.

[0124] Compared with the prior art, the above-mentioned pathological hyperspectral image detection method based on the cross-scale spatial-spectral feature fusion network has the following beneficial effects:

[0125] 1. The pathological hyperspectral image detection method based on the cross-scale spatial-spectral feature fusion network adopts dilated convolution to extract multi-scale spatial-spectral features in the encoder part, so that the network can better obtain the feature information of multiple receptive fields;

[0126] 2. In the jump connection stage between the encoder and decoder, the transformer deformation is used to extract cross-scale features, which can better preserve the global context information and layout information of the input image samples;

[0127] 3. To address the problem of unequal information importance between different layers, a feature attention mechanism is proposed in the decoder stage to process the outputs of different layers of the decoder, obtain richer cross-layer features, and extract more important feature information between different layers, thereby improving the segmentation performance of the network.

[0128] The above is a detailed introduction to the pathological hyperspectral image detection method based on a cross-scale spatial-spectral feature fusion network provided by the present invention. This article uses specific examples to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the core idea of ​​the present invention. It should be pointed out that for ordinary technicians in this technical field, without departing from the principles of the present invention, the present invention can also be improved and modified, and these improvements and modifications also fall within the scope of protection of the claims of the present invention.

Claims

1. A pathological hyperspectral image detection method based on a cross-scale spatial-spectral feature fusion network, characterized in that: The method comprises the following steps: S100: Acquire a hyperspectral image to form a data set, preprocess the data set, and obtain a preprocessed data set; S200: Build a hyperspectral image detection model, including an encoder based on multi-scale feature extraction, a transformer-based cross-scale feature extraction network, and a feature attention decoder; S300: inputting the preprocessed data set into an encoder based on multi-scale feature extraction, extracting image features, and obtaining a multi-scale encoding feature map output; S400: inputting the multi-scale coding features into a transformer-based cross-scale feature extraction network, performing cross-scale feature extraction on the coding features, and obtaining a final cross-scale feature map; S500: inputting the cross-scale feature map into the feature attention decoder for feature attention mechanism decoding processing to obtain the segmentation result; S600: The hyperspectral image detection model is trained using a cross-validation method combined with a preset loss function. When a preset training end condition is reached, a trained detection model is obtained, and the real-time pathological hyperspectral image segmentation is completed according to the trained detection model.

2. The method according to claim 1, characterized in that S100 includes: S110: performing a scaling operation on the data set to scale it to a quarter of its original size; S120: Perform data enhancement processing operations using the albumenation library to obtain a pre-training data set; the data enhancement processing operations include rotation, scaling, inversion, color conversion, and blurring; S130: Divide the pre-training data set according to a certain ratio to obtain corresponding training sets and test sets, and cut the training set into a preset size.

3. The method according to claim 2, characterized in that The encoder based on multi-scale feature extraction includes 3D adaptive pooling layer, depth-wise separable convolution and multi-scale convolution. S300 includes: S310: inputting the preprocessed data set into an encoder based on multi-scale feature extraction, and performing pooling processing on the images in the preprocessed data set using a 3D adaptive pooling layer to obtain an initial feature image; S320: Processing the initial feature image using two depth-wise separable convolutions to change the dimension of the initial feature image; S330: Use multi-scale dilated convolution to extract features from the feature map after the dimension is changed to obtain the multi-scale coding features of the first stage. S340: Input the multi-scale coding features of the first stage into the 3D adaptive pooling layer, repeat steps S310 to S330 to obtain the multi-scale coding features of the second stage, use the output of the previous stage as the input of the next stage, repeat this four times, and obtain four coding feature map outputs of different sizes.

4. The method according to claim 3, characterized in that S310 is specifically: The images in the input preprocessed data set are pooled using a 3D adaptive pooling layer. The 3D adaptive pooling layer is a pooling operation that automatically adjusts the pooling area according to the size of the input feature map so that each dimension of the output feature map can reach a preset size, thereby obtaining the initial feature image after pooling. The two depth-wise separable convolutions include depth-wise convolution and point-wise convolution. Specifically, S320 is as follows: assuming that the dimension of the original feature image is H×W×C, the depth-wise convolution applies a convolution kernel of size K×K to each channel, performs spatial convolution, and generates an output of H×W×C. Then, the point-wise convolution uses a convolution kernel of size 1×1×C to combine the output of the depth-wise convolution into a new feature map, and converts the output into H×W×D, where D is the number of output channels. S330 specifically includes: applying three dilated convolutions with different dilation rates to perform feature extraction on the feature map after the dimension is changed, respectively extracting feature maps of different scales, combining the feature maps of different scales by weighted averaging, and obtaining a comprehensive multi-scale coding feature map.

5. The method according to claim 4, characterized in that The transformer-based cross-scale feature extraction network includes a multi-head attention module, a cross-scale feature extraction module, and a multi-layer perceptron module. S400 includes: S410: using a multi-head self-attention module to perform self-attention mechanism processing on the multi-scale encoding feature maps other than the first encoding feature map; S420: Using interpolation to convert the feature maps of different sizes obtained in S410 into a uniform size, and using transformer-based cross attention to perform cross-scale feature extraction on the feature maps of the uniform size to obtain a cross-scale feature fusion output; S430: Using a multi-layer perceptron to process and transform the obtained cross-scale feature fusion output layer by layer, enhance the representation of local information, and obtain the final cross-scale feature output.

6. The method according to claim 5, characterized in that S410 specifically includes: inputting the multi-scale encoded feature maps other than the first encoded feature map into different multi-head self-attention mechanisms respectively, performing feature segmentation processing, evenly dividing them into a plurality of spaces according to the number of attention heads, performing parallel self-attention mechanism processing, and finally splicing and merging their outputs; S420 is as follows: for each encoded feature map, other encoded feature maps are adjusted to a uniform size using bilinear interpolation, and the bilinear interpolation calculates the value of the target pixel by using the values ​​of the four surrounding pixels; then, the feature map output by S410 is subjected to cross-scale feature extraction using the transformer-based cross attention, and the currently processed encoded feature map is regarded as the value vector, and the other two encoded feature maps are regarded as the query vector and the key vector respectively, and three learnable projection matrices W are used. q , W k and W v Mapping is performed separately, and then the query vector Q and the key vector K are used for dot product operation, and the attention weight is obtained through the Relu function. The most useful attention is aggregated and the sparsity of the attention weight is automatically guaranteed. Finally, the attention weight is multiplied by the value vector V and the results are summed to obtain the cross-scale feature fusion output corresponding to the number of encoded feature maps. S430: The multi-layer perceptron module corresponding to the cross-scale feature fusion output input corresponding to the number of encoded feature maps processes and transforms the input features layer by layer to obtain the final cross-scale feature output corresponding to the number of cross-scale feature fusion outputs.

7. The method according to claim 6, characterized in that In S420, for each coding feature map, other coding feature maps are adjusted to a uniform size using bilinear interpolation. The bilinear interpolation calculates the value of the target pixel by using the values ​​of the surrounding four pixels: I′(x′,y′)=(1-Δx)(1-Δy)I(x1,y1)+Δx(1-Δy)I(x2,y1)+Δy(1-Δx)I(x1,y2)+ΔxΔyI(x2,y2) Among them, (x′, y′) is the pixel coordinate in the target image, (x1, y1) and (x2, y2) are the coordinates of the four pixels adjacent to the target pixel in the input feature map, and Δx and Δy are the offsets of the target coordinates relative to the input coordinates. In S420, the currently processed encoding feature map is regarded as the value vector, and the other two encoding feature maps are regarded as the query vector and the key vector respectively. Three learnable projection matrices W are used. q , W k and W v Mapping is performed separately, specifically: Q=W q ×q,K=W k ×k,V=W v ×v Among them, q, k, and v represent the value vector, query vector, and key vector of the feature graph before processing, respectively, and Q, K, and V represent the value vector, query vector, and key vector of the feature graph after mapping processing, respectively.

8. The method according to claim 7, characterized in that S500 includes: S510: upsampling the cross-scale feature map, and concatenating it with the encoder feature map of the corresponding size, and using interpolation to unify the size of all the concatenated feature maps to obtain a decoded feature map; S520: concatenate all decoded feature maps, pass through a pooling layer, obtain a pooled feature map, and perform an attention operation on the original concatenated feature map to obtain a corresponding feature attention map; S530: Finally, the feature attention map is subjected to 3D convolution, sigmoid and spectral average pooling operations to obtain the segmentation result.

9. The method according to claim 8, characterized in that S510 is specifically: Receive the last multi-scale encoded feature map and the corresponding cross-scale feature output map from the encoder. The decoder first upsamples the cross-scale feature map to double the spatial resolution of the feature map. Then, the decoder concatenates the upsampled feature map with the encoded feature map of the corresponding layer in the encoder. Repeat the above process until the size of the original concatenated feature map is restored to the size of the input image. Use the bilinear interpolation method to transform the four decoded feature maps to a uniform size. S520 is specifically as follows: the decoded feature maps after the uniform size are spliced ​​along the spectral direction to obtain the spliced ​​feature map, which is sent to the pooling layer for pooling operation to obtain the corresponding pooling feature map, and then the corresponding Q is respectively sent to the linear layer to obtain the corresponding Q decoder ,K decoder , therefore, the feature attention weight matrix can be calculated as follows: W decoder =conv2D(Q decoder K decoder T ) Where conv2D(·) represents a 2D convolution with a kernel of 3×3 and padding set to 1. This operation is used to merge the relevant information between adjacent pixels. The original concatenated feature map is linearly operated to obtain the corresponding V decoder , thus obtaining the corresponding feature attention map Out decoder , calculated by the following formula:

10. The method according to claim 9, characterized in that S600 includes: The data set is randomly divided into four non-overlapping subsets. Then, four training and validation are performed. Each time, one subset is used as the validation set, and the remaining three subsets are used as the training set. The result of each validation is recorded. Finally, the average of the four validation results is calculated as the final evaluation index of the model. The above steps are repeated continuously, and the loss function obtained from the training set is continuously iterated until the loss is less than a certain threshold or the predetermined number of iterations is completed. The final model weights are recorded, and the real-time pathological hyperspectral image segmentation is completed according to the trained detection model.

Images