Spatial spectrum context feature enhancement method and device for remote sensing image semantic segmentation

Abstract

The invention discloses a remote sensing image semantic segmentation-oriented spatial spectrum context feature enhancement method and device. The method comprises the following steps of: constructing a neural network model based on an encoder-decoder architecture; inputting the preprocessed remote sensing image set into a feature coding block, and extracting hierarchical features; according to the model, spectrum attention and position self-attention mechanisms are integrated through a collaborative attention module, and the sensitivity of network spectrum and spatial context information is remarkably enhanced; the network coding module efficiently compresses and codes the features before entering a decoding stage; a cross fusion module is applied to intelligently fuse features between a coding block and a decoding block to ensure that information is not lost in a decoding process; the feature map is mapped into a pixel-level category label through a Softmax layer, precise semantic segmentation is achieved, and a high-quality classification result is obtained. According to the method, through collaborative modeling of the spectrum and the spatial context, high-frequency details are effectively reserved, and the precision of semantic segmentation is improved.

Description

Technical Field

[0001] The present invention belongs to the field of computer vision technology, and in particular relates to a method and device for enhancing spatial spectral context features for semantic segmentation of remote sensing images. Background Art

[0002] Accurate segmentation from remote sensing images is of great significance for land resource management, land area planning, and human socio-economic development. In remote sensing images, there is a challenging problem that intra-class differences are significant while inter-class differences are subtle. Specifically, even geographical entities belonging to the same category may show obvious visual inconsistencies due to multiple factors such as their growth environment and seasonal changes; on the contrary, geographical entities belonging to different categories may show very similar visual features in certain specific situations. For example, the same type of vegetation may have significant changes in surface texture and color properties if it is in different ecological environments or affected by different climatic conditions. In addition, the shadow areas created by buildings may have high similarity in color characteristics with certain types of impervious ground, which undoubtedly adds considerable difficulty to the semantic segmentation task. Traditional segmentation methods often fail to effectively capture the subtle spatial details and spectral features in images, which leads to a decrease in segmentation accuracy and blurred boundaries.

[0003] To address these challenges, existing research methods focus on starting with spatial domain features, digging deep into and utilizing contextual information to extract and enhance pixel-level category features. The core of this method is to help the model more accurately identify and respond to various intra-class changes, while also being able to finely distinguish tiny inter-class differences, thereby achieving the goal of improving segmentation accuracy.

[0004] However, it is worth noting that the internal features of an image usually contain low-frequency information, while the edge features are rich in high-frequency information. When using image information, these current methods always ignore an important dimension, namely, spectral domain information. Summary of the invention

[0005] In view of the difficulty that remote sensing images have significant differences within classes but subtle differences between classes, and faced with the problems that high-frequency details are easily lost and semantic boundaries are highly uncertain in existing methods, the purpose of the present invention is to provide a spatial-spectral context feature enhancement method and device for semantic segmentation of remote sensing images, which adopts an encoder-decoder architecture, combined with an innovative collaborative attention module and a cross-fusion module, to achieve enhanced acquisition of high-frequency information in remote sensing images, enhance the certainty of semantic boundaries, and achieve the purpose of optimizing deep visual features of remote sensing images.

[0006] To achieve the above object, the technical solution adopted by the present invention is as follows:

[0007] In a first aspect, the present invention provides a method for enhancing spatial spectral context features for semantic segmentation of remote sensing images, comprising:

[0008] Acquire remote sensing image datasets and perform preprocessing;

[0009] Build a deep visual feature optimization neural network model based on encoder-decoder architecture;

[0010] Inputting the preprocessed remote sensing image data set into the deep visual feature optimization neural network model, extracting image features and performing spectral and spatial context enhancement, reconstructing key features, and outputting semantic segmentation results of the image based on the key features;

[0011] Calculate the loss value based on the output semantic segmentation result and the true label, update the model parameters, iterate the training until the end, save the optimal model parameters, and obtain a trained semantic segmentation model;

[0012] The remote sensing image to be tested is input into the trained semantic segmentation model to obtain the semantic segmentation result of the image.

[0013] Preferably, the acquiring of remote sensing image data sets and preprocessing includes:

[0014] Acquire a remote sensing image data set and divide it into a training data set and a test data set, and preprocess the training data set and the test data set before each iteration;

[0015] The preprocessing refers to performing random cropping, random horizontal sliding, random flipping, rotation and color enhancement operations on the training data set and the test data set.

[0016] Preferably, the deep visual feature optimization neural network model based on the encoder-decoder architecture includes:

[0017] A feature extraction module is used to extract a hierarchical feature map of an input remote sensing image;

[0018] A collaborative attention module is used to enhance the spectral and spatial context of the extracted feature maps and reconstruct key features;

[0019] The network coding module is used to compress and encode the features before entering the decoding stage;

[0020] Cross-fusion module, used to fuse encoding features and decoding features;

[0021] and,

[0022] The result output module is used to output the semantic segmentation results.

[0023] Preferably, the data processing process of the collaborative attention module is:

[0024] The input features are processed by the spectral attention module and the intermediate features are obtained by pixel summing with the input features;

[0025] The intermediate features are processed by the position self-attention module, and the output features are obtained by pixel summing with the intermediate features and input features.

[0026] Preferably, the step of processing the input feature through a spectral attention module and performing pixel summation with the input feature to obtain an intermediate feature comprises:

[0027] The input feature map is projected into the frequency domain through discrete wavelet transform to obtain four independent frequency domain components, namely low-frequency component LL, horizontal component LH, vertical component HL and high-frequency component HH;

[0028] Capture the contextual information in each frequency domain component and reshape the frequency domain components into vertical features and horizontal features ; is the height of the feature map, is the width of the feature map, is the number of channels of the feature map;

[0029] The vertical feature and horizontal features Multiply them with their own transposed matrices to get the initial attention map and ;

[0030] Multiply the initial attention map and the convolution feature element by element to get the feature map and , and then apply element-by-element summation with the input feature map to obtain the frequency domain features enhanced by bidirectional spectral attention:

[0031] , ;

[0032] Indicates that a convolution operation is performed on the input feature map;

[0033] Using 3×3 convolution, the number of channels of the output feature is expanded to 4C, expressed as: ;

[0034] Four frequency domain features enhanced by bidirectional spectral attention , , and Cascading, we get: ;

[0035] Will and Add, use inverse discrete wavelet transform to project the frequency domain features back to the normal domain; then add the input feature map and the re-projected features to obtain the intermediate features .

[0036] Preferably, the processing of the intermediate features by a position self-attention module and performing pixel summation with the intermediate features and the input features to obtain the output features includes:

[0037] The input features are projected through three 1×1 convolutional layers. In the top branch, the projected features are reshaped and transposed and recorded as ,in ; Reshape the result after projecting the intermediate branch into ;

[0038] Will and The result of matrix multiplication is mapped to (0,1) through the Softmax function to obtain the position-related attention map ;

[0039] Reshape the result after the third branch projection into ,Will and Perform matrix multiplication and reshape the result back The shape of ;

[0040] Will Added to the input features to get the output features.

[0041] Preferably, the data processing process of the cross-fusion module is:

[0042] ;

[0043] in, Indicates encoding blocks and decoding blocks, Indicates The feature map of the encoding block, Indicates The feature map of the decoded block, and The sizes are consistent.

[0044] In a second aspect, the present invention provides a spatial spectral context feature enhancement device for remote sensing image semantic segmentation, which is used to implement the above-mentioned spatial spectral context feature enhancement method for remote sensing image semantic segmentation, and the device includes:

[0045] Data acquisition module, used to acquire remote sensing image data sets and perform preprocessing;

[0046] A deep visual feature optimization neural network model is used to extract remote sensing image features and perform spectral and spatial context enhancement, reconstruct key features, and output semantic segmentation results of the image based on the key features;

[0047] A training module, used to train the deep visual feature optimization neural network model based on the remote sensing image data set, optimize model parameters, and obtain a trained semantic segmentation model;

[0048] The prediction module is used to input the remote sensing image to be tested into the trained semantic segmentation model to obtain the semantic segmentation result of the image.

[0049] Preferably, the deep visual feature optimization neural network model includes:

[0050] The feature extraction module extracts the hierarchical feature map of the input remote sensing image based on the global residual structure;

[0051] A collaborative attention module is used to perform spectral and spatial context enhancement on the extracted feature graph and reconstruct key features; the collaborative attention module includes a spectral attention module and a spatial position self-attention module, the spectral attention module is used to perform deep feature extraction on the input image based on a bidirectional spectral attention mechanism, and perform frequency domain feature cascade to obtain intermediate features; the spatial position self-attention module is used to enhance the spectral features of the frequency domain component and refine the output features using position-related context;

[0052] The network coding module is used to compress and encode the features before entering the decoding stage;

[0053] Cross-fusion module, used to bridge the feature information of encoding and decoding stages;

[0054] The result output module is used to map the features of each pixel to the corresponding category label, achieve accurate pixel-level classification, and output semantic segmentation results.

[0055] Preferably, the spectrum attention module is specifically used to:

[0056] The input feature map is projected into the frequency domain through discrete wavelet transform to obtain four independent frequency domain components, namely low-frequency component LL, horizontal component LH, vertical component HL and high-frequency component HH;

[0057] Capture the contextual information in each frequency domain component and reshape the frequency domain components into vertical features and horizontal features ; is the height of the feature map, is the width of the feature map, is the number of channels of the feature map;

[0058] The vertical feature and horizontal features Multiply them with their own transposed matrices to get the initial attention map and ;

[0059] Multiply the initial attention map and the convolution feature element by element to get the feature map and , and then apply element-by-element summation with the input feature map to obtain the frequency domain features enhanced by bidirectional spectral attention:

[0060] , ;

[0061] Indicates that a convolution operation is performed on the input feature map;

[0062] Using 3×3 convolution, the number of channels of the output feature is expanded to 4C, expressed as: ;

[0063] Four frequency domain features enhanced by bidirectional spectral attention , , and Cascading, we get: ;

[0064] Will and Add, use inverse discrete wavelet transform to project the frequency domain features back to the normal domain; then add the input feature map and the re-projected features to obtain the intermediate features ;

[0065] The spatial position self-attention module is specifically used to:

[0066] The input features are projected through three 1×1 convolutional layers. In the top branch, the projected features are reshaped and transposed and recorded as ,in ; Reshape the result after projecting the intermediate branch into ;

[0067] Will and The result of matrix multiplication is mapped to (0,1) through the Softmax function to obtain the position-related attention map ;

[0068] Reshape the result after the third branch projection into ,Will and Perform matrix multiplication and reshape the result back The shape of ;

[0069] Will Added to the input features to get the output features.

[0070] The beneficial effects of the present invention are:

[0071] The present invention significantly improves the segmentation performance of remote sensing images by introducing a collaborative attention module and a cross-fusion module, while retaining high-frequency details and improving the accuracy of semantic boundaries. The design of the spectral-spatial context network not only overcomes the limitations of traditional methods in processing complex images, but also enhances the robustness and adaptability of the model by comprehensively modeling spectral and spatial context, thereby greatly improving the effect of semantic segmentation of remote sensing images. BRIEF DESCRIPTION OF THE DRAWINGS

[0072] Figure 1 It is a schematic diagram of the structure of the deep visual feature optimization neural network model provided by the present invention;

[0073] Figure 2 is a schematic diagram of the structure of the collaborative attention module provided by the present invention;

[0074] Figure 3 It is a schematic diagram of the structure of the frequency domain spectral attention module provided by the present invention;

[0075] Figure 4 It is a schematic diagram of the structure of the bidirectional spectral attention module provided by the present invention;

[0076] Figure 5 It is a schematic diagram of the structure of the spatial position self-attention module provided by the present invention;

[0077] Figure 6 It is a schematic diagram of the cross-fusion module structure provided by the present invention. DETAILED DESCRIPTION

[0078] The specific implementation of the present invention will be further described in detail below in conjunction with the accompanying drawings and embodiments. The embodiments described below with reference to the accompanying drawings are illustrative and intended to be used to explain the present invention, but should not be construed as limiting the present invention.

[0079] It should be noted that the term "one embodiment" or "embodiment" of the present invention refers to a specific feature, structure or characteristic that can be included in at least one implementation of the present invention. The term "in one embodiment" that appears in different places in this specification does not refer to the same embodiment, nor is it a separate or selective embodiment that is mutually exclusive with other embodiments.

[0080] It should be emphasized here that the step marks mentioned below are not intended to limit the order of the steps, but it should be understood that the steps can be executed in the order mentioned in the embodiment, or in a different order from the embodiment, or several steps can be executed simultaneously.

[0081] Example 1

[0082] This embodiment 1 provides a spatial spectral context feature enhancement method for remote sensing image semantic segmentation. Figure 1 , the method comprising:

[0083] (1) Obtain remote sensing image datasets and perform preprocessing;

[0084] (2) Construct a deep visual feature optimization neural network model based on an encoder-decoder architecture;

[0085] (3) Input the preprocessed remote sensing image dataset into the constructed deep visual feature optimization neural network model, extract image features and perform spectral and spatial context enhancement, reconstruct key features, and output the semantic segmentation results of the image based on the key features;

[0086] (4) Calculate the loss value based on the model output results and the true label, and update the model parameters;

[0087] (5) Perform iterative training in the above manner until the maximum number of iterations is reached, save the optimal model parameters, and obtain a trained semantic segmentation model;

[0088] (6) Input the remote sensing image to be tested into the trained semantic segmentation model to obtain the semantic segmentation result of the image.

[0089] In this embodiment, the Potsdam semantic segmentation dataset is selected as the benchmark. The Potsdam dataset is provided by ISPRS (International Society for Photogrammetry and Remote Sensing). It covers a wide area of ​​Potsdam, Germany, and mainly includes six categories: background, car, tree, low, vegetation, building, and impervious surfaces.

[0090] In this embodiment, the remote sensing image data set is divided into a training data set and a test data set, and the training data set and the test data set are preprocessed before each iteration to enhance the generalization performance of the model.

[0091] Specifically, preprocessing refers to performing random cropping, random horizontal sliding, random flipping, rotation, and color enhancement operations before each iteration to enhance the generalization performance of the model.

[0092] In this embodiment, the maximum number of iterations is set to 500 times.

[0093] In this embodiment, the deep visual feature optimization neural network model based on the encoder-decoder architecture includes a feature extraction module, multiple collaborative attention modules, four cross-fusion modules, a network encoding module and a result output module.

[0094] Specifically, the feature extraction module extracts hierarchical features based on the global residual structure. The feature extraction module enables shallow features in the convolutional neural network model to be transferred to deeper network layers through global residual connections, which can effectively avoid gradient explosion or gradient disappearance.

[0095] The SYnergetic Attention (SYA) module is the core component of SSCBNet, which consists of two key sub-modules: the Frequent Spectral Attention (FSA) module and the Position-wise Self-Attention (PSA) module. These two modules are complementary in function and work together to improve the network's ability to process complex spectral and spatial contextual information.

[0096] In the internal structure of the SYA module, the residual connection between FSA and PSA is designed quite ingeniously. This connection adopts a position-dependent summation method, which aims to maximize the retention of original feature information while giving the network the ability to learn and emphasize features that are critical to specific tasks. The SYA module also further enhances the expressiveness of features through global residual connections.

[0097] In FSA, the conversion process of feature maps is particularly important. The feature map is first converted to the frequency domain in order to more effectively represent and embed spectral context information in the frequency domain. This conversion is achieved through a mechanism called Dual Directional Spectral Attention (DDSA).

[0098] The collaborative attention module comprehensively models the spectral-spatial context by continuously deploying spectral attention and spatial position self-attention, thereby supplementing the ratio of effective information in the features. The collaborative attention module includes a spectral attention module and a spatial position self-attention module. The spectral attention module is used to extract the four frequency components of the input feature map based on the bidirectional spectral attention mechanism, and perform deep feature extraction on the four frequency components, capturing the intrinsic information of the image from different frequency domain perspectives; the spatial position self-attention module is used to enhance the spectral features of the frequency domain components and refine the output features using position-related context.

[0099] The network coding module is used to compress and encode features more efficiently before entering the decoding stage.

[0100] The cross-fusion module is used to efficiently bridge the feature information of the encoding and decoding stages, achieving efficient feature transmission and significantly reducing information loss in the feature map recovery process.

[0101] The result output module is used to map the features of each pixel to its corresponding category label to achieve accurate pixel-level classification, thereby outputting high-quality semantic segmentation results.

[0102] Furthermore, the collaborative attention module performs spectral and spatial context enhancement and reconstructs key features. The specific processing process is as follows:

[0103] (21) Input feature map Through discrete wavelet transform, it is projected into the frequency domain to obtain four independent frequency domain components, including low-frequency component LL, horizontal component LH, vertical component HL and high-frequency component HH. The size of the frequency domain component characteristics is , is the height of the feature map, is the width of the feature map, is the number of channels of the feature map.

[0104] (22) The four frequency domain components are fed into the bidirectional spectral attention module respectively to capture the contextual information in each frequency domain component and reshape the feature map into a vertical feature map. and horizontal features .

[0105] (23) Vertical features and horizontal features Multiply them with their own transposed matrices to get the initial attention map and , , .in, refers to the feature after reshaping along the horizontal axis, that is, Can be considered as The shape is The matrices are stacked. The transpose of The shape is Therefore, the initial attention map calculation can be expressed as Second-rate Matrix and Matrix multiplication of .

[0106] (24) Multiply the initial attention map and the convolution feature element by element to obtain the feature map and , we apply element-wise summation to obtain the output features of the bidirectional spectral attention module:

[0107] , ,

[0108] Represents a convolution operation on the input feature map of the bidirectional spectral attention module.

[0109] By using bidirectional spectral attention for deep feature extraction, the spectral information in the feature map is captured and enhanced more finely.

[0110] (25) Use a 3×3 convolutional layer to output features The number of channels is expanded to 4C, resulting in a more comprehensive and detailed feature map .

[0111] (26) The four frequency domain features enhanced by bidirectional spectral attention are , , and Perform cascade operations to form a more comprehensive and comprehensive feature set: ,in .

[0112] (27) and Add, and use the inverse discrete wavelet transform to project the frequency domain features back into the normal domain. Then the input features Add to the reprojected features to get the output features .

[0113] (28) In the spatial position self-attention module, the input feature Projection is performed through three 1×1 convolutional layers. In the top branch, the projected features are reshaped and transposed and recorded as ,in , and similarly we can get the intermediate branch features .Will and The result of matrix multiplication is mapped to (0,1) through the Softmax function to obtain the position-related attention map ; Reshape the result after the third branch projection into , and Perform the matrix multiplication and reshape the result back The shape of .at last With input features Add together to get the feature map after PSA refinement .

[0114] (29) The result is mapped to (0,1) through the Softmax function, and the features of each pixel are mapped to the corresponding category label to achieve accurate pixel-level classification and output high-quality segmentation results.

[0115] Furthermore, the working principle of the cross-fusion module (CFM) is to accurately calculate the attention weights based on the similarity measurement between the query component and the key component. This process is completed through highly complex mathematical operations, ensuring the accuracy and rationality of the weight distribution. These carefully calculated attention weights are then used to accurately extract key information from the value component. This process realizes the precise interaction and deep fusion of contextual information, greatly improving the efficiency and accuracy of information processing.

[0116] The data processing process of the cross-fusion module is as follows:

[0117] (31) The attention weight is calculated based on the similarity measure between the query value component (Value) and the key component (Key).

[0118] (32) Use attention weights to accurately extract key information in the value component.

[0119] (33) is obtained through residual connection .

[0120] The cross-fusion module can be formally expressed as follows:

[0121] ;

[0122] in, Indicates encoding blocks and decoding blocks, Indicates The feature map of the encoding block, Indicates The feature map of the decoded block, and The sizes are consistent.

[0123] Example 2

[0124] Based on the same inventive concept as that of Embodiment 1, Embodiment 2 provides a deep visual feature optimization device based on spatial spectral context enhancement, the device comprising:

[0125] Data acquisition module, used to acquire remote sensing image data sets and perform preprocessing;

[0126] A deep visual feature optimization neural network model is used to extract remote sensing image features and perform spectral and spatial context enhancement, reconstruct key features, and output semantic segmentation results of the image based on the key features;

[0127] A training module, used to train the deep visual feature optimization neural network model based on a remote sensing image data set, optimize model parameters, and obtain a trained semantic segmentation model;

[0128] The prediction module is used to input the remote sensing image to be tested into the trained semantic segmentation model to obtain the semantic segmentation result of the image.

[0129] In this embodiment, the deep visual feature optimization neural network model includes a feature extraction module, multiple collaborative attention modules, four cross-fusion modules, a network encoding module and a result output module;

[0130] Specifically,

[0131] The feature extraction module extracts the hierarchical features of the input remote sensing image based on the global residual structure.

[0132] The collaborative attention module is used to significantly enhance the network's sensitivity to spectral and spatial context information by integrating spectral attention and position self-attention mechanisms. The collaborative attention module includes a spectral attention module and a spatial position self-attention module.

[0133] The frequency-domain spectral attention module is used to perform in-depth feature extraction of four frequency-domain components based on the bidirectional spectral attention mechanism. It captures the intrinsic information of the image from different frequency-domain perspectives and performs frequency-domain feature cascade to obtain reconstructed features.

[0134] The spatial position self-attention module is used to enhance the spectral features of the frequency domain components and refine the output features using position-related context.

[0135] Network coding module: used to compress and encode features more efficiently before entering the decoding stage.

[0136] Cross-fusion module: used to efficiently bridge the feature information of the encoding and decoding stages, significantly reducing the information loss in the feature map recovery process.

[0137] Result output module: used to map the features of each pixel to its corresponding category label to achieve accurate pixel-level classification, thereby outputting high-quality semantic segmentation results.

[0138] See also Figure 2In this embodiment, the collaborative attention module processes the input feature through the spectral attention module and performs pixel summation with the input feature to obtain the intermediate feature. The intermediate feature is processed by the position self-attention module and performs pixel summation with the intermediate feature and the input feature to obtain the output feature.

[0139] See also Figure 3 In this embodiment, the data processing process of the spectrum attention module is:

[0140] Let the input feature map Through discrete wavelet transform, it is projected into the frequency domain to obtain four independent frequency domain components, including low-frequency component LL, horizontal component LH, vertical component HL and high-frequency component HH. The size of the frequency domain component characteristics is .in, Feature map extracted by feature extraction module;

[0141] The four frequency domain components are sent to the bidirectional spectral attention (DDSA) module for deep feature extraction to obtain the frequency domain features enhanced by bidirectional spectral attention. , , and ;

[0142] Use a 3×3 convolutional layer to output features The number of channels is expanded to 4C, resulting in a more comprehensive and detailed feature map .

[0143] Four frequency domain features enhanced by bidirectional spectral attention , , and Perform cascade operations to form a more comprehensive and comprehensive feature set: ,in ;

[0144] Will and Add, use the inverse discrete wavelet transform to project the frequency domain features back into the normal domain, then add the input feature map to the reprojected features to get the output features .

[0145] See also Figure 4 In this embodiment, the data processing process of the bidirectional spectral attention module is:

[0146] The four frequency domain components are fed into the bidirectional spectral attention module respectively to capture the contextual information in each frequency domain component and reshape the frequency domain components into vertical features. and horizontal features ;

[0147] Vertical feature and horizontal features Multiply them with their own transposed matrices to get the initial attention map and ;

[0148] Multiply the initial attention map and the convolution feature element by element to get the feature map and , we apply element-wise summation to obtain the output features of the bidirectional spectral attention module: , .

[0149] By using bidirectional spectral attention for deep feature extraction, the spectral information in the feature map is captured and enhanced more finely.

[0150] See also Figure 5 In this embodiment, the data processing process of the spatial position self-attention module is:

[0151] The input features are projected through three 1×1 convolutional layers. In the top branch, the projected features are reshaped and transposed and recorded as ,in , and similarly we can get the intermediate branch features ;

[0152] Will and The result of matrix multiplication is mapped to (0,1) through the Softmax function to obtain the position-related attention map ; Reshape the result after the third branch projection into , and Perform the matrix multiplication and reshape the result back The shape of ;at last With input features Add together to get the feature map after PSA refinement .

[0153] See also Figure 6 In this embodiment, the data processing process of the cross-fusion module is as follows:

[0154] Calculate the attention weight based on the similarity measure between the query value component (Value) and the key component (Key);

[0155] Use attention weights to accurately extract key information in the value component;

[0156] Through residual connection ;

[0157] The cross-fusion module can be formally expressed as follows:

[0158] ;

[0159] in, Indicates encoding blocks and decoding blocks, Indicates The feature map of the encoding block, Indicates The feature map of the decoded block, and The sizes are consistent.

[0160] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.

[0161] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0162] These computer program instructions may also be stored in a computer readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0163] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0164] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the above embodiments, ordinary technicians in the relevant field should understand that the specific implementation methods of the present invention can still be modified or replaced by equivalents. Any modification or equivalent replacement that does not depart from the spirit and scope of the present invention should be covered within the scope of protection of the claims of the present invention.

Claims

1. A spatial spectral context feature enhancement method for remote sensing image semantic segmentation, characterized in that: include: Acquire remote sensing image datasets and perform preprocessing; Build a deep visual feature optimization neural network model based on encoder-decoder architecture; Inputting the preprocessed remote sensing image data set into the deep visual feature optimization neural network model, extracting image features and performing spectral and spatial context enhancement, reconstructing key features, and outputting semantic segmentation results of the image based on the key features; Calculate the loss value based on the output semantic segmentation result and the true label, update the model parameters, iterate the training until the end, save the optimal model parameters, and obtain a trained semantic segmentation model; The remote sensing image to be tested is input into the trained semantic segmentation model to obtain the semantic segmentation result of the image.

2. The method for enhancing spatial spectral context features for semantic segmentation of remote sensing images according to claim 1, characterized in that: The step of obtaining a remote sensing image data set and performing preprocessing includes: Acquire a remote sensing image data set and divide it into a training data set and a test data set, and preprocess the training data set and the test data set before each iteration; The preprocessing refers to performing random cropping, random horizontal sliding, random flipping, rotation and color enhancement operations on the training data set and the test data set.

3. The method for enhancing spatial spectral context features for semantic segmentation of remote sensing images according to claim 2, characterized in that: The deep visual feature optimization neural network model based on the encoder-decoder architecture includes: A feature extraction module is used to extract a hierarchical feature map of an input remote sensing image; A collaborative attention module is used to enhance the spectral and spatial context of the extracted feature maps and reconstruct key features; The network coding module is used to compress and encode the features before entering the decoding stage; Cross-fusion module, used to fuse encoding features and decoding features; and, The result output module is used to output the semantic segmentation results.

4. The method for enhancing spatial spectral context features for semantic segmentation of remote sensing images according to claim 3, characterized in that: The data processing process of the collaborative attention module is: The input features are processed by the spectral attention module and the intermediate features are obtained by pixel summing with the input features; The intermediate features are processed by the position self-attention module, and the output features are obtained by pixel summing with the intermediate features and input features.

5. The method for enhancing spatial spectral context features for semantic segmentation of remote sensing images according to claim 4, characterized in that: The input feature is processed by the spectrum attention module and pixel summed with the input feature to obtain the intermediate feature, including: The input feature map is projected into the frequency domain through discrete wavelet transform to obtain four independent frequency domain components, namely low-frequency component LL, horizontal component LH, vertical component HL and high-frequency component HH; Capture the contextual information in each frequency domain component and reshape the frequency domain components into vertical features and horizontal features ; is the height of the feature map, is the width of the feature map, is the number of channels of the feature map; The vertical feature and horizontal features Multiply them with their own transposed matrices to get the initial attention map and ; Multiply the initial attention map and the convolution feature element by element to get the feature map and , and then apply element-by-element summation with the input feature map to obtain the frequency domain features enhanced by bidirectional spectral attention: ， ； Indicates that a convolution operation is performed on the input feature map; Using 3×3 convolution, the number of channels of the output feature is expanded to 4C, expressed as: ; Four frequency domain features enhanced by bidirectional spectral attention , , and Cascading, we get: ; Will and Add, use inverse discrete wavelet transform to project the frequency domain features back to the normal domain; then add the input feature map and the re-projected features to obtain the intermediate features .

6. The method for enhancing spatial spectral context features for semantic segmentation of remote sensing images according to claim 5, characterized in that: The intermediate features are processed by the position self-attention module, and the output features are obtained by pixel summing with the intermediate features and the input features, including: The input features are projected through three 1×1 convolutional layers. In the top branch, the projected features are reshaped and transposed and recorded as ,in ; Reshape the result after projecting the intermediate branch into ; Will and The result of matrix multiplication is mapped to (0,1) through the Softmax function to obtain the position-related attention map ; Reshape the result after the third branch projection into ,Will and Perform matrix multiplication and reshape the result back The shape of ; Will Added to the input features to get the output features.

7. The method for enhancing spatial spectral context features for semantic segmentation of remote sensing images according to claim 6, characterized in that: The data processing process of the cross-fusion module is as follows: ； in, Indicates encoding blocks and decoding blocks, Indicates The feature map of the encoding block, Indicates The feature map of the decoded block, and The sizes are consistent.

8. A spatial spectral context feature enhancement device for remote sensing image semantic segmentation, characterized in that: The device is used to implement the spatial spectral context feature enhancement method for remote sensing image semantic segmentation according to any one of claims 1 to 7, comprising: Data acquisition module, used to acquire remote sensing image data sets and perform preprocessing; A deep visual feature optimization neural network model is used to extract remote sensing image features and perform spectral and spatial context enhancement, reconstruct key features, and output semantic segmentation results of the image based on the key features; A training module, used to train the deep visual feature optimization neural network model based on the remote sensing image data set, optimize model parameters, and obtain a trained semantic segmentation model; The prediction module is used to input the remote sensing image to be tested into the trained semantic segmentation model to obtain the semantic segmentation result of the image.

9. The spatial spectral context feature enhancement device for remote sensing image semantic segmentation according to claim 8, characterized in that: The deep visual feature optimization neural network model includes: The feature extraction module extracts the hierarchical feature map of the input remote sensing image based on the global residual structure; A collaborative attention module is used to perform spectral and spatial context enhancement on the extracted feature graph and reconstruct key features; the collaborative attention module includes a spectral attention module and a spatial position self-attention module, the spectral attention module is used to perform deep feature extraction on the input image based on a bidirectional spectral attention mechanism, and perform frequency domain feature cascade to obtain intermediate features; the spatial position self-attention module is used to enhance the spectral features of the frequency domain component and refine the output features using position-related context; The network coding module is used to compress and encode the features before entering the decoding stage; Cross-fusion module, used to bridge the feature information of encoding and decoding stages; The result output module is used to map the features of each pixel to the corresponding category label, achieve accurate pixel-level classification, and output semantic segmentation results.

10. The spatial spectral context feature enhancement device for remote sensing image semantic segmentation according to claim 8, characterized in that: The spectrum attention module is specifically used to: The input feature map is projected into the frequency domain through discrete wavelet transform to obtain four independent frequency domain components, namely low-frequency component LL, horizontal component LH, vertical component HL and high-frequency component HH; Capture the contextual information in each frequency domain component and reshape the frequency domain components into vertical features and horizontal features ; is the height of the feature map, is the width of the feature map, is the number of channels of the feature map; The vertical feature and horizontal features Multiply them with their own transposed matrices to get the initial attention map and ; Multiply the initial attention map and the convolution feature element by element to get the feature map and , and then apply element-by-element summation with the input feature map to obtain the frequency domain features enhanced by bidirectional spectral attention: ， ； Indicates that a convolution operation is performed on the input feature map; Using 3×3 convolution, the number of channels of the output feature is expanded to 4C, expressed as: ; Four frequency domain features enhanced by bidirectional spectral attention , , and Cascading, we get: ; Will and Add, use inverse discrete wavelet transform to project the frequency domain features back to the normal domain; then add the input feature map and the re-projected features to obtain the intermediate features ; The spatial position self-attention module is specifically used to: The input features are projected through three 1×1 convolutional layers. In the top branch, the projected features are reshaped and transposed and recorded as ,in ; Reshape the result after projecting the intermediate branch into ; Will and The result of matrix multiplication is mapped to (0,1) through the Softmax function to obtain the position-related attention map ; Reshape the result after the third branch projection into ,Will and Perform matrix multiplication and reshape the result back The shape of ; Will Added to the input features to get the output features.

Images