Remote sensing image fine-grained semantic segmentation method and device
By introducing cross-domain coupling layer and cross-level fusion modules into the encoder-decoder network of remote sensing images, the problem that traditional methods are difficult to capture complex terrain fine-grained textures and periodic patterns is solved, and higher semantic segmentation accuracy is achieved.
Patent Information
- Application Number
- CN202411291639.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-09-14
- Publication Date
- 2025-02-11
AI Technical Summary
Traditional remote sensing image semantic segmentation methods are difficult to accurately capture fine-grained textures and periodic patterns in complex terrain, resulting in unsatisfactory semantic segmentation accuracy.
By introducing a cross-domain coupling layer into the encoder-decoder network, frequency domain and spatial domain information are captured, and feature maps of encoder and decoder are aggregated through cross-stage fusion modules, the gap between encoder and decoder is bridged.
Effectively combining frequency and spatial domain features, enhance the discernibility and contextual understanding of the learned feature representation, improve semantic segmentation accuracy, and solve the limitations of capturing fine-grained textures and periodic patterns.
Smart Images
Figure CN119418043A_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the technical field of remote sensing image processing, and in particular relates to a remote sensing image fine-grained semantic segmentation method and device. Background Art
[0002] With the rapid development of remote sensing technology, semantic segmentation technology of remote sensing images has shown irreplaceable importance in key areas such as urban planning, agricultural monitoring and disaster management. By accurately identifying elements such as buildings, roads, vegetation, and water bodies in images, semantic segmentation technology provides detailed surface information. Therefore, how to perform semantic segmentation on remote sensing images is one of the hot research topics in the field.
[0003] Many researchers have conducted extensive research in the field of semantic segmentation of remote sensing images. The diversity of information in images and the complexity of ground features make it difficult for traditional methods based on pixels or simple texture features to accurately segment different categories of targets. Machine learning techniques such as support vector machines and random forests provide better accuracy and flexibility, but still face the challenges of large data sets and extensive feature engineering.
[0004] In recent years, deep learning, especially convolutional neural networks, has transformed the semantic segmentation of remote sensing images, achieving high accuracy and efficiency. These methods mainly rely on convolutional neural networks to extract spatial domain features. Although convolutional neural networks are effective in detecting edges and shapes, they focus on local receptive fields and hierarchical features, often losing subtle textures and repetitive patterns that are critical for distinguishing similar areas. These regions exhibit unique texture patterns—forests with varying tree densities, repetitive crop layouts in fields, and grid-like urban structures—which are critical for accurate land cover classification and environmental monitoring, but cannot be effectively captured by spatial domain models alone, resulting in suboptimal results for complex terrain. Summary of the invention
[0005] The purpose of the present invention is to provide a method and device for fine-grained semantic segmentation of remote sensing images, which improves the learning feature representation in the frequency domain and spatial domain by embedding a cross-domain coupling layer into an encoder-decoder network, and bridges the gap between the encoder and decoder through cross-level fusion, so as to achieve the purpose of accurately extracting fine-grained semantic segmentation targets of remote sensing images.
[0006] To achieve the above object, the technical solution adopted by the present invention is as follows: On one hand, the present invention provides a remote sensing image fine-grained semantic segmentation method, comprising: Acquire a target remote sensing image, wherein the target remote sensing image is a remote sensing image including a target object and a background object; Inputting the target remote sensing image data into a pre-trained fine-grained semantic segmentation model to obtain a fine-grained semantic segmentation result of the target remote sensing image; the fine-grained semantic segmentation result refers to dividing and labeling different objects or regions in the image according to semantic categories to obtain a prediction map containing semantic information; The fine-grained semantic segmentation model is built based on a cross-domain encoder-cross-domain decoder neural network. The cross-domain encoder and the cross-domain decoder are both provided with a cross-domain coupling layer, and the cross-domain coupling layer is used to capture the frequency domain information and spatial domain information of the input feature map to form a feature mapping matrix; the neural network is provided with a cross-level fusion module, and the cross-level fusion module is used to aggregate the feature maps output by the cross-domain encoder and the cross-domain decoder as the input of the next feature output layer of the cross-domain decoder.
[0007] Preferably, the method further comprises: Pre-train a fine-grained semantic segmentation model as follows: Obtain an Earth observation remote sensing image dataset and randomly divide it into a training set and a test set; Constructing a neural network based on a cross-domain encoder-cross-domain decoder, inputting the training set images into the neural network of the cross-domain encoder-cross-domain decoder for training until the model converges, and obtaining a network model with optimal parameters as a fine-grained semantic segmentation model; The neural network training process is as follows: Input the training set image into the cross-domain encoder, and obtain a feature map after convolution, cross-domain coupling layer and maximum pooling processing; The feature map output by the cross-domain encoder and the feature map output by the cross-domain decoder are fused through a cross-level fusion module and used as the input of the next feature output layer of the cross-domain decoder. After convolution, cross-domain coupling layer and convolution processing, the spatial resolution of the feature map is restored to its original size. According to the category with the maximum probability, the semantics of the ground object in the current feature map is determined as the output result of semantic segmentation.
[0008] Preferably, the training set image is input into the cross-domain encoder, and a feature map is obtained after convolution, cross-domain coupling layer and maximum pooling processing, including: Input the cross-domain encoder i The feature map of the feature extraction layer is convolved by 1×1, 3×3 and 1×1 to obtain the feature map E i1 , i =1, 2, 3, 4; where the input of the first feature extraction layer is the original image, i The input of the +1 feature extraction layer is i The output of the feature extraction layer; The feature map Ei1 Input the cross-domain coupling layer to get the feature map E i2 , i =1, 2, 3, 4; The feature map E i2 Use the maximum pooling operation to get the feature map E i , i =1, 2, 3, 4; The feature map E The 5th feature extraction layer of the 4-input cross-domain encoder performs 1×1, 3×3 and 1×1 convolution operations to obtain the feature map E 51 ; The feature map E 51 Input the cross-domain coupling layer to get the feature map E 5.
[0009] Preferably, the step of fusing the feature map output by the cross-domain encoder and the feature map output by the cross-domain decoder through a cross-level fusion module as the input of the next feature output layer of the cross-domain decoder comprises: The feature map output by the cross-domain encoder E i And the feature map output by the cross-domain decoder D i+1 After element-wise multiplication and 1×1 convolution, the cross-domain decoder i The input feature map of the feature output layer, i ∈[1,5), where the input feature map of the fifth feature output layer of the cross-domain decoder is the feature map output by the fifth feature extraction layer of the cross-domain encoder E 5.
[0010] Preferably, the processing process of the feature output layer of the cross-domain decoder is as follows: The first i The feature map of the feature output layer is convolved with 1×1, 3×3 and 1×1 to obtain the feature map D i1 , i ∈[1,5]; The feature map D i1 Input the cross-domain coupling layer to get the feature map D i2 ; The feature map D i2 A 1×1 convolution operation is used to obtain the feature map Di .
[0011] Preferably, the processing process of the cross-domain coupling layer is as follows: The input feature map of the cross-domain coupling layer is used to obtain the frequency domain output domain feature map in parallel F f out And the spatial domain output feature map F s out ; The frequency domain output domain feature map F f out And the spatial domain output feature map F s out The input feature map of the cross-domain coupling layer is concatenated and then a 1×1 convolution is used to obtain the output feature map of the cross-domain coupling layer. F out .
[0012] Preferably, obtain the frequency domain output domain feature map F f out ,include: The input feature map of the cross-domain coupling layer is converted into a frequency domain feature map using a two-dimensional discrete cosine transform F freq ; The frequency domain feature map F freq Three 1×1 convolutions with different kernels are used to reshape the features in the channel dimension: F f1 ，F f2 , F f3 ； The feature map F f1 After transposition and the feature map F f2 Perform element-wise multiplication and normalization with the feature map F f3 Perform element-wise multiplication to obtain the feature map A f ; The feature map A f With the characteristic diagram F freq After addition, it is converted into a frequency domain output domain feature map through inverse discrete cosine transform F f out .
[0013] Preferably, obtain the spatial domain output feature map F s out ,include: The input feature map of the cross-domain coupling layer is convolved by 1×1 to generate a spatial domain feature map F spa ; The feature map F spa Three 1×1 convolutions with different kernels are used to reshape the features in the channel dimension: F s1 , F s2 , F s3 ; The feature map F s1 After transposition and the feature map F s2 Perform element-wise multiplication and normalization with the feature map F s3 Perform element-wise multiplication to obtain the feature map A s ; The feature map A s With the characteristic diagram F spa Add to get the spatial domain output feature map F s out .
[0014] On the other hand, the present invention also provides a remote sensing image fine-grained semantic segmentation device, which is used to implement the above remote sensing image fine-grained semantic segmentation method, and the device comprises: A data acquisition module is used to acquire a target remote sensing image, wherein the target remote sensing image is a remote sensing image including a target object and a background object; A prediction output module is used to input the target remote sensing image data into a fine-grained semantic segmentation model to obtain a fine-grained semantic segmentation result of the target remote sensing image; the fine-grained semantic segmentation result refers to dividing and labeling different objects or regions in the image according to semantic categories to obtain a prediction map containing semantic information; The fine-grained semantic segmentation model is built based on a cross-domain encoder-cross-domain decoder neural network, including: A cross-domain coupling encoder is provided, which is used to encode the remote sensing image and capture the frequency domain information and spatial domain information of the input feature map to form a feature map; A cross-level fusion module, used for aggregating the feature maps output by the cross-domain encoder and the cross-domain decoder as input to the next feature output layer of the cross-domain decoder; The cross-domain coupling decoder sets a cross-domain coupling layer to decode the input feature map, fuse the frequency domain information and the spatial domain information, restore the feature map to the same size as the original input remote sensing image, and obtain the semantic segmentation result.
[0015] Preferably, the device further comprises: Model training module, used to Obtain an Earth observation remote sensing image dataset and randomly divide it into a training set and a test set; Constructing a neural network based on a cross-domain encoder-cross-domain decoder, inputting the training set images into the neural network of the cross-domain encoder-cross-domain decoder for training until the model converges, and obtaining a network model with optimal parameters as a fine-grained semantic segmentation model; The neural network training process is as follows: Input the training set image into the cross-domain encoder, and obtain a feature map after convolution, cross-domain coupling layer and maximum pooling processing; The feature map output by the cross-domain encoder and the feature map output by the cross-domain decoder are fused through a cross-level fusion module and used as the input of the next feature output layer of the cross-domain decoder. After convolution, cross-domain coupling layer and convolution processing, the spatial resolution of the feature map is restored to its original size. According to the category with the maximum probability, the semantics of the ground object in the current feature map is determined as the output result of semantic segmentation.
[0016] The beneficial effects of the present invention are: The present invention constructs a neural network based on a cross-domain encoder-cross-domain decoder, and trains to obtain a fine-grained semantic segmentation model. The neural network replaces ordinary convolution blocks with cross-domain coupling layers to more comprehensively model fine-grained textures and periodic patterns in the frequency domain and edges, shapes, and a wide range of structural elements in the spatial domain, which is used to enhance the discriminability of learned representations; the feature representations of the encoder and decoder are aggregated through a cross-level fusion module to bridge the gap between the encoder and decoder. The present invention can effectively combine frequency domain and spatial domain features, enhance the discriminability and contextual understanding of the learned feature representations, solve the limitations of capturing fine-grained textures and periodic patterns, and improve the accuracy of semantic segmentation. BRIEF DESCRIPTION OF THE DRAWINGS
[0017] Figure 1 It is a fine-grained semantic segmentation model architecture provided by an embodiment of the present invention; Figure 2 This is a cross-domain coupling layer data processing process provided by an embodiment of the present invention; Figure 3is a schematic diagram of LoveDA experimental data and results in an embodiment of the present invention; Figure 4 It is a schematic diagram of ISPRS Potsdam experimental data and results in an embodiment of the present invention. DETAILED DESCRIPTION
[0018] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the embodiments and the accompanying drawings. Here, the illustrative embodiments of the present invention and their descriptions are used to explain the present invention, but are not intended to limit the present invention.
[0019] It should be emphasized here that the step marks mentioned below are not intended to limit the order of the steps, but it should be understood that the steps can be executed in the order mentioned in the embodiment, or in a different order from the embodiment, or several steps can be executed simultaneously.
[0020] Secondly, the term "one embodiment" or "embodiment" as used herein refers to a specific feature, structure or characteristic that may be included in at least one implementation of the present invention. The term "in one embodiment" that appears in different places in this specification does not necessarily refer to the same embodiment, nor does it refer to a separate or selective embodiment that is mutually exclusive with other embodiments.
[0021] The specific embodiments of the present invention are described in further detail below in conjunction with the accompanying drawings. The following examples are used to illustrate the present invention but are not intended to limit the scope of the present invention. Example 1
[0022] This embodiment 1 provides a remote sensing image fine-grained semantic segmentation method, including: Acquire a target remote sensing image, wherein the target remote sensing image is a remote sensing image including a target object and a background object; The target remote sensing image is input into a pre-trained fine-grained semantic segmentation model to obtain a fine-grained semantic segmentation result of the target remote sensing image.
[0023] It should be noted that this embodiment performs semantic segmentation on high-resolution remote sensing images, such as images in the LoveDA and ISPRSPotsdam datasets.
[0024] It should be noted that the fine-grained semantic segmentation result of remote sensing images means that after the remote sensing image semantic segmentation performs in-depth analysis on the remote sensing image, different objects or regions in the image are accurately divided and labeled according to their semantic categories (such as water bodies, forests, urban buildings, roads, farmlands, etc.), generating an image prediction map containing rich semantic information. In the prediction map, objects or regions of the same category are given the same label or color, and different colors represent different types of objects.
[0025] See also Figure 1 In one embodiment, a fine-grained semantic segmentation model is pre-trained, and the specific implementation method is as follows: S1. Construct a neural network based on cross-domain encoder-cross-domain decoder, such as Figure 1 As shown, the neural network includes a cross-domain coupling layer and a cross-level fusion module, wherein the cross-domain coupling layer simultaneously captures the context information of the frequency domain and the spatial domain in a parallel manner for fusion, and the cross-level fusion module aggregates the feature representations of the encoder and the decoder; S2, obtain the earth observation remote sensing image data set, and randomly divide it into training set and test set according to a fixed ratio; S3, input the training set images into the constructed neural network for training until the model converges, and obtain the network model with the optimal parameters as the fine-grained semantic segmentation model; S4. Use the test set to verify the performance of the constructed fine-grained semantic segmentation model.
[0026] For details, see the neural network training process. Figure 1 , as follows: S31, let the original remote sensing image obtained by input be F ∈ H×W×C .in, H, W, C are the height, width and number of channels of the input image respectively; The training set image is input into the cross-domain encoder and then subjected to 1×1, 3×3 and 1×1 convolution blocks, cross-domain coupling layer and maximum pooling operations to obtain the feature map. E i , and E i As i+ 1 Input of the feature extraction layer, E i ∈ H i ×W i ×C i .in H i , W i , C i Represents the cross-domain encoder i The height, width and number of channels of the feature map output by the feature extraction layer, i ∈[1,5); It should be noted that the feature map of the fifth feature extraction layer of the input cross-domain decoder E 4 feature maps are obtained through 1×1, 3×3, 1×1 convolution blocks and cross-domain coupling layers E 5.
[0027] The specific refinement process is: Input the cross-domain encoder i The feature map of the feature extraction layer is convolved by 1×1, 3×3 and 1×1 to obtain the feature map E i1 , i =1, 2, 3, 4; where the input of the first feature extraction layer is the original image, i The input of the +1 feature extraction layer is i The output of the feature extraction layer; The feature map E i1 Input the cross-domain coupling layer to get the feature map E i2 , i =1, 2, 3, 4; The feature map E i2 Use the maximum pooling operation to get the feature map E i , i =1, 2, 3, 4; The feature map E The 5th feature extraction layer of the 4-input cross-domain encoder performs 1×1, 3×3 and 1×1 convolution operations to obtain the feature map E 51 ; The feature map E 51 Input the cross-domain coupling layer to get the feature map E 5.
[0028] S32, the feature map output by the cross-domain encoder E i And the feature map output by the cross-domain decoder D i+1 Input the cross-level fusion module to obtain the cross-domain decoder i The input feature map of the feature output layer D in ,in i ∈[1,5).
[0029] Among them, the cross-level fusion module combines the feature maps output by the cross-domain encoder E i And the feature map output by the cross-domain decoder D i+1 Element-wise multiplication and 1×1 convolution are performed as the input feature map of the next feature output layer of the cross-domain decoder. D in ,in i ∈[1,5), expressed as Din =Conv 1×1 ( Concat ( E i ，D i+1 )).
[0030] It should be noted that the input feature map input to the fifth feature output layer of the cross-domain decoder is the feature map output by the fifth feature extraction layer of the cross-domain encoder. E 5.
[0031] S33, feature map D in The input cross-domain decoder restores the spatial resolution to its original size through 1×1, 3×3 and 1×1 convolution blocks, cross-domain coupling layers and 1×1 convolution operations, and determines the semantics of the object in the current feature map according to the category with the maximum probability as the output result of semantic segmentation.
[0032] The specific processing process of the feature output layer of the cross-domain decoder is as follows: Input the cross-domain decoder i The feature map of the feature output layer D in Perform 1×1, 3×3, and 1×1 convolution operations to obtain feature maps D i1 , i ∈[1,5); The feature map D i1 Input the cross-domain coupling layer to get the feature map D i2 ; The feature map D i2 A 1×1 convolution operation is used to obtain the feature map D i .
[0033] It should be noted that this is only used after the first cross-domain encoder outputs the feature map, that is, after the spatial resolution of the feature map has been restored to the original size of the input image. Softmax Determine the semantics of the object in the current feature map as the output result of semantic segmentation.
[0034] join Figure 2 , the detailed process of extracting frequency domain information and spatial domain information through the cross-domain coupling layer is as follows: S311, the input image is subjected to 1×1, 3×3 and 1×1 convolutional block operations to obtain the input feature map of the cross-domain coupling layer F in , the feature map is transformed intoF in From spatial domain to frequency domain feature map F freq , expressed as: F freq =2DDCT(F in ） ; The feature map F freq Three 1×1 convolutions with different kernels are used to reshape the feature representation in the channel dimension. F f1 ， F f2 , F f3 ; The feature map F f1 After transposition and feature map F f2 Perform element-wise multiplication and normalization with the feature map F f3 Perform element-wise multiplication to obtain the feature map A f , expressed as: A f =Softmax(F f1 T ×F f2 ）×F f3 ; The feature map A f With feature map F freq After addition, the frequency domain features are converted into spatial features through inverse discrete cosine transform F f out , get the frequency domain output domain feature map F f out , expressed as: F f out =iDCT(A f +F freq ） .
[0035] S312: The input image is subjected to 1×1, 3×3 and 1×1 convolutional block operations to obtain an input feature map of the cross-domain coupling layer. F in , 1×1 convolution is used to extract spatial domain features, which can be expressed as: Fspa =Conv 1×1 （F in ); The feature map F spa Three 1×1 convolutions with different kernels are used to reshape the feature representation in the channel dimension. F s1 , F s2 , F s3 ; The feature map F s1 After transposition and feature map F s2 Perform element-wise multiplication and normalization with the feature map F s3 Perform element-wise multiplication to obtain the feature map A s , expressed as: A s =Softmax(F s1 T ×F s2 ）×F s3 ; The feature map A s With feature map F spa Add to get the spatial domain output feature map F s out , expressed as: F s out =A s +F spa .
[0036] S313, output the frequency domain feature map F f out , spatial domain output feature map F s out as well as F in The output feature map of the cross-domain coupling layer is obtained by concatenating and performing a 1×1 convolution operation. F out , expressed as: F out =Conv 1×1 (Concat ( F in ，F f out ， F s out )).
[0037] It should be noted that the operation of the cross-domain coupling layer in S33 is consistent with the operation of the cross-domain coupling layer in S31, so it will not be repeated here.
[0038] In one embodiment, the cross-domain coupled decoder restores the spatial resolution of the input feature map to its original size, and determines the object semantics of the current feature map according to the category with the maximum probability as the output result of semantic segmentation. Example 2
[0039] Based on the same inventive concept as the above-mentioned embodiment 1, this embodiment 2 provides a remote sensing image fine-grained semantic segmentation device, which is used to implement the above-mentioned remote sensing image fine-grained semantic segmentation method, and the device includes: A data acquisition module is used to acquire a target remote sensing image, wherein the target remote sensing image is a remote sensing image including a target object and a background object; A prediction output module is used to input the target remote sensing image data into a fine-grained semantic segmentation model to obtain a fine-grained semantic segmentation result of the target remote sensing image; the fine-grained semantic segmentation result refers to dividing and labeling different objects or regions in the image according to semantic categories to obtain a prediction map containing semantic information; The fine-grained semantic segmentation model is built based on a cross-domain encoder-cross-domain decoder neural network, including: A cross-domain coupling encoder is provided, which is used to encode the remote sensing image and capture the frequency domain information and spatial domain information of the input feature map to form a feature map; A cross-level fusion module, used for aggregating the feature maps output by the cross-domain encoder and the cross-domain decoder as input to the next feature output layer of the cross-domain decoder; The cross-domain coupling decoder sets a cross-domain coupling layer to decode the input feature map, fuse the frequency domain information and the spatial domain information, restore the feature map to the same size as the original input remote sensing image, and obtain the semantic segmentation result.
[0040] In one embodiment, the device further comprises: a model training module, for: Obtain an Earth observation remote sensing image dataset and randomly divide it into a training set and a test set; Constructing a neural network based on a cross-domain encoder-cross-domain decoder, inputting the training set images into the neural network of the cross-domain encoder-cross-domain decoder for training until the model converges, and obtaining a network model with optimal parameters as a fine-grained semantic segmentation model; The neural network training process is as follows: The training set images are input into the cross-domain encoder, and the feature maps are obtained after convolution, cross-domain coupling layer and maximum pooling processing; The feature map output by the cross-domain encoder and the feature map output by the cross-domain decoder are fused through the cross-level fusion module and used as the input of the next feature output layer of the cross-domain decoder. After convolution, cross-domain coupling layer and convolution processing, the spatial resolution of the feature map is restored to its original size. According to the category with the maximum probability, the semantics of the object in the current feature map is determined as the output result of semantic segmentation.
[0041] It is worth pointing out that the device embodiment corresponds to the above-mentioned method embodiment, and the implementation methods of the above-mentioned method embodiments are all applicable to the device embodiment and can achieve the same or similar technical effects, so they will not be repeated here. Example 3
[0042] This embodiment uses two different data sets, namely the LoveDA satellite remote sensing image data set and the ISPRSPotsdam UAV remote sensing image data set. The method of Example 1 is used for semantic segmentation. The final semantic segmentation result is as follows: Figure 3 and Figure 4 As shown. It can be seen that the method of the embodiment of the present invention can be applied to satellite remote sensing images and UAV remote sensing images, which verifies that the method of domain-specific extraction and cross-domain coupling effectively combines frequency domain and spatial domain features. The fine-grained textures and periodic patterns represented in the frequency domain and the edges, shapes and extensive structural elements represented in the spatial domain are well modeled and used for pixel semantic reasoning, which enhances the discriminability and contextual understanding of the learned representation. Example 4
[0043] This embodiment provides a computing device, including a processor and a computer program stored in a memory and executable on the processor, wherein when the processor executes the program, a remote sensing image semantic segmentation method in the above-mentioned embodiment 1 is implemented.
[0044] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.
[0045] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.
[0046] These computer program instructions may also be stored in a computer readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.
[0047] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.
[0048] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that the specific implementation modes of the present invention can still be modified or replaced by equivalents. Any modification or equivalent replacement that does not depart from the spirit and scope of the present invention should be included in the scope of protection of the claims of the present invention.
Claims
1. A remote sensing image fine-grained semantic segmentation method, characterized in that: include: Acquire a target remote sensing image, wherein the target remote sensing image is a remote sensing image including a target object and a background object; Inputting the target remote sensing image data into a pre-trained fine-grained semantic segmentation model to obtain a fine-grained semantic segmentation result of the target remote sensing image; the fine-grained semantic segmentation result refers to dividing and labeling different objects or regions in the image according to semantic categories to obtain a prediction map containing semantic information; The fine-grained semantic segmentation model is built based on a neural network of a cross-domain encoder and a cross-domain decoder, and both the cross-domain encoder and the cross-domain decoder are provided with a cross-domain coupling layer, and the cross-domain coupling layer is used to capture the frequency domain information and the spatial domain information of the input feature map to form a feature mapping matrix; The neural network is provided with a cross-level fusion module, and the cross-level fusion module is used to aggregate the feature maps output by the cross-domain encoder and the cross-domain decoder as the input of the next feature output layer of the cross-domain decoder.
2. The remote sensing image fine-grained semantic segmentation method according to claim 1, characterized in that: The method further comprises, Pre-train a fine-grained semantic segmentation model as follows: Obtain an Earth observation remote sensing image dataset and randomly divide it into a training set and a test set; Constructing a neural network based on a cross-domain encoder-cross-domain decoder, inputting the training set images into the neural network of the cross-domain encoder-cross-domain decoder for training until the model converges, and obtaining a network model with optimal parameters as a fine-grained semantic segmentation model; The neural network training process is as follows: Input the training set image into the cross-domain encoder, and obtain a feature map after convolution, cross-domain coupling layer and maximum pooling processing; The feature map output by the cross-domain encoder and the feature map output by the cross-domain decoder are fused through a cross-level fusion module and used as the input of the next feature output layer of the cross-domain decoder. After convolution, cross-domain coupling layer and convolution processing, the spatial resolution of the feature map is restored to its original size. According to the category with the maximum probability, the semantics of the ground object in the current feature map is determined as the output result of semantic segmentation.
3. The remote sensing image fine-grained semantic segmentation method according to claim 2, characterized in that: The training set image is input into the cross-domain encoder, and a feature map is obtained after convolution, cross-domain coupling layer and maximum pooling processing, including: Input the cross-domain encoder i The feature map of the feature extraction layer is convolved by 1×1, 3×3 and 1×1 to obtain the feature map E i1 , i =1, 2, 3, 4; where the input of the first feature extraction layer is the original image, i The input of the +1 feature extraction layer is i The output of the feature extraction layer; The feature map E i1 Input cross-domain coupling layer to get feature map E i2 , i =1, 2, 3, 4; The feature map E i2 Use the maximum pooling operation to get the feature map E i , i =1, 2, 3, 4; The feature map E The 5th feature extraction layer of the 4-input cross-domain encoder performs 1×1, 3×3 and 1×1 convolution operations to obtain the feature map E 51 ; The feature map E 51 Input the cross-domain coupling layer to get the feature map E 5.
4. The remote sensing image fine-grained semantic segmentation method according to claim 3, characterized in that: The step of fusing the feature map output by the cross-domain encoder and the feature map output by the cross-domain decoder through a cross-level fusion module as input to a next feature output layer of the cross-domain decoder includes: The feature map output by the cross-domain encoder E i And the feature map output by the cross-domain decoder D i+1 After element-wise multiplication and 1×1 convolution, the cross-domain decoder i The input feature map of the feature output layer, i ∈[1,5), where the input feature map of the fifth feature output layer of the cross-domain decoder is the feature map output by the fifth feature extraction layer of the cross-domain encoder E 5.
5. The remote sensing image fine-grained semantic segmentation method according to claim 4, characterized in that: The processing process of the feature output layer of the cross-domain decoder is as follows: The first i The feature map of the feature output layer is convolved with 1×1, 3×3 and 1×1 to obtain the feature map D i1 , i ∈[1,5]; The feature map D i1 Input cross-domain coupling layer to get feature map D i2 ; The feature map D i2 A 1×1 convolution operation is used to obtain the feature map D i .
6. A remote sensing image fine-grained semantic segmentation method according to claim 3 or 5, characterized in that: The processing process of the cross-domain coupling layer is as follows: The input feature map of the cross-domain coupling layer is used to obtain the frequency domain output domain feature map in parallel F f out And the spatial domain output feature map F s out ; The frequency domain output domain feature map F f out And the spatial domain output feature map F s out The input feature map of the cross-domain coupling layer is concatenated and then a 1×1 convolution is used to obtain the output feature map of the cross-domain coupling layer. F out .
7. A remote sensing image fine-grained semantic segmentation method according to claim 6, characterized in that: Get the frequency domain output domain feature map F f out ,include: The input feature map of the cross-domain coupling layer is converted into a frequency domain feature map using a two-dimensional discrete cosine transform F freq ; The frequency domain feature map F freq Three 1×1 convolutions with different kernels are used to reshape the features in the channel dimension: F f1 ，F f2 , F f3 ； The feature map F f1 After transposition and the feature map F f2 Perform element-wise multiplication and normalization with the feature map F f3 Perform element-wise multiplication to obtain the feature map A f ; The feature map A f With the characteristic diagram F freq After addition, it is converted into a frequency domain output domain feature map through inverse discrete cosine transform F f out .
8. The remote sensing image fine-grained semantic segmentation method according to claim 6, characterized in that: Get spatial domain output feature map F s out ,include: The input feature map of the cross-domain coupling layer is convolved by 1×1 to generate a spatial domain feature map F spa ; The feature map F spa Three 1×1 convolutions with different kernels are used to reshape the features in the channel dimension: F s1 , F s2 , F s3 ; The feature map F s1 After transposition and the feature map F s2 Perform element-wise multiplication and normalization with the feature map F s3 Perform element-wise multiplication to obtain the feature map A s ; The feature map A s With the characteristic diagram F spa Add to get the spatial domain output feature map F s out .
9. A remote sensing image fine-grained semantic segmentation device, characterized in that: For implementing the remote sensing image fine-grained semantic segmentation method according to any one of claims 1 to 8, the device comprises: A data acquisition module is used to acquire a target remote sensing image, wherein the target remote sensing image is a remote sensing image including a target object and a background object; A prediction output module is used to input the target remote sensing image data into a fine-grained semantic segmentation model to obtain a fine-grained semantic segmentation result of the target remote sensing image; the fine-grained semantic segmentation result refers to dividing and labeling different objects or regions in the image according to semantic categories to obtain a prediction map containing semantic information; The fine-grained semantic segmentation model is built based on a cross-domain encoder-cross-domain decoder neural network, including: A cross-domain coupling encoder is provided, which is used to encode the remote sensing image and capture the frequency domain information and spatial domain information of the input feature map to form a feature map; A cross-level fusion module, used for aggregating the feature maps output by the cross-domain encoder and the cross-domain decoder as input to the next feature output layer of the cross-domain decoder; The cross-domain coupling decoder sets a cross-domain coupling layer to decode the input feature map, fuse the frequency domain information and the spatial domain information, restore the feature map to the same size as the original input remote sensing image, and obtain the semantic segmentation result.
10. The remote sensing image fine-grained semantic segmentation device according to claim 9, characterized in that: The device also includes: Model training module, used to Obtain an Earth observation remote sensing image dataset and randomly divide it into a training set and a test set; Constructing a neural network based on a cross-domain encoder-cross-domain decoder, inputting the training set images into the neural network of the cross-domain encoder-cross-domain decoder for training until the model converges, and obtaining a network model with optimal parameters as a fine-grained semantic segmentation model; The neural network training process is as follows: Input the training set image into the cross-domain encoder, and obtain a feature map after convolution, cross-domain coupling layer and maximum pooling processing; The feature map output by the cross-domain encoder and the feature map output by the cross-domain decoder are fused through a cross-level fusion module and used as the input of the next feature output layer of the cross-domain decoder. After convolution, cross-domain coupling layer and convolution processing, the spatial resolution of the feature map is restored to its original size. According to the category with the maximum probability, the semantics of the ground object in the current feature map is determined as the output result of semantic segmentation.