A remote sensing image semantic segmentation method based on multimodal data fusion

Abstract

The present invention provides a remote sensing image semantic segmentation method based on multimodal data fusion, comprising: obtaining multimodal remote sensing data of a target area; the multimodal remote sensing data includes at least two different types of remote sensing image data; preprocessing the multimodal remote sensing data to obtain preprocessed data; extracting features from the preprocessed data of each modality to obtain features of each modality; fusing the features of each modality to obtain fused features; inputting the fused features into a trained semantic segmentation network to obtain a segmentation result. The present invention can make full use of the complementarity of each modality data to improve the accuracy and robustness of semantic segmentation.

Description

Technical Field

[0001] The present invention relates to the technical field of remote sensing image processing, and in particular to a remote sensing image semantic segmentation method based on multimodal data fusion. Background Art

[0002] Remote sensing image semantic segmentation is the process of assigning each pixel in a remote sensing image to a specific category, which is crucial for fields such as object classification and environmental monitoring. Since remote sensing images usually contain complex surface features and different imaging conditions, semantic segmentation becomes a challenging task.

[0003] Single-modality remote sensing images (such as remote sensing images using only optical satellites) often fail to provide enough information for accurate segmentation. Therefore, how to fuse multimodal data and utilize the complementary information provided by different sensors has become a technical problem that needs to be solved to improve the accuracy of semantic segmentation of remote sensing images. Summary of the invention

[0004] In order to overcome the deficiencies of the prior art, the object of the present invention is to provide a remote sensing image semantic segmentation method based on multimodal data fusion.

[0005] To achieve the above object, the present invention provides the following solutions:

[0006] A remote sensing image semantic segmentation method based on multimodal data fusion, comprising:

[0007] Acquiring multimodal remote sensing data of a target area; the multimodal remote sensing data comprising at least two different types of remote sensing image data;

[0008] Preprocessing the multimodal remote sensing data to obtain preprocessed data;

[0009] Performing feature extraction on the preprocessed data of each mode respectively to obtain features of each mode;

[0010] The features of each modality are fused to obtain fused features;

[0011] The fused features are input into the trained semantic segmentation network to obtain the segmentation result.

[0012] Preferably, the multimodal remote sensing data includes: optical remote sensing data, radar remote sensing data and millimeter wave remote sensing data.

[0013] Preferably, preprocessing the multimodal remote sensing data to obtain preprocessed data includes:

[0014] Adopting an adaptive threshold filtering method to filter the optical remote sensing data to obtain the preprocessed data of the corresponding modality;

[0015] The radar remote sensing data and the millimeter wave remote sensing data are respectively subjected to data correction, data registration and data filtering to obtain the preprocessed data of the corresponding modality.

[0016] Preferably, the optical remote sensing data is filtered using an adaptive threshold filtering method to obtain the preprocessed data of the corresponding modality, including:

[0017] Performing wavelet decomposition on the optical remote sensing data at multiple scales to obtain corresponding wavelet coefficients;

[0018] Constructing filtering thresholds according to the size and decomposition scale of the optical remote sensing data respectively;

[0019] constructing an image filtering model using the filtering threshold;

[0020] The optical remote sensing data are filtered using the image filtering model to obtain the preprocessed image.

[0021] Preferably, the calculation formula of the filtering threshold is: Among them, λ j represents the filtering threshold at the jth decomposition scale, σ j represents the standard deviation of the noise at the jth decomposition scale, N j represents the signal length at the jth decomposition scale, and j represents the decomposition scale.

[0022] Preferably, the calculation formula of the image filtering model is:

[0023]

[0024] Wherein, sign is the sign function, a is the first preset coefficient, b is the second preset coefficient, ω j,k represents the kth wavelet coefficient at the jth decomposition scale, Represents the wavelet coefficients after filtering.

[0025] Preferably, the features of each modality are fused to obtain fused features, including:

[0026] Standardizing the features of each mode to obtain initial features at a standard scale;

[0027] Assigning a weight to each of the initial features, and then summing the initial features according to the corresponding weights to obtain a weighted feature vector;

[0028] The weighted feature vector is input into a trained decision tree model to modify the weighted feature vector to obtain the fusion feature.

[0029] Preferably, a weight is assigned to each of the initial features, and then the initial features are summed up according to the corresponding weights to obtain a weighted feature vector, including:

[0030] According to the historical optical remote sensing data, historical radar remote sensing data and historical millimeter wave remote sensing data, optical remote sensing data set, radar remote sensing data set and millimeter wave remote sensing data set are constructed respectively;

[0031] Respectively training a convolutional neural network deep learning model for multi-target detection according to the optical remote sensing data set, the radar remote sensing data set, and the millimeter wave remote sensing data set to obtain a trained first classifier, a second classifier, and a third classifier;

[0032] Using the first classifier to identify objects in the optical remote sensing data, obtaining a first confidence set for all objects in the optical remote sensing data, and using the second classifier to identify objects in the radar remote sensing data, obtaining a second confidence set for all objects in the radar remote sensing data, and using the third classifier to identify objects in the millimeter wave remote sensing data, obtaining a third confidence set for all objects in the millimeter wave remote sensing data;

[0033] Summing the confidence levels of each object in the first confidence set, the second confidence set, and the third confidence set to obtain the sum of object confidences in the first confidence set, the second confidence set, and the third confidence set, respectively;

[0034] Allocating weights to the corresponding initial features according to the ratio of the total confidence level of the object;

[0035] Each of the initial features is multiplied by the corresponding weight, and each multiplication result is summed to obtain the weighted feature vector.

[0036] According to the specific embodiments provided by the present invention, the present invention discloses the following technical effects:

[0037] The present invention provides a remote sensing image semantic segmentation method based on multimodal data fusion, comprising: obtaining multimodal remote sensing data of a target area; the multimodal remote sensing data includes at least two different types of remote sensing image data; preprocessing the multimodal remote sensing data to obtain preprocessed data; extracting features from the preprocessed data of each modality to obtain features of each modality; fusing the features of each modality to obtain fused features; inputting the fused features into a trained semantic segmentation network to obtain a segmentation result. The present invention can make full use of the complementarity of each modality data to improve the accuracy and robustness of semantic segmentation. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0039] Figure 1 A flow chart of a method provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0040] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0041] The purpose of the present invention is to provide a remote sensing image semantic segmentation method based on multimodal data fusion, which can make full use of the complementarity of each modality data and improve the accuracy and robustness of semantic segmentation.

[0042] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0043] Figure 1 A flow chart of a method provided by an embodiment of the present invention, such as Figure 1 As shown, the present invention provides a remote sensing image semantic segmentation method based on multimodal data fusion, comprising:

[0044] Step 100: Acquire multimodal remote sensing data of a target area; the multimodal remote sensing data includes at least two different types of remote sensing image data;

[0045] Step 200: preprocessing the multimodal remote sensing data to obtain preprocessed data;

[0046] Step 300: extracting features from the preprocessed data of each mode to obtain features of each mode;

[0047] Step 400: Fusing the features of each modality to obtain fused features;

[0048] Step 500: Input the fused features into a trained semantic segmentation network to obtain a segmentation result.

[0049] Preferably, the multimodal remote sensing data includes: optical remote sensing data, radar remote sensing data and millimeter wave remote sensing data.

[0050] Specifically, in this embodiment, the optical remote sensing data includes image data in visible light, infrared and ultraviolet bands; radar remote sensing data is microwave band data obtained using radar technology; and millimeter wave remote sensing data is data obtained using millimeter wave bands.

[0051] Preferably, preprocessing the multimodal remote sensing data to obtain preprocessed data includes:

[0052] Adopting an adaptive threshold filtering method to filter the optical remote sensing data to obtain the preprocessed data of the corresponding modality;

[0053] The radar remote sensing data and the millimeter wave remote sensing data are respectively subjected to data correction, data registration and data filtering to obtain the preprocessed data of the corresponding modality.

[0054] Specifically, the preprocessing method for radar remote sensing data and millimeter wave remote sensing data in this embodiment includes:

[0055] Data correction: Correct radar remote sensing data and millimeter wave remote sensing data, including atmospheric attenuation correction, terrain correction, etc., to eliminate stray signals and noise in the data.

[0056] Data registration: Register radar remote sensing data and millimeter wave remote sensing data to align them spatially for subsequent data fusion and analysis.

[0057] Data filtering: Filter radar remote sensing data and millimeter wave remote sensing data to remove clutter and interference signals in the data to improve data quality and availability.

[0058] Optionally, the specific steps of preprocessing depend on the needs and target tasks of subsequent processing. For example, for machine learning tasks, frame extraction and normalization may be required, while for video editing, more color correction and stabilization may be required. Preprocessing tools and libraries include OpenCV, FFmpeg, PIL (Python Imaging Library), etc.

[0059] Preferably, the optical remote sensing data is filtered using an adaptive threshold filtering method to obtain the preprocessed data of the corresponding modality, including:

[0060] Performing wavelet decomposition on the optical remote sensing data at multiple scales to obtain corresponding wavelet coefficients;

[0061] Constructing filtering thresholds according to the size and decomposition scale of the optical remote sensing data respectively;

[0062] constructing an image filtering model using the filtering threshold;

[0063] The optical remote sensing data are filtered using the image filtering model to obtain the preprocessed image.

[0064] Specifically, in addition to the adaptive threshold filtering process, in this embodiment, the optical remote sensing data may also be subjected to image processing including grayscale conversion and normalization.

[0065] Preferably, the calculation formula of the filtering threshold is: Among them, λ j represents the filtering threshold at the jth decomposition scale, σ j represents the standard deviation of the noise at the jth decomposition scale, N j represents the signal length at the jth decomposition scale, and j represents the decomposition scale.

[0066] Specifically, it can be seen from the above filtering threshold formula that as the decomposition scale increases, the threshold of each layer decreases accordingly, which is in line with the law of wavelet denoising. Therefore, this embodiment can improve the performance of wavelet denoising.

[0067] Preferably, the calculation formula of the image filtering model is:

[0068]

[0069] Wherein, sign is the sign function, a is the first preset coefficient, b is the second preset coefficient, ω j,k represents the kth wavelet coefficient at the jth decomposition scale, Represents the wavelet coefficients after filtering.

[0070] Optionally, this embodiment utilizes an image filtering model to perform a self-adjusting threshold transformation on the wavelet coefficients at each scale, thereby highlighting the detailed features of the image at different decomposition levels, suppressing image noise, and enhancing the layering of the image.

[0071] Preferably, the features of each modality are fused to obtain fused features, including:

[0072] Standardizing the features of each mode to obtain initial features at a standard scale;

[0073] Assigning a weight to each of the initial features, and then summing the initial features according to the corresponding weights to obtain a weighted feature vector;

[0074] The weighted feature vector is input into a trained decision tree model to modify the weighted feature vector to obtain the fusion feature.

[0075] Specifically, in this embodiment, feature fusion may further include the following steps:

[0076] Feature concatenation: directly concatenate all feature vectors end-to-end to form a longer feature vector.

[0077] Feature average: Calculate the average value of all feature vectors to get a new feature vector.

[0078] Feature weighting: assign a weight to each feature vector, then add them up to get a new feature vector. The weight can be set according to the importance of each feature. The feature weighting method is used in this embodiment.

[0079] Using a neural network: You can design a small neural network that takes the feature vector as input and train the network to learn how to best fuse these features.

[0080] Preferably, a weight is assigned to each of the initial features, and then the initial features are summed up according to the corresponding weights to obtain a weighted feature vector, including:

[0081] According to the historical optical remote sensing data, historical radar remote sensing data and historical millimeter wave remote sensing data, optical remote sensing data set, radar remote sensing data set and millimeter wave remote sensing data set are constructed respectively;

[0082] Respectively training a convolutional neural network deep learning model for multi-target detection according to the optical remote sensing data set, the radar remote sensing data set, and the millimeter wave remote sensing data set to obtain a trained first classifier, a second classifier, and a third classifier;

[0083] Using the first classifier to identify objects in the optical remote sensing data, obtaining a first confidence set for all objects in the optical remote sensing data, and using the second classifier to identify objects in the radar remote sensing data, obtaining a second confidence set for all objects in the radar remote sensing data, and using the third classifier to identify objects in the millimeter wave remote sensing data, obtaining a third confidence set for all objects in the millimeter wave remote sensing data;

[0084] Summing the confidence levels of each object in the first confidence set, the second confidence set, and the third confidence set to obtain the sum of object confidences in the first confidence set, the second confidence set, and the third confidence set, respectively;

[0085] Allocating weights to the corresponding initial features according to the ratio of the total confidence level of the object;

[0086] Each of the initial features is multiplied by the corresponding weight, and each multiplication result is summed to obtain the weighted feature vector.

[0087] Optionally, the convolutional neural network deep learning model for multi-target detection in this embodiment can be a YoloV3 model. This embodiment builds a classifier based on the YoloV3 neural network, builds an optical remote sensing data set, a radar remote sensing data set, and a millimeter wave remote sensing data set, and operates on the three data sets as follows: 70% of the data are randomly set as a training set, and the remaining 30% of the data are set as a validation set. Use the YoloV3 model to train on the training set until the model converges. In order to further improve the detection effect, a multi-scale training method and a data enhancement method are used. Through these training methods, the performance and generalization ability of the trained model are greatly improved.

[0088] Specifically, in this embodiment, by calculating the confidence level, the allocated weights can be made more accurate, thereby improving the accuracy and confidence level of feature fusion.

[0089] Furthermore, this embodiment uses the fused features for training models (such as SVM, decision tree, deep learning model, etc.). By comparing the performance of models trained using fused features and single features, the effect of the fusion strategy can be evaluated.

[0090] The beneficial effects of the present invention are as follows:

[0091] The present invention can fully utilize the complementarity of each modality data to improve the accuracy and robustness of semantic segmentation.

[0092] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0093] The principles and implementation methods of the present invention are described in this article using specific examples. The description of the above embodiments is only used to help understand the method and core idea of ​​the present invention. At the same time, for those skilled in the art, according to the idea of ​​the present invention, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as limiting the present invention.

Claims

1. A remote sensing image semantic segmentation method based on multimodal data fusion, characterized in that: include: Acquiring multimodal remote sensing data of a target area; the multimodal remote sensing data comprising at least two different types of remote sensing image data; Preprocessing the multimodal remote sensing data to obtain preprocessed data; Performing feature extraction on the preprocessed data of each mode respectively to obtain features of each mode; The features of each modality are fused to obtain fused features; Inputting the fused features into a trained semantic segmentation network to obtain a segmentation result; The multimodal remote sensing data includes: optical remote sensing data, radar remote sensing data and millimeter wave remote sensing data; The features of each modality are fused to obtain fused features, including: Standardizing the features of each mode to obtain initial features at a standard scale; Assigning a weight to each of the initial features, and then summing the initial features according to the corresponding weights to obtain a weighted feature vector; Inputting the weighted feature vector into a trained decision tree model to modify the weighted feature vector to obtain the fusion feature; Assign a weight to each of the initial features, and then sum the initial features according to the corresponding weights to obtain a weighted feature vector, including: According to the historical optical remote sensing data, historical radar remote sensing data and historical millimeter wave remote sensing data, optical remote sensing data set, radar remote sensing data set and millimeter wave remote sensing data set are constructed respectively; The preset convolutional neural network deep learning model for multi-target detection is trained according to the optical remote sensing data set, the radar remote sensing detection data set and the millimeter wave remote sensing detection data set to obtain a trained first classifier, a second classifier and a third classifier; Using the first classifier to identify objects in the optical remote sensing data, obtaining a first confidence set for all objects in the optical remote sensing data, and using the second classifier to identify objects in the radar remote sensing data, obtaining a second confidence set for all objects in the radar remote sensing data, and using the third classifier to identify objects in the millimeter wave remote sensing data, obtaining a third confidence set for all objects in the millimeter wave remote sensing data; Summing the confidence levels of each object in the first confidence set, the second confidence set, and the third confidence set to obtain the sum of object confidences in the first confidence set, the second confidence set, and the third confidence set, respectively; Allocating weights to the corresponding initial features according to the ratio of the total confidence level of the object; Each of the initial features is multiplied by the corresponding weight, and each multiplication result is summed to obtain the weighted feature vector.

2. The remote sensing image semantic segmentation method based on multimodal data fusion according to claim 1, characterized in that: Preprocessing the multimodal remote sensing data to obtain preprocessed data includes: Adopting an adaptive threshold filtering method to filter the optical remote sensing data to obtain the preprocessed data of the corresponding modality; The radar remote sensing data and the millimeter wave remote sensing data are respectively subjected to data correction, data registration and data filtering to obtain the preprocessed data of the corresponding modality.

3. The remote sensing image semantic segmentation method based on multimodal data fusion according to claim 2, characterized in that: The optical remote sensing data is filtered using an adaptive threshold filtering method to obtain the pre-processed data of the corresponding modality, including: Performing wavelet decomposition on the optical remote sensing data at multiple scales to obtain corresponding wavelet coefficients; Constructing filtering thresholds according to the size and decomposition scale of the optical remote sensing data respectively; constructing an image filtering model using the filtering threshold; The optical remote sensing data are filtered using the image filtering model to obtain the preprocessed image.

4. The remote sensing image semantic segmentation method based on multimodal data fusion according to claim 3, characterized in that: The calculation formula of the filtering threshold is: Among them, λ j represents the filtering threshold at the jth decomposition scale, σ j represents the standard deviation of the noise at the jth decomposition scale, N j represents the signal length at the jth decomposition scale, and j represents the decomposition scale.

5. The remote sensing image semantic segmentation method based on multimodal data fusion according to claim 4, characterized in that: The calculation formula of the image filtering model is: Wherein, sign is the sign function, a is the first preset coefficient, b is the second preset coefficient, ω j,k represents the kth wavelet coefficient at the jth decomposition scale, Represents the wavelet coefficients after filtering.

Images