An airborne multi-dimensional synthetic aperture radar image ground object classification method

Abstract

The present invention relates to a method for classifying ground objects in airborne multi-dimensional synthetic aperture radar images based on deep learning, comprising the following steps: Step S1, preprocess the simultaneously acquired airborne high-resolution multi-dimensional SAR images, label the ground object type tags, and divide the data set; Step S2, respectively construct corresponding ground object classification models for the SAR images of each dimension, and use deep learning methods to train with the training set to make the loss function value tend to be stable and reduce as much as possible; Step S3, respectively perform ground object classification test evaluations using the trained ground object classification models, and obtain the classification accuracy and sensitivity of each ground object classification model for various types of ground objects according to the test evaluation results; Step S4, fuse the ground object classification models to achieve ground object classification of airborne multi-dimensional SAR images. Compared with the prior art, the method of the present invention can improve the application efficiency of airborne high-resolution multi-dimensional SAR data in ground object classification and improve the accuracy of ground object classification.

Description

Technical Field

[0001] The present invention relates to the field of airborne synthetic aperture radar imaging remote sensing applications, and particularly to a method for classifying ground objects in airborne multi-dimensional synthetic aperture radar images based on deep learning. Background Art

[0002] Synthetic aperture radar (SAR) imaging has attracted much attention due to its advantages of all-weather and all-time. Multi-dimensional SAR is a joint observation technique that obtains multiple observables respectively within at least two of the polarization, frequency, angle, and phase dimensions of its basic observation method. In contrast, the detection method of obtaining single / multiple observables within a single dimension is single-dimensional SAR. Through signal and information integrated processing, multi-dimensional SAR may be able to more accurately distinguish different scattering mechanisms of the observed objects, and thus more accurately obtain their geometric and physical characteristics. In the application of SAR ground object classification, the ground object target information contained in single-dimensional data is relatively less, and there is usually a certain degree of ambiguity between different types of ground objects. Using multi-dimensional SAR data for ground object classification can obtain richer target information. Therefore, using multi-dimensional SAR images for ground object classification has become an important research content of SAR image interpretation and an important development direction in the practical application of SAR technology.

[0003] SAR ground object classification plays an indispensable role in fields such as vegetation detection, land use, urban planting, and sea ice monitoring. The existing polarization features for SAR ground object classification extraction can be obtained through target decomposition, such as Pauli decomposition, Freeman decomposition, etc. Some other existing methods can also use edge detection, polarization scattering mechanism, etc. to achieve SAR ground object classification.

[0004] Compared with the existing methods, using deep learning can process a large amount of SAR data and achieve high-precision ground object classification. However, on the other hand, deep learning has high requirements for the amount of data, and at the same time, the training of the model takes a long time. SAR data has the disadvantage of insufficient data volume compared with optical data that is relatively easy to obtain and widely used, especially airborne multi-dimensional SAR data, which is even more difficult to obtain. Therefore, there are relatively few methods for classifying ground objects using airborne multi-dimensional SAR data, and the used bands are not rich enough. Summary of the Invention

[0005] The purpose of the present invention is to overcome the above-mentioned defects existing in the prior art and provide a method for classifying ground objects in airborne multi-dimensional synthetic aperture radar images.

[0006] The purpose of the present invention can be achieved by the following technical solutions:

[0007] A method for classifying ground objects in airborne multi-dimensional synthetic aperture radar images, comprising the following steps:

[0008] Step S1: Preprocess the simultaneously acquired airborne high-resolution multi-dimensional SAR images, label the ground object type tags, and divide the dataset into a training set and a test set.

[0009] Step S2: For each dimension of the SAR images, construct corresponding ground object classification models respectively, and use the deep learning method to train with the training set to make the loss function value tend to be stable and be reduced as much as possible.

[0010] Step S3: Use the trained ground object classification models to conduct ground object classification test evaluations with the test set respectively, and obtain the classification accuracy sensitivities of each ground object classification model for various types of ground objects according to the test evaluation results.

[0011] Step S4: Fuse the classification results of each ground object classification model to achieve ground object classification of airborne multi-dimensional SAR images.

[0012] The specific steps of the said Step S1 include the following steps:

[0013] Step S1-1: Simultaneously obtain airborne high-resolution multi-dimensional SAR image data through an aerial remote sensing system, generate corresponding pixel category masks, and label the masks to generate a dataset.

[0014] Step S1-2: On the premise of counting the category proportions, divide the dataset into a training set and a test set according to the semantic segmentation dataset ratio requirements.

[0015] In the said Step S1, the airborne high-resolution multi-dimensional SAR images include optical images, C-band images, Ka-band images, L-band images, P-band images, and S-band images.

[0016] The specific steps of the said Step S2 include the following steps:

[0017] Step S2-1: Use the semantic segmentation network HRNet for feature extraction and multi-scale feature fusion.

[0018] Step S2-2: Stack and upsample the extracted features to generate a prediction map and calculate the loss with the ground truth. The expression of the loss function is:

[0019]

[0020] where y i is the ground truth of the i-th class, is the predicted value of the i-th class, and n is the total number of classes.

[0021] The specific steps of the said Step S3 include the following steps:

[0022] Step S3-1: For the ground object classification model corresponding to the SAR image of each dimension, verify it on the corresponding test set to obtain the test visualization classification result without training data.

[0023] Step S3-2: Evaluate the classification result using semantic segmentation evaluation metrics.

[0024] In the above-mentioned Step S3-2, the semantic segmentation evaluation metrics include Intersection over Union (IoU), Frequency Weighted Intersection over Union (FWIoU), and Pixel Accuracy (PA).

[0025] The specific content of Step S4 is as follows:

[0026] Input the SAR training images of each dimension into the respective ground object classification models for classification to obtain classification test probabilities, and perform weighted superposition on all test probabilities. Take the maximum probability after weighted superposition as the category of the corresponding image pixel.

[0027] In the above-mentioned Step S4, the expression for fusing the respective ground object classification models is:

[0028]

[0029] where M represents the ground object classification model, D represents the image data, b represents the band, and f r (M b , D b ) represents the scaling operation, Class represents the category, and axis represents the direction for obtaining the maximum value.

[0030] In the above-mentioned Step S4, determine the weights based on the classification test results of each ground object classification model for various ground objects and perform weighted fusion. Specifically:

[0031] Taking the Ka band with the best single-band classification test result as the benchmark, and using the bands with the best individual classification results for various ground objects as assistance, weight the classification results of each ground object into the classification result of the Ka band.

[0032] To prevent pixel category conflicts during fusion, define the category priority. The specific definition of the category priority is:

[0033] priority = α·max_iou + β·dif

[0034] α + β = 1

[0035] where priority is the category priority, α and β are weights, max_iou is the optimal Intersection over Union (IoU) of each ground object classification model for each ground object target classification, and dif is the difference between the optimal Intersection over Union (IoU) and the sub-optimal Intersection over Union (IoU) of each ground object classification model for each ground object target classification.

[0036] Compared with the prior art, the present invention has the following advantages:

[0037] The method for airborne high-resolution multi-dimensional SAR ground object classification based on deep learning provided by the present invention applies deep learning to the ground object classification of multi-dimensional SAR. First, the ground object classification of each single-band multi-polarization is performed to obtain the classification sensitivity of each band to different ground object types, and then the ground object classification models of each band are used for fusion to achieve the optimal ground object classification, and the result fusion is performed based on the prior knowledge of the single-band classification results.

[0038] In addition, the method of the present invention can also improve the application efficiency of airborne high-resolution multi-dimensional SAR data in ground object classification, and give an optimized acquisition scheme for airborne multi-dimensional SAR imaging data according to the ground object classification requirements of different industry users. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Figure 1 It is a flowchart of the method for airborne high-resolution multi-dimensional SAR ground object classification based on deep learning in an embodiment of the present invention.

[0040] Figure 2 It is the optical image, Ka-band SAR image and corresponding mask of two regions in an embodiment of the present invention. Among them, Fig. (2a) is the optical image of the first region, Fig. (2b) is the Ka-band SAR image of the first region, Fig. (2c) is the mask corresponding to Fig. (2b), Fig. (2d) is the optical image of the second region, Fig. (2e) is the Ka-band SAR image of the second region, and Fig. (2f) is the mask corresponding to Fig. (2e).

[0041] Figure 3 It is the deep learning network algorithm structure adopted in an embodiment of the present invention. Among them, Fig. (3a) is the feature extraction and fusion structure HRNet, and Fig. (3b) is the stacking and upsampling network structure.

[0042] Figure 4 It is the display of data samples used for testing in an embodiment of the present invention.

[0043] Figure 5 It is the final multi-dimensional SAR ground object classification result map obtained by using the model fusion and result fusion algorithms in an embodiment of the present invention.

[0044] Figure 6 It is a schematic diagram of the definition of category priority in the result fusion of an embodiment of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0045] The present invention will be described in detail below with reference to the accompanying drawings and specific embodiments.

[0046] Embodiment

[0047] The present invention provides a method for airborne high-resolution multi-dimensional SAR ground object classification based on deep learning, which is used to achieve SAR ground object classification. The following combines Figure 1 to illustrate the process of the method for airborne high-resolution multi-dimensional SAR ground object classification based on deep learning in this embodiment.

[0048] The method includes the following steps:

[0049] Step S1, preprocess the acquired airborne high-resolution multi-dimensional SAR image data to generate labels and divide the data set. Among them, the data region information collected in this embodiment is shown in Table 1;

[0050] Table 1 Data region information adopted in the embodiment

[0051]

[0052]

[0053] Step S1 includes the following sub-steps:

[0054] Step S1-1, use the multi-dimensional SAR data obtained by the airborne remote sensing system to generate corresponding pixel category masks. The obtained high-resolution optical map and polarimetric Ka-band SAR image are as Figure 2 shown, Figure 2 The mask generation in

[0055] is generated by annotating according to multiple bands, high-resolution optical images and maps. The tool used for annotation is the imageLabeler of MATLAB. Figure 2 Step S1-2, on the premise of counting the category proportion, divide the training set and the test set according to the proportion requirements of the semantic segmentation data set. The division requirement is to ensure that the category proportions of the test set and the training set are not very different. The selection of the test set is shown in

[0056] the white square in the mask. Figure 3 Step S2, as

[0057] shown, use the deep learning network model to train the data of each band, so that the loss function value tends to be stable and is reduced as much as possible.

[0058] Step S2 includes the following sub-steps:

[0059] Step S2-1, use the feature extraction and fusion structure HRNet in Fig. (3a) to perform feature extraction and multi-scale feature fusion, and simultaneously obtain the low-level semantic features and high-level semantic features of the image. While ensuring high resolution, obtain rich image features required for pixel-level classification.

[0059] Step S2-2: Using the network structure in Fig. (3b), stack and upsample the extracted features to generate a prediction map and calculate the loss with the ground truth. The loss function is as follows:

[0060]

[0061] where $y$ i is the ground truth of the $i$-th class, is the predicted value of the $i$-th class, and $n$ is the total number of classes.

[0062] Step S3: Use the models trained with each band to perform land cover classification tests on the data of each band. In this embodiment, the SAR data of the Ka, C, L, P, and S bands and the high-resolution optical data are trained separately to obtain 6 models with different dimensions, and then these 6 models are tested and verified on their respective test sets to obtain the visualization test results of a single band. Finally, quantitative analysis and research are carried out on the results of all bands to obtain the classification sensitivity of different bands to different land cover types, that is, which land cover types each model has a high classification accuracy for.

[0063] The present invention aims at the advantage that the national major scientific and technological infrastructure "aerial remote sensing system" can simultaneously acquire multi-dimensional SAR imaging data. In this example, by analyzing the classification sensitivity of multi-dimensional SAR images to different land cover types, on the one hand, it can give full play to the application efficiency of multi-dimensional SAR data in fine land cover classification and explore a more accurate land cover classification method; on the other hand, it can take into account both the land cover classification accuracy and the economic efficiency, provide support for the design and construction of SAR imaging systems for users in various industries, and help select a suitable SAR imaging system design scheme for different industry applications.

[0064] Step S3 includes the following sub-steps:

[0065] Step S3-1: Use the models trained with each band on the training set to verify on its test set to obtain the test visualization results without training data. Some test set samples are as Figure 4 shown. The first row is the SAR image sample of the Ka band, and the second row is its corresponding mask image.

[0066] Step S3-2: Evaluate the classification results using semantic segmentation evaluation metrics. The metrics used are Intersection over Union (IoU):

[0067]

[0068] and Frequency Weighted Intersection over Union (FWIoU):

[0069]

[0070] Pixel Accuracy (PA):

[0071]

[0072] where n is the number of target categories, and p ij is the number of pixels belonging to class i but predicted as class j. The quantitative analysis results of the single - band land cover classification results are shown in Table 2 and Figure 5 as follows.

[0073] Table 2 Quantitative Results of Single - Band Land Cover Classification

[0074]

[0075] Step S4: Perform model fusion on the trained models of each band to achieve land cover classification.

[0076] Step S4 includes the following sub - steps:

[0077] Step S4 - 1: In the embodiment of the present invention, read all the models obtained by single - band training, and read the test images of all bands. Then, use the model of each band to test each band. Instead of obtaining the final result, scale the result matrix to the same size.

[0078] Step S4 - 2: Weightedly superimpose the test probabilities of the images tested by all band models, and take the maximum probability as the class of the pixel. The model fusion formula is as follows:

[0079]

[0080] where M represents the land cover classification model, D represents the image data, b represents the band, and f r (M b , D b ) represents the scaling operation, Class represents the class, axis represents the direction of obtaining the maximum value. Scale the results of multiple bands to the same size, and then weightedly classify the probabilities of all bands. The visualization results of model fusion for land cover classification are shown in Figure 5 as follows. Figure 5 In it, the first row represents the image sample for testing, the second row represents the corresponding mask image of the image sample, and the third row is the test result of the model fusion algorithm.

[0081] Step S5: Define the class priority using the prior information of the test results of each band to achieve result fusion.

[0082] Step S5 includes the following sub - steps:

[0083] Step S5-1: In the embodiment of the present invention, taking the Ka band with the optimal comprehensive classification result in a single band as the benchmark (with the highest weight), and using the bands with the optimal individual classification results for each ground object category as the assistance (determined successively according to the test and evaluation results in step S3), the classification results of different ground objects are weighted and fused into the classification result of the Ka band.

[0084] Step S5-2: In this embodiment, pixel category conflicts may occur during the fusion. That is, for a certain pixel point, the classification result in the Ka band is category A, but the classification result in the C band is category B. Then, the specific category of this pixel point needs to be defined with priorities. The definition of category priorities is as Figure 6 shown, which consists of two factors. One is the highest IoU of the classification result, and the other is the difference between the optimal IoU and the sub-optimal IoU. Figure 6 In, the red circle indicates that if a category conflict occurs between water area and road, the priority of the water area is greater than that of the road. The formula for defining category priorities is as follows:

[0085] priority = α·max_iou + β·dif

[0086] In the formula, α + β = 1, max_iou is the optimal IoU of each ground object target classification by different bands, and dif is the difference between the optimal IoU and the sub-optimal IoU of each ground object target classification by different bands. Both are values after normalization.

[0087] The above embodiments are only used to illustrate the specific implementation manners of the present invention, and the present invention is not limited to the description scope of the above embodiments.

Claims

1. An airborne multi-dimensional synthetic aperture radar image ground object classification method, characterized in that It includes the following steps: Step S1: Preprocess the simultaneously acquired airborne high-resolution multi-dimensional SAR images, label the ground object type tags, and divide the dataset into a training set and a test set; Step S2: For each dimension of the SAR images, construct corresponding ground object classification models respectively, and use deep learning methods to train with the training set to make the loss function value tend to be stable and reduce it as much as possible; Step S3: Use the trained ground object classification models to conduct ground object classification test evaluations with the test set respectively, and obtain the classification accuracy sensitivities of each ground object classification model for various types of ground objects according to the test evaluation results; Step S4: Fuse the classification results of each ground object classification model to achieve ground object classification of airborne multi-dimensional SAR images; In the above-mentioned step S1, the airborne high-resolution multi-dimensional SAR images include optical images, C-band images, Ka-band images, L-band images, P-band images, and S-band images; The specific content of the above-mentioned step S4 is as follows: Input the SAR training images of each dimension into the corresponding ground object classification models for classification to obtain classification test probabilities, and perform weighted superposition on all test probabilities, and take the maximum probability after weighted superposition as the category of the corresponding image pixel; In the above-mentioned step S4, the expression for fusing each ground object classification model is: Among them, M represents the ground object classification model, D represents the image data, b represents the band, and f r (M b ,D b ) represents the scaling operation, Class represents the category, and axis represents the direction for obtaining the maximum value; In the above-mentioned step S4, determine the weights according to the classification test results of each ground object classification model for various types of ground objects in step S3 and perform weighted fusion. Specifically: Taking the Ka-band with the best single-band classification test result as the benchmark, and using the band with the best single-classification result for each type of ground object as an auxiliary, weight the classification results of each ground object into the classification result of the Ka-band.

2. The method for classifying ground objects in an airborne multi-dimensional synthetic aperture radar image according to claim 1, wherein, The specific content of the above-mentioned step S1 includes the following steps: Step S1-1: Simultaneously obtain airborne high-resolution multi-dimensional SAR image data through an airborne remote sensing system, generate corresponding pixel category masks, and label the masks to generate a dataset; Step S1-2: On the premise of counting the category proportions, divide the dataset into a training set and a test set according to the semantic segmentation dataset ratio requirements.

3. The method for classifying ground objects in an airborne multi-dimensional synthetic aperture radar image according to claim 1, characterized in that The specific content of the above-mentioned step S2 includes the following steps: Step S2-1: Use the semantic segmentation network HRNet for feature extraction and multi-scale feature fusion; Step S2-2: Stack and upsample the extracted features to generate a prediction map and calculate the loss with the ground truth. The expression of the loss function is: where y i is the true value of the i-th class, is the predicted value of the i-th class, and n is the total number of classes.

4. A method for classifying ground objects in airborne multi-dimensional synthetic aperture radar images according to claim 1, characterized in that, The specific content of the above-mentioned step S3 includes the following steps: Step S3-1: For the ground object classification model corresponding to each dimension of the SAR images, verify on the corresponding test set to obtain the test visualization classification results without training data; Step S3-2: Evaluate the classification results using semantic segmentation evaluation metrics.

5. The method for classifying ground objects in an airborne multi-dimensional synthetic aperture radar image according to claim 4, wherein In the above-mentioned step S3-2, the semantic segmentation evaluation metrics include intersection over union IoU, frequency weighted intersection over union FWIoU, and pixel accuracy PA.

6. The method for classifying ground objects in an airborne multi-dimensional synthetic aperture radar image according to claim 1, wherein, To prevent pixel category conflicts during fusion, define the category priority. The specific definition of the category priority is: priority = α·max_iou + β·dif α+β＝1 Among them, priority is the category priority, α and β are weights, max_iou is the optimal intersection over union (IoU) of each object classification by each object classification model, and dif is the difference between the optimal IoU and the sub-optimal IoU of each object classification by each object classification model.

Images