Ground feature classification method, device, medium and product based on optical image and synthetic aperture radar image

Abstract

The application discloses a ground object classification method and device based on optical images and synthetic aperture radar images, a medium and a product, relates to the ground object classification field, and acquires optical images and synthetic aperture radar images of a region to be classified; according to the optical images and the synthetic aperture radar images, a ground object classification model is used to determine the ground object category of the region to be classified; the ground object classification model is obtained by training a ground object classification initial model by using a training data set; the ground object classification initial model comprises a first feature extraction network, a second feature extraction network, a feature fusion module and a classification module; the feature extraction network extracts deep features and shallow features of the synthetic aperture radar images and deep features and shallow features of the optical images; the feature fusion module fuses the extracted deep features and shallow features to obtain a feature information fusion image; and the classification module determines the ground object category of the region to be classified according to the feature information fusion image. The application improves the accuracy of ground object classification.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of ground object classification, and in particular to a ground object classification method and device based on optical images and synthetic aperture radar images, a medium and a product. BACKGROUND

[0002] Remote sensing images provide extremely rich and complex data of a target area. Remote sensing image classification is a key link in the target interpretation of remote sensing images. With the continuous development of remote sensing technology, the observation mode has evolved from single ground observation to multi-platform, multi-source and multi-scale observation. Therefore, remote sensing images can provide more comprehensive and accurate information. When performing image classification, the use of single modal data is still insufficient to meet the actual classification needs. Therefore, researchers pay more and more attention to the collaborative processing of optical images and synthetic aperture radar images. Synthetic aperture radar (SAR) images can show good structural features and have the advantages of all-weather and all-day operation and strong penetration. However, the definition of SAR images is relatively low. Optical images mainly use spectral information, which makes it easier to classify ground objects from remote sensing images. However, optical sensors are easily affected by climate and observation time when imaging. The fusion of SAR images and optical images can make full use of the information in different data types and offset the interference of single data type. In addition, the general rules of ground object features can be learned from multiple data types, the generalization ability is improved, and the accuracy is also improved.

[0003] However, due to the different imaging principles of the two sensors, there is heterogeneity between the heterogeneous data, the expression forms of the same ground object features are different, and the data scales are different. Therefore, using a simple feature fusion method, such as directly adding multiple source images at the pixel level and then classifying, cannot achieve ideal results. How to classify the optical images and synthetic aperture radar images has become a challenge in the field of image processing. SUMMARY

[0004] The purpose of the present application is to provide a ground object classification method, device, medium and product based on optical images and synthetic aperture radar images to improve the accuracy of ground object classification.

[0005] To achieve the above purpose, the present application provides the following solutions.

[0006] In a first aspect, the present application provides a ground object classification method based on optical images and synthetic aperture radar images, comprising:

[0007] obtaining optical images and synthetic aperture radar images of a region to be classified;

[0008] According to the optical image and the synthetic aperture radar image, a ground object class of the region to be classified is determined by using a ground object classification model; wherein the ground object classification model is obtained by training a ground object classification initial model by using a training data set; the training data set comprises optical images and synthetic aperture radar images of a training region and corresponding ground object class labels; the ground object classification initial model comprises a first feature extraction network, a second feature extraction network, a feature fusion module and a classification module; the first feature extraction network and the second feature extraction network are connected with the feature fusion module; the feature fusion module is connected with the classification module; the first feature extraction network is used to extract deep features and shallow features of the optical image; the second feature extraction network is used to extract deep features and shallow features of the synthetic aperture radar image; the feature fusion module is used to fuse the deep features and shallow features of the optical image and the deep features and shallow features of the synthetic aperture radar image to obtain a feature information fusion image; and the classification module is used to classify ground objects of the region to be classified according to the feature information fusion image to determine the ground object class.

[0009] Optionally, the feature extraction network comprises: a first jump dense connection module, a first down-sampling transition layer, a second jump dense connection module, a second down-sampling transition layer, a third jump dense connection module, a third down-sampling transition layer, a fourth jump dense connection module, a first up-sampling transition layer, a first feature extraction module, a second up-sampling transition layer, a second feature extraction module, a third up-sampling transition layer and a third feature extraction module connected in sequence; the fourth jump dense connection module and the third feature extraction module are connected with the feature fusion module.

[0010] Optionally, the feature fusion module comprises: a first self-attention mechanism based separation fusion gating module, a second self-attention mechanism based separation fusion gating module and a fourth up-sampling transition layer; the first self-attention mechanism based separation fusion gating module is connected with the fourth jump dense connection module; the second self-attention mechanism based separation fusion gating module is connected with the fourth up-sampling transition layer and the third feature extraction module respectively; the self-attention mechanism based separation fusion gating module and the fourth up-sampling transition layer are connected with the classification module through a feature map concatenation operation.

[0011] Optionally, the classification module is a softmax classifier.

[0012] Optionally, the ground object classification initial model is trained by using the training data set, and specifically comprises:

[0013] The training data set is obtained.

[0014] inputting the optical image and the synthetic aperture radar image of the training area into a current land cover classification initial model to obtain a land cover predicted category of the training area;

[0015] determining a loss function value of the land cover predicted category of the training area and the land cover category label of the training area by using a loss function;

[0016] determining whether an end training condition is met; the end training condition is that a maximum iteration number is reached or the loss function value is less than a preset value;

[0017] if yes, taking the current land cover classification initial model as the land cover classification model;

[0018] if no, adjusting a model parameter of the current land cover classification initial model according to the loss function value, and returning to the step of inputting the optical image and the synthetic aperture radar image of the training area into the current land cover classification initial model to obtain the land cover predicted category of the training area.

[0019] In a second aspect, a computer device is provided, which comprises a memory, a processor, a computer program stored in the memory and executable on the processor, and the processor executes the computer program to implement the land cover classification method based on the optical image and the synthetic aperture radar image according to any one of the above embodiments.

[0020] In a third aspect, a computer readable storage medium is provided, which stores a computer program, and the computer program is executed by a processor to implement the land cover classification method based on the optical image and the synthetic aperture radar image according to any one of the above embodiments.

[0021] In a fourth aspect, a computer program product is provided, which comprises a computer program, and the computer program is executed by a processor to implement the land cover classification method based on the optical image and the synthetic aperture radar image according to any one of the above embodiments.

[0022] According to the embodiments provided in the present application, the following technical effects are achieved:

[0023] The application provides a ground object classification method and device based on optical images and synthetic aperture radar images, a medium and a product. The optical images and the synthetic aperture radar images of a region to be classified are obtained. According to the optical images and the synthetic aperture radar images, a ground object classification model is used to determine the ground object category of the region to be classified. The ground object classification model is obtained by training a ground object classification initial model using a training data set. The ground object classification initial model includes a first feature extraction network, a second feature extraction network, a feature fusion module and a classification module. The first feature extraction network is used to extract deep features and shallow features of the optical images. The second feature extraction network is used to extract deep features and shallow features of the synthetic aperture radar images. The feature fusion module is used to fuse the deep features and the shallow features of the optical images and the deep features and the shallow features of the synthetic aperture radar images to obtain a feature information fusion image. The classification module is used to classify the ground objects of the region to be classified according to the feature information fusion image to determine the ground object category. The application improves the accuracy of ground object classification. BRIEF DESCRIPTION OF DRAWINGS

[0024] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings needed in the embodiments will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative effort.

[0025] Figure 1 A flowchart of a ground object classification method based on optical images and synthetic aperture radar images according to an embodiment of the present application is provided.

[0026] Figure 2 A general structure diagram of the ground object classification model in the present application is provided.

[0027] Figure 3 A feature extraction network structure diagram in the present application is provided.

[0028] Figure 4 A skip-dense connection module structure diagram in the present application is provided.

[0029] Figure 5 A SSAF feature fusion classification model structure diagram in the present application is provided.

[0030] Figure 6 A SSA-Gate module structure diagram in the present application is provided.

[0031] Figure 7Figures are schematic diagrams of the forestland classification results of the WHU-OPT-SAR dataset; wherein (a) is the forestland optical image in the WHU-OPT-SAR dataset; (b) is the forestland label in the WHU-OPT-SAR dataset; (c) is a schematic diagram of the forestland classification results of the WHU-OPT-SAR dataset on the UNet network; (d) is a schematic diagram of the forestland classification results of the WHU-OPT-SAR dataset on the DeeplabV3+ network; (e) is a schematic diagram of the forestland classification results of the WHU-OPT-SAR dataset on the PSPNet network; (f) is a schematic diagram of the forestland classification results of the WHU-OPT-SAR dataset on the U-DSDNet network;

[0032] Figure 8 Figures are schematic diagrams of the along-water classification results of the WHU-OPT-SAR dataset; wherein (a) is the along-water optical image in the WHU-OPT-SAR dataset; (b) is the along-water label in the WHU-OPT-SAR dataset; (c) is a schematic diagram of the along-water classification results of the WHU-OPT-SAR dataset on the UNet network; (d) is a schematic diagram of the along-water classification results of the WHU-OPT-SAR dataset on the DeeplabV3+ network; (e) is a schematic diagram of the along-water classification results of the WHU-OPT-SAR dataset on the PSPNet network; (f) is a schematic diagram of the along-water classification results of the WHU-OPT-SAR dataset on the U-DSDNet network;

[0033] Figure 9 Figures are schematic diagrams of the one-class optical image classification results of the SEN-OPT-SAR dataset; wherein (a) is the one-class optical image in the SEN-OPT-SAR dataset; (b) is the one-class optical image label in the SEN-OPT-SAR dataset; (c) is a schematic diagram of the one-class optical image classification results of the SEN-OPT-SAR dataset on the UNet network; (d) is a schematic diagram of the one-class optical image classification results of the SEN-OPT-SAR dataset on the DeeplabV3+ network; (e) is a schematic diagram of the one-class optical image classification results of the SEN-OPT-SAR dataset on the PSPNet network; (f) is a schematic diagram of the one-class optical image classification results of the SEN-OPT-SAR dataset on the U-DSDNet network;

[0034] Figure 10Schematic diagram of the classification results of another type of optical images of the SEN-OPT-SAR dataset; (a) is another type of optical image in the SEN-OPT-SAR dataset; (b) is another type of optical image label in the SEN-OPT-SAR dataset; (c) is a schematic diagram of the classification results of another type of optical images of the SEN-OPT-SAR dataset on the UNet network; (d) is a schematic diagram of the classification results of another type of optical images of the SEN-OPT-SAR dataset on the DeeplabV3+ network; (e) is a schematic diagram of the classification results of another type of optical images of the SEN-OPT-SAR dataset on the PSPNet network; (f) is a schematic diagram of the classification results of another type of optical images of the SEN-OPT-SAR dataset on the U-DSDNet network;

[0035] Figure 11 A schematic diagram of the structure of a computer device provided in one embodiment of the present application. DETAILED DESCRIPTION

[0036] The following will be combined with the drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are only part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0037] In order to make the above-mentioned purposes, features and advantages of the present application more obvious and easy to understand, the present application is further described in detail below with reference to the accompanying drawings and specific implementation methods.

[0038] In an exemplary embodiment, Figure 1 As shown, a method for ground object classification based on optical images and synthetic aperture radar images is provided, comprising the following steps:

[0039] S1: Obtain optical images and synthetic aperture radar images of the area to be classified.

[0040] S2: determining a ground object category of the region to be classified according to the optical image and the synthetic aperture radar image by using a ground object classification model; wherein the ground object classification model is obtained by training a ground object classification initial model by using a training data set; the training data set comprises optical images and synthetic aperture radar images of training regions and corresponding ground object category labels; the ground object classification initial model comprises a first feature extraction network, a second feature extraction network, a feature fusion module and a classification module; the first feature extraction network and the second feature extraction network are connected with the feature fusion module; the feature fusion module is connected with the classification module; the first feature extraction network is used to extract deep features and shallow features of the optical image; the second feature extraction network is used to extract deep features and shallow features of the synthetic aperture radar image; the feature fusion module is used to fuse the deep features and shallow features of the optical image and the deep features and shallow features of the synthetic aperture radar image to obtain a feature information fusion image; and the classification module is used to classify ground objects of the region to be classified according to the feature information fusion image to determine the ground object category.

[0041] As an optional implementation, the ground object classification initial model is trained by using the training data set, and specifically includes:

[0042] The training data set is obtained.

[0043] The optical image and the synthetic aperture radar image of the training region are input into a current ground object classification initial model to obtain a ground object predicted category of the training region.

[0044] A loss function value of the ground object predicted category of the training region and the ground object category label of the training region is determined by using a loss function.

[0045] It is determined whether an end training condition is met; the end training condition is that a maximum iteration number is reached or the loss function value is less than a preset value.

[0046] If yes, the current ground object classification initial model is taken as the ground object classification model.

[0047] If no, a model parameter of the current ground object classification initial model is adjusted according to the loss function value, and the step of inputting the optical image and the synthetic aperture radar image of the training region into the current ground object classification initial model to obtain the ground object predicted category of the training region is returned.

[0048] In the field of remote sensing, data fusion is a commonly used technique, which aims to integrate information with different spatial and spectral resolutions obtained from different platforms such as satellites, aircraft, and ground. After data fusion, the integrated data contains more comprehensive information than a single image source. Data fusion methods include pixel-level fusion, feature-level fusion, and decision-level fusion, each with its own application advantages and application fields. The present application adopts a feature-level fusion method, and its structure is shown in Figure 2 The advantages of feature fusion in the present application are:

[0049] (1) Reducing data dimensionality: The goal of feature-level fusion is to extract and capture the core features of the original data, which helps to reduce the dimensionality of the data and thus shorten the processing time required for information integration.

[0050] (2) Retaining most information: Feature-level fusion retains most of the information while reducing the dimensionality of the data, which helps to improve the processing efficiency and real-time performance of the fusion system.

[0051] (3) Improving the accuracy of decision analysis: Different feature values can be extracted according to different decision analyses, and the required feature values can be obtained according to decision analysis, which helps to improve the accuracy of decision analysis.

[0052] (4) Flexibility: Feature-level fusion is suitable for multiple information sources and has high flexibility.

[0053] The present application inputs synthetic aperture radar images and optical images into a feature extraction network to extract synthetic aperture radar image feature information and optical image feature information, respectively. The feature fusion separation module (feature fusion module) is used to fuse the feature information (deep and shallow features) of the synthetic aperture radar image and the feature information of the optical image extracted in the feature extraction network by weighted merging, to obtain a feature information fusion image containing optical image-synthetic aperture radar image feature information. Finally, based on the feature information fusion image, ground object classification is performed.

[0054] The present application proposes a U-DSDNet feature extraction and fusion classification model for fusion classification of optical images and synthetic aperture radar images, and its process is shown in Figure 2 .

[0055] In the feature extraction stage, the present application adopts a double-branch feature extraction method according to the characteristics of heterogeneous data. This method allows the model to extract image features of synthetic aperture radar images and optical images separately.

[0056] By using a dual-branch feature extraction approach, the model fully leverages the characteristics of these two different data sources, providing rich feature information for subsequent feature fusion and classification. During feature integration and classification, the U-DSDNet (U-Double-Skip-Dense-Network) model (comprising a feature fusion module, a feature extraction network, and a classification module) first simplifies the feature extraction portion of the dual-branch network. Subsequently, the model effectively concatenates the feature maps obtained from the two branches, achieving a preliminary integration of different feature types. This step aims to combine feature information from different data sources to form a more comprehensive and rich feature representation. To further improve feature validity and classification accuracy, the U-DSDNet model also incorporates an attention mechanism. This performs a weighted fusion of input feature information, allowing the U-DSDNet model to focus on important features and suppress the influence of less important information. This process helps the model better capture key information in the fused image, thereby improving classification accuracy and robustness. Finally, after processing by the feature fusion and classification modules, the model outputs a classification result for the feature-fused image. This result is based on a comprehensive analysis and utilization of features from different data sources, enabling effective classification of optical and synthetic aperture radar images. This is done in two steps: feature extraction from optical and synthetic aperture radar images, and fusion classification of these features.

[0057] 1. Feature extraction of optical images and synthetic aperture radar images.

[0058] The imaging principles of synthetic aperture radar images and optical images are different, and the feature expressions for the same information are different. In order to ensure that the feature information of the two images is not confused during the feature extraction process, this application proposes a dual-branch feature extraction strategy and adopts an encoding-decoding structure. Under this strategy and structure, synthetic aperture radar images and optical images are input into the first feature extraction network and the second feature extraction network respectively. They remain independent of each other in the feature extraction stage and do not interfere with each other. Each branch will capture the deep and shallow features of the input image. These feature information is important information when performing classification. The dual-branch feature extraction structure is as follows: Figure 3 shown.

[0059] As an optional implementation, the feature extraction network comprises, in sequence, a first skip dense connection module, a first downsampling transition layer, a second skip dense connection module, a second downsampling transition layer, a third skip dense connection module, a third downsampling transition layer, a fourth skip dense connection module, a first upsampling transition layer, a first feature extraction module, a second upsampling transition layer, a second feature extraction module, a third upsampling transition layer, and a third feature extraction module; the fourth skip dense connection module and the third feature extraction module are connected with the feature fusion module.

[0060] Each branch structure is designed the same and is composed of a downsampling transition layer, an upsampling transition layer, and a skip dense connection module. The specific structure of each branch is shown in Table 1.

[0061] Table 1: Feature extraction network structure table

[0062]

[0063]

[0064] Suppose that the input image size is 4r×4r×C, where C is the number of bands, which is used as the input of the feature extraction network part. When the image enters the feature extraction network, it will first undergo a downsampling process. Each downsampling has 3 skip dense connection modules (Sk-DCM) and 3 downsampling transition layers (D-TL). The skip dense connection module absorbs the idea of skip connection, selects the initial layer and the previous two layers as input, and forms a skip dense network (Skip DenseNet). This modification effectively alleviates the problem of gradient disappearance, further reduces the calculation parameters, improves the running speed of the network, and reduces the time of single cycle. While ensuring the performance of the network, the efficiency of the model is improved. The specific structure is shown in Figure 4 .

[0065] In the downsampling process, the first two skip dense connection modules use 1×1 convolution kernels and 5×5 convolution kernels, respectively, and the number of convolutions of the two is different, consisting of 5 groups and 10 groups, respectively. In order to maintain the stability of the output size of the image, it is realized by adjusting the step of padding. As for the third skip dense connection module, 1×1 convolution kernels and 3×3 convolution kernels are selected, a total of 20 groups. The purpose of this is to more effectively capture deep features in smaller size images.

[0066] The down-sampling transition layer is composed of a series of consecutive operations, including batch normalization (BN), ReLU activation function, 1x1 convolution, and 2x2 max-pooling operation.

[0067] In the up-sampling process, the network adopts a combination of convolution-normalization-activation. This method can more effectively extract and analyze the features of the input image while maintaining the spatial resolution and detail information of the image. Through the up-sampling transition layer, the size of the image is gradually expanded, and finally restored to the same size as the original image. Through deep feature extraction and layer-by-layer up-sampling, the network can generate an output image with the same size as the original image but containing more rich feature information, providing strong support for subsequent image processing and analysis.

[0068] The up-sampling path includes 3 feature extraction modules (Decoding convolution module, DCM) and 3 up-sampling transition layers (Upsampling Transitions Layer, U-TL) to correspond to the down-sampling path. The feature extraction module of the up-sampling part includes two layers of convolution, two layers of BN layer, two layers of RELU activation layer, and reduces the complexity of the model. In the up-sampling path, the output size of the feature extraction module is consistent with the output size of the same module in the down-sampling path, both are 4r×4r×C, ensuring the effective correspondence and fusion between feature maps. The up-sampling transition layer is composed of 3x3 transpose convolution with a step of 2.

[0069] In order to gradually restore the target detail information in the up-sampling process, a skip connection is added between the corresponding layers of the down-sampling path and the up-sampling path to realize the effective combination of shallow feature maps and deep feature maps. The structure of the connection layer is the same as the skip dense connection module in the down-sampling part, which is composed of 1x1 convolution kernel and 3x3 convolution kernel, a total of 15 groups, forming a bottleneck structure for transition.

[0070] Meanwhile, at the overall level, the present application redesigns the skip connection. Taking DCM(1) in Table 1 as an example, the input feature channel not only includes the output feature image from the transition layer Sk-DCM(4) and the output feature image of the same layer Sk-DCM(3) in the down-sampling path, but also includes the output feature image from the higher layer in the down-sampling path after down-sampling as part of its input feature image. Through the comprehensive arrangement of feature information at different levels, the feature extraction network can fully extract the feature information of the input image.

[0071] The advantages of the feature extraction network are as follows:

[0072] (1) Feature reuse. An optimized version of Sk-DenseNet structure is adopted for feature extraction. In the decoding stage, a mechanism is designed to enable each level in the network to establish a connection with the previous level. This mechanism not only realizes the reuse of features, but also promotes the fusion of multi-layer features. This design is beneficial to enhance the robustness of features and help improve the detail accuracy of the segmented image. In this way, the model can make more full use of existing feature information, thereby improving the overall segmentation performance.

[0073] (2) Encoder-decoder structure is adopted. The optimized dense connection network is used in the encoding stage to efficiently extract features. An improved skip connection mechanism is introduced between the corresponding levels of the encoder and the decoder. In the decoding stage, the model integrates various feature information extracted in the encoding stage and performs feature integration again, combining detailed features of the shallow layer with semantic features of the deep layer. By using the transpose convolution operation for upsampling, the model can gradually restore the feature map to the full resolution of the original image to generate an accurate segmentation map.

[0074] The extracted synthetic aperture radar image feature information and optical image feature information are input into the SSAF feature fusion classification model (feature fusion module and classification module).

[0075] 2. Optical image, synthetic aperture radar image feature fusion classification.

[0076] The SSAF feature fusion classification model is used to fuse the synthetic aperture radar image feature information and the optical image feature information and perform classification. The principle is as follows:

[0077] (1) First, the feature information of different levels output by the encoder module (feature extraction network) is added pixel by pixel according to the same level feature, i.e., the optical image feature information and the synthetic aperture radar image feature information are superimposed.

[0078] (2) In the feature integration stage, the self-attention mechanism module is introduced, which plays a key role in identifying and highlighting more critical feature information and giving them greater weight.

[0079] (3) Then, the deep layer features are upsampled to match the dimensions of the shallow layer features. Finally, all the integrated feature maps are merged in the channel layer to serve as the input of the subsequent classifier.

[0080] (4) The integrated and merged feature maps are classified by the softmax classifier.

[0081] As an optional implementation, the feature fusion module comprises: a first self-attention mechanism-based separation fusion gate module, a second self-attention mechanism-based separation fusion gate module and a fourth up-sampling transition layer; the first self-attention mechanism-based separation fusion gate module is connected with the fourth skip dense connection module; the second self-attention mechanism-based separation fusion gate module is connected with the fourth up-sampling transition layer and the third feature extraction module respectively; the self-attention mechanism-based separation fusion gate module and the fourth up-sampling transition layer are connected with the classification module through a feature map concatenation operation.

[0082] In actual application, the overall structure of the SSAF feature fusion classification model is as shown in Figure 5

[0083] The process is represented by formula (1)-formula (3):

[0084]

[0085]

[0086] Wherein, S low represents the shallow features of the SAR image, MS low represents the shallow features of the multi-spectral image (optical image), S high represents the deep features of the SAR image, MS high represents the deep features of the multi-spectral image, φ(·) represents the feature information after the same level fusion of the encoder module, represents the operation through the SSA-Gate feature fusion module, F1 and F2 respectively represent the shallow features and the deep features after the fusion of the decoder module, the same level feature information is superimposed, φ u-Fi represents the information after the up-sampling, the feature information of the deep features is processed and superimposed and then up-sampled to the same scale. C(·) represents the feature map channel concatenation operation, φ F represents the final fused feature map (feature information fusion image).

[0087] Due to the different imaging mechanisms of the synthetic aperture radar sensor and the optical sensor, the expression ways of the feature information are also different. The effect of the feature fusion of the heterogeneous remote sensing images obtained by the previous feature fusion mechanism needs to be improved, while the SSAF feature fusion classification model of the present application can maximize the accuracy and comprehensiveness of the extracted feature information. The present application designs a self-attention mechanism-based separation fusion gate (SSA-Gate) module to complete the fusion of the cross-modal image features, and the structure is as shown in Figure 6 ​​

[0088] The SSA-Gate is an attention gate mechanism. It has two inputs, which are connected to the synthetic aperture radar image feature map and the optical image feature map at the same level, respectively. After the synthetic aperture radar image feature information and the optical image feature information at the same level are input into the SSA-Gate module, the synthetic aperture radar image feature information and the optical image feature information are first spliced together along the channel dimension to form an optical-synthetic aperture radar feature information fusion map. Then, a 1x1 convolution kernel is applied to the optical-synthetic aperture radar feature information fusion map to perform convolution operation, which helps to realize the preliminary integration of features and further obtain a new feature map that integrates various features.

[0089] Down-sampling processing is performed on the merged feature map using average pooling and maximum pooling, and then the feature maps obtained by the two operations are superimposed to generate a new feature map. The latest feature map is passed to self-attention calculation to calculate the attention weight. After reshaping the attention weight and normalizing it using the Softmax function, up-sampling is performed.

[0090] Then, by performing element-wise multiplication of the synthetic aperture radar image feature information and the optical image feature information with the Softmax normalized attention weight, the weights are applied to the synthetic aperture radar image feature map and the optical image feature map. The feature maps with attention weights are spliced together along the channel dimension to generate the final fusion feature to obtain the final output.

[0091] The SSA-Gate module enables the model to effectively learn the important parts in the synthetic aperture radar image and optical image feature maps and give them more attention in the fusion process, which is beneficial in tasks where different feature maps may contain complementary or conflicting information.

[0092] The comparative experiments were carried out on two optical-synthetic aperture radar image datasets, namely the WHU-OPT-SAR dataset and the SEN-OPT-SAR dataset. According to the experimental results, among the four models participating in the experiment, the U-DSDNet model has a leading advantage in overall classification accuracy and Kappa coefficient. In the WHU-OPT-SAR dataset, the U-DSDNet model leads the three models (UNet, DeeplabV3+, and PSPNet) by 6.3%, 4.1%, and 5.5% in overall accuracy, respectively. In the SEN-OPT-SAR dataset, the U-DSDNet model leads the three models by 3.8%, 0.2%, and 1.6% in overall accuracy, respectively.

[0093] The classification results of the WHU-OPT-SAR dataset are shown in Table 1, Table 2, and Table 3. Figure 7 Figure 8 and Table 2.​

[0094] Table 2 Statistics of accuracy and Kappa coefficient of each algorithm in WHU-OPT-SAR dataset

[0095] Class UNet DeeplabV3+ PSPNet U-DSDNet Farmland 0.759 0.816 0.729 0.854 City 0.587 0.627 0.628 0.634 Village 0.704 0.799 0.754 0.669 Water body 0.658 0.713 0.728 0.732 Forest 0.789 0.712 0.818 0.861 Road 0.774 0.656 0.669 0.765 Others 0.592 0.486 0.538 0.598 Overall accuracy 0.737 0.759 0.745 0.800 Kappa coefficient 0.631 0.656 0.645 0.705

[0096] The classification results of the SEN-OPT-SAR dataset are shown in Figure 9 , Figure 10 and Table 3:

[0097] Table 3 Statistics of accuracy and Kappa coefficient of each algorithm in SEN-OPT-SAR dataset

[0098]

[0099] The reasons why the U-DSDNet model achieves better results compared to other methods are as follows:

[0100] SAR images can better distinguish water bodies and other ground objects. For the water body category included in both datasets, the U-DSDNet model ranks first in classification accuracy, thus proving that the model proposed in the present application can better apply synthetic aperture radar images to the multi-source image fusion classification task.

[0101] In tree forest, grassland, farmland and other ground object categories that can be easily detected by multispectral images, the classification effect of the model of the present application is also in the leading position, which proves that the U-DSDNet model does a good job in the distribution of feature information weights.

[0102] Through the feature fusion module and the classification module of the present application, different weights can be adaptively assigned to the feature information extracted from multi-source images according to the different recognition abilities of optical images and synthetic aperture radar images for the same features, thereby achieving more accurate classification.

[0103] The ground object classification method based on optical images and synthetic aperture radar images of the present application includes constructing a double-branch feature information extraction network and designing a feature fusion classification module for optical image and synthetic aperture radar image feature fusion classification, effectively aggregating synthetic aperture radar image feature information and optical image feature information, and the classification result is obviously more accurate and detailed.

[0104] In an exemplary embodiment, a computer device, which can be a server or a terminal, is provided, and an internal structure diagram thereof can be as shown in Figure 11As shown in the figure. The computer device includes a processor, a memory, an input / output interface (I / O for short) and a communication interface. Among them, the processor, the memory and the input / output interface are connected through the system bus, and the communication interface is connected to the system bus through the input / output interface. Among them, the processor of the computer device is used to provide computing and control capability. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The input / output interface of the computer device is used to exchange information between the processor and the external device. The communication interface of the computer device is used to communicate with the terminal outside through the network connection. The computer program is executed by the processor to realize a ground feature classification method based on optical image and synthetic aperture radar image.

[0105] Those skilled in the art can understand that, Figure 11 The structure shown in the figure is only a block diagram of part of the structure related to the scheme of the present application, and does not constitute a limitation on the computer device to which the scheme of the present application is applied. The specific computer device can include more or fewer components than those shown in the figure, or combine certain components, or have a different component arrangement. In one exemplary embodiment, a computer device is provided, including a memory and a processor, the memory storing a computer program, and the processor executing the computer program to realize the above-mentioned ground feature classification method based on optical image and synthetic aperture radar image.

[0106] In one exemplary embodiment, a computer readable storage medium is provided, storing a computer program, which is executed by a processor to realize the above-mentioned ground feature classification method based on optical image and synthetic aperture radar image.

[0107] In one exemplary embodiment, a computer program product is provided, including a computer program, which is executed by a processor to realize the above-mentioned ground feature classification method based on optical image and synthetic aperture radar image.

[0108] It should be noted that the user information (including but not limited to user equipment information, user personal information, etc.) and data (including but not limited to data for analysis, stored data, displayed data, etc.) involved in the present application are all information and data authorized by the user or authorized by all parties, and the collection, use and processing of related data need to comply with relevant regulations.

[0109] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a non-volatile computer readable storage medium, and when the computer program is executed, the processes of the above-mentioned embodiments of the methods can be included. Any reference to memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical storage, high-density embedded non-volatile memory, resistive memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration but not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM), etc.

[0110] The database involved in the embodiments provided in the present application can include at least one of a relational database and a non-relational database. The non-relational database can include a distributed database based on a blockchain, etc., without being limited thereto. The processor involved in the embodiments provided in the present application can be a general-purpose processor, a central processing unit, a graphics processing unit, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., without being limited thereto.

[0111] The technical features of the above embodiments can be combined arbitrarily. To make the description concise, all possible combinations of the technical features in the above embodiments are not described, however, as long as the combinations of the technical features do not exist contradictory, they should be considered as the scope of the present application.

[0112] The principles and implementation modes of the present application are described by applying specific examples herein, and the above-mentioned embodiments are only used to help understand the method and its core idea of the present application; meanwhile, for those skilled in the art, according to the idea of the present application, the specific implementation mode and application range can be changed. In conclusion, the content of the present application should not be understood as a limitation.

Claims

1. A method for ground feature classification based on optical images and synthetic aperture radar images, characterized in that: include: Obtain optical images and synthetic aperture radar images of the area to be classified; According to the optical image and the synthetic aperture radar image, the ground object category of the area to be classified is determined using a ground object classification model; wherein the ground object classification model is obtained by training an initial ground object classification model using a training data set; the training data set includes the optical image and synthetic aperture radar image of the training area and the corresponding ground object category labels; the initial ground object classification model includes a first feature extraction network, a second feature extraction network, a feature fusion module and a classification module; the first feature extraction network and the second feature extraction network are both connected to the feature fusion module; the feature fusion module is connected to the classification module; the first feature extraction network is used to extract deep features and shallow features of the optical image; the second feature extraction network is used to extract deep features and shallow features of the synthetic aperture radar image; the feature fusion module is used to fuse the deep features and shallow features of the optical image and the deep features and shallow features of the synthetic aperture radar image to obtain a feature information fusion image; the classification module is used to classify the ground objects in the area to be classified according to the feature information fusion image Classification, determine the category of the ground object; the feature extraction network includes: a first jump dense connection module, a first downsampling transition layer, a second jump dense connection module, a second downsampling transition layer, a third jump dense connection module, a third downsampling transition layer, a fourth jump dense connection module, a first upsampling transition layer, a first feature extraction module, a second upsampling transition layer, a second feature extraction module, a third upsampling transition layer and a third feature extraction module connected in sequence; the fourth jump dense connection module and the third feature extraction module are both connected to the feature fusion module; the feature fusion module includes: a first separation and fusion gating module based on the self-attention mechanism, a second separation and fusion gating module based on the self-attention mechanism and a fourth upsampling transition layer; the first separation and fusion gating module based on the self-attention mechanism is connected to the fourth jump dense connection module; the second separation and fusion gating module based on the self-attention mechanism is respectively connected to the fourth upsampling transition layer and the third feature extraction module; the separation and fusion gating module based on the self-attention mechanism and the fourth upsampling transition layer are connected to the classification module through a feature map cascade operation.

2. The method for classifying objects based on optical images and synthetic aperture radar images according to claim 1, characterized in that: The classification module is a softmax classifier.

3. The method for classifying objects based on optical images and synthetic aperture radar images according to claim 1, characterized in that: The initial model for object classification is trained using the training dataset, including: Get the training dataset; Inputting the optical image and the synthetic aperture radar image of the training area into the current initial model for ground object classification to obtain the predicted ground object category of the training area; Determining loss function values ​​of the predicted ground object category and the ground object category label of the training area using a loss function; Determine whether the training end condition is met; the training end condition is that the maximum number of iterations is reached or the loss function value is less than a preset value; If yes, the current initial model of land feature classification is used as the land feature classification model; If not, adjust the model parameters of the current initial model for ground object classification according to the loss function value, and return to "input the optical image and synthetic aperture radar image of the training area into the current initial model for ground object classification to obtain the predicted ground object category of the training area." 4. A computer device comprising: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the method for classifying objects based on optical images and synthetic aperture radar images as described in any one of claims 1 to 3.

5. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method for classifying objects based on optical images and synthetic aperture radar images described in any one of claims 1 to 3 is implemented.

6. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the method for classifying objects based on optical images and synthetic aperture radar images described in any one of claims 1 to 3 is implemented.

Images