Ground feature classification method and device based on optical image and synthetic aperture radar image, medium and product

Abstract

The invention discloses a ground object classification method and device based on an optical image and a synthetic aperture radar image, a medium and a product, and relates to the field of ground object classification, and the method comprises the steps: obtaining the optical image and the synthetic aperture radar image of a to-be-classified region; according to the optical image and the synthetic aperture radar image, utilizing a ground object classification model to determine a ground object category of the to-be-classified region; the ground feature classification model is obtained by training a ground feature classification initial model by using a training data set; the surface feature classification initial model comprises a first feature extraction network, a second feature extraction network, a feature fusion module and a classification module; the feature extraction network extracts deep features and shallow features of the synthetic aperture radar image and deep features and shallow features of the optical image; the feature fusion module fuses the extracted deep features and shallow features to obtain a feature information fusion image; and the classification module determines the surface feature category of the to-be-classified region according to the feature information fusion image. The accuracy of ground feature classification is improved.

Description

Technical Field

[0001] The present application relates to the field of land object classification, and in particular to a land object classification method, equipment, medium and product based on optical images and synthetic aperture radar images. Background Art

[0002] Remote sensing images provide a large amount of extremely rich and complex data of the target area. Remote sensing image classification is a key link in the interpretation of remote sensing image targets. With the continuous development of remote sensing technology, its observation method has evolved from a single ground observation method to a multi-platform, multi-source, and multi-scale observation method. Therefore, remote sensing images can provide more comprehensive and accurate information. When classifying images, the use of single-modal data is not enough to meet the needs of actual classification. Therefore, the collaborative processing of optical images and synthetic aperture radar images has attracted more and more attention from researchers. Synthetic aperture radar (SAR) images can show relatively good structural characteristics and have the advantages of all-day, all-weather, and strong penetration, but its clarity is relatively low; optical images are mainly based on spectral information, which makes it easy to classify remote sensing images, but optical sensors are easily affected by climate and observation time when imaging. Fusion of SAR images and optical images can make full use of information in different data types and offset the interference of a single data type; and use information in multiple data types to learn the general laws of ground feature characteristics, improve generalization ability, and improve accuracy.

[0003] However, due to the different imaging principles of the two sensors, there is heterogeneity between the heterogeneous data, different forms of expression for the same ground features, and different data scales. Therefore, using simple feature fusion methods, such as directly adding and classifying multi-source images at the pixel level, does not achieve ideal results. How to fuse and classify optical images and synthetic aperture radar images has become a challenge in the field of image processing. Summary of the invention

[0004] The purpose of this application is to provide a method, device, medium and product for ground object classification based on optical images and synthetic aperture radar images to improve the accuracy of ground object classification.

[0005] To achieve the above objectives, this application provides the following solutions:

[0006] In a first aspect, the present application provides a method for classifying objects based on optical images and synthetic aperture radar images, comprising:

[0007] Obtain optical images and synthetic aperture radar images of the area to be classified;

[0008] According to the optical image and the synthetic aperture radar image, the ground object category of the area to be classified is determined by using a ground object classification model; wherein the ground object classification model is obtained by training the ground object classification initial model using a training data set; the training data set includes the optical image and the synthetic aperture radar image of the training area and the corresponding ground object category label; the ground object classification initial model includes a first feature extraction network, a second feature extraction network, a feature fusion module and a classification module; the first feature extraction network and the second feature extraction network are both connected to the feature fusion module; the feature fusion module is connected to the classification module; the first feature extraction network is used to extract deep features and shallow features of the optical image; the second feature extraction network is used to extract deep features and shallow features of the synthetic aperture radar image; the feature fusion module is used to fuse the deep features and shallow features of the optical image and the deep features and shallow features of the synthetic aperture radar image to obtain a feature information fusion image; the classification module is used to classify the ground objects in the area to be classified according to the feature information fusion image to determine the ground object category.

[0009] Optionally, the feature extraction network includes: a first jump dense connection module, a first downsampling transition layer, a second jump dense connection module, a second downsampling transition layer, a third jump dense connection module, a third downsampling transition layer, a fourth jump dense connection module, a first upsampling transition layer, a first feature extraction module, a second upsampling transition layer, a second feature extraction module, a third upsampling transition layer and a third feature extraction module, which are connected in sequence; the fourth jump dense connection module and the third feature extraction module are both connected to the feature fusion module.

[0010] Optionally, the feature fusion module includes: a first separation and fusion gating module based on the self-attention mechanism, a second separation and fusion gating module based on the self-attention mechanism and a fourth upsampling transition layer; the first separation and fusion gating module based on the self-attention mechanism is connected to the fourth jump dense connection module; the second separation and fusion gating module based on the self-attention mechanism is respectively connected to the fourth upsampling transition layer and the third feature extraction module; the separation and fusion gating module based on the self-attention mechanism and the fourth upsampling transition layer are connected to the classification module through a feature map cascade operation.

[0011] Optionally, the classification module is a softmax classifier.

[0012] Optionally, the initial model for ground object classification is trained using the training data set, specifically including:

[0013] Get the training dataset;

[0014] Inputting the optical image and synthetic aperture radar image of the training area into the current initial model of ground object classification to obtain the predicted ground object category of the training area;

[0015] Determine the loss function value of the ground object prediction category and the ground object category label of the training area by using the loss function;

[0016] Determine whether a training termination condition is met; the training termination condition is that a maximum number of iterations is reached or the loss function value is less than a preset value;

[0017] If yes, the current initial model of land object classification is used as the land object classification model;

[0018] If not, adjust the model parameters of the current initial model for land object classification according to the loss function value, and return to "input the optical image and synthetic aperture radar image of the training area into the current initial model for land object classification to obtain the predicted category of land objects in the training area".

[0019] In a second aspect, the present application provides a computer device, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement any of the above-mentioned methods for classifying objects based on optical images and synthetic aperture radar images.

[0020] In a third aspect, the present application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements any of the above-mentioned methods for classifying objects based on optical images and synthetic aperture radar images.

[0021] In a fourth aspect, the present application provides a computer program product, including a computer program, which, when executed by a processor, implements any of the above-mentioned methods for classifying objects based on optical images and synthetic aperture radar images.

[0022] According to the specific embodiments provided in this application, this application discloses the following technical effects:

[0023] The present application provides a method, device, medium and product for classifying objects based on optical images and synthetic aperture radar images, which obtains optical images and synthetic aperture radar images of the area to be classified; according to the optical images and synthetic aperture radar images, the object classification model is used to determine the object category of the area to be classified; wherein the object classification model is obtained by training the initial object classification model using the training data set; the initial object classification model includes a first feature extraction network, a second feature extraction network, a feature fusion module and a classification module; the first feature extraction network is used to extract deep features and shallow features of the optical image; the second feature extraction network is used to extract deep features and shallow features of the synthetic aperture radar image; the feature fusion module is used to fuse the deep features and shallow features of the optical image and the deep features and shallow features of the synthetic aperture radar image to obtain a feature information fusion image; the classification module is used to classify the objects in the area to be classified according to the feature information fusion image and determine the object category. The present application improves the accuracy of object classification. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0025] Figure 1 A schematic diagram of a flow chart of a method for classifying objects based on optical images and synthetic aperture radar images provided in one embodiment of the present application;

[0026] Figure 2 This is the overall structure diagram of the land feature classification model in this application;

[0027] Figure 3 This is a schematic diagram of the feature extraction network structure in this application;

[0028] Figure 4 This is a schematic diagram of the structure of the jump dense connection module in this application;

[0029] Figure 5 This is a schematic diagram of the SSAF feature fusion classification model structure in this application;

[0030] Figure 6 This is a schematic diagram of the SSA-Gate module structure in this application;

[0031] Figure 7Schematic diagram of the inland forest classification results of the WHU-OPT-SAR dataset; where (a) is the inland forest optical image in the WHU-OPT-SAR dataset; (b) is the inland forest label in the WHU-OPT-SAR dataset; (c) is the inland forest classification results of the WHU-OPT-SAR dataset on the UNet network; (d) is the inland forest classification results of the WHU-OPT-SAR dataset on the DeeplabV3+ network; (e) is the inland forest classification results of the WHU-OPT-SAR dataset on the PSPNet network; (f) is the inland forest classification results of the WHU-OPT-SAR dataset on the U-DSDNet network;

[0032] Figure 8 Schematic diagram of the waterside classification results of the WHU-OPT-SAR dataset; where (a) is the waterside optical image in the WHU-OPT-SAR dataset; (b) is the waterside label in the WHU-OPT-SAR dataset; (c) is a schematic diagram of the waterside classification results of the WHU-OPT-SAR dataset on the UNet network; (d) is a schematic diagram of the waterside classification results of the WHU-OPT-SAR dataset on the DeeplabV3+ network; (e) is a schematic diagram of the waterside classification results of the WHU-OPT-SAR dataset on the PSPNet network; (f) is a schematic diagram of the waterside classification results of the WHU-OPT-SAR dataset on the U-DSDNet network;

[0033] Fig. 9 Schematic diagram of the classification results of a class of optical images of the SEN-OPT-SAR dataset; (a) is a class of optical images in the SEN-OPT-SAR dataset; (b) is a class of optical image labels in the SEN-OPT-SAR dataset; (c) is a schematic diagram of the classification results of a class of optical images of the SEN-OPT-SAR dataset on the UNet network; (d) is a schematic diagram of the classification results of a class of optical images of the SEN-OPT-SAR dataset on the DeeplabV3+ network; (e) is a schematic diagram of the classification results of a class of optical images of the SEN-OPT-SAR dataset on the PSPNet network; (f) is a schematic diagram of the classification results of a class of optical images of the SEN-OPT-SAR dataset on the U-DSDNet network;

[0034] Fig.10Schematic diagram of another type of optical image classification results of the SEN-OPT-SAR dataset; (a) is another type of optical image in the SEN-OPT-SAR dataset; (b) is another type of optical image label in the SEN-OPT-SAR dataset; (c) is a schematic diagram of another type of optical image classification results of the SEN-OPT-SAR dataset on the UNet network; (d) is a schematic diagram of another type of optical image classification results of the SEN-OPT-SAR dataset on the DeeplabV3+ network; (e) is a schematic diagram of another type of optical image classification results of the SEN-OPT-SAR dataset on the PSPNet network; (f) is a schematic diagram of another type of optical image classification results of the SEN-OPT-SAR dataset on the U-DSDNet network;

[0035] Fig.11 A schematic diagram of the structure of a computer device provided in one embodiment of the present application. DETAILED DESCRIPTION

[0036] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0037] In order to make the above-mentioned objects, features and advantages of the present application more obvious and easy to understand, the present application is further described in detail below with reference to the accompanying drawings and specific implementation methods.

[0038] In an exemplary embodiment, Figure 1 As shown, a method for classifying objects based on optical images and synthetic aperture radar images is provided, comprising the following steps:

[0039] S1: Obtain optical images and synthetic aperture radar images of the area to be classified.

[0040] S2: According to the optical image and the synthetic aperture radar image, the ground object category of the area to be classified is determined using a ground object classification model; wherein the ground object classification model is obtained by training the ground object classification initial model using a training data set; the training data set includes the optical image and the synthetic aperture radar image of the training area and the corresponding ground object category labels; the ground object classification initial model includes a first feature extraction network, a second feature extraction network, a feature fusion module and a classification module; the first feature extraction network and the second feature extraction network are both connected to the feature fusion module; the feature fusion module is connected to the classification module; the first feature extraction network is used to extract deep features and shallow features of the optical image; the second feature extraction network is used to extract deep features and shallow features of the synthetic aperture radar image; the feature fusion module is used to fuse the deep features and shallow features of the optical image and the deep features and shallow features of the synthetic aperture radar image to obtain a feature information fusion image; the classification module is used to classify the ground objects in the area to be classified according to the feature information fusion image to determine the ground object category.

[0041] As an optional implementation method, the initial model for ground object classification is trained using a training data set, specifically including:

[0042] Get the training dataset.

[0043] The optical image and the synthetic aperture radar image of the training area are input into the current initial model of ground object classification to obtain the predicted ground object category of the training area.

[0044] The loss function is used to determine the loss function value of the predicted ground object category of the training area and the ground object category label of the training area.

[0045] Determine whether a training termination condition is met; the training termination condition is that a maximum number of iterations is reached or the loss function value is less than a preset value.

[0046] If so, the current initial model of land object classification is used as the land object classification model.

[0047] If not, adjust the model parameters of the current initial model for land object classification according to the loss function value, and return to "input the optical image and synthetic aperture radar image of the training area into the current initial model for land object classification to obtain the predicted category of land objects in the training area".

[0048] In the field of remote sensing, data fusion is a commonly used technology, which aims to integrate information with different spatial and spectral resolutions obtained from different platforms such as satellites, aircraft and ground. After data fusion, the resulting comprehensive data contains richer and more comprehensive information than a single image source. Data fusion methods include pixel-level fusion, feature-level fusion and decision-level fusion. Each fusion method has its own application advantages and application fields. This application adopts the feature-level fusion method, and its structure is as follows: Figure 2 The advantages of feature fusion in this application are:

[0049] (1) Reducing data dimension: The goal of feature-level fusion is to extract and capture the core features in the original data. This process helps to reduce the dimension of the data and thus shorten the processing time required for information integration.

[0050] (2) Retaining most of the information: Feature-level fusion retains most of the information while reducing the data dimension, which helps to improve the processing efficiency and real-time performance of the fusion system.

[0051] (3) Improving the accuracy of decision analysis: Different eigenvalues ​​can be extracted according to different decision analyses. Obtaining the required eigenvalues ​​based on decision analysis helps improve the accuracy of decision analysis.

[0052] (4) Flexibility: Feature-level fusion is applicable to a variety of information sources and has high flexibility.

[0053] The present application inputs synthetic aperture radar images and optical images into a feature extraction network to extract synthetic aperture radar image feature information and optical image feature information respectively. The feature fusion separation module (feature fusion module) is used to fuse the feature information (deep features and shallow features) of the synthetic aperture radar image extracted in the feature extraction network with the feature information of the optical image by weighted merging to obtain a feature information fusion image containing the feature information of the optical image and synthetic aperture radar image. Finally, the ground object classification is performed based on the feature information fusion image.

[0054] The present invention proposes a U-DSDNet feature extraction and fusion classification model to fuse and classify optical images and synthetic aperture radar images. The process is as follows: Figure 2 shown.

[0055] In the feature extraction stage, the present application adopts a dual-branch feature extraction method based on the characteristics of heterogeneous data. This method allows the model to extract the image features of synthetic aperture radar images and optical images respectively.

[0056] Through the dual-branch feature extraction method, the model can make full use of the characteristics of these two different types of data sources to provide rich feature information for subsequent feature fusion and classification. In the process of feature integration and classification, the U-DSDNet (U-Double-Skip-Dense-Network, dual-branch skip connection network model) model (including feature fusion module, feature extraction network and classification module) first simplifies the part responsible for feature extraction in the dual-branch network. Subsequently, the model effectively splices the feature maps obtained from the two branches to achieve a preliminary combination of different types of features. This step aims to combine the feature information of different data sources to form a more comprehensive and rich feature representation. In order to further improve the effectiveness of features and the accuracy of classification, the U-DSDNet model also introduces an attention mechanism. Its approach is to perform weighted fusion of input feature information, so that the U-DSDNet model can pay more attention to important feature information and suppress the influence of secondary information. This process helps the model better capture key information in the fused image, thereby improving the accuracy and robustness of classification. Finally, after processing by the feature fusion module and the classification module, the model will output the classification result of the feature information fused image. This result is based on the comprehensive analysis and utilization of the features of different data sources, thus achieving effective classification of optical images and synthetic aperture radar images. The steps are divided into two steps, namely, feature extraction of optical images and synthetic aperture radar images and feature fusion classification of optical images and synthetic aperture radar images.

[0057] 1. Feature extraction of optical images and synthetic aperture radar images.

[0058] The imaging principles of synthetic aperture radar images and optical images are different, and the feature expressions for the same information are different. In order to ensure that the feature information of the two images is not confused during the feature extraction process, this application proposes a dual-branch feature extraction strategy and adopts an encoding-decoding structure. Under this strategy and structure, the synthetic aperture radar image and the optical image are respectively input into the first feature extraction network and the second feature extraction network. They remain independent of each other and do not interfere with each other during the feature extraction stage. Each branch captures the deep and shallow features of the input image. These feature information is important information when performing classification. The dual-branch feature extraction structure is as follows: Figure 3 shown.

[0059] As an optional implementation, the feature extraction network includes: a first jump dense connection module, a first downsampling transition layer, a second jump dense connection module, a second downsampling transition layer, a third jump dense connection module, a third downsampling transition layer, a fourth jump dense connection module, a first upsampling transition layer, a first feature extraction module, a second upsampling transition layer, a second feature extraction module, a third upsampling transition layer and a third feature extraction module, which are connected in sequence; the fourth jump dense connection module and the third feature extraction module are both connected to the feature fusion module.

[0060] The structural design of each branch is the same, consisting of a downsampling transition layer, an upsampling transition layer and a jump dense connection module. The specific structure of each branch is shown in Table 1.

[0061] Table 1 Feature extraction network structure

[0062]

[0063]

[0064] Assume that the input image size is 4r×4r×C, where C is the number of bands, and is used as the input of the feature extraction network. When the image enters the feature extraction network, it will first undergo a downsampling process. Each downsampling has 3 skip dense connection modules (Skip Dense Convolution Module, Sk-DCM) and 3 downsampling transition layers (Downsampling Transition Layer, D-TL). The skip dense connection module absorbs the idea of ​​skip connection and selects the initial layer and the first two layers as input to form a skip dense connection (Skip DenseNet). This modification not only effectively alleviates the problem of gradient disappearance, but also further reduces the calculation parameters, improves the running speed of the network, and reduces the time of a single cycle. While ensuring network performance, it also improves the efficiency of the model. The specific structure is as follows Figure 4 shown.

[0065] In the process of downsampling, the first two jump dense connection modules use 1×1 convolution kernel and 5×5 convolution kernel respectively. The number of convolutions is different between the two modules, which are composed of 5 and 10 groups respectively. In order to maintain the stability of the output size of the image, the padding step size is adjusted. As for the third jump dense connection module, 1×1 convolution kernel and 3×3 convolution kernel are selected, with a total of 20 groups. The purpose of this is to more effectively capture deep features in smaller-sized images.

[0066] The downsampling transition layer consists of a series of consecutive operations, including batch normalization (BatchNormalization, BN), ReLU activation function, 1×1 convolution, and 2×2 maximum pooling operations.

[0067] During the upsampling process, the network uses a combination of convolution-normalization-activation. This method can more effectively extract and analyze the features of the input image while maintaining the spatial resolution and detail information of the image. Through the upsampling transition layer, the size of the image is gradually expanded and finally restored to the same size as the original image. Through deep feature extraction and layer-by-layer upsampling, the network can generate an output image with the same size as the original image but with richer feature information, providing strong support for subsequent image processing and analysis.

[0068] The upsampling path includes three feature extraction modules (Decoding convolution module, DCM) and three upsampling transition layers (Upsampling Transitions Layer, U-TL) to correspond to the downsampling path. The feature extraction module of the upsampling part includes two layers of convolution, two layers of BN layers, and two layers of RELU activation layers to reduce the complexity of the model. In the upsampling path, the output size of the feature extraction module is consistent with the output size of the same module in the downsampling path, both of which are 4r×4r×C, ensuring effective correspondence and fusion between feature maps. The upsampling transition layer is composed of 3×3 transposed convolutions with a stride of 2.

[0069] In order to gradually restore the target detail information during the upsampling process, a jump connection is added between the corresponding layers of the downsampling path and the upsampling path to effectively combine the shallow feature map with the deep feature map. The structure of the connection layer is the same as the jump dense connection module of the downsampling part, consisting of 1×1 convolution kernels and 3×3 convolution kernels, a total of 15 groups, forming a bottleneck structure for transition.

[0070] At the same time, at the overall level, the present application redesigns the jump connection. Taking DCM(1) ​​in Table 1 as an example, its input feature channel not only includes the output feature image of Sk-DCM(4) from the transition layer and the output feature image of Sk-DCM(3) of the same layer in the downsampling path, but also includes the output feature image from the higher layer in the downsampling path after downsampling as part of its input feature image. Through the comprehensive arrangement of feature information at different levels, the feature extraction network can fully extract the feature information of the input image.

[0071] Some advantages of feature extraction network are as follows:

[0072] (1) Feature reuse. An optimized version of the Sk-DenseNet structure is used for feature extraction. In the decoding stage, each layer in the network is designed to establish a connection with the previous layer. This mechanism not only achieves feature reuse, but also facilitates the fusion of multi-layer features. This design is conducive to enhancing the robustness of features and helps to improve the detail of segmented images. In this way, the model can make more full use of existing feature information, thereby improving the overall segmentation performance.

[0073] (2) An encoder-decoder structure is adopted. The encoding stage uses an optimized densely connected network to efficiently extract features. An improved jump connection mechanism is introduced between the corresponding layers of the encoder and decoder. In the decoding stage, the model integrates various feature information extracted in the encoding stage and performs another feature integration, combining shallow detailed features with deep semantic features. By using transposed convolution operations for upsampling, the model can gradually restore the feature map to the full resolution of the original image to generate an accurate segmentation map.

[0074] The extracted synthetic aperture radar image feature information and optical image feature information are input into the SSAF feature fusion classification model (feature fusion module and classification module).

[0075] 2. Fusion and classification of optical image and synthetic aperture radar image features.

[0076] The function of the SSAF feature fusion classification model is to fuse the feature information of synthetic aperture radar images and optical images and classify them. Its principle is as follows:

[0077] (1) First, the feature information of different levels output by the encoder module (feature extraction network) is added pixel by pixel according to the features of the same level, that is, the optical image feature information and the synthetic aperture radar image feature information are superimposed.

[0078] (2) In the stage of feature information integration, the self-attention mechanism module is introduced. Its core function is to identify and highlight the more critical feature information and give them greater weight.

[0079] (3) Then, the deep features are upsampled to match their dimensions with the shallow features. Finally, all the integrated feature maps are merged at the channel level as the input of the subsequent classifier.

[0080] (4) The integrated and merged feature maps are classified through a softmax classifier.

[0081] As an optional implementation, the feature fusion module includes: a first separation and fusion gating module based on a self-attention mechanism, a second separation and fusion gating module based on a self-attention mechanism, and a fourth upsampling transition layer; the first separation and fusion gating module based on the self-attention mechanism is connected to the fourth jump dense connection module; the second separation and fusion gating module based on the self-attention mechanism is respectively connected to the fourth upsampling transition layer and the third feature extraction module; the separation and fusion gating module based on the self-attention mechanism and the fourth upsampling transition layer are connected to the classification module through a feature map cascade operation.

[0082] In practical applications, the overall structure of the SSAF feature fusion classification model is as follows: Figure 5 shown.

[0083] The process is expressed by formula (1)-formula (3):

[0084]

[0085]

[0086] Among them, S low Represents the shallow features of SAR images, MS low Represents the shallow features of multispectral images (optical images), S high Represents the deep features of SAR images, MS high represents the deep features of multispectral images, φ(·) represents the feature information after the encoder module is fused at the same level, Indicates the operation through the SSA-Gate feature fusion module. F1 and F2 represent the shallow features and deep features fused by the decoder module, respectively. The feature information at the same level is superimposed. u-Fi represents the upsampled information, which processes and superimposes the feature information of the deep features and then upsamples them to the same scale. C(·) represents the feature map channel cascade operation, φ F Represents the final fused feature map (feature information fusion image).

[0087] Since the imaging mechanisms of synthetic aperture radar sensors and optical sensors are different, the expression methods of their feature information are also different. The effect of using the previous feature fusion mechanism to fuse the features of heterogeneous remote sensing images needs to be improved. The SSAF feature fusion classification model of this application can maximize the accuracy and comprehensiveness of the extracted feature information. This application designs a separation fusion gating (Self-Separation-and-AggregationGate, SSA-Gate) module based on the self-attention mechanism to complete the fusion of cross-modal image features. Its structure is as follows Figure 6 shown.

[0088] SSA-Gate is an attention gate mechanism. It has two input ports, which are connected to the synthetic aperture radar image feature map and the optical image feature map at the same level. When the synthetic aperture radar image feature information and the optical image feature information at the same level are input into the SSA-Gate module, it first splices the synthetic aperture radar image feature information and the optical image feature information together along the channel dimension to form an optical-synthetic aperture radar feature information fusion map. Then, a 1×1 convolution kernel is applied to the spliced ​​optical-synthetic aperture radar feature information fusion map for convolution operation. This step helps to achieve the initial integration of features, and then obtain a new feature map that integrates all aspects of features.

[0089] The merged feature map is downsampled using average pooling and max pooling, and then the feature maps obtained by these two operations are superimposed to generate a new feature map. The latest feature map is passed to the self-attention calculation to calculate the attention weight. The attention weight is reshaped and normalized using the Softmax function before upsampling.

[0090] These weights are then applied to the SAR image feature map and the optical image feature map by element-wise multiplication of the SAR image feature information and the optical image feature information with the Softmax normalized attention weights. The feature maps with attention weights are concatenated along the channel dimension to generate the final fused features to obtain the final output.

[0091] The SSA-Gate module enables the model to effectively learn the important parts of the SAR and optical image feature maps and give them more attention during the fusion process, which is beneficial in tasks where different feature maps may contain complementary or conflicting information.

[0092] Comparative experiments were conducted on two optical-synthetic aperture radar image datasets, namely the WHU-OPT-SAR dataset and the SEN-OPT-SAR dataset. The experimental results show that among the four models participating in the experiment, the U-DSDNet model has a leading advantage in overall classification accuracy and Kappa coefficient. In the WHU-OPT-SAR dataset, the U-DSDNet model leads the three models (UNet, DeeplabV3+, PSPNet) by 6.3%, 4.1% and 5.5% in overall accuracy respectively; in the SEN-OPT-SAR dataset, the U-DSDNet model leads the three models by 3.8%, 0.2% and 1.6% in overall accuracy respectively.

[0093] The classification results of the WHU-OPT-SAR dataset are as follows Figure 7 , Figure 8 As shown in Table 2:

[0094] Table 2 Statistics of the experimental results of each algorithm accuracy and Kappa coefficient of the WHU-OPT-SAR dataset

[0095] category UNet DeeplabV3+ PSPNet U-DSDNet farmland 0.759 0.816 0.729 0.854 City 0.587 0.627 0.628 0.634 village 0.704 0.799 0.754 0.669 Water 0.658 0.713 0.728 0.732 forest 0.789 0.712 0.818 0.861 the way 0.774 0.656 0.669 0.765 other 0.592 0.486 0.538 0.598 Overall accuracy 0.737 0.759 0.745 0.800 Kappa coefficient 0.631 0.656 0.645 0.705

[0096] The classification results of the SEN-OPT-SAR dataset are as follows Fig. 9 , Fig.10 As shown in Table 3:

[0097] Table 3 Statistics of the experimental results of each algorithm accuracy and Kappa coefficient of SEN-OPT-SAR dataset

[0098]

[0099] Compared with other methods, the U-DSDNet model achieves better results for the following reasons:

[0100] SAR images can better distinguish water bodies from other landforms. For the water body categories included in both datasets, the U-DSDNet model ranks first among the four models in classification accuracy. This proves that the model proposed in this application can better apply synthetic aperture radar images to participate in multi-source image fusion classification tasks.

[0101] The classification effect of the model in this application is also in the leading position in the categories of objects that are easily detected by multispectral images, such as forests, grasslands, and cultivated land, proving that the U-DSDNet model performs well in the distribution of feature information weights.

[0102] Through the feature fusion module and classification module of the present application, different weights can be adaptively assigned to the feature information extracted from multi-source images based on the different recognition capabilities of optical images and synthetic aperture radar images for the same features, thereby achieving more accurate classification.

[0103] The object classification method based on optical images and synthetic aperture radar images of the present application includes constructing a dual-branch feature information extraction network and designing a feature fusion classification module for optical image and synthetic aperture radar image feature fusion classification, which effectively aggregates synthetic aperture radar image feature information and optical image feature information, and the classification results are significantly more accurate and detailed.

[0104] In an exemplary embodiment, a computer device is provided. The computer device may be a server or a terminal. The internal structure diagram thereof may be as follows: Fig.11As shown. The computer device includes a processor, a memory, an input / output interface (Input / Output, referred to as I / O) and a communication interface. The processor, the memory and the input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a method for classifying objects based on optical images and synthetic aperture radar images is implemented.

[0105] Those skilled in the art will understand that Fig.11 The structure shown in the figure is only a block diagram of a part of the structure related to the present application scheme, and does not constitute a limitation on the computer device to which the present application scheme is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components. In an exemplary embodiment, a computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and when the processor executes the computer program, the above-mentioned method for classifying objects based on optical images and synthetic aperture radar images is implemented.

[0106] In an exemplary embodiment, a computer-readable storage medium is provided, storing a computer program, which, when executed by a processor, implements the above-mentioned method for classifying objects based on optical images and synthetic aperture radar images.

[0107] In an exemplary embodiment, a computer program product is provided, including a computer program, which implements the above-mentioned method for classifying objects based on optical images and synthetic aperture radar images when executed by a processor.

[0108] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant regulations.

[0109] Those of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM may be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM).

[0110] The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. The non-relational database may include a distributed database based on blockchain, etc., but is not limited thereto. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., but is not limited thereto.

[0111] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0112] This article uses specific examples to illustrate the principles and implementation methods of this application. The description of the above embodiments is only used to help understand the method and core ideas of this application. At the same time, for those skilled in the art, according to the ideas of this application, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as limiting this application.

Claims

1. A method for classifying objects based on optical images and synthetic aperture radar images, characterized in that: include: Obtain optical images and synthetic aperture radar images of the area to be classified; According to the optical image and the synthetic aperture radar image, the ground object category of the area to be classified is determined by using a ground object classification model; wherein the ground object classification model is obtained by training the ground object classification initial model using a training data set; the training data set includes the optical image and the synthetic aperture radar image of the training area and the corresponding ground object category label; the ground object classification initial model includes a first feature extraction network, a second feature extraction network, a feature fusion module and a classification module; the first feature extraction network and the second feature extraction network are both connected to the feature fusion module; the feature fusion module is connected to the classification module; the first feature extraction network is used to extract deep features and shallow features of the optical image; the second feature extraction network is used to extract deep features and shallow features of the synthetic aperture radar image; the feature fusion module is used to fuse the deep features and shallow features of the optical image and the deep features and shallow features of the synthetic aperture radar image to obtain a feature information fusion image; the classification module is used to classify the ground objects in the area to be classified according to the feature information fusion image to determine the ground object category.

2. The method for classifying objects based on optical images and synthetic aperture radar images according to claim 1, characterized in that: The feature extraction network includes: a first jump dense connection module, a first downsampling transition layer, a second jump dense connection module, a second downsampling transition layer, a third jump dense connection module, a third downsampling transition layer, a fourth jump dense connection module, a first upsampling transition layer, a first feature extraction module, a second upsampling transition layer, a second feature extraction module, a third upsampling transition layer and a third feature extraction module, which are connected in sequence; the fourth jump dense connection module and the third feature extraction module are both connected to the feature fusion module.

3. The method for classifying objects based on optical images and synthetic aperture radar images according to claim 2, characterized in that: The feature fusion module includes: a first separation and fusion gating module based on the self-attention mechanism, a second separation and fusion gating module based on the self-attention mechanism and a fourth upsampling transition layer; the first separation and fusion gating module based on the self-attention mechanism is connected to the fourth jump dense connection module; the second separation and fusion gating module based on the self-attention mechanism is respectively connected to the fourth upsampling transition layer and the third feature extraction module; the separation and fusion gating module based on the self-attention mechanism and the fourth upsampling transition layer are connected to the classification module through a feature map cascade operation.

4. The method for classifying objects based on optical images and synthetic aperture radar images according to claim 1, characterized in that: The classification module is a softmax classifier.

5. The method for classifying objects based on optical images and synthetic aperture radar images according to claim 1, characterized in that: The initial model of ground object classification is trained using the training data set, including: Get the training dataset; Inputting the optical image and synthetic aperture radar image of the training area into the current initial model of ground object classification to obtain the predicted ground object category of the training area; Determine the loss function value of the ground object prediction category and the ground object category label of the training area by using the loss function; Determine whether a training termination condition is met; the training termination condition is that a maximum number of iterations is reached or the loss function value is less than a preset value; If yes, the current initial model of land object classification is used as the land object classification model; If not, the model parameters of the current initial model for land object classification are adjusted according to the loss function value, and "the optical image and synthetic aperture radar image of the training area are input into the current initial model for land object classification to obtain the predicted category of land objects in the training area" is returned.

6. A computer device comprising: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the method for classifying objects based on optical images and synthetic aperture radar images as described in any one of claims 1 to 5.

7. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method for classifying objects based on optical images and synthetic aperture radar images described in any one of claims 1 to 5 is implemented.

8. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the method for classifying objects based on optical images and synthetic aperture radar images described in any one of claims 1 to 5 is implemented.

Images