Dual-channel-based multi-modal feature fusion remote sensing image classification method and device

Abstract

The invention provides a dual-channel-based multi-modal feature fusion remote sensing image classification method and device, and relates to the technical field of remote sensing image processing and analysis. Comprising the following steps: preprocessing remote sensing images of different modes and combining the images to obtain a plurality of image groups; a preset classification model is constructed, the preset classification model comprises a backbone network, a fusion module, a gating mechanism layer and a decoding layer which are connected in sequence, and the structure type of the backbone network comprises a twin backbone network, an isomorphic backbone network and a heterogeneous backbone network; according to the plurality of image groups, selecting one of a twin backbone network, an isomorphic backbone network and a heterogeneous backbone network from a preset classification model as a backbone network to obtain a corresponding target classification model; training the target classification model through the training set to obtain a trained target classification model; and inputting the verification set into the trained target classification model, and outputting a classification result. In this way, the accuracy of remote sensing image classification is high.

Description

Technical Field

[0001] The present invention relates to the technical field of remote sensing image processing and analysis, and in particular to a dual-channel based multi-modal feature fusion remote sensing image classification method and device. Background Art

[0002] Remote sensing image classification is a crucial technology in modern earth observation and environmental monitoring. The main purpose of remote sensing image classification is to extract useful information from remote sensing images and apply the useful information to multiple fields such as land use, environmental management and disaster monitoring. Traditional remote sensing image classification methods usually rely on a single type of data, such as optical images or radar data. Although traditional remote sensing image classification methods perform well in some applications, they may face challenges when dealing with complex surface features and environmental conditions due to the limitations of a single data source.

[0003] At present, the traditional remote sensing image classification method usually adopts the traditional machine learning method to perform the classification task of optical images. However, the traditional machine learning method is not flexible enough when dealing with multi-scale features, and it is difficult to capture the complex relationships and high-level abstract features in the image, which limits the full use of contextual information and feature extraction, resulting in low accuracy of remote sensing image classification. Summary of the invention

[0004] The purpose of the embodiments of the present invention is to provide a dual-channel multimodal feature fusion remote sensing image classification method and device to solve the problem of low accuracy in remote sensing image classification.

[0005] To solve the above technical problems, the embodiments of the present invention provide the following technical solutions:

[0006] The first aspect of the present invention provides a dual-channel multimodal feature fusion remote sensing image classification method, the method comprising:

[0007] Remote sensing images of different modalities are acquired, and the remote sensing images of different modalities are preprocessed and combined in sequence to obtain a plurality of image groups, and the plurality of image groups are divided into a training set and a validation set, and each image group is a combination of remote sensing images of two different modalities;

[0008] Constructing a preset classification model, the preset classification model includes a backbone network, a fusion module, a gating mechanism layer and a decoding layer connected in sequence, the structural types of the backbone network include a twin backbone network, a homogeneous backbone network and a heterogeneous backbone network, the twin backbone network includes two identical encoders with the same weights and model parameters, the homogeneous backbone network includes two identical encoders with different weights and model parameters, and the heterogeneous backbone network includes two different encoders with different weights and model parameters;

[0009] According to multiple image groups in the training set, one of the twin backbone network, the homogeneous backbone network and the heterogeneous backbone network is selected in the preset classification model as the backbone network to obtain the corresponding target classification model;

[0010] The target classification model is trained by using the training set to obtain a trained target classification model;

[0011] Input the validation set into the trained target classification model, use the trained target classification model to classify the validation set, and output the classification results.

[0012] A second aspect of the present invention provides a dual-channel multimodal feature fusion remote sensing image classification device, the device comprising:

[0013] A division unit is used to obtain remote sensing images of different modalities, and pre-process and combine the remote sensing images of different modalities in sequence to obtain multiple image groups, and divide the multiple image groups into a training set and a verification set, each image group is a combination of remote sensing images of two different modalities;

[0014] A construction unit, used to construct a preset classification model, the preset classification model includes a backbone network, a fusion module, a gating mechanism layer and a decoding layer connected in sequence, the structural types of the backbone network include a twin backbone network, a homogeneous backbone network and a heterogeneous backbone network, the twin backbone network includes two identical encoders with the same weights and model parameters, the homogeneous backbone network includes two identical encoders with different weights and model parameters, and the heterogeneous backbone network includes two different encoders with different weights and model parameters;

[0015] A selection unit is used to select one of a twin backbone network, a homogeneous backbone network and a heterogeneous backbone network in a preset classification model as a backbone network according to multiple image groups in a training set, so as to obtain a corresponding target classification model;

[0016] A training unit, used for training the target classification model through a training set to obtain a trained target classification model;

[0017] The classification unit is used to input the verification set into the trained target classification model, classify the verification set using the trained target classification model, and output the classification result.

[0018] Compared with the prior art, the dual-channel multimodal feature fusion remote sensing image classification method and device provided by the present invention obtain remote sensing images of different modalities, and pre-process and combine the remote sensing images of different modalities in sequence to obtain multiple image groups, and divide the multiple image groups into training sets and verification sets; construct a preset classification model, the preset classification model includes a backbone network, a fusion module, a gating mechanism layer and a decoding layer connected in sequence, the structural types of the backbone network include a twin backbone network, an isomorphic backbone network and a heterogeneous backbone network, the twin backbone network includes two identical encoders with the same weights and model parameters, and the same The twin backbone network includes two identical encoders with different weights and model parameters, and the heterogeneous backbone network includes two different encoders with different weights and model parameters; according to multiple image groups in the training set, one of the twin backbone network, the isomorphic backbone network and the heterogeneous backbone network is selected in the preset classification model as the backbone network to obtain the corresponding target classification model; the target classification model is trained with the training set to obtain the trained target classification model; the verification set is input into the trained target classification model, the trained target classification model is used to classify the verification set, and the classification result is output. In this way, the use of remote sensing images of different modalities can utilize the complementary information between different modalities, understand the data more comprehensively, and enhance the representation ability of the model; the preset classification model can support three different target classification models corresponding to the twin backbone network, the isomorphic backbone network, and the heterogeneous backbone network, as well as two-way cross-multi-scale fusion through the fusion modules in the three different target classification models, so that information can be exchanged between different feature maps, which helps different target classification models to learn richer feature representations, better capture the correlation between multimodal remote sensing images, and make full use of the complementarity of each modality information, so that the accuracy of remote sensing image classification is higher. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] By reading the detailed description below with reference to the accompanying drawings, the above and other objects, features and advantages of the exemplary embodiments of the present invention will become readily understood. In the accompanying drawings, several embodiments of the present invention are shown in an exemplary and non-limiting manner, and the same or corresponding reference numerals represent the same or corresponding parts, wherein:

[0020] Figure 1 The flowchart of the multimodal feature fusion remote sensing image classification method based on dual channels is schematically shown;

[0021] Figure 2 The execution diagram of the fusion module is schematically shown;

[0022] Figure 3 Schematically showing an execution diagram of a first target classification model;

[0023] Figure 4A diagram schematically shows a training target classification model diagram based on image input classification;

[0024] Figure 5 A schematic diagram of weighted fusion is shown schematically;

[0025] Figure 6 The structure of a dual-channel multimodal feature fusion remote sensing image classification device is schematically shown. DETAILED DESCRIPTION

[0026] The exemplary embodiments of the present invention will be described in more detail below with reference to the accompanying drawings. Although the exemplary embodiments of the present invention are shown in the accompanying drawings, it should be understood that the present invention can be implemented in various forms and should not be limited by the embodiments described herein. On the contrary, these embodiments are provided in order to enable a more thorough understanding of the present invention and to fully convey the scope of the present invention to those skilled in the art.

[0027] It should be noted that, unless otherwise specified, the technical terms or scientific terms used in the present invention should have the common meanings understood by those skilled in the art to which the present invention belongs.

[0028] The method in the embodiment of the present invention is described in detail below.

[0029] Figure 1 The flowchart of the multi-modal feature fusion remote sensing image classification method based on dual channels in an embodiment of the present invention is schematically shown. Figure 1 As shown, the method may include:

[0030] S101, acquiring remote sensing images of different modalities, and sequentially preprocessing and combining the remote sensing images of different modalities to obtain multiple image groups, and dividing the multiple image groups into a training set and a validation set.

[0031] Each image group is a combination of remote sensing images of two different modalities.

[0032] Remote sensing images of different modalities can be obtained through the Landsat 8 satellite launched in 2013 by the National Aeronautics and Space Administration (NASA), the Sentinel satellite series of the European Space Agency (ESA), and high-resolution cameras carried by drones. Remote sensing images of different modalities include synthetic aperture radar images, infrared images, optical images, and multispectral images. All remote sensing images should be acquired within the growing season of the designated study area, and all acquired remote sensing images need to cover the same study area to ensure that remote sensing images of different modalities come from the same geographical location.

[0033] Synthetic Aperture Radar (SAR) images have a spatial resolution of at least 5 meters and at least one of the following polarization modes: horizontal transmission and horizontal reception (HH), horizontal transmission and vertical reception (HV), vertical transmission and horizontal reception (VH), and vertical transmission and vertical reception (VV). SAR can provide high-resolution surface information all day and all weather, and is particularly suitable for data collection under conditions of severe cloud cover or at night. It can penetrate vegetation and soil surface to provide surface and subsurface information.

[0034] The infrared images obtained by infrared sensors have a spatial resolution of at least 30 meters and include thermal infrared bands. They can detect thermal radiation information and are suitable for temperature change monitoring, fire detection, and thermal pollution monitoring. They can distinguish between different materials and surface features.

[0035] Optical images provide rich visible spectrum information and should include RGB bands. They are suitable for natural resource monitoring, urban planning and land use analysis. They can intuitively reflect the shape, color and texture characteristics of surface objects and are widely used in ecological environment protection, agriculture and water resources management.

[0036] Multispectral images contain information from multiple spectral bands with a spatial resolution of at least 10 meters, including at least red, green, blue and near-infrared bands. They can reflect the spectral characteristics of surface objects and are suitable for precision agriculture, forest monitoring and mineral exploration. Through analysis of different band combinations, they can reveal plant health, soil type and water quality, and are widely used in environmental monitoring and resource management.

[0037] Preprocessing remote sensing images of different modalities can ensure that remote sensing images of different modalities can be effectively fused in subsequent processing.

[0038] Specifically, remote sensing images of different modalities are preprocessed and combined in sequence to obtain multiple image groups, including:

[0039] Step A1: perform spatial alignment, time point pairing, time alignment, image cropping and denoising on remote sensing images of different modalities in sequence to obtain denoised remote sensing images.

[0040] To align remote sensing images of different modalities in space, image registration technology can be used to ensure that remote sensing images of different modalities are aligned in the same geographic space. Image registration technology includes feature point-based registration, phase correlation method, and transformation model-based registration, which ensures that the pixels of remote sensing images of different modalities at the same geographic location correspond to each other.

[0041] The time point pairing of the remote sensing images after spatial alignment can be ensured by timestamp matching and interpolation methods. The closest time point is selected for pairing first. The time point paired remote sensing images acquired at different times are time aligned by linear interpolation and spline interpolation methods to ensure that the surface information corresponding to the remote sensing images of each modality is consistent.

[0042] The remote sensing images of different modalities after time alignment are cropped to the same size, which can be 512*512. The cropping size is not specifically limited here. The cropped remote sensing images are denoised using median filtering.

[0043] Step A2: geometric transformation, color transformation and noise injection are performed on the denoised remote sensing image in sequence to obtain a preprocessed remote sensing image.

[0044] Generate new remote sensing images through geometric transformation (rotation, flipping, cropping) and color transformation (color jittering, brightness adjustment, contrast adjustment), as well as noise injection (Gaussian noise), which can increase the diversity of remote sensing images and provide rich training samples during the training process. Rotation in geometric transformation can change the direction of the denoised remote sensing image. Flipping in geometric transformation can change the structure of the denoised remote sensing image. Cropping in geometric transformation can focus on a specific area. Color jittering can change the color distribution, brightness adjustment can change the brightness of the image, and contrast adjustment can change the contrast of the remote sensing image. Gaussian noise injection simulates the noise in actual shooting. The injection of Gaussian noise provides rich and diverse data samples for the preset classification model, improves the denoising ability of the preset classification model, and thus improves the robustness of the trained preset classification model.

[0045] Step A3: In the preprocessed remote sensing images, two remote sensing images of different modalities are taken as one image group to obtain a plurality of image groups.

[0046] It is necessary to ensure that the remote sensing images of two different modes in multiple image groups are the same samples in different modes. According to the specific application requirements, two remote sensing images of different modes are arbitrarily selected as an image group, and all preprocessed remote sensing images are divided into image groups to obtain multiple image groups. The multiple image groups are used as the input of the subsequent classification model. The multiple image groups can be a combination of optical images and SAR images, optical images and infrared images, SAR images and infrared images, optical images and multispectral images, SAR images and multispectral images, and infrared images and multispectral images. There are 6 combinations of multiple image groups. Among them, optical images provide visible light information of ground objects, SAR images provide microwave reflection information of ground objects, infrared images provide thermal radiation information, and multispectral images provide spectral information of multiple bands.

[0047] Four data sources (i.e., synthetic aperture radar images, infrared images, optical images, and multispectral images) of 5,000 each can be prepared, and two remote sensing images of different modalities can be taken as one image group to obtain multiple image groups. From the six combinations of multiple image groups of each modality, 80% of the data are randomly selected as training sets, and the remaining 20% ​​of the data are used as validation sets.

[0048] The use of remote sensing images of different modalities enables the characteristics of each modality to complement each other and capture diverse features.

[0049] S102: Build a preset classification model.

[0050] Among them, the preset classification model includes a backbone network, a fusion module, a gating mechanism layer and a decoding layer connected in sequence. The structural types of the backbone network include a twin backbone network, an isomorphic backbone network and a heterogeneous backbone network. The twin backbone network includes two identical encoders with the same weights and model parameters. The isomorphic backbone network includes two identical encoders with different weights and model parameters. The heterogeneous backbone network includes two different encoders with different weights and model parameters.

[0051] The twin backbone network, homogeneous backbone network and heterogeneous backbone network are all dual-channel collaborative encoding networks. The twin backbone network, homogeneous backbone network and heterogeneous backbone network can be selected from the following encoders: Multi-scale Interactive Image Transformer (MiT), Improved Visual Image Transformer (PVTv2) based on Pyramid Structure, Vision Transformer (ViT), Swin Transformer, Improved Swin Transformer (Swin Transformer V2), Visual Large Model Based on Deformable Convolution (InternImage), Visual State Space Model (VMamba), Convolutional Network with Fusion Transformer Structure (ConvNeXt), Large Kernel Convolutional Sequence Network (LSKNet) and Large Kernel Convolutional Network Across Multimodal (UniRepLKNet).

[0052] Among them, the fusion module includes a first multi-head attention layer, a first normalization layer, a first multi-layer perception mechanism layer, a second normalization layer, a second multi-head attention layer, a third normalization layer, a second multi-layer perception mechanism layer, a fourth normalization layer, a splicing layer and a first 1×1 convolutional layer connected in sequence.

[0053] The gating mechanism layer includes the second 1×1 convolution layer, the ReLU activation function layer, the third 1×1 convolution layer, the Tanh activation function layer and the dot product layer. The decoding layer includes a fully connected layer.

[0054] S103. According to multiple image groups in the training set, one of the twin backbone network, the isomorphic backbone network and the heterogeneous backbone network is selected in the preset classification model as the backbone network to obtain the corresponding target classification model.

[0055] Among them, the multiple image groups in the training set include multiple first image groups, multiple second image groups and multiple third image groups, the multiple first image groups include a combination of optical images and SAR images and a combination of optical images and infrared images, the multiple second image groups include a combination of SAR images and infrared images and a combination of optical images and multispectral images, and the multiple third image groups include a combination of SAR images and multispectral images and a combination of infrared images and multispectral images.

[0056] Specifically, according to multiple image groups in the training set, one of the twin backbone network, the homogeneous backbone network and the heterogeneous backbone network is selected in the preset classification model as the backbone network to obtain the corresponding target classification model, including:

[0057] Step B1: When the image groups in the training set are a combination of optical images and SAR images and a combination of optical images and infrared images, a twin backbone network is selected as the backbone network to obtain a corresponding first target classification model.

[0058] The combination of optical images and SAR images, as well as the combination of optical images and infrared images, are highly correlated between modalities.

[0059] The twin backbone network specifically builds two parameter-sharing branches to ensure that the two modal data undergo consistent feature extraction. The same convolutional neural network structure is used to build two branches, and the two branches share the same weight parameters. Each branch is responsible for processing data of one modality, ensuring that the two modal data have a consistent processing method during the feature extraction process, capturing the similarities and differences between different modal data, and is suitable for similarity detection and matching tasks.

[0060] The first target classification model includes a twin backbone network, a fusion module, a gating mechanism layer and a decoding layer, and the twin backbone network includes two identical encoders.

[0061] Step B2: When the image group in the training set is a combination of SAR images and infrared images and a combination of optical images and multispectral images, a homogeneous backbone network is selected as the backbone network to obtain a corresponding second target classification model.

[0062] The combination of SAR images and infrared images, as well as the combination of optical images and multispectral images, belong to the modes with similar correlation.

[0063] The homogeneous backbone network uses the same but independent network structure to process two modal data and extract their respective features. The two branches use independent weight parameters, and each branch is responsible for processing data of one modality, allowing the network to independently learn and optimize according to the characteristics of different modal data. Before feature fusion, the features of each modal data are fully processed in an independent network structure to ensure the integrity and effectiveness of the features.

[0064] The second target classification model includes a homogeneous backbone network, a fusion module, a gating mechanism layer and a decoding layer, and the homogeneous backbone network includes two identical encoders.

[0065] Step B3: When the image groups in the training set are a combination of SAR images and multispectral images and a combination of infrared images and multispectral images, a heterogeneous backbone network is selected as the backbone network to obtain a corresponding third target classification model.

[0066] The combination of SAR image and multispectral image, as well as the combination of infrared image and multispectral image, have large differences between modalities.

[0067] The heterogeneous backbone network uses different network structures to process data of different modalities according to the characteristics of different modal data, and selects the most suitable network structure and optimization method to extract features. The network structure and parameter settings of each branch are designed and optimized according to the characteristics of the respective modal data. The features extracted through the heterogeneous network are highly discriminative and complementary, which can provide a solid foundation for the subsequent cross-modal feature fusion, so that data of different modalities can better complement and integrate each other.

[0068] The third target classification model includes a heterogeneous backbone network, a fusion module, a gating mechanism layer and a decoding layer, and the heterogeneous backbone network includes two different encoders.

[0069] S104. Train the target classification model using the training set to obtain a trained target classification model.

[0070] Among them, the decoding layer includes a fully connected layer.

[0071] Specifically, the target classification model is trained by the training set to obtain a trained target classification model, including:

[0072] Step C1: The first target classification model is trained by combining the optical image with the SAR image and combining the optical image with the infrared image to obtain a trained first target classification model.

[0073] Specifically, the first target classification model is trained by combining the optical image with the SAR image and combining the optical image with the infrared image to obtain the trained first target classification model, including:

[0074] Step C111: Combine the optical image with the SAR image and combine the optical image with the infrared image, and input them into two identical encoders of the twin backbone network respectively, so that the twin backbone network outputs a first feature map combination.

[0075] Among them, the first feature map combination includes a first preset channel feature map and a first target channel feature map.

[0076] The first preset channel feature map may be represented as feature map A1, and the first target channel feature map may be represented as feature map B1.

[0077] Step C112: input the first preset channel feature map and the first target channel feature map into the fusion module, so that the fusion module outputs a first fusion feature map.

[0078] Specifically, the first preset channel feature map and the first target channel feature map are input into the fusion module so that the fusion module outputs a first fusion feature map, including:

[0079] Step C1121: Use the first preset channel feature map as query information and the first target channel feature map as key and value information, input them into the first multi-head attention layer, the first normalization layer, the first multi-layer perception mechanism layer and the second normalization layer in sequence, and output the first fusion result.

[0080] The query information is recorded as (Query, Q), the key and value information are recorded as (Key, K) and (Value, V) respectively, and the first fusion result is recorded as result1_1 (1 is the size pair index of feature map A1 and feature map B1).

[0081] Step C1122: Use the first fusion result as key and value information and the first target channel feature map as query information, and input them into the second multi-head attention layer, the third normalization layer, the second multi-layer perception mechanism layer and the fourth normalization layer in sequence, and output the second fusion result.

[0082] The second fusion result is recorded as result2_1.

[0083] Step C1123: Input the first fusion result and the second fusion result into the concatenation layer and the first 1×1 convolution layer in sequence, and output the first fusion feature map.

[0084] Result1_1 and result2_1 are input into the concatenation layer for concatenation, and then a first 1x1 convolution layer is used to perform channel dimension reduction to obtain the first fused feature map with the same dimension as the input feature map (i.e., feature map A1 and feature map B1), which is recorded as final_result_1.

[0085] Figure 2The execution diagram of the fusion module is schematically shown. The first preset channel feature map (feature map A1) is used as the query information Q, and the first target channel feature map (feature map B1) is used as the key and value information (K and V), which are sequentially input into the first multi-head attention layer, the first normalization layer, the first multi-layer perception mechanism layer, and the second normalization layer, and the first fusion result is output as result1_1. Result1_1 is used as the key and value information (K and V), and the feature map B1 is used as the query information, which are sequentially input into the second multi-head attention layer, the third normalization layer, the second multi-layer perception mechanism layer, and the fourth normalization layer, and the second fusion result is output as result2_1. Result1_1 and result2_1 are input into the splicing layer for splicing operation, and then a first 1x1 convolution layer is used for channel dimension reduction, and the first fusion feature map is output as final_result_1.

[0086] Step C113: Input the first fused feature map into the gating mechanism layer so that the gating mechanism layer outputs the first intermediate feature layer.

[0087] Specifically, the first fused feature map is input into the gating mechanism layer so that the gating mechanism layer outputs a first intermediate feature layer, including:

[0088] The first fused feature map is sequentially input into the second 1×1 convolution layer, the ReLU activation function layer, the third 1×1 convolution layer, the Tanh activation function layer and the dot product layer, and the first intermediate feature layer is output.

[0089] Step C114: Add the first intermediate feature layer, the first preset channel feature map, and the first target channel feature map, respectively, and input the added first preset channel feature map and the added first target channel feature map into the twin backbone network, so that the twin backbone network outputs a second feature map combination.

[0090] Among them, the second feature map combination includes a second preset channel feature map and a second target channel feature map.

[0091] The added first feature map is input into the twin backbone network for encoding operation, so that the twin backbone network outputs a second feature map combination, namely, a second preset channel feature map (represented as feature map A2) and a second target channel feature map (represented as feature map B2).

[0092] Step C115: input the second preset channel feature map and the second target channel feature map into the fusion module, so that the fusion module outputs a second fusion feature map.

[0093] The second fused feature map is recorded as final_result_2.

[0094] For the specific operation of inputting the second preset channel feature map and the second target channel feature map into the fusion module so that the fusion module outputs the second fusion feature map, please refer to the specific operations of steps C1121 to C1123.

[0095] Step C116: Input the second fused feature map into the gating mechanism layer so that the gating mechanism layer outputs a second intermediate feature layer.

[0096] Step C117: Add the second intermediate feature layer, the second preset channel feature map, and the second target channel feature map, respectively, and input the added second preset channel feature map and the added second target channel feature map into the twin backbone network, so that the twin backbone network outputs a third feature map combination.

[0097] Among them, the third feature map combination includes a third preset channel feature map and a third target channel feature map.

[0098] The third preset channel feature map is represented as feature map A3, and the third target channel feature map is represented as feature map B3.

[0099] Step C118: input the third preset channel feature map and the third target channel feature map into the fusion module, so that the fusion module outputs a third fusion feature map.

[0100] The third fused feature map is recorded as final_result_3.

[0101] For the specific operation of inputting the third preset channel feature map and the third target channel feature map into the fusion module so that the fusion module outputs the third fusion feature map, please refer to the specific operations of steps C1121 to C1123.

[0102] Step C119: Input the third fused feature map into the gating mechanism layer so that the gating mechanism layer outputs a third intermediate feature layer.

[0103] Step C120: Add the third intermediate feature layer, the third preset channel feature map, and the third target channel feature map, respectively, and input the added third preset channel feature map and the added third target channel feature map into the twin backbone network, so that the twin backbone network outputs a fourth feature map combination.

[0104] Among them, the fourth feature map combination includes a fourth preset channel feature map and a fourth target channel feature map.

[0105] The fourth preset channel feature map is represented as feature map A4, and the fourth target channel feature map is represented as feature map B4.

[0106] Four different levels of feature maps, i.e., four different scales of feature maps, can be extracted, including low-level features (such as shape, object) at 1 / 4 and 1 / 8 times the original size and high-level features (such as edge, texture) at 1 / 16 and 1 / 32 times the original size. Feature maps A1 and B1 are 1 / 4 of the original size, feature maps A2 and B2 are 1 / 8 of the original size, feature maps A3 and B3 are 1 / 16 of the original size, and feature maps A4 and B4 are 1 / 32 of the original size.

[0107] The extracted feature maps A3, B3, A4, and B4 are high-dimensional feature representations that can capture details and complex patterns of the image and improve classification accuracy.

[0108] Step C121: input the fourth preset channel feature map and the fourth target channel feature map into the fusion module, so that the fusion module outputs a fourth fusion feature map.

[0109] The third fused feature map is recorded as final_result_4.

[0110] For the specific operation of inputting the fourth preset channel feature map and the fourth target channel feature map into the fusion module so that the fusion module outputs the fourth fusion feature map, please refer to the specific operations of steps C1121 to C1123.

[0111] Step C122: input the fourth fused feature map into the fully connected layer so that the fully connected layer outputs a preset classification result.

[0112] The fourth fused feature map contains rich information of the input data, including spatial, spectral and other useful features. The fourth fused feature map is input into the fully connected layer, i.e., the decoding layer, for decoding, and the features are mapped to the category space, so that the fully connected layer outputs a preset classification result.

[0113] Figure 3 The execution diagram of the first target classification model is schematically shown, see Figure 3 As shown, the combination of optical image and SAR image and the combination of optical image and infrared image are recorded as input Figure 1 and input Figure 2, respectively input into the two identical encoders 1 in the twin backbone network, so that the encoder of the twin backbone network outputs the first preset channel feature map, i.e., feature map A1, and the first target channel feature map, i.e., feature map B1. Feature map A1 and feature map B1 are input into the fusion module and the gating mechanism layer in sequence; the first intermediate feature layer output by the gating mechanism layer is added with feature map A1 and feature map B1 respectively; the added feature map A1 and the added feature map B1 are respectively input into the twin backbone network, so that the twin backbone network outputs the second preset channel feature map (expressed as feature map A2) and the second target channel feature map (expressed as feature map B2). The feature map A2 and the feature map B2 are sequentially input into the fusion module and the gating mechanism layer; the second intermediate feature layer output by the gating mechanism layer is summed with the second preset channel feature map and the second target channel feature map; the summed feature map A2 and the summed feature map B2 are respectively input into the twin backbone network, so that the twin backbone network outputs the third preset channel feature map (expressed as feature map A3) and the third target channel feature map (expressed as feature map B3). The feature map A3 and the feature map B3 are sequentially input into the fusion module and the gating mechanism layer; the third intermediate feature layer output by the gating mechanism layer is summed with the feature map A3 and the feature map B3; the summed feature map A3 and the summed feature map B3 are input into the twin backbone network, so that the twin backbone network outputs the fourth preset channel feature map (expressed as feature map A4) and the fourth target channel feature map (expressed as feature map B4). The feature map A4 and the feature map B4 are sequentially input into the fusion module and the fully connected layer, so that the fully connected layer outputs the preset classification result.

[0114] Step C123: Calculate the loss function according to the preset classification result, update the parameters of the first target classification model according to the loss function and the total number of training rounds, and save the parameters of the first target classification model to obtain a trained first target classification model.

[0115] You can choose a suitable loss function according to the specific application task. The loss function can be cross entropy loss and mean square error to ensure that the first target classification model converges to the optimal state during the training process. Calculate the loss function according to the preset classification results. Select a commonly used optimizer, which can be Adam with weight decay (AdamW), Adaptive Moment Estimation Optimizer (Adaptive Moment Estimation, Adam) and Stochastic Gradient Descent (Stochastic Gradient Descent, SGD) optimizer, and adjust the learning rate and other hyperparameters (batch size, total number of training rounds, i.e. number of iterations, weight decay and learning rate multiplication) according to the specific application scenario to obtain the best training effect. Support cosine annealing and learning rate decay learning rate scheduling strategies to improve training results and make the model converge faster in the later stage of training. Set the total number of training rounds and other related parameters, complete the relevant configuration settings, and start training. You can update the parameters of the first target classification model according to the loss function and the total number of training rounds. During the training process, you need to use the validation set to evaluate the model performance to ensure the generalization ability and robustness of the model.

[0116] After training is completed, the parameters of the first target classification model are saved to a file to obtain a trained first target classification model and provide real-time classification services. When the first target classification model needs to be used for inference, the saved model and parameters are loaded from the file, and the trained first target classification model is used to infer the new remote sensing image to generate classification results. The model performance is quantitatively evaluated by using the evaluation indicators Accuracy (Acc), Top-1 highest accuracy (Top-1) and Overall Accuracy (OA) to ensure the effectiveness of the model in different application scenarios. Accuracy is the ratio of the number of correct samples to the total number of samples, which is the most intuitive evaluation indicator in the classification task. Top-1 refers to the proportion of the correct category ranked first in the prediction probability ranking of the correct category among all categories. This indicator measures the ability of the model to directly and correctly predict the most likely category among all possible categories. Overall Accuracy is the ratio of the number of samples correctly classified in all categories to the total number of samples.

[0117] Step C2: The second target classification model is trained by combining the SAR image with the infrared image and the optical image with the multispectral image to obtain a trained second target classification model.

[0118] For the specific operations of training the second target classification model by combining the SAR image with the infrared image and the optical image with the multispectral image to obtain the trained second target classification model, refer to the specific operations of steps C111 to C123.

[0119] Step C3: The third target classification model is trained by combining the SAR image and the multispectral image and the infrared image and the multispectral image to obtain a trained third target classification model.

[0120] For the specific operations of training the third target classification model by combining the SAR image and the multispectral image and the infrared image and the multispectral image to obtain the trained third target classification model, refer to the specific operations of steps C111 to C123.

[0121] Figure 4 The following diagram schematically shows a training target classification model diagram based on image input classification, see Figure 4 As shown, by combining the optical image with the SAR image and the optical image with the infrared image as the input of the first target classification model (the first target classification model is a model that selects the twin backbone network as the backbone network), the first target classification model is trained to obtain a trained first target classification model. By combining the SAR image with the infrared image and the optical image with the multispectral image as the input of the second target classification model (the second target classification model is a model that selects the homogeneous backbone network as the backbone network), the second target classification model is trained to obtain a trained second target classification model. By combining the SAR image with the multispectral image and the infrared image with the multispectral image as the input of the third target classification model (the third target classification model is a model that selects the heterogeneous backbone network as the backbone network), the third target classification model is trained to obtain a trained third target classification model.

[0122] S105: Input the verification set into the trained target classification model, classify the verification set using the trained target classification model, and output the classification result.

[0123] Figure 5 The weighted fusion diagram is shown schematically, see Figure 5 As shown, specifically, the validation set is classified using the trained target classification model, and the classification results are output, including:

[0124] Step D1: Use the trained first target classification model to classify the validation set and output the first classification result.

[0125] Step D2: Use the trained second target classification model to classify the validation set and output the second classification result.

[0126] Step D3: Use the trained third target classification model to classify the verification set and output the third classification result.

[0127] See also Figure 5As shown, specifically, after the verification set is input into the trained target classification model, the verification set is classified using the trained target classification model, and the classification result is output, the classification result can also be post-processed, specifically including:

[0128] Step E1: Test Time Augmentation (TTA) is performed on the first classification result, the second classification result and the third classification result respectively to obtain the corresponding first enhanced classification result, the second enhanced classification result and the third enhanced classification result.

[0129] Step E2: weighted fusion is performed on the first enhanced classification result, the second enhanced classification result and the third enhanced classification result to obtain a final classification result.

[0130] After using multiple classification models (i.e., the first target classification model, the second target classification model, and the third target classification model), the output results of the multiple classification models after test time enhancement (i.e., the first enhanced classification result, the second enhanced classification result, and the third enhanced classification result) can be weighted fused. Specifically, weighted fusion can perform weighted summation of the output probability distribution of each classification model according to a certain weight to obtain the final classification result, or a voting mechanism can be used, that is, multiple classification models classify the same input, and the final result is determined by voting.

[0131] Based on the above Figure 1It can be seen from the implementation method that the embodiment of the present invention obtains remote sensing images of different modalities, and preprocesses and combines the remote sensing images of different modalities in sequence to obtain multiple image groups, and divides the multiple image groups into training sets and verification sets; constructs a preset classification model, the preset classification model includes a backbone network, a fusion module, a gating mechanism layer and a decoding layer connected in sequence, the structural types of the backbone network include a twin backbone network, an isomorphic backbone network and a heterogeneous backbone network, the twin backbone network includes two identical encoders, and the weights and model parameters are the same, the isomorphic backbone network includes two identical encoders, and the weights and model parameters are different, and the heterogeneous backbone network includes two different encoders, and the weights and model parameters are different; according to the multiple image groups in the training set, select one of the twin backbone network, the isomorphic backbone network and the heterogeneous backbone network in the preset classification model as the backbone network to obtain the corresponding target classification model; train the target classification model through the training set to obtain a trained target classification model; input the verification set into the trained target classification model, classify the verification set using the trained target classification model, and output the classification result. In this way, the use of remote sensing images of different modalities can utilize the complementary information between different modalities, understand the data more comprehensively, and enhance the representation ability of the model; the preset classification model can support three different target classification models corresponding to the twin backbone network, the isomorphic backbone network, and the heterogeneous backbone network, as well as two-way cross-multi-scale fusion through the fusion modules in the three different target classification models, so that information can be exchanged between different feature maps, which helps different target classification models to learn richer feature representations, better capture the correlation between multimodal remote sensing images, and make full use of the complementarity of each modality information, so that the accuracy of remote sensing image classification is higher.

[0132] Based on the same inventive concept, as an implementation of the above-mentioned dual-channel based multimodal feature fusion remote sensing image classification method, an embodiment of the present invention also provides a dual-channel based multimodal feature fusion remote sensing image classification method device. Figure 6 FIG. 1 is a structural diagram of a multi-modal feature fusion remote sensing image classification method device based on dual channels in an embodiment of the present invention, see Figure 6 As shown, the device may include:

[0133] The division unit 601 is used to obtain remote sensing images of different modalities, and sequentially pre-process and combine the remote sensing images of different modalities to obtain multiple image groups, and divide the multiple image groups into a training set and a verification set, each image group being a combination of remote sensing images of two different modalities;

[0134] A construction unit 602 is used to construct a preset classification model, which includes a backbone network, a fusion module, a gating mechanism layer and a decoding layer connected in sequence. The structural types of the backbone network include a twin backbone network, a homogeneous backbone network and a heterogeneous backbone network. The twin backbone network includes two identical encoders with the same weights and model parameters, the homogeneous backbone network includes two identical encoders with different weights and model parameters, and the heterogeneous backbone network includes two different encoders with different weights and model parameters.

[0135] A selection unit 603 is used to select one of a twin backbone network, a homogeneous backbone network and a heterogeneous backbone network in a preset classification model as a backbone network according to multiple image groups in a training set, so as to obtain a corresponding target classification model;

[0136] A training unit 604 is used to train the target classification model using a training set to obtain a trained target classification model;

[0137] The classification unit 605 is used to input the verification set into the trained target classification model, classify the verification set using the trained target classification model, and output the classification result.

[0138] The selection unit 603 is specifically used to select a twin backbone network as a backbone network to obtain a corresponding first target classification model when the image group in the training set is a combination of optical images and SAR images and a combination of optical images and infrared images; when the image group in the training set is a combination of SAR images and infrared images and a combination of optical images and multispectral images, select a homogeneous backbone network as a backbone network to obtain a corresponding second target classification model; when the image group in the training set is a combination of SAR images and multispectral images and a combination of infrared images and multispectral images, select a heterogeneous backbone network as a backbone network to obtain a corresponding third target classification model; the multiple image groups in the training set include multiple first image groups, multiple second image groups and multiple third image groups, the multiple first image groups include a combination of optical images and SAR images and a combination of optical images and infrared images, the multiple second image groups include a combination of SAR images and infrared images and a combination of optical images and multispectral images, and the multiple third image groups include a combination of SAR images and multispectral images and a combination of infrared images and multispectral images.

[0139] The training unit 604 is specifically used to train the first target classification model by combining the optical image with the SAR image and the optical image with the infrared image to obtain a trained first target classification model; to train the second target classification model by combining the SAR image with the infrared image and the optical image with the multispectral image to obtain a trained second target classification model; and to train the third target classification model by combining the SAR image with the multispectral image and the infrared image with the multispectral image to obtain a trained third target classification model.

[0140] The training unit 604, the decoding layer includes a fully connected layer, and the first target classification model is trained by combining the optical image with the SAR image and the optical image with the infrared image to obtain the trained first target classification model, including: inputting the optical image with the SAR image and the optical image with the infrared image into two identical encoders in the twin backbone network respectively, so that the twin backbone network outputs a first feature map combination, and the first feature map combination includes a first preset channel feature map and a first target channel feature map; inputting the first preset channel feature map and the first target channel feature map into the fusion module, so that the fusion module outputs a first fused feature map; inputting the first fused feature map into the gate The gating mechanism layer is configured to enable the gating mechanism layer to output a first intermediate feature layer; the first intermediate feature layer is added with the first preset channel feature map and the first target channel feature map, respectively, and the summed preset channel first feature map and the summed first target channel feature map are respectively input into the twin backbone network, so that the twin backbone network outputs a second feature map combination, and the second feature map combination includes a second preset channel feature map and a second target channel feature map; the second preset channel feature map and the second target channel feature map are input into the fusion module, so that the fusion module outputs a second fused feature map; the second fused feature map is input into the gating mechanism layer, so that the gating mechanism layer outputs a second intermediate feature layer; the second intermediate feature layer is combined with The second preset channel feature map and the second target channel feature map are added respectively, and the added second preset channel feature map and the added second target channel feature map are respectively input into the twin backbone network, so that the twin backbone network outputs a third feature map combination, and the third feature map combination includes a third preset channel feature map and a third target channel feature map; the third preset channel feature map and the third target channel feature map are input into the fusion module, so that the fusion module outputs a third fused feature map; the third fused feature map is input into the gating mechanism layer, so that the gating mechanism layer outputs a third intermediate feature layer; the third intermediate feature layer is added with the third preset channel feature map and the third target channel feature map, and the added The third preset channel feature map and the added third target channel feature map are respectively input into the twin backbone network, so that the twin backbone network outputs a fourth feature map combination, and the fourth feature map combination includes a fourth preset channel feature map and a fourth target channel feature map; the fourth preset channel feature map and the fourth target channel feature map are input into the fusion module, so that the fusion module outputs a fourth fused feature map; the fourth fused feature map is input into the fully connected layer, so that the fully connected layer outputs a preset classification result; the loss function is calculated according to the preset classification result, the parameters of the first target classification model are updated according to the loss function and the total number of training rounds, and the parameters of the first target classification model are saved to obtain a trained first target classification model.

[0141] Building unit 602, the fusion module includes a first multi-head attention layer, a first normalization layer, a first multi-layer perception mechanism layer, a second normalization layer, a second multi-head attention layer, a third normalization layer, a second multi-layer perception mechanism layer, a fourth normalization layer, a splicing layer and a first 1×1 convolution layer connected in sequence.

[0142] The training unit 604 inputs the first preset channel feature map and the first target channel feature map into the fusion module so that the fusion module outputs the first fused feature map, including: using the first preset channel feature map as query information and the first target channel feature map as key and value information, sequentially inputting into the first multi-head attention layer, the first normalization layer, the first multi-layer perception mechanism layer and the second normalization layer, and outputting the first fusion result; using the first fusion result as key and value information and the first target channel feature map as query information, sequentially inputting into the second multi-head attention layer, the third normalization layer, the second multi-layer perception mechanism layer and the fourth normalization layer, and outputting the second fusion result; inputting the first fusion result and the second fusion result into the splicing layer and the first 1×1 convolution layer, and outputting the first fusion feature map.

[0143] Building unit 602, the gating mechanism layer includes a second 1×1 convolution layer, a ReLU activation function layer, a third 1×1 convolution layer, a Tanh activation function layer and a dot product layer.

[0144] The device may also include: a post-processing unit, which is specifically used to input the verification set into the trained target classification model, classify the verification set using the trained target classification model, and after outputting the classification result, perform test time enhancement on the first classification result, the second classification result, and the third classification result, respectively, to obtain the corresponding first enhanced classification result, the second enhanced classification result, and the third enhanced classification result; and perform weighted fusion on the first enhanced classification result, the second enhanced classification result, and the third enhanced classification result to obtain the final classification result.

[0145] The division unit 601 is specifically used to sequentially perform spatial alignment, time point pairing, time alignment, image cropping and denoising on remote sensing images of different modalities to obtain a denoised remote sensing image; sequentially perform geometric transformation, color transformation and noise injection on the denoised remote sensing image to obtain a preprocessed remote sensing image; in the preprocessed remote sensing image, two remote sensing images of different modalities are taken as an image group to obtain multiple image groups.

[0146] It should be pointed out here that: the above description of the embodiment of the multimodal feature fusion remote sensing image classification device based on dual channels is similar to the description of the embodiment of the multimodal feature fusion remote sensing image classification method based on dual channels, and has similar beneficial effects as the embodiment of the multimodal feature fusion remote sensing image classification method based on dual channels. For technical details not disclosed in the embodiment of the multimodal feature fusion remote sensing image classification device based on dual channels of the embodiment of the present invention, please refer to the description of the embodiment of the multimodal feature fusion remote sensing image classification method based on dual channels of the present invention for understanding.

[0147] The above are only specific embodiments of the present invention, but the protection scope of the present invention is not limited thereto. Any person skilled in the art who is familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed by the present invention, which should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims.

Claims

1. A dual-channel multimodal feature fusion remote sensing image classification method, characterized in that: The method comprises: Acquire remote sensing images of different modalities, and preprocess and combine the remote sensing images of different modalities in sequence to obtain a plurality of image groups, divide the plurality of image groups into a training set and a validation set, each image group being a combination of remote sensing images of two different modalities; Constructing a preset classification model, the preset classification model includes a backbone network, a fusion module, a gating mechanism layer and a decoding layer connected in sequence, the structural types of the backbone network include a twin backbone network, a homogeneous backbone network and a heterogeneous backbone network; the twin backbone network includes two identical encoders with the same weights and model parameters; the homogeneous backbone network includes two identical encoders with different weights and model parameters; the heterogeneous backbone network includes two different encoders with different weights and model parameters; According to the plurality of image groups in the training set, one of the twin backbone network, the homogeneous backbone network and the heterogeneous backbone network is selected in the preset classification model as the backbone network to obtain a corresponding target classification model; Training the target classification model using the training set to obtain a trained target classification model; The verification set is input into the trained target classification model, the verification set is classified using the trained target classification model, and the classification result is output.

2. The method for remote sensing image classification based on dual-channel multimodal feature fusion according to claim 1 is characterized in that: The plurality of image groups in the training set include a plurality of first image groups, a plurality of second image groups, and a plurality of third image groups, the plurality of first image groups include a combination of optical images and SAR images and a combination of the optical images and infrared images, the plurality of second image groups include a combination of the SAR images and the infrared images and a combination of the optical images and multispectral images, and the plurality of third image groups include a combination of the SAR images and the multispectral images and a combination of the infrared images and the multispectral images; The method of selecting one of the twin backbone network, the homogeneous backbone network and the heterogeneous backbone network in the preset classification model as the backbone network according to the multiple image groups in the training set to obtain the corresponding target classification model includes: When the image group in the training set is a combination of the optical image and the SAR image and a combination of the optical image and the infrared image, selecting the twin backbone network as the backbone network to obtain a corresponding first target classification model; When the image group in the training set is a combination of the SAR image and the infrared image and a combination of the optical image and the multispectral image, selecting the isomorphic backbone network as the backbone network to obtain a corresponding second target classification model; When the image groups in the training set are a combination of the SAR image and the multispectral image and a combination of the infrared image and the multispectral image, the heterogeneous backbone network is selected as the backbone network to obtain a corresponding third target classification model.

3. The method for remote sensing image classification based on dual-channel multimodal feature fusion according to claim 2 is characterized in that: The target classification model is trained by the training set to obtain a trained target classification model, including: The first target classification model is trained by combining the optical image with the SAR image and combining the optical image with the infrared image to obtain a trained first target classification model; The second target classification model is trained by combining the SAR image with the infrared image and combining the optical image with the multispectral image to obtain a trained second target classification model; The third target classification model is trained by combining the SAR image and the multispectral image and combining the infrared image and the multispectral image to obtain a trained third target classification model; Accordingly, the validation set is classified using the trained target classification model and the classification result is output, including: Classify the verification set using the trained first target classification model, and output a first classification result; Classifying the verification set using the trained second target classification model, and outputting a second classification result; The validation set is classified using the trained third target classification model, and a third classification result is output.

4. The method for remote sensing image classification based on dual-channel multimodal feature fusion according to claim 3 is characterized in that: The decoding layer includes a fully connected layer, and the first target classification model is trained by combining the optical image with the SAR image and combining the optical image with the infrared image to obtain a trained first target classification model, including: Combining the optical image with the SAR image and combining the optical image with the infrared image, and inputting the combined images into two identical encoders of the twin backbone network, respectively, so that the twin backbone network outputs a first feature map combination, wherein the first feature map combination includes a first preset channel feature map and a first target channel feature map; Inputting the first preset channel feature map and the first target channel feature map into the fusion module, so that the fusion module outputs a first fusion feature map; Inputting the first fused feature map into the gating mechanism layer so that the gating mechanism layer outputs a first intermediate feature layer; The first intermediate feature layer is added with the first preset channel feature map and the first target channel feature map, respectively, and the summed first preset channel feature map and the summed first target channel feature map are respectively input into the twin backbone network, so that the twin backbone network outputs a second feature map combination, wherein the second feature map combination includes a second preset channel feature map and a second target channel feature map; Inputting the second preset channel feature map and the second target channel feature map into the fusion module so that the fusion module outputs a second fusion feature map; Inputting the second fused feature map into the gating mechanism layer so that the gating mechanism layer outputs a second intermediate feature layer; The second intermediate feature layer is added with the second preset channel feature map and the second target channel feature map, and the added second preset channel feature map and the added second target channel feature map are respectively input into the twin backbone network, so that the twin backbone network outputs a third feature map combination, and the third feature map combination includes a third preset channel feature map and a third target channel feature map; Inputting the third preset channel feature map and the third target channel feature map into the fusion module, so that the fusion module outputs a third fusion feature map; Inputting the third fused feature map into the gating mechanism layer so that the gating mechanism layer outputs a third intermediate feature layer; The third intermediate feature layer is added with the third preset channel feature map and the third target channel feature map, and the added third preset channel feature map and the added third target channel feature map are respectively input into the twin backbone network, so that the twin backbone network outputs a fourth feature map combination, and the fourth feature map combination includes a fourth preset channel feature map and a fourth target channel feature map; Inputting the fourth preset channel feature map and the fourth target channel feature map into the fusion module, so that the fusion module outputs a fourth fusion feature map; Inputting the fourth fused feature map into the fully connected layer so that the fully connected layer outputs a preset classification result; A loss function is calculated according to the preset classification result, parameters of the first target classification model are updated according to the loss function and the total number of training rounds, and the parameters of the first target classification model are saved to obtain the trained first target classification model.

5. The method for remote sensing image classification based on dual-channel multimodal feature fusion according to claim 4 is characterized in that: The fusion module includes a first multi-head attention layer, a first normalization layer, a first multi-layer perception mechanism layer, a second normalization layer, a second multi-head attention layer, a third normalization layer, a second multi-layer perception mechanism layer, a fourth normalization layer, a splicing layer and a first 1×1 convolution layer, which are connected in sequence.

6. The method for remote sensing image classification based on dual-channel multimodal feature fusion according to claim 5 is characterized in that: The step of inputting the first preset channel feature map and the first target channel feature map into the fusion module so that the fusion module outputs a first fusion feature map comprises: Taking the first preset channel feature map as query information and the first target channel feature map as key and value information, sequentially inputting them into the first multi-head attention layer, the first normalization layer, the first multi-layer perception mechanism layer, and the second normalization layer, and outputting a first fusion result; Using the first fusion result as the key and value information and the first target channel feature map as the query information, inputting them into the second multi-head attention layer, the third normalization layer, the second multi-layer perception mechanism layer and the fourth normalization layer in sequence, and outputting a second fusion result; The first fusion result and the second fusion result are sequentially input into the concatenation layer and the first 1×1 convolution layer, and the first fusion feature map is output.

7. The method for remote sensing image classification based on dual-channel multimodal feature fusion according to claim 1, characterized in that: The gating mechanism layer includes a second 1×1 convolution layer, a ReLU activation function layer, a third 1×1 convolution layer, a Tanh activation function layer and a dot product layer.

8. The method for remote sensing image classification based on dual-channel multimodal feature fusion according to claim 3 is characterized in that: After inputting the verification set into the trained target classification model, classifying the verification set using the trained target classification model, and outputting the classification result, the dual-channel based multimodal feature fusion remote sensing image classification method further includes: Performing test time enhancement on the first classification result, the second classification result, and the third classification result, respectively, to obtain corresponding first enhanced classification result, second enhanced classification result, and third enhanced classification result; The first enhanced classification result, the second enhanced classification result and the third enhanced classification result are weightedly fused to obtain a final classification result.

9. The method for remote sensing image classification based on dual-channel multimodal feature fusion according to claim 1, characterized in that: The remote sensing images of different modalities are preprocessed and combined in sequence to obtain a plurality of image groups, including: The remote sensing images of different modalities are sequentially subjected to spatial alignment, time point pairing, time alignment, image cropping and denoising to obtain denoised remote sensing images; Performing geometric transformation, color transformation and noise injection on the denoised remote sensing image in sequence to obtain a preprocessed remote sensing image; In the preprocessed remote sensing images, two remote sensing images of different modalities are taken as one image group to obtain the multiple image groups.

10. A dual-channel multimodal feature fusion remote sensing image classification device, characterized in that: The dual-channel multi-modal feature fusion remote sensing image classification device comprises: A division unit is used to obtain remote sensing images of different modalities, and preprocess and combine the remote sensing images of different modalities in sequence to obtain multiple image groups, and divide the multiple image groups into a training set and a verification set, each image group is a combination of remote sensing images of two different modalities; A construction unit, used to construct a preset classification model, wherein the preset classification model includes a backbone network, a fusion module, a gating mechanism layer and a decoding layer connected in sequence, wherein the structural types of the backbone network include a twin backbone network, an isomorphic backbone network and a heterogeneous backbone network, wherein the twin backbone network includes two identical encoders with the same weights and model parameters, the isomorphic backbone network includes two identical encoders with different weights and model parameters, and the heterogeneous backbone network includes two different encoders with different weights and model parameters; A selection unit, configured to select one of the twin backbone network, the homogeneous backbone network and the heterogeneous backbone network in the preset classification model as the backbone network according to the multiple image groups in the training set, so as to obtain a corresponding target classification model; A training unit, used to train the target classification model using the training set to obtain a trained target classification model; The classification unit is used to input the verification set into the trained target classification model, classify the verification set using the trained target classification model, and output the classification result.

Images