A remote sensing image classification method and device based on dual-channel multimodal feature fusion.

Abstract

This invention provides a method and apparatus for remote sensing image classification based on dual-channel multimodal feature fusion, relating to the field of remote sensing image processing and analysis technology. It includes: preprocessing and combining remote sensing images of different modalities to obtain multiple image groups; constructing a preset classification model, which includes a backbone network, a fusion module, a gating mechanism layer, and a decoding layer connected in sequence; the backbone network structure includes twin backbone networks, homogeneous backbone networks, and heterogeneous backbone networks; selecting one of the twin backbone network, homogeneous backbone network, and heterogeneous backbone network from the preset classification model based on the multiple image groups as the backbone network to obtain the corresponding target classification model; training the target classification model using a training set to obtain a trained target classification model; inputting a validation set into the trained target classification model and outputting the classification result. This results in high accuracy for remote sensing image classification.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of remote sensing image processing and analysis, and in particular to a multi-modal feature fusion remote sensing image classification method and device based on a double channel. BACKGROUND

[0002] Remote sensing image classification is a crucial technology in modern earth observation and environmental monitoring. The main purpose of remote sensing image classification is to extract useful information from remote sensing images and apply it to various fields such as land use, environmental management, and disaster monitoring. Traditional remote sensing image classification methods usually rely on a single type of data, such as optical images or radar data. Although traditional remote sensing image classification methods perform well in some applications, they may face challenges in handling complex surface features and environmental conditions due to the limitations of a single data source.

[0003] Currently, traditional remote sensing image classification methods typically use traditional machine learning methods for optical image classification tasks. However, traditional machine learning methods are not flexible enough when dealing with multi-scale features, making it difficult to capture complex relationships and high-level abstract features in images, limiting the full utilization of contextual information and feature extraction, and resulting in low accuracy of remote sensing image classification. SUMMARY

[0004] The purpose of the embodiments of the present application is to provide a multi-modal feature fusion remote sensing image classification method and device based on a double channel, which solves the problem of low accuracy of remote sensing image classification.

[0005] To solve the above technical problems, the embodiments of the present application provide the following technical solutions:

[0006] The first aspect of the present application provides a multi-modal feature fusion remote sensing image classification method based on a double channel, the method comprising:

[0007] Obtaining remote sensing images of different modalities and sequentially pre-processing and combining the remote sensing images of different modalities to obtain a plurality of image groups, dividing the plurality of image groups into a training set and a validation set, and each image group being a combination of remote sensing images of two different modalities;

[0008] Constructing a preset classification model, the preset classification model comprising a backbone network, a fusion module, a gating mechanism layer, and a decoding layer connected in sequence, the structure type of the backbone network comprising a twin backbone network, an isomorphic backbone network, and a heterogeneous backbone network, the twin backbone network comprising two identical encoders with the same weights and model parameters, the isomorphic backbone network comprising two identical encoders with different weights and model parameters, and the heterogeneous backbone network comprising two different encoders with different weights and model parameters;

[0009] According to the multiple image groups in the training set, one of the twin backbone network, the isomorphic backbone network and the heterogeneous backbone network in the preset classification model is selected as the backbone network to obtain a corresponding target classification model;

[0010] The target classification model is trained by the training set to obtain a trained target classification model;

[0011] The verification set is input into the trained target classification model, and the trained target classification model is used for classifying the verification set to output a classification result.

[0012] The second aspect of the application provides a dual-channel-based multi-modal feature fusion remote sensing image classification device, which comprises:

[0013] The division unit is used for acquiring remote sensing images of different modalities, and sequentially pre-processing and combining images of different modalities to obtain multiple image groups, and dividing the multiple image groups into a training set and a verification set, and each image group is a combination of remote sensing images of two different modalities.

[0014] The construction unit is used for constructing a preset classification model, and the preset classification model comprises a backbone network, a fusion module, a gating mechanism layer and a decoding layer connected in sequence, the structure type of the backbone network comprises a twin backbone network, an isomorphic backbone network and a heterogeneous backbone network, the twin backbone network comprises two identical encoders, and the weights and model parameters are the same, the isomorphic backbone network comprises two identical encoders, and the weights and model parameters are different, and the heterogeneous backbone network comprises two different encoders, and the weights and model parameters are different.

[0015] The selection unit is used for selecting one of the twin backbone network, the isomorphic backbone network and the heterogeneous backbone network in the preset classification model as the backbone network according to the multiple image groups in the training set to obtain a corresponding target classification model.

[0016] The training unit is used for training the target classification model by the training set to obtain a trained target classification model.

[0017] The classification unit is used for inputting the verification set into the trained target classification model, and using the trained target classification model to classify the verification set to output a classification result.

[0018] Compared with the prior art, the dual-channel-based multi-modal feature fusion remote sensing image classification method and device provided by the present application can obtain remote sensing images of different modalities, and sequentially pre-process and combine the remote sensing images of different modalities to obtain a plurality of image groups. The plurality of image groups are divided into a training set and a validation set. A preset classification model is constructed, which includes a backbone network, a fusion module, a gating mechanism layer and a decoding layer connected in sequence. The structure type of the backbone network includes a twin backbone network, an isomorphic backbone network and a heterogeneous backbone network. The twin backbone network includes two identical encoders with the same weights and model parameters. The isomorphic backbone network includes two identical encoders with different weights and model parameters. The heterogeneous backbone network includes two different encoders with different weights and model parameters. According to the plurality of image groups in the training set, one of the twin backbone network, the isomorphic backbone network and the heterogeneous backbone network is selected as the backbone network in the preset classification model to obtain a corresponding target classification model. The target classification model is trained by the training set to obtain a trained target classification model. The validation set is input into the trained target classification model, and the trained target classification model is used to classify the validation set to output a classification result. In this way, the complementary information between different modalities can be utilized to more comprehensively understand the data and enhance the representation ability of the model using remote sensing images of different modalities. The preset classification model can support three different target classification models corresponding to the twin backbone network, the isomorphic backbone network and the heterogeneous backbone network, and perform bidirectional cross multi-scale fusion through the fusion module in the three different target classification models, so that the information between different feature maps is exchanged, which helps the different target classification models to learn more rich feature representations, better captures the correlation between multi-modal remote sensing images, fully utilizes the complementarity of the information of each modality, and makes the accuracy of remote sensing image classification higher. BRIEF DESCRIPTION OF DRAWINGS

[0019] The above and other objects, features and advantages of the present application will become more apparent from the following detailed description read in conjunction with the accompanying drawings, in which several embodiments of the application are shown. The detailed description and drawings are merely illustrative of the present application, since the present application can take many forms. In the drawings, like or corresponding elements are referred to by like or corresponding reference numerals, in which:

[0020] Figure 1 A flowchart of a dual-channel-based multi-modal feature fusion remote sensing image classification method is schematically shown;

[0021] Figure 2 An execution diagram of a fusion module is schematically shown;

[0022] Figure 3 An execution diagram of a first target classification model is schematically shown;

[0023] Figure 4A diagram of a training target classification model based on image input classification is schematically shown;

[0024] Figure 5 A diagram of a weighted fusion schematic is schematically shown;

[0025] Figure 6 A structural diagram of a remote sensing image classification device based on dual-channel multi-modal feature fusion is schematically shown. DETAILED DESCRIPTION

[0026] Exemplary embodiments of the present application will be described in detail with reference to the drawings. Although exemplary embodiments of the present application are shown in the drawings, it should be understood that the present application can be implemented in various forms and should not be limited by the embodiments set forth herein. Rather, these embodiments are provided so that this application will be thorough and complete, and will fully convey the scope of the application to those skilled in the art.

[0027] It should be noted that: unless otherwise specified, the technical terms or scientific terms used by the present application should be understood as the usual meaning understood by the skilled person in the field of the present application.

[0028] The method in the embodiments of the present application will be described in detail below.

[0029] Figure 1 A flowchart of a remote sensing image classification method based on dual-channel multi-modal feature fusion in the embodiments of the present application is schematically shown, referring to Figure 1 As shown, the method can include:

[0030] S101, acquiring remote sensing images of different modalities, and sequentially pre-processing and combining the remote sensing images of different modalities to obtain a plurality of image groups, and dividing the plurality of image groups into a training set and a validation set.

[0031] Each image group is a combination of remote sensing images of two different modalities.

[0032] The remote sensing images of different modalities can be acquired by the United States Landsat 8 launched by the National Aeronautics and Space Administration (NASA) in 2013, the Sentinel series of satellites by the European Space Agency (ESA), and high-resolution cameras carried by unmanned aerial vehicles. The remote sensing images of different modalities include synthetic aperture radar images, infrared images, optical images, and multispectral images. All remote sensing images should be acquired within the growing season of the designated study area, and all acquired remote sensing images need to cover the same study area to ensure that the remote sensing images of different modalities come from the same geographical location.

[0033] Synthetic Aperture Radar (SAR) images are at least 5-meter spatial resolution, horizontal transmit and horizontal receive (HH), horizontal transmit and vertical receive (HV), vertical transmit and horizontal receive (VH), vertical transmit and vertical receive (VV) polarization modes at least one. Synthetic Aperture Radar can provide all-weather high-resolution ground information, especially suitable for data collection under heavy cloud cover or night conditions, can penetrate vegetation and soil surface, and provide surface and subsurface information.

[0034] Infrared images obtained by infrared sensors, at least 30-meter spatial resolution, contain thermal infrared bands, can detect thermal radiation information, suitable for temperature change monitoring, fire detection and thermal pollution monitoring, can distinguish different materials and surface characteristics.

[0035] Optical images provide rich visible spectral information, should contain RGB bands, suitable for natural resource monitoring, urban planning and land use analysis, can intuitively reflect the shape, color and texture characteristics of surface objects, widely used in ecological environment protection, agriculture and water resource management.

[0036] Multispectral images contain information of multiple spectral bands, at least 10-meter spatial resolution, at least including red, green, blue and near-infrared bands, can reflect the spectral characteristics of surface objects, suitable for precision agriculture, forest monitoring and mineral exploration, through different band combination analysis, can reveal plant health status, soil type and water quality, widely used in environmental monitoring and resource management.

[0037] Pretreatment of different modal remote sensing images can ensure that different modal remote sensing images can be effectively fused in subsequent processing.

[0038] Specifically, the different modal remote sensing images are sequentially pretreated and combined images to obtain a plurality of image groups, including:

[0039] Step A1: sequentially performing spatial alignment, time point pairing, time alignment, image cropping and denoising on different modal remote sensing images to obtain denoised remote sensing images.

[0040] Spatial alignment of different modal remote sensing images can use image registration technology to ensure that different modal remote sensing images are aligned on the same geographical space. Image registration technology includes feature point-based registration, phase correlation method and transformation model-based registration, to ensure that the pixel points of the same geographical position in different modal remote sensing images correspond.

[0041] The time point matching and interpolation method can ensure the spatial alignment of the remote sensing images of different modalities, and the closest time point is selected for pairing. The linear interpolation and spline interpolation methods are used to time-align the remote sensing images paired at different time points, to ensure that the corresponding ground information of the remote sensing images of each modality is consistent.

[0042] The time-aligned remote sensing images of different modalities are cropped to the same size, which can be 512*512. The size of the cropped remote sensing images is not limited here. The cropped remote sensing images are denoised using median filtering.

[0043] Step A2: The denoised remote sensing images are sequentially subjected to geometric transformation, color transformation and noise injection to obtain preprocessed remote sensing images.

[0044] The geometric transformation (rotation, flipping, cropping) and color transformation (color jittering, brightness adjustment, contrast adjustment) and noise injection (Gaussian noise) generate new remote sensing images, which can increase the diversity of remote sensing images and provide rich training samples in the training process. The rotation in geometric transformation can change the direction of the denoised remote sensing images. The flipping in geometric transformation can change the structure of the denoised remote sensing images. The cropping in geometric transformation can focus on a specific area. Color jittering can change the color distribution, brightness adjustment can change the brightness of the image, and contrast adjustment can change the contrast of the remote sensing image. Gaussian noise injection can simulate the noise in actual shooting, and the injection of Gaussian noise can provide a variety of data samples for the preset classification model, improve the denoising ability of the preset classification model, and thus improve the robustness of the trained preset classification model.

[0045] Step A3: In the preprocessed remote sensing images, two remote sensing images of different modalities are taken as an image group to obtain multiple image groups.

[0046] To ensure that the two remote sensing images of different modalities in the multiple image groups are the same samples in different modalities. According to the specific application requirements, two remote sensing images of different modalities are arbitrarily selected as an image group, and all preprocessed remote sensing images are divided into image groups to obtain multiple image groups. The multiple image groups are used as the input of the subsequent classification model. The multiple image groups can be combinations of optical images and SAR images, optical images and infrared images, SAR images and infrared images, optical images and multispectral images, SAR images and multispectral images, and infrared images and multispectral images, and the multiple image groups have 6 combinations. Among them, the optical image provides visible light information of the ground object, the SAR image provides microwave reflection information of the ground object, the infrared image provides thermal radiation information, and the multispectral image provides spectral information of multiple wavebands.

[0047] 5000 pieces of four kinds of data sources (i.e. synthetic aperture radar images, infrared images, optical images and multispectral images) can be prepared, two remote sensing images of different modalities are taken as an image group to obtain multiple image groups, 80% of the data is randomly selected from 6 combinations of multiple image groups of each modality as a training set, and the remaining 20% of the data is taken as a verification set.

[0048] Different modalities of remote sensing images are used so that the characteristics of each modality are complementary and have diversified feature capture.

[0049] S102, construct a preset classification model.

[0050] The preset classification model includes a backbone network, a fusion module, a gating mechanism layer and a decoding layer connected in sequence. The structure type of the backbone network includes a twin backbone network, an isomorphic backbone network and a heterogeneous backbone network. The twin backbone network includes two identical encoders, and the weights and model parameters are the same. The isomorphic backbone network includes two identical encoders, and the weights and model parameters are different. The heterogeneous backbone network includes two different encoders, and the weights and model parameters are different.

[0051] The twin backbone network, the isomorphic backbone network and the heterogeneous backbone network are all double-channel collaborative coding networks. The twin backbone network, the isomorphic backbone network and the heterogeneous backbone network can be selected from the following encoders: multi-scale interactive image transformer (MI T), improved visual image transformer based on pyramid structure (PVTv2), visual transformer (ViT), sliding window transformer (Swin Transformer), improved sliding window transformer (Swin Transformer V2), visual large model based on deformable convolution (InternImage), visual state space model (VMamba), convolution network with fusion transformer structure (ConvNeXt), large kernel convolution sequence network (LSKNet) and cross multi-modal large kernel convolution network (UniRepLKNet).

[0052] The fusion module includes a first multi-head attention layer, a first normalization layer, a first multi-layer perception mechanism layer, a second normalization layer, a second multi-head attention layer, a third normalization layer, a second multi-layer perception mechanism layer, a fourth normalization layer, a splicing layer and a first 1×1 convolution layer connected in sequence.

[0053] The gating mechanism layer includes a second 1×1 convolution layer, a ReLU activation function layer, a third 1×1 convolution layer, a Tanh activation function layer and a point multiplication layer. The decoding layer includes a full connection layer.

[0054] S103. Select one of the twin backbone network, the isomorphic backbone network and the heterogeneous backbone network in the preset classification model as a backbone network according to the plurality of image groups in the training set, to obtain a corresponding target classification model.

[0055] The plurality of image groups in the training set includes a plurality of first image groups, a plurality of second image groups and a plurality of third image groups, the plurality of first image groups include optical image and SAR image combination and optical image and infrared image combination, the plurality of second image groups include SAR image and infrared image combination and optical image and multispectral image combination, and the plurality of third image groups include SAR image and multispectral image combination and infrared image and multispectral image combination.

[0056] Specifically, selecting one of the twin backbone network, the isomorphic backbone network and the heterogeneous backbone network in the preset classification model as a backbone network according to the plurality of image groups in the training set, to obtain a corresponding target classification model, includes:

[0057] Step B1: when the image groups in the training set are optical image and SAR image combination and optical image and infrared image combination, selecting the twin backbone network as the backbone network to obtain a corresponding first target classification model.

[0058] The optical image and SAR image combination and the optical image and infrared image combination belong to modalities with high correlation.

[0059] The twin backbone network specifically constructs two parameter-shared branches to ensure consistent feature extraction of two modalities of data. Two branches are constructed using the same convolutional neural network structure, and the two branches share the same weight parameters. Each branch is responsible for processing data of one modality, ensuring that two modalities of data have consistent processing methods in the feature extraction process, capturing the similarities and differences between different modalities of data, and being suitable for similarity detection and matching tasks.

[0060] The first target classification model includes a twin backbone network, a fusion module, a gating mechanism layer and a decoding layer, and the twin backbone network includes two identical encoders.

[0061] Step B2: when the image groups in the training set are SAR image and infrared image combination and optical image and multispectral image combination, selecting the isomorphic backbone network as the backbone network to obtain a corresponding second target classification model.

[0062] The SAR image and infrared image combination and the optical image and multispectral image combination belong to modalities with similar correlation.

[0063] The isomorphic backbone network specifically uses the same but independent network structure to process two modal data and extract respective features. Two branches use independent weight parameters, and each branch is responsible for processing data of one modal, allowing the network to independently learn and optimize according to the characteristics of different modal data. Before feature fusion, the features of each modal data are fully processed in the independent network structure, ensuring the integrity and effectiveness of the features.

[0064] The second target classification model includes an isomorphic backbone network, a fusion module, a gating mechanism layer and a decoding layer, and the isomorphic backbone network includes two same encoders.

[0065] Step B3: When the image group in the training set is a combination of SAR images and multispectral images and a combination of infrared images and multispectral images, an isomorphic backbone network is selected as the backbone network to obtain a corresponding third target classification model.

[0066] The combination of SAR images and multispectral images and the combination of infrared images and multispectral images belong to a larger difference between modalities.

[0067] The isomorphic backbone network specifically uses different network structures to process data of different modalities according to the characteristics of different modal data, selects the most suitable network structure and optimization method to extract features, and designs and optimizes the network structure and parameter setting of each branch according to the characteristics of the respective modal data. The features extracted by the isomorphic network have high discriminability and complementarity, which can provide a solid foundation for subsequent cross-modal feature fusion, so that data of different modalities can better complement and integrate with each other.

[0068] The third target classification model includes an isomorphic backbone network, a fusion module, a gating mechanism layer and a decoding layer, and the isomorphic backbone network includes two different encoders.

[0069] S104, training the target classification model through the training set to obtain a trained target classification model.

[0070] The decoding layer includes a fully connected layer.

[0071] Specifically, training the target classification model through the training set to obtain a trained target classification model includes:

[0072] Step C1: training the first target classification model through a combination of optical images and SAR images and a combination of optical images and infrared images to obtain a trained first target classification model.

[0073] Specifically, training the first target classification model through a combination of optical images and SAR images and a combination of optical images and infrared images to obtain a trained first target classification model includes:

[0074] Step C111: inputting the combination of the optical image and the SAR image and the combination of the optical image and the infrared image into two same encoders of the twin backbone network respectively, so that the twin backbone network outputs a first feature map combination.

[0075] The first feature map combination includes a first preset channel feature map and a first target channel feature map.

[0076] The first preset channel feature map can be represented as a feature map A1, and the first target channel feature map can be represented as a feature map B1.

[0077] Step C112: inputting the first preset channel feature map and the first target channel feature map into a fusion module, so that the fusion module outputs a first fusion feature map.

[0078] Specifically, inputting the first preset channel feature map and the first target channel feature map into the fusion module so that the fusion module outputs the first fusion feature map includes:

[0079] Step C1121: inputting the first preset channel feature map as query information, the first target channel feature map as key and value information into a first multi-head attention layer, a first normalization layer, a first multi-layer perception mechanism layer and a second normalization layer in sequence, and outputting a first fusion result.

[0080] The query information is denoted as (Query, Q), the key and value information are denoted as (Key, K) and (Value, V) respectively, and the first fusion result is denoted as result1_1 (1 is the size index of the feature map A1 and the feature map B1).

[0081] Step C1122: inputting the first fusion result as key and value information, and the first target channel feature map as query information into a second multi-head attention layer, a third normalization layer, a second multi-layer perception mechanism layer and a fourth normalization layer in sequence, and outputting a second fusion result.

[0082] The second fusion result is denoted as result2_1.

[0083] Step C1123: inputting the first fusion result and the second fusion result into a concatenation layer and a first 1x1 convolution layer in sequence, and outputting a first fusion feature map.

[0084] The result1_1 and the result2_1 are input into the concatenation layer for concatenation operation, and then a first 1x1 convolution layer is used for channel dimension reduction, so that the first fusion feature map with the same dimension as the input feature map (i.e. the feature map A1 and the feature map B1) is obtained, and is denoted as final_result_1.

[0085] Figure 2The execution graph of the fusion module is schematically shown. The first preset channel feature map (feature map A1) is taken as the query information Q, and the first target channel feature map (feature map B1) is taken as the key and value information (K and V), which are sequentially input into the first multi-head attention layer, the first normalization layer, the first multi-layer perception mechanism layer and the second normalization layer, and the first fusion result is recorded as result1_1. The result1_1 is taken as the key and value information (K and V), and the feature map B1 is taken as the query information, which are sequentially input into the second multi-head attention layer, the third normalization layer, the second multi-layer perception mechanism layer and the fourth normalization layer, and the second fusion result is recorded as result2_1. The result1_1 and the result2_1 are input into the splicing layer for splicing operation, and then a first 1x1 convolution layer is used for channel dimension reduction, and the first fusion feature map is recorded as final_result_1.

[0086] Step C113: inputting the first fusion feature map into the gating mechanism layer, so that the gating mechanism layer outputs a first intermediate feature layer.

[0087] Specifically, the first fusion feature map is input into the gating mechanism layer, so that the gating mechanism layer outputs the first intermediate feature layer, including:

[0088] The first fusion feature map is sequentially input into a second 1x1 convolution layer, a ReLU activation function layer, a third 1x1 convolution layer, a Tanh activation function layer and a dot multiplication layer, and the first intermediate feature layer is output.

[0089] Step C114: adding the first intermediate feature layer with the first preset channel feature map and the first target channel feature map respectively, and inputting the added first preset channel feature map and the added first target channel feature map into the twin backbone network respectively, so that the twin backbone network outputs a second feature map combination.

[0090] The second feature map combination includes a second preset channel feature map and a second target channel feature map.

[0091] The added first feature map is input into the twin backbone network for encoding operation, so that the twin backbone network outputs the second feature map combination, i.e. the second preset channel feature map (denoted as feature map A2) and the second target channel feature map (denoted as feature map B2).

[0092] Step C115: inputting the second preset channel feature map and the second target channel feature map into the fusion module, so that the fusion module outputs a second fusion feature map.

[0093] The second fusion feature map is recorded as final_result_2.

[0094] The second preset channel feature map and the second target channel feature map are input into the fusion module, so that the fusion module outputs a second fusion feature map. The specific operation can refer to the specific operation of steps C1121 to C1123.

[0095] Step C116: The second fusion feature map is input into the gating mechanism layer, so that the gating mechanism layer outputs a second intermediate feature layer.

[0096] Step C117: The second intermediate feature layer is added to the second preset channel feature map and the second target channel feature map respectively, and the added second preset channel feature map and the added second target channel feature map are input into the twin backbone network respectively, so that the twin backbone network outputs a third feature map combination.

[0097] The third feature map combination includes a third preset channel feature map and a third target channel feature map.

[0098] The third preset channel feature map is denoted as feature map A3, and the third target channel feature map is denoted as feature map B3.

[0099] Step C118: The third preset channel feature map and the third target channel feature map are input into the fusion module, so that the fusion module outputs a third fusion feature map.

[0100] The third fusion feature map is denoted as final_result_3.

[0101] The third preset channel feature map and the third target channel feature map are input into the fusion module, so that the fusion module outputs a third fusion feature map. The specific operation can refer to the specific operation of steps C1121 to C1123.

[0102] Step C119: The third fusion feature map is input into the gating mechanism layer, so that the gating mechanism layer outputs a third intermediate feature layer.

[0103] Step C120: The third intermediate feature layer is added to the third preset channel feature map and the third target channel feature map respectively, and the added third preset channel and the added third target channel feature map are input into the twin backbone network respectively, so that the twin backbone network outputs a fourth feature map combination.

[0104] The fourth feature map combination includes a fourth preset channel feature map and a fourth target channel feature map.

[0105] The fourth preset channel feature map is denoted as feature map A4, and the fourth target channel feature map is denoted as feature map B4.

[0106] Four different levels of feature maps, i.e. four different scales of feature maps, can be extracted, including low-level features (such as shape, object) of 1 / 4, 1 / 8 times of the original size and high-level features (such as edge, texture) of 1 / 16, 1 / 32 times of the original size. Feature map A1 and feature map B1 are 1 / 4 times of the original size, feature map A2 and feature map B2 are 1 / 8 times of the original size, feature map A3 and feature map B3 are 1 / 16 times of the original size, and feature map A4 and feature map B4 are 1 / 32 times of the original size.

[0107] The extracted feature map A3, feature map B3, feature map A4 and feature map B4 are high-dimensional feature representations, which can capture the details and complex patterns of the image and improve the accuracy of classification.

[0108] Step C121: input the fourth preset channel feature map and the fourth target channel feature map into the fusion module, so that the fusion module outputs a fourth fusion feature map.

[0109] The third fusion feature map is denoted as final_result_4.

[0110] The specific operation of inputting the fourth preset channel feature map and the fourth target channel feature map into the fusion module to make the fusion module output the fourth fusion feature map can refer to the specific operation of steps C1121 to C1123.

[0111] Step C122: input the fourth fusion feature map into the fully connected layer, so that the fully connected layer outputs a preset classification result.

[0112] The fourth fusion feature map contains rich information of the input data, including space, spectrum and other useful features. The fourth fusion feature map is input into the fully connected layer, i.e. the decoding layer, for decoding, mapping the features to the category space, so that the fully connected layer outputs a preset classification result.

[0113] Figure 3 The execution graph of the first target classification model is schematically shown, referring to Figure 3 As shown, the combination of the optical image and the SAR image and the combination of the optical image and the infrared image are denoted as input Figure 1 and input Figure 2, respectively input the two same encoders 1 of the twin backbone network, so that the encoders of the twin backbone network output the first preset channel feature map, i.e., feature map A1, and the first target channel feature map, i.e., feature map B1, respectively. The feature map A1 and the feature map B1 are sequentially input into the fusion module and the gating mechanism layer; the first intermediate feature layer output by the gating mechanism layer is added with the feature map A1 and the feature map B1, respectively; the added feature map A1 and the added feature map B1 are input into the twin backbone network, respectively, so that the twin backbone network outputs the second preset channel feature map (denoted as feature map A2) and the second target channel feature map (denoted as feature map B2). The feature map A2 and the feature map B2 are sequentially input into the fusion module and the gating mechanism layer; the second intermediate feature layer output by the gating mechanism layer is added with the second preset channel feature map and the second target channel feature map, respectively; the added feature map A2 and the added feature map B2 are input into the twin backbone network, respectively, so that the twin backbone network outputs the third preset channel feature map (denoted as feature map A3) and the third target channel feature map (denoted as feature map B3). The feature map A3 and the feature map B3 are sequentially input into the fusion module and the gating mechanism layer; the third intermediate feature layer output by the gating mechanism layer is added with the feature map A3 and the feature map B3, respectively; the added feature map A3 and the added feature map B3 are input into the twin backbone network, so that the twin backbone network outputs the fourth preset channel feature map (denoted as feature map A4) and the fourth target channel feature map (denoted as feature map B4). The feature map A4 and the feature map B4 are sequentially input into the fusion module and the fully connected layer, so that the fully connected layer outputs the preset classification result.

[0114] Step C123: calculating a loss function according to the preset classification result, updating the parameters of the first target classification model according to the loss function and the total number of training rounds, and saving the parameters of the first target classification model to obtain a trained first target classification model.

[0115] The appropriate loss function can be selected according to the specific application task, and the loss function can be cross-entropy loss and mean square error, to ensure that the first target classification model converges to the optimal state in the training process. The loss function is calculated according to the preset classification result. Select a commonly used optimizer, which can be Adam with weight decay (AdamW), Adaptive Moment Estimation (Adam), and Stochastic Gradient Descent (SGD) optimizer. Adjust the learning rate and other hyperparameters (batch size, total number of training rounds, i.e., the number of iterations, weight decay, and learning rate multiplication) according to the specific application scenario to obtain the best training effect. Support cosine annealing, learning rate scheduling strategy with learning rate decay, improve training effect, and make the model converge faster in the later training. Set the total number of training rounds and other related parameters, complete the related configuration settings, and start training. The parameters of the first target classification model can be updated according to the loss function and the total number of training rounds. During the training process, the performance of the model needs to be evaluated using the validation set to ensure the generalization ability and robustness of the model.

[0116] After training, save the parameters of the first target classification model to a file to obtain the trained first target classification model, and provide real-time classification services. When the first target classification model is needed for inference, load the saved model and parameters from the file, and use the trained first target classification model to infer new remote sensing images to generate classification results. The model performance is quantitatively evaluated using evaluation indicators such as accuracy (Accuracy, Acc), Top-1 highest accuracy (Top-1), and overall accuracy (Overall Accuracy, OA) to ensure the effectiveness of the model in different application scenarios. The accuracy (Accuracy) is the proportion of correct samples to the total number of samples, which is the most intuitive evaluation indicator in the classification task. Top-1 refers to the proportion of the correct class being ranked first in the prediction probability ranking of all classes. This indicator measures the model's ability to directly correctly predict the most likely class among all possible classes. The overall accuracy (Overall Accuracy) is the proportion of samples classified correctly in all classes to the total number of samples.

[0117] Step C2: Train the second target classification model by combining SAR images and infrared images and combining optical images and multispectral images to obtain a trained second target classification model.

[0118] For training the second target classification model by combining SAR images and infrared images and combining optical images and multispectral images to obtain a trained second target classification model, refer to the specific operations of steps C111 to C123.

[0119] Step C3: training the third target classification model by SAR image and multispectral image combination and infrared image and multispectral image combination, to obtain the trained third target classification model.

[0120] For training the third target classification model by SAR image and multispectral image combination and infrared image and multispectral image combination, to obtain the trained third target classification model, the specific operation can refer to the specific operation of step C111 to step C123.

[0121] Figure 4 The training target classification model diagram based on image input classification is schematically shown, referring to Figure 4 As shown, by optical image and SAR image combination and optical image and infrared image combination, as the input of the first target classification model (the first target classification model is a model selecting a twin backbone network as a backbone network), the first target classification model is trained to obtain the trained first target classification model. By SAR image and infrared image combination and optical image and multispectral image combination, as the input of the second target classification model (the second target classification model is a model selecting an isomorphic backbone network as a backbone network), the second target classification model is trained to obtain the trained second target classification model. By SAR image and multispectral image combination and infrared image and multispectral image combination, as the input of the third target classification model (the third target classification model is a model selecting a heterogeneous backbone network as a backbone network), the third target classification model is trained to obtain the trained third target classification model.

[0122] S105, input the verification set into the trained target classification model, classify the verification set by using the trained target classification model, and output the classification result.

[0123] Figure 5 The weighted fusion schematic diagram is schematically shown, referring to Figure 5 As shown, specifically, the verification set is classified by using the trained target classification model, and the classification result is output, including:

[0124] Step D1: classifying the verification set by using the trained first target classification model, and outputting the first classification result.

[0125] Step D2: classifying the verification set by using the trained second target classification model, and outputting the second classification result.

[0126] Step D3: classifying the verification set by using the trained third target classification model, and outputting the third classification result.

[0127] Referring to Figure 5As shown, specifically, after inputting the verification set into the trained target classification model, classifying the verification set by using the trained target classification model, and outputting the classification result, the classification result can be post-processed, specifically including:

[0128] Step E1: Test time augmentation (TTA) is performed on the first classification result, the second classification result, and the third classification result, respectively, to obtain corresponding first enhanced classification result, second enhanced classification result, and third enhanced classification result.

[0129] Step E2: The first enhanced classification result, the second enhanced classification result, and the third enhanced classification result are weighted and fused to obtain the final classification result.

[0130] After using multiple classification models (i.e., the first target classification model, the second target classification model, and the third target classification model), the output results (i.e., the first enhanced classification result, the second enhanced classification result, and the third enhanced classification result) of the multiple classification models after test time augmentation can be weighted and fused. Specifically, the output probability distribution of each classification model can be weighted and summed according to a certain weight to obtain the final classification result, or a voting mechanism can be used, that is, multiple classification models classify the same input, and the final result is determined by voting.

[0131] Based on the above Figure 1As can be seen from the implementation mode, the embodiment of the application obtains remote sensing images of different modalities, and sequentially pre-processes and combines images of different modalities to obtain a plurality of image groups, and divides the plurality of image groups into a training set and a validation set; a preset classification model is constructed, the preset classification model includes a backbone network, a fusion module, a gating mechanism layer and a decoding layer connected in sequence, the structure type of the backbone network includes a twin backbone network, an isomorphic backbone network and a heterogeneous backbone network, the twin backbone network includes two identical encoders, and the weights and model parameters are the same, the isomorphic backbone network includes two identical encoders, and the weights and model parameters are different from each other, and the heterogeneous backbone network includes two different encoders, and the weights and model parameters are different from each other; according to the plurality of image groups in the training set, one of the twin backbone network, the isomorphic backbone network and the heterogeneous backbone network in the preset classification model is selected as the backbone network to obtain a corresponding target classification model; the target classification model is trained through the training set to obtain a trained target classification model; the validation set is input into the trained target classification model, and the trained target classification model is used for classifying the validation set to output a classification result. In this way, the remote sensing images of different modalities can utilize the complementary information between different modalities to more comprehensively understand the data and enhance the representation ability of the model; the preset classification model can support three different target classification models corresponding to the twin backbone network, the isomorphic backbone network and the heterogeneous backbone network, and the fusion module in the three different target classification models performs bidirectional cross multi-scale fusion, so that the information is exchanged between different feature maps, which helps the different target classification models to learn more rich feature representations, can better capture the correlation between the multi-modal remote sensing images, fully utilizes the complementarity of the information of each modality, and makes the accuracy of remote sensing image classification higher.

[0132] Based on the same inventive concept, as an implementation of the above-mentioned multi-modal feature fusion remote sensing image classification method based on double channels, the embodiment of the application further provides a multi-modal feature fusion remote sensing image classification method device based on double channels. Figure 6 For the structural diagram of the multi-modal feature fusion remote sensing image classification method device based on double channels in the embodiment of the application, refer to Figure 6 As shown in the figure, the device can include:

[0133] The division unit 601 is configured to obtain remote sensing images of different modalities, and sequentially pre-process and combine images of different modalities to obtain a plurality of image groups, and divide the plurality of image groups into a training set and a validation set, each image group being a combination of remote sensing images of two different modalities;

[0134] The constructing unit 602 is configured to construct a preset classification model, the preset classification model comprising a backbone network, a fusion module, a gating mechanism layer and a decoding layer connected in sequence, and the backbone network comprises a twin backbone network, an isomorphic backbone network and a heterogeneous backbone network, the twin backbone network comprising two identical encoders with the same weight and model parameter, the isomorphic backbone network comprising two identical encoders with different weight and model parameter, and the heterogeneous backbone network comprising two different encoders with different weight and model parameter.

[0135] The selecting unit 603 is configured to select one of the twin backbone network, the isomorphic backbone network and the heterogeneous backbone network in the preset classification model as the backbone network according to a plurality of image groups in the training set, to obtain a corresponding target classification model.

[0136] The training unit 604 is configured to train the target classification model by using the training set, to obtain a trained target classification model.

[0137] The classification unit 605 is configured to input the verification set into the trained target classification model, and classify the verification set by using the trained target classification model, to output a classification result.

[0138] The selecting unit 603 is specifically configured to select the twin backbone network as the backbone network when the image groups in the training set are optical image and SAR image combination and optical image and infrared image combination, to obtain a corresponding first target classification model; select the isomorphic backbone network as the backbone network when the image groups in the training set are SAR image and infrared image combination and optical image and multispectral image combination, to obtain a corresponding second target classification model; and select the heterogeneous backbone network as the backbone network when the image groups in the training set are SAR image and multispectral image combination and infrared image and multispectral image combination, to obtain a corresponding third target classification model; the plurality of image groups in the training set comprising a plurality of first image groups, a plurality of second image groups and a plurality of third image groups, the plurality of first image groups comprising optical image and SAR image combination and optical image and infrared image combination, the plurality of second image groups comprising SAR image and infrared image combination and optical image and multispectral image combination, and the plurality of third image groups comprising SAR image and multispectral image combination and infrared image and multispectral image combination.

[0139] The training unit 604 is specifically configured to train the first target classification model through combination of the optical image and the SAR image and combination of the optical image and the infrared image, to obtain the trained first target classification model; train the second target classification model through combination of the SAR image and the infrared image and combination of the optical image and the multispectral image, to obtain the trained second target classification model; and train the third target classification model through combination of the SAR image and the multispectral image and combination of the infrared image and the multispectral image, to obtain the trained third target classification model.

[0140] The training unit 604, the decoding layer includes a full connection layer, the first target classification model is trained through the combination of optical images and SAR images and the combination of optical images and infrared images, to obtain the trained first target classification model, comprising: inputting the combination of optical images and SAR images and the combination of optical images and infrared images into the two same encoders of the twin backbone network respectively, so that the twin backbone network outputs a first feature map combination, the first feature map combination includes a first preset channel feature map and a first target channel feature map; the first preset channel feature map and the first target channel feature map are input into the fusion module, so that the fusion module outputs a first fusion feature map; the first fusion feature map is input into the gate mechanism layer, so that the gate mechanism layer outputs a first intermediate feature layer; the first intermediate feature layer is added with the first preset channel feature map and the first target channel feature map respectively, and the added first preset channel feature map and the added first target channel feature map are input into the twin backbone network respectively, so that the twin backbone network outputs a second feature map combination, the second feature map combination includes a second preset channel feature map and a second target channel feature map; the second preset channel feature map and the second target channel feature map are input into the fusion module, so that the fusion module outputs a second fusion feature map; the second fusion feature map is input into the gate mechanism layer, so that the gate mechanism layer outputs a second intermediate feature layer; the second intermediate feature layer is added with the second preset channel feature map and the second target channel feature map respectively, and the added second preset channel feature map and the added second target channel feature map are input into the twin backbone network respectively, so that the twin backbone network outputs a third feature map combination, the third feature map combination includes a third preset channel feature map and a third target channel feature map; the third preset channel feature map and the third target channel feature map are input into the fusion module, so that the fusion module outputs a third fusion feature map; the third fusion feature map is input into the gate mechanism layer, so that the gate mechanism layer outputs a third intermediate feature layer; the third intermediate feature layer is added with the third preset channel feature map and the third target channel feature map respectively, and the added third preset channel feature map and the added third target channel feature map are input into the twin backbone network respectively, so that the twin backbone network outputs a fourth feature map combination, the fourth feature map combination includes a fourth preset channel feature map and a fourth target channel feature map; the fourth preset channel feature map and the fourth target channel feature map are input into the fusion module, so that the fusion module outputs a fourth fusion feature map; the fourth fusion feature map is input into the full connection layer, so that the full connection layer outputs a preset classification result; the loss function is calculated according to the preset classification result, the parameters of the first target classification model are updated according to the loss function and the total number of training rounds, and the parameters of the first target classification model are saved, to obtain the trained first target classification model.

[0141] The construction unit 602, the fusion module includes sequentially connected first multi-head attention layer, first normalization layer, first multi-layer perception mechanism layer, second normalization layer, second multi-head attention layer, third normalization layer, second multi-layer perception mechanism layer, fourth normalization layer, splicing layer and first 1*1 convolution layer.

[0142] The training unit 604, the first preset channel feature map and the first target channel feature map are input into the fusion module to make the fusion module output the first fusion feature map, including: taking the first preset channel feature map as query information, taking the first target channel feature map as key and value information, and inputting into the first multi-head attention layer, the first normalization layer, the first multi-layer perception mechanism layer and the second normalization layer in turn, outputting the first fusion result; taking the first fusion result as key and value information, taking the first target channel feature map as query information, inputting into the second multi-head attention layer, the third normalization layer, the second multi-layer perception mechanism layer and the fourth normalization layer in turn, outputting the second fusion result; inputting the first fusion result and the second fusion result into the splicing layer and the first 1*1 convolution layer in turn, outputting the first fusion feature map.

[0143] The construction unit 602, the gating mechanism layer includes a second 1*1 convolution layer, a ReLU activation function layer, a third 1*1 convolution layer, a Tanh activation function layer and a dot multiplication layer.

[0144] The device can further include a post-processing unit, specifically configured to input the verification set into the trained target classification model, classify the verification set by using the trained target classification model, and output a classification result; and respectively test time enhancement on the first classification result, the second classification result and the third classification result to obtain corresponding first enhanced classification result, second enhanced classification result and third enhanced classification result; and perform weighted fusion on the first enhanced classification result, the second enhanced classification result and the third enhanced classification result to obtain a final classification result.

[0145] The division unit 601 is specifically configured to sequentially perform spatial alignment, time point pairing, time alignment, image cropping and denoising processing on remote sensing images of different modalities to obtain denoised remote sensing images; sequentially perform geometric transformation, color transformation and noise injection on the denoised remote sensing images to obtain preprocessed remote sensing images; and take two remote sensing images of different modalities as an image group in the preprocessed remote sensing images to obtain a plurality of image groups.

[0146] It should be noted that the above description of the embodiment of the remote sensing image classification device based on multi-modal feature fusion of dual channels is similar to the above description of the embodiment of the remote sensing image classification method based on multi-modal feature fusion of dual channels, and has similar beneficial effects as the embodiment of the remote sensing image classification method based on multi-modal feature fusion of dual channels. For technical details of the embodiment of the remote sensing image classification device based on multi-modal feature fusion of dual channels of the present application that are not disclosed, please refer to the description of the embodiment of the remote sensing image classification method based on multi-modal feature fusion of dual channels of the present application for understanding.

[0147] The above is only a specific embodiment of the present application, but the protection scope of the present application is not limited thereto, and any person skilled in the art can easily think of changes or replacements within the technical scope disclosed by the present application, which should be covered within the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. A dual-channel based multi-modal feature fusion remote sensing image classification method, characterized in that, The method comprises: acquiring remote sensing images of different modalities, and sequentially pre-processing and combining the remote sensing images of different modalities to obtain a plurality of image groups, and dividing the plurality of image groups into a training set and a validation set, each image group being a combination of remote sensing images of two different modalities; constructing a preset classification model, the preset classification model comprising a backbone network, a fusion module, a gating mechanism layer and a decoding layer connected in sequence, the structure type of the backbone network comprising a twin backbone network, an isomorphic backbone network and a heterogeneous backbone network; the twin backbone network comprising two identical encoders, and the weights and model parameters being the same; the isomorphic backbone network comprising two identical encoders, and the weights and model parameters being different; the heterogeneous backbone network comprising two different encoders, and the weights and model parameters being different; selecting one of the twin backbone network, the isomorphic backbone network and the heterogeneous backbone network in the preset classification model as the backbone network according to the plurality of image groups in the training set to obtain a corresponding target classification model; training the target classification model through the training set to obtain a trained target classification model; inputting the validation set into the trained target classification model, classifying the validation set by using the trained target classification model, and outputting a classification result; the plurality of image groups in the training set comprising a plurality of first image groups, a plurality of second image groups and a plurality of third image groups, the plurality of first image groups comprising combinations of optical images and SAR images and combinations of the optical images and infrared images, the plurality of second image groups comprising combinations of the SAR images and the infrared images and combinations of the optical images and multispectral images, and the plurality of third image groups comprising combinations of the SAR images and the multispectral images and combinations of the infrared images and the multispectral images; the selecting one of the twin backbone network, the isomorphic backbone network and the heterogeneous backbone network in the preset classification model as the backbone network according to the plurality of image groups in the training set to obtain a corresponding target classification model comprising: when the image groups in the training set are the combinations of the optical images and the SAR images and the combinations of the optical images and the infrared images, selecting the twin backbone network as the backbone network to obtain a corresponding first target classification model; when the image groups in the training set are the combinations of the SAR images and the infrared images and the combinations of the optical images and the multispectral images, selecting the isomorphic backbone network as the backbone network to obtain a corresponding second target classification model; when the image groups in the training set are the combinations of the SAR images and the multispectral images and the combinations of the infrared images and the multispectral images, selecting the heterogeneous backbone network as the backbone network to obtain a corresponding third target classification model; the training the target classification model through the training set to obtain a trained target classification model comprising: The first target classification model is trained through combination of the optical image and the SAR image and combination of the optical image and the infrared image, to obtain a trained first target classification model; The second target classification model is trained through combination of the SAR image and the infrared image and combination of the optical image and the multispectral image, to obtain a trained second target classification model; The third target classification model is trained through combination of the SAR image and the multispectral image and combination of the infrared image and the multispectral image, to obtain a trained third target classification model; Correspondingly, the classification of the verification set by using the trained target classification model and output of a classification result include: The first target classification model is trained through combination of the optical image and the SAR image and combination of the optical image and the infrared image, to obtain a trained first target classification model; The second target classification model is trained through combination of the SAR image and the infrared image and combination of the optical image and the multispectral image, to obtain a trained second target classification model; The third target classification model is trained through combination of the SAR image and the multispectral image and combination of the infrared image and the multispectral image, to obtain a trained third target classification model; The decoding layer includes a full connection layer, and the first target classification model is trained through combination of the optical image and the SAR image and combination of the optical image and the infrared image, to obtain a trained first target classification model, including: The optical image and the SAR image are combined and the optical image and the infrared image are combined, and are respectively input into two same encoders of the twin backbone network, so that the twin backbone network outputs a first feature map combination, and the first feature map combination includes a first preset channel feature map and a first target channel feature map; The first preset channel feature map and the first target channel feature map are input into the fusion module, so that the fusion module outputs a first fusion feature map; The first fusion feature map is input into the gating mechanism layer, so that the gating mechanism layer outputs a first intermediate feature layer; The first intermediate feature layer is added with the first preset channel feature map and the first target channel feature map respectively, and the added first preset channel feature map and the added first target channel feature map are respectively input into the twin backbone network, so that the twin backbone network outputs a second feature map combination, and the second feature map combination includes a second preset channel feature map and a second target channel feature map; The second preset channel feature map and the second target channel feature map are input into the fusion module, so that the fusion module outputs a second fusion feature map; The second fusion feature map is input into the gating mechanism layer, so that the gating mechanism layer outputs a second intermediate feature layer; The second intermediate feature layer is added with the second preset channel feature map and the second target channel feature map respectively, and the added second preset channel feature map and the added second target channel feature map are respectively input into the twin backbone network, so that the twin backbone network outputs a third feature map combination, and the third feature map combination includes a third preset channel feature map and a third target channel feature map; The third preset channel feature map and the third target channel feature map are input into the fusion module, so that the fusion module outputs a third fusion feature map; The third fusion feature map is input into the gating mechanism layer, so that the gating mechanism layer outputs a third intermediate feature layer; The third intermediate feature layer is added with the third preset channel feature map and the third target channel feature map respectively, and the added third preset channel feature map and the added third target channel feature map are input into the twin backbone network respectively, so that the twin backbone network outputs a fourth feature map combination, the fourth feature map combination including a fourth preset channel feature map and a fourth target channel feature map; The fourth preset channel feature map and the fourth target channel feature map are input into the fusion module, so that the fusion module outputs a fourth fusion feature map; The fourth fusion feature map is input into the full connection layer, so that the full connection layer outputs a preset classification result; According to the preset classification result, a loss function is calculated, the parameters of the first target classification model are updated according to the loss function and the total number of training rounds, and the parameters of the first target classification model are saved to obtain the trained first target classification model; After inputting the verification set into the trained target classification model and classifying the verification set by using the trained target classification model to output a classification result, the method further comprises: The first classification result, the second classification result and the third classification result are respectively tested for time enhancement to obtain corresponding first enhanced classification results, second enhanced classification results and third enhanced classification results; The first enhanced classification result, the second enhanced classification result and the third enhanced classification result are weighted fused to obtain a final classification result. The operation of the weighted fusion refers to weighted summation of the output probability distribution of the first target classification model, the output probability distribution of the second target classification model and the output probability distribution of the third target classification model according to certain weights to obtain the final classification result, or the operation of the weighted fusion refers to using the first target classification model, the second target classification model and the third target classification model to classify the same input to obtain the final classification result, and the final classification result is determined by voting.

2. The dual-channel based multi-modal feature fusion remote sensing image classification method according to claim 1, characterized in that, The fusion module comprises a first multi-head attention layer, a first normalization layer, a first multi-layer perception mechanism layer, a second normalization layer, a second multi-head attention layer, a third normalization layer, a second multi-layer perception mechanism layer, a fourth normalization layer, a splicing layer and a first 1×1 convolution layer connected in sequence.

3. The dual-channel based multi-modal feature fusion remote sensing image classification method according to claim 2, characterized in that, The operation of inputting the first preset channel feature map and the first target channel feature map into the fusion module to make the fusion module output a first fusion feature map comprises: The first preset channel feature map is taken as query information, and the first target channel feature map is taken as key and value information, which are sequentially input into the first multi-head attention layer, the first normalization layer, the first multi-layer perception mechanism layer and the second normalization layer, to output a first fusion result; The first fusion result is taken as the key and value information, and the first target channel feature map is taken as the query information, which are sequentially input into the second multi-head attention layer, the third normalization layer, the second multi-layer perception mechanism layer and the fourth normalization layer, to output a second fusion result; The first fusion result and the second fusion result are sequentially input into the splicing layer and the first 1*1 convolution layer, to output the first fusion feature map.

4. The dual-channel based multi-modal feature fusion remote sensing image classification method according to claim 1, characterized in that, The gating mechanism layer comprises a second 1*1 convolution layer, a ReLU activation function layer, a third 1*1 convolution layer, a Tanh activation function layer and a point multiplication layer.

5. The dual-channel based multi-modal feature fusion remote sensing image classification method according to claim 1, characterized in that, After the verification set is input into the trained target classification model, the trained target classification model is used to classify the verification set, and a classification result is output, the double-channel-based multi-modal feature fusion remote sensing image classification method further comprises: The first classification result, the second classification result and the third classification result are respectively tested for time enhancement, to obtain corresponding first enhanced classification results, second enhanced classification results and third enhanced classification results; The first enhanced classification result, the second enhanced classification result and the third enhanced classification result are weightedly fused, to obtain a final classification result.

6. The dual-channel based multi-modal feature fusion remote sensing image classification method according to claim 1, characterized in that, The different modal remote sensing images are sequentially preprocessed and combined into images, to obtain a plurality of image groups, comprising: The different modal remote sensing images are sequentially subjected to spatial alignment, time point pairing, time alignment, image cropping and denoising processing, to obtain denoised remote sensing images; The denoised remote sensing images are sequentially subjected to geometric transformation, color transformation and noise injection, to obtain preprocessed remote sensing images; In the preprocessed remote sensing images, two different modal remote sensing images are taken as an image group, to obtain the plurality of image groups.

7. A dual-channel based multi-modal feature fusion remote sensing image classification device, characterized in that, The double-channel-based multi-modal feature fusion remote sensing image classification device comprises: A division unit is configured to obtain different modal remote sensing images, sequentially preprocess and combine the different modal remote sensing images into images, obtain a plurality of image groups, and divide the plurality of image groups into a training set and a verification set, each image group being a combination of two different modal remote sensing images; A construction unit is configured to construct a preset classification model, the preset classification model comprising a backbone network, a fusion module, a gating mechanism layer and a decoding layer connected in sequence, the structure type of the backbone network comprising a twin backbone network, an isomorphic backbone network and a heterogeneous backbone network, the twin backbone network comprising two identical encoders with the same weight and model parameter, the isomorphic backbone network comprising two identical encoders with different weight and model parameter, and the heterogeneous backbone network comprising two different encoders with different weight and model parameter; The selecting unit is configured to select one of the twin backbone network, the isomorphic backbone network and the heterogeneous backbone network in the preset classification model as the backbone network according to the image groups in the training set, so as to obtain a corresponding target classification model; The training unit is configured to train the target classification model by using the training set, so as to obtain a trained target classification model; The classification unit is configured to input the verification set into the trained target classification model, classify the verification set by using the trained target classification model, and output a classification result; The selecting unit is specifically configured to select the twin backbone network as the backbone network when the image groups in the training set are combinations of optical images and SAR images and combinations of the optical images and infrared images, so as to obtain a corresponding first target classification model; select the isomorphic backbone network as the backbone network when the image groups in the training set are combinations of the SAR images and the infrared images and combinations of the optical images and multispectral images, so as to obtain a corresponding second target classification model; and select the heterogeneous backbone network as the backbone network when the image groups in the training set are combinations of the SAR images and the multispectral images and combinations of the infrared images and the multispectral images, so as to obtain a corresponding third target classification model; the image groups in the training set include a plurality of first image groups, a plurality of second image groups and a plurality of third image groups, the first image groups include combinations of optical images and SAR images and combinations of the optical images and infrared images, the second image groups include combinations of the SAR images and the infrared images and combinations of the optical images and multispectral images, and the third image groups include combinations of the SAR images and the multispectral images and combinations of the infrared images and the multispectral images; The training unit is specifically configured to train the first target classification model by using the combinations of the optical images and the SAR images and the combinations of the optical images and the infrared images, so as to obtain a trained first target classification model; train the second target classification model by using the combinations of the SAR images and the infrared images and the combinations of the optical images and the multispectral images, so as to obtain a trained second target classification model; and train the third target classification model by using the combinations of the SAR images and the multispectral images and the combinations of the infrared images and the multispectral images, so as to obtain a trained third target classification model; correspondingly, the classification unit is configured to classify the verification set by using the trained first target classification model, output a first classification result; classify the verification set by using the trained second target classification model, output a second classification result; and classify the verification set by using the trained third target classification model, output a third classification result. The decoding layer in the training unit includes a full connection layer, and the training of the first target classification model through the combination of the optical image and the SAR image and the combination of the optical image and the infrared image obtains a trained first target classification model, including: inputting the combination of the optical image and the SAR image and the combination of the optical image and the infrared image into two same encoders of the twin backbone network respectively, so that the twin backbone network outputs a first feature map combination, the first feature map combination including a first preset channel feature map and a first target channel feature map; inputting the first preset channel feature map and the first target channel feature map into the fusion module, so that the fusion module outputs a first fusion feature map; inputting the first fusion feature map into the gating mechanism layer, so that the gating mechanism layer outputs a first intermediate feature layer; respectively adding the first intermediate feature layer and the first preset channel feature map and the first target channel feature map, and inputting the added first preset channel feature map and the added first target channel feature map into the twin backbone network respectively, so that the twin backbone network outputs a second feature map combination, the second feature map combination including a second preset channel feature map and a second target channel feature map; inputting the second preset channel feature map and the second target channel feature map into the fusion module, so that the fusion module outputs a second fusion feature map; inputting the second fusion feature map into the gating mechanism layer, so that the gating mechanism layer outputs a second intermediate feature layer; respectively adding the second intermediate feature layer and the second preset channel feature map and the second target channel feature map, and inputting the added second preset channel feature map and the added second target channel feature map into the twin backbone network respectively, so that the twin backbone network outputs a third feature map combination, the third feature map combination including a third preset channel feature map and a third target channel feature map; inputting the third preset channel feature map and the third target channel feature map into the fusion module, so that the fusion module outputs a third fusion feature map; inputting the third fusion feature map into the gating mechanism layer, so that the gating mechanism layer outputs a third intermediate feature layer; respectively adding the third intermediate feature layer and the third preset channel feature map and the third target channel feature map, and inputting the added third preset channel feature map and the added third target channel feature map into the twin backbone network respectively, so that the twin backbone network outputs a fourth feature map combination, the fourth feature map combination including a fourth preset channel feature map and a fourth target channel feature map; inputting the fourth preset channel feature map and the fourth target channel feature map into the fusion module, so that the fusion module outputs a fourth fusion feature map; inputting the fourth fusion feature map into the full connection layer, so that the full connection layer outputs a preset classification result;According to the preset classification result, a loss function is calculated, parameters of the first target classification model are updated according to the loss function and a total number of rounds of training, and the parameters of the first target classification model are saved to obtain the trained first target classification model. The classification unit is further configured to, after inputting the verification set into the trained target classification model, classifying the verification set by using the trained target classification model, and outputting a classification result, respectively test the first classification result, the second classification result and the third classification result, and obtain corresponding first enhanced classification result, second enhanced classification result and third enhanced classification result; and perform weighted fusion on the first enhanced classification result, the second enhanced classification result and the third enhanced classification result to obtain a final classification result; wherein the operation of the weighted fusion refers to performing weighted summation on the output probability distribution of the first target classification model, the output probability distribution of the second target classification model and the output probability distribution of the third target classification model according to certain weights to obtain the final classification result, or the operation of the weighted fusion refers to using a voting mechanism to classify the same input by using the first target classification model, the second target classification model and the third target classification model to obtain the final classification result, and the final classification result is determined by voting.

Images