Multi-source remote sensing image classification method and device based on intra-source and inter-source interaction representation learning network

Abstract

This invention proposes a multi-source remote sensing image classification method and apparatus based on a representation learning network with intra-source and inter-source interactions. Through a trained representation learning model based on intra-source and inter-source interactions, features of real-time remote sensing images are extracted during the feature extraction stage. Simultaneously, the internal dependencies of single-source features are strengthened, making the model more focused on information-rich regions within the source. By learning the correlations between multi-source features, interactions between multi-source and multi-level features are achieved, realizing bidirectional enhanced fusion and mitigating the semantic gap between deep features from multiple sources. Based on the similarity and differences between multi-source features, an optimization strategy is established to retain information-rich single-source features, while another source gradually filters out complementary information, effectively promoting complementary multi-source representation, reducing information redundancy, and thus improving the network's classification performance. Finally, a relatively accurate classification result for real-time remote sensing images is obtained.

Description

Technical Field

[0001] This invention belongs to the field of image information processing technology, specifically relating to a multi-source remote sensing image classification method and apparatus based on a representation learning network for intra-source and inter-source interactions. Background Technology

[0002] With the rapid development of remote sensing technology, the types of satellites used in China, including high-resolution satellites, communication satellites, and radars, have been increasing in recent years. Simultaneously, the methods for acquiring image data have become more diversified. We can easily obtain multi-source data with complementary information, especially panchromatic (PAN) data with rich spatial information and multispectral (MS) data with rich spectral information. Multi-source remote sensing image classification technology is a prominent field in the current remote sensing technology field, with significant application value in urban planning, agricultural development, and military applications. Its goal is to interpret the categories and distribution of ground features based on their electromagnetic radiation characteristics. However, establishing relationships between multi-source remote sensing images, fully utilizing data complementarity, and reducing information redundancy to improve the performance of classification models remains a challenge.

[0003] For multi-source image classification tasks based on panchromatic PAN images and multispectral MS images, there are generally two strategies. Traditional methods focus on fusing multi-source data to obtain a fused image with rich spatial and spectral features. For example, the component substitution (CS) method typically replaces the spatial components of the MS with PAN in the transformed spatial domain, and then performs an inverse transformation to obtain the fused HR-MS image. Commonly used component substitution methods include Intensity-Hue-Saturation Transform (IHS) and Principal Component Analysis (PCA). In the Multi-Resolution Analysis (MRA) method, wavelet transform or Laplacian pyramid transform is used to decompose the PAN and MS images to obtain multi-scale, multi-resolution images, and then a specific fusion strategy is used to fuse the multi-resolution images. However, in practical applications, they all suffer from certain spectral distortion and spatial artifacts, affecting the fusion results. In recent years, variational optimization-based methods have been widely studied and applied. By using prior regularization constraints to establish an observation model, the data fusion problem is transformed into an optimization problem of minimizing an energy function. The first variational fusion method is the P+XS model. Its core idea is to superimpose the spatial information of the PAN image with the MS image to improve the spatial resolution of the MS image. Compared to CS- or MRA-based methods, VO-based methods can achieve better spatial spectral quality. However, current VO-based methods are mainly linear and cannot accurately describe the complex relationship between the fused image and the observations.

[0004] In deep learning methods, features are typically extracted directly from multi-source images and then fused for classification. Existing algorithms mostly focus on the design of the feature extraction network, while incorporating joint optimization strategies to utilize single-source attribute features as supplementary supervisory information to reduce multi-source discrepancies. During the feature extraction stage, spatial and spectral matching learning establishes interactions between multiple sources. This often involves a three-branch network structure: the first two branches enhance the spatial features of large targets and the spectral features between channels, while the third branch progressively strengthens the purely independent features extracted by the PAN and MS branches. Compared to two-branch networks, three-branch networks have more parameters and a heavier training burden. While these designs can effectively improve the model's classification accuracy, they do not fully consider the complementarity and redundancy between multi-source features, resulting in limited classification performance.

[0005] Current technologies have not fully explored the complementarity and redundancy among multi-source data, leading to semantic gaps and information redundancy among deep features from multiple sources. Due to differences in imaging mechanisms, the shallow feature representations of multi-source remote sensing data often differ significantly, resulting in semantic gaps among deep features and hindering the integration of multi-source information. Furthermore, without enhancement of individual source features, they may fail to focus on regions rich in feature information. On the other hand, multi-source remote sensing data typically describe the same scene, often resulting in information redundancy between sources during feature fusion, causing useful classification information to be buried under a large amount of redundant information. Summary of the Invention

[0006] To address the aforementioned problems in the existing technology, this invention provides a multi-source remote sensing image classification method and apparatus based on a representation learning network for intra-source and inter-source interactions. The technical problem to be solved by this invention is achieved through the following technical solution:

[0007] In a first aspect, the present invention provides a multi-source remote sensing image classification method based on a representation learning network for intra-source and inter-source interactions, comprising:

[0008] S100, acquires real-time remote sensing images;

[0009] S200, the real-time remote sensing image is used as input to a trained representation learning model based on intra-source and inter-source interactions. The representation learning model extracts dual-source features from the real-time remote sensing image through a dual-branch feature extraction network composed of multiple cascaded residual modules. Each residual module inputs the extracted dual-source features into a cross-source interaction enhancement fusion module for feature interaction. During the feature interaction process, a complementary representation supervision learning strategy is used for supervision, and then the feedback is fed back to the next level residual module until the last residual module outputs the final fused features. The final fused features of the dual branches are concatenated and then passed through the output layer to obtain the category of the same object in the real-time remote sensing image.

[0010] Secondly, the present invention provides a multi-source remote sensing image classification device based on a representation learning network for intra-source and inter-source interactions, comprising:

[0011] The acquisition module is configured to acquire real-time remote sensing images;

[0012] The classification module is configured to take the real-time remote sensing image as input to a trained representation learning model based on intra-source and inter-source interactions. The representation learning model extracts intermediate features from the real-time remote sensing image through a dual-branch feature extraction network composed of multiple cascaded residual modules. Each residual module inputs the dual-source features extracted by each residual module of each branch into a cross-source interaction enhancement fusion module for feature interaction. During the feature interaction process, a complementary representation-supervised learning strategy is used for supervision, which is then fed back to the next level residual module until the last residual module outputs the final fused feature. The final fused features from the two branches are concatenated and then passed through an output layer to obtain the category of the same object in the real-time remote sensing image.

[0013] Beneficial effects:

[0014] This invention addresses two problems in current multi-source remote sensing image classification technologies: the semantic gap between multi-source features and information redundancy. It proposes a multi-source remote sensing image classification method and apparatus based on a representation learning network for intra-source and inter-source interactions. Through a trained representation learning model based on intra-source and inter-source interactions, features of real-time remote sensing images are extracted during the feature extraction stage. The internal dependencies of single-source features are strengthened, allowing the model to focus more on information-rich regions within a single source. By learning the correlations between multi-source features, interactions between multi-source and multi-level features are achieved, realizing bidirectional enhanced fusion and mitigating the semantic gap between deep features from multiple sources. Based on the similarity and differences between multi-source features, an optimization strategy is established to retain information-rich single-source features, while another source gradually filters out complementary information, effectively promoting complementary multi-source representation, reducing information redundancy, and thus improving the network's classification performance. Finally, a more accurate classification result for real-time remote sensing images is obtained.

[0015] The present invention will be further described in detail below with reference to the accompanying drawings and embodiments. Attached Figure Description

[0016] Figure 1 This is a flowchart illustrating the multi-source remote sensing image classification method based on a representation learning network for intra-source and inter-source interactions provided by the present invention.

[0017] Figure 2 This is the overall network structure diagram of the representation learning model based on intra-source and inter-source interaction provided by the present invention;

[0018] Figure 3 This is a structural diagram of the residual block provided by the present invention;

[0019] Figure 4 This is a schematic diagram of the processing procedure of the cross-source interaction enhancement fusion module provided by the present invention;

[0020] Figure 5 This is a schematic diagram of the NCSM(·) operation process provided by the present invention;

[0021] Figure 6 This is a schematic diagram of the process of the complementary representation supervised learning strategy provided by the present invention;

[0022] Figure 7 This is a schematic diagram of the training process provided by the present invention. Detailed Implementation

[0023] The present invention will be further described in detail below with reference to specific embodiments, but the implementation of the present invention is not limited thereto.

[0024] Combination Figure 1 and Figure 2 This invention provides a multi-source remote sensing image classification method based on a representation learning network for intra-source and inter-source interactions, comprising:

[0025] S100, acquires real-time remote sensing images;

[0026] S200, the real-time remote sensing image is used as input to a trained representation learning model based on intra-source and inter-source interactions. The representation learning model extracts intermediate features from the real-time remote sensing image through a dual-branch feature extraction network composed of multiple cascaded residual modules. Each residual module inputs the dual-source features extracted by each residual module of each branch into a cross-source interaction enhancement fusion module for feature interaction. During the feature interaction process, a complementary representation supervision learning strategy is used for supervision, and then the feedback is fed back to the next level residual module until the last residual module outputs the final fused feature. The final fused features of the dual branches are concatenated and then passed through the output layer to obtain the category of the same object in the real-time remote sensing image.

[0027] The trained representation learning model based on intra-source and inter-source interactions described in this invention is jointly supervised by the fusion loss function and the cross-entropy classification loss function, and then trained using the training set.

[0028] refer to Figures 2 to 4 The representation learning model based on intra-source and inter-source interaction of the present invention includes a dual-branch feature extraction network, a cross-source interaction enhancement fusion module, a fully connected layer, and an output layer;

[0029] The dual-branch feature extraction network comprises two branches with non-shared weights, each branch consisting of multiple cascaded residual modules. Each residual module is connected to a corresponding cross-source interaction enhancement fusion module. The dual-source features extracted by the residual modules at the same level of the two branches are input to the corresponding cross-source interaction enhancement fusion module for feature interaction. The fused features output by the cross-source interaction enhancement fusion module are input to the residual modules of the two branches and summed with the input features of the residual modules. During the feature interaction process, the cross-source interaction enhancement fusion module generates two intermediate variables and is supervised by the output control of the complementary representation supervised learning strategy. The addition unit connected to the last residual module is used to output the final fused features. The final fused features are converted into a one-dimensional vector through a fully connected layer and then classified through the output layer to obtain the classification result.

[0030] This invention can acquire PAN and MS image patches from real-time remote sensing images. Then, multiple residual modules (ResB) with identical structures but non-shared weights are used to extract multi-level PAN and MS features. Since the resolution ratio of PAN to MS features is 4:1, for features of the same level, the PAN features are downsampled using the Davg(·) operation to make their resolution size the same as the MS features, facilitating subsequent dual-source interaction and feature fusion. The dual-source features extracted by each residual module are fed into the Cross-Source Interaction Enhancement Fusion Module (CIEF-Module) for interaction. The output features are summed with the input features, and then passed through the residual modules for further feature extraction. The two intermediate variables Vm and Vp generated by the CIEF-Module are supervised by a Complementary Representation Supervised Learning Strategy (CRSL-Strategy) to reduce information redundancy. The output of this strategy is Lp, which, together with the cross-entropy classification loss function Lc, supervises the training of the network. Finally, the dual-source deep features obtained through layer-by-layer interaction are subjected to global average pooling downsampling operation Pavg(·), and then cascaded to obtain fused features. The fused features are then converted into a one-dimensional vector through a fully connected layer (FC), and then input into a softmax classifier to obtain the classification result.

[0031] refer to Figure 3 Each residual module includes two branches: a residual part and an identity mapping. The residual part consists of a convolutional layer, a ReLU function, another convolutional layer, and a BN layer in sequence, which is used to learn the difference between the input and the output. The identity mapping part directly adds the input to the output of the residual part. All residual modules have the same structure.

[0032] refer to Figure 4 The cross-source interaction enhancement fusion module consists of three parts: a preprocessing module, a source-intra-source dependency enhancement module, and an inter-source enhancement fusion module.

[0033] The preprocessing module uses the self-attention mechanism in Transformer to process the global dependency of the intra-source features of the dual-source features extracted by the residual module to obtain the autocorrelation matrix;

[0034] The preprocessing module applies three parallel 1×1 convolutions to the dual-source features input from the residual module to obtain three components, and then performs a size transformation operation to obtain intermediate variables; the autocorrelation matrix is ​​obtained by using the intermediate variables to perform scaling dot product attention calculation.

[0035] The source dependency enhancement module multiplies the autocorrelation matrix of the two branches by the corresponding coefficients, thereby updating each correlation score in the autocorrelation matrix to obtain the enhanced single-source feature.

[0036] refer to Figure 4 and Figure 5 To address the semantic gap between deep features from multiple sources, a novel Cross-Source Interaction Enhancement Fusion Module (CIEF-Module) is introduced to facilitate interaction between multi-level PAN and MS features. By strengthening the internal dependencies of single-source features, refining related features within a source, and learning the correlations between multi-source features, connections are established between intra-source and inter-source relationships. Figure 4 As shown, this module consists of three parts: preprocessing, enhancement of intra-source dependencies, and enhanced fusion between sources. The preprocessing part uses the self-attention mechanism in Transformer to model the global dependencies of intra-source features for the dual-source multi-level features fp and fm obtained after residual block extraction. Specifically, taking the PAN branch as an example, three parallel 1×1 convolutions are applied to the downsampled PAN feature fp to obtain three components, and then a size transformation R(·) operation is performed to obtain Qp, Kp, and Vp. The autocorrelation matrix Ap is calculated based on the scaled dot product attention, as shown in the following formula:

[0037]

[0038] The enhancement of source-related dependencies aims to make the network pay more attention to relevant features within the source, focusing on regions rich in source information. Self-attention mechanisms lack this focus, so we designed the NCSM(·) operation, such as... Figure 5 As shown, the autocorrelation matrices Am and Ap obtained from the two branches are multiplied by two coefficients α and β, respectively. Figure 5 Taking Ap as an example, α and β are defined as feature pairs.<fi,fj> The ratio of the dot product similarity score aij to the highest similarity score max(i:) and max(:j) in the i-th row and j-th column is used to multiply aij element-wise with α and β, thereby updating each correlation score in the autocorrelation matrix to pay more attention to the relevant features within the source.

[0039] The source enhancement fusion module performs a fusion operation by multiplying the two enhanced single-source features element by element, then obtains the weight matrix of the fused features through the Sigmoid function, and performs implicit feature transfer through the weight matrix to achieve bidirectional feature enhancement between multi-source features to obtain the fused features of the two branches, which are then fed back to the output of the residual module.

[0040] This invention performs a gated weighting operation on each value of the transmitted feature, where the transmitted feature is the component multiplication matrix weight matrix corresponding to the single-source feature; then it passes through a 1×1 convolutional layer, a BN layer, and a Sigmoid function for gated weighting in sequence, and multiplies with the transmitted feature element by element to achieve bidirectional feature enhancement between multi-source features, resulting in the fused features of the two branches.

[0041] Inter-source enhancement fusion: The two enhanced single-source features are multiplied element-wise and then fused. The resulting matrix is ​​then passed through a Sigmoid function to obtain a weight matrix C for the fused features. This weight matrix is ​​then used to perform implicit feature transfer, achieving bidirectional feature enhancement between multi-source features. Considering the adaptability of the transferred features to potential changes in appearance information such as noise, a gated weighting G(·) operation is performed on each value of the transferred features. The two transferred features are their respective V components Vp and Vm multiplied by matrix C, then subjected to 1×1 convolution, BN, and Sigmoid for gated weighting, and multiplied element-wise with the transferred features to determine how much of the transferred feature will be received. The formula is expressed as follows:

[0042]

[0043] The output of the inter-source enhancement fusion module is directly summed with the input of the module, and then the subsequent feature extraction operation continues.

[0044] This invention employs a novel complementary representation-supervised learning strategy during cross-source interaction.

[0045] The complementary representation supervised learning strategy performs a global average pooling operation along the channel direction on the intermediate variables of the dual-branch to obtain the global average pooling operation result.

[0046] Calculate the information entropy H(·) of the global average pooling operation result, and take the difference between the information entropy H(·) of the two branches to obtain the difference in information richness;

[0047] Calculate the cosine similarity parameter of the intermediate variable in the two-branch system;

[0048] The difference in information richness and the cosine similarity parameter are imported into the max-min normalization optimization algorithm. The max-min normalization algorithm is then used to optimize the result of the global average pooling operation to adjust the optimization value range. Based on the optimized value range, a fusion loss function is output to supervise the feature interaction process.

[0049] like Figure 6 As shown, for the two V components Vp and Vm of the calculated PAN and MS, a global average pooling operation Pavg(·) along the channel direction is first performed on them, and then the following optimization is performed: minL=γ p ||V P ||2+γ m ||V M ||2, where the coefficient γ p γ m Based on the formal definition of the Sigmoid function, its value range is [0, 1], and the formula is as follows:

[0050]

[0051] In the formula, the parameter κ is defined as the reciprocal of the square of the cosine similarity of Vp and Vm:

[0052]

[0053] The parameter ξ is defined as the difference in information richness between Vp and Vm, where information richness is characterized by information entropy I. Specifically, after pooling, the range of values ​​for Vp and Vm is adjusted using max-min normalization N(·), and then the information entropy H(·) is calculated separately, with the difference between the two calculated to obtain ξ.

[0054]

[0055] Based on the above design, when the similarity between the two V components is low, and if the PAN features are richer than the MS features, γ is calculated. p γ is closer to 0, while γ m Closer to 1. This means the network will penalize MS features more to avoid MS features containing more similar information to PAN features. Conversely, a smaller κ value indicates a smaller γ value. From the perspective of the sigmoid function form, the larger the κ value, the faster the function decreases, leading to an imbalance in the weight distribution and an imbalance in the penalty for PAN and MS features. Furthermore, if the κ value is small, meaning the PAN and MS features are similar, then their information richness is also similar. Therefore, the calculated κ value will be close to 1, and γ... p and γ m The value will be close to 0.5. Therefore, the penalty for PAN and MS features is similar. The output Lp of this strategy, along with the cross-entropy classification loss Lc, jointly supervises the training of the network.

[0056] refer to Figure 7 The training process of the trained representation learning model based on intra-source and inter-source interactions of the present invention is as follows:

[0057] a) Read PAN and MS plots from multiple different sensors;

[0058] This invention reads PAN and MS maps from a dataset, including PAN and MS maps of the target region after registration, and corresponding class label GT maps of only a portion of the region;

[0059] b. Normalize the PAN image and the MS image so that the values ​​of all pixels are within a fixed range to obtain a normalized image;

[0060] Since the PAN and MS images come from different sensors and their data distribution ranges differ significantly, the values ​​of all pixels in the PAN and MS image matrices are scaled to the range [0, 1] using the following formula:

[0061]

[0062] Where, p i p represents a pixel value min p is the minimum value among all pixels. max It is the maximum value among all pixels.

[0063] c. Construct training and testing sets using the normalized images;

[0064] In the normalized image, pixels with corresponding class labels are identified. For each pixel, a 64×64 region in the PAN image and a 16×16 region in the MS image are selected as paired sample blocks. The neighborhood relationship of each block provides a basis for classifying the central pixel. Based on this, 2% of the samples from each class are selected as the training set, and the remaining 98% are selected as the test set.

[0065] d. Construct a representation learning model based on intra-source and inter-source interactions. The learning network includes a residual module, a cross-source interaction enhancement fusion module, and a cascaded module.

[0066] e. The representation learning model based on intra-source and inter-source interactions is trained using the training set, and the prediction error is backpropagated during the training process to update the parameters of the representation learning model, thereby obtaining the trained representation learning model.

[0067] The learning model parameters are set as follows:

[0068] PAN branch:

[0069] Pre-adjustment:

[0070] The input size of the PAN image block is 64×64×1;

[0071] 3×3 convolution with stride of 1, padding of 1, 1 input channel, and 64 output channels;

[0072] Feature extraction:

[0073] Residual block 1: 3×3 convolution, stride 1, padding 1, input channels 64, output channels 64; adaptive average pooling downsampling;

[0074] Residual block 2: 3×3 convolution, stride 1, padding 1, input channels 64, output channels 128; adaptive average pooling downsampling;

[0075] Residual Block 3: 3×3 convolution, stride 1, padding 1, input channels 128, output channels 256; adaptive average pooling downsampling;

[0076] Residual Block 4: 3×3 convolution, stride 1, padding 1, input channels 256, output channels 512; adaptive average pooling downsampling;

[0077] Interactive module:

[0078] Three parallel 1×1 convolutional layers with a stride of 1 and no padding, and the number of input channels is the same as the number of output channels (used to obtain the three components of self-attention);

[0079] A 1×1 convolutional layer with a stride of 1 and no padding, where the number of input channels is the same as the number of output channels (gated weighted convolution);

[0080] Feature fusion:

[0081] Global average pooling layer;

[0082] MS branch:

[0083] Pre-adjustment:

[0084] The input size of the PAN image block is 16×16×4;

[0085] 3×3 convolution with stride of 1, padding of 1, 4 input channels, and 64 output channels;

[0086] Feature extraction:

[0087] Residual block 1: 3×3 convolution, stride 1, padding 1, input channels 64, output channels 64;

[0088] Residual block 2: 3×3 convolution, stride 1, padding 1, input channels 64, output channels 128;

[0089] Residual Block 3: 3×3 convolution, stride 1, padding 1, input channels 128, output channels 256;

[0090] Residual block 4: 3×3 convolution, stride 1, padding 1, input channels 256, output channels 512;

[0091] Interactive module:

[0092] Three parallel 1×1 convolutional layers with a stride of 1 and no padding, and the number of input channels is the same as the number of output channels (used to obtain the three components of self-attention);

[0093] A 1×1 convolutional layer with a stride of 1 and no padding, where the number of input channels is the same as the number of output channels (gated weighted convolution);

[0094] Feature fusion:

[0095] Global average pooling layer;

[0096] Classification:

[0097] Fully connected layer: Input 1024 nodes, output a number of nodes representing categories.

[0098] The model uses paired PAN and MS image patches as inputs to its two branches. The output is the model's predicted classification result for each pixel. The error between the model output and the true class of each pixel is calculated and backpropagated to update the model parameters, thus obtaining the optimal solution for the classification model.

[0099] The input is each sample in the test set. The test is performed on the trained model for the category to which each pixel belongs. Once the test is completed, the classification and recognition process can begin.

[0100] This invention provides a multi-source remote sensing image classification device based on a representation learning network for intra-source and inter-source interactions, comprising:

[0101] The acquisition module is configured to acquire real-time remote sensing images;

[0102] The classification module is configured to take the real-time remote sensing image as input to a trained representation learning model based on intra-source and inter-source interactions. The representation learning model extracts dual-source features from the real-time remote sensing image through a dual-branch feature extraction network composed of multiple cascaded residual modules. Each residual module inputs its extracted dual-source features into a cross-source interaction enhancement fusion module for feature interaction. During the feature interaction process, a complementary representation-supervised learning strategy is used for supervision, which is then fed back to the next level residual module until the last residual module outputs the final fused feature. The final fused features from the dual branches are concatenated and then passed through an output layer to obtain the category of the same object in the real-time remote sensing image.

[0103] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of this invention, "a plurality of" means two or more, unless otherwise explicitly specified.

[0104] Although this application has been described herein in conjunction with various embodiments, those skilled in the art will understand and implement other variations of the disclosed embodiments by reviewing the accompanying drawings, the disclosure, and the appended claims in carrying out the claimed application. In the claims, the word "comprising" does not exclude other components or steps, and "a" or "an" does not exclude a plurality.

[0105] The above description, in conjunction with specific preferred embodiments, provides a further detailed explanation of the present invention. It should not be construed that the specific implementation of the present invention is limited to these descriptions. For those skilled in the art, various simple deductions or substitutions can be made without departing from the concept of the present invention, and all such modifications and substitutions should be considered within the scope of protection of the present invention.

Claims

1. A multi-source remote sensing image classification method based on a representation learning network for intra-source and inter-source interactions, characterized in that, include: S100, acquires real-time remote sensing images; S200, the real-time remote sensing image is used as the input of the trained representation learning model based on intra-source and inter-source interaction, so that the representation learning model extracts the dual-source features of the real-time remote sensing image through a dual-branch feature extraction network composed of multiple cascaded residual modules. Each residual module inputs the extracted dual-source features into the cross-source interaction enhancement fusion module for feature interaction. During the feature interaction process, a complementary representation supervision learning strategy is used for supervision, and then the feedback is fed back to the next level residual module until the last residual module outputs the final fused features. The final fused features of the two branches are stitched together and then passed through the output layer to obtain the category of the same object in the real-time remote sensing image; the representation learning model based on intra-source and inter-source interaction includes a two-branch feature extraction network, a cross-source interaction enhancement fusion module, a fully connected layer, and an output layer; The dual-branch feature extraction network comprises two branches with non-shared weights, each branch consisting of multiple cascaded residual modules. Each residual module is connected to a corresponding cross-source interaction enhancement fusion module. The dual-source features extracted by the residual modules at the same level of the two branches are input to the corresponding cross-source interaction enhancement fusion module for feature interaction. The fused features output by the cross-source interaction enhancement fusion module are input to the residual modules of the two branches and summed with the input features of the residual modules. During the feature interaction process, the cross-source interaction enhancement fusion module generates two intermediate variables and is supervised by the output control of a complementary representation supervised learning strategy. The addition unit connected to the last residual module is used to output the final fused features. The final fused features are converted into a one-dimensional vector through a fully connected layer and then classified through an output layer to obtain the classification result. The cross-source interaction enhancement fusion module consists of three parts: a preprocessing module, a source intra-dependency enhancement module, and an inter-source enhancement fusion module. The preprocessing module uses the self-attention mechanism in Transformer to process the global dependency of the intra-source features of the dual-source features extracted by the residual module to obtain the autocorrelation matrix; The source dependency enhancement module multiplies the autocorrelation matrix of the two branches by the corresponding coefficients, thereby updating each correlation score in the autocorrelation matrix to obtain the enhanced single-source feature. The source enhancement fusion module performs a fusion operation by multiplying the two enhanced single-source features element by element, then obtains the weight matrix of the fused features through the Sigmoid function, and performs implicit feature transfer through the weight matrix to achieve bidirectional feature enhancement between multi-source features to obtain the fused features of the two branches, which are then fed back to the output of the residual module.

2. The multi-source remote sensing image classification method based on representation learning networks with intra-source and inter-source interactions according to claim 1, characterized in that, Each residual module includes two branches: a residual part and an identity mapping. The residual part consists of a convolutional layer, a ReLU function, another convolutional layer, and a BN layer in sequence, which is used to learn the difference between the input and the output. The identity mapping part directly adds the input to the output of the residual part. All residual modules have the same structure.

3. The multi-source remote sensing image classification method based on representation learning networks with intra-source and inter-source interactions according to claim 1, characterized in that, The preprocessing module uses the self-attention mechanism in Transformer to perform global dependency processing on the intra-source features of the dual-source features extracted by the residual module, resulting in an autocorrelation matrix including: The preprocessing module applies three parallel 1×1 convolutions to the dual-source features input from the residual module to obtain three components, and then performs a size transformation operation to obtain intermediate variables. The autocorrelation matrix is ​​obtained by using the intermediate variable to perform scaled dot product attention calculation.

4. The multi-source remote sensing image classification method based on representation learning networks with intra-source and inter-source interactions according to claim 1, characterized in that, The method of performing implicit feature transfer through a weight matrix to achieve bidirectional feature enhancement between multi-source features and obtain the fused features of the two branches includes: Each value of the transmitted feature is subjected to a gated weighting operation, where the transmitted feature is the component multiplication matrix weight matrix corresponding to the single-source feature; then it is passed through a 1×1 convolutional layer, a BN layer, and a Sigmoid function for gated weighting in sequence, and multiplied element-wise with the transmitted feature to achieve bidirectional feature enhancement between multi-source features, resulting in the fused features of the two branches.

5. The multi-source remote sensing image classification method based on representation learning networks with intra-source and inter-source interactions according to claim 3, characterized in that, The complementary representation supervised learning strategy performs a global average pooling operation along the channel direction on the intermediate variables of the dual-branch to obtain the global average pooling operation result. Calculate the information entropy H(•) of the global average pooling operation result, and take the difference between the information entropy H(•) of the two branches to obtain the difference in information richness. Calculate the cosine similarity parameter of the intermediate variable in the two-branch system; The difference in information richness and the cosine similarity parameter are imported into the max-min normalization optimization algorithm. The max-min normalization algorithm is then used to optimize the result of the global average pooling operation to adjust the optimization value range. Based on the optimized value range, a fusion loss function is output to supervise the feature interaction process.

6. The multi-source remote sensing image classification method based on representation learning networks with intra-source and inter-source interactions according to claim 5, characterized in that, The trained representation learning model based on intra-source and inter-source interactions is jointly supervised by the fusion loss function and the cross-entropy classification loss function, and is then trained using the training set.

7. The multi-source remote sensing image classification method based on representation learning networks with intra-source and inter-source interactions according to claim 6, characterized in that, The training process of the trained representation learning model based on intra-source and inter-source interactions is as follows: a) Read PAN and MS plots from multiple different sensors; b. Normalize the PAN image and the MS image so that the values ​​of all pixels are within a fixed range to obtain a normalized image; c. Construct training and testing sets using the normalized images; d. Construct a representation learning model based on intra-source and inter-source interactions. The learning network includes a residual module, a cross-source interaction enhancement fusion module, and a cascaded module. e. The representation learning model based on intra-source and inter-source interactions is trained using the training set, and the prediction error is backpropagated during the training process to update the parameters of the representation learning model, thereby obtaining the trained representation learning model.

8. A multi-source remote sensing image classification device based on a representation learning network for intra-source and inter-source interactions, characterized in that, include: The acquisition module is configured to acquire real-time remote sensing images; The classification module is configured to take the real-time remote sensing image as input to a trained representation learning model based on intra-source and inter-source interaction, so that the representation learning model extracts the dual-source features of the real-time remote sensing image through a dual-branch feature extraction network composed of multiple cascaded residual modules. Each residual module inputs the extracted dual-source features into the cross-source interaction enhancement fusion module for feature interaction. During the feature interaction process, a complementary representation supervision learning strategy is used for supervision, and then the feedback is fed back to the next level residual module until the last residual module outputs the final fused features. The final fused features of the two branches are stitched together and then passed through the output layer to obtain the category of the same object in the real-time remote sensing image; the representation learning model based on intra-source and inter-source interaction includes a two-branch feature extraction network, a cross-source interaction enhancement fusion module, a fully connected layer, and an output layer; The dual-branch feature extraction network comprises two branches with non-shared weights, each branch consisting of multiple cascaded residual modules. Each residual module is connected to a corresponding cross-source interaction enhancement fusion module. The dual-source features extracted by the residual modules at the same level of the two branches are input to the corresponding cross-source interaction enhancement fusion module for feature interaction. The fused features output by the cross-source interaction enhancement fusion module are input to the residual modules of the two branches and summed with the input features of the residual modules. During the feature interaction process, the cross-source interaction enhancement fusion module generates two intermediate variables and is supervised by the output control of a complementary representation supervised learning strategy. The addition unit connected to the last residual module is used to output the final fused features. The final fused features are converted into a one-dimensional vector through a fully connected layer and then classified through an output layer to obtain the classification result. The cross-source interaction enhancement fusion module consists of three parts: a preprocessing module, a source intra-dependency enhancement module, and an inter-source enhancement fusion module. The preprocessing module uses the self-attention mechanism in Transformer to process the global dependency of the intra-source features of the dual-source features extracted by the residual module to obtain the autocorrelation matrix; The source dependency enhancement module multiplies the autocorrelation matrix of the two branches by the corresponding coefficients, thereby updating each correlation score in the autocorrelation matrix to obtain the enhanced single-source feature. The source enhancement fusion module performs a fusion operation by multiplying the two enhanced single-source features element by element, then obtains the weight matrix of the fused features through the Sigmoid function, and performs implicit feature transfer through the weight matrix to achieve bidirectional feature enhancement between multi-source features to obtain the fused features of the two branches, which are then fed back to the output of the residual module.

Images