Multi-source remote sensing image collaborative classification method and system

Abstract

The application discloses a kind of multi-source remote sensing image collaborative classification method and system, comprising: with each pixel point as center extraction local neighborhood window, construct training set and test set, and initialize global information guide vector.Joint convolution layer and Transformer structure, local and global features of multi-source remote sensing image are extracted and updated layer by layer.Make full use of the advantages of quaternion and long short time memory network, extract effective multi-modal fusion features that describe the complementarity of multi-source remote sensing image.Construct cross-modal contrast learning structure, impose consistency constraint on global information learning process, guide the fusion of multi-modal features.Integrate the above structure, propose cross-modal memory Transformer network framework, realize the collaborative classification of multi-source remote sensing image through end-to-end training.The application has the advantages of maintaining multi-source image complementary relationship, strong general framework applicability, improving remote sensing intelligent processing capability, considering multi-source image characteristics, improving classification accuracy and the like.

Description

Technical Field

[0001] This invention relates to the field of remote sensing intelligent processing technology, and in particular to a multi-source remote sensing image collaborative classification method and system using a cross-modal memory Transformer network. Background Technology

[0002] With the development of remote sensing and imaging technologies, Earth observation data from multiple sensors has gradually increased, providing researchers with abundant multi-source remote sensing data, such as hyperspectral, lidar, and synthetic aperture radar data. However, due to differences in imaging mechanisms and observation methods among different sensors, a single sensor cannot simultaneously describe the high-resolution spectral and spatial information of ground features. Therefore, my country has successively issued the "National Civil Space Infrastructure Medium and Long-Term Development Plan (2015-2025)" and the "Ecological Environment Satellite Medium and Long-Term Development Plan (2021-2035)," both of which clearly emphasize the vigorous development of multi-source remote sensing collaborative monitoring technologies such as hyperspectral and radar. Consequently, the combined use of multi-sensor technologies for comprehensive detection and analysis has gradually attracted widespread attention. Among these, multi-source remote sensing image collaborative classification, as an important Earth observation information acquisition technology, is currently a research hotspot in the field of intelligent remote sensing processing.

[0003] Compared to traditional machine learning methods, deep learning-based multi-source remote sensing image collaborative classification methods exhibit stronger multimodal information modeling capabilities and higher classification accuracy. Most existing collaborative classification methods are directly designed based on feature-level fusion methods. However, due to the strong heterogeneity between data from different sensors, these feature fusion methods are easily affected by data heterogeneity, failing to fully exploit the complementary relationships between multi-source remote sensing images and limiting their collaborative classification performance. Furthermore, existing multimodal feature fusion methods often employ self-attention or cross-attention to achieve multimodal feature fusion. However, these methods only focus on the input data itself, ignoring prior knowledge between different modalities or insufficient learning of complementary information due to inappropriate feature fusion methods. This, to some extent, restricts the accuracy of multi-source remote sensing image collaborative classification, especially in scenarios with multiple different sensors. Therefore, how to design an effective multimodal collaborative classification framework for multi-source remote sensing image fusion and collaborative classification is a key problem that urgently needs to be solved by engineers in the field of remote sensing intelligent processing. Summary of the Invention

[0004] This invention addresses the shortcomings of existing technologies by providing a multi-source remote sensing image collaborative classification method and system. Based on fully mining complementary information from multi-source remote sensing images, suppressing multimodal heterogeneity, and improving collaborative classification performance, it effectively mines discriminative representations of multi-source remote sensing images and achieves multimodal fusion, thereby improving collaborative classification accuracy.

[0005] To achieve the above-mentioned objectives, the technical solution adopted by the present invention is as follows:

[0006] A collaborative classification method for multi-source remote sensing images includes the following steps:

[0007] S1. Extract the local neighborhood window centered on each pixel in the multi-source remote sensing image to obtain two three-dimensional data cubes, and randomly initialize the global information guiding vector. Combine the corresponding category information to construct the training set and the test set.

[0008] S2. A multimodal feature extraction backbone network is built by combining convolutional layers and Transformer structure, and local and global features of multi-source remote sensing images are updated layer by layer across modalities;

[0009] S3 combines the advantages of quaternions and long short-term memory networks to design a memory Transformer module to extract multimodal fusion features that effectively describe the complementarity of multi-source remote sensing images;

[0010] S4. Construct a cross-modal contrastive learning structure to impose consistency constraints on the global information learning process and guide the multimodal feature fusion process;

[0011] S5. Integrating the above structures, a cross-modal memory Transformer network framework is proposed to achieve collaborative classification of multi-source remote sensing images through end-to-end training.

[0012] Furthermore, the specific steps of S1 are as follows:

[0013] Let the dimensions of the mode 1 hyperspectral image and the mode 2 synthetic aperture radar / lidar image in the multi-source remote sensing image be S, respectively. h ×H×W and S s ×H×W, where H and W are the height and width respectively, and S is the width. h and S s These represent the number of channels. First, the s×s neighborhood centered on the i-th pixel is selected as the spatial information. Principal component analysis is used to process the hyperspectral image, resulting in two three-dimensional data cubes. and The corresponding label is y i Where K is the number of spectral bands after dimensionality reduction.

[0014] Then, the training data are constructed separately:

[0015]

[0016] and test data:

[0017]

[0018] The corresponding training labels are

[0019] and test tags

[0020] Obtain the training set [χ] train Y train ] and test set [χ test Y test ]. and This represents the number of training and testing samples.

[0021] Furthermore, the specific steps of S2 are as follows:

[0022] S21. A preprocessing module is designed using convolutional layers to map multimodal data to the same dimension, adapting to different sensors and obtaining primary features. C represents the number of channels. The global feature is defined as a trainable global information guiding vector Z, randomly initialized to... n t m and m represent the number of tags and the dimension, respectively.

[0023] S22. By combining convolutional layers and Transformer structures, a multimodal feature extraction backbone network is constructed to extract local and global features from multi-source remote sensing images layer by layer:

[0024]

[0025] in, and These are the local modal feature extraction branches, This branch is used for modeling global cross-modal information, where l is the number of network layers. c l h l and w l Specify the number of output channels, height, and width.

[0026] S23. Based on the multi-head attention mechanism, construct a multi-modal feature fusion module to update the aforementioned local features layer by layer across modalities: and and global features: and

[0027] Furthermore, to update the global features, the input local features are... Mapped to key K respectively H Sum V H ;Utilizing fully connected layer f fc-Q (·) global features Mapping to query Q z , and the key K H Multiply and obtain the attention matrix A through the Softmax function. z . Az and V H Multiplying them together gives O. z Through a fully connected layer We obtain local to global labels and use the residual structure to obtain updated global features.

[0028]

[0029] in, For the l-th layer Transformer, d k For key K H The dimension. Similarly, the updated global features are obtained. Attention(·) is a multi-head attention function.

[0030] Furthermore, to update local features, the local features are... Projection for query Q H ;Utilizing fully connected layer f fc-K and f fc-V global features Mapping to key K z Sum V z Using K z and Q H Calculate attention matrix A H , will A H and V z Multiplying them together gives O. H Through the fully connected layer f fc-O We obtain global to local features and combine them with the residual structure to obtain updated local features. Similarly, the updated local features are obtained.

[0031]

[0032] Furthermore, the specific steps of S3 are as follows:

[0033] S31. Extract the multimodal local features from step S2. and Tokenization. Along and The channel dimension is used to convert it into vector form. Trainable weights W are introduced. H and W S Extract the feature labels of mode 1 respectively and mode 2 feature markers

[0034] S32. Define trainable category labels and Concatenated with feature markers and combined with shared location encoding P, we obtain:

[0035]

[0036] Where [·, ·] represent cascading operations. This yields multimodal local labels. and and global tags and

[0037] S33. Design a memory-enhancing attention module to achieve cross-modal information interaction and complete feature fusion.

[0038] For input Category tags First, through the fully connected layer f fc-inp Alignment Dimensions, and through a fully connected layer f fc-q Mapped to query q h ; For input Middle feature markers Through the fully connected layer f fc-k and f fc-v Mapping, and introducing additional extended "memory slots" m k and m v Additional encoding of prior information yields the key z. k Sum z v .

[0039] Multi-head attention is computed through a fully connected layer f. fc-out Alignment Dimension, generating updated modality 1 local category labels based on global information. h :

[0040]

[0041] Similarly, the updated global category label o is obtained. z ;

[0042] Mark the local category of mode 1 as o h Global category marker o z The updated local features are obtained by fusing them with the original category labels and feature labels. and global features Used to represent Mode 1 information:

[0043]

[0044] Obtain the local update after intramodal interaction and global features Used to represent modality 2 information, local information updated after intermodal interactions. and local features Used to jointly represent two modal information.

[0045] S34, To fuse intra-modal and inter-modal features and Quaternion convolutional layers are introduced to uncover their complex and nonlinear relationships, and the final multimodal fusion feature F is extracted through a feedforward network. HS .

[0046] Furthermore, the specific steps of S5 are as follows:

[0047] S51. Integrate the above-mentioned multimodal feature extraction backbone network, memory Transformer module and cross-modal contrastive learning structure to build a multi-source remote sensing image collaborative classification framework based on cross-modal memory Transformer network.

[0048] S52, Based on the multimodal fusion feature F obtained in step S3 HS Extract its category tags The softmax function is used to predict the corresponding ground object category, and the cross-entropy loss function is used to design the classification loss. Combining the global consistency constraint loss and the cross-entropy classification loss, the overall framework optimization function is designed as follows:

[0049]

[0050] in, and These represent the global consistency constraint loss and the cross-entropy classification loss, respectively.

[0051] S53, The training set [χ] constructed based on step S1 train Y train The above formula is optimized end-to-end, so that the proposed multi-source collaborative classification network framework can automatically extract multimodal fusion features from the input multi-source remote sensing images to achieve collaborative classification, and thus obtain a trained multi-source collaborative classification model.

[0052] S54, Test set χ test Input the data into a trained multi-source collaborative classification model to obtain predicted labels. and the real label Y test The performance of the proposed model in multi-source remote sensing collaborative classification was evaluated by comparison.

[0053] This invention discloses a multi-source remote sensing image collaborative classification system, which can be used to implement the above-mentioned multi-source remote sensing image collaborative classification method, specifically including:

[0054] Data preprocessing module: It is responsible for taking each pixel in the multi-source remote sensing image as the center, extracting the local neighborhood window, and obtaining two three-dimensional data cubes.

[0055] The global information guidance vector is randomly initialized, and training and test sets are constructed by combining the corresponding category information.

[0056] Multimodal feature extraction backbone network module: Combines convolutional layers and Transformer structure to extract local and global features of multi-source remote sensing images layer by layer.

[0057] Memory Transformer Module: Leveraging the advantages of quaternions and long short-term memory networks, a memory Transformer module is established to extract multimodal fusion features that effectively describe the complementarity of multi-source remote sensing images.

[0058] Cross-modal contrastive learning structure module: Constructs a cross-modal contrastive learning structure, applies consistency constraints to the global information learning process, and guides the multimodal feature fusion process.

[0059] Cross-modal memory Transformer network framework module: Integrates all modules to achieve collaborative classification of multi-source remote sensing images through end-to-end training.

[0060] The present invention also discloses a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the above-described multi-source remote sensing image collaborative classification method.

[0061] The present invention also discloses a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the above-described multi-source remote sensing image collaborative classification method.

[0062] Compared with the prior art, the advantages of the present invention are as follows:

[0063] 1. Maintaining the ability to learn complementary relationships between multi-source remote sensing images: This invention utilizes the cross-modal memory Transformer network framework, which can effectively maintain the ability to learn complementary relationships between multi-source remote sensing images, thereby fully exploring the information provided by different sensors.

[0064] 2. Strong applicability of the general framework: The designed multimodal feature extraction backbone network serves as a general framework, which can be flexibly applied to processing different types of data such as hyperspectral, lidar, and synthetic aperture radar, thereby improving the versatility and applicability of the algorithm.

[0065] 3. Enhance remote sensing intelligent processing capabilities: By integrating the advantages of convolutional layers, Transformer structures, quaternions, and long short-term memory networks, this method can enhance the remote sensing intelligent processing capabilities of multi-sensor collaboration, making the classification of multi-source remote sensing images more accurate and efficient.

[0066] 4. Fully consider the characteristics of multi-source remote sensing images: This invention comprehensively considers the inherent data attribute characteristics between multi-source remote sensing images, including data heterogeneity under different sensor scenarios, and can suppress multimodal heterogeneity and improve collaborative classification performance.

[0067] 5. Improved classification accuracy: Compared with existing collaborative classification models, this invention can better adapt to different sensor scenarios, thereby improving the accuracy and stability of collaborative classification of multi-source remote sensing images and providing higher-level technical support for remote sensing image processing. Attached Figure Description

[0068] Figure 1 This is a flowchart of the multi-source remote sensing collaborative classification method according to an embodiment of the present invention.

[0069] Figure 2 This is a structural diagram of the cross-modal memory Transformer network framework according to an embodiment of the present invention. Detailed Implementation

[0070] To make the objectives, technical solutions, and advantages of the present invention clearer, the present invention will be further described in detail below with reference to the accompanying drawings and examples.

[0071] Example 1

[0072] This embodiment illustrates a multi-source remote sensing image collaborative classification method using a cross-modal memory Transformer network, as provided by the present invention. Figure 1 As shown, it includes the following steps:

[0073] S1. Extract the local neighborhood window centered on each pixel in the multi-source remote sensing image to obtain two three-dimensional data cubes, and randomly initialize the global information guiding vector. Combine the corresponding category information to construct the training set and the test set.

[0074] Assuming that in multi-source remote sensing images, the dimensions of mode 1 (hyperspectral image) and mode 2 (synthetic aperture radar / lidar image) are S and S respectively. h ×H×W and S s ×H×W, where H and W are the height and width respectively, and S is the width. h and S s These represent the number of channels. First, the s×s neighborhood centered on the i-th pixel is selected as the spatial information. Principal component analysis is used to process the hyperspectral image, resulting in two three-dimensional data cubes. and The corresponding label is y i Where K is the number of spectral bands after dimensionality reduction.

[0075] Then, the training data are constructed separately:

[0076]

[0077] and test data:

[0078]

[0079] The corresponding training labels are

[0080] and test tags

[0081] Obtain the training set [χ] train Y train ] and test set [X test Y test ]. and This represents the number of training and testing samples.

[0082] S2. A multimodal feature extraction backbone network is built by combining convolutional layers and Transformer structures, and local and global features of multi-source remote sensing images are updated layer by layer across modalities.

[0083] First, a preprocessing module is designed using convolutional layers to map multimodal data to the same dimension, adapting to different sensors and obtaining primary features. C represents the number of channels. The global feature is defined as a trainable global information guiding vector Z, randomly initialized to... n t m and m represent the number of tags and the dimension, respectively.

[0084] Then, by combining convolutional layers and the Transformer structure, a multimodal feature extraction backbone network is constructed to extract local and global features from multi-source remote sensing images layer by layer:

[0085]

[0086] in, and These are the local modal feature extraction branches, This branch is used for modeling global cross-modal information, where l is the number of network layers. c l h l and w l Specify the number of output channels, height, and width.

[0087] Then, based on the multi-head attention mechanism, a multimodal feature fusion module is constructed to update the aforementioned local features layer by layer across modalities: and and global features: and

[0088] Specifically, to update global features, local features are input. and global features For example. First, Mapped to key K respectively H Sum V H ;Utilizing fully connected layer f fc-Q (·)Will Mapping to query Q z to K H Multiply and obtain the attention matrix A through the Softmax function. z Then, A z and V H Multiplying them together gives O. z Finally, through a fully connected layer We obtain local to global labels and use the residual structure to obtain updated global features.

[0089]

[0090] in, For the l-th layer Transformer, d k For key K H The dimension. Similarly, the updated global features are obtained. Attention(·) is a multi-head attention function.

[0091] Furthermore, to update local features, input local features are used. and global features For example. First, Mapped to query Q H ;Utilizing fully connected layer f fc-K and f fc-V Will Mapping to key K z Sum V z Then, using K z and Q H Calculate attention matrix A H , will A H and V z Multiplying them together gives O. H Finally, through the fully connected layer f fc-O We obtain global to local features and combine them with the residual structure to obtain updated local features. (Similarly, the updated local features are obtained) ):

[0092]

[0093] S3 combines the advantages of quaternions and long short-term memory networks to design a memory Transformer module, which extracts multimodal fusion features that effectively describe the complementarity of multi-source remote sensing images.

[0094] Based on the aforementioned multimodal features, we combine attention mechanisms, quaternions, and long short-term memory networks to construct a memory Transformer. By designing a memory-enhanced attention module, we can achieve cross-modal information interaction and fusion.

[0095] First, due to global features and All are Transformer-based notation formats; the multimodal local features extracted in step S2 must be processed first. and Tokenization. First, along with... and The channel dimension is used to convert it into vector form. Then, trainable weights W are introduced. H and W S Extract the feature labels of mode 1 respectively and mode 2 feature markers

[0096] Then, define trainable category labels. and Concatenated with feature markers and combined with shared location encoding P, we obtain:

[0097]

[0098] in, This is a cascaded operation. Through the above tokenization process, multimodal local tags are obtained. and and global tags and

[0099] Then, multimodal fusion is implemented based on the designed memory Transformer module. Taking the first memory-enhanced attention module as an example, for the input... Category tags First, through the fully connected layer f fc-inp Alignment Dimensions, and through a fully connected layer f fc-q Mapped to query q h For input Middle feature markers Through the fully connected layer f fc-k and f fc-v Mapping, and introducing additional extended "memory slots" m k and m v Additional encoding of prior information yields the key z. k Sum z v Calculate multi-head attention and pass it through a fully connected layer f. fc-out Alignment Dimension, generating updated modality 1 local category labels based on global information. h :

[0100]

[0101] Similarly, the updated global category tag o is obtained. z The local category label for mode 1 is o. h Global category marker o z The updated local features are obtained by fusing them with the original category labels and feature labels. and global features Used to represent Mode 1 information:

[0102]

[0103] Similar to formulas (6) and (7), the local data after intramodal interaction update is further obtained. and global features Used to represent modality 2 information, local information updated after intermodal interactions. and local features Used to jointly represent two modal information.

[0104] Finally, to integrate the above intra-modal and inter-modal features and Quaternion convolutional layers are introduced to uncover their complex and nonlinear relationships, and the final multimodal fusion feature F is extracted through a feedforward network. HS .

[0105] Therefore, based on the memory Transformer module designed above, multimodal fusion features F that effectively describe the complementarity of multi-source remote sensing images can be extracted from multi-source remote sensing images. HS and updated global features and

[0106] S4. Construct a cross-modal contrastive learning structure to impose consistency constraints on the global information learning process and guide the multimodal feature fusion process.

[0107] Considering the consistency of global information, a contrastive learning approach is introduced. Utilizing contrastive learning loss, similarity metric loss, and other methods, a global consistency constraint loss is designed to construct a cross-modal contrastive learning structure for the aforementioned layer-by-layer extracted global features. and By imposing consistency constraints, the entire network learns global information under dynamic supervision, while simultaneously guiding the extraction and updating of local information, in order to uncover more discriminative and expressive multimodal fusion features F. HS .

[0108] S5. Integrating the above three structures, a cross-modal memory Transformer network framework is proposed, which achieves collaborative classification of multi-source remote sensing images through end-to-end training.

[0109] First, by integrating the aforementioned multimodal feature extraction backbone network, memory Transformer module, and cross-modal contrastive learning structure, a multi-source remote sensing image collaborative classification framework based on the cross-modal memory Transformer network is constructed.

[0110] Then, based on the multimodal fusion feature F obtained in step S4 HS Extract its category tags The softmax function is used to predict the corresponding ground object category, and the cross-entropy loss function is used to design the classification loss. Combining the global consistency constraint loss and the cross-entropy classification loss, the overall framework optimization function is designed as follows:

[0111]

[0112] in, and These represent the global consistency constraint loss and the cross-entropy classification loss, respectively.

[0113] Finally, based on the training set [χ] constructed in step S1 train Y train The above formula (7) is optimized end-to-end so that the proposed multi-source collaborative classification network framework can automatically extract multi-modal fusion features from the input multi-source remote sensing images to achieve collaborative classification, and thus obtain a trained multi-source collaborative classification model.

[0114] Finally, the test set χ test Input the data into a trained multi-source collaborative classification model to obtain predicted labels. and the real label Y test The performance of the proposed model in multi-source remote sensing collaborative classification was evaluated by comparison.

[0115] Example 2

[0116] Building upon Example 1, this embodiment selects a publicly available multi-source remote sensing dataset (Augsburg dataset) to conduct simulation experiments on the multi-source remote sensing collaborative classification method proposed in this invention. This dataset, collected in Augsburg, Germany, includes spaceborne hyperspectral images, dual-polarization synthetic aperture radar images, and digital surface model images. The entire dataset contains 7 categories and 332×485 pixels, with a spatial resolution of 30 meters. In this embodiment, standard training and testing sets are used for model training and testing. Furthermore, this embodiment selects two novel multi-source remote sensing image collaborative classification algorithms from the past three years as comparison methods, including: Dual-channel Spatial, Spectral, and Multiscale AttentionConvLSTM Neural Network (Dual-channel A... 3 CLNN.IEEE Trans.NeuralNetw.Learn.Syst.2022), Asymmetric Feature Fusion Network (AsyFFNet.IEEETrans.Neural Netw.Learn.Syst..2023).

[0117] Table 1. Classification results (%) of different classification algorithms on the Augsburg dataset.

[0118]

[0119] Table 1 presents the classification results (%) of the algorithm of this invention and different comparison methods on this dataset, including overall accuracy (OA), average accuracy (AA), and Kappa coefficient. Compared with existing Dual-channel A... 3 Compared with the CLNN model, the cross-modal memory Transformer network proposed in this invention improves the OA index by 0.97% and 2.10%, respectively, the AA index by 6.64% and 1.88%, respectively, and the Kappa coefficient distribution by 1.55% and 2.96%.

[0120] This invention addresses the field of intelligent remote sensing processing by proposing a multi-source remote sensing image collaborative classification method using a cross-modal memory Transformer network. Through network structure derivation, experimental results, and comparative analysis, the feasibility and effectiveness of the proposed method in cross-modal feature fusion and multi-modal collaborative classification of multi-source remote sensing images are demonstrated.

[0121] In another embodiment of the present invention, a multi-source remote sensing image collaborative classification system using a cross-modal memory Transformer network is provided. This system can be used to implement the above-described multi-source remote sensing image collaborative classification method, specifically including:

[0122] Data preprocessing module: It is responsible for taking each pixel in the multi-source remote sensing image as the center, extracting the local neighborhood window, and obtaining two three-dimensional data cubes.

[0123] The global information guidance vector is randomly initialized, and training and test sets are constructed by combining the corresponding category information.

[0124] Multimodal feature extraction backbone network module: Combines convolutional layers and Transformer structure to extract local and global features of multi-source remote sensing images layer by layer.

[0125] Memory Transformer Module: Leveraging the advantages of quaternions and long short-term memory networks, a memory Transformer module is established to extract multimodal fusion features that effectively describe the complementarity of multi-source remote sensing images.

[0126] Cross-modal contrastive learning structure module: Constructs a cross-modal contrastive learning structure, applies consistency constraints to the global information learning process, and guides the multimodal feature fusion process.

[0127] Cross-modal memory Transformer network framework module: Integrates all modules to achieve collaborative classification of multi-source remote sensing images through end-to-end training.

[0128] In another embodiment of the present invention, a terminal device is provided, comprising a processor and a memory. The memory stores a computer program, which includes program instructions. The processor executes the program instructions stored in the computer storage medium. The processor may be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. It is the computing and control core of the terminal, suitable for implementing one or more instructions, specifically suitable for loading and executing one or more instructions to achieve a corresponding method flow or corresponding function. The processor described in this embodiment of the present invention can be used for the operation of a multi-source remote sensing image collaborative classification method, including the following steps:

[0129] S1. Extract the local neighborhood window centered on each pixel in the multi-source remote sensing image to obtain two three-dimensional data cubes, and randomly initialize the global information guiding vector. Combine the corresponding category information to construct the training set and the test set.

[0130] S2. A multimodal feature extraction backbone network is built by combining convolutional layers and Transformer structure, and local and global features of multi-source remote sensing images are updated layer by layer across modalities;

[0131] S3 combines the advantages of quaternions and long short-term memory networks to design a memory Transformer module to extract multimodal fusion features that effectively describe the complementarity of multi-source remote sensing images;

[0132] S4. Construct a cross-modal contrastive learning structure to impose consistency constraints on the global information learning process and guide the multimodal feature fusion process;

[0133] S5. Integrating the above structures, a cross-modal memory Transformer network framework is proposed to achieve collaborative classification of multi-source remote sensing images through end-to-end training.

[0134] In another embodiment of the present invention, a storage medium is provided, specifically a computer-readable storage medium (Memory). This computer-readable storage medium is a memory device in a terminal device used to store programs and data. It is understood that the computer-readable storage medium here can include both the built-in storage medium in the terminal device and extended storage media supported by the terminal device. The computer-readable storage medium provides storage space that stores the terminal's operating system. Furthermore, this storage space also stores one or more instructions suitable for loading and execution by a processor. These instructions can be one or more computer programs (including program code). It should be noted that the computer-readable storage medium here can be high-speed random access memory (RAM) or non-volatile memory, such as at least one disk storage device.

[0135] One or more instructions stored in a computer-readable storage medium can be loaded and executed by a processor to implement the corresponding steps of the multi-source remote sensing image collaborative classification method in the above embodiments; one or more instructions in the computer-readable storage medium are loaded and executed by the processor in the following steps:

[0136] S1. Extract the local neighborhood window centered on each pixel in the multi-source remote sensing image to obtain two three-dimensional data cubes, and randomly initialize the global information guiding vector. Combine the corresponding category information to construct the training set and the test set.

[0137] S2. A multimodal feature extraction backbone network is built by combining convolutional layers and Transformer structure, and local and global features of multi-source remote sensing images are updated layer by layer across modalities;

[0138] S3 combines the advantages of quaternions and long short-term memory networks to design a memory Transformer module to extract multimodal fusion features that effectively describe the complementarity of multi-source remote sensing images;

[0139] S4. Construct a cross-modal contrastive learning structure to impose consistency constraints on the global information learning process and guide the multimodal feature fusion process;

[0140] S5. Integrating the above structures, a cross-modal memory Transformer network framework is proposed to achieve collaborative classification of multi-source remote sensing images through end-to-end training.

[0141] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, read-only optical disc storage, optical storage, etc.) containing computer-usable program code.

[0142] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0143] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0144] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0145] Those skilled in the art will recognize that the embodiments described herein are intended to help the reader understand the implementation methods of the present invention, and should be understood that the scope of protection of the present invention is not limited to such specific statements and embodiments. Those skilled in the art can make various other specific modifications and combinations based on the technical teachings disclosed in this invention without departing from the spirit of the invention, and these modifications and combinations are still within the scope of protection of the present invention.

Claims

1. A collaborative classification method for multi-source remote sensing images, characterized in that, Includes the following steps: S1. Extract the local neighborhood window centered on each pixel in the multi-source remote sensing image to obtain two three-dimensional data cubes, and randomly initialize the global information guiding vector. Combine the corresponding category information to construct the training set and the test set. S2. A multimodal feature extraction backbone network is built by combining convolutional layers and Transformer structure, and local and global features of multi-source remote sensing images are updated layer by layer across modalities; S3 combines the advantages of quaternions and long short-term memory networks to design a memory Transformer module to extract multimodal fusion features that effectively describe the complementarity of multi-source remote sensing images; The specific steps are as follows: S31. Convert the multimodal local features into vector form and extract modal feature labels; S32. Define a concatenation of trainable category labels and feature labels, and combine it with shared location encoding to obtain multimodal local labels and global labels; S33. Design a memory-enhanced attention module to achieve cross-modal information interaction. Introduce additional extended "memory slots" to encode prior information to obtain keys and values. Based on multi-head attention, generate updated local and global category labels and fuse them to obtain updated local and global features, thereby realizing intra-modal and inter-modal interaction. S34. Quaternion convolutional layers are introduced to explore their complex and nonlinear interaction relationships and extract the final multimodal fusion features; S4. Construct a cross-modal contrastive learning structure to impose consistency constraints on the global information learning process and guide the multimodal feature fusion process; S5. Integrating the above structures, a cross-modal memory Transformer network framework is proposed to achieve collaborative classification of multi-source remote sensing images through end-to-end training.

2. The multi-source remote sensing image collaborative classification method according to claim 1, characterized in that: The specific steps of S1 are as follows: Let the dimensions of the mode 1 hyperspectral image and the mode 2 synthetic aperture radar / lidar image in the multi-source remote sensing image be respectively... and H and W represent height and width, respectively, and S... h and S s These represent the number of channels; firstly, the s×s neighborhood centered on the i-th pixel is selected as the spatial information, and principal component analysis is used to process the hyperspectral image to obtain two three-dimensional data cubes. and The corresponding tag is ;in, This represents the number of spectral bands after dimensionality reduction. Then, the training data are constructed separately: ， And test data: ， The corresponding training labels are , and test tags , Obtain the training set and test set ; and This represents the number of training and testing samples.

3. The multi-source remote sensing image collaborative classification method according to claim 2, characterized in that: The specific steps of S2 are as follows: S21. A preprocessing module is designed using convolutional layers to map multimodal data to the same dimension, adapting to different sensors and obtaining primary features. , C represents the number of channels; the global feature is defined as a trainable global information guiding vector Z, randomly initialized to... n t and m are the number of labels and the dimension, respectively; S22. By combining convolutional layers and Transformer structures, a multimodal feature extraction backbone network is constructed to extract local and global features from multi-source remote sensing images layer by layer: ， in, and These are the local modal feature extraction branches, For the global cross-modal information modeling branch, For network layers, , c l h l and Output the number of channels, height, and width; S23. Based on the multi-head attention mechanism, construct a multi-modal feature fusion module to update the aforementioned local features layer by layer across modalities: and and global features: and .

4. The multi-source remote sensing image collaborative classification method according to claim 3, characterized in that: To update global features, input local features are... Mapped to keys respectively Sum ; Utilizing fully connected layers global features Mapping to query , and key Multiply and obtain the attention matrix through the Softmax function. ;Will and Multiply, we get ; through fully connected layers We obtain local to global labels and use the residual structure to obtain updated global features. : ， in, For the l-th layer Transformer, For key The dimension; similarly, the updated global features are obtained. , For multi-head attention functions.

5. The multi-source remote sensing image collaborative classification method according to claim 4, characterized in that: To update local features, the local features are... Projection for query ; Utilizing fully connected layers and global features Map to key Sum ;use and Calculate the attention matrix ,Will and Multiply, we get ; through fully connected layers We obtain global to local features and combine them with the residual structure to obtain updated local features. Similarly, the updated local features are obtained. : 。 6. The multi-source remote sensing image collaborative classification method according to claim 5, characterized in that: S31 specifically involves: extracting the multimodal local features from step S2. and Tokenization; along and Transform it into vector form using the channel dimension; introduce trainable weights. and Extract the feature labels of mode 1 respectively and mode 2 feature markers ; S32 specifically defines trainable class labels. and Concatenated with feature markers and combined with shared location encoding P, we obtain: ， in, For cascading operations, multimodal local labels are obtained. and and global tags and ; Specifically, S33 involves designing a memory-enhancing attention module to achieve cross-modal information interaction and complete feature fusion. For input Category tags First, through the fully connected layer Alignment Dimensions, and through fully connected layers Mapped to query ; For input Middle feature markers Through the fully connected layer and Mapping, and introducing additional extended "memory slots". and Additional encoding of prior information yields the key. Sum ; Multi-head attention is computed through a fully connected layer. Alignment Dimension, generating updated modality 1 local category labels based on global information. Based on multi-head attention ， Similarly, we obtain the updated global category label. ; Local category labeling for mode 1 Global category tags The updated local features are obtained by fusing them with the original category labels and feature labels. and global features Used to represent Mode 1 information: ， Obtain the local update after intramodal interaction and global features Used to represent modality 2 information, local information updated after intermodal interactions. and local features Used to jointly represent two modal information; S34 specifically refers to: fusing intra-modal and inter-modal features. , , and Quaternion convolutional layers are introduced to explore their complex and nonlinear interaction relationships, and the final multimodal fusion features are extracted through a feedforward network. .

7. The multi-source remote sensing image collaborative classification method according to claim 6, characterized in that: The specific steps for S5 are as follows: S51. Integrate the above-mentioned multimodal feature extraction backbone network, memory Transformer module and cross-modal contrastive learning structure to build a multi-source remote sensing image collaborative classification framework based on cross-modal memory Transformer network; S52. Multimodal fusion features obtained in step S3 Extract its category tags The softmax function is used to predict the corresponding ground object category, and the cross-entropy loss function is used to design the classification loss. Combining the global consistency constraint loss and the cross-entropy classification loss, the overall framework optimization function is designed as follows: ， in, and These represent the global consistency constraint loss and the cross-entropy classification loss, respectively. S53, Training set constructed based on step S1 The above formula is optimized end-to-end so that the proposed multi-source collaborative classification network framework can automatically extract multi-modal fusion features from the input multi-source remote sensing images to achieve collaborative classification, thereby obtaining a well-trained multi-source collaborative classification model. S54, Test set Input the data into a trained multi-source collaborative classification model to obtain predicted labels. and with real labels The performance of the proposed model in multi-source remote sensing collaborative classification was evaluated by comparison.

8. A multi-source remote sensing image collaborative classification system using a cross-modal memory Transformer network, characterized in that: This system can be used to implement the multi-source remote sensing image collaborative classification method according to any one of claims 1 to 7, specifically including: The data preprocessing module is responsible for extracting local neighborhood windows from each pixel in the multi-source remote sensing image to obtain two three-dimensional data cubes. The global information guidance vector is randomly initialized, and training and test sets are constructed by combining the corresponding category information; Multimodal feature extraction backbone network module: combines convolutional layers and Transformer structure to extract local and global features of multi-source remote sensing images layer by layer; Memory Transformer Module: Leveraging the advantages of quaternions and long short-term memory networks, a memory Transformer module is established to extract multimodal fusion features that effectively describe the complementarity of multi-source remote sensing images; Specifically, this includes: converting multimodal local features into vector form and extracting modal feature labels; Define a concatenation of trainable category labels and feature labels, and combine it with shared location encoding to obtain multimodal local labels and global labels; The design of the memory-enhanced attention module enables cross-modal information interaction. It introduces an additional extended "memory slot" to encode prior information, obtaining keys and values. Based on multi-head attention, it generates updated local and global category labels, which are then fused to obtain updated local and global features, enabling intra-modal and inter-modal interaction. Quaternion convolutional layers are introduced to explore their complex and nonlinear interaction relationships and extract the final multimodal fusion features; Cross-modal contrastive learning structure module: Constructs a cross-modal contrastive learning structure, applies consistency constraints to the global information learning process, and guides the multimodal feature fusion process; Cross-modal memory Transformer network framework module: Integrates all modules to achieve collaborative classification of multi-source remote sensing images through end-to-end training.

9. A computer device, characterized in that: It includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the program, implements the multi-source remote sensing image collaborative classification method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: It stores a computer program that, when executed by a processor, implements the multi-source remote sensing image collaborative classification method according to any one of claims 1 to 7.

Images