A multi-source remote sensing image classification method based on multimodal quaternion representation network

Abstract

The present invention provides a multi-source remote sensing image classification method based on a multi-modal quaternion representation network, which relates to the technical field of remote sensing image processing, including obtaining a hyperspectral image and a laser radar image data set, normalizing and reducing the dimension of the data set, and dividing the data set into a training set and a test set; sequentially constructing a multi-modal quaternion representation module, a three-level fusion framework, and a dual-branch quaternion convolutional Transformer network, and constructing a multi-modal quaternion representation network based on the multi-modal quaternion representation module, the three-level fusion framework, and the dual-branch quaternion convolutional Transformer network; training the multi-modal quaternion representation network according to preset training parameters and loss functions of the training set to obtain a trained multi-source remote sensing image classification model, and testing the test set according to the trained multi-source remote sensing image classification model to obtain a classification result. The present invention can simply and effectively fuse multi-modal features.

Description

Technical Field

[0001] The present invention relates to the technical field of remote sensing image processing, and in particular to a multi-source remote sensing image classification method based on a multi-modal quaternion representation network. Background Art

[0002] With the rapid development of satellite sensor technology, remote sensing images from multiple sensors provide information with different characteristics for the same observation area. For example, hyperspectral data and lidar data provide rich spatial spectral information and accurate elevation spatial information of land cover, respectively. The joint application of multi-source remote sensing data can make full use of its complementary information to improve the accuracy of classification, and has been widely used in environmental monitoring, urban planning and other fields.

[0003] In the joint classification task of multi-source remote sensing data, there is complementarity between the two different modal features, traditional manual features and deep semantic features. The effective fusion of these two different modal features has very important research value and practical significance for improving classification performance. However, the existing multimodal feature fusion methods have problems such as insufficient interaction between features, redundancy caused by feature stacking, and complex fusion architecture. How to effectively fuse multimodal features needs to be solved urgently.

[0004] Quaternions are hypercomplex numbers with one real part and three imaginary parts, which are naturally suitable for representing three-dimensional or four-dimensional vectors. The algebraic properties of quaternions can capture the potential interdependencies between components and realize compact interactions between components. There is an urgent need for a multi-source remote sensing image classification method based on a multimodal quaternion representation network to simply and effectively fuse multimodal features. Summary of the invention

[0005] The purpose of the present invention is to provide a multi-source remote sensing image classification method based on a multimodal quaternion representation network to improve the above problems. In order to achieve the above purpose, the technical solution adopted by the present invention is as follows:

[0006] In a first aspect, the present application provides a multi-source remote sensing image classification method based on a multimodal quaternion representation network, comprising:

[0007] Obtain hyperspectral image and lidar image datasets, normalize and reduce their dimensionality, and divide them into training and test sets;

[0008] A multimodal quaternion representation module, a three-level fusion framework, and a dual-branch quaternion convolutional Transformer network are sequentially constructed, and a multimodal quaternion representation network is constructed based on the multimodal quaternion representation module, the three-level fusion framework, and the dual-branch quaternion convolutional Transformer network;

[0009] The training set is trained according to the preset training parameters and loss function to train the multimodal quaternion representation network to obtain a trained multi-source remote sensing image classification model, and the test set is tested according to the trained multi-source remote sensing image classification model to obtain a classification result.

[0010] Optionally, the training set is trained according to preset training parameters and loss function to train the multimodal quaternion representation network, including:

[0011] Inputting the data in the training set into a multimodal quaternion representation module for processing to obtain an output of the multimodal quaternion representation module, wherein the output of the multimodal quaternion representation module includes a spatial feature quaternion representation of a hyperspectral image, a spatial feature quaternion representation of a lidar image, and a spectral feature quaternion representation of a hyperspectral image;

[0012] The output of the multimodal quaternion representation module is respectively input into the first-level multimodal feature cross-fusion module and the second-level multimodal feature cross-fusion module of the three-level fusion framework for processing to obtain the first-level fused multimodal feature output and the second-level fused multimodal feature output;

[0013] The multimodal feature output of the first-level fusion and the multimodal feature output of the second-level fusion are respectively input into a dual-branch quaternion convolutional Transformer network for processing to obtain the output of the dual-branch quaternion convolutional Transformer network;

[0014] The output of the dual-branch quaternion convolutional Transformer network is sent to the third-level multimodal feature cross-fusion module for processing, and the output of the third-level multimodal feature cross-fusion module and the output of the dual-branch quaternion convolutional Transformer network are sequentially input into a preset fully connected layer for classification to obtain a classification result, wherein the preset fully connected layer includes two quaternion fully connected layers and one fully connected layer.

[0015] Optionally, obtain hyperspectral image and lidar image datasets, normalize and reduce their dimensionality, and divide them into training and test sets, including:

[0016] The acquired hyperspectral image and lidar image are normalized so that the value range of each pixel in the hyperspectral image and lidar image is within the preset value;

[0017] The principal component analysis method is used to reduce the spectral dimension of the normalized hyperspectral image to obtain a low-dimensional hyperspectral image;

[0018] Taking each pixel point of the low-dimensional hyperspectral image and the laser radar image as the center, cutting out the low-dimensional hyperspectral image and the laser radar image data blocks according to the preset sampling window size;

[0019] The captured low-dimensional hyperspectral image and lidar image data blocks are divided into a training set and a test set according to the preset number of training samples and test samples.

[0020] Optionally, the step of inputting the data in the training set into a multimodal quaternion representation module for processing to obtain an output of the multimodal quaternion representation module comprises:

[0021] Determine the spatial feature quaternion representation of the hyperspectral image and the spatial feature quaternion representation of the lidar image based on a preset mathematical expression of the spatial feature quaternion representation of the hyperspectral image;

[0022] The quaternion representation of the spectral characteristics of the hyperspectral image is constructed based on the preset mathematical expression of the quaternion representation of the spectral characteristics of the hyperspectral image.

[0023] Optionally, the output of the multimodal quaternion representation module is inputted into a first-level multimodal feature cross-fusion module and a second-level multimodal feature cross-fusion module of a three-level fusion framework for processing, including:

[0024] Based on the three-dimensional quaternion convolution of the first-level multimodal feature cross-fusion module, feature extraction is performed on each position represented by the quaternion of the hyperspectral image space feature and each position represented by the quaternion of the lidar image space feature, to obtain the hyperspectral image position feature and the lidar image position feature;

[0025] Calculate the correlation value between the hyperspectral image position feature and the lidar image position feature using a softmax function;

[0026] Multiplying the correlation value and the hyperspectral image position feature based on Hamiltonian multiplication to obtain a weighted response of the lidar image spatial feature quaternion representation to the hyperspectral image spatial feature quaternion representation;

[0027] Performing an element-by-element addition operation based on the weighted response and the quaternion representation of the hyperspectral image spatial feature to obtain a first-level fused multimodal feature output;

[0028] The quaternion representation of the spectral features of the hyperspectral image and the quaternion representation of the spatial features of the lidar image are input into the second-level multimodal feature cross-fusion module for processing, wherein the above-mentioned feature extraction, correlation value calculation, weighted response calculation and element-by-element addition operations are repeated to obtain the second-level fused multimodal feature output.

[0029] Optionally, the multimodal feature output of the first-level fusion and the multimodal feature output of the second-level fusion are respectively input into a dual-branch quaternion convolution Transformer network for processing, including:

[0030] Sending the multimodal feature output of the first-level fusion to a preset spatial quaternion convolution Transformer block in a dual-branch quaternion convolution Transformer network for processing to obtain an output result of the spatial quaternion convolution Transformer block;

[0031] The multimodal feature output of the second-level fusion is sent to the preset spectral quaternion convolution Transformer block in the dual-branch quaternion convolution Transformer network for processing to obtain the output result of the spectral quaternion convolution Transformer block.

[0032] Optionally, the multimodal feature output of the first-level fusion is sent to a preset spatial quaternion convolution Transformer block in a dual-branch quaternion convolution Transformer network for processing, including:

[0033] A spatial quaternion convolutional label embedding block based on the spatial quaternion convolutional Transformer block is used to normalize the labeled spatial label based on layer normalization to obtain a standardized spatial label;

[0034] Calculate the standardized spatial markup according to the preset mathematical expressions of the query, key, and value components of the single-head three-dimensional quaternion convolution spatial attention to obtain the query, key, and value components of the single-head three-dimensional quaternion convolution spatial attention;

[0035] Calculate the query, key, and value components of the single-head 3D quaternion convolution spatial attention based on a preset mathematical expression of the single-head 3D quaternion convolution spatial attention score to obtain the single-head 3D quaternion convolution spatial attention score;

[0036] The single-head three-dimensional quaternion convolution spatial attention score is calculated based on the preset mathematical expression of the output of the single-head three-dimensional quaternion convolution spatial attention to obtain the output of the single-head three-dimensional quaternion convolution spatial attention;

[0037] The output of the single-head three-dimensional quaternion convolution spatial attention is calculated based on the mathematical expression of the output of the multi-head three-dimensional quaternion convolution spatial attention to obtain the output of the multi-head three-dimensional quaternion convolution spatial attention;

[0038] Normalize the output of the multi-head 3D quaternion convolution spatial attention layer to obtain the first eigenvector;

[0039] Processing the first feature vector based on two three-dimensional quaternion convolution layers with preset convolution kernel sizes to obtain a processed first feature vector;

[0040] The processed first eigenvector is nonlinearly transformed based on the Gaussian error linear unit activation function and the gating mechanism to obtain the output result of the spatial quaternion convolution Transformer block.

[0041] Optionally, the second-level fused multimodal feature output is sent to a preset spectral quaternion convolution Transformer block in a dual-branch quaternion convolution Transformer network for processing, including:

[0042] A spectral quaternion convolutional tag embedding block based on a spectral quaternion convolutional Transformer block is used to normalize the labeled spectral tags based on layer normalization to obtain standardized spectral tags;

[0043] Calculate the standardized spectral label according to the preset mathematical expressions of the query, key and value components of the single-head three-dimensional quaternion convolution spectral attention to obtain the query, key and value components of the single-head three-dimensional quaternion convolution spectral attention;

[0044] Based on the preset mathematical expression of the single-head three-dimensional quaternion convolution spectral attention score, the query, key and value components of the single-head three-dimensional quaternion convolution spectral attention are calculated to obtain the single-head three-dimensional quaternion convolution spectral attention score;

[0045] The single-head three-dimensional quaternion convolution spectral attention score is calculated based on the preset mathematical expression of the output of the single-head three-dimensional quaternion convolution spectral attention to obtain the output of the single-head three-dimensional quaternion convolution spectral attention;

[0046] The output of the single-head three-dimensional quaternion convolution spectral attention is calculated based on the mathematical expression of the output of the multi-head three-dimensional quaternion convolution spectral attention to obtain the output of the multi-head three-dimensional quaternion convolution spectral attention;

[0047] The output of the multi-head three-dimensional quaternion convolution spectral attention is layer-normalized to obtain the second eigenvector;

[0048] Processing the second feature vector based on two three-dimensional quaternion convolution layers with preset convolution kernel sizes to obtain a processed second feature vector;

[0049] The processed second eigenvector is nonlinearly transformed based on the Gaussian error linear unit activation function and the gating mechanism to obtain the output result of the spectral quaternion convolution Transformer block.

[0050] Optionally, the output of the dual-branch quaternion convolutional Transformer network is sent to a third-level multimodal feature cross-fusion module for processing, including:

[0051] Performing feature extraction on the output of a dual-branch quaternion convolution Transformer network based on a three-dimensional quaternion convolution of a third-level multimodal feature cross-fusion module, wherein the output of the dual-branch quaternion convolution Transformer network includes an output of a spatial branch and an output of a spectral branch;

[0052] The correlation between the output of the spatial branch and the output of the spectral branch is calculated based on the softmax function, and the correlation between the output of the spatial branch and the output of the spectral branch is multiplied by the feature of the output result of the spectral branch based on the Hamiltonian multiplication to obtain a weighted response of the output result of the spectral branch to the output result of the spatial branch;

[0053] The output of the spectral branch is added element by element to the weighted response of the output of the spatial branch and the output of the spatial branch to obtain the output of the third-level multimodal feature cross-fusion module.

[0054] Optionally, the output of the third-level multimodal feature cross-fusion module and the output of the dual-branch quaternion convolutional Transformer network are sequentially input into a preset fully connected layer for classification, including:

[0055] Based on the output of the third-level multimodal feature cross-fusion module and the output of the dual-branch quaternion convolutional Transformer network, cross entropy calculation is performed to obtain three classification losses;

[0056] The three classification losses are added together to obtain the multi-task loss function. The network is trained using the preset Adam optimizer and the multi-task loss function until convergence to obtain the trained multi-source remote sensing image classification model.

[0057] The beneficial effects of the present invention are:

[0058] First, the multimodal quaternion representation module designed by the present invention encodes multimodal features into the imaginary part of pure quaternions to explore spatial and spectral features, and utilizes the ability of quaternion networks in exploring the relationships between and within hidden spaces of quaternions to naturally integrate multimodal features, capture potential interdependencies between features, and encourage compact interactions. Secondly, the multimodal feature cross-fusion module designed by the present invention integrates multi-level features through three-level fusion, reduces the impact of heterogeneity on multimodal fusion, and maximizes the use of multi-source remote sensing image information. In addition, the dual-branch quaternion convolution Transformer network designed by the present invention combines the advantages of three-dimensional quaternion convolution in capturing local features and the ability of Transformer to model long-term dependencies, and simultaneously captures refined local and global features from spatial and spectral perspectives, which can simply and effectively fuse multimodal features.

[0059] Other features and advantages of the present invention will be described in the following description, and partly become apparent from the description, or be understood by implementing the embodiments of the present invention. The purpose and other advantages of the present invention can be realized and obtained by the structures particularly pointed out in the written description, claims, and drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0060] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for use in the embodiments are briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present invention and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without creative work.

[0061] Figure 1 It is a schematic flow chart of a multi-source remote sensing image classification method based on a multi-modal quaternion representation network described in an embodiment of the present invention;

[0062] Figure 2 It is a schematic diagram of a multi-source remote sensing image processing flow of a multi-modal quaternion representation network described in an embodiment of the present invention;

[0063] Figure 3 It is a schematic diagram of the processing flow of the multimodal space quaternion representation module described in an embodiment of the present invention;

[0064] Figure 4 It is a schematic diagram of the processing flow of the multimodal spectral quaternion representation module described in an embodiment of the present invention;

[0065] Figure 5 Schematic diagram of the processing flow of the multimodal feature cross-fusion module described in an embodiment of the present invention;

[0066] Figure 6Schematic diagram of the processing flow of the quaternion convolution Transformer module described in an embodiment of the present invention. DETAILED DESCRIPTION

[0067] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, rather than all of the embodiments. The components of the embodiments of the present invention generally described and shown in the drawings here can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present invention provided in the drawings is not intended to limit the scope of the claimed invention, but merely represents selected embodiments of the present invention. Based on the embodiments in the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0068] It should be noted that similar reference numerals and letters represent similar items in the following drawings, so once an item is defined in one drawing, it does not need to be further defined and explained in the subsequent drawings. At the same time, in the description of the present invention, the terms "first", "second", etc. are only used to distinguish the description and cannot be understood as indicating or implying relative importance.

[0069] Embodiment 1:

[0070] This embodiment provides a multi-source remote sensing image classification method based on a multimodal quaternion representation network.

[0071] See also Figure 1 , Figure 2 , Figure 3 , Figure 4 , Figure 5 and Figure 6 , the figure shows that the method includes step S1, step S2 and step S3.

[0072] Step S1, obtaining a hyperspectral image and a lidar image data set, normalizing and reducing the dimension thereof, and dividing them into a training set and a test set;

[0073] Specifically, step S1 includes step S11, step S12, step S13 and step S14.

[0074] Step S11, normalizing the acquired hyperspectral image and laser radar image so that the value range of each pixel in the hyperspectral image and the laser radar image is between preset values;

[0075] Step S12, using principal component analysis to reduce the spectral dimension of the normalized hyperspectral image to obtain a low-dimensional hyperspectral image;

[0076] Step S13, taking each pixel point of the low-dimensional hyperspectral image and the laser radar image as the center, and cutting out the low-dimensional hyperspectral image and the laser radar image data blocks according to a preset sampling window size;

[0077] In this embodiment, a preset sampling window size of 11×11 is used, and the number of retained principal components is preset to 30.

[0078] Step S14: The captured low-dimensional hyperspectral image and lidar image data blocks are divided into a training set and a test set according to the preset number of training samples and test samples.

[0079] Step S2, sequentially constructing a multimodal quaternion representation module, a three-level fusion framework, and a dual-branch quaternion convolutional Transformer network, and constructing a multimodal quaternion representation network based on the multimodal quaternion representation module, the three-level fusion framework, and the dual-branch quaternion convolutional Transformer network;

[0080] It can be understood that the multimodal quaternion representation module in this step includes a spatial feature quaternion representation and a spectral feature quaternion representation, and the spatial feature quaternion representation includes three parallel spatial feature extraction branches: the first branch includes a three-dimensional convolution layer for extracting depth features and a three-dimensional convolution layer for its feature dimension; the second branch includes a local binary pattern algorithm layer for extracting manual features and a three-dimensional convolution layer for its feature dimension; the third branch includes a Gabor filter layer for extracting manual features and a three-dimensional convolution layer for its feature dimension. Finally, the outputs of the three branches are embedded in the three imaginary parts of the pure quaternion as the output of the spatial feature quaternion representation.

[0081] The spectral feature quaternion representation includes shallow feature extraction of hyperspectral image data blocks through three parallel spectral feature extraction branches: the first branch contains an average pooling layer, a one-dimensional convolution layer with adaptive kernel size, and a Sigmoid activation function; the second branch contains a maximum pooling layer, a one-dimensional convolution layer with adaptive kernel size, and a Sigmoid activation function; the third branch contains two Convolutional layer, a ReLU activation layer and a Sigmoid activation function. Finally, the outputs of the three branches are embedded into the three imaginary parts of the pure quaternion as the output of the spectral feature quaternion representation.

[0082] Step S3: train the multimodal quaternion representation network according to preset training parameters and loss function on the training set to obtain a trained multi-source remote sensing image classification model, and test the test set according to the trained multi-source remote sensing image classification model to obtain a classification result.

[0083] Specifically, step S3 includes step S31, step S32, step S33 and step S34.

[0084] Step S31, inputting the data in the training set into a multimodal quaternion representation module for processing to obtain the output of the multimodal quaternion representation module, wherein the output of the multimodal quaternion representation module includes a spatial feature quaternion representation of a hyperspectral image, a spatial feature quaternion representation of a lidar image, and a spectral feature quaternion representation of a hyperspectral image;

[0085] Specifically, step S31 includes step S311 and step S312.

[0086] Step S311, determining the spatial feature quaternion representation of the hyperspectral image and the spatial feature quaternion representation of the laser radar image based on a preset mathematical expression of the hyperspectral image spatial feature quaternion representation;

[0087] It is understandable that reference Figure 3 , the mathematical expression of the quaternion representation of the spatial feature of the hyperspectral image is:

[0088] ;

[0089] ;

[0090] ;

[0091] ;

[0092] in, Output representing the quaternion representation of the spatial features of a hyperspectral image data block; , , Three orthogonal imaginary units representing quaternions; represents a hyperspectral image, , , Respectively represent the spatial coordinates and band number of the hyperspectral image; Represents the first modal feature obtained by performing shallow feature extraction on the input data using three-dimensional quaternion convolution; It represents the second modal feature obtained by shallow feature extraction of input data using the local binary pattern algorithm; It represents the third modal feature obtained by shallow feature extraction of input data using Gabor filtering algorithm; 3DConv represents three-dimensional convolution; represents the local binary pattern operator; represents the center pixel; Indicates the number of surrounding pixels; represents the Gabor filter operator; is the center frequency of the sinusoidal plane wave; and is the angle; Represents a convolution operation.

[0093] It should be noted that , , Follow the following operation rules:

[0094] ;

[0095] ;

[0096] ;

[0097] It should be noted that the local binary pattern operator The mathematical expression is:

[0098] ;

[0099] ;

[0100] in, represents the center pixel; Indicates the number of surrounding pixels; Represents the index number of the neighborhood pixel; Represents the gray value of a pixel; represents a step function.

[0101] It should be noted that the Gabor filter operator The mathematical expression is:

[0102] ;

[0103] ;

[0104] in, is the center frequency of the sinusoidal plane wave, and is the wave vector and Axis and Plane in frequency domain The angle in , , Respectively represent the spatial coordinates and band number of the hyperspectral image, yes The width of the Gaussian envelope in the domain. To simplify the experimental analysis, we set , and .

[0105] It should be noted that the mathematical expression of the spatial feature quaternion representation of the lidar image is the same as the mathematical expression of the spatial feature quaternion representation of the hyperspectral image.

[0106] Step S312: constructing a quaternion representation of the spectral characteristics of the hyperspectral image based on a preset mathematical expression of the quaternion representation of the spectral characteristics of the hyperspectral image.

[0107] Specifically, refer to Figure 2 , the mathematical expression of the quaternion representation of the spectral characteristics of the hyperspectral image is:

[0108] ;

[0109] ;

[0110] ;

[0111] ;

[0112] in, The output representing the quaternion representation of the spectral features of the hyperspectral image data block, It represents the first modal feature obtained by extracting the input data using average pooling, a one-dimensional convolutional layer with an adaptively determined kernel size, and a Sigmoid activation function. It represents the second modal feature obtained by extracting the input data using maximum pooling, a one-dimensional convolutional layer with an adaptively determined kernel size, and a Sigmoid activation function. Indicates the use of two The convolution kernel, ReLU activation function and Sigmoid activation function extract the input data to obtain the third modal feature. , , Represent the three orthogonal imaginary units of quaternions, represents a hyperspectral image, is element-wise multiplication, is the Sigmoid activation function, 1DConv represents one-dimensional convolution, Avgpooling represents average pooling operation, Maxpooling represents maximum pooling operation, ReLU represents ReLU activation function, and 2DConv represents two-dimensional convolution.

[0113] Step S32, inputting the output of the multimodal quaternion representation module into the first-level multimodal feature cross-fusion module and the second-level multimodal feature cross-fusion module of the three-level fusion framework for processing, to obtain the first-level fused multimodal feature output and the second-level fused multimodal feature output;

[0114] Specifically, step S32 includes step S321, step S322, step S323, step S324 and step S325.

[0115] Step S321, based on the three-dimensional quaternion convolution of the first-level multimodal feature cross-fusion module, feature extraction is performed on each position represented by the hyperspectral image spatial feature quaternion and each position represented by the laser radar image spatial feature quaternion, to obtain the hyperspectral image position feature and the laser radar image position feature;

[0116] Step S322, using a softmax function to calculate the correlation value between the hyperspectral image position feature and the laser radar image position feature;

[0117] Step S323, multiplying the correlation value and the hyperspectral image position feature based on Hamiltonian multiplication to obtain a weighted response of the lidar image spatial feature quaternion representation to the hyperspectral image spatial feature quaternion representation;

[0118] Step S324, performing element-by-element addition operation based on the weighted response and the quaternion representation of the hyperspectral image spatial feature to obtain a first-level fused multimodal feature output;

[0119] It can be understood that the mathematical expression of the multimodal feature output of the first level fusion is as follows:

[0120] ;

[0121] ;

[0122] in, represents the weighted response matrix; and Respectively represent each position of the LiDAR image and the hyperspectral image spatial feature quaternion representation, and They are the quaternion representations of the spatial features of LiDAR images and hyperspectral images respectively; Indicates that 3D quaternion convolution with kernel size; Indicates the use 3D quaternion convolution with kernel size; and Indicates the use 3D quaternion convolution with kernel size; represents element-wise addition; Represents the multimodal feature output of the first level fusion.

[0123] Step S325: input the quaternion representation of the spectral features of the hyperspectral image and the quaternion representation of the spatial features of the lidar image into the second-level multimodal feature cross-fusion module for processing, wherein the above-mentioned feature extraction, correlation value calculation, weighted response calculation and element-by-element addition operations are repeated to obtain the second-level fused multimodal feature output.

[0124] It can be understood that the specific operation method of step S325 is as follows:

[0125] Use three-dimensional quaternion convolution to extract features for each position represented by the quaternion of the spectral feature of the hyperspectral image. Use three-dimensional quaternion convolution to extract features for each position represented by the quaternion of the spatial feature of the lidar image. Use the softmax function to calculate the correlation between the features. Use Hamiltonian multiplication to multiply the correlation with the features of each position represented by the quaternion of the spectral feature of the hyperspectral image to obtain the weighted response of the quaternion of the spatial feature of the lidar image to the quaternion of the spectral feature of the hyperspectral image. Perform element-by-element addition operation on the weighted response and the quaternion representation of the spectral feature of the hyperspectral image to obtain the multimodal feature output of the second-level fusion, and its mathematical expression is as follows:

[0126] ;

[0127] ;

[0128] in, represents the weighted response matrix; Output representing the quaternion representation of the spectral features of the hyperspectral image data block; Represents the quaternion representation of the spatial features of the LiDAR image; Indicates that 3D quaternion convolution with kernel size; Indicates the use 3D quaternion convolution with kernel size; and Indicates the use 3D quaternion convolution with kernel size; represents element-wise addition; Represents the multimodal feature output of the second level fusion.

[0129] Step S33, inputting the multimodal feature output of the first-level fusion and the multimodal feature output of the second-level fusion into a dual-branch quaternion convolutional Transformer network for processing, to obtain the output of the dual-branch quaternion convolutional Transformer network;

[0130] Specifically, step S33 includes step S331 and step S332.

[0131] Step S331, sending the multimodal feature output of the first-level fusion to a preset spatial branch in a dual-branch quaternion convolutional Transformer network for processing to obtain an output result of the spatial branch;

[0132] It is understandable that, referring to Figure 2 and Figure 6 , the spatial branch consists of N spatial quaternion convolutional Transformer blocks, including spatial quaternion convolutional label embedding blocks, layer normalization, multi-head 3D quaternion convolutional spatial attention, and spatial quaternion feedforward network.

[0133] Among them, step S331 includes step S3311, step S3312, step S3313, step S3314, step S3315, step S3316, step S3317, step S3318 and step S3319.

[0134] Step S3311, based on the spatial quaternion convolution Transformer block, the spatial quaternion convolution label embedding block is used, and the labeled spatial label is standardized based on layer normalization to obtain a standardized spatial label;

[0135] Among them, get the first space mark The mathematical expression is as follows:

[0136] ;

[0137] in, and Represents 3D quaternion convolution with kernels of different spatial sizes and padding operations that keep the spatial dimensions unchanged; Represents the multimodal feature output of the first level fusion; each convolutional layer is followed by a batch normalization layer and a Gaussian error linear unit layer.

[0138] Step S3312: Calculate the first standardized spatial tag according to the preset mathematical expressions of the query, key and value components of the single-head three-dimensional quaternion convolution spatial attention to obtain the query, key and value components of the single-head three-dimensional quaternion convolution spatial attention;

[0139] Among them, the query, key and value components of the preset single-head 3D quaternion convolution spatial attention The mathematical expression is as follows:

[0140] ;

[0141] in, Represents the number of multi-head attention mechanisms; Indicates that 3D quaternion convolution with kernel size; represents the learnable weights used to compute the query, key, and value components;

[0142] Step S3313, calculating the query, key, and value components of the single-head three-dimensional quaternion convolution spatial attention based on a preset mathematical expression of the single-head three-dimensional quaternion convolution spatial attention score to obtain the single-head three-dimensional quaternion convolution spatial attention score;

[0143] Among them, the preset single-head three-dimensional quaternion convolution spatial attention score The mathematical expression is as follows:

[0144] ;

[0145] in, , Respectively represent the spatial branch The query and key components of a multi-head attention mechanism; Indicates the number of feature dimensions assigned to each detection head;

[0146] Step S3314: Calculate the single-head three-dimensional quaternion convolution spatial attention score based on the preset mathematical expression of the output of the single-head three-dimensional quaternion convolution spatial attention to obtain the output of the single-head three-dimensional quaternion convolution spatial attention;

[0147] Among them, the output of the preset single-head three-dimensional quaternion convolution spatial attention The mathematical expression is as follows:

[0148] ;

[0149] in, , Respectively represent the spatial branch Spatial attention scores and value components of a multi-head attention mechanism.

[0150] Step S3315: Calculate the output of the single-head three-dimensional quaternion convolution spatial attention based on the mathematical expression of the output of the multi-head three-dimensional quaternion convolution spatial attention to obtain the output of the multi-head three-dimensional quaternion convolution spatial attention;

[0151] Among them, the output of multi-head three-dimensional quaternion convolution spatial attention The mathematical expression is as follows:

[0152] ;

[0153] in, , , Represents the query, key, and value components of the 3D quaternion convolution spatial attention; Indicates a connection operation. Represents the output of a single-head 3D quaternion convolution spatial attention;

[0154] Step S3316, normalize the output of the multi-head three-dimensional quaternion convolution spatial attention layer to obtain an intermediate vector;

[0155] Among them, get the intermediate vector The mathematical expression is as follows:

[0156] ;

[0157] in, Representation layer normalization operation, Represents the output of multi-head 3D quaternion convolution spatial attention.

[0158] Step S3317: Processing the intermediate vector based on two three-dimensional quaternion convolution layers with preset convolution kernel sizes to obtain a processed intermediate vector;

[0159] The mathematical expression for processing the intermediate vector based on the three-dimensional quaternion convolution layer with two preset convolution kernel sizes is as follows:

[0160] ;

[0161] in, represents the processed intermediate vector, and Indicates that and 3D quaternion convolution with kernel size, Represents the unprocessed intermediate vector from the previous step.

[0162] Step S3318: Perform a nonlinear transformation on the processed first eigenvector based on the Gaussian error linear unit activation function and the gating mechanism to obtain the output result of the first spatial quaternion convolution Transformer block.

[0163] The mathematical expression for nonlinear transformation is as follows:

[0164] ;

[0165] in, is the output result of the first spatial quaternion convolution Transformer block; is the Gaussian error linear unit activation function; represents the intermediate vector after processing; Represents element-wise multiplication.

[0166] Step S3319: Based on steps S3311 to S3318, the output results of N spatial quaternion convolution Transformer blocks are obtained. .

[0167] Step S332: Send the second-level fused multimodal feature output to the preset spectral branch in the dual-branch quaternion convolutional Transformer network for processing to obtain the output result of the spectral quaternion convolutional Transformer block.

[0168] It is understandable that, referring to Figure 2 and Figure 6 , the spectral branch consists of N spectral quaternion convolutional Transformer blocks, including spectral quaternion convolutional label embedding blocks, layer normalization, multi-head 3D quaternion convolutional spectral attention, and spectral quaternion feedforward network.

[0169] Specifically, step S332 includes step S3321, step S3322, step S3323, step S3324, step S3325, step S3326, step S3327, step S3328 and step S3329.

[0170] Step S3321, embedding a spectral quaternion convolution label block based on the spectral quaternion convolution Transformer block, and standardizing the labeled spectral labels based on layer normalization to obtain standardized spectral labels;

[0171] Among them, the mathematical expression of the spectral quaternion convolutional label embedding block based on the spectral quaternion convolutional Transformer block is as follows:

[0172] ;

[0173] in, Indicates the first spectral marker; and Represents 3D quaternion convolutions with different spectral sizes; each convolution layer is followed by a batch normalization layer and a Gaussian error linear unit layer; Represents the multimodal feature output of the second level fusion.

[0174] Step S3322: Calculate the standardized spectral label according to the preset mathematical expressions of the query, key and value components of the single-head three-dimensional quaternion convolution spectral attention to obtain the query, key and value components of the single-head three-dimensional quaternion convolution spectral attention;

[0175] Among them, the query, key and value components of the preset single-head 3D quaternion convolution spectral attention The mathematical expression is as follows:

[0176] ;

[0177] in, Represents the number of multi-head attention mechanisms; Indicates that 3D quaternion convolution with kernel size; Indicates the first spectral marker; represents the learnable weights used to compute the query, key, and value components;

[0178] Step S3323, calculating the query, key and value components of the single-head three-dimensional quaternion convolution spectrum attention based on the preset mathematical expression of the single-head three-dimensional quaternion convolution spectrum attention score to obtain the single-head three-dimensional quaternion convolution spectrum attention score;

[0179] Among them, the mathematical expression of the preset single-head three-dimensional quaternion convolution spectral attention score is as follows:

[0180] ;

[0181] in, Represents the single-head 3D quaternion convolution spectral attention score; represents the softmax function; , Represents the spectral branch The query and key components of a multi-head attention mechanism; Indicates the number of feature dimensions assigned to each detection head;

[0182] Step S3324: Calculate the single-head three-dimensional quaternion convolution spectral attention score based on the preset mathematical expression of the output of the single-head three-dimensional quaternion convolution spectral attention to obtain the output of the single-head three-dimensional quaternion convolution spectral attention;

[0183] Among them, the mathematical expression of the output of the preset single-head three-dimensional quaternion convolution spectral attention is as follows:

[0184] ;

[0185] in, Represents the output of a single-head 3D quaternion convolution spectral attention; , Represents the spectral branch Spectral attention score and value components of a multi-head attention mechanism.

[0186] Step S3325, calculating the output of the single-head three-dimensional quaternion convolution spectral attention based on the mathematical expression of the output of the multi-head three-dimensional quaternion convolution spectral attention, to obtain the output of the multi-head three-dimensional quaternion convolution spectral attention;

[0187] Among them, the output of the multi-head three-dimensional quaternion convolution spectral attention The mathematical expression is:

[0188] ;

[0189] in, , , Represents the query, key, and value components of 3D quaternion convolutional spectral attention; Indicates a connection operation. Represents the output of single-head 3D quaternion convolution spectral attention;

[0190] Step S3326, normalize the output of the multi-head three-dimensional quaternion convolution spectral attention layer to obtain an intermediate vector;

[0191] Among them, get the intermediate vector The mathematical expression is as follows:

[0192] ;

[0193] in, Representation layer normalization operation, Represents the output of multi-head 3D quaternion convolution spectral attention.

[0194] Step S3327: Process the intermediate vector based on two three-dimensional quaternion convolution layers with preset convolution kernel sizes to obtain a processed intermediate vector;

[0195] The mathematical expression for processing the second feature vector based on the three-dimensional quaternion convolution layer with two preset convolution kernel sizes is as follows:

[0196] ;

[0197] in, represents the processed intermediate vector, and Indicates that and 3D quaternion convolution with kernel size, Represents the unprocessed intermediate vector from the previous step.

[0198] Step S3328: Perform a nonlinear transformation on the processed second eigenvector based on the Gaussian error linear unit activation function and the gating mechanism to obtain the output result of the spectral quaternion convolution Transformer block.

[0199] Among them, the mathematical expression of nonlinear transformation is as follows:

[0200] ;

[0201] in, is the output result of the first spectral quaternion convolution Transformer block; is the Gaussian error linear unit activation function; represents the intermediate vector after processing; Represents element-wise multiplication.

[0202] Step S3329: Based on steps S3321 to S3328, the output results of N spectral quaternion convolution Transformer blocks are obtained. .

[0203] Step S34, sending the output of the dual-branch quaternion convolutional Transformer network to the third-level multimodal feature cross-fusion module for processing, and inputting the output of the third-level multimodal feature cross-fusion module and the output of the dual-branch quaternion convolutional Transformer network into the preset fully connected layer for classification in sequence to obtain a classification result, wherein the preset fully connected layer includes two quaternion fully connected layers and one fully connected layer.

[0204] Specifically, step S34 includes step S341, step S342, step S343, step S344 and step S345.

[0205] It is understandable that reference Figure 5 In this step, the output results of the spatial branch and the spectral branch of the dual-branch quaternion convolutional Transformer network are input into the multimodal feature cross-fusion module to perform the third-level fusion in the three-level fusion framework. The specific steps are as follows:

[0206] Step S341, performing feature extraction on the output of the dual-branch quaternion convolution Transformer network based on the three-dimensional quaternion convolution of the third-level multimodal feature cross-fusion module, wherein the output of the dual-branch quaternion convolution Transformer network includes the output of the spatial branch and the output of the spectral branch;

[0207] Step S342: Calculate the correlation between the output of the spatial branch and the output of the spectral branch based on the softmax function, and multiply the correlation between the output of the spatial branch and the output of the spectral branch by the feature of the spectral branch output result based on the Hamiltonian multiplication to obtain a weighted response of the output result of the spectral branch to the output result of the spatial branch;

[0208] Step S343: perform element-by-element addition operation on the output result of the spectral branch to the weighted response of the output result of the spatial branch and the output result of the spatial branch to obtain the output of the third-level multimodal feature cross-fusion module.

[0209] It can be understood that the expression for the output of the third-level multimodal feature cross-fusion module is as follows:

[0210] ;

[0211] ;

[0212] in, represents the weighted response matrix; and represents the output results of the spatial branch and the spectral branch in step S33; and Respectively and Each position of , , and Indicates the use 3D quaternion convolution with kernel size; Represents the multimodal feature output of the third-level fusion.

[0213] It can be understood that in this step, after step S343, the following steps are also included:

[0214] Step S344, performing cross entropy calculation based on the output of the third-level multimodal feature cross fusion module and the output of the dual-branch quaternion convolution Transformer network to obtain three classification losses;

[0215] Step S345: Add the three classification losses to obtain a multi-task loss function, use the preset Adam optimizer and the multi-task loss function to train the network until convergence, and obtain a trained multi-source remote sensing image classification model.

[0216] It is understandable that the mathematical expression of the multi-task loss function is as follows:

[0217] ;

[0218] in, represents the overall loss, represents the loss of the spectral branch of the dual-branch quaternion convolutional Transformer network, represents the loss of the spatial branch of the dual-branch quaternion convolutional Transformer network, Represents the loss after three-level fusion.

[0219] Embodiment 2:

[0220] This example is conducted on the Trento multi-source remote sensing dataset, where average classification accuracy, overall classification accuracy, and Kappa coefficient are used as evaluation indicators of model classification accuracy. As shown in Table 1, the present invention has higher classification accuracy than the comparison model.

[0221] Experimental results show that the multimodal quaternion representation network designed by the present invention uses the mathematical properties of quaternions to naturally capture the potential interdependence between features and encourage compact interactions. In addition, the three-level fusion framework proposed by the present invention integrates multimodal features at the feature level and decision level from different network layers, suppresses the negative impact of multimodal heterogeneity, and enhances the representation ability and robustness of the network. Finally, the dual-branch quaternion convolutional Transformer network designed by the present invention simultaneously captures refined local and global features from spatial and spectral perspectives. Therefore, the present invention has excellent classification performance.

[0222] Table 1 Comparison results of different algorithms on the Trento dataset (fixed training samples)

[0223]

[0224] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

[0225] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any person skilled in the art can easily think of changes or substitutions within the technical scope disclosed by the present invention, which should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims.

Claims

1. A multi-source remote sensing image classification method based on a multimodal quaternion representation network, characterized in that: include: Obtain hyperspectral image and lidar image datasets, normalize and reduce their dimensionality, and divide them into training and test sets; A multimodal quaternion representation module, a three-level fusion framework, and a dual-branch quaternion convolutional Transformer network are sequentially constructed, and a multimodal quaternion representation network is constructed based on the multimodal quaternion representation module, the three-level fusion framework, and the dual-branch quaternion convolutional Transformer network; The training set is trained with a multimodal quaternion representation network according to preset training parameters and loss functions to obtain a trained multi-source remote sensing image classification model, and the test set is tested according to the trained multi-source remote sensing image classification model to obtain a classification result; Among them, the training set is trained according to the preset training parameters and loss function to train the multimodal quaternion representation network, including: Inputting the data in the training set into a multimodal quaternion representation module for processing to obtain an output of the multimodal quaternion representation module, wherein the output of the multimodal quaternion representation module includes a spatial feature quaternion representation of a hyperspectral image, a spatial feature quaternion representation of a lidar image, and a spectral feature quaternion representation of a hyperspectral image; The output of the multimodal quaternion representation module is respectively input into the first-level multimodal feature cross-fusion module and the second-level multimodal feature cross-fusion module of the three-level fusion framework for processing to obtain the first-level fused multimodal feature output and the second-level fused multimodal feature output; The multimodal feature output of the first-level fusion and the multimodal feature output of the second-level fusion are respectively input into a dual-branch quaternion convolutional Transformer network for processing to obtain the output of the dual-branch quaternion convolutional Transformer network; The output of the dual-branch quaternion convolutional Transformer network is sent to the third-level multimodal feature cross-fusion module for processing, and the output of the third-level multimodal feature cross-fusion module and the output of the dual-branch quaternion convolutional Transformer network are sequentially input into a preset fully connected layer for classification to obtain a classification result, wherein the preset fully connected layer includes two quaternion fully connected layers and one fully connected layer; Wherein, the data in the training set is input into the multimodal quaternion representation module for processing to obtain the output of the multimodal quaternion representation module, including: Determine the spatial feature quaternion representation of the hyperspectral image and the spatial feature quaternion representation of the lidar image based on a preset mathematical expression of the spatial feature quaternion representation of the hyperspectral image; Constructing the spectral characteristic quaternion representation of the hyperspectral image based on the preset mathematical expression of the spectral characteristic quaternion representation of the hyperspectral image; The output of the multimodal quaternion representation module is respectively inputted into the first-level multimodal feature cross-fusion module and the second-level multimodal feature cross-fusion module of the three-level fusion framework for processing, including: Based on the three-dimensional quaternion convolution of the first-level multimodal feature cross-fusion module, feature extraction is performed on each position represented by the quaternion of the hyperspectral image space feature and each position represented by the quaternion of the lidar image space feature, to obtain the hyperspectral image position feature and the lidar image position feature; Calculate the correlation value between the hyperspectral image position feature and the lidar image position feature using a softmax function; Multiplying the correlation value and the hyperspectral image position feature based on Hamiltonian multiplication to obtain a weighted response of the lidar image spatial feature quaternion representation to the hyperspectral image spatial feature quaternion representation; Performing an element-by-element addition operation based on the weighted response and the quaternion representation of the hyperspectral image spatial feature to obtain a first-level fused multimodal feature output; The quaternion representation of the spectral feature of the hyperspectral image and the quaternion representation of the spatial feature of the laser radar image are input into the second-level multimodal feature cross-fusion module for processing, wherein the above-mentioned feature extraction, correlation value calculation, weighted response calculation and element-by-element addition operations are repeated to obtain the second-level fused multimodal feature output; The multimodal feature output of the first-level fusion and the multimodal feature output of the second-level fusion are respectively input into a dual-branch quaternion convolution Transformer network for processing, including: Sending the multimodal feature output of the first-level fusion to a preset spatial quaternion convolution Transformer block in a dual-branch quaternion convolution Transformer network for processing to obtain an output result of the spatial quaternion convolution Transformer block; The multimodal feature output of the second-level fusion is sent to the preset spectral quaternion convolution Transformer block in the dual-branch quaternion convolution Transformer network for processing to obtain the output result of the spectral quaternion convolution Transformer block.

2. The multi-source remote sensing image classification method based on a multimodal quaternion representation network according to claim 1 is characterized in that ,Acquire the hyperspectral image and lidar image datasets, normalize and reduce their dimensionality, and divide them into training and test sets, including: The acquired hyperspectral image and lidar image are normalized so that the value range of each pixel in the hyperspectral image and lidar image is within the preset value; The principal component analysis method is used to reduce the spectral dimension of the normalized hyperspectral image to obtain a low-dimensional hyperspectral image; Taking each pixel point of the low-dimensional hyperspectral image and the laser radar image as the center, cutting out the low-dimensional hyperspectral image and the laser radar image data blocks according to the preset sampling window size; The captured low-dimensional hyperspectral image and lidar image data blocks are divided into a training set and a test set according to the preset number of training samples and test samples.

3. The multi-source remote sensing image classification method based on a multimodal quaternion representation network according to claim 1 is characterized in that , the multimodal feature output of the first-level fusion is sent to the preset spatial quaternion convolution Transformer block in the dual-branch quaternion convolution Transformer network for processing, including: A spatial quaternion convolutional label embedding block based on the spatial quaternion convolutional Transformer block is used to normalize the labeled spatial label based on layer normalization to obtain a standardized spatial label; Calculate the standardized spatial label according to the preset mathematical expressions of the query, key and value components of the single-head three-dimensional quaternion convolution spatial attention to obtain the query, key and value components of the single-head three-dimensional quaternion convolution spatial attention; Calculate the query, key, and value components of the single-head 3D quaternion convolution spatial attention based on a preset mathematical expression of the single-head 3D quaternion convolution spatial attention score to obtain the single-head 3D quaternion convolution spatial attention score; The single-head three-dimensional quaternion convolution spatial attention score is calculated based on the preset mathematical expression of the output of the single-head three-dimensional quaternion convolution spatial attention to obtain the output of the single-head three-dimensional quaternion convolution spatial attention; The output of the single-head three-dimensional quaternion convolution spatial attention is calculated based on the mathematical expression of the output of the multi-head three-dimensional quaternion convolution spatial attention to obtain the output of the multi-head three-dimensional quaternion convolution spatial attention; Normalize the output of the multi-head 3D quaternion convolution spatial attention layer to obtain the first eigenvector; Processing the first feature vector based on two three-dimensional quaternion convolution layers with preset convolution kernel sizes to obtain a processed first feature vector; The processed first eigenvector is nonlinearly transformed based on the Gaussian error linear unit activation function and the gating mechanism to obtain the output result of the spatial quaternion convolution Transformer block.

4. The multi-source remote sensing image classification method based on a multimodal quaternion representation network according to claim 1 is characterized in that , the second-level fused multimodal feature output is sent to the preset spectral quaternion convolution Transformer block in the dual-branch quaternion convolution Transformer network for processing, including: A spectral quaternion convolutional tag embedding block based on a spectral quaternion convolutional Transformer block is used to normalize the labeled spectral tags based on layer normalization to obtain standardized spectral tags; Calculate the standardized spectral label according to the preset mathematical expressions of the query, key and value components of the single-head three-dimensional quaternion convolution spectral attention to obtain the query, key and value components of the single-head three-dimensional quaternion convolution spectral attention; The query, key and value components of the single-head three-dimensional quaternion convolution spectral attention are calculated based on the preset mathematical expression of the single-head three-dimensional quaternion convolution spectral attention score to obtain the single-head three-dimensional quaternion convolution spectral attention score; The single-head three-dimensional quaternion convolution spectral attention score is calculated based on the preset mathematical expression of the output of the single-head three-dimensional quaternion convolution spectral attention to obtain the output of the single-head three-dimensional quaternion convolution spectral attention; The output of the single-head three-dimensional quaternion convolution spectral attention is calculated based on the mathematical expression of the output of the multi-head three-dimensional quaternion convolution spectral attention to obtain the output of the multi-head three-dimensional quaternion convolution spectral attention; The output of the multi-head three-dimensional quaternion convolution spectral attention is layer-normalized to obtain the second eigenvector; Processing the second feature vector based on two three-dimensional quaternion convolution layers with preset convolution kernel sizes to obtain a processed second feature vector; The processed second eigenvector is nonlinearly transformed based on the Gaussian error linear unit activation function and the gating mechanism to obtain the output result of the spectral quaternion convolution Transformer block.

5. The multi-source remote sensing image classification method based on a multimodal quaternion representation network according to claim 1 is characterized in that , sending the output of the dual-branch quaternion convolutional Transformer network to the third-level multimodal feature cross-fusion module for processing, including: Performing feature extraction on the output of a dual-branch quaternion convolution Transformer network based on a three-dimensional quaternion convolution of a third-level multimodal feature cross-fusion module, wherein the output of the dual-branch quaternion convolution Transformer network includes an output of a spatial branch and an output of a spectral branch; The correlation between the output of the spatial branch and the output of the spectral branch is calculated based on the softmax function, and the correlation between the output of the spatial branch and the output of the spectral branch is multiplied by the feature of the output result of the spectral branch based on the Hamiltonian multiplication to obtain a weighted response of the output result of the spectral branch to the output result of the spatial branch; The output of the spectral branch is added element by element to the weighted response of the output of the spatial branch and the output of the spatial branch to obtain the output of the third-level multimodal feature cross-fusion module.

6. The multi-source remote sensing image classification method based on a multimodal quaternion representation network according to claim 1 is characterized in that , and the output of the third-level multimodal feature cross-fusion module and the output of the dual-branch quaternion convolution Transformer network are sequentially input into the preset fully connected layer for classification, including: Based on the output of the third-level multimodal feature cross-fusion module and the output of the dual-branch quaternion convolutional Transformer network, cross entropy calculation is performed to obtain three classification losses; The three classification losses are added together to obtain the multi-task loss function. The network is trained using the preset Adam optimizer and the multi-task loss function until convergence to obtain the trained multi-source remote sensing image classification model.

Images