A transformer multimodal data feature fusion method

Abstract

This invention discloses a Transformer-based multimodal data feature fusion method. The method includes: a first context optimization using a shallow spatial-spectral feature interpretation module, extracting shallow multimodal features from spectral to spatial information using a local spatial mask obtained from LiDAR data as guiding information; a second context feature fusion using a mid-layer adaptive feature fusion module, fusing CLS tags from multiple data sources using an adaptive cross-Transformer to obtain collaborative abstract information; and a third cross-fusion using a high-level multi-stage feature transfer module based on the shallow multimodal features and collaborative abstract information to obtain the final global feature fusion result. This invention is more reliable and flexible, and can be widely applied in the field of computer technology.

Description

Technical Field

[0001] This invention relates to the field of computer technology, and in particular to a Transformer multimodal data feature fusion method. Background Technology

[0002] With the development of sensor technology, multimodal data fusion plays a crucial role in scene understanding and target recognition. Fusion perception has become an important development direction in the discipline and is widely applied in many fields such as road classification and medical image processing. Classical machine learning data fusion methods use handcrafted features for fusion classification, which are often designed based on specific domain knowledge, limiting portability across different datasets. In contrast, deep learning-based techniques can extract discriminative features ranging from shallow textures to deep abstract features without prior knowledge, offering superior portability compared to traditional models. Currently, deep learning-based techniques have made significant progress and have become an important technical means for multimodal data fusion and classification.

[0003] There are two main types of deep learning-based multimodal data fusion: multi-branch feature overlay fusion strategy and context-aware feature cross-fusion strategy. The former uses non-overlapping network branches to independently learn multimodal heterogeneous features. In contrast, the context-aware feature cross-structure learns multimodal features simultaneously by using context-aware cross-connection and information guidance for each modality, thus possessing greater capability in multimodal feature representation. However, existing networks neglect the necessity of context fusion strategies during local feature fusion in the early feature extraction stage, i.e., they lack global context-aware feature transmission. Summary of the Invention

[0004] In view of this, embodiments of the present invention provide a highly flexible and accurate Transformer multimodal data feature fusion method.

[0005] One aspect of this invention provides a Transformer multimodal data feature fusion method, comprising:

[0006] The first context optimization is performed through the shallow spatial spectral feature interpretation module. Based on the local spatial mask obtained from LiDAR data as guiding information, shallow multimodal features are extracted from spectral information to spatial information.

[0007] A second contextual feature fusion is performed through the mid-layer adaptive feature fusion module, and the CLS tags from multiple data sources are fused using an adaptive cross-transformer to obtain collaborative abstract information.

[0008] Based on the shallow multimodal features and the collaborative abstract information, a third cross-fusion is performed through the high-level multi-stage feature transfer module to obtain the final global feature fusion result.

[0009] Optionally, the first context optimization via the shallow spatial spectral feature interpretation module, using a local spatial mask obtained from LiDAR data as guiding information, extracts shallow multimodal features from spectral information to spatial information, including:

[0010] The spectral attention module is used to determine the global correlation of a single image channel dimension;

[0011] Based on the global correlation of a single image channel, a dynamic region-aware strategy is introduced to generate a guiding mask for LiDAR.

[0012] The spatial discrimination capability of image features is enhanced by using the guiding mask as guiding information.

[0013] Optionally, determining the global correlation of a single image channel dimension using the spectral attention module includes:

[0014] For input HSI image features X HSI Applying a d-dimensional attention vector yields spectral enhancement features;

[0015] Based on the spectral enhancement features, the HSI and LiDAR data for each training sample are determined.

[0016] Optionally, the step of generating a LiDAR guiding mask by introducing a dynamic region-aware strategy based on the global correlation of a single image channel dimension includes:

[0017] Features are extracted from LiDAR data using convolutional layers with ReLU activation function;

[0018] A guided mask for learning LiDAR from input features is based on a data dependency mechanism.

[0019] Optionally, the step of enhancing the spatial discrimination capability of image features based on the guiding mask as guiding information includes:

[0020] Calculate the enhancement features of the HSI image;

[0021] The HSI channels are divided into multiple groups, each sharing a Class Token. The channel attribute representation is refined by embedding local spectra along the channel domain, thus completing the LiDAR feature enhancement.

[0022] Optionally, the second context feature fusion is performed through the mid-layer adaptive feature fusion module, using an adaptive cross-transformer to fuse CLS tags from multiple data sources to obtain collaborative abstract information, including:

[0023] The MHT encoder was applied to HSI data, and the MLT encoder was applied to LiDAR data to obtain spectral and spatial features.

[0024] Based on the adaptive CLS label mixer mechanism, the CLS labels of different groups of HSI data are adaptively fused with LiDAR data to obtain collaborative abstract information.

[0025] Optionally, applying the MHT encoder to HSI data and the MLT encoder to LiDAR data to obtain spectral and spatial features includes:

[0026] For the MHT encoder, the image features of each group are discretized into a sequence along the channel by linear projection to obtain the encoder input features;

[0027] For the MLT encoder, after transforming the spatial features into a 2D feature matrix, two linear projections are performed to unify the input format with the MHT encoder.

[0028] The MHT encoder and MLT encoder are used to extract features from the input tensor to obtain spectral features and spatial features, respectively.

[0029] Optionally, the step of adaptively fusing the CLS tags of different groups of HSI data with LiDAR data according to the adaptive CLS tag mixer mechanism to obtain collaborative abstract information includes:

[0030] The learnable CLS tag is appended to the first position of the encoder to calculate the features of the HSI data and the features of the LiDAR data.

[0031] A second feature context fusion is performed by coordinating abstract information in multimodal data through an adaptive CLS label mixer.

[0032] Based on the result of the second feature context fusion, the CLS tag and the transmitted feature tensor are obtained;

[0033] By connecting the features of HSI data from each group with the features of LiDAR data, collaborative abstract information is obtained.

[0034] Optionally, the step of performing a third cross-fusion based on the shallow multimodal features and the collaborative abstract information through a high-level multi-stage feature transfer module to obtain the final global feature fusion result includes:

[0035] Calculate high-level texture features, second-highest level interpretable features, and highest-level semantic features;

[0036] The final fused feature is calculated based on the high-level texture features, the second-highest level interpretable features, and the highest-level semantic features.

[0037] The final fused features are input into the classification module to obtain class labels, which are then converted into one-dimensional vectors after global average pooling and fully connected layers.

[0038] The one-dimensional vector is transformed into a probability distribution using the softmax function to obtain a prediction vector, and classification is performed based on the maximum probability of the prediction vector.

[0039] The consistency between the predicted vector and the corresponding true vector is measured using a marginal loss function, and the iterative process is updated based on the consistency measurement result.

[0040] Another aspect of this invention provides a Transformer multimodal data feature fusion device with a Transformer structure, comprising:

[0041] The first module is used to perform the first context optimization through the shallow spatial spectral feature interpretation module. It uses the local spatial mask obtained from LiDAR data as guiding information to extract shallow multimodal features from spectral information to spatial information.

[0042] The second module is used to perform a second context feature fusion through the mid-layer adaptive feature fusion module. It uses an adaptive cross-transformer to fuse CLS tags from multiple data sources to obtain collaborative abstract information.

[0043] The third module is used to perform a third cross-fusion based on the shallow multimodal features and the collaborative abstract information through the high-level multi-stage feature transfer module to obtain the final global feature fusion result.

[0044] This invention also discloses a computer program product or computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device can read the computer instructions from the computer-readable storage medium and execute the computer instructions, causing the computer device to perform the aforementioned method.

[0045] In this invention, the first context optimization is performed through a shallow spatial-spectral feature interpretation module, using a local spatial mask obtained from LiDAR data as guiding information to extract shallow multimodal features from spectral information to spatial information. A second contextual feature fusion is performed through a mid-layer adaptive feature fusion module, using an adaptive cross-transformer to fuse CLS markers from multiple data sources to obtain collaborative abstract information. Based on the shallow multimodal features and the collaborative abstract information, a third cross-fusion is performed through a high-level multi-stage feature transfer module to obtain the final global feature fusion result. This invention employs a multimodal Transformer fusion framework, which, compared to CNN-based models, is more reliable and flexible in modeling long-term spectral sequence properties and global spatial correlations. This invention introduces dynamic region-aware convolution into the spatial guiding mask generation and uses the acquired mask as an elevation saliency guiding image to guide another branch in exploring spatial context-aware information. Attached Figure Description

[0046] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0047] Figure 1 This is a flowchart of the Transformer-based adaptive mutual learning multimodal data fusion method of the present invention;

[0048] Figure 2 This is a flowchart of the shallow spatial spectral feature interpretation module (SSFI module) of the present invention;

[0049] Figure 3 This is a flowchart of the MHT and MLT encoders of the present invention. Detailed Implementation

[0050] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0051] To address the problems existing in the prior art, one aspect of the present invention provides a Transformer multimodal data feature fusion method, comprising:

[0052] The first context optimization is performed through the shallow spatial spectral feature interpretation module. Based on the local spatial mask obtained from LiDAR data as guiding information, shallow multimodal features are extracted from spectral information to spatial information.

[0053] A second contextual feature fusion is performed through the mid-layer adaptive feature fusion module, and the CLS tags from multiple data sources are fused using an adaptive cross-transformer to obtain collaborative abstract information.

[0054] Based on the shallow multimodal features and the collaborative abstract information, a third cross-fusion is performed through the high-level multi-stage feature transfer module to obtain the final global feature fusion result.

[0055] Optionally, the first context optimization via the shallow spatial spectral feature interpretation module, using a local spatial mask obtained from LiDAR data as guiding information, extracts shallow multimodal features from spectral information to spatial information, including:

[0056] The spectral attention module is used to determine the global correlation of a single image channel dimension;

[0057] Based on the global correlation of a single image channel, a dynamic region-aware strategy is introduced to generate a guiding mask for LiDAR.

[0058] The spatial discrimination capability of image features is enhanced by using the guiding mask as guiding information.

[0059] Optionally, determining the global correlation of a single image channel dimension using the spectral attention module includes:

[0060] For input HSI image features X HSI Applying a d-dimensional attention vector yields spectral enhancement features;

[0061] Based on the spectral enhancement features, the HSI and LiDAR data for each training sample are determined.

[0062] Optionally, the step of generating a LiDAR guiding mask by introducing a dynamic region-aware strategy based on the global correlation of a single image channel dimension includes:

[0063] Features are extracted from LiDAR data using convolutional layers with ReLU activation function;

[0064] A guided mask for learning LiDAR from input features is based on a data dependency mechanism.

[0065] Optionally, the step of enhancing the spatial discrimination capability of image features based on the guiding mask as guiding information includes:

[0066] Calculate the enhancement features of the HSI image;

[0067] The HSI channels are divided into multiple groups, each sharing a Class Token. The channel attribute representation is refined by embedding local spectra along the channel domain, thus completing the LiDAR feature enhancement.

[0068] Optionally, the second context feature fusion is performed through the mid-layer adaptive feature fusion module, using an adaptive cross-transformer to fuse CLS tags from multiple data sources to obtain collaborative abstract information, including:

[0069] The MHT encoder was applied to HSI data, and the MLT encoder was applied to LiDAR data to obtain spectral and spatial features.

[0070] Based on the adaptive CLS label mixer mechanism, the CLS labels of different groups of HSI data are adaptively fused with LiDAR data to obtain collaborative abstract information.

[0071] Optionally, applying the MHT encoder to HSI data and the MLT encoder to LiDAR data to obtain spectral and spatial features includes:

[0072] For the MHT encoder, the image features of each group are discretized into a sequence along the channel by linear projection to obtain the encoder input features;

[0073] For the MLT encoder, after transforming the spatial features into a 2D feature matrix, two linear projections are performed to unify the input format with the MHT encoder.

[0074] The MHT encoder and MLT encoder are used to extract features from the input tensor to obtain spectral features and spatial features, respectively.

[0075] Optionally, the step of adaptively fusing the CLS tags of different groups of HSI data with LiDAR data according to the adaptive CLS tag mixer mechanism to obtain collaborative abstract information includes:

[0076] The learnable CLS tag is appended to the first position of the encoder to calculate the features of the HSI data and the features of the LiDAR data.

[0077] A second feature context fusion is performed by coordinating abstract information in multimodal data through an adaptive CLS label mixer.

[0078] Based on the result of the second feature context fusion, the CLS tag and the transmitted feature tensor are obtained;

[0079] By connecting the features of HSI data from each group with the features of LiDAR data, collaborative abstract information is obtained.

[0080] Optionally, the step of performing a third cross-fusion based on the shallow multimodal features and the collaborative abstract information through a high-level multi-stage feature transfer module to obtain the final global feature fusion result includes:

[0081] Calculate high-level texture features, second-highest level interpretable features, and highest-level semantic features;

[0082] The final fused feature is calculated based on the high-level texture features, the second-highest level interpretable features, and the highest-level semantic features.

[0083] The final fused features are input into the classification module to obtain class labels, which are then converted into one-dimensional vectors after global average pooling and fully connected layers.

[0084] The one-dimensional vector is transformed into a probability distribution using the softmax function to obtain a prediction vector, and classification is performed based on the maximum probability of the prediction vector.

[0085] The consistency between the predicted vector and the corresponding true vector is measured using a marginal loss function, and the iterative process is updated based on the consistency measurement result.

[0086] Another aspect of this invention provides a Transformer multimodal data feature fusion device with a Transformer structure, comprising:

[0087] The first module is used to perform the first context optimization through the shallow spatial spectral feature interpretation module. It uses the local spatial mask obtained from LiDAR data as guiding information to extract shallow multimodal features from spectral information to spatial information.

[0088] The second module is used to perform a second context feature fusion through the mid-layer adaptive feature fusion module. It uses an adaptive cross-transformer to fuse CLS tags from multiple data sources to obtain collaborative abstract information.

[0089] The third module is used to perform a third cross-fusion based on the shallow multimodal features and the collaborative abstract information through the high-level multi-stage feature transfer module to obtain the final global feature fusion result.

[0090] This invention also discloses a computer program product or computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device can read the computer instructions from the computer-readable storage medium and execute the computer instructions, causing the computer device to perform the aforementioned method.

[0091] The specific implementation process of the present invention will now be described in detail with reference to the accompanying drawings:

[0092] To address the aforementioned technical problems in existing technologies, this invention proposes a Transformer multimodal data feature fusion method with a Transformer structure. The method is rationally designed, overcomes the shortcomings of existing technologies, and has good performance.

[0093] like Figure 1 As shown, this invention is a Transformer multimodal data feature fusion method with a Transformer structure, comprising the following three steps:

[0094] The S1 shallow spatial spectral feature interpretation module (SSFI module) performs the first context optimization, using the local spatial mask obtained from LiDAR data as guiding information to extract shallow multimodal features from spectral information to spatial information.

[0095] The S2 mid-layer adaptive feature fusion module (MAFF module) performs a second contextual feature fusion, using an adaptive cross-transformer to fuse CLS tags from multiple data sources to provide collaborative abstraction information.

[0096] The S3 high-level multi-stage feature transfer module (DMFI module) and the deep cross-fusion module perform a third cross-fusion to improve the framework's representation capabilities.

[0097] Furthermore, such as Figure 2 As shown, the present invention further includes the following in step S1:

[0098] S1.1 Spectral Attention: Uses the spectral attention module to measure the global relevance of a single image channel dimension;

[0099] S1.2 Dynamic Region Awareness Guided Mask Generation: A dynamic region awareness strategy is introduced to generate LiDAR guided masks.

[0100] S1.3 Spectral to spatial feature enhancement: Using a guiding mask as guiding information to enhance the spatial discriminative ability of image features.

[0101] It should be noted that step S1.1 also includes...

[0102] S1.1.1 Input features X HSI Applying a d-dimensional attention vector yields the spectral enhancement feature X. HSI’ It can be calculated using the following formula:

[0103]

[0104] Where: Softmax(·) is the activation function, W Q W K and W V It is the encoding weight matrix;

[0105] S1.1.2 Based on the spectral enhancement in step S1.1.1, for each training sample, the HSI data with an image patch size of m×n is X. hsi′ ∈R m×n×d The corresponding LiDAR data can be represented as X. lid′ ∈R m×n×1 .

[0106] Furthermore, the present invention further includes the following in step S1.2:

[0107] S1.2.1 Using convolutional layers with ReLU activation to extract features X from LiDAR data lid′ It can be calculated using the following formula:

[0108] X lid′ =Conv(X lid′ W 1→(64 / G) );

[0109] Among them: W 1→(64 / G) It is a convolution filter, X lid′ ∈R m×n×(64 / G) ;

[0110] S1.2.2 Based on the data dependency mechanism, from the input feature X lid′ The learning guide mask M can be calculated using the following formula:

[0111] M = argmax(conv(X) lid′ W (64 / G)→(64 / G) ));

[0112] Among them: W (64 / G)→(64 / G) It is a convolution filter, and argmax(·) outputs the maximum value of the index.

[0113] Furthermore, the present invention further includes the following in step S1.3:

[0114] S1.3.1 Image feature enhancement can be calculated using the following formula:

[0115]

[0116] in: F represents element-wise multiplication. hsi ∈R m×n×64 This indicates the enhanced features of HSI;

[0117] S1.3.2 refines the channel attribute representation by embedding local spectra along the channel domain, thus reducing the computational burden on subsequent encoder blocks. Specifically, HSI channels can be divided into G groups, each sharing a key and representative CLS marker, as specified in the following formula:

[0118] F hsi ={F hsi,1 ,…,F hsi,g ,…,F hsi,G};

[0119] Wherein: F hsi,g ∈R m×n×(64 / G) It is the g-th group F hsi ;

[0120] S1.3.3 LiDAR feature enhancement can be calculated using the following formula:

[0121]

[0122] in: F represents element-wise multiplication. lid ∈R m×n×64 This indicates the enhancement features of LiDAR.

[0123] Furthermore, the present invention further includes the following in step S2:

[0124] S2.1 as Figure 3 As shown, the H / L encoder, model-H Transformer encoder (MHT), and model-L Transformer encoder (MLT) were applied to HSI and LiDAR data, respectively, to obtain spectral features and spatial features with local elevation information.

[0125] S2.2 Adaptive CLS tag mixer mechanism, where G-group HSI CLS tags are adaptively fused with LiDAR CLS tags respectively.

[0126] Furthermore, the present invention further includes step S2.1.

[0127] S2.1.1 For the MHT encoder, the g-th group of image features F hsi,g The encoder input features are obtained by discretizing the data along the channel using linear projection. It can be calculated using the following formula:

[0128]

[0129] in, E is a linear projection function, E pos It is a set of position embeddings;

[0130] S2.1.2 For the MLT encoder, spatial feature F lid The matrix is ​​transformed into a 2D feature matrix, and then subjected to two linear projections to unify it with the MHT encoder input format. The encoder input features MLT can then be obtained based on the following formula. in :

[0131] MLT in ∈R m×n×(64 / G) ←R (1×1×(64 / G))×(64 / G) ←R (1×1×(64 / G))×m×n ;

[0132] The S2.1.3MHT and MLT encoders respectively process the input tensor and MLT in Feature extraction can be calculated based on the following formula:

[0133] MultiHead(Q,K,V)=Concat(z1,…,z i ,…,z h W O ;

[0134]

[0135]

[0136] Where h is the number of multi-head attention, W O It is a learnable parameter matrix. They are different parameter matrices, d k It is the feature dimension of each attention head.

[0137] Furthermore, the present invention further includes, in step S2.2:

[0138] S2.2.1 The learnable CLS tag is appended to the first position of the encoder, combined with the resulting HSI feature F′. hsi,g and LiDAR features F′ lid It can be calculated based on the following formula:

[0139]

[0140] F′ lid =[cls lid MLT out ];

[0141] Where || represents the join operation, cls hsi,g and cls lid It is the g-th group of HSI and LiDAR CLS markers, MLT out , These are block markers for HSI spectral and LiDAR spatial information, respectively.

[0142] S2.2.2 The adaptive CLS label mixer coordinates the abstract information in multimodal data and performs a second feature context fusion based on the following formula:

[0143] Q hsi =F′hsi,g W q ,K hsi =F′ hsi,g W k V hsi =F′ hsi,g W v ,

[0144] Q lid =F′ lid W q ,K lid =F′ lid W k V lid =F′ lid W v ,

[0145] Q fusion =α1·Q lid +α2·Q hsi ;

[0146] Among them, Q fusion This represents a fusion query, where α1 and α2 represent adaptive query weights with different HSI and LiDAR features;

[0147] S2.2.3 The final CLS marker and the transmitted feature tensor can be obtained based on the following formula:

[0148]

[0149]

[0150] Among them, cls′ hsi,g ,cls′ lid,g These are updated HSI and LiDAR CLS notations, where FC(·) denotes a linear projection function;

[0151] The S2.2.4G group HSI and LiDAR subset features can be joined based on the following formula:

[0152]

[0153] Among them, F″ hsi ∈R m×n×64 ,F″ lid ∈R m×n×64 .

[0154] Furthermore, step S3 of the present invention also includes:

[0155] S3.1 Advanced Texture Features F ATF Secondary-level interpretable features F SIF and the highest level semantic feature F SSF They can be calculated based on the following formulas:

[0156] F ATF =F″ hsi ⊕F″ lid ,

[0157] F SIF =DMFI(F ATF ),

[0158] F SSF =DMFI(F SIF );

[0159] Where ⊕ represents the tensor join operation along the channel axis, and DMFI(·) represents the DMFI operation;

[0160] S3.2 The final fused feature O can be obtained based on the following formula:

[0161] O=g1(F ATF )+g2(F SIF )+g3(F SSF );

[0162] Where g(·) is the dimension matching function;

[0163] S3.3 The feature O is input to the classification module to obtain the class label. After global average pooling and fully connected layer, it is converted into a (1,1,C) vector, where C represents the maximum number of ground value classes.

[0164] S3.4 Based on the vector obtained in step S3.3, use the softmax function to transform it into a probability distribution to obtain the prediction vector. Classification is based on the maximum probability of the predicted vector.

[0165] S3.5 The inference process uses a marginal loss function to measure the true value y and The degree of inconsistency determines whether to update the iteration process; the loss function is as follows:

[0166]

[0167] Where one_hot(·) denotes the hot probabilistic constructor, where scalar y∈{0,1}, n + =0.9 represents the lower bound of the correct label, n - =0.1 indicates the upper bound of the error labels, and μ is the hyperparameter that controls the backpropagation of gradients during the training process.

[0168] In summary, the present invention has the following advantages:

[0169] 1. This invention designs an innovative multimodal Transformer fusion framework, which is more reliable and flexible in modeling long-term spectral sequence properties and global spatial correlations compared with CNN-based models.

[0170] 2. This invention is the first to introduce dynamic region-aware convolution into spatial guided mask generation, and uses the acquired mask as an elevation saliency guide image to guide another branch to explore spatial context-aware information.

[0171] 3. This invention designs a local spectral embedding strategy that follows the Transformer encoder, which highlights the inherent attributes of CLS tags within the same group and expands the differences of CLS tags between different groups, which is beneficial to improving the classification accuracy of complex scenes.

[0172] 4. The present invention further proposes an innovative adaptive CLS label fusion mechanism, in which the CLS labels of HSI and LiDAR are adaptively fused to collaboratively carry the modality of integrated features.

[0173] In some alternative embodiments, the functions / operations mentioned in the block diagrams may not occur in the order shown in the operation diagrams. For example, depending on the functions / operations involved, two consecutively shown blocks may actually be executed substantially simultaneously, or the blocks may sometimes be executed in reverse order. Furthermore, the embodiments presented and described in the flowcharts of this invention are provided by way of example to provide a more comprehensive understanding of the technology. The disclosed methods are not limited to the operations and logic flows presented herein. Alternative embodiments are contemplated in which the order of various operations is altered and sub-operations described as part of a larger operation are executed independently.

[0174] Furthermore, although the invention has been described in the context of functional modules, it should be understood that, unless otherwise stated, one or more of the described functions and / or features may be integrated into a single physical device and / or software module, or one or more functions and / or features may be implemented in a separate physical device or software module. It is also understood that a detailed discussion of the actual implementation of each module is unnecessary for understanding the invention. Rather, given the properties, functions, and internal relationships of the various functional modules in the apparatus disclosed herein, the actual implementation of the module will be understood within the scope of conventional skill of an engineer. Therefore, those skilled in the art can implement the invention as set forth in the claims using ordinary techniques without excessive experimentation. It is also understood that the specific concepts disclosed are merely illustrative and not intended to limit the scope of the invention, which is determined by the full scope of the appended claims and their equivalents.

[0175] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, essentially, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0176] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (such as a computer-based system, a processor-included system, or other system that can fetch and execute instructions from, an instruction execution system, apparatus, or device). For the purposes of this specification, "computer-readable medium" can be any means that can contain, store, communicate, propagate, or transmit programs for use by, or in conjunction with, an instruction execution system, apparatus, or device.

[0177] More specific examples of computer-readable media (a non-exhaustive list) include: electrical connections (electronic devices) having one or more wires, portable computer disk drives (magnetic devices), random access memory (RAM), read-only memory (ROM), erasable and editable read-only memory (EPROM or flash memory), fiber optic devices, and portable optical disc read-only memory (CDROM). Furthermore, computer-readable media can even be paper or other suitable media on which the program can be printed, because the program can be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, interpreting, or otherwise processing as necessary, and then stored in computer memory.

[0178] It should be understood that various parts of the present invention can be implemented in hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented in software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0179] In the description of this specification, references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0180] Although embodiments of the invention have been shown and described, those skilled in the art will understand that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the claims and their equivalents.

[0181] The above is a detailed description of the preferred embodiments of the present invention, but the present invention is not limited to the embodiments described. Those skilled in the art can make various equivalent modifications or substitutions without departing from the spirit of the present invention, and these equivalent modifications or substitutions are all included within the scope defined by the claims of this application.

Claims

1. A method for Transformer multimodal data feature fusion, characterized in that, The method comprises the following steps: First context optimization is performed by a shallow spectral-spatial feature interpretation module, shallow multi-modal features are extracted from spectral information to spatial information according to local spatial masks obtained from LiDAR data as guide information; Second context feature fusion is performed by a middle-layer adaptive feature fusion module, and collaborative abstract information is obtained by using adaptive cross-Transformer to fuse CLS labels of multiple data sources; Third cross-fusion is performed by a high-level multi-stage feature transmission module according to the shallow multi-modal features and the collaborative abstract information, and a final global feature fusion result is obtained; The first context optimization performed by the shallow spectral-spatial feature interpretation module, the shallow multi-modal features are extracted from spectral information to spatial information according to local spatial masks obtained from LiDAR data as guide information, comprising: The global correlation of a single image channel dimension is determined by using a spectral attention module; According to the global correlation of a single image channel dimension, a dynamic region perception strategy is introduced to generate a guide mask of LiDAR; According to the guide mask as guide information, the spatial differentiation ability of image features is enhanced; The second context feature fusion performed by the middle-layer adaptive feature fusion module, the collaborative abstract information is obtained by using adaptive cross-Transformer to fuse CLS labels of multiple data sources, comprising: The MHT encoder is applied to HSI data, and the MLT encoder is applied to LiDAR data to obtain spectral features and spatial features; According to an adaptive CLS label mixer mechanism, the CLS labels of different groups of HSI data are respectively adaptively fused with LiDAR data to obtain collaborative abstract information; The third cross-fusion performed by the high-level multi-stage feature transmission module according to the shallow multi-modal features and the collaborative abstract information, and the final global feature fusion result is obtained, comprising: High-level texture features, sub-high-level interpretable features and highest-level semantic features are calculated; According to the high-level texture features, the sub-high-level interpretable features and the highest-level semantic features, a final fusion feature is calculated; The final fusion feature is input into a classification module to obtain a class label, which is converted into a one-dimensional vector after global average pooling and a fully connected layer; The one-dimensional vector is converted into a probability distribution by using a softmax function to obtain a prediction vector, and classification is performed based on the maximum probability of the prediction vector; The consistency of the prediction vector and a corresponding true value vector is measured by using a margin loss function, and the iteration process is updated according to the measurement result of the consistency.

2. The method of claim 1, wherein, The global correlation of a single image channel dimension is determined by using a spectral attention module, comprising: Applying the d-dimensional attention vector to the input HSI image features X HSI Applying the d-dimensional attention vector to the input HSI image features X According to the spectral enhancement features, HSI data and LiDAR data of each training sample are determined.

3. The method of claim 1, wherein, The guide mask of LiDAR is generated by introducing a dynamic region perception strategy according to the global correlation of a single image channel dimension, comprising: Features of LiDAR data are extracted by using a convolutional layer with a ReLu activation function; The guide mask of LiDAR is learned from the input features based on a data dependency mechanism.

4. The Transformer multi-modal data feature fusion method of claim 1, characterized in that, The spatial discrimination ability of image features is enhanced according to the guide mask as guide information, and the enhancement processing includes: The enhanced features of the HSI image are calculated; The HSI channel is divided into multiple groups, each group shares a Class Token label, and the channel attribute representation is refined by local spectral embedding along the channel domain to complete LiDAR feature enhancement.

5. The method of claim 1, wherein, The MHT encoder is applied to the HSI data, and the MLT encoder is applied to the LiDAR data to obtain spectral features and spatial features, including: For the MHT encoder, the image features of each group are discretized into sequences along the channel by linear projection to obtain encoder input features; For the MLT encoder, the spatial features are transformed into a 2D feature matrix, and then linearly projected twice to unify the input format with the MHT encoder; The MHT encoder and the MLT encoder respectively extract features from the input tensor to obtain spectral features and spatial features.

6. The Transformer multi-modal data feature fusion method of claim 1, wherein, According to the adaptive CLS label mixer mechanism, the CLS labels of different groups of HSI data are respectively adaptively fused with LiDAR data to obtain collaborative abstract information, including: The learnable CLS label is attached to the first position of the encoder, and the features of the HSI data and the features of the LiDAR data are calculated; The abstract information in the multi-modal data is coordinated by the adaptive CLS label mixer to perform a second feature context fusion; According to the result of the second feature context fusion, the CLS label and the transmitted feature tensor are obtained; The features of the HSI data and the features of the LiDAR data of each group are connected to obtain collaborative abstract information.

7. An apparatus for implementing the method of claim 1-6, wherein, Including: The first module is configured to perform first context optimization by a shallow spectral feature interpretation module, extract shallow multi-modal features from spectral information to spatial information according to local spatial masks obtained from LiDAR data as guide information; The second module is configured to perform second context feature fusion by a middle layer adaptive feature fusion module, use adaptive cross Transformer to fuse CLS labels of multiple data sources to obtain collaborative abstract information; The third module is configured to perform third cross fusion by a high layer multi-stage feature transmission module according to the shallow multi-modal features and the collaborative abstract information to obtain a final global feature fusion result.

Images