Cross-modal multi-information fusion classification network, system, medium and equipment
By using a cross-modal multi-information fusion classification network, the problems of mode-specific information loss and insufficient feature extraction in the joint classification of hyperspectral images and lidar are solved, and high-precision feature representation and classification are achieved.
Patent Information
- Application Number
- CN202410960833.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-17
- Publication Date
- 2024-11-12
AI Technical Summary
Existing technologies suffer from insufficient classification accuracy due to the loss of modality-specific information and inadequate extraction of modality features in joint classification tasks using hyperspectral images and lidar.
A cross-modal multi-information fusion classification network is adopted, including a two-branch CNN network, a two-branch cross-modal attention fusion module, an adaptive mask modulation module, and a semantically guided feature fusion module. By adaptively adjusting the modality learning rate and exchanging semantic information, the modality feature representation and complementary information mining are enhanced.
It achieves accurate joint classification of hyperspectral images and lidar data, improves classification accuracy and the comprehensiveness of feature representation, and alleviates the learning imbalance problem.
Smart Images

Figure CN118941846A_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the technical field of image processing and relates to a cross-modal multi-information fusion classification network, system, medium and equipment. Background Art
[0002] For the joint classification task of hyperspectral images and lidar, common strategies include feature fusion and information interaction. In existing methods, information interaction is often processed directly on each modal feature, and interaction is achieved by narrowing the similarity of modal features, which may lead to the loss of modality-specific information. In addition, due to the differences in different data information, the dominant modality will inhibit the learning of other modalities, which is also ignored by many methods. Simple linear fusion strategies are also a certain obstacle to the accurate understanding of the content of hyperspectral images because they cannot fully explore the complementary information of different modalities.
[0003] In the existing technical solutions, the hyperspectral image and the lidar image are first input into different convolutional layers respectively, so that the two have the same number of channels, and then a weight-shared convolutional neural network is used to obtain the features of different modalities. Considering that although hyperspectral images and lidar images have different imaging mechanisms, they essentially describe the same scene, so there is a certain similarity between the two. Therefore, based on the idea of contrastive learning, the features of the two modalities are regarded as positive pairs, and the contrast loss is calculated for unsupervised pre-training. Then, after adding the classification head, a small number of labeled samples are used for fine-tuning. Summary of the invention
[0004] The present invention provides a cross-modal multi-information fusion classification network to reduce the loss of modality-specific information in the similarity interaction process of multimodal data and the technical problem of insufficient extraction of different modal features.
[0005] The present invention adopts the following technical solutions:
[0006] The present invention provides a cross-modal multi-information fusion classification network, comprising:
[0007] Dual-branch CNN network, two dual-branch cross-modal attention fusion modules, adaptive mask modulation module, semantic-guided feature fusion module, fully connected layer network;
[0008] The dual-branch CNN network is used to take the patch images corresponding to the HSI and LiDAR modal data as input, extract the features of the two patches respectively, and output the hyperspectral data features. and lidar data features
[0009] Two dual-branch cross-modal attention fusion modules are used to collect global context from hyperspectral data features through a two-layer CMAFT network. Extract and output the first fusion feature and features from LiDAR data And output the second fusion feature
[0010] Adaptive mask modulation module, used to transform the first fusion feature and the second fusion feature As input, the learning rate of the two modal data is adaptively adjusted according to the imbalance rate of the two modal features, and the first modulation feature z corresponding to the HSI data is re-output h The second modulation feature z corresponding to the LiDAR data l ;
[0011] The semantically guided feature fusion module is based on the self-attention mechanism according to the first modulation feature z corresponding to the HSI data. h The second modulation feature z corresponding to the LiDAR data l Perform feature fusion and output multimodal features
[0012] Fully connected layer network, used to integrate the fused multimodal features As input, the final classification result is output.
[0013] Optionally, the structure of the dual-branch CNN network is: a dual-branch CNN network, one branch sequentially stacks a 3D convolution with a kernel size of 5×1×1 and a 2D convolution with a kernel size of 1×1 to extract the hyperspectral data features of the HSI patch data The other branch stacks two 1×12D convolutions in sequence to obtain the lidar data features.
[0014] Optionally, the cross-modal attention fusion module is a two-layer CMAFT network, where one layer is used to integrate the hyperspectral data features As input, another layer of network is used to transform the lidar data features As input, the two features are first flattened and concatenated with all-zero learnable classification labels to generate corresponding sequence features; then the corresponding sequence features are added to the learning position to obtain the input features and Where h represents HSI data, l represents LiDAR data, and each layer of the double-layer CMAFT network contains two components, one of which is used to and Extract global context information, and another component is used to extract global context information based on input features. and Nonlinear transformation is performed, and the double-layer CMAFT network fuses the attention maps of different modes through the Project module, and finally outputs the first fusion feature and the second fusion feature
[0015] Optionally, the adaptive mask modulation module is used to calculate the distance between the classification labels of the original HSI data and the LiDAR data and the corresponding prototypes to obtain the category distribution probability of each modality;
[0016] Secondly, the imbalance rate between the two modalities is measured according to the class distribution probability of each modality;
[0017] Thirdly, the mask probabilities β and γ of HSI and LiDAR features are calculated respectively according to the imbalance rates of the two modalities;
[0018] Finally, the first fusion feature and the second fusion feature Perform modulation with probability β and γ, and output the first modulation feature z h and the second modulation characteristic z l .
[0019] Optionally, the semantic guided feature fusion module uses the SGFF module to first transform the unique feature z corresponding to the HSI branch h The key vector K and the value vector V are derived through two fully connected layers, and then the pseudo prediction probability P from the LiDAR branch is l Input into the fully connected layer to generate the query vector Q, then Q is multiplied by K and input into the softmax function to obtain the aggregation weight Then calculate the aggregation weight S and multiply it by V to get the LiDAR-assisted HSI modal representation; finally, the feature z h Classification mark Residual connection is performed with the LiDAR-assisted HSI modality representation to obtain In the same way
[0020] The pseudo prediction probability of the HSI branch is input into the LiDAR branch to obtain the HSI-assisted LiDAR modal representation.
[0021] Optionally, a fully connected layer network is used to represent the HSI-assisted LiDAR modality and LiDAR-assisted HSI modality representation Connect rows to get multimodal features
[0022] The present invention also provides a cross-modal multi-information fusion classification method, comprising the following steps:
[0023] The corresponding patches are extracted from HSI and LiDAR data and input into the dual-branch CNN network, and batch normalization and leaky rectified linear unit functions are applied sequentially after each convolution layer;
[0024] Two dual-branch cross-modal attention fusion Transformers are used to obtain f h and f l Collect global contextual content in the model while enhancing the interaction between modalities through attention fusion and latent similarity constraints;
[0025] An adaptive mask modulation module is applied to identify the learning rates of the two modalities according to the classification accuracy, and adaptively adjust them to alleviate the unbalanced learning. Semantic-guided feature fusion is then used to fully explore the complementary information between different modalities. By exchanging semantics between HSI and LiDAR, valuable clues from the two branches are mined to achieve comprehensive feature integration.
[0026] The discriminative fusion features are mapped into semantic vectors through the fully connected layer to achieve pixel-level classification.
[0027] The present invention also provides a cross-modal multi-information fusion classification system, comprising:
[0028] The processing module extracts the corresponding patches from the HSI and LiDAR data and inputs them into a two-branch CNN network, applying batch normalization and leaky rectified linear unit functions in sequence after each convolution layer;
[0029] The fusion module uses two dual-branch cross-modal attention fusion Transformers to obtain f h and f l Collect global contextual content in the model while enhancing the interaction between modalities through attention fusion and latent similarity constraints;
[0030] The adaptive module identifies the learning rates of the two modalities based on the classification accuracy and adjusts them adaptively to alleviate the imbalanced learning. It then uses semantic-guided feature fusion to fully explore the complementary information between different modalities. It also mines valuable clues from the two branches through the mutual exchange of semantics between HSI and LiDAR to achieve comprehensive feature integration.
[0031] The classification module maps the discriminative fusion features into semantic vectors through a fully connected layer to achieve pixel-level classification.
[0032] The present invention also provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the function of any cross-modal multi-information fusion classification network in the above-mentioned embodiments is implemented.
[0033] The present invention also provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, it implements the functions of any cross-modal multi-information fusion classification network in the above embodiments.
[0034] The beneficial effects of the present invention are:
[0035] It should be noted that the present invention proposes a cross-modal multi-information fusion classification network as an end-to-end multi-modal fusion network to achieve accurate joint classification of HSI and LiDAR data. By fully extracting the intrinsic features of each modality and the potential correlation between the modalities, high-quality discriminant features can be obtained and used for the final classification.
[0036] Secondly, a dual-branch CMAFT is used in the network to mine global context clues and interact information based on spatial relations. It not only integrates the attention information of the two modalities, but also promotes the interaction of modalities by restricting the similarity of spatial relations. Therefore, effective interaction is achieved while preserving modality specificity and enhancing feature representation.
[0037] Thirdly, the AMM module is also used in the network to dynamically adjust the learning rate of each modality feature to alleviate the problem of learning imbalance in multimodal fusion classification. In this way, the under-optimization of the weaker modality is effectively alleviated, which is conducive to the subsequent feature fusion.
[0038] Finally, the SGFF module is also used in this network to merge the feature representations of different modalities. Unlike simple linear fusion strategies, it uses the semantics of one modality to guide the features of another modality. Therefore, the complementary information of the two modalities can be explored, highly discriminative semantic features can be generated, and the performance of the model can be improved. BRIEF DESCRIPTION OF THE DRAWINGS
[0039] Figure 1 A schematic diagram of the overall structure of a cross-modal multi-information fusion classification network provided by the present invention;
[0040] Figure 2 A schematic diagram of the structure of a cross-modal attention fusion module provided by the present invention (taking the HSI branch as an example);
[0041] Figure 3 A schematic diagram of a cross-modal attention fusion Transfomer structure provided by the present invention;
[0042] Figure 4 A schematic diagram of a semantically guided feature fusion module provided by the present invention;
[0043] Figure 5 A schematic diagram of a computer device provided by an embodiment of the present invention;
[0044] Figure 6 The present invention is a block diagram of an electronic device provided according to an embodiment. DETAILED DESCRIPTION
[0045] The present invention is described in detail below with reference to the accompanying drawings and specific embodiments.
[0046] The present invention provides a cross-modal multi-information fusion classification network, comprising:
[0047] Dual-branch CNN network, two dual-branch cross-modal attention fusion modules, adaptive mask modulation module, semantic-guided feature fusion module, fully connected layer network;
[0048] The dual-branch CNN network is used to take the patch images corresponding to the HSI and LiDAR modal data as input, extract the features of the two patches respectively, and output and lidar data features
[0049] Two dual-branch cross-modal attention fusion modules are used to collect global context from hyperspectral data features through a two-layer CMAFT network. Extract and output the first fusion feature and features from LiDAR data And output the second fusion feature
[0050] Adaptive mask modulation module, used to transform the first fusion feature and the second fusion feature As input, the learning rate of the two modal data is adaptively adjusted according to the imbalance rate of the two modal features, and the first modulation feature z corresponding to the HSI data is re-output h The second modulation feature z corresponding to the LiDAR data l ;
[0051] The semantically guided feature fusion module is based on the self-attention mechanism according to the first modulation feature corresponding to the HSI data. The second modulation feature corresponding to the LiDAR data Perform feature fusion and output multimodal features
[0052] Fully connected layer network, used to integrate the fused multimodal features As input, the final classification result is output.
[0053] In one embodiment, if Figure 1As shown in Figure 2, in order to discover more useful knowledge and improve the feasibility of joint classification of hyperspectral imagery (HSI) and laser radar data (LiDAR), a new network named cross-modal multi-information collaborative fusion network (MICF-Net) is proposed. Its overall framework is as follows: Figure 1 shown.
[0054] First, extract the corresponding patches from HSI and LiDAR data and Input to a simple two-branch CNN network, where s×s represents the block size, c1 and c2 are the original channel numbers of HSI and LiDAR. For the HSI branch, 3D convolutions with a kernel size of 5×1×1 and 2D convolutions with a kernel size of 1×1 are stacked in sequence to extract spectral information from HSI. Differently, since LiDAR is a single-channel image, two 1×12D convolutions are stacked to extract elevation information from LiDAR.
[0055] Here, the number of filters in the two convolutional layers in the HSI branch is 8 and 64, respectively, while the number of filters in the two convolutional layers in the LiDAR branch is 32 and 64, respectively. In addition, in order to speed up the convergence of the model, batch normalization (BN) and leaky rectified linear unit (LeakyReLU) functions are applied sequentially after each convolutional layer. After this step, patches from the two modalities are mapped to hyperspectral data features. and lidar data features Where c=64.
[0056] Secondly, two dual-branch cross-modal attention fusion Transformer (CMAFT) are used to obtain the hyperspectral data features f h and the lidar data feature f l The global context content is collected from the hyperspectral data, and the interaction between modalities is further enhanced through the fusion of attention and the constraint of potential similarity. Extract and output the first fusion feature and features from LiDAR data And output the second fusion feature
[0057] Next, the adaptive mask modulation module (AMM) is applied to transform the first fusion feature and the second fusion feature As input, the learning rates of the two modalities are identified according to the classification accuracy, and they are adaptively adjusted to alleviate the unbalanced learning. This ensures the validity of each branch information and re-outputs the first modulation feature z corresponding to the HSI data h The second modulation feature z corresponding to the LiDAR data l It should be noted that the present invention develops an AMM module to dynamically adjust the learning rate of each modality feature to alleviate the problem of learning imbalance in multimodal fusion classification. It effectively alleviates the under-optimization of the weaker modality and is conducive to subsequent feature fusion.
[0058] Then, semantic-guided feature fusion (SGFF) will be used to fully explore the complementary information between different modalities. Based on the self-attention mechanism, the first modulation feature corresponding to the HSI data is The second modulation feature corresponding to the LiDAR data Perform feature fusion and output multimodal features By exchanging semantics between HSI data and LiDAR data, valuable clues of the two branches can be mined and comprehensive feature integration can be achieved. It should be noted that the present invention develops an SGFF module to merge feature representations of different modalities. Unlike simple linear fusion strategies, it uses the semantics of one modality to guide the features of another modality. Therefore, the complementary information of the two modalities can be explored, highly distinguishable semantic features can be generated, and the performance of the model can be improved.
[0059] Finally, the discriminative multimodal features are transformed into Mapping into semantic vectors to achieve pixel-level classification.
[0060] It should be noted that the present invention proposes a cross-modal multi-information fusion classification network to achieve accurate joint classification of HSI and LiDAR data. By fully extracting the intrinsic features of each modality and the potential correlation between the modalities, high-quality discriminant features can be obtained and used for the final classification.
[0061] Optionally, the structure of the dual-branch CNN network is: a dual-branch CNN network, one branch sequentially stacks a 3D convolution with a kernel size of 5×1×1 and a 2D convolution with a kernel size of 1×1 to extract the hyperspectral data features of the HSI patch data The other branch stacks two 1×12D convolutions in sequence to obtain the lidar data features.
[0062] Optionally, the cross-modal attention fusion module is a two-layer CMAFT network, where one layer is used to integrate the hyperspectral data features As input, another layer of network is used to transform the lidar data features As input, the two features are first flattened and concatenated with all-zero learnable classification labels to generate corresponding sequence features; then the corresponding sequence features are added to the learning position to obtain the input features and Where h represents HSI data and l represents LiDAR data.
[0063] In one embodiment, if Figure 1 As shown in Figure 2, although HSI data and LiDAR data record target information from different angles, they essentially describe the same area. Therefore, they should be similar in the transformation space. Inspired by this observation, a dual-branch cross-modal attention fusion Transformer is proposed to extract global contextual knowledge while mining the similarity of spatial relations between the two modalities.
[0064] Specifically, characteristic hyperspectral data features and features from LiDAR data It is first flattened and concatenated with the all-zero learnable classification markers to generate a sequence feature, which is then added to the learnable position to obtain the input feature and Next, they are sequentially input into D dual-branch CMAFTs to mine global contextual knowledge while enhancing the interaction between the two modalities.
[0065] Optionally, each layer of the two-layer CMAFT network contains two components, one for and Extract global context information, and another component is used to extract global context information based on input features. and Nonlinear transformation is performed, and the double-layer CMAFT network fuses the attention maps of different modes through the Project module, and finally outputs the first fusion feature and the second fusion feature
[0066] In one embodiment, D is set to 2. Since all CMAFTs have the same structure, the CMAFT of the first HSI branch is taken as an example for explanation to facilitate clarity.
[0067] Like the standard ViT encoder, CMAFT is divided into two components. The former is responsible for mining the global context of the input, while the latter focuses on further nonlinear transformation. Unlike the standard ViT encoder, CMAFT expands the original self-attention to cross-modal fusion attention, effectively enhancing the information interaction between modalities.
[0068] Its structure is as Figure 2As shown, the two components calculate the following two formulas in sequence, and the process can be expressed as:
[0069]
[0070] where CMAF(·) represents cross-modal attention fusion (CMAF), LN(·) represents layer normalization (LN) and MLP(·) represents multi-layer perceptron (MLP) function. Represents the self-attention map obtained from the first LiDAR branch.
[0071] To this end, in addition to the self-attention within the modality, it also fuses the attention maps of different modalities through the Project module to effectively utilize the complementary information of the two modalities. Its specific structure in the i-th head space is as follows Figure 3 As shown, the complete CMAF is represented as:
[0072]
[0073] Among them, head i is the output of the ith head, represents the cross-modal attention of the i-th head, and represents the self-attention of the i-th head of the current HSI branch and LiDAR branch, Q i , K i and V i yes The i-th query, key and value of and Represents a learnable parameter matrix. [·] represents concatenation, Softmax(·) and Project(·), and mapping operations. By connecting all head spaces, self-attention Can get A l The acquisition process is similar to A h The Project block consists of two layers of 1×12D convolutions with m and n convolution kernels respectively.
[0074] Besides, BN and LeakyReLU layers are added between them.
[0075] In the present invention, the number of convolution kernels of the two convolution layers of the Project block is set to m=8 and n=1.
[0076] Note that to further enhance the interactions between modes, the Project modules in dual-branch CMAFT share weights.
[0077] In one embodiment, considering that HSI and LiDAR data cover the same location, their attention maps should have similarities. To illustrate this point, the present invention first obtains the cross-modal attention map CA corresponding to the dth CMAFT of the HSI and LiDAR branches by concatenating the cross-modal attention maps collected from each head space. h_d and CA l_d Then, the Kullback-Leibler (KL) divergence is introduced to constrain the correlation between two cross-modal attention maps.
[0078] For D two-branch CMAFTs, the network adopts Kullback-Leibler (KL) divergence to constrain the correlation between two cross-modal attention maps, and its related loss is expressed as:
[0079]
[0080] Among them, CA h_d is the cross-modal attention map of HSI data, CA l_d Cross-modal attention map for LiDAR data.
[0081] Optionally, the adaptive mask modulation module is used to calculate the distance between the classification labels of the original HSI data and the LiDAR data and the corresponding prototypes to obtain the category distribution probability of each modality;
[0082] Secondly, the imbalance rate between the two modalities is measured according to the class distribution probability of each modality;
[0083] Thirdly, the mask probabilities β and γ of HSI and LiDAR features are calculated respectively according to the imbalance rates of the two modalities;
[0084] Finally, the first fusion feature and the second fusion feature Perform modulation with probability β and γ and output the first modulation feature and the second modulation characteristic
[0085] In one embodiment, for the joint classification task of multimodal data, due to the heterogeneity of the modalities, the learning imbalance between the modalities is a challenge. This imbalance occurs for two main reasons:
[0086] First, the convergence rates of different modes are inherently inconsistent.
[0087] Second, the modality with better performance usually produces a smaller loss, which hinders the learning of other modalities.
[0088] To alleviate this problem, we develop an AMM module whose purpose is to adaptively select and suppress the dominant modality, thereby ensuring comprehensive feature learning of all modalities.
[0089] The first step in modulation is to determine the dominant mode. The present invention uses two non-parametric prototype classifiers to generate pseudo predictions for HSI and LiDAR and calculate the modal imbalance rate, specifically:
[0090] First, based on the original HSI data and LiDAR data sample data, the classification labels are selected from the two modal features output by D dual-branch CMAFT. and
[0091] Then, by calculating the distance between the classification mark of each modality and the corresponding prototype, the category distribution probability of each modality is obtained. This process is expressed as:
[0092]
[0093] Where k = 1, 2, ..., K, P h and P l denote the pseudo-prediction probability distribution of HSI and LiDAR, respectively, K denotes the number of categories, d(·,·) denotes the Euclidean distance, and are the prototypes of the kth category of HSI and LiDAR data, respectively.
[0094] Next, the imbalance rate between the two modalities is measured according to the confidence ratio of the accurate predictions corresponding to each modality in each batch, defined as:
[0095]
[0096] Among them, B represents the batch size, y i Represents the true value of the i-th sample.
[0097] When the imbalance rate is obtained, the present invention further uses it to implement a feature mask modulation operation and adjust the learning speed of different modes.
[0098] First, the imbalance rate is converted into the mask probability β of the HSI feature and the mask probabilities β and γ of the LiDAR feature according to the following formula:
[0099]
[0100] Where tanh(·) represents the hyperbolic tangent function and α is the penalty coefficient.
[0101] Then and Modulation with probability β and γ is performed, that is:
[0102]
[0103] in and Denotes a Bernoulli random variable with probabilities β and γ. When the learning rates of the two modalities are unbalanced, the mask probability of the dominant modality will be smaller. This means that more channels of the dominant modality will be suppressed. In this way, the AMM module can effectively align the learning states of different modalities, thereby facilitating the comprehensive extraction of multimodal features.
[0104] Furthermore, please note that the present invention updates the prototypes between each batch in a momentum manner, expressed as:
[0105]
[0106] in, Indicates calculating the average and corresponding features of the HSI data of the k-th sample in the current batch It means calculating the average value of the corresponding features of the LiDAR data of the k-th sample in the current batch, and ε is the momentum parameter, which is fixed to 0.9 in the present invention.
[0107] It should be noted that, in this embodiment, the cross-modal multi-information fusion classification network provided by the present invention adopts a dual-branch CMAFT to mine global context clues and perform information interaction based on spatial relationships. It not only integrates the attention information of the two modalities, but also promotes the interaction of modalities by limiting the similarity of spatial relationships. Therefore, effective interaction is achieved while retaining modality specificity and enhancing feature representation.
[0108] Optionally, the semantic guided feature fusion module uses the SGFF module to first transform the unique feature z corresponding to the HSI branch h The key vector K and the value vector V are derived through two fully connected layers, and then the pseudo prediction probability P from the LiDAR branch is l Input into the fully connected layer to generate the query vector Q, then Q is multiplied by K and input into the softmax function to obtain the aggregation weight Then calculate the aggregation weight S and multiply it by V to get the LiDAR-assisted HSI modal representation; finally, the feature z h Classification mark Residual connection is performed with the LiDAR-assisted HSI modality representation to obtain In the same way, the pseudo-prediction probability of the HSI branch is input into the LiDAR branch to obtain the HSI-assisted LiDAR modality representation
[0109] In one embodiment, after obtaining the feature z corresponding to HSI and LiDAR h and z l After that, the next step for the final classification is to fuse them into multimodal features. In order to enhance the discrimination of multimodal features, the SGFF module is introduced. Unlike simple integration, SGFF encourages each modality to utilize semantic information from other modalities to effectively extract multimodal complementary information, thereby enhancing the fused multimodal features.
[0110] The present invention takes the HSI branch as an example to illustrate the SGFF, and its structure is as follows Figure 4 shown.
[0111] In order to explore as many beneficial classification clues as possible from HSI and LiDAR data, SGFF considers the pseudo-prediction distribution probabilities of the two modalities obtained in the AMM module during feature fusion. Specifically:
[0112] SGFF first converts the unique feature z corresponding to the HSI branch h The key vector K and the value vector V are derived through two fully connected layers; at the same time, SGFF converts the pseudo prediction probability P from the LiDAR branch l Input into the fully connected layer to generate the query vector Q;
[0113] Then, Q is multiplied by K and input into the softmax function to get the aggregate weight
[0114] Afterwards, S is multiplied by V to obtain the LiDAR-assisted HSI modal representation;
[0115] In addition, z h Classification mark Residual connection is performed with it to obtain This further highlights the unique characteristics of HSI data.
[0116] In general, this process can be described as:
[0117]
[0118] K=z h W k , V = z h W v ,
[0119] in, and Represents a learnable parameter matrix. Similarly, by inputting the pseudo-prediction probability of the HSI branch into the LiDAR branch, the HSI-assisted LiDAR modality representation is obtained.
[0120] Optionally, a fully connected layer network is used to represent the HSI-assisted LiDAR modality and LiDAR-assisted HSI modality representation Connect to get multimodal features
[0121] In one embodiment, the HSI-assisted LiDAR modality is represented as and LiDAR-assisted HSI modality representation Concatenate to get multimodal features That is, F = [gh, gl], and then the fused feature F is input into the fully connected layer to obtain the final classification result.
[0122] It should be noted that the present invention uses cross entropy loss as the classification loss, which is defined as:
[0123]
[0124] Among them, p i represents the probability of correct prediction. Considering the designed correlation loss L COR , the overall loss is expressed as:
[0125] L=L CE +λL COR ,
[0126] Among them, λ is the balance coefficient that controls the importance of classification loss and correlation loss.
[0127] It should be noted that compared with the existing multimodal joint classification network, the cross-modal multi-information fusion classification network provided by the present invention designs an information interaction module CMAFT based on attention graph, which not only mines the global contextual content, but also promotes the similarity of the attention associations of the two modalities, retains the modality-specific information, and realizes efficient information interaction.
[0128] Many existing HSI and LiDAR joint classification methods ignore the problem of unbalanced learning rates due to modal heterogeneity. The present invention designs an AMM module, which adaptively modulates its learning rate by calculating the classification capabilities of different modalities, effectively alleviating the sub-optimization problem of the weak modality and facilitating subsequent feature fusion.
[0129] Some existing joint classification models use simple linear fusion strategies, which are difficult to mine the complex complementary relationship between HSI and LiDAR data. To this end, the present invention designs a SGFF module, which effectively mines the complementary information through the exchange of semantic information between modalities and improves the final classification accuracy.
[0130] In another embodiment of the present invention, a cross-modal multi-information fusion classification system is provided, which can be used to implement the above-mentioned cross-modal multi-information fusion classification method. Specifically, the cross-modal multi-information fusion classification system includes a processing module, a fusion module, an adaptive module and a classification module.
[0131] Among them, the processing module extracts the corresponding patches from the HSI and LiDAR data and inputs them into the dual-branch CNN network, and applies batch normalization and leaky rectified linear unit functions in sequence after each convolution layer;
[0132] The fusion module uses two dual-branch cross-modal attention fusion Transformers to obtain f h and f l Collect global contextual content in the model while enhancing the interaction between modalities through attention fusion and latent similarity constraints;
[0133] The adaptive module identifies the learning rates of the two modalities based on the classification accuracy and adjusts them adaptively to alleviate the imbalanced learning. It then uses semantic-guided feature fusion to fully explore the complementary information between different modalities. It also mines valuable clues from the two branches through the mutual exchange of semantics between HSI and LiDAR to achieve comprehensive feature integration.
[0134] The classification module maps the discriminative fusion features into semantic vectors through a fully connected layer to achieve pixel-level classification.
[0135] In another embodiment of the present invention, in another embodiment of the present invention, a terminal device is provided, the terminal device includes a processor and a memory, the memory is used to store a computer program, the computer program includes program instructions, and the processor is used to execute the program instructions stored in the computer storage medium. The processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc., which are the computing core and control core of the terminal, which are suitable for implementing one or more instructions, and are specifically suitable for loading and executing one or more instructions to implement the corresponding method flow or corresponding functions; the processor described in the embodiment of the present invention can be used to implement the operation of a cross-modal multi-information fusion classification network.
[0136] In another embodiment of the present invention, the present invention also provides a storage medium, specifically a computer-readable storage medium (Memory), which is a memory device in a terminal device for storing programs and data. It is understandable that the computer-readable storage medium here can include both the built-in storage medium in the terminal device and the extended storage medium supported by the terminal device, and can be any tangible medium containing or storing a program, which can be used by an instruction execution system, device or device or used in combination with it. The computer-readable storage medium provides a storage space, which stores the operating system of the terminal. In addition, one or more instructions suitable for being loaded and executed by the processor are also stored in the storage space, and these instructions can be one or more computer programs (including program codes). It should be noted that more specific examples (non-exhaustive list) of the computer-readable storage medium here include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.
[0137] Computer readable storage media also include data signals propagated in baseband or as part of a carrier wave, which carry readable program codes. Such propagated data signals can take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. The readable storage medium can also be any readable medium other than a readable storage medium, which can send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, device, or device. The program code contained on the readable storage medium can be transmitted using any appropriate medium, including but not limited to wireless, wired, optical cable, RF, etc., or any suitable combination of the above.
[0138] Program code for performing the operations of the present invention may be written in any combination of one or more programming languages, including object-oriented programming languages such as Java, C++, and the like, as well as conventional procedural programming languages such as "C" or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as a separate software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server. In the case of a remote computing device, the remote computing device may be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computing device (e.g., via the Internet using an Internet service provider).
[0139] The processor may load and execute one or more instructions stored in a computer-readable storage medium to implement operations related to implementing a cross-modal multi-information fusion classification network in the above embodiments.
[0140] See also Figure 5 , the terminal device is a computer device, and the computer device 60 of this embodiment includes: a processor 61, a memory 62, and a computer program 63 stored in the memory 62 and executable on the processor 61. When the computer program 63 is executed by the processor 61, the method for calculating the composition of fluid in the reservoir transformation wellbore in the embodiment is realized. To avoid repetition, it is not described one by one here. Alternatively, when the computer program 63 is executed by the processor 61, the functions of each model / unit in the calculation system for the composition of fluid in the reservoir transformation wellbore in the embodiment are realized. To avoid repetition, it is not described one by one here.
[0141] The computer device 60 may be a computing device such as a desktop computer, a notebook, a PDA, or a cloud server. The computer device 60 may include, but is not limited to, a processor 61 and a memory 62. Those skilled in the art will appreciate that Figure 5 This is only an example of the computer device 60 and does not constitute a limitation of the computer device 60. It may include more or fewer components than shown in the figure, or a combination of certain components, or different components. For example, the computer device may also include input and output devices, network access devices, buses, etc.
[0142] The processor 61 may be a central processing unit (CPU), or other general-purpose processors, central processing units, graphics processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, data processing logic based on quantum computing, discrete hardware components, etc. A general-purpose processor may be a microprocessor or the processor may be any conventional processor, etc.
[0143] The memory 62 may be an internal storage unit of the computer device 60, such as a hard disk or memory of the computer device 60. The memory 62 may also be an external storage device of the computer device 60, such as a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc., equipped on the computer device 60.
[0144] Furthermore, the memory 62 may include both an internal storage unit of the computer device 60 and an external storage device. The memory 62 is used to store computer programs and other programs and data required by the computer device. The memory 62 may also be used to temporarily store data that has been output or is to be output.
[0145] Any reference to a memory, a database or other medium used in the embodiments provided in the present application may include at least one of a non-volatile and a volatile memory. Non-volatile memory may include a read-only memory (ROM), a magnetic tape, a floppy disk, a flash memory, an optical memory, a high-density embedded non-volatile memory, a resistive random access memory (ReRAM), a magnetic random access memory (MRAM), a ferroelectric random access memory (FRAM), a phase change memory (PCM), a graphene memory, etc. Volatile memory may include a random access memory (RAM) or an external cache memory, etc. As an illustration and not limitation, RAM may be in various forms, such as a static random access memory (SRΔM) or a dynamic random access memory (DRAM), etc.
[0146] The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. The non-relational database may include a distributed database based on blockchain, etc., but is not limited thereto. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., but is not limited thereto.
[0147] See also Figure 6The terminal device 600 is an electronic device, and the electronic device is in the form of a general computing device. The components of the electronic device may include but are not limited to: at least one processing unit 610, at least one storage unit 620, a bus 630 connecting different platform components (including the storage unit 620 and the processing unit 610), a display unit 640, etc.
[0148] The storage unit stores program codes, which can be executed by the processing unit 610, so that the processing unit 610 performs the steps according to various exemplary embodiments of the present invention described in the above method section of this specification. For example, the processing unit 610 can perform the following steps: Figure 1 Follow the steps shown in .
[0149] The storage unit 620 may include a readable medium in the form of a volatile storage unit, such as a random access memory unit (RAM) 6201 and / or a cache memory unit 6202 , and may further include a read-only memory unit (ROM) 6203 .
[0150] The storage unit 620 may also include a program / utility 6204 having a set (at least one) of program modules 6205, such program modules 6205 including but not limited to: an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment.
[0151] Bus 630 may represent one or more of several types of bus structures, including a memory unit bus or memory unit controller, a peripheral bus, an accelerated graphics port, a processing unit, or a local bus using any of a variety of bus architectures.
[0152] The electronic device 600 may also communicate with one or more external devices 700 (e.g., keyboards, pointing devices, Bluetooth devices, etc.), may also communicate with one or more devices that enable a user to interact with the electronic device 600, and / or communicate with any device that enables the electronic device 600 to communicate with one or more other computing devices (e.g., routers, modems, etc.). Such communication may be performed via an input / output (I / O) interface 650. Furthermore, the electronic device 600 may also communicate with one or more networks (e.g., a local area network (LAN), a wide area network (WAN), and / or a public network, such as the Internet) via a network adapter 660. The network adapter 660 may communicate with other modules of the electronic device 600 via a bus 630. It should be understood that, although not shown in the figure, other hardware and / or software modules may be used in conjunction with the electronic device 600, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage platforms, etc.
[0153] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.
[0154] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.
[0155] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.
[0156] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process in the computer or other programmable device. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.
[0157] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the above embodiments, ordinary technicians in the relevant field should understand that the specific implementation methods of the present invention can still be modified or replaced by equivalents. Any modification or equivalent replacement that does not depart from the spirit and scope of the present invention should be covered within the scope of protection of the claims of the present invention.
Claims
1. A cross-modal multi-information fusion classification network, characterized in that: include: Dual-branch CNN network, two dual-branch cross-modal attention fusion modules, adaptive mask modulation module, semantic-guided feature fusion module, fully connected layer network; The dual-branch CNN network is used to take the patch images corresponding to the HSI and LiDAR modal data as input, extract the features of the two patches respectively, and output the hyperspectral data features f h and the lidar data feature f l ; The two dual-branch cross-modal attention fusion modules are used to collect global context from the hyperspectral data features f through a double-layer CMAFT network. h Extract and output the first fusion feature and the feature f from the lidar data l And output the second fusion feature An adaptive mask modulation module is used to transform the first fusion feature And the second fusion feature As input, the learning rate of the two modal data is adaptively adjusted according to the imbalance rate of the two modal features, and the first modulation feature z corresponding to the HSI data is re-output h The second modulation feature z corresponding to the LiDAR data l ; The semantically guided feature fusion module is based on the self-attention mechanism according to the first modulation feature corresponding to the HSI data The second modulation feature corresponding to the LiDAR data Perform feature fusion and output multimodal feature F; The fully connected layer network is used to take the fused multimodal feature F as input and output the final classification result.
2. A cross-modal multi-information fusion classification network according to claim 1, characterized in that: The structure of the dual-branch CNN network is as follows: a dual-branch CNN network, in which one branch sequentially stacks a 3D convolution with a kernel size of 5×1×1 and a 2D convolution with a kernel size of 1×1 to extract the hyperspectral data features f of the HSI patch data. h ; The other branch stacks two 1×12D convolutions in sequence to extract the lidar data feature f l .
3. A cross-modal multi-information fusion classification network according to claim 1, characterized in that: The cross-modal attention fusion module is a double-layer CMAFT network, in which one layer of the network is used to integrate the hyperspectral data features f h As input, another layer of network is used to transform the lidar data features f l As input; The two features are first flattened and concatenated with the all-zero learnable categorical labels to generate the corresponding sequence features; Then add the corresponding sequence features and learning positions to get the input features and h represents HSI data, l represents LiDAR data; Each layer of the double-layer CMAFT network contains two components, one of which is used to calculate the input features. and Extracting global context information, another component is used to extract global context information based on the input features. and Nonlinear transformation is performed, and the double-layer CMAFT network fuses the attention maps of different modes through the Project module, and finally outputs the first fusion feature And the second fusion feature 4. A cross-modal multi-information fusion classification network according to claim 1, characterized in that: The adaptive mask modulation module is used to calculate the distance between the classification marks of the original HSI data and the LiDAR data and the corresponding prototypes to obtain the category distribution probability of each mode; Secondly, the imbalance rate of the two modalities is measured according to the class distribution probability of each modality; Thirdly, the mask probabilities β and γ of the HSI and LiDAR features are calculated respectively according to the imbalance rates of the two modes; Finally, the first fusion feature And the second fusion feature Perform modulation with probability β and γ, and output the first modulation feature and the second modulation characteristic 5. The cross-modal multi-information fusion classification network according to claim 1, characterized in that: The semantic guided feature fusion module first uses the SGFF module to transform the unique feature z corresponding to the HSI branch h The key vector K and the value vector V are derived through two fully connected layers, and then the pseudo prediction probability P from the LiDAR branch is l Input to the fully connected layer to generate the query vector Q, then Q is multiplied by K and input into the softmax function to obtain the aggregation weight S, and then the aggregation weight S is calculated and multiplied by V to obtain the LiDAR-assisted HSI modality representation; finally, the feature z h Classification mark Perform residual connection with the LiDAR-assisted HSI modality representation to obtain g h ; In the same way, the pseudo prediction probability of the HSI branch is input into the LiDAR branch to obtain the HSI-assisted LiDAR modal representation g l .
6. A cross-modal multi-information fusion classification network according to claim 1, characterized in that: The fully connected layer network is used to represent the HSI-assisted LiDAR modality g l and LiDAR-assisted HSI modality representation g h The rows are connected to obtain the multimodal feature F.
7. A cross-modal multi-information fusion classification method, characterized in that: The following steps are involved: The corresponding patches are extracted from HSI and LiDAR data and input into the dual-branch CNN network, and batch normalization and leaky rectified linear unit functions are applied sequentially after each convolution layer; Two dual-branch cross-modal attention fusion Transformers are used to obtain f h and f l Collect global contextual content in the model while enhancing the interaction between modalities through attention fusion and latent similarity constraints; The learning rates of the two modalities are identified based on the classification accuracy and adaptively adjusted to alleviate unbalanced learning. Semantic-guided feature fusion is then used to fully explore the complementary information between different modalities. By exchanging semantics between HSI and LiDAR, valuable clues from the two branches are mined to achieve comprehensive feature integration. The discriminative fusion features are mapped into semantic vectors through the fully connected layer to achieve pixel-level classification.
8. A cross-modal multi-information fusion classification system, characterized in that: include: The processing module extracts the corresponding patches from the HSI and LiDAR data and inputs them into a two-branch CNN network, applying batch normalization and leaky rectified linear unit functions in sequence after each convolution layer; The fusion module uses two dual-branch cross-modal attention fusion Transformers to obtain f h and f l Collect global contextual content in the model while enhancing the interaction between modalities through attention fusion and latent similarity constraints; The adaptive module identifies the learning rates of the two modalities based on the classification accuracy and adjusts them adaptively to alleviate the imbalanced learning. It then uses semantic-guided feature fusion to fully explore the complementary information between different modalities. It also mines valuable clues from the two branches through the mutual exchange of semantics between HSI and LiDAR to achieve comprehensive feature integration. The classification module maps the discriminative fusion features into semantic vectors through a fully connected layer to achieve pixel-level classification.
9. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the function of the cross-modal multi-information fusion classification network as described in any one of claims 1 to 6 is realized.
10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the functions of the cross-modal multi-information fusion classification network as described in any one of claims 1 to 6 are realized.
Citation Information
Patent Citations
Multi-source remote sensing image classification method based on two-way attention fusion neural network
CN109993220A
Hyperspectral and laser radar data fusion classification method based on interactive Transform-CNN (Convolutional Neural Network)
CN117994616A
Cited By
Target echo classification method and system based on multi-modal data fusion
CN120524418A
Merculation environment-based road network extraction and topological optimization method based on fusion of visual large model and graph neural network
CN121767864A
A visual large model and graph neural network fusion road network extraction and topology optimization method based on an ascendance environment
CN121767864B
