An unmanned aerial vehicle perspective infrared-visible light cross-modal data fusion method

Abstract

The application relates to the technical field of unmanned aerial vehicle signal processing and computer intelligent image fusion, in particular to an infrared-visible light cross-modal data fusion method under an unmanned aerial vehicle visual angle. The method comprises the following steps: eliminating the visual angle sensitivity and motion interference of unmanned aerial vehicle data through normalization preprocessing; performing preliminary global feature extraction on input visible light images and infrared images through a self-attention mechanism; then performing information interaction and feature fusion on visible light image features and infrared image features through a cross-attention mechanism; and finally constructing an optimization target through contrast learning and isometry consistency, enhancing the complementarity and discriminability of multi-modal features, so as to improve the robustness and effectiveness of infrared-visible light cross-modal image fusion under the unmanned aerial vehicle visual angle.

Description

Technical Field

[0001] This application relates to the field of UAV signal processing and computer intelligent image fusion technology, and in particular to a method for infrared-visible cross-modal data fusion from the perspective of a UAV. Background Technology

[0002] Unmanned aerial vehicles (UAVs), with their flexibility and maneuverability, have played a vital role in fields such as forest fire prevention, urban security, remote sensing monitoring, and situational awareness. In practical UAV applications, infrared and visible light images are the most commonly used dual-modal data. Infrared images can capture target thermal radiation information and are unaffected by day / night cycles, fog, snow, or other lighting conditions, but they suffer from low spatial resolution and a lack of texture detail. Visible light images, on the other hand, provide high-resolution texture features, but are easily affected by changes in lighting, shooting distance, and weather, resulting in suboptimal imaging performance in unfavorable environments. Therefore, achieving effective complementary fusion of infrared and visible light modalities is crucial for improving target situational awareness performance from the UAV's perspective. Traditional infrared-visible light image fusion methods mainly rely on manually designed loss functions or generative models, which suffer from two major problems: first, the lack of ground truth values ​​for the fused images makes supervised training impossible, causing the fusion results to be biased towards a single modality; second, the failure to distinguish the modal specificity of infrared and visible light results in weak discriminative power of the fused features, making effective modal fusion difficult. In recent years, deep learning-based methods have achieved certain fusion effects by aligning the fused image with images from two different modalities. However, these methods cannot solve the problems of flight motion blur and dynamic changes in viewpoint caused by drone flight, and they do not fully utilize the complementarity between infrared thermal radiation information and visible light texture features. Summary of the Invention

[0003] Therefore, it is necessary to address the aforementioned technical problems by providing a method for infrared-visible cross-modal image fusion from the perspective of a UAV, which can improve the robustness and effectiveness of infrared-visible cross-modal image fusion.

[0004] A method for infrared-visible cross-modal data fusion from the perspective of an unmanned aerial vehicle (UAV), the method comprising:

[0005] Infrared and visible light images acquired by UAVs are preprocessed by normalization to eliminate the viewpoint sensitivity and motion interference of UAV data, resulting in normalized infrared and visible light images. A data fusion model is then constructed, which includes a feature extraction model based on self-attention mechanism, a feature interaction module based on cross-attention mechanism, a feature enhancement module based on contrastive learning, and a feature optimization module based on equivariant consistency.

[0006] Global feature extraction is performed on the normalized infrared image and the normalized visible light image based on the feature extraction model based on the self-attention mechanism to obtain infrared features and visible light features;

[0007] Infrared and visible light features are input into a feature interaction module based on a cross-attention mechanism. Information interaction and fusion of the two features are achieved through dual-path cross-attention to obtain fused features.

[0008] In the feature enhancement module based on contrastive learning, modality contrastive learning loss is used to enhance the fused features.

[0009] In the feature optimization module based on equivariant consistency, the fusion features are optimized using equivariant consistency loss to ensure that the enhanced fusion features are consistent with the distribution of infrared and visible light features. The optimized fusion features are then used to complete the infrared-visible cross-modal data fusion from the perspective of the UAV.

[0010] The aforementioned method for cross-modal data fusion of infrared and visible light from a UAV perspective first eliminates the viewpoint sensitivity and motion interference of UAV data through normalization preprocessing, laying a stable data foundation for subsequent feature extraction and fusion. This addresses the shortcomings of existing deep learning methods, such as ghosting and target offset in the fusion results due to unprocessed dynamic interference, directly improving the robustness of the fusion process. Secondly, a feature extraction model based on a self-attention mechanism captures the local correlation and global correlation of infrared and visible light images through multiple multi-head self-attention layers. This model can extract the core thermal radiation features of infrared images while preserving the high-resolution texture features of visible light images, avoiding the loss of single-modal features. Simultaneously, layer normalization and nonlinear transformation of the feedforward layer enhance effective feature selection, ensuring that key information from both modalities can be accurately represented, providing high-quality feature input for subsequent fusion and guaranteeing the effectiveness of the fusion from the feature source. The feature interaction module based on the cross-attention mechanism addresses the pain point of existing methods not fully utilizing the complementarity of infrared and visible light through dual-path cross-attention, namely infrared-visible light and visible light-infrared. The dual-path design, each primarily driven by a single modality, deeply mines the correlation features between the two modalities, achieving precise interaction between infrared thermal radiation information and visible light texture information. This avoids the problem of traditional methods biasing the fusion results towards a single modality, giving the fused features the dual advantages of resistance to environmental interference and clear details, further enhancing the practical value of the fused features. Finally, the feature enhancement module based on contrastive learning and the feature optimization module based on equivariant consistency construct optimization objectives through dual loss functions, forming a closed-loop optimization of the fused features: contrastive learning, through modal contrastive loss, brings features from the same modality closer and pushes away features from dissimilar negative samples, solving the problem of weak discriminability of the fused features and making them more distinguishable in the feature space; equivariant consistency, through bidirectional KL divergence, ensures the consistency of the distribution of the fused features with infrared and visible light features, preventing the fused features from deviating from the core information of the original modality and ensuring that the complementarity of the two modalities is fully utilized. The combination of these two approaches not only improves the quality of the fused features but also enhances the model's adaptability to complex drone scenarios, ultimately achieving a dual improvement in fusion robustness and effectiveness, and providing reliable technical support for practical engineering applications of drones in forest fire prevention, urban security, and other fields. Attached Figure Description

[0011] Figure 1 This is a flowchart illustrating an infrared-visible cross-modal data fusion method from the perspective of an unmanned aerial vehicle (UAV) in one embodiment.

[0012] Figure 2 This is a schematic diagram of a feature extraction model based on a self-attention mechanism in one embodiment;

[0013] Figure 3 This is a structural diagram of a feature interaction module based on a cross-attention mechanism in one embodiment. Detailed Implementation

[0014] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0015] In one embodiment, such as Figure 1 As shown, an infrared-visible cross-modal data fusion method from the perspective of an unmanned aerial vehicle (UAV) is provided, including the following steps:

[0016] Step 102: Normalize the infrared and visible light images collected by the UAV to eliminate the viewpoint sensitivity and motion interference of the UAV data, and obtain normalized infrared and visible light images; construct a data fusion model; the data fusion model includes a feature extraction model based on self-attention mechanism, a feature interaction module based on cross-attention mechanism, a feature enhancement module based on contrastive learning, and a feature optimization module based on equivariant consistency.

[0017] This application mainly comprises four modules: a feature extraction model based on a self-attention mechanism, a feature interaction module based on a cross-attention mechanism, a feature enhancement module based on contrastive learning, and a feature optimization module based on equivariant consistency. The overall research approach is as follows: first, normalization preprocessing is used to eliminate the viewpoint sensitivity and motion interference of UAV data; then, a self-attention mechanism is used to perform preliminary global feature extraction on the input visible light and infrared images; next, a cross-attention mechanism is used to perform information interaction and feature fusion between the visible light and infrared image features; finally, contrastive learning and equivariant consistency are used to construct an optimization objective, enhancing the complementarity and discriminability of multimodal features.

[0018] Step 104: Perform global feature extraction on the normalized infrared image and the normalized visible light image according to the feature extraction model based on the self-attention mechanism to obtain infrared features and visible light features.

[0019] The self-attention mechanism possesses excellent long-range dependency representation capabilities, effectively capturing local correlations and global associations between visible and infrared image regions, thus solving the problem of scattered target features from the perspective of UAVs. This module constructs a feature extraction model with a multi-layer multi-head self-attention layer as its core for infrared (I) and visible (V) dual-modal images. Each layer consists of three parts: layer normalization, multi-head attention layer, and feedforward layer, as shown in Figure 2.

[0020] To mitigate the issues of viewpoint shift and amplitude differences in UAV images, this application first normalizes the mean and variance of the infrared and visible light images to eliminate amplitude sensitivity, which can be expressed as:

[0021] ;

[0022] ;

[0023] in, and These represent the mean and standard deviation of the infrared images, respectively. and These represent the mean and standard deviation of the visible light image, respectively. and This represents the normalized infrared and visible light image data. Indicates pixel position. The normalized image is then processed according to... The pixel block is divided into sub-regions, and each sub-region is flattened as follows: 3D feature vectors are used to obtain infrared feature sequences. With visible light feature sequences (N represents the number of sub-regions), which serves as the input to the feature extraction layer.

[0024] Layer normalization is used to alleviate the gradient vanishing problem in deep networks. It standardizes the input feature sequence, as shown in the following formula:

[0025] ;

[0026] ;

[0027] in, and (Hereafter referred to as) () indicates the output of the layer normalization. Presentation layer normalization operation.

[0028] Multi-head self-attention layers capture the local and global relationships of input data through a multi-head attention mechanism, enhancing the representation ability of local details and global structure of the target. Specifically, the multi-head self-attention layer first normalizes the output of the layer... Projected onto the query matrix respectively Key matrix Value matrix ,Right now:

[0029] ;

[0030] ;

[0031] ;

[0032] in, , , , , The output of the normalized layer is represented.

[0033] The resulting three matrices A self-attention module using cosine similarity as a metric, through a scaling factor The attention weights between subsequences are obtained through softmax operations, and then the value matrices are weighted and fused to obtain the final output of the self-attention layer.

[0034] ;

[0035] The multi-head self-attention layer will query the matrix. Key matrix Sum matrix The projections are respectively projected onto multiple feature subspaces, and attention is calculated for each subspace before concatenation to obtain the final multi-head attention layer output. This is used to enable the model to learn diverse projection methods that fit the characteristics of each modality, which can be specifically expressed as:

[0036] ;

[0037] in, This is the final output of the multi-head self-attention layer. This indicates a splicing operation. Indicates the first The output of each attention head, Indicates the number of heads of attention. This represents the recovery matrix, ensuring that the input and output remain consistent in the feature dimension. The feedforward layer of the feature extraction module is implemented using a Multi Layer Perceptron (MLP) and a normalization layer to process the output of the multi-head self-attention layer. Nonlinear transformations are performed to enhance the selection of effective features. The specific process of the feedforward layer can be defined as follows:

[0038] ;

[0039] in, This represents a feedforward layer network. This represents the forward operation of a multilayer perceptron with two fully connected layers. The final output of the overall feature extraction layer can be represented as:

[0040] ;

[0041] ;

[0042] in, and These represent the extracted infrared and visible light features, respectively. This indicates the proposed feature extraction layer based on the self-attention mechanism. This represents a multi-head attention mechanism extraction network.

[0043] This model, centered on a multi-layered multi-head self-attention layer, possesses excellent long-range dependency representation capabilities, accurately capturing the local regional correlations and global structural relationships between infrared and visible light images. On one hand, addressing the modal specificity of thermal radiation features in infrared images and texture features in visible light images, it alleviates the gradient vanishing problem in deep networks through layer normalization, achieves multi-dimensional feature extraction using multi-head self-attention layers, and further enhances effective feature selection through nonlinear transformation of feedforward layers, ensuring that the core information of both modalities (target contours in infrared and detailed textures in visible light) is fully extracted. On the other hand, the residual connection design improves the model's training stability, effectively solving the problem of scattered target features from the UAV's perspective. This provides high-quality, highly discriminative single-modal features for subsequent feature interaction and fusion, fundamentally avoiding poor fusion results caused by insufficient feature extraction.

[0044] Step 106: Input the infrared features and visible light features into the feature interaction module based on the cross-attention mechanism. Through dual-path cross-attention, the information interaction and fusion of the two features are realized to obtain the fused features.

[0045] To achieve information interaction and feature fusion between visible and infrared images and enhance modal feature complementarity, this application proposes a dual-path cross-attention mechanism ("infrared-visible light" and "visible light-infrared") to mine intermodal correlation features, with the structure as follows: Figure 3 As shown, the infrared-visible cross-attention branch is dominated by the infrared mode, emphasizing the thermal radiation characteristics of infrared modal data. The visible-infrared cross-attention branch is dominated by the visible light mode, emphasizing the thermal radiation characteristics of infrared modal data.

[0046] This application innovatively designs a dual-path cross-attention branch, namely "infrared-visible light" and "visible light-infrared," breaking through the limitation of insufficient modal information interaction in traditional fusion methods. The infrared-dominant branch emphasizes infrared thermal radiation characteristics, while the visible light-dominant branch highlights the advantages of visible light texture. The dual-path design uses one modality as the core to mine the correlation features with the other modality, achieving deep interaction and accurate fusion of the two modalities. By concatenating the dual-path cross-attention outputs and adjusting the dimensions through convolution, the final fused feature simultaneously retains the advantages of infrared images in resisting illumination / weather interference and the high-resolution detail advantages of visible light images. This effectively solves the problem of traditional methods tending to bias towards a single modality, significantly enhancing modal complementarity and laying a high-quality foundation for subsequent feature optimization.

[0047] Step 108: In the feature enhancement module based on contrastive learning, modal contrastive learning loss is used to enhance the fused features.

[0048] This module constructs positive and negative sample pairs and proposes a Model Contrastive Learning (MCL) loss to enhance the discriminative power of infrared and visible light features. In each batch of training data, a specific pair of samples ( , ) Fusion features obtained from the input model As the anchor point, its positive sample is the corresponding and The other samples in this training batch are considered negative samples, and the modality contrastive learning loss function... The formula is as follows:

[0049] ;

[0050] in, This indicates that, apart from the anchor point, this batch contains... The set of all other samples, express The samples in This represents the temperature hyperparameter. The modal contrastive learning loss function narrows down the features of two corresponding modes in the feature space. and This method pushes away all other samples in the batch, enhancing the discriminability of infrared and visible light features, allowing features of homologous modal data to be tightly aggregated in space, and significantly separating features of heterologous negative samples. This loss improves the discriminative power and robustness of feature representation, highlighting the fusion effect of homologous modal data under unsupervised conditions, and alleviating the performance bottleneck caused by cross-modal feature differences.

[0051] This module constructs a contrastive learning system of anchor point-positive sample-negative sample and designs a modality contrastive learning (MCL) loss function to achieve the aggregation of homologous modal features and the separation of heterologous modal features in the feature space. Specifically, it uses the fused feature as the anchor point, treats the corresponding bimodal branch features as positive samples, and other sample features in the same batch as negative samples. The loss function optimizes and narrows the feature distance between the anchor point and the positive samples, while widening the feature distance between the anchor point and the negative samples. This process significantly improves the discriminability of infrared features, visible light features, and fused features, allowing features of homologous modal data to be more tightly aggregated in the feature space and heterologous negative sample features to be more significantly separated. It effectively solves the problem of weak discriminability of fused features in traditional methods, improves the distinguishability and robustness of feature representation, and further highlights the value of bimodal data fusion under unsupervised conditions.

[0052] Step 110: In the feature optimization module based on equivariant consistency, the fusion features are optimized using equivariant consistency loss to ensure that the enhanced fusion features are consistent with the distribution of infrared and visible light features, thereby obtaining the optimized fusion features and completing the infrared-visible cross-modal data fusion from the perspective of the UAV.

[0053] This module achieves distribution consistency between fused features and infrared and visible light features by constructing an Equal Consistency Loss (ECL). Specifically, the Equal Consistency Loss... Achieved using bidirectional Kullback-Leibler divergence, this measure assesses the distributional differences between the fused features and infrared features, and between the fused features and visible light features, aiming to make the distributions of the three as similar as possible. Isovariant consistency loss. The calculation formula can be expressed as:

[0054] ;

[0055] in, For batch sample sets, Indicates batch size, This represents the set of all fused features obtained in this batch. and These represent the sets of all infrared and visible light features obtained in this batch, respectively. This is the formula for calculating the KL divergence, used to describe two probability distributions. and The smaller the KL divergence value, the more likely the probability distribution will be divided into two distributions. P and Q The more similar they are.

[0056] This module constructs an Equal Variation Consistency (ECL) loss function using bidirectional KL divergence. Its core function is to ensure the consistency of the distribution of fused features with infrared and visible light features. By measuring the difference in probability distribution between the fused features and the two single-modal features, it optimizes the loss function to make the distributions of the three as similar as possible, avoiding the problem of fused features deviating from the core information of the original single-modal features. This operation ensures that the fused features inherit the target recognition advantages of infrared images and the detail restoration advantages of visible light images, while preventing feature distortion caused by the fusion process. It further enhances the complementarity of multimodal features, ensuring that the fused results can accurately identify targets and clearly present details in complex UAV scenarios, such as forest fire prevention and urban security. Ultimately, it improves the practicality and reliability of the fused results, providing technical support for practical engineering applications.

[0057] The aforementioned method for cross-modal data fusion of infrared and visible light from a UAV perspective first eliminates the viewpoint sensitivity and motion interference of UAV data through normalization preprocessing, laying a stable data foundation for subsequent feature extraction and fusion. This addresses the shortcomings of existing deep learning methods, such as ghosting and target offset in the fusion results due to unprocessed dynamic interference, directly improving the robustness of the fusion process. Secondly, a feature extraction model based on a self-attention mechanism captures the local correlation and global correlation of infrared and visible light images through multiple multi-head self-attention layers. This model can extract the core thermal radiation features of infrared images while preserving the high-resolution texture features of visible light images, avoiding the loss of single-modal features. Simultaneously, layer normalization and nonlinear transformation of the feedforward layer enhance effective feature selection, ensuring that key information from both modalities can be accurately represented, providing high-quality feature input for subsequent fusion and guaranteeing the effectiveness of the fusion from the feature source. The feature interaction module based on the cross-attention mechanism addresses the pain point of existing methods not fully utilizing the complementarity of infrared and visible light through dual-path cross-attention, namely infrared-visible light and visible light-infrared. The dual-path design, each primarily driven by a single modality, deeply mines the correlation features between the two modalities, achieving precise interaction between infrared thermal radiation information and visible light texture information. This avoids the problem of traditional methods biasing the fusion results towards a single modality, giving the fused features the dual advantages of resistance to environmental interference and clear details, further enhancing the practical value of the fused features. Finally, the feature enhancement module based on contrastive learning and the feature optimization module based on equivariant consistency construct optimization objectives through dual loss functions, forming a closed-loop optimization of the fused features: contrastive learning, through modal contrastive loss, brings features from the same modality closer and pushes away features from dissimilar negative samples, solving the problem of weak discriminability of the fused features and making them more distinguishable in the feature space; equivariant consistency, through bidirectional KL divergence, ensures the consistency of the distribution of the fused features with infrared and visible light features, preventing the fused features from deviating from the core information of the original modality and ensuring that the complementarity of the two modalities is fully utilized. The combination of these two approaches not only improves the quality of the fused features but also enhances the model's adaptability to complex drone scenarios, ultimately achieving a dual improvement in fusion robustness and effectiveness, and providing reliable technical support for practical engineering applications of drones in forest fire prevention, urban security, and other fields.

[0058] In one embodiment, the infrared and visible light images acquired by the UAV are subjected to normalization preprocessing to eliminate the viewpoint sensitivity and motion interference of the UAV data, resulting in normalized infrared and visible light images, including:

[0059] The infrared and visible light images acquired by the UAV were preprocessed using normalization to obtain the normalized infrared and visible light images, respectively:

[0060] ;

[0061] ;

[0062] in, and These represent the mean and standard deviation of the infrared images, respectively. and These represent the mean and standard deviation of the visible light image, respectively. and This represents the normalized infrared image and the normalized visible light image. Indicates pixel position, Represents an infrared image. Represents a visible light image.

[0063] In one embodiment, the feature extraction model based on the self-attention mechanism is centered on a multi-layer multi-head self-attention layer. Each layer includes a layer normalization unit, a multi-head self-attention unit, and a feedforward layer unit. Global feature extraction is performed on the normalized infrared image and the normalized visible light image according to the feature extraction model based on the self-attention mechanism to obtain infrared features and visible light features, including:

[0064] The normalized image is divided into... The pixel block is divided into sub-regions, and each sub-region is flattened as follows: 3D feature vectors are used to obtain infrared feature sequences. With visible light feature sequences N represents the number of subregions;

[0065] The infrared and visible light feature sequences are standardized using a layer normalization unit to obtain the layer normalized output.

[0066] In the multi-head self-attention unit, the layer-normalized output is projected onto the query matrix, key matrix, and value matrix, respectively.

[0067] The self-attention module, which measures the query matrix, key matrix, and value matrix using cosine similarity, obtains attention weights between subsequences through scaling factors and softmax operations. Then, it performs weighted fusion on the value matrix to obtain the final output of the self-attention layer.

[0068] A multi-head self-attention layer is used to project the query matrix, key matrix, and value matrix to multiple feature subspaces respectively, and then the attention is calculated separately and concatenated to obtain the output of the multi-head attention layer.

[0069] The output of the multi-head attention layer is input into the feedforward layer to obtain infrared and visible light features.

[0070] In one embodiment, the multi-head self-attention unit projects the layer-normalized output onto the query matrix, key matrix, and value matrix, respectively, including:

[0071] The layer-normalized outputs of the multi-head self-attention unit are projected onto the query matrix respectively. Key matrix Sum matrix for:

[0072] ;

[0073] ;

[0074] ;

[0075] in, , , , , The output of the normalized layer is represented.

[0076] In one embodiment, the self-attention module, which measures the query matrix, key matrix, and value matrix using cosine similarity, obtains attention weights between subsequences through scaling factors and softmax operations. Then, the value matrices are weighted and fused to obtain the final output of the self-attention layer, including:

[0077] query matrix Key matrix Sum matrix A self-attention module using cosine similarity as a metric, through a scaling factor The attention weights between subsequences are obtained through softmax operations, and then the value matrices are weighted and fused to obtain the final output of the self-attention layer.

[0078] .

[0079] In one embodiment, a multi-head self-attention layer is used to project the query matrix, key matrix, and value matrix onto multiple feature subspaces, respectively. Attention is then calculated for each subspace and concatenated to obtain the multi-head attention layer output, including:

[0080] The query matrix, key matrix, and value matrix are projected onto multiple feature subspaces using a multi-head self-attention layer. Attention is then calculated for each subspace and concatenated to obtain the output of the multi-head attention layer.

[0081] ;

[0082] in, This is the final output of the multi-head self-attention layer. This indicates a splicing operation. Indicates the first The output of each attention head, Indicates the number of heads of attention. This represents the recovery matrix.

[0083] In one embodiment, the output of the final multi-head attention layer is input into the feedforward layer to obtain infrared and visible light features, including:

[0084] The output of the final multi-head attention layer is input into the feedforward layer to obtain the infrared and visible light features, respectively:

[0085] ;

[0086] ;

[0087] in, This represents a feedforward layer network. and These represent the extracted infrared and visible light features, respectively. This indicates the proposed feature extraction layer based on the self-attention mechanism. This indicates a multi-head attention mechanism extraction network. and This represents the normalized infrared and visible light images. Presentation layer normalization operation.

[0088] In one embodiment, the dual-path cross-attention includes an infrared-visible cross-attention branch dominated by the infrared mode and a visible-infrared cross-attention branch dominated by the visible mode. Infrared and visible features are input into a feature interaction module based on the cross-attention mechanism. Through dual-path cross-attention, information interaction and fusion of the two features are achieved to obtain fused features, including:

[0089] In the infrared-visible cross-attention branch, infrared features are used as the basis. Visible light characteristics are bonds and Features are obtained by computing across attention mechanisms. , is represented as:

[0090] ;

[0091] In the visible-infrared cross-attention branch, infrared features are used as the basis. Visible light characteristics are bonds and Features are obtained by computing across attention mechanisms. , is represented as:

[0092] ;

[0093] Features and After concatenation, the feature dimensions are adjusted to obtain the fused features. , is represented as:

[0094] ;

[0095] in, This represents the convolution operation. These are the convolution weights.

[0096] In one embodiment, the modality contrastive learning loss is:

[0097] ;

[0098] in, Indicates fusion features, and This indicates that positive samples are features corresponding to the two modalities. This indicates that, apart from the anchor point, this batch contains... The set of all other samples, express The samples in This indicates the temperature hyperparameter.

[0099] In one embodiment, the equivariant consistency loss is:

[0100] ;

[0101] in, For batch sample sets, Indicates batch size, This represents the set of all fused features obtained in a batch. and These represent the sets of all infrared and visible light features obtained in each batch. This is the formula for calculating the KL divergence.

[0102] In a specific embodiment, the training process of this application can be divided into three steps: ① Data preprocessing and feature extraction: First, the infrared images collected by the UAV are normalized to eliminate the influence of viewpoint shift and amplitude difference, and then input into the feature extraction model based on self-attention mechanism and the dual-path cross-attention module to obtain infrared, visible light and fused features; ② Loss function calculation: Modality contrast learning loss and equivariance consistency loss are calculated for the obtained features; ③ Iterative update of model parameters: The backpropagation algorithm is used to update the network parameters in the feature extraction module and feature interaction module, and iterative training is carried out until the total loss converges to complete the training of the fusion model.

[0103] The testing process of this application can be divided into two steps. First, the parameters of the trained fusion model are fixed. Then, the test set is preprocessed, features are extracted and interacted with to generate fused images. The performance index is calculated by the evaluation module to verify the fusion robustness of the model under dynamic perspectives and complex weather conditions.

[0104] The beneficial effects of this application are as follows:

[0105] First, the fusion technology based on equivariant consistency and contrastive learning provided by this invention can realize deep interaction of cross-modal features from the perspective of UAVs, improve modal complementarity, and enhance the quality of fused features and model robustness.

[0106] Second, the fusion technology based on equivariant consistency and contrastive learning provided in this application can extract good feature representations of infrared and visible light data and fuse them to obtain effective fused features;

[0107] Third, the fusion technology based on equivariant consistency and contrastive learning provided in this application can be adapted to the actual application scenarios of UAVs, enhance its practical value, and provide reliable technical support for practical engineering applications. Table 1 shows a comparison of image fusion index results.

[0108] Table 1

[0109]

[0110] It should be understood that, although Figure 1 The steps in the flowchart are shown sequentially as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order in which these steps are executed, and they can be performed in other orders. Figure 1 At least some of the steps in the process may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least some of the sub-steps or stages of other steps.

[0111] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0112] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these modifications and improvements all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A method for infrared-visible cross-modal data fusion from the perspective of an unmanned aerial vehicle (UAV), characterized in that, The method includes: Infrared and visible light images acquired by the UAV are preprocessed by normalization to eliminate the viewpoint sensitivity and motion interference of the UAV data, resulting in normalized infrared and visible light images; a data fusion model is constructed; the data fusion model includes a feature extraction model based on self-attention mechanism, a feature interaction module based on cross-attention mechanism, a feature enhancement module based on contrastive learning, and a feature optimization module based on equivariant consistency. Global feature extraction is performed on the normalized infrared image and the normalized visible light image according to the feature extraction model based on the self-attention mechanism to obtain infrared features and visible light features; The infrared and visible light features are input into a feature interaction module based on a cross-attention mechanism. Information interaction and fusion of the two features are achieved through dual-path cross-attention to obtain fused features. The feature enhancement module based on contrastive learning uses modality contrastive learning loss to enhance the fused features. The feature optimization module based on equivariant consistency optimizes the fused features using equivariant consistency loss, so that the enhanced fused features are consistent with the distribution of infrared and visible light features, and the optimized fused features are obtained to complete the infrared-visible cross-modal data fusion from the perspective of the UAV. The equivalent consistency loss is: in, For batch sample sets, Indicates batch size, This represents the set of all fused features obtained in a batch. and These represent the sets of all infrared and visible light features obtained in each batch. This is the formula for calculating the KL divergence.

2. The method according to claim 1, characterized in that, Infrared and visible light images acquired by the UAV are preprocessed using normalization to eliminate viewpoint sensitivity and motion interference in the UAV data, resulting in normalized infrared and visible light images, including: The infrared and visible light images acquired by the UAV were preprocessed using normalization to obtain the normalized infrared and visible light images, respectively: in, and These represent the mean and standard deviation of the infrared images, respectively. and These represent the mean and standard deviation of the visible light image, respectively. and This represents the normalized infrared image and the normalized visible light image. Indicates pixel position, Represents an infrared image. Represents a visible light image.

3. The method according to claim 1, characterized in that, The feature extraction model based on the self-attention mechanism is centered on a multi-layer multi-head self-attention layer. Each layer contains a layer normalization unit, a multi-head self-attention unit, and a feedforward layer unit. Global feature extraction is performed on the normalized infrared image and the normalized visible light image according to the self-attention mechanism-based feature extraction model to obtain infrared features and visible light features, including: The normalized image is divided into... The pixel block is divided into sub-regions, and each sub-region is flattened as follows: 3D feature vectors are used to obtain infrared feature sequences. With visible light feature sequences N represents the number of subregions; The infrared and visible light feature sequences are standardized using a layer normalization unit to obtain the layer normalized output. In the multi-head self-attention unit, the normalized output of the layer is projected onto the query matrix, the key matrix, and the value matrix, respectively. The self-attention module, which measures the query matrix, key matrix, and value matrix using cosine similarity, obtains attention weights between subsequences through scaling factors and softmax operations. Then, it performs weighted fusion on the value matrix to obtain the final output of the self-attention layer. A multi-head self-attention layer is used to project the query matrix, key matrix, and value matrix to multiple feature subspaces respectively, and then the attention is calculated separately and concatenated to obtain the output of the multi-head attention layer. The output of the multi-head attention layer is input into the feedforward layer to obtain infrared and visible light features.

4. The method according to claim 3, characterized in that, The multi-head self-attention unit projects the layer-normalized output onto the query matrix, key matrix, and value matrix, respectively, including: The multi-head self-attention unit projects the layer-normalized outputs onto the query matrix. Key matrix Sum matrix for: in, , , , , The output of the normalized layer is represented.

5. The method according to claim 3, characterized in that, The self-attention module, which uses cosine similarity as a metric for the query matrix, key matrix, and value matrix, obtains attention weights between subsequences through scaling factors and softmax operations. Then, it performs weighted fusion of the value matrices to obtain the final output of the self-attention layer, including: query matrix Key matrix Sum matrix A self-attention module using cosine similarity as a metric, through a scaling factor The attention weights between subsequences are obtained through softmax operations, and then the value matrix is ​​weighted and fused to obtain the final output of the self-attention layer. 。 6. The method according to claim 3, characterized in that, A multi-head self-attention layer is used to project the query matrix, key matrix, and value matrix onto multiple feature subspaces, respectively. Attention is then calculated for each subspace and concatenated to obtain the multi-head attention layer output, which includes: The query matrix, key matrix, and value matrix are projected onto multiple feature subspaces using a multi-head self-attention layer. Attention is then calculated for each subspace and concatenated to obtain the output of the multi-head attention layer. in, This is the final output of the multi-head self-attention layer. This indicates a splicing operation. Indicates the first The output of each attention head, Indicates the number of heads of attention. This represents the recovery matrix.

7. The method according to claim 3, characterized in that, The output of the final multi-head attention layer is input into the feedforward layer to obtain infrared and visible light features, including: The output of the final multi-head attention layer is input into the feedforward layer to obtain the infrared and visible light features, respectively: in, This represents a feedforward layer network. and These represent the extracted infrared and visible light features, respectively. This indicates the proposed feature extraction layer based on the self-attention mechanism. This indicates a multi-head attention mechanism extraction network. and This represents the normalized infrared and visible light images. Presentation layer normalization operation.

8. The method according to claim 1, characterized in that, The dual-path cross-attention includes an infrared-visible cross-attention branch dominated by the infrared mode and a visible-infrared cross-attention branch dominated by the visible mode. The infrared and visible features are input into a feature interaction module based on a cross-attention mechanism. Through dual-path cross-attention, information interaction and fusion of the two features are achieved to obtain fused features, including: In the infrared-visible cross-attention branch, infrared features are used as the basis. Visible light characteristics are bonds and Features are obtained by computing across attention mechanisms. , is represented as: In the visible-infrared cross-attention branch, infrared features are used as the basis. Visible light characteristics are bonds and Features are obtained by computing across attention mechanisms. , is represented as: Features and After concatenation, the feature dimensions are adjusted to obtain the fused features. , is represented as: in, This represents the convolution operation. These are the convolution weights.

9. The method according to claim 1, characterized in that, The modality contrastive learning loss is: in, Indicates fusion features, and This indicates that positive samples are features corresponding to the two modalities. This indicates that, apart from the anchor point, this batch contains... The set of all other samples, express The samples in This indicates the temperature hyperparameter.

Images