Complex environment infrared and visible light unmanned aerial vehicle cooperative sensing method based on hybrid expert model

Abstract

The invention discloses a complex environment infrared and visible light unmanned aerial vehicle cooperative sensing method based on a hybrid expert model, and belongs to the field of infrared and visible image fusion. The system comprises three parts: a double-branch image encoder network carrying an environment perception hybrid expert model, a domain constraint interaction fusion module, and a decoder network based on visual Transform. The environment perception hybrid expert model is composed of a special route and a space-channel expert, and processing paths can be dynamically allocated according to the characteristics of an input image through the special route; and the space-channel expert module can perform dissimilation processing on different types of random environment interferences, so that the influence of the random environment on the input image is removed, and the quality of the fused image is remarkably improved. And the domain constraint interaction fusion layer is used for realizing stable feature interaction fusion by constraining the domain consistency of the features.

Description

Technical Field

[0001] This invention relates to the field of infrared and image fusion, specifically to a collaborative perception method for infrared and visible light UAVs in complex environments based on a hybrid expert model. Background Technology

[0002] Infrared and visible light images possess unique characteristics due to their different imaging mechanisms. Infrared images provide stable imaging results in low-light or nighttime conditions, while visible light images capture rich details and color information in well-lit conditions. Typically, these images are acquired from target areas using equipment such as drones, vehicles, and surveillance systems equipped with infrared or visible light sensors.

[0003] The fusion of infrared and visible light images is of great significance. Through fusion, complementary information from the two images can be organically combined, thereby improving scene understanding and target recognition capabilities. Especially in complex environments, the fused image can compensate for the limitations of a single imaging mode, providing high-quality data support for subsequent image analysis and processing.

[0004] Devices such as drones equipped with infrared or visible light sensors are subject to random environmental disturbances when acquiring images of a target area. For example, changes in wind speed, lighting conditions, or weather factors (such as rain or fog) can all affect the quality of the acquired images. These random disturbances significantly increase the difficulty of image processing, causing the loss of important information about the target in the images.

[0005] Existing infrared and visible light image fusion methods typically use clean images as input. DIFNet (Jung H, Kim Y, Jang H, et al. Unsupervised deep image fusion with structure tensor representations[J]. IEEE Transactions on Image Processing, 2020, 29: 3845-3858.) was the first to utilize deep networks to handle multimodal and multi-task image fusion tasks, introducing a parameterized fusion framework that can independently optimize feature extraction, fusion, and reconstruction, thereby improving performance. U2Fusion (Xu H, Ma J, Jiang J, et al. U2Fusion: A unified unsupervised image fusion network[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2020, 44(1): 502-518.) proposed an adaptive information preservation mechanism for multimodal, multi-exposure, and multi-focus scenes, using the EWC mechanism to preserve the memory of each task without being affected. Zhang et al. (Zhang H, Xu H, Xiao Y, et al. Rethinking the image fusion: A fast unified image fusion network based on proportional maintenance of gradient and intensity[C] / / Proceedings of the AAAI conference on artificial intelligence. 2020, 34(07): 12797-12804.) reformulated the fusion task as preserving the texture and intensity ratio and proposed a unified and efficient fusion network. However, the above methods fail to separately consider the imaging perturbations of infrared and visible light sensors in unknown environments, generating fused images that include noise information from the source images. This makes them difficult to directly deploy in real, complex scenarios for collaborative perception between infrared and visible light UAVs.

[0006] The Transformer architecture based on the Mixture of Experts (MoE) model demonstrates strong adaptability in multi-task scenarios. This architecture can select appropriate paths for processing based on the characteristics of the input data, making it suitable for image feature extraction and fusion. However, its classification mechanism struggles to effectively distinguish the specific types of noise when faced with different environmental noise levels, which limits its denoising capabilities in complex environments. Summary of the Invention

[0007] To address the shortcomings of existing technologies, this invention proposes a hybrid expert-based environment-aware Transformer for fusing infrared and visible light images acquired in complex environments. It comprises three parts: a dual-branch image encoder network incorporating an environment-aware hybrid expert model, a domain-constrained interactive fusion module, and a visual Transformer-based decoder network. The environment-aware hybrid expert model consists of dedicated routing and spatial-channel experts. Dedicated routing dynamically allocates processing paths based on the characteristics of the input image; while the spatial-channel expert module provides specialized processing for different types of random environmental interference, thereby removing the influence of random environments on the input image and significantly improving the quality of the fused image. The domain-constrained interactive fusion layer achieves stable feature interaction fusion by constraining the domain consistency of features.

[0008] A collaborative sensing method for infrared and visible light UAVs in complex environments based on a hybrid expert model, comprising the following steps:

[0009] S1: Acquire infrared and visible light images of multiple scenes, and randomly add a type of interference to each image.

[0010] Furthermore, in step S1, the interference added to the visible light image is rain, random noise, or blurring. The interference added to the infrared image is random noise or stripe noise.

[0011] S2, Constructing a backbone model and domain constraint interaction fusion module:

[0012] The backbone model uses ViT (Vision Transformer) as the backbone network, which includes a dual-branch image encoder and a single-branch image decoder. Infrared and visible light images are input into the infrared encoder branch and the visible light encoder branch, respectively. Features are extracted by the dual-branch image encoder, and the processed features are fused by the domain-constrained interactive fusion module and reconstructed by the single-branch decoder to obtain the fused result.

[0013] Domain-Constrained Interactive Fusion Module: This module fuses the output features of a dual-branch image encoder. First, it constrains the mapping distribution of the output features using cosine similarity. Second, it enhances the feature representations of the two branches through an interactive attention layer. Finally, it passes the preliminarily fused features after addition through multiple convolutional layers to achieve deep feature fusion.

[0014] Furthermore, to improve the computational efficiency of the network, the features are reduced in dimensionality through a fully connected layer before the corresponding cosine similarity is calculated.

[0015] Furthermore, the processing procedure for the backbone model is as follows:

[0016] Step 2.1: Encoding by the dual-branch image encoder

[0017] The image is divided into fixed-size image patches. Each image patch is mapped to a low-dimensional embedding space through a linear transformation, resulting in image patch embeddings. These image patch embeddings, along with learnable perturbation class labels, are input into stacked Transformer blocks for sequence processing. The stacked Transformer blocks consist of multiple ViT modules, each containing a multi-head self-attention mechanism and a feedforward network. Residual connections and LayerNorm are used between ViT modules to ensure training stability and the network's expressive power.

[0018] In the dual-branch infrared and visible light encoder, the infrared encoder and the visible light encoder respectively receive image inputs of their corresponding modalities and extract features through the corresponding ViT backbone network. During the feature extraction process, image patch embedding incorporates positional information encoding to preserve spatial information. After passing through M layers of ViT modules, the depth feature representation of each layer is obtained.

[0019] Step 2.2, Integration of the Domain Constraint Interaction Fusion Module

[0020] First, the depth feature representations of the infrared and visible light branches are fused through a domain-constrained interactive fusion module. The fusion method includes cosine similarity constraints and interactive attention mechanisms.

[0021] Furthermore, the integration of the domain constraint interaction fusion module is as follows:

[0022] Step 2.2.1: The output features of the dual-branch image encoder are... The distribution of output features of the dual-branch image encoder is constrained by cosine similarity, which is expressed as:

[0023]

[0024] in, This represents the dot product of two eigenvectors. Characteristic Norm.

[0025] Step 2.2.2: The interactive attention mechanism generates an attention weight matrix by calculating the dot product between features. For infrared features... and visible light characteristics The formula for calculating attention weights is as follows:

[0026]

[0027] in, and These represent the attention weights from infrared features to visible light features and from visible light features to infrared features, respectively. is a scaling factor for the feature dimension, used to balance the magnitude of the dot product. The interactive feature representation is updated as follows:

[0028]

[0029] Step 2.2.3: Enhance the infrared features and visible light characteristics Adding them together, we obtain the initial fusion characteristics:

[0030]

[0031] Step 2.2.4: Perform in-depth feature processing on the initial fused features through a series of convolutional layers to finally generate high-quality fused features:

[0032]

[0033] in, This represents a convolution operation consisting of three 1×1 convolutions at the beginning and end, and a 3×3 convolution in the middle.

[0034] Step 2.3: Decoding by a single-branch image decoder

[0035] The fused features are input into a single-branch image decoder, and then processed through multiple ViT modules to generate a fused image.

[0036] Furthermore, to improve the feature representation capability of the encoder, the ViT backbone network is initialized by introducing pre-trained weights to enhance feature extraction capabilities.

[0037] S3, Constructing an Environment-Aware Hybrid Expert Model: In the dual-branch image encoder, an environment-aware hybrid expert model is inserted after every two ViT modules to remove perturbations from different input images. The environment-aware hybrid expert model consists of a dedicated routing module and a spatial-channel expert module. The dedicated routing module predicts the category of environmental perturbations in the input image based on intermediate features and generates corresponding expert weights. The spatial-channel expert module removes image perturbations at both spatial and channel scales by combining spatial experts and shared channel experts, and then fuses them according to the expert weights to generate the final feature representation.

[0038] The dedicated routing module mentioned in step S3: The dedicated routing module uses intermediate features of the m-th layer. As input, weights are extracted by adjusting learnable parameters.

[0039] The space-channel expert module mentioned in step S3 consists of regular experts and shared channel experts, and is responsible for removing disturbances at different scales. Regular experts are... Each of the linear layers has a conventional expert specializing in a specific type of perturbation removal operation. Input features Compression to spatial dimensions: . No. The output of each expert is represented as The shared channel expert consists of multiple convolutional layers, used to maintain the same number of channels while simultaneously increasing the feature size. Compress to The outputs of the regular expert and the shared channel expert are fused based on the weights generated by the dedicated routing module. The resulting outputs are then fused using the following formula:

[0040]

[0041] in, For the environment-aware hybrid expert model inserted at layer m, the features The processing results.

[0042] S4, Constructing the Loss Function: Used to constrain model convergence during training, mainly including domain constraint loss, fusion loss, and reconstruction loss. Domain constraint loss ensures domain consistency of features extracted by the dual-branch image encoder. Fusion loss ensures the fused image retains contrast information from the infrared image and texture details from the visible light image. Reconstruction loss preserves the overall information of the fused image.

[0043] S5, Training and Inference Phase: During the training phase, a clean fused image is learned from the input perturbed image by supervising the network through domain constraint loss, fusion loss, and reconstruction loss. During the inference phase, only the perturbed infrared and visible light image pairs are input to obtain the corresponding clean fused image.

[0044] During the training phase, infrared and visible light images are randomly perturbed and used as inputs to the two branches.

[0045] During the inference phase, the network automatically determines the perturbation category of the input image, selects appropriate experts and weights, and generates a clean fused image.

[0046] The beneficial effects of this invention are as follows: By introducing an environmental perception hybrid expert model, this invention addresses common interference factors in the fusion of infrared and visible light images in complex environments, such as random noise, rain noise, stripe noise, and image blurring. It designs a dual-branch image encoder with dynamic feature processing capabilities and a domain-constrained interactive fusion module, effectively improving image fusion quality and enhancing the system's adaptability in various complex scenarios. The dedicated routing module used in this invention can adaptively adjust the processing path of the expert model based on input features, while the spatial-channel expert module achieves perturbation removal and feature enhancement at multiple scales, promoting the full fusion of complementary features from infrared and visible light images. Through the domain-constrained interactive fusion mechanism, modal consistency constraints and interactive enhancements are achieved in the feature space, further improving the overall performance of the fused image in terms of contrast, structural restoration, and detail rendering. Compared with existing technologies, this invention not only improves the subjective visual effect of the fused image but also achieves excellent performance in objective evaluation indicators (including information entropy, standard deviation, peak signal-to-noise ratio, structural similarity, feature space contrast, and spatial frequency), verifying the effectiveness and application value of the proposed method in collaborative perception tasks of UAVs in complex environments, and demonstrating good prospects for widespread application. Attached Figure Description

[0047] The accompanying drawings, which form part of this invention, are used to provide a further understanding of the invention. The illustrative embodiments of the invention and their descriptions are used to explain the invention and do not constitute an improper limitation of the invention.

[0048] Figure 1 This is a flowchart of the collaborative perception method for infrared and visible light UAVs in complex environments based on a hybrid expert model, according to the present invention.

[0049] Figure 2 These are instances of the training and testing datasets;

[0050] Figure 3 This is an overall framework diagram of the present invention;

[0051] Figure 4This is a diagram illustrating the reasoning and application of the collaborative sensing method for infrared and visible light UAVs in complex environments based on a hybrid expert model, as described in this invention.

[0052] Figure 5 This is the fusion result of the present invention. Detailed Implementation

[0053] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0054] It should be noted that the following detailed description is illustrative and intended to provide further explanation of the invention. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains.

[0055] Where there is no conflict, the embodiments and features in the embodiments of the present invention can be combined with each other.

[0056] Example 1

[0057] like Figure 1 As shown, Embodiment 1 of the present invention provides a flowchart of a collaborative perception method for infrared and visible light UAVs in complex environments based on a hybrid expert model, including the following steps:

[0058] S1. Acquire infrared and visible light images of multiple scenes, introduce random perturbations into each image, and simulate the impact of environmental, equipment and other factors on the image.

[0059] S2. Construct the backbone model. The model uses the ViT model as the backbone network. The constructed model includes a two-branch image encoder and a single-branch decoder.

[0060] S3. Construct an environment-aware hybrid expert model that classifies and processes input features through dedicated routing and spatial-channel experts.

[0061] S4. Construct a domain-constrained interactive fusion module, which includes a domain-constrained module and an interactive fusion model, respectively maintaining the domain consistency of input features and performing interactive feature fusion.

[0062] S5. Construct a loss function that includes domain constraint loss, fusion loss, and reconstruction loss, and ensure the joint constraint model converges.

[0063] S6. Training and Inference Phase: The training phase utilizes multi-scenario data to train the model, while the inference phase integrates infrared and visible light images of complex environments collected by drones.

[0064] Specifically, S1 is:

[0065] This invention utilizes three publicly available infrared and visible light image datasets and adds random environmental perturbations to each image to train and infer the model.

[0066] The LLVIP dataset contains 16,836 pairs of visible light and infrared images, totaling 33,672 images, primarily captured in extremely low-light scenes. All images are strictly aligned temporally and spatially and annotated with pedestrian information. This dataset can be used for low-light vision tasks such as image fusion, pedestrian detection, and image-to-image translation.

[0067] The MSRS dataset contains 1083 training pairs and 361 test pairs. This dataset primarily consists of rigorously registered infrared and visible light image pairs representing traffic scenes, both daytime and nighttime.

[0068] The DroneVehicle dataset contains 56,878 images taken by drones, divided into RGB and infrared categories. The dataset includes directional bounding box annotations for five vehicle types (cars, trucks, buses, vans, and freight trains). Specifically, the car category has 389,779 and 428,086 annotations in the RGB and infrared images, respectively, with a resolution of 840×712.

[0069] When utilizing the above datasets, this invention uses 12,025 image pairs from the LLVIP dataset and 1,083 image pairs from the MSRS dataset as training sets. Fifty image pairs from the DroneVehicle dataset are used as a test set to verify the effectiveness of the invention's technology for collaborative perception of complex environments using infrared and visible light UAVs.

[0070] This invention adds five types of random perturbations to images based on common complex environments. These are stripe noise and random noise for infrared images, and random noise, blurring, and rain noise for visible light images.

[0071] Using the MSRS dataset as an example, the original image and the image with added noise are shown below. Figure 2 As shown in the figure, the first row displays the visualization results of visible light images subjected to three random perturbations: blur, rain, and random noise. The second row displays the visualization results of infrared images subjected to two perturbations: stripe noise and random noise. The following section details how to generate these perturbations.

[0072] Random noise: By adding three types of random perturbations—Gaussian noise, Poisson noise, and salt-and-pepper noise—to the image, random interference in real-world scenes is simulated. Different types of noise are added to the image sequentially: Gaussian noise is generated by superimposing normally distributed random values ​​onto each pixel; Poisson noise simulates the randomness of photon statistics; and salt-and-pepper noise is generated by randomly setting some pixels to black or white.

[0073] Image Rain Addition: This feature adds rain noise to images by simulating raindrop effects, generating realistic rainy scenes. The rain noise is created by randomly generating noise points, combining Gaussian blur and rotation operations to simulate the falling shape and angle changes of raindrops. The generated rain noise is then superimposed onto the original image with a certain degree of transparency. The length, angle, density, and blur level of the raindrops are adjusted according to hyperparameters to generate diverse rainy-day effect images.

[0074] Stripe noise: By adding stripe noise to an image, the stripe interference effect that imaging devices may encounter in complex environments is simulated. Stripe noise consists of randomly generated column vectors, which are replicated and expanded to form the stripe pattern of the image. Random Gaussian noise can be superimposed to enhance realism. The density and intensity of the stripes can be adjusted by parameters, making it suitable for generating image samples with stripe interference. This allows for the study of the impact and robustness of noise on image processing algorithms or deep learning models.

[0075] Image blurring: A blurred image is generated by applying Gaussian blur and downsampling to the image. The blurring is achieved using a Gaussian filter, followed by a 2x downsampling and resampling to enhance the blur effect. This simulates low-resolution and blurred image scenes.

[0076] S2 specifically refers to:

[0077] like Figure 3 As shown, preprocessed infrared and visible light image data and their corresponding perturbation category labels are input into a dual-branch image encoder for feature extraction and a decoder for feature reconstruction. The basic model structure consists of ViT modules; it includes a two-branch encoder and a single-branch decoder; the feature processing process of the two encoders and decoders is as follows.

[0078] The dual-branch encoding process is as follows: In the infrared and visible light dual-branch encoder, the infrared encoder and the visible light encoder respectively receive the image input of their corresponding modalities and extract features through the ViT backbone network. During feature extraction, image patch embedding incorporates positional information encoding to preserve spatial information. After creating layers of ViT Blocks, the depth feature representation of each layer is obtained, which is formalized as follows:

[0079]

[0080] in, Let M represent the feature representations of the infrared and visible light branches at the m-th layer, respectively, where M is the depth of the encoder.

[0081] The decoding process is as follows: First, the final depth features of the infrared and visible light branches are represented. Feature fusion is performed through a feature fusion module, employing methods including cosine similarity constraints and interactive attention mechanisms to enhance the expressive power of bimodal features. The fused feature representation is then presented. The input decoder, after passing through multiple ViT modules, generates a high-quality fused image. The process is as follows:

[0082]

[0083] During the decoding process, feature fusion further enhances the contrast information of the infrared image and the texture details of the visible light image, ultimately achieving the conversion from perturbed infrared and visible light images to a clear fused image. The model is initialized using pre-trained parameters from the ViT-Base model, effectively improving the performance and robustness of feature extraction.

[0084] Specifically, S3 describes an environment-aware hybrid expert model designed to enhance the feature processing capabilities of infrared and visible light image encoders in complex environments. This model is inserted after every two ViT modules in the dual-branch image encoder to remove perturbations from the input image. The model consists of a dedicated routing module and a spatial-channel expert module. First, the dedicated routing module predicts the perturbation category based on the intermediate features of the input image and generates corresponding expert weights. In the fusion stage, the outputs of the spatial expert and the shared channel expert are fused according to their weights to generate the final feature representation.

[0085] Dedicated routing module: The dedicated routing module uses the characteristics of layer m. For input, Representation of features By learning functions Output expert weights ,in The weights for the number of experts are obtained using the following formula:

[0086]

[0087] in Indicates the use of generating the first A routing function with expert weights, These are learnable parameters. Expert weights. This indicates the degree of contribution each expert makes to the input features. Indicates the first Each of the top K experts is assigned a weight. In this way, the dedicated path module can effectively distinguish different types of disturbances, providing guidance for subsequent feature processing. Subsequently, we retain the weights only for the top K experts, setting the weights of the other experts to 0. ,in .

[0088] Spatial-Channel Expert Module: The Spatial-Channel Expert module focuses on removing perturbations from input features, aiming to preserve both spatial and channel information of the features. This module consists of two parts: a regular expert and a shared channel expert. The regular expert performs specific processing on different types of local perturbations (such as noise, blur, etc.) through multiple independent linear layers. Each linear layer works independently and outputs the processed feature representation. Specifically, the regular expert consists of... It consists of several linear layers, with each specialist focusing on a specific type of perturbation removal operation. A typical expert will input features Compressed to spatial dimensions The specific formula is as follows:

[0089]

[0090] in, It is the first The weight matrix of each expert.

[0091] The shared channel expert adds feature processing capabilities based on convolution operations to extract global features, especially important information in cross-channel and large-scale feature distributions.

[0092]

[0093] in, This represents a convolution operation with three 3×3 kernels, a stride, and padding of 1. .

[0094] Shared channel experts extract more comprehensive feature representations at both spatial and channel scales by maintaining the number of channels and performing deep feature compression. The combination of these two approaches enables the module to handle both local perturbations and extract global information, forming an efficient and robust removal mechanism for various perturbation types. Finally, the outputs of the regular expert and the shared channel expert are fused according to the weights generated by the dedicated routing module to obtain the final feature representation.

[0095]

[0096] in, For the environment-aware hybrid expert model inserted at layer m, the features The processing results. Through this design, the spatial-channel expert module can efficiently remove disturbances at multiple scales while preserving the spatial and channel information of the input features.

[0097] S4 specifically refers to:

[0098] The domain-constrained interactive fusion module proposed in this invention is used to fuse the output features of a dual-branch image encoder. This module aims to enhance the complementarity and consistency of infrared and visible light modal features and consists of the following three steps:

[0099] First, the module constrains the distribution of the output features of the dual-branch image encoder using cosine similarity to ensure domain consistency between infrared and visible light features. Let the output features of the dual-branch image encoder be... The cosine similarity is expressed as:

[0100]

[0101] in, This represents the dot product of two eigenvectors. Characteristic Norm. By maximizing cosine similarity, modules can make the distributions of features from two modalities more similar in the feature space, thereby enhancing domain consistency. Preferably, to improve computational efficiency, features are reduced in dimensionality through a fully connected layer before cosine similarity calculation to reduce computational complexity while preserving key semantic information.

[0102] Furthermore, after initially aligning the feature distributions, the module further enhances the representational power of the two modalities through an interactive attention mechanism. The main goal of the interactive attention layer is to capture the correlation between the two modalities and emphasize shared salient features.

[0103] The interactive attention mechanism generates an attention weight matrix by calculating the dot product between features. For infrared features... and visible light characteristics The formula for calculating attention weights is as follows:

[0104]

[0105] in, and These represent the attention weights from infrared features to visible light features and from visible light features to infrared features, respectively. is a scaling factor for the feature dimension, used to balance the magnitude of the dot product. These weight matrices allow the module to focus on the parts of the two-modal features that contribute to the fusion, thereby improving the expressive power of feature interactions. The updated feature representation after interaction is:

[0106]

[0107] Furthermore, after enhancing the features through the interactive attention mechanism, the module further fuses the features. First, the enhanced infrared features are... and visible light characteristics Adding them together, we obtain the initial fusion characteristics:

[0108]

[0109] Then, a series of convolutional layers are used to perform in-depth feature processing on the initially fused features to extract high-order semantic information and remove redundant features. Convolutional operations not only further integrate complementary information between modalities but also improve the overall consistency of the fused features, ultimately generating high-quality fused features.

[0110]

[0111] in, This represents a convolution operation consisting of three 1×1 convolutions at the beginning and end, and a 3×3 convolution in the middle.

[0112] S5 specifically refers to:

[0113] This invention designs a comprehensive loss function incorporating multiple loss mechanisms to constrain the convergence of the model during training and ensure that the generated fused image retains both the contrast information of the infrared image and the texture details of the visible light image. The loss function mainly includes domain constraint loss, fusion loss, and reconstruction loss.

[0114] Domain constraint loss: The domain constraint loss ensures the consistency of infrared and visible light modal features in the shared feature space by aligning the features output by the dual-branch image encoder. Cosine similarity is used as the alignment metric, and the alignment loss between the infrared and visible light branches is defined as follows:

[0115]

[0116] Fusion Loss: The fusion loss measures the structural and pixel similarity between the generated fused image and the original infrared and visible light images, ensuring that the fused image retains key features of both modalities. Specifically, structural loss is achieved through... constraint:

[0117]

[0118] in, It is a multi-scale structural similarity index. Infrared image, This is a visible light image. Simultaneously, we utilize pixel loss... Measuring the absolute difference between the fused image and the infrared and visible light images at the pixel level:

[0119]

[0120] Reconstruction Loss: The reconstruction loss improves the overall quality of the fused image through two strategies: pixel-maximum loss and gradient-maximum loss. Gradient-maximum loss is used to reconstruct important gradient information from the original image.

[0121]

[0122] in, Represents the Sobel operator. This indicates a maximum value operation. The maximum pixel loss is used to reconstruct important contrast information from the infrared image.

[0123]

[0124] The final comprehensive loss function is a weighted combination of the above-mentioned losses:

[0125]

[0126] in, To assist in loss calculation, this method is used to balance the load distribution of expert models, ensuring a more even distribution of load among experts in the model, thereby improving the training efficiency and generalization ability of the model.

[0127] S6 specifically demonstrates the superior performance of this invention in collaborative perception tasks involving infrared and visible light UAVs in complex environments. In experiments with 50 images from the DroneVehicle dataset, the generated fused images exhibited stable and excellent results across multiple objective evaluation metrics. Specifically, as shown in Table 1, the fused images maintained an information entropy (EN) of 7.28 to 7.29, indicating rich information content; the standard deviation (SD) remained stable at 10.38 to 10.41, showing uniform grayscale distribution; the peak signal-to-noise ratio (PSNR) reached 29.19 to 29.48; the structural similarity (SSIM) ranged from 0.58 to 0.65, with blurred scenes having the greatest impact on structural similarity, but overall still retaining image structural information well; the feature spatial contrast (SCD) and spatial frequency (SF) fluctuated within the ranges of 1.15 to 1.25 and 6.88 to 7.26, respectively, further validating the fused images' performance in feature contrast and texture detail. In summary, this invention demonstrates efficient processing capabilities for input features under various perturbation scenarios. The generated fused image not only retains the contrast information of the infrared image but also presents the texture details of the visible light image, fully demonstrating the robustness and superior performance of the model.

[0128] As shown in Figure 5, this invention demonstrates excellent performance in handling infrared and visible light image fusion tasks under various complex environmental disturbances. Under conditions of random noise, rain noise, blurring, and stripe noise, the fused image effectively preserves the contrast information of the infrared image and the texture details of the visible light image. Visually, the fusion result clearly presents the details of vehicles and roads. Particularly when the infrared and visible light images are subjected to different types of disturbances, the fused image generated by the model shows significant advantages in noise removal and detail restoration. Furthermore, regardless of whether the infrared or visible light image is disturbed, the global contrast and detail performance of the fused image are superior to the single-modal image, verifying the robustness and efficient processing capability of this invention in complex environments.

[0129] Table 1: Objective evaluation metrics of the present invention on a dataset of 50 DroneVehicles

[0130]

Claims

1. A collaborative perception method for infrared and visible light UAVs in complex environments based on a hybrid expert model, characterized in that, The steps are as follows: S1, acquire infrared and visible light images of multiple scenes, and randomly add a type of interference to each image; S2, Constructing a backbone model and domain constraint interaction fusion module: The backbone model uses ViT as the backbone network, which includes a dual-branch image encoder and a single-branch image decoder. Infrared and visible light images are input into the infrared encoder branch and the visible light encoder branch, respectively. The dual-branch image encoder extracts features, and the processed features are fused by the domain-constrained interactive fusion module and reconstructed by the single-branch decoder to obtain the fused result. Domain-constrained interactive fusion module: used to fuse the output features of a dual-branch image encoder. First, the mapping distribution of the output features is constrained by cosine similarity; second, the feature representation of the two branches is enhanced by an interactive attention layer; finally, the preliminary fused features are passed through multiple convolutional layers to achieve deep feature fusion. S3, Construct an environment-aware hybrid expert model: In the dual-branch image encoder, an environment-aware hybrid expert model is inserted after every two ViT modules to remove perturbations from different input images; the environment-aware hybrid expert model consists of a dedicated routing module and a spatial-channel expert module. The dedicated routing module predicts the category of environmental perturbations to the input image based on intermediate features and generates corresponding expert weights. The spatial-channel expert module removes image perturbations at both spatial and channel scales by combining spatial experts and shared channel experts, and then fuses them according to expert weights to generate the final feature representation. S4, Construct the loss function: used to constrain model convergence during training, mainly including domain constraint loss, fusion loss and reconstruction loss; domain constraint loss ensures the domain consistency of features extracted by the dual-branch image encoder; fusion loss ensures that the fused image retains the contrast information of the infrared image and the texture detail information of the visible light image; reconstruction loss ensures the overall information of the fused image; S5, Training and Inference Phase: During the training phase, a clean fused image is learned by supervising the network through domain constraint loss, fusion loss, and reconstruction loss using the input perturbed image; during the inference phase, the corresponding clean fused image can be obtained by inputting only the perturbed infrared and visible light image pairs.

2. The method for collaborative perception of complex environments using infrared and visible light UAVs based on a hybrid expert model according to claim 1, characterized in that, In step S1, the interference added to the visible light image is rain, random noise, or blur; the interference added to the infrared image is random noise or stripe noise.

3. The method for collaborative perception of complex environments using infrared and visible light UAVs based on a hybrid expert model according to claim 1, characterized in that, In step S2, the processing procedure for the backbone model is as follows: Step 2.1: Encoding by the dual-branch image encoder The image is divided into fixed-size image blocks. Each image block is mapped to a low-dimensional embedding space through a linear transformation to obtain image block embeddings. These image block embeddings, along with learnable perturbation category labels, are input into stacked Transformer blocks for sequence processing. The stacked Transformer block consists of multiple ViT modules, each containing a multi-head self-attention mechanism and a feedforward network. Residual connections and LayerNorm are used between ViT modules to ensure training stability and the network's expressive power. In the dual-branch encoder of infrared and visible light, the infrared encoder and the visible light encoder respectively receive the image input of the corresponding modality and extract features through the corresponding ViT backbone network; During the feature extraction process, image patch embedding incorporates location information encoding to preserve spatial information. After passing through M layers of ViT modules, the depth feature representation of each layer is obtained. Step 2.2, Integration of the Domain Constraint Interaction Fusion Module First, the depth feature representations of the infrared and visible light branches are fused through a domain-constrained interactive fusion module. The fusion method includes cosine similarity constraints and interactive attention mechanisms. Step 2.3: Decoding by a single-branch image decoder The fused features are input into a single-branch image decoder, and then processed through multiple ViT modules to generate a fused image.

4. The method for collaborative perception of complex environments using infrared and visible light UAVs based on a hybrid expert model according to claim 3, characterized in that, In step S2, the fusion of the domain constraint interaction fusion module is as follows: Step 2.2.1: The output features of the dual-branch image encoder are... The distribution of output features of the dual-branch image encoder is constrained by cosine similarity, which is expressed as: ， in, This represents the dot product of two eigenvectors. Characteristic Norm; Step 2.2.2: The interactive attention mechanism generates an attention weight matrix by calculating the dot product between features; for infrared features... and visible light characteristics The formula for calculating attention weights is as follows: ， in, and These represent the attention weights from infrared features to visible light features and from visible light features to infrared features, respectively. is a scaling factor for the feature dimension, used to balance the magnitude of the dot product; the interactive feature representation is updated as follows: ， Step 2.2.3: Enhance the infrared features and visible light characteristics Adding them together, we obtain the initial fusion characteristics: ， Step 2.2.4: Perform in-depth feature processing on the initial fused features through a series of convolutional layers to finally generate high-quality fused features: ， in, This represents a convolution operation consisting of three 1×1 convolutions at the beginning and end, and a 3×3 convolution in the middle.

5. The method for collaborative perception of complex environments using infrared and visible light UAVs based on a hybrid expert model according to claim 1, characterized in that, The dedicated routing module mentioned in step S3: The dedicated routing module uses intermediate features of the m-th layer. As input, weights are extracted by adjusting learnable parameters.

6. The method for collaborative perception of complex environments using infrared and visible light UAVs based on a hybrid expert model according to claim 1, characterized in that, The space-channel expert module mentioned in step S3: The space-channel expert module consists of regular experts and shared channel experts, and is responsible for removing disturbances at different scales; the regular experts are... Each of the linear layers has a conventional expert specializing in a specific type of perturbation removal operation. Input features Compression to spatial dimensions: ;No. The output of each expert is represented as The shared channel expert consists of multiple convolutional layers, used to maintain the same number of channels while simultaneously increasing the feature density. Compress to ; The outputs of the regular expert and the shared channel expert are fused based on the weights generated by the dedicated routing module. After obtaining the outputs of the regular and shared channel experts, the expert outputs are fused using the following formula: ， in, For the environment-aware hybrid expert model inserted at layer m, the features The processing results.

Images