A multi-modal adaptive fusion method based on visual-linguistic semantic adversarial guidance

Abstract

The application discloses a kind of multi-modal adaptive fusion methods based on vision-language semantic confrontation guide, belong to multi-modal perception technical field.For the poor self-adaptive ability of existing dual light fusion algorithm in thick smoke, fog, night dark light and other complex all-weather environment, easy to be interfered by modal degradation problem, construct "front-end semantic block+back-end task-driven fusion" framework.Introduce vision-language model as all-weather semantic guide, realize semantic confrontation by the cross-attention similarity of double-flow visual features and positive and negative environmental semantic prompt, dynamically generate modal modulation weight, and the inferior is stored before feature interaction.Optimizing again, high-dimensional semantics is injected across dimension to downstream detection network, and bidirectional fusion of visible light texture and infrared thermal radiation is realized by light cross-attention, and feature reconstruction is completed by combining physical constraints such as thermal energy enhancement and texture denoising.The application effectively breaks through the bottleneck of multi-sensor fusion in extreme environment, and significantly improves the robustness and feature reliability of the system.

Description

Technical Field

[0001] This invention relates to the fields of computer vision, image processing and infrared-visible light image fusion technology, and in particular to a multimodal adaptive fusion method based on visual-language semantic adversarial guidance. Background Technology

[0002] In open scenarios such as autonomous driving, security monitoring, and drone reconnaissance, a single visual sensor often struggles to cope with complex and ever-changing all-weather environments. Infrared and visible light image fusion technology aims to combine the sensitivity of infrared sensors to thermal radiation targets with the ability of visible light sensors to capture high-frequency texture details. However, the reliability of information from visible light and infrared sensors fluctuates drastically with weather and lighting conditions (such as strong light, extreme darkness, rain, snow, and fog).

[0003] Existing fusion algorithms largely rely on local image gradients or statistical features to assign fusion weights, lacking a macroscopic understanding of the current global "environmental or weather semantics." This leads to situations where, in severely degraded scenarios (such as heavy fog at night), the system may still incorrectly assign excessive weights to low-quality visible light backgrounds, introducing significant noise and obscuring the originally clear infrared target features. How to enable the system to possess "weather perception common sense" similar to humans and adaptively suppress degraded modes while enhancing reliable modes is a critical technical bottleneck that urgently needs to be addressed in the field of multimodal fusion. Summary of the Invention

[0004] To address the problems of existing multimodal fusion technologies being severely affected by extreme environmental interference and lacking adaptive modality selection capabilities, this invention proposes a multimodal adaptive fusion method guided by visual-linguistic semantic adversarial approaches.

[0005] The core mechanism of this invention is as follows: at the feature front end, a prompt learning framework based on the visual-language model (VLM) is used to perform high-dimensional semantic alignment between image features and text descriptions, and positive and negative prompt semantic adversarial methods are used to guide features to move closer to positive prompts and away from negative prompts, thereby achieving early soft gating blocking; at the feature interaction back end, task-level high-dimensional semantics are injected into a lightweight cross-attention module, and combined with physical heuristic constraints, high-quality bidirectional feature fusion and reconstruction are completed.

[0006] The beneficial effects of this invention are as follows:

[0007] 1. All-weather intelligent perception: By utilizing the open-world semantic knowledge of the vision-language model, the system is given the ability to perceive the current weather and lighting conditions, breaking through the limitations of traditional methods that rely solely on the weights extracted from low-level image features.

[0008] 2. Adversarial Elimination and Plug-and-Play: Innovatively employing adversarial logic that subtracts positive from negative semantic similarity, it not only proactively enhances the similarity between features and clear semantics but also forcibly suppresses their similarity to severe weather semantics. The generated adaptive weights effectively perform spatial and channel-level soft gating of single-modal features before feature interaction, preventing degradation noise from contaminating the fusion channel.

[0009] 3. Dual guarantee of task-driven and physical constraints: The high-level semantics of the downstream detection network are embedded into the fusion process, so that the fused image actively caters to the downstream preferences; at the same time, a thermal enhancement layer and a texture denoising layer are introduced after cross-attention fusion, which effectively suppresses the "thermal halo" phenomenon and decouples high-frequency shot noise, ensuring the high fidelity of the final image structure and the extremely strong saliency of the target. Attached Figure Description

[0010] Figure 1 This is a schematic diagram of the overall process of the method of the present invention;

[0011] Figure 2 This is a schematic diagram of the semantic adversarial and weight generation process of the present invention;

[0012] Figure 3 This is a schematic diagram of the lightweight cross-attention fusion module of the present invention; Detailed Implementation

[0013] The present invention will now be described in further detail with reference to specific implementation steps.

[0014] Step 1: Feature extraction.

[0015] The system receives aligned infrared images and visible light images Spatial features of the underlying layer are extracted through parallel convolutional encoders. and .

[0016] Step 2: Construct the environmental semantic prompt embedding.

[0017] Establish two sets of textual descriptions reflecting the extremes of the environment: a set of positive semantic cues. (e.g., "clear scene", "high contrast", "well-lit") and negative semantic cues (e.g., "low light at night," "dense fog," "overexposure"). Using a pre-trained text encoder with frozen weights (such as the Text Encoder in the CLIP model), these discrete text prompts are transformed into positive text embedding matrices in a continuous semantic space. and negative text embedding matrix .

[0018] Step 3: Semantic spatial projection of visual features.

[0019] To compute cross-modal attention, the underlying visual features are... and After being mapped to the same dimensional space as the text embedding through a linear projection layer, a visual query vector is generated. and .

[0020] Step 4: Semantic adversarial and weight generation.

[0021] Taking the visible light branch as an example, the cross-attention between the visual query vector and the text embedding matrix is ​​calculated. The system calculates... With forward embedding The dot product similarity is calculated, and its similarity to the negative embedding is subtracted. The dot product similarity. Specifically, the physical operation logic of the adaptive modal modulation weights is as follows:

[0022]

[0023] in This is a scaling factor to prevent gradient vanishing. Normalize adversarial similarity to Within the range. When visible light images are contaminated by heavy fog. and The similarity will increase dramatically, causing the adversarial value within the brackets to be negative or extremely small. After Sigmoid activation, It will approach 0, thus achieving adaptive discarding of unreliable features. Infrared branch weights The calculation is similar.

[0024] Step 5: Feature modulation application.

[0025] The extracted low-level features are subjected to element-wise spatial and channel-weighted modulation to complete semantically guided filtering preprocessing, resulting in enhanced dual-stream features after environmental modulation.

[0026]

[0027]

[0028] in, Represents element-wise multiplication.

[0029] Step 6: Cross-dimensional embedding of task semantics.

[0030] To enable the fused image to proactively cater to the feature preferences of the downstream object detection network, cross-dimensional embedding of high-dimensional semantics is performed. High-dimensional detection features are extracted from the internal detector backbone network. Then, spatial alignment slicing is performed to extract local perceptual features that match the spatial scale of the current fused feature map. The query-key-value projection transformation logic is adopted: the dimensionality-reduced detection features are transformed... As a query vector, the environmental modulated infrared features output in step S5 are used. (or visible light characteristics) Semantic injection is performed using key and value vectors. Two independent feature streams carrying high-dimensional semantic priors are generated: enhanced infrared features. and enhance visible light characteristics .

[0031] Step 7: Lightweight cross-attention fusion mechanism.

[0032] To address the dimensionality explosion problem caused by high-resolution image fusion, this invention introduces a lightweight cross-attention module. This module utilizes depthwise separable convolution to extract feature maps and transforms the traditional spatial global dot product matrix into similarity aggregation along the channel dimension, thereby compressing the theoretical time complexity to a fraction of the total time. Cross-integration consists of two symmetrical and complementary paths:

[0033] Injection of visible light textures into infrared thermal radiation: to enhance infrared features as query vector To enhance visible light characteristics as a key vector Sum value vector By calculating attention weights along the channel dimension, the output features possessing both high temperature gradients and high-fidelity geometric structures. :

[0034]

[0035] Injection of infrared saliency into visible light texture: Reverse the above query logic, force the visible light feature stream to precisely focus on key physical entities using infrared thermal coordinates, and output texture features with extremely strong target saliency. .

[0036] Step 8: Physical heuristic constraints and feature reconstruction.

[0037] The cross-fused features, due to the complex channel mixing process, are prone to local energy overshoot (such as infrared thermal halos) or microstructural breaks. Therefore, this method applies specific physical heuristic constraints to the dual-stream features before decoding output:

[0038] Thermal enhancement and corona suppression of the infrared branch: targeting the fused infrared features A thermal enhancement layer is introduced. To prevent the "thermal halo" phenomenon of texture engulfment in the fused image, a hyperbolic tangent function is used. Nonlinear smoothing is applied to locally extremely high pixel values. As a preferred implementation, this step employs the following constraint transformation formula for the residual structure:

[0039]

[0040] in, This represents the final infrared signature after physical constraint processing. This represents a learnable weight matrix in the thermal enhancement layer, used to adaptively adjust the threshold and intensity of the nonlinear cutoff. This formula is derived through... The saturation characteristics limit the disordered expansion of thermal radiation intensity, while the original temperature gradient information is preserved through the residual branch.

[0041] Texture denoising and decoupling of visible light branches: targeting the fused visible light features A texture denoising layer is introduced. Through successive depth-group convolutions, the structured real-scene edges are decoupled from disordered high-frequency shot noise in the frequency domain, outputting high-fidelity texture features. .

[0042] Finally, the dual-flow features, after being processed by physical constraints, are... and The images are stitched together along the channel dimension and then fed into a decoder network for image reconstruction, outputting a high-quality, all-weather, interference-resistant final fused image. .

Claims

1. A multimodal adaptive fusion method based on visual-linguistic semantic adversarial guidance, characterized in that, Includes the following steps: S1. Acquire aligned infrared and visible light images, and extract infrared and visible light low-level features respectively using a dual-stream encoder; S2. Construct a set of positive semantic cue and a set of negative semantic cue to describe environmental conditions, and use a text encoder to map them into a positive text embedding matrix and a negative text embedding matrix, respectively; S3. Project the infrared underlying features and visible light underlying features into the same semantic space as the text embedding matrix to generate infrared visual query vectors and visible light visual query vectors; S4. Calculate the cross-attention similarity between the visual query vector and the positive and negative text embedding matrices, and generate infrared adaptive modulation weights and visible light adaptive modulation weights based on the adversarial subtraction operation of the positive and negative similarities. S5. The corresponding low-level features are weighted and modulated using the adaptive modulation weights to obtain enhanced dual-stream features after environmental modulation; S6. Extract the high-dimensional detection features of the downstream network, and use the high-dimensional detection features as a guide to perform semantic injection on the enhanced dual-stream features after environmental modulation, so as to generate enhanced infrared features and enhanced visible light features carrying high-dimensional semantic priors; S7. Input the enhanced infrared features and enhanced visible light features into the cross-attention module, perform bidirectional feature interactive injection, and output the cross-fused dual-stream features; S8. Apply physical heuristic constraints to the cross-fused dual-stream features, and perform feature stitching and reconstruction to output the final fused image.

2. The method according to claim 1, characterized in that, The text encoder described in step S2 keeps its parameters frozen during feature extraction and weight generation.

3. The method according to claim 1, characterized in that, The calculation process for the visible light adaptive modulation weight and the infrared adaptive modulation weight mentioned in step S4 is as follows: Visible light adaptive modulation weights The calculation formula is as follows: in, This represents a visible light visual query vector. This represents a positive text embedding matrix. This represents a negative text embedding matrix. It is a scaling constant. Indicates matrix transpose. This represents the Sigmoid activation function; The calculation process for the infrared adaptive modulation weight is the same as that for the visible light branch, using the infrared visual query vector to replace the visible light visual query vector in the above formula.

4. The method according to claim 1, characterized in that, The semantic injection described in step S6 is specifically calculated as follows: High-dimensional detection features generated by the downstream target detection network are extracted and spatially aligned and sliced. Using the sliced ​​local perceptual features as query vectors, attention is calculated by using the environmentally modulated infrared and visible light features as key and value vectors, respectively, to generate the enhanced infrared features. With enhanced visible light characteristics .

5. The method according to claim 1 or 4, characterized in that, The bidirectional feature interaction injection described in step S7 is calculated as follows: To enhance infrared signature as query vector To enhance visible light characteristics as a key vector Sum value vector The process of injecting visible light textures into infrared thermal radiation is expressed mathematically as follows: in, This represents the infrared features after injecting visible light texture. Represents the weight matrix. This represents matrix multiplication along the channel dimension; The process of injecting infrared saliency into visible light texture is the symmetrical inversion of this process, using enhanced visible light features as the query vector and enhanced infrared features as the key and value vectors, and outputting the cross-fused visible light features. .

6. The method according to claim 1 or 5, characterized in that, The cross-attention module mentioned in step S7 is a lightweight module. This module uses depthwise separable convolution to extract feature maps and converts the calculation of the spatial global dot product matrix into the calculation of attention similarity aggregation along the channel dimension to reduce computational complexity.

7. The method according to claim 1, characterized in that, The physical heuristic constraints mentioned in step S8 include: For the infrared features after cross-fusion, a hyperbolic tangent function (tanh) is used for nonlinear smoothing to suppress local thermal halo phenomenon; For the visible light features after cross-fusion, continuous depth grouping convolution is used to decouple the scene edge structure from high-frequency shot noise in the frequency domain.

Images