A Multimodal Image Fusion and Detection Method for Constrained Scenes Based on Two-Branch Diffusion Recombination

Abstract

A multimodal image fusion and detection method for constrained scenes based on bi-branch diffusion recombination is proposed. This method acquires infrared and visible light images from constrained scenes and stacks them as multi-channel inputs, then performs a forward diffusion noise addition process. A context-anchored attention module is embedded in the noise prediction network, capturing long-range dependencies through pooling and depthwise separable convolutions to generate attention weight modulation features. A parallel bi-branch structure is constructed, with a common branch extracting modality-independent scene structural features and a specific branch extracting modality-specific detail features, which are adaptively aggregated through a dynamic gating mechanism. A composite loss function is constructed, including bidirectional structural similarity loss, gradient loss, and intensity loss, and the noise prediction network is optimized through backpropagation. The reconstructed high-quality fused image is input into a pre-trained target detection model to obtain detection accuracy, and the running state of the actual scene is determined. This method significantly improves the fusion quality of infrared and visible light images in constrained scenes.

Description

Technical Field

[0001] This invention belongs to the field of industrial image processing and computer vision technology, specifically relating to a method for multimodal image fusion and detection in constrained scenes based on bi-branch diffusion recombination. Background Technology

[0002] With the widespread application of computer vision tasks in key fields such as autonomous driving and intelligent coal mining, the complexity and diversity of scenarios have brought significant challenges to visual perception tasks. Visible light images can provide intuitive details and scene information, serving as a core data source for visual perception. However, in constrained scenarios such as coal mine monitoring and nighttime security, the acquired visible light images often suffer from low quality and missing effective features due to limitations such as ambient light, dust interference, and sensor performance. This makes it difficult to clearly highlight key targets, severely impacting the accuracy of subsequent target detection. In contrast, infrared images rely on the difference in thermal radiation between the target and the environment to distinguish targets, possessing the advantage of being unaffected by lighting conditions and inclement weather. They can effectively capture target outlines in dark or complex environments. However, their inherent limitation lies in their inability to accurately identify the texture details of objects, making it difficult to provide refined semantic information about targets. Therefore, in constrained scenarios, relying solely on a single modality image cannot provide comprehensive and effective perceptual information, making it difficult for target detection tasks to meet industrial-grade high-precision requirements and satisfying the practical application needs of downstream scenarios.

[0003] Currently, deep learning-driven infrared and visible light image fusion methods have become a research hotspot in this field, with related technologies mainly revolving around three major frameworks: autoencoders, convolutional neural networks (CNNs), and generative adversarial networks (GANs). Although existing fusion methods have achieved certain breakthroughs in fusion performance, they still face many technical bottlenecks: blurred details in the fused image, severe intermodal interference, and a lack of effective semantic guidance; most methods focus on preserving the contour information of infrared targets and the background structure of visible light images, but neglect the preservation of color information in visible light images, while color features play an indispensable role in visual semantic understanding and target differentiation; in addition, existing generative model-based fusion methods, due to the additional constraints imposed on the generator, struggle to accurately construct the real data distribution of infrared and visible light images, resulting in deviations between the fusion effect and the real scene.

[0004] In recent years, diffusion models have shown superior performance to GAN models in image generation tasks. However, when its standard backbone network, U-Net, is applied to the fusion of infrared and visible light images in confined scenes, it still reveals many inherent defects. First, standard convolutional operations lack selectivity for feature responses and lack dynamic attention mechanisms to focus on key information of specific modalities (such as thermal target signals in infrared images), resulting in key features being diluted by redundant information and failing to effectively highlight the target. Second, fusion methods commonly used in existing fusion methods, such as feature concatenation and feature summation, are prone to intermodal interference, destroying the independence and integrity of features of each modality and affecting the fusion quality. Finally, a single encoder structure is difficult to effectively separate the features of the two modalities, and cannot independently represent the shared scene structure (common features) and the unique texture, thermal radiation, and other information (specific features) of each modality. This feature coupling phenomenon limits the model's full exploitation and utilization of the advantages of each modality.

[0005] In summary, to address the practical needs of constrained scenarios, developing an infrared and visible light image fusion and detection method that can achieve modal feature decoupling, enhance the detail representation of fused images, and maintain color consistency in visible light images has significant theoretical and engineering application value for improving the accuracy of downstream visual perception tasks and promoting the large-scale application of computer vision technology in constrained scenarios. Summary of the Invention

[0006] To address the problems existing in the prior art, this invention provides a method for multimodal image fusion and detection in confined scenes based on bi-branch diffusion recombination. This method can significantly improve the fusion quality of infrared and visible light images in confined scenes and effectively support downstream target detection tasks.

[0007] To achieve the above objectives, this invention provides a method for multimodal image fusion and detection in constrained scenes based on bi-branch diffusion recombination, comprising the following steps: Step 1: Construct a cross-modal feature extraction module based on an improved diffusion model; By acquiring infrared and visible light images in confined scenarios and stacking them into multi-channel inputs, a forward diffusion noise addition process is performed to provide noisy samples for subsequent noise prediction networks. Step 2: Introduce a context-anchored attention mechanism into the noise prediction network; A context-anchored attention module is embedded in the noise prediction network. Long-range dependencies are captured through pooling and depthwise separable convolutions, generating attention weight modulation features to enhance the response to texture details and edge features. Step 3: Design and train the cross-modal feature decoupling and adaptive fusion module; A parallel dual-branch structure is constructed, with a common branch extracting modality-independent scene structural features and a specific branch extracting modality-specific detailed features. The features are then adaptively aggregated through a dynamic gating mechanism to achieve efficient feature fusion. Step 4: Design a composite loss function to guide the model to perform the reverse noise reduction process; A composite loss function is constructed, including bidirectional structural similarity loss, gradient loss, and intensity loss. The noise prediction network is optimized through backpropagation, thereby reconstructing a high-quality fused image. Step 5: Based on the reconstructed fused image, perform object detection using a target detection network; The reconstructed high-quality fused image is input into a pre-trained target detection model to obtain the detection accuracy, and the running status of the actual scene is determined based on the detection results.

[0008] As a preferred embodiment, the process of constructing the cross-modal feature extraction module based on the improved diffusion model in step 1 is as follows: S11: Image Acquisition and Channel Stacking; Acquire infrared and visible light images under limited conditions, and stack the three visible light channels and the single infrared channel along the channel dimension to form a 4-channel input sample. ; S12: Forward diffusion noise addition process; for the initial sample Add Gaussian noise to obtain the time step. Noisy images at different times ; ; In the formula, , This represents the total number of diffusion steps; , , For predefined variance scheduling; It is standard Gaussian noise.

[0009] As a preferred option, in step 2, the process of introducing a context-anchored attention mechanism into the noise prediction network is as follows: S21: Local region feature extraction; Input feature map First, global average pooling is used to obtain After Convolution yields local region features As shown in the following formula: ; S22: Long-range spatially dependent capture; Depthwise separable convolution is used, based on horizontal convolution kernels. With vertical convolution kernel Scanning in the width and height directions generates attention weights. ; S23: Feature modulation; The attention weights are multiplied element-wise with the original feature map to obtain the output features. As shown in the following formula: ; In the formula, This indicates element-wise multiplication.

[0010] As a preferred option, in S22, the long-distance spatially dependent capture process is as follows: right Perform horizontal depthwise convolution, then vertical depthwise convolution, and finally sigmoid activation to obtain attention weights. .

[0011] As a preferred option, the process of designing and training the cross-modal feature decoupling and adaptive fusion module in step 3 is as follows: S31: Common Feature Extraction Branch; The common feature extraction branch is composed of multiple stacked residual modules, input... Output modality-independent scene structure features As shown in the following formula: ; In the formula, Indicates by A sequence consisting of residual blocks For diffusion time steps; S32: Feature-Specific Extraction Branch; This branch uses cascaded residual blocks and context-anchored attention modules to extract modality-specific features. As shown in the following formula: ; In the formula, That is, the attention module defined in step 2; S33: Dynamic Gated Aggregation; Generating a Spatial Weight Graph using a dynamic gating mechanism. and To achieve adaptive fusion and obtain the final output features As shown in the following formula: ; In the formula, the gating weights are obtained by convolution and softmax of the feature maps: ; This indicates channel splicing.

[0012] As a preferred option, in step 4, the process of designing a composite loss function to guide the model in performing the inverse noise reduction process is as follows: S41: Bidirectional structural similarity loss; Constructing a bidirectional structural similarity loss function. Constrained fusion of images With infrared source image Visible light source image The structural consistency between them is shown in the following formula: ; In the formula, It is a structural similarity index; S42: Gradient loss and intensity loss; Gradient loss is defined according to the following formula. : ; Strength loss is defined according to the following formula. : ; S43: Total loss function; combined with bidirectional structural similarity loss. gradient loss and intensity loss Construct the total loss function As shown in the following formula: ; In the formula, , and These are the balancing hyperparameters for bidirectional structural similarity loss, gradient loss, and intensity loss, respectively. The noise prediction network parameters are optimized through backpropagation, and the fused image is reconstructed by performing an inverse denoising process. .

[0013] As a preferred embodiment, in step 5, the process of using an object detection network to perform detection based on the reconstructed fused image is as follows: S51: Object detection; Input the fused image into the pre-trained YOLOv8 detection model, output the detection results, and calculate the mean accuracy as an evaluation metric to evaluate the object detection performance; S52: Status determination; Determine the operating status of the production environment or monitoring scenario based on the detection results.

[0014] As a preferred option, the state determination process in S52 is as follows: If a pedestrian is detected entering a restricted area, an alarm is triggered; if the target detection confidence is lower than a set threshold, an alert is issued indicating that the lighting is too bright or obstructed and that supplemental lighting is needed. At the same time, long-term monitoring results are statistically analyzed to assess the effective coverage of the monitoring system.

[0015] Compared with the prior art, the present invention has the following technical advantages: 1. A diffusion model and context-anchored attention (CAA) are introduced to improve the ability to preserve texture and edge in constrained scenes. The diffusion model is used for cross-modal feature extraction for the first time, and a CAA module is embedded in the noise prediction network. Long-distance spatial dependencies are captured through horizontal and vertical depth-separable convolutions to generate attention weight modulation features, which effectively enhances the edge and texture response of infrared and visible light images in constrained scenes such as low light, haze, and occlusion, and avoids the problems of blurred details or loss of thermal information in traditional methods.

[0016] 2. A dual-branch feature decoupling and adaptive fusion module (DFM) was designed to separate and dynamically aggregate modal common and unique features. A parallel dual-branch structure was constructed, in which the common branch extracts modality-independent scene structural features, while the unique branch extracts modality-specific detailed features through cascaded residuals and CAA. Simultaneously, a spatial weight map is generated using a dynamic gating mechanism to adaptively adjust the contribution ratio of common / unique features. This design effectively suppresses redundant interference between modalities, preserves complementary information, and achieves a fusion effect superior to simple superposition with fixed weights. 3. A composite loss function (bidirectional SSIM, gradient loss, and intensity loss) was constructed to effectively guide the high-quality reconstruction process of backdiffusion. Bidirectional SSIM was used to constrain the structural consistency between the fused image and the infrared and visible light source images. The gradient loss adopted a maximum value guidance strategy to maintain significant edges. The intensity loss controlled the brightness distribution. By weighting and combining the three, the problems of excessively smooth texture or thermal information attenuation that may occur in backsampling of the diffusion model were solved, and the reconstructed image has both high contrast and rich details.

[0017] 4. An end-to-end closed loop for fusion and detection is formed, improving target detection accuracy and system practicality in constrained scenarios. The reconstructed and fused image is directly input into the YOLOv8 detection network, evaluated using mAP quantization, and the status of the monitoring scene is determined based on the detection results (number of targets, confidence level) (such as intrusion alarm, insufficient lighting warning). This integrated solution effectively solves the problem of low detection rate of single modalities under dark or occluded conditions, providing a feasible technical path for constrained scenarios such as intelligent security and autonomous driving.

[0018] 5. The model has good generalization and engineering adaptability; the number of steps in the forward noise addition and reverse noise reduction process of the diffusion model can be flexibly adjusted; the number of parameters in the DFM module is controllable; the hyperparameters of the composite loss are easy to tune; and the whole process does not rely on additional labeled data (fused into unsupervised / self-supervised data), which is convenient for rapid deployment in different restricted scenarios (low illumination, rain and fog, heat cross, etc.).

[0019] This method innovatively integrates context-anchored attention, a two-branch decoupling gating mechanism, and a composite loss function within the diffusion model framework, significantly improving the fusion quality of infrared and visible light images in constrained scenarios and effectively supporting downstream target detection tasks. Compared with existing deep learning fusion methods, it has significant advantages in detail preservation, modal complementarity utilization, and end-to-end practicality. Attached Figure Description

[0020] Figure 1 This is an overall framework diagram of the multimodal constrained scene image fusion and detection method in this invention; Figure 2 This is a schematic diagram of the context-anchored attention (CAA) mechanism in this invention; Figure 3 This is an overall structural diagram of the cross-modal feature decoupling and adaptive fusion module (DFM) in this invention; Figure 4 This invention compares the performance of fused images with other fusion models in low-light environments. Figure 1 .

[0021] Figure 5 This invention compares the performance of fused images with other fusion models in low-light environments. Figure 2 . Detailed Implementation

[0022] To further illustrate the technical means and effects of the present invention in achieving its intended purpose, the following detailed description of the specific implementation methods, structures, features, and effects of the present invention, in conjunction with the accompanying drawings and preferred embodiments, is provided below.

[0023] like Figures 1 to 5 As shown, this invention provides a method for multimodal image fusion and detection in constrained scenes based on dual-branch diffusion recombination, comprising the following steps: Step 1: Construct a cross-modal feature extraction module based on an improved diffusion model; By acquiring infrared and visible light images in confined scenarios and stacking them into multi-channel inputs, a forward diffusion noise addition process is performed to provide noisy samples for subsequent noise prediction networks. As a preferred option, the process of constructing a cross-modal feature extraction module based on an improved diffusion model is as follows: S11: Image Acquisition and Channel Stacking; Acquire infrared images (IR single channel) and visible light images (R, G, B three channels) under limited scenarios (low light, haze, occlusion, etc.), and stack the visible light three channels and the infrared single channel along the channel dimension to form a 4-channel input sample. This provides multi-channel input samples; S12: Forward diffusion noise addition process; for the initial sample Gaussian noise was systematically added to obtain the time step. Noisy images at different times ; ; In the formula, , This represents the total number of diffusion steps (usually taken as 1000). , , For predefined variance scheduling (such as linear or cosine scheduling); Standard Gaussian noise; In this technical solution, the three visible light channels and the single infrared channel are stacked along the channel dimension to form a four-channel input sample, which makes full use of the complementarity between infrared thermal radiation information and visible light texture details. At the same time, a forward diffusion process is used to systematically add Gaussian noise to the initial sample, gradually transforming the original data into a noise distribution. This provides a probabilistic modeling foundation for the subsequent noise prediction network to learn cross-modal features, and enhances the robustness to noise and degradation in limited scenarios such as low light, haze, and occlusion. Meanwhile, the progressive noise addition strategy of the diffusion framework helps the model learn the ability to align cross-modal features from coarse to fine.

[0024] Step 2: Introduce Context Anchored Attention (CAA) mechanism into the Noise Prediction Network (U-Net); A context-anchored attention module is embedded in the noise prediction network. Long-range dependencies are captured through pooling and depthwise separable convolution, generating attention weight modulation features to dynamically enhance the responsiveness to texture details and edge features. As a preferred approach, the process of introducing a context-anchored attention mechanism into a noise prediction network is as follows: S21: Local region feature extraction; Input feature map First, global average pooling is used to obtain After Convolution yields local region features As shown in the following formula: ; S22: Long-range spatially dependent capture; Depthwise separable convolution is used, based on horizontal convolution kernels. With vertical convolution kernel Scanning in the width and height directions generates attention weights. ; Specific steps: First, […] Perform horizontal depthwise convolution, then vertical depthwise convolution, and finally sigmoid activation to obtain attention weights. .

[0025] S23: Feature modulation; The attention weights are multiplied element-wise with the original feature map to obtain the output features. As shown in the following formula: ; In the formula, This indicates element-wise multiplication; In this technical solution, local region features are extracted through global average pooling and 1×1 convolution, and depthwise separable convolutions in the horizontal and vertical directions are used to efficiently capture long-distance spatial dependencies, generating attention weights. Finally, the original features are modulated element-wise. This context-anchored attention mechanism enhances the model's responsiveness to texture details and edge features while significantly reducing the computational complexity of traditional self-attention mechanisms, effectively improving the accuracy and efficiency of cross-modal feature extraction in constrained scenarios.

[0026] Step 3: Design and train the cross-modal feature decoupling and adaptive fusion module (DFM); A parallel dual-branch structure is constructed, with a common branch extracting modality-independent scene structural features and a specific branch extracting modality-specific detailed features. The features are then adaptively aggregated through a dynamic gating mechanism to achieve efficient feature fusion. As a preferred option, the process of designing and training the cross-modal feature decoupling and adaptive fusion module is as follows: S31: Common Feature Extraction Branch; The common feature extraction branch is composed of multiple residual modules (ResNet-Block) stacked together, input... (i.e., the output in step 2) (or features from the previous layer), outputting modality-independent scene structure features. As shown in the following formula: ; In the formula, Indicates by A sequence consisting of residual blocks This is the diffusion time step (used for conditional diffusion). S32: Specific Feature Extraction Branch; This branch employs cascaded residual blocks and context-anchored attention (CAA) modules to extract modality-specific features such as texture information. As shown in the following formula: ; In the formula, That is, the attention module defined in step 2; S33: Dynamic Gated Aggregation; Generating a Spatial Weight Graph using a dynamic gating mechanism. and (The sum of the two is 1), achieving adaptive fusion and obtaining the final output feature. As shown in the following formula: ; In the formula, the gating weights are obtained by convolution and softmax of the feature maps: ; Indicates channel splicing; This technical solution constructs a parallel dual-branch structure. The common branch utilizes a residual module to extract modality-independent scene structural features, while the specific branch employs a cascaded residual and context-anchored attention (CAA) module to extract modality-specific texture and other detailed features. A dynamic gating mechanism is introduced to generate a spatial weight map for adaptive aggregation of the two types of features, achieving explicit decoupling and complementary fusion of cross-modal information. This method effectively suppresses intermodal redundant interference, preserves the advantages of both infrared thermal radiation and visible light texture, and enhances the model's adaptability to noise levels by introducing conditions at the diffusion time step, significantly improving the structural integrity and detail clarity of the fused image.

[0027] Step 4: Design a composite loss function to guide the model to perform the reverse noise reduction process; A composite loss function is constructed, including bidirectional structural similarity loss, gradient loss, and intensity loss. The noise prediction network is optimized through backpropagation, thereby reconstructing a high-quality fused image. As a preferred approach, the process of designing a composite loss function to guide the model in performing the inverse noise reduction process is as follows: S41: Bidirectional Structural Similarity Loss; Constructing Bidirectional Structural Similarity (SSIM) Loss Constrained fusion of images With infrared source image Visible light source image The structural consistency between them is shown in the following formula: ; In the formula, This is a structural similarity index, with values ​​ranging from [0,1]. S42: Gradient loss and intensity loss; Gradient loss (Multi-scale Contrastive Gradient) is used to preserve edge sharpness, and the gradient loss is defined according to the following formula. : ; Strength loss (Multi-scale Contrastive Intensity) is used to maintain consistency between luminance and thermal radiation information, and the intensity loss is defined according to the following formula. : ; As an alternative, L2 loss can also be used; S43: Total loss function; combined with bidirectional structural similarity loss. gradient loss and intensity loss Construct the total loss function This ensures that the fused image retains high color fidelity while preserving high-frequency details, as shown in the following formula: ; In the formula, , and These are the balancing hyperparameters for bidirectional structural similarity loss, gradient loss, and intensity loss, respectively. The noise prediction network parameters are optimized through backpropagation, and the inverse denoising process (DDPM sampling) is performed to reconstruct the fused image. .

[0028] In this technical solution, a composite total loss function is formed to guide backdiffusion reconstruction by constructing a bidirectional structural similarity loss (simultaneously constraining the structural consistency between the fused image and infrared and visible light source images), a gradient loss (using a maximum value guidance strategy to preserve significant edges), and an intensity loss (using a maximum value guidance strategy to preserve thermal radiation and brightness information). This design effectively solves the problems of overly smoothed textures or attenuated thermal information that may occur in backsampling of diffusion models, enabling the fused image to maintain high-frequency details while possessing high color fidelity and structural integrity, significantly improving the perceptual quality of the fused image in constrained scenarios and its applicability to downstream tasks.

[0029] Step 5: Based on the reconstructed fused image, perform object detection using a target detection network; The reconstructed high-quality fused image is input into a pre-trained target detection model to obtain the detection accuracy, and the running status of the actual scene is determined based on the detection results.

[0030] As a preferred option, the process of using an object detection network to perform detection based on the reconstructed fused image is as follows: S51: Object detection; Input the fused image into the pre-trained YOLOv8 detection model, output the detection results, and calculate the mean average precision (mAP) as the evaluation metric to evaluate the detection performance of targets such as pedestrians and vehicles; S52: Status determination; Determine the operating status of the production environment or monitoring scenario based on the detection results (such as the number of targets, confidence level, and location).

[0031] Specifically, the following measures are taken: if a pedestrian is detected entering a restricted area, an alarm is triggered; if the target detection confidence is below the threshold, it indicates that the lighting or occlusion is severe and supplementary lighting needs to be enhanced; long-term detection results are statistically analyzed to evaluate the effective coverage of the monitoring system.

[0032] In this technical solution, the fused and reconstructed images are input into a pre-trained YOLOv8 detection model. The detection performance of targets such as pedestrians and vehicles is quantitatively evaluated using the mean average accuracy (mAP). Based on the detection results (target quantity, confidence level, and location), intelligent status determination is achieved: triggering alarms in prohibited areas, providing supplementary lighting for low-confidence warnings, and evaluating monitoring coverage through long-term statistical analysis. This design forms an integrated closed loop of "fusion-detection-decision," effectively solving the problem of insufficient detection capabilities of single modalities in limited scenarios such as low light and occlusion, and significantly improving the accuracy of target detection and the intelligence level of the monitoring system.

[0033] The technical solution of the present invention will be described in detail below with reference to the embodiments, and its feasibility will be verified.

[0034] Example: In constrained environments such as coal mines and nighttime surveillance, limitations imposed by the environment and sensor conditions often result in low-quality visible light images with a lack of effective information, while infrared images fail to identify the texture details of objects. This invention addresses the challenge of effectively integrating multimodal image information in constrained environments by utilizing cross-modal feature decoupling and context-anchored attention mechanisms. This allows for more effective extraction of shared structures and modality-specific features, resulting in more accurate and robust generation of high-quality fused images for target detection.

[0035] 1) Experimental Setup and Offline State Evaluation Model Training: All experiments in this embodiment were conducted on a workstation equipped with a 13th generation Intel Core i9-13900K processor, an NVIDIA GeForce RTX 4080 Super GPU, and 64GB of RAM. The LLVIP dataset, collected under low-light conditions, was used, selecting 5000 pairs of infrared and visible light images from this dataset, covering pedestrian targets in 26 scenes, to comprehensively evaluate the performance of the algorithm. For evaluation metrics, six quantitative metrics were used: mutual information (MI), visual information fidelity (VIF), spatial frequency (SF), edge preservation quality (Qabf), standard deviation (SD), and color difference (Delta E). Except for a lower Delta E value, which indicates better color fidelity, higher values ​​for the other metrics indicate better fusion results.

[0036] 2) Online testing of fused images based on dual-branch diffusion recombination: A schematic diagram of the structure of the model of this invention is shown below. Figures 1 to 3As shown, this includes a cross-modal feature decoupling and adaptive fusion module (DFM) and a context-anchored attention mechanism (CAA). Figure 4 and Figure 5 These figures represent a qualitative comparison of the fusion performance of the model of this invention and several advanced fusion models in low-light-constrained scenarios and underground coal mine scenarios. (Combined with...) Figure 4 and Figure 5 As can be seen, this invention can aggregate multimodal information while preserving color texture and key image features. For road icons, tree branch textures, and light-dark boundaries, this method better maintains the color distribution and structure of the source image, resulting in richer texture and layering, and is also superior in handling glare at headlights or traffic lights.

[0037] To further verify the quantitative results, the method in this application was quantitatively compared with existing mainstream fusion methods, and the results are shown in Table 1.

[0038] Table 1: Comparison of Image Fusion Metrics Based on the LLVIP Dataset As shown in Table 1, the measured data of this invention significantly improves mutual information (MI) and visual information fidelity (VIF), and has the lowest color difference (Delta E). This indicates that by using cross-modal decoupling and bidirectional structural consistency loss, this invention effectively alleviates the problems of feature conflict and information annihilation in deep networks.

[0039] 3) Validation of the effectiveness of image fusion for downstream object detection tasks. The ultimate goal of image fusion is to improve the performance of subsequent computer vision tasks under constrained environments. In this embodiment, the downstream object detector YOLOv8 is used as an example. All models are trained independently without loading pre-trained weights. The mAP@0.5 metric is used to quantitatively evaluate the detection performance. The measured performance comparison of the fused images obtained by different models in the object detection task is shown in Table 2.

[0040] Table 2: Comparison of measured performance of different input types in object detection tasks As shown in Table 2, among various advanced fusion algorithms, the method of this invention achieved the highest mAP@0.5 of 97.5%, significantly outperforming other comparative methods. Compared to the single visible light modality, this invention improves detection accuracy by 14%. This performance gain demonstrates that this invention not only generates visually superior images but also produces inputs that are more user-friendly for downstream machine perception tasks and have more distinctive features.

[0041] The above experimental examples demonstrate the significant effectiveness of this invention. By implementing high-quality image fusion in constrained environments, it can provide crucial decision-making and front-end data support for downstream perception tasks such as intelligent coal mine monitoring and nighttime autonomous driving.

[0042] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention in any way. Although the present invention has been disclosed above with reference to preferred embodiments, it is not intended to limit the present invention. Any person skilled in the art can make some modifications or alterations to the above-disclosed technical content to create equivalent embodiments without departing from the scope of the present invention. Any simple modifications, equivalent changes, and alterations made to the above embodiments based on the technical essence of the present invention without departing from the scope of the present invention shall still fall within the scope of the present invention.

Claims

1. A method for multimodal image fusion and detection in confined scenes based on bi-branch diffusion recombination, characterized in that, Includes the following steps: Step 1: Construct a cross-modal feature extraction module based on an improved diffusion model; By acquiring infrared and visible light images in confined scenarios and stacking them into multi-channel inputs, a forward diffusion noise addition process is performed to provide noisy samples for subsequent noise prediction networks. Step 2: Introduce a context-anchored attention mechanism into the noise prediction network; A context-anchored attention module is embedded in the noise prediction network. Long-range dependencies are captured through pooling and depthwise separable convolutions, generating attention weight modulation features to enhance the response to texture details and edge features. Step 3: Design and train the cross-modal feature decoupling and adaptive fusion module; A parallel dual-branch structure is constructed, with a common branch extracting modality-independent scene structural features and a specific branch extracting modality-specific detailed features. The features are then adaptively aggregated through a dynamic gating mechanism to achieve efficient feature fusion. Step 4: Design a composite loss function to guide the model to perform the reverse noise reduction process; A composite loss function is constructed, including bidirectional structural similarity loss, gradient loss, and intensity loss. The noise prediction network is optimized through backpropagation, thereby reconstructing a high-quality fused image. Step 5: Based on the reconstructed fused image, perform object detection using a target detection network; The reconstructed high-quality fused image is input into a pre-trained target detection model to obtain the detection accuracy, and the running status of the actual scene is determined based on the detection results.

2. The method for multimodal image fusion and detection in confined scenes based on bi-branch diffusion recombination according to claim 1, characterized in that, In step 1, the process of constructing the cross-modal feature extraction module based on the improved diffusion model is as follows: S11: Image Acquisition and Channel Stacking; Acquire infrared and visible light images under limited conditions, and stack the three visible light channels and the single infrared channel along the channel dimension to form a 4-channel input sample. ; S12: Forward diffusion noise addition process; for the initial sample Add Gaussian noise to obtain the time step. Noisy images at different times ; ； In the formula, , This represents the total number of diffusion steps. , , For predefined variance scheduling; It is standard Gaussian noise.

3. The method for multimodal image fusion and detection in confined scenes based on bi-branch diffusion recombination according to claim 1, characterized in that, In step 2, the process of introducing the context-anchored attention mechanism into the noise prediction network is as follows: S21: Local region feature extraction; Input feature map First, global average pooling is used to obtain After Convolution yields local region features As shown in the following formula: ； S22: Long-range spatially dependent capture; Depthwise separable convolution is used, based on horizontal convolution kernels. With vertical convolution kernel Scanning in the width and height directions generates attention weights. ; S23: Feature modulation; The attention weights are multiplied element-wise with the original feature map to obtain the output features. As shown in the following formula: ； In the formula, This indicates element-wise multiplication.

4. The method for multimodal image fusion and detection in confined scenes based on bi-branch diffusion recombination according to claim 3, characterized in that, In S22, the long-range spatial dependency capture process is as follows: right Perform horizontal depthwise convolution, then vertical depthwise convolution, and finally sigmoid activation to obtain attention weights. .

5. The method for multimodal image fusion and detection in confined scenes based on bi-branch diffusion recombination according to claim 1, characterized in that, In step 3, the process of designing and training the cross-modal feature decoupling and adaptive fusion module is as follows: S31: Common feature extraction branch; The common feature extraction branch is composed of multiple residual modules stacked together, input... Output modality-independent scene structure features As shown in the following formula: ； In the formula, Indicates by A sequence consisting of residual blocks For diffusion time steps; S32: Specific feature extraction branch; The unique branch employs cascaded residual blocks and a context-anchored attention module to extract modality-specific features. As shown in the following formula: ； In the formula, That is, the attention module defined in step 2; S33: Dynamic Gated Aggregation; Generating a Spatial Weight Graph using a dynamic gating mechanism. and To achieve adaptive fusion and obtain the final output features As shown in the following formula: ； In the formula, the gating weights are obtained by convolution and softmax of the feature maps: ; This indicates channel splicing.

6. The method for multimodal image fusion and detection in confined scenes based on bi-branch diffusion recombination according to claim 1, characterized in that, In step 4, the process of designing a composite loss function to guide the model in performing the inverse noise reduction process is as follows: S41: Bidirectional structural similarity loss; Constructing a bidirectional structural similarity loss function. Constrained fusion of images With infrared source image Visible light source image The structural consistency between them is shown in the following formula: ； In the formula, It is a structural similarity index; S42: Gradient loss and intensity loss; Gradient loss is defined according to the following formula. : ； Strength loss is defined according to the following formula. : ； S43: Total loss function; combined with bidirectional structural similarity loss. gradient loss and intensity loss Construct the total loss function As shown in the following formula: ； In the formula, , and These are the balancing hyperparameters for bidirectional structural similarity loss, gradient loss, and intensity loss, respectively. The noise prediction network parameters are optimized through backpropagation, and the fused image is reconstructed by performing an inverse denoising process. .

7. The method for multimodal image fusion and detection in confined scenes based on bi-branch diffusion recombination according to claim 1, characterized in that, In step 5, the process of using an object detection network to perform detection based on the reconstructed fused image is as follows: S51: Object detection; Input the fused image into the pre-trained YOLOv8 detection model, output the detection results, and calculate the mean accuracy as an evaluation metric to evaluate the object detection performance; S52: Status determination; Determine the operating status of the production environment or monitoring scenario based on the detection results.

8. The method for multimodal image fusion and detection in confined scenes based on bi-branch diffusion recombination according to claim 7, characterized in that, In S52, the state determination process is as follows: If a pedestrian is detected entering a restricted area, an alarm is triggered; if the target detection confidence is lower than a set threshold, an alert is issued indicating that the lighting is too bright or obstructed and that supplemental lighting is needed. At the same time, long-term monitoring results are statistically analyzed to assess the effective coverage of the monitoring system.

Images