Dual-optical image fusion method based on asymmetric high-frequency injection and gradient-aware constraint

Abstract

The application discloses a dual-optical image fusion method based on asymmetric high-frequency injection and gradient perception constraint, and specifically comprises the following steps: step one: acquiring and pre-processing infrared source images and visible light source images collected synchronously to obtain infrared images and visible light images; step two: constructing a dual-optical image fusion network, wherein the dual-optical image fusion network comprises an encoder and a decoder, the encoder performs feature extraction and outputs deep representation features, and the decoder decodes and reconstructs the deep representation features to generate dual-optical fusion images; step three: constructing a composite loss function comprising gradient perception constraint for training; and inputting the preprocessed infrared images and visible light images into the dual-optical image fusion network to output dual-optical fusion images. Through the construction of the dual-optical image fusion network and the composite loss function comprising gradient perception constraint, the application can realize efficient fusion of infrared thermal target features and visible light structural texture features, and enhance edge retention and high-frequency detail expression capability.

Description

Technical Field

[0001] This invention pertains to the field of image processing technology, specifically relating to a dual-light image fusion method based on asymmetric high-frequency injection and gradient-aware constraints. Background Technology

[0002] In industrial site safety monitoring, worker identification, abnormal behavior detection, and hazardous area early warning are directly related to production safety and on-site management effectiveness. Real-world industrial scenarios typically feature low light conditions at night, smoke and dust, partial obstructions, complex backgrounds, and limited bandwidth, making it difficult for traditional monitoring methods to reliably achieve all-weather target perception and continuous early warning. Especially in scenarios such as factory roads, construction areas, and equipment operating areas, monitoring systems not only need to identify worker targets but also maintain good real-time performance and accuracy in complex environments.

[0003] Current industrial monitoring largely relies on single-modal image analysis. While visible light images alone can provide relatively clear scene textures, equipment outlines, and personnel appearance information, their imaging quality is heavily dependent on ambient lighting. In conditions such as nighttime, backlighting, or smoke / dust obstruction, issues like target loss, blurred edges, and decreased recognition accuracy are common. Infrared images alone, while highlighting thermal targets in low light and complex environments, suffer from low spatial resolution and insufficient texture information, making it difficult to accurately reflect background structure and environmental layers. Furthermore, they are susceptible to interference from equipment heat dissipation and thermal reflection against complex heat source backgrounds. Therefore, single-modal monitoring methods cannot simultaneously meet the requirements of industrial sites for all-weather perception, clear target representation, and stable recognition.

[0004] Based on the aforementioned issues, infrared and visible light dual-light fusion, as a technical approach that balances the saliency of thermal targets and the readability of background structures, has gradually become an important development direction for industrial site safety monitoring. Existing dual-light fusion methods mainly include traditional fixed-rule methods, shallow learning methods, and deep learning methods. Among these, traditional methods have poor scene adaptability, shallow learning methods have limited ability to preserve complex backgrounds and local details, and while deep learning methods have significantly improved fusion quality, they still have shortcomings in specific industrial site applications. In particular, advanced fusion methods represented by CDDFuse in recent years have proven the effectiveness of dual-light fusion based on deep feature decomposition and recombination; however, in the specific scenario of industrial site safety monitoring, these methods are still prone to problems such as smooth target edges, loss of high-frequency details, high model complexity, and insufficient integration with downstream detection tasks.

[0005] Therefore, based on the existing advanced dual-light fusion technology, it is necessary to propose an improved infrared and visible light image fusion method to address key issues in industrial site safety monitoring scenarios, such as edge preservation, lightweight deployment, and fusion results serving downstream identification tasks, in order to improve the monitoring accuracy, real-time performance, and engineering application effectiveness in complex industrial environments. Summary of the Invention

[0006] The purpose of this invention is to provide a two-light image fusion method based on asymmetric high-frequency injection and gradient-aware constraints. By constructing a two-light image fusion network and a composite loss function based on gradient-aware constraints, this invention can achieve efficient fusion of infrared thermal target features and visible light structural texture features while maintaining the naturalness of the visible light background structure and the integrity of the scene texture, thereby enhancing the edge preservation and high-frequency detail expression capabilities of the fused image.

[0007] The technical solution of this invention: a two-light image fusion method based on asymmetric high-frequency injection and gradient-aware constraints, specifically including the following steps: Step 1: Acquire and preprocess the synchronously acquired infrared source image and visible light source image to obtain the infrared image and visible light image; Step 2: Construct a dual-light image fusion network, which includes an encoder and a decoder. The encoder extracts features from the infrared image and the visible light image respectively, and outputs deep representation features. The decoder adopts an asymmetric high-frequency injection structure to decode and reconstruct the deep representation features. Step 3: Construct a composite loss function that includes gradient-aware constraints, and train the dual-light image fusion network based on the composite loss function to obtain the trained dual-light image fusion network; Step 4: Input the preprocessed infrared image and visible light image into the dual-light image fusion network, and output the dual-light fused image.

[0008] In the aforementioned dual-light image fusion method based on asymmetric high-frequency injection and gradient-aware constraints, the preprocessing includes size alignment, grayscale normalization, and basic registration processing.

[0009] In the aforementioned dual-light image fusion method based on asymmetric high-frequency injection and gradient-aware constraints, the encoder employs a lightweight twin encoder with weight sharing. When performing feature extraction, the encoder specifically performs the following operations: The encoder maps the infrared image or the visible light image into an input feature map through shallow convolution. The input feature map is then subjected to layer normalization, multi-head self-attention calculation, weighted by a projection matrix, and connected to the residual of the input feature map to obtain an intermediate feature map. The obtained feature map is then subjected to layer normalization again, transformed by a gated feedforward network, weighted by a projection matrix, and connected to the residual again to output the deep representation feature.

[0010] In the aforementioned two-light image fusion method based on asymmetric high-frequency injection and gradient-aware constraints, the intermediate feature map The calculation formula is as follows: ; in, For the input feature map, Presentation layer normalization operation, This indicates multi-head attention calculation. To learn the projection matrix; The deep characterization features The calculation formula is as follows: ; in, Indicates gated feedforward transform. To learn the projection matrix.

[0011] In the aforementioned dual-light image fusion method based on asymmetric high-frequency injection and gradient-aware constraints, the encoder further includes a convolutional block attention module composed of a channel attention submodule and a spatial attention submodule. The convolutional block attention module is connected in series in the intermediate extraction layer of the encoder to enhance the expressive power of the target region and the texture region.

[0012] In the aforementioned dual-light image fusion method based on asymmetric high-frequency injection and gradient-aware constraints, the asymmetric high-frequency injection structure includes a visible light structure preservation path and an infrared high-frequency texture enhancement path; the visible light structure preservation path transmits the visible light image... The output fusion terminal of the decoder is directly introduced to maintain the global background brightness and structural information of the fused image; the infrared high-frequency texture enhancement path enhances the infrared image... Mean filtering is performed to extract low-frequency background components. And through infrared images and low-frequency background components Extracting high-frequency detail components by subtraction .

[0013] In the aforementioned two-light image fusion method based on asymmetric high-frequency injection and gradient-aware constraints, the extraction of low-frequency background components... The formula is as follows: ; in, This indicates an average pooling or mean filtering operation. Represents an infrared image; The formula for extracting high-frequency detail components from infrared images is as follows: ; in, This represents high-frequency detail components.

[0014] In the aforementioned two-light image fusion method based on asymmetric high-frequency injection and gradient-aware constraints, the calculation formula for the two-light fused image is as follows: ; in, For dual-light fusion images, This represents the decoder reconstruction function. This represents the deep representation features of the encoder output. This represents the Sigmoid activation function. Visible light image, Gain factor to enhance infrared detail This represents high-frequency detail components.

[0015] In the aforementioned two-light image fusion method based on asymmetric high-frequency injection and gradient-aware constraints, the composite loss function It consists of three parts: strength loss, structural loss, and gradient loss, specifically expressed as follows: ; ; in, , and This represents the weighting coefficient of each loss term. For mean square error loss, For structural similarity loss, For gradient operators, To fuse the gradient magnitudes of the images, This represents the maximum value of the pixel-by-pixel gradient amplitude in both the infrared and visible light images. It is an L1 norm.

[0016] In the aforementioned dual-light image fusion method based on asymmetric high-frequency injection and gradient-aware constraints, the training step adopts a two-stage training strategy: the first stage freezes the parameters of the fusion layer, the visible light structure preservation path, and the infrared high-frequency texture enhancement path, and performs self-reconstruction training on the infrared image and the visible light image respectively to initialize the encoder and decoder parameters; the second stage unfreezes all network parameters, inputs pairs of infrared images and visible light images, and uses the composite loss function to jointly fine-tune the dual-light image fusion network.

[0017] Compared with the prior art, the present invention has the following beneficial effects: This invention constructs a dual-light image fusion network equipped with gradient-aware constraints. While fully preserving the naturalness of the background structure and the integrity of the scene texture in visible light images, it accurately enhances the edge contours and high-frequency details of infrared thermal targets. This effectively solves the problems of blurred target contours, weakened edges, and distorted background textures commonly found in existing fusion methods, achieving efficient collaborative fusion of infrared thermal target features and visible light structural texture features. This invention introduces a gradient-aware composite loss function to enhance the fused image's ability to retain significant edge information, effectively mitigating edge smoothing phenomena. This results in fusion results that not only possess excellent visual effects but also efficiently adapt to downstream target detection tasks, significantly improving the recognition accuracy of targets such as workers and vehicles at night and in complex industrial environments. Furthermore, this invention employs an asymmetric high-frequency injection structure decoder, achieving precise complementarity of dual-modal features through a differentiated dual-path design. The visible light structure preservation path directly introduces the visible light image, stably maintaining the global background brightness, scene structure, and texture information of the fused image, ensuring a natural and realistic background presentation. The infrared high-frequency texture enhancement path accurately extracts high-frequency detail components from the infrared image through mean filtering and subtraction processing, specifically enhancing the edges and local textures of thermal targets. Structurally, this avoids the feature neutralization and detail loss problems common in traditional symmetric decoders, improving the target clarity and detail representation of the fused image. Simultaneously, this invention uses a weight-sharing lightweight twin encoder and integrates a convolutional block attention module, significantly reducing the number of network parameters while ensuring feature extraction accuracy, making it more suitable for the lightweight deployment requirements of industrial edge devices. Attached Figure Description

[0018] Figure 1 This is a diagram of the overall architecture of the dual-light image fusion network of the present invention; Figure 2 This is a schematic diagram of the asymmetric high-frequency injection module structure of the present invention; Figure 3 This is a visual comparison of different fusion methods of the present invention in a nighttime scene; Figure 4 This is a schematic diagram of the edge-cloud collaborative processing framework for industrial site safety monitoring of the present invention; Figure 5 This is a diagram showing the downstream YOLO recognition and detection results of this invention. Detailed Implementation

[0019] The present invention will be further described below with reference to the accompanying drawings and embodiments, but this should not be construed as limiting the present invention.

[0020] Example 1: A two-light image fusion method based on asymmetric high-frequency injection and gradient-aware constraints, such as... Figure 1 As shown, the specific steps include: Step 1: Acquire and preprocess the synchronously acquired infrared source image and visible light source image to obtain the infrared image and visible light image; In this step, infrared images are acquired simultaneously by the dual-light cameras. and visible light images Preprocessing includes size alignment, grayscale normalization, and basic registration to reduce differences in resolution, viewing angle, and grayscale distribution between different sensors, so that infrared and visible light images meet the same input requirements.

[0021] Step 2: Construct a dual-light image fusion network, which includes an encoder and a decoder. The encoder extracts features from the infrared image and the visible light image respectively, and outputs deep representation features. The decoder adopts an asymmetric high-frequency injection structure to decode and reconstruct the deep representation features. In this step, during the feature extraction stage, the present invention employs a lightweight twin encoder structure with weight sharing to perform multi-scale feature extraction on infrared and visible light images respectively. Compared with traditional independent dual-branch encoders, the weight sharing method can significantly reduce the number of parameters, making it more suitable for deployment in industrial edge devices. The encoder backbone preferably adopts a lightweight Restormer structure, which extracts dual-modal deep representation features through shallow convolutional mapping, multi-head self-attention mechanism, and gated feedforward network.

[0022] Specifically, the encoder performs the following operations when performing feature extraction: The encoder maps the infrared image or the visible light image to an input feature map through shallow convolution. The input feature map is then normalized, multi-head self-attention is performed, and the result is weighted by a projection matrix and concatenated with the residual of the input feature map to obtain an intermediate feature map. The calculation formula is as follows: ; in, For the input feature map, Presentation layer normalization operation, This indicates multi-head attention calculation. To learn the projection matrix.

[0023] The deep characterization features The calculation formula is as follows: ; in, Indicates gated feedforward transform. To learn the projection matrix.

[0024] Then, the obtained features are subjected to layer normalization again, transformed by a gated feedforward network, weighted by a projection matrix, and reconnected with residuals to output the deep representation features.

[0025] This invention, through a lightweight twin encoder structure with weight sharing, can extract thermal target saliency information from infrared images and background texture and structural information from visible light images while maintaining the lightweight nature of the dual-light image fusion network.

[0026] To further enhance the expressive power of target and key texture regions, this invention introduces a Convolutional Block Attention (CBAM) module into the encoder. This module consists of a channel attention submodule and a spatial attention submodule connected in series. It is used to highlight the prominent channels where infrared bright targets are located and the key spatial regions with rich textures and clear edges, thereby suppressing background noise interference and improving the joint expressive power of dual-modal features for the contours of workers and environmental structures.

[0027] To address the issues of smooth target edges and loss of high-frequency details that often occur in existing dual-light fusion methods in industrial scenarios, the decoder employs an asymmetric high-frequency injection structure. The visible light branch is primarily responsible for maintaining the background brightness, environmental structure, and scene texture of the fused image, while the infrared branch is mainly responsible for enhancing the high-frequency edges and local details of thermal targets. This differentiated processing approach using the asymmetric high-frequency injection structure avoids the feature neutralization and detail loss phenomena commonly found in traditional symmetrical reconstruction, making it more suitable for safety monitoring tasks of industrial field workers.

[0028] Specifically, the asymmetric high-frequency injection structure includes a visible light structure preservation path and an infrared high-frequency texture enhancement path; the visible light structure preservation path transmits the visible light image... The output fusion terminal of the decoder is directly introduced to maintain the global background brightness and structural information of the fused image; the infrared high-frequency texture enhancement path enhances the infrared image... Mean filtering is performed to extract low-frequency background components. Extracting low-frequency background components The formula is as follows: ; in, This indicates an average pooling or mean filtering operation. Represents an infrared image.

[0029] Then through infrared images and low-frequency background components Extracting high-frequency detail components by subtraction The formula for extracting high-frequency detail components (high-frequency texture and thermal edge information) from infrared images is as follows: ; in, This represents high-frequency detail components.

[0030] During the decoding and reconstruction stage, the deep representation features fused from the coding layer, the visible light structure-preserving pathway, and the infrared high-frequency enhancement pathway are all injected into the output to generate a dual-light fused image. The calculation formula is as follows: ; in, For dual-light fusion images, This represents the decoder reconstruction function. This represents the deep representation features of the encoder output. This represents the Sigmoid activation function. Visible light image, The gain factor for enhancing infrared detail is preferably set to 1.5; Represents high-frequency detail components. For example... Figure 2 As shown, through this asymmetric high-frequency injection mechanism, the present invention can significantly enhance the outline edges and local high-frequency details of the worker while maintaining the naturalness of the visible light background structure.

[0031] Step 3: Construct a composite loss function that includes gradient-aware constraints, and train the dual-light image fusion network based on the composite loss function to obtain the trained dual-light image fusion network; In this step, to avoid the fused image becoming overly smooth due to traditional pixel-level reconstruction loss, this invention introduces gradient-aware constraints during the training phase to construct a composite loss function. The fused image should not only maintain consistency with the source image in pixel intensity and local structure, but also retain as much of the most significant edge gradient information as possible from the infrared and visible light images. First, the gradient magnitudes of the source images are compared pixel-by-pixel, and the maximum value is used to construct a significant gradient map. The composite loss function... It consists of three parts: strength loss, structural loss, and gradient loss, specifically expressed as follows: ; ; in, , and Represents the weighting coefficients of each loss term, preferably... , and ; This is the mean square error loss, used to ensure consistency of constraint strength. This is a structural similarity loss, used to preserve local structural similarity; Used to enhance the ability of fused images to preserve significant edge information; For gradient operators, the Sobel operator is preferred for computation; To fuse the gradient magnitudes of the images, This represents the maximum value of the pixel-by-pixel gradient amplitude in both the infrared and visible light images. It is an L1 norm. By introducing this composite loss function, the edge smoothing phenomenon in the fusion process can be effectively alleviated, making the outlines of workers and the boundaries of key structures clearer.

[0032] The training steps employ a two-stage training strategy: In the first stage, the parameters of the fusion layer, the visible light structure preservation pathway, and the infrared high-frequency texture enhancement pathway are frozen, and self-reconstruction training is performed on the infrared and visible light images respectively to initialize the encoder and decoder parameters, enabling the dual-light image fusion network to possess basic feature representation capabilities; In the second stage, all network parameters are unfrozen, paired infrared and visible light images are input, and the dual-light image fusion network is jointly fine-tuned using the composite loss function, enabling the dual-light image fusion network to learn to enhance the edge and detail information of the worker target while maintaining the naturalness of the background structure.

[0033] This invention employs a two-stage training strategy based on deep learning to train a dual-light image fusion network. The dual-light image fusion network is implemented in the PyTorch framework, runs in a GPU environment (CUDA), and uses a single-card training method.

[0034] In terms of training settings, the optimizer uses the Adam optimization algorithm, and each sub-module (encoder, decoder, basic feature fusion layer, and detail feature fusion layer) is optimized independently, with an initial learning rate of 1×10⁻. 4 The weight decay factor is 0. The total number of training epochs is 120, and the batch size is 8. The learning rate adopts the StepLR scheduling strategy, decaying to 0.5 times its original value every 20 epochs. To improve training stability, a gradient pruning strategy is used during backpropagation, with a pruning threshold of 0.01 (L2 norm).

[0035] In terms of data, the training data uses an infrared-visible light paired dataset, and the input data is stored in H5 format. During training, image patches of size 128×128 are cropped from the original images with a stride of 200, and the input images are normalized (pixel values ​​are scaled to the [0,1] range). The input to the dual-light image fusion network is a single-channel image, where the visible light image is taken from the luminance (Y) channel, and the infrared image is in grayscale format.

[0036] In terms of network structure, the encoder adopts a Transformer-based Restormer structure with 1 input channel, 64 feature dimensions, and two levels of Transformer Blocks, each containing 4 Transformer sub-modules. The multi-head self-attention mechanism has 8 heads, and the feedforward network expansion factor is 2. The encoder output is decomposed into basic features and detail features. The basic features are enhanced by a feature extraction module with a CBAM attention mechanism, while the detail features are modeled by a detail extraction module based on an invertible neural network (INN) structure. The basic feature fusion module uses a multi-head attention structure (8 heads), and the detail fusion module contains one invertible layer.

[0037] The decoder employs an improved Restormer structure, introducing a high-frequency information injection mechanism during feature reconstruction. Specifically, low-frequency and high-frequency information are separated by mean filtering (5×5 average pooling), and the high-frequency component is injected into the decoded features with a 1.5x weight during the fusion stage to enhance the ability to express image edges and texture details.

[0038] The training process is divided into two phases: The first stage is the reconstruction stage (the first 40 rounds), where self-reconstruction training is performed on visible light and infrared images respectively. The loss function consists of a weighted average of structural similarity loss (SSIM) and mean squared error loss (MSE), with SSIM having a weight of 5 and MSE having a weight of 1, and a weight coefficient of 20 applied to both visible light and infrared images. Furthermore, a gradient loss (based on the first-order spatial gradient, L1 norm) is introduced to enhance edge preservation capabilities, and an eigenvalue decomposition loss is introduced to constrain the correlation between basic and detailed features, with a weight coefficient of 2. Feature fusion and high-frequency injection are not performed in this stage to ensure the network has stable reconstruction capabilities.

[0039] The second stage is the fusion stage (last 80 rounds), where the dual-light image fusion network is trained to fuse infrared and visible light images. The fusion features are input into the fusion module after the basic features and detail features are added separately. The loss function consists of three parts: (1) Fusion Loss, which consists of pixel-level maximum response constraint (max(VIS, IR)) and gradient preservation term, where the gradient is calculated by the Sobel operator; (2) Feature decomposition loss, with a weight coefficient of 2; (3) Max-Gradient Loss, which is used to constrain the gradient of the fused image to be close to the pixel-wise maximum value of the gradient of the input image, with a weight coefficient of 20, in order to improve the edge sharpness and spatial frequency index of the fused image.

[0040] During training, to avoid excessive smoothing of high-frequency information, the total variational loss (TV Loss) is turned off and its weight is set to 0.

[0041] During the testing phase, for color visible light images, they are first converted to the YCrCb color space, and only the luminance channel (Y) is fused. The fused result is then recombined with the original chrominance channels (Cr, Cb) and converted back to the RGB space, thereby improving the quality of structural details while maintaining color information.

[0042] Step 4: Input the preprocessed infrared image and visible light image into the dual-light image fusion network, and output the dual-light fused image.

[0043] In this step, after the dual-light image fusion network is trained, infrared and visible light images are input at the edge, and a dual-light fused image is output. This fused image is then transmitted to the cloud and further input into the target detection model for downstream recognition and verification. Preferably, the detection model uses YOLOv5 to perform bounding box localization and category recognition for targets such as workers and vehicles. By directly using the dual-light fused image for downstream detection tasks, it can be verified that this invention not only improves the fusion quality at the visual level but also effectively enhances target detection performance in nighttime, low-light, and complex industrial environments.

[0044] This invention addresses safety monitoring applications for industrial field workers, constructing an edge-cloud collaborative processing framework of "sensor-end data acquisition, edge-end fusion, and cloud-based detection," such as... Figure 4 As shown, the sensing end simultaneously acquires infrared and visible light images through a fixed dual-light camera, the edge end completes dual-modal fusion, and the cloud end completes worker identification, safety analysis, and anomaly warning.

[0045] To verify the effectiveness of the method of this invention, experiments were conducted using the standard MSRS dataset. This dataset contains infrared and visible light image pairs under multiple illumination conditions and provides target annotation information for pedestrians and vehicles, which can effectively simulate nighttime monitoring scenarios such as industrial plants and construction areas. Experiments were conducted under uniform training set conditions, and the following metrics were used: information entropy (EN), standard deviation (SD), spatial frequency (SF), mutual information (MI), and visual information fidelity (VIF). And structural similarity (SSIM) is used as an evaluation index for fusion performance; at the same time, it adopts Pedestrian category AP is used as a downstream detection performance evaluation indicator.

[0046] DenseFuse is an early representative method for infrared and visible light image fusion based on deep learning. It adopts an encoder-decoder structure and introduces densely connected blocks to achieve multi-scale feature extraction. It reconstructs the fused image through a simple feature fusion strategy and decoder, and has the advantages of simple structure and easy training. However, the fusion strategy of this method is relatively basic and does not make sufficient use of the complementary information of infrared thermal targets and visible light textures. This can easily lead to the loss of high-frequency details in the fused image, making it difficult to meet the needs of industrial scenarios for clear expression of target details.

[0047] DIDFuse is an unsupervised image fusion method based on deep image decomposition. It decomposes the source image into low-frequency background features and high-frequency detail features through an autoencoder, performs feature fusion separately, and then reconstructs the image. It can effectively separate high and low frequency information to a certain extent and improve the target brightness performance of the fused image. However, this method is not adaptable to complex industrial scenarios, and the contrast and information retention of the fused image are generally poor, making it difficult to support the high-precision perception requirements of all-weather monitoring scenarios.

[0048] U2Fusion is a general unsupervised image fusion framework that can be adapted to various fusion tasks such as infrared / visible light fusion and multi-exposure fusion. It adaptively evaluates the importance of source image information through feature extraction and information measurement, without the need to design a special loss function for specific tasks, and has strong versatility. However, this method has not been specifically optimized for infrared and visible light fusion scenarios, and its ability to balance the prominence of thermal targets and the preservation of texture details is limited. The fusion effect in complex industrial environments needs to be improved.

[0049] SDNet is a squeeze-decomposition image fusion network that models the fusion task as the extraction and reconstruction of gradient texture information and intensity target information. By adjusting the gradient optimization target through adaptive decision blocks, it can balance real-time performance and versatility to a certain extent. However, the training of this method is highly dependent on data, and artifacts are prone to occur in complex industrial scenarios such as smoke and low light. The information retention of the fused image is low, making it difficult to guarantee the accuracy of subsequent detection tasks.

[0050] DeFusion is an unsupervised image fusion method based on self-supervised feature decomposition. It does not require paired training data or complex loss function constraints. It achieves image fusion by decomposing the source image into common features and unique features. It has the advantages of low data dependence and strong versatility. However, this method is prone to problems such as smooth edges of fused images and loss of high-frequency details. Its visual quality is not well adapted to downstream target detection tasks, making it difficult to meet the actual needs of industrial safety monitoring.

[0051] FusionGAN is an infrared and visible light image fusion method based on generative adversarial networks. Through adversarial training between the generator and the discriminator, the fused image is constrained to simultaneously contain infrared thermal target intensity information and visible light texture gradient information. The generated fused image has a relatively natural visual effect. However, this method has the defects of unstable GAN training and easy mode collapse. In some scenarios, the loss of detailed information is serious, and the model complexity is high, making it difficult to adapt to the lightweight deployment requirements of industrial sites.

[0052] like Figure 3 As shown, compared with existing methods such as DenseFuse, FusionGAN, U2Fusion, and DeFusion, the method of this invention can more clearly maintain the outline edge of the worker in complex nighttime scenes, while better preserving the texture of background leaves, roads, and environmental structures, significantly improving the problems of blurred target edges and overly smoothed background areas in traditional symmetrical fusion methods.

[0053] Table 1. Comparison of objective evaluation metrics for different fusion methods on the test set.

[0054] As shown in Table 1, the method of this invention achieved superior results in several objective evaluation indicators, including an information entropy (EN) of 6.6924, a standard deviation (SD) of 42.7786, a spatial frequency (SF) of 11.5683, a mutual information (MI) of 3.7867, a visual information fidelity (VIF) of 1.0666, and an edge correlation index of [missing data]. The result reached 0.7167. These results demonstrate that the present invention exhibits good overall performance in terms of information preservation, detail enhancement, and edge sharpness.

[0055] Table 2 Comparison of Nighttime Worker Recognition Performance Based on Cloud-based YOLOv5

[0056] Table 2 shows the overall detection performance of the cloud-based YOLOv5 detector when the fused image generated by the method of this invention is used as input. The accuracy rate reached 55.7%, with the pedestrian category AP reaching 69.0%, both superior to most comparative methods. This demonstrates that the method of this invention can not only improve the visual quality of fused images but also effectively enhance downstream detection performance in complex industrial scenarios.

[0057] like Figure 5As shown, compared to inputting infrared or visible light images alone, the fused image generated by the method of this invention not only preserves the background of the industrial site environment, but also enhances the outline edges of workers and vehicles, identifying more people and objects. This enables the cloud detector to maintain higher detection confidence and reduce missed detections and false detections at night and under complex conditions.

[0058] Table 3 shows that removing the CBAM module decreases the feature extraction capability and the mutual information (MI) index; removing the asymmetric high-frequency injection module decreases the fidelity of the fused image and the VIF index; removing the maximum gradient constraint decreases the structure preservation capability and the SSIM index. These results indicate that the saliency enhancement module, asymmetric high-frequency injection structure, and maximum gradient loss introduced in this invention all have a positive effect on improving the performance of industrial two-light fusion.

[0059] Table 3. Results of CBAM module ablation experiment

[0060] Furthermore, to evaluate the feasibility of deploying this invention on industrial edge devices, its model complexity and inference efficiency were analyzed. Thanks to the lightweight twin encoder structure employing weight sharing, the number of parameters in the fusion network can be controlled within 2M, preferably approximately 1.54M; under standard resolution infrared-visible image pairs, the single-frame processing latency can be controlled within tens of milliseconds. Simultaneously, since dual-light fusion is performed first at the edge before uploading the single-path result, the system's data transmission volume can be compressed by approximately 50%, effectively meeting the practical application requirements of continuous, low-latency sensing in industrial settings.

[0061] This invention constructs a dual-light image fusion network equipped with gradient-aware constraints. While fully preserving the naturalness of the background structure and the integrity of the scene texture in visible light images, it accurately enhances the edge contours and high-frequency details of infrared thermal targets. This effectively solves the problems of blurred target contours, weakened edges, and distorted background textures commonly found in existing fusion methods, achieving efficient collaborative fusion of infrared thermal target features and visible light structural texture features. This invention introduces a gradient-aware composite loss function to enhance the fused image's ability to retain significant edge information, effectively mitigating edge smoothing phenomena. This results in fusion results that not only possess excellent visual effects but also efficiently adapt to downstream target detection tasks, significantly improving the recognition accuracy of targets such as workers and vehicles at night and in complex industrial environments. Furthermore, this invention employs an asymmetric high-frequency injection structure decoder, achieving precise complementarity of dual-modal features through a differentiated dual-path design. The visible light structure preservation path directly introduces the visible light image, stably maintaining the global background brightness, scene structure, and texture information of the fused image, ensuring a natural and realistic background presentation. The infrared high-frequency texture enhancement path accurately extracts high-frequency detail components from the infrared image through mean filtering and subtraction processing, specifically enhancing the edges and local textures of thermal targets. Structurally, this avoids the feature neutralization and detail loss problems common in traditional symmetric decoders, improving the target clarity and detail representation of the fused image. Simultaneously, this invention uses a weight-sharing lightweight twin encoder and integrates a convolutional block attention module, significantly reducing the number of network parameters while ensuring feature extraction accuracy, making it more suitable for the lightweight deployment requirements of industrial edge devices.

[0062] In summary, this invention, by constructing a dual-light image fusion network and a composite loss function with gradient-aware constraints, can achieve efficient fusion of infrared thermal target features and visible light structural texture features, thereby enhancing the edge preservation and high-frequency detail representation capabilities of the fused image.

Claims

1. A two-light image fusion method based on asymmetric high-frequency injection and gradient-aware constraints, characterized in that, Specifically, the following steps are included: Step 1: Acquire and preprocess the synchronously acquired infrared source image and visible light source image to obtain the infrared image and visible light image; Step 2: Construct a dual-light image fusion network, which includes an encoder and a decoder. The encoder extracts features from the infrared image and the visible light image respectively, and outputs deep representation features. The decoder adopts an asymmetric high-frequency injection structure to decode and reconstruct the deep representation features. Step 3: Construct a composite loss function that includes gradient-aware constraints, and train the dual-light image fusion network based on the composite loss function to obtain the trained dual-light image fusion network; Step 4: Input the preprocessed infrared image and visible light image into the dual-light image fusion network, and output the dual-light fused image.

2. The dual-light image fusion method based on asymmetric high-frequency injection and gradient-aware constraints according to claim 1, characterized in that: In step one, the preprocessing includes size alignment, grayscale normalization, and basic registration.

3. The dual-light image fusion method based on asymmetric high-frequency injection and gradient-aware constraints according to claim 1, characterized in that: In step two, the encoder adopts a lightweight twin encoder with weight sharing. When performing feature extraction, the encoder specifically performs the following operations: The encoder maps the infrared image or the visible light image into an input feature map through shallow convolution. The input feature map is then subjected to layer normalization, multi-head self-attention calculation, weighted by a projection matrix, and connected to the residual of the input feature map to obtain an intermediate feature map. The obtained feature map is then subjected to layer normalization again, transformed by a gated feedforward network, weighted by a projection matrix, and connected to the residual again to output the deep representation feature.

4. The dual-light image fusion method based on asymmetric high-frequency injection and gradient-aware constraints according to claim 3, characterized in that: The intermediate feature map The calculation formula is as follows: ； in, For the input feature map, Presentation layer normalization operation, This indicates multi-head attention calculation. To learn the projection matrix; The deep characterization features The calculation formula is as follows: ； in, Indicates gated feedforward transform. To learn the projection matrix.

5. The dual-light image fusion method based on asymmetric high-frequency injection and gradient-aware constraints according to claim 3, characterized in that: The encoder also includes a convolutional block attention module composed of a channel attention submodule and a spatial attention submodule. The convolutional block attention module is connected in series in the intermediate extraction layer of the encoder to enhance the expressive power of the target region and the texture region.

6. The dual-light image fusion method based on asymmetric high-frequency injection and gradient-aware constraints according to claim 1, characterized in that: The asymmetric high-frequency injection structure includes a visible light structure preservation path and an infrared high-frequency texture enhancement path; the visible light structure preservation path transmits visible light images... The output fusion terminal of the decoder is directly introduced to maintain the global background brightness and structural information of the fused image; the infrared high-frequency texture enhancement path enhances the infrared image... Mean filtering is performed to extract low-frequency background components. And through infrared images and low-frequency background components Extracting high-frequency detail components by subtraction .

7. The dual-light image fusion method based on asymmetric high-frequency injection and gradient-aware constraints according to claim 6, characterized in that: The extraction of low-frequency background components The formula is as follows: ； in, This indicates an average pooling or mean filtering operation. Represents an infrared image; The formula for extracting high-frequency detail components from infrared images is as follows: ； in, This represents high-frequency detail components.

8. The dual-light image fusion method based on asymmetric high-frequency injection and gradient-aware constraints according to claim 6, characterized in that: The calculation formula for the dual-light fused image is as follows: ； in, For dual-light fusion images, Indicates the decoder reconstruction function, This represents the deep representation features of the encoder output. This represents the Sigmoid activation function. Visible light image, Gain factor to enhance infrared detail This represents high-frequency detail components.

9. The dual-light image fusion method based on asymmetric high-frequency injection and gradient-aware constraints according to claim 8, characterized in that: In step three, the composite loss function It consists of three parts: strength loss, structural loss, and gradient loss, specifically expressed as follows: ； ； in, , and This represents the weighting coefficient of each loss term. For mean square error loss, For structural similarity loss, For gradient operators, To fuse the gradient magnitudes of the images, This represents the maximum value of the pixel-by-pixel gradient amplitude in both the infrared and visible light images. It is an L1 norm.

10. The dual-light image fusion method based on asymmetric high-frequency injection and gradient-aware constraints according to claim 6, characterized in that: The training steps employ a two-stage training strategy: In the first stage, the parameters of the fusion layer, the visible light structure preservation pathway, and the infrared high-frequency texture enhancement pathway are frozen, and self-reconstruction training is performed on the infrared and visible light images respectively to initialize the encoder and decoder parameters; In the second stage, all network parameters are unfrozen, pairs of infrared and visible light images are input, and the dual-light image fusion network is jointly fine-tuned using the composite loss function.

Images