Cascade deep learning method and device for infrared and visible image fusion

Abstract

The application discloses a cascaded deep learning method and equipment for infrared and visible light image fusion, and belongs to the technical field of image processing. The method is as follows: acquiring vehicle-mounted infrared images and visible light images and performing geometric alignment; performing spatial frequency separation on the aligned images respectively, constructing a multi-scale low-frequency feature pyramid and a high-frequency feature pyramid through step-by-step average pooling downsampling; realizing lossless cross-modal feature fusion of visible light and infrared low-frequency components in the low-frequency feature pyramid through reversible transformation, and element-by-element addition fusion of visible light and infrared high-frequency components in the high-frequency feature pyramid; recombining the fused low-frequency and high-frequency features at the same scale, and generating a final fusion image through multi-layer convolution step-by-step decoding. Based on the method, a corresponding equipment is also provided. The application guarantees zero information loss in the fusion process, adaptively highlights infrared significant targets to generate high-quality fusion images, and improves the perception performance in a complex environment of automatic driving.

Description

Technical Field

[0001] This invention belongs to the field of image processing technology, and specifically relates to a cascaded deep learning method and device for fusing infrared and visible light images. Background Technology

[0002] In autonomous driving systems, environmental perception is a core component for ensuring driving safety. Image fusion technology, by combining infrared and visible light images, can significantly improve the robustness and accuracy of the perception system under different environmental conditions.

[0003] However, existing technologies still face many challenges. First, a single sensor is insufficient to meet the perception needs of complex scenarios. Visible light sensors cannot provide effective information in low-light or drastically changing lighting environments, especially in low-light scenarios such as nighttime, tunnels, and fog. Traditional visible light sensors often suffer image degradation, affecting target detection and perception. While infrared sensors are unaffected by lighting and can operate effectively in harsh weather conditions, their images lack sufficient texture details and cannot provide crucial information such as the shape of object surfaces. Second, there is physical parallax between infrared and visible light sensors, and vehicle vibrations during driving can cause non-rigid deformation of the image. Existing technologies mainly rely on registration methods based on feature point matching and mutual information, but these methods typically assume that the input images are strictly aligned, making it difficult to handle scenarios with large parallax and non-rigid deformation. Furthermore, traditional fusion methods suffer from information loss at the feature fusion level. Commonly used fusion methods directly perform image-level superposition calculations after filtering such as Gaussian filtering, which can easily cause abrupt low-frequency energy and texture masking. Direct fusion of infrared and visible light features can easily lead to information loss and texture masking, making it difficult to maintain the texture clarity of visible light images while preserving infrared thermal radiation information.

[0004] Therefore, how to achieve efficient and accurate image alignment and perform lossless multimodal feature fusion remains a significant challenge for current technologies. Summary of the Invention

[0005] To address the aforementioned technical challenges, this invention proposes a cascaded deep learning method and device for fusing infrared and visible light images. By effectively compensating for physical parallax and non-rigid deformation between infrared and visible light images, a reversible interactive architecture is utilized to achieve lossless fusion of multimodal features. This ultimately generates a high-quality fused image that simultaneously preserves infrared thermal radiation contrast and visible light texture clarity, significantly improving perception performance in complex autonomous driving environments.

[0006] To achieve the above objectives, the present invention adopts the following technical solution: In a first aspect, the present invention proposes a cascaded deep learning method for the fusion of infrared and visible light images, comprising the following steps: The infrared and visible light images acquired by the vehicle-mounted image acquisition device are obtained, and the infrared and visible light images are geometrically aligned to obtain infrared-aligned images and visible light-aligned images. Spatial frequency separation is performed on the infrared aligned image and the visible light aligned image respectively. Low frequency components and high frequency components are extracted at each scale, and the low frequency components and high frequency components are downsampled by average pooling to enter the next scale. The above process is repeated until the low frequency components of the preset scale and the high frequency components of the corresponding scale are obtained. The low frequency components of each scale are arranged in scale order to form a low frequency feature pyramid, and the high frequency components of each scale are arranged in the same scale order to form a high frequency feature pyramid. The visible light low-frequency features and infrared low-frequency features in the low-frequency feature pyramid are subjected to lossless cross-modal feature fusion to obtain the fused multi-scale low-frequency features; the visible light high-frequency features and infrared high-frequency features in the high-frequency feature pyramid are added element by element according to the corresponding scale to obtain the fused multi-scale high-frequency features. The fused multi-scale low-frequency features and the fused multi-scale high-frequency features at the same scale are recombined to merge low-frequency and high-frequency information, resulting in reconstructed features at each scale. The reconstructed features at each scale are then decoded step by step into the image space through multi-layer convolution operations to generate the final fused image.

[0007] Furthermore, infrared and visible light images are acquired by the vehicle-mounted image acquisition equipment, specifically by simultaneously acquiring infrared and visible light images using an infrared camera and a visible light camera installed at the same location on the vehicle.

[0008] Furthermore, geometric alignment processing is performed on the infrared image and the visible light image to obtain infrared-aligned images and visible light-aligned images, specifically as follows: Feature points are extracted and matched from the infrared image and the visible light image, and the homography transformation matrix is ​​calculated to compensate for the physical parallax between the infrared camera and the visible light camera. The initial alignment results are finely adjusted using a spatial transformation network to compensate for the non-rigid deformations that occur during vehicle movement. Output spatially aligned infrared and visible light aligned images.

[0009] Furthermore, the specific process of obtaining the low-frequency component at the preset scale and the high-frequency component at the corresponding scale includes: For each of the infrared alignment images and the visible light alignment images respectively , in scale Below, a Gaussian blur kernel is used. For the aligned image Perform convolutional filtering, and then perform average pooling downsampling on the filtered result to obtain the scale. low-frequency components , represented as:

[0010] The aligned image With the low-frequency components at the current scale Subtracting pixel by pixel yields the scale. high frequency components , represented as:

[0011] in, This represents the convolution operation; This indicates an average pooling downsampling operation; Indicates the total number of preset scales; Indicates the first Low-frequency components of size; Indicates the first High-frequency components of size.

[0012] Furthermore, the low-frequency components at each scale are arranged in scale order to form a low-frequency feature pyramid, and the high-frequency components at each scale are arranged in the same scale order to form a high-frequency feature pyramid, specifically as follows: The low-frequency components obtained from the infrared aligned image at each scale are arranged in order from coarse to fine to form an infrared low-frequency feature pyramid; the low-frequency components obtained from the visible light aligned image at each scale are arranged in the same order to form a visible light low-frequency feature pyramid. The high-frequency components obtained from the infrared aligned image at each scale are arranged in order from coarse to fine to form an infrared high-frequency feature pyramid; the high-frequency components obtained from the visible light aligned image at each scale are arranged in the same order to form a visible light high-frequency feature pyramid.

[0013] Furthermore, the visible light low-frequency features and infrared low-frequency features in the low-frequency feature pyramid are subjected to lossless cross-modal feature fusion to obtain fused multi-scale low-frequency features, specifically as follows: The visible light low-frequency component at this scale in the low-frequency feature pyramid. and infrared low-frequency components As input, the forward interaction operation is: ; ; ; in, This represents the first learnable convolution operation, used to transform infrared low-frequency features and then additively inject them into the visible light branch; This represents the second learnable convolution operation, used to map the fused visible light branch features into a gated modulation signal; This represents the third learnable convolution operation, used to additively inject the infrared branch after transforming the fused visible light branch features; This represents the element-wise multiplication operation; This represents an exponentially gated modulation signal, which performs element-wise adaptive weighting on the low-frequency infrared features. Representing scale Low-frequency characteristics of the visible light branch after fusion; Representing scale The low-frequency characteristics of the infrared branch after fusion; scale Low-frequency characteristics after fusion.

[0014] Furthermore, the visible light high-frequency features and infrared high-frequency features in the high-frequency feature pyramid are added element-wise according to their corresponding scales to obtain the fused multi-scale high-frequency features, specifically: For each scale The first visible light high-frequency feature pyramid High-frequency components of size , and the first in the infrared high-frequency feature pyramid High-frequency components of scale Element-wise addition is performed to obtain the high-frequency features after fusion at this scale. , represented as: ; in, Representing scale High-frequency characteristics after fusion.

[0015] Furthermore, the fused multi-scale low-frequency features and the fused multi-scale high-frequency features at the same scale are recombined, merging the low-frequency and high-frequency information to obtain the reconstructed features at each scale, specifically as follows: scale Low-frequency characteristics after fusion With scale High-frequency features after fusion By adding elements one by one, the scale is obtained. Reconstruction features under ; indicates as: ; in, Representing scale The reconstructed features below.

[0016] Furthermore, the reconstructed features at each scale are progressively decoded into the image space through multi-layer convolution operations to generate the final fused image; specifically: scale Reconstruction features under The input features are sequentially fed into the multi-layer convolutional decoding module in order of scale from coarse to fine. The multi-layer convolutional decoding module includes at least one cascaded 3×3 convolutional layer and at least one 1×1 convolutional layer. The 3×3 convolutional layer is used to refine the spatial texture of the input features, and the 1×1 convolutional layer is used to adjust the number of feature channels to fit the image space. The multi-layer convolutional decoding module performs upsampling and convolution operations on the reconstructed features at each scale step by step to restore the original image resolution, and decodes the high-dimensional hybrid features to the pixel-level image space, outputting the final fused image. .

[0017] Secondly, the present invention also proposes a cascaded deep learning device for infrared and visible light image fusion, comprising: At least one processor and a memory; the memory stores a computer program, characterized in that the computer program, when executed by the at least one processor, implements the cascaded deep learning method for infrared and visible light image fusion.

[0018] Compared with the prior art, the present invention has the following significant advantages. 1. It breaks through the bottleneck of traditional fusion based on filtering extraction. Commonly used image fusion methods directly perform image-level superposition calculations after Gaussian filtering and other methods, which can easily lead to abrupt low-frequency energy and texture masking. Although some studies have attempted to use reversible neural networks (INNs) for image fusion, they have mostly focused on using INNs as single feature extraction or fusion modules, failing to form a complete workflow of "frequency decoupling - reversible fusion - feature reconstruction". This invention first decouples the image into the basic energy matrix and edge detail matrix through spatial frequency separation, and then imports the low-frequency components into a reversible interactive architecture for deep fusion, while the high-frequency components are independently retained through element-wise addition. This architecture mathematically guarantees zero information loss in the deep fusion of multimodal representations, achieving deep decoupling and reconstruction of basic thermal radiation features and visible light structural details.

[0019] 2. Unique exponential gating mechanism enables adaptive fusion. Existing fusion methods based on attention mechanisms or Transformers often employ fixed weights or global attention distributions, making it difficult to dynamically adjust the contribution ratios of different modal features according to local scene content. This is especially problematic in low-light conditions or scenes with extreme target-background contrast, where infrared targets can easily be obscured by the visible light background, or vice versa. To address the difficulty of achieving optimal fusion in handling low-frequency scenarios, this invention designs a gating operator. This mechanism uses an exponential transform to treat the feature representation of the visible light branch as a modulation signal, adaptively weighting the infrared low-frequency features element-wise. Significant infrared target regions receive high weights and are thus highlighted, while background clutter regions are effectively suppressed with weights approaching zero, clearly solving the specific information interference problem in this application scenario.

[0020] 3. A cascade alignment strategy that is robust to input image registration errors. Most existing image fusion methods assume that the input images are strictly aligned, which is difficult to satisfy in real-world autonomous driving scenarios. The physical parallax between infrared and visible light sensors, as well as the non-rigid deformation caused by driving vibrations, cause a sharp decline in the performance of traditional registration methods based on feature point matching. This invention employs a two-stage cascaded alignment strategy at the front end of the framework: "coarse alignment using a homography transformation matrix to compensate for the physical parallax between the infrared and visible light cameras; and fine adjustment using a spatial transformation network to compensate for the non-rigid deformation generated during vehicle movement." This makes the entire fusion framework robust to the registration errors of the input images, extending the technical solution from ideal laboratory conditions to real-world engineering scenarios.

[0021] 4. The organic synergy between multi-scale pyramids and reversible fusion Existing methods either employ single-scale feature extraction, which struggles to capture both global structure and local details, or they use multi-scale decomposition but process each scale independently, neglecting cross-scale information correlation. This invention constructs a dual-branch feature pyramid of low and high frequencies through stepwise average pooling downsampling. It then reassembles the fused low-frequency features with high-frequency features at the same scale, and decodes the reconstructed features at each scale stepwise from coarse to fine, achieving full-band information preservation from global thermal radiation structure to local texture details. Experimental results show that this invention significantly outperforms existing technologies in four evaluation metrics: MI, VIF, SSIM, and CC. Attached Figure Description

[0022] Figure 1 This is a flowchart of the cascaded deep learning method for fusion of infrared and visible light images proposed in Embodiment 1 of the present invention; Figure 2 This is a schematic diagram of the cascaded deep learning method framework for infrared and visible light image fusion proposed in Embodiment 1 of the present invention. Figure 3 This is a framework diagram of spatial frequency separation proposed in Embodiment 1 of the present invention; Figure 4 This is a visual comparison chart showing the integration results of this invention and existing technologies; Figure 5 This is a schematic diagram of a cascaded deep learning device for fusing infrared and visible light images, as proposed in Embodiment 2 of the present invention. Detailed Implementation

[0023] To clearly illustrate the technical features of this solution, the invention will be described in detail below through specific embodiments and in conjunction with the accompanying drawings. The following disclosure provides many different embodiments or examples for implementing different structures of the invention. To simplify the disclosure of the invention, components and arrangements of specific examples are described below. Furthermore, reference numerals and / or letters may be repeated in different examples. This repetition is for simplification and clarity and does not in itself indicate a relationship between the various embodiments and / or arrangements discussed. It should be noted that the components illustrated in the drawings are not necessarily drawn to scale. Descriptions of well-known components, processing techniques, and processes are omitted in this invention to avoid unnecessarily limiting the invention.

[0024] Example 1 Embodiment 1 of this invention proposes a cascaded deep learning method for fusing infrared and visible light images. The technical problem it solves is: in autonomous driving scenarios, how to overcome the physical parallax and non-rigid deformation between infrared and visible light images to achieve accurate registration, and avoid the loss of multimodal information and texture masking during feature fusion, so as to generate a high-quality fused image that retains both thermal radiation contrast and texture clarity.

[0025] Figure 1 This is a flowchart of the cascaded deep learning method for fusion of infrared and visible light images proposed in Embodiment 1 of the present invention; Figure 2 This is a schematic diagram of the cascaded deep learning method framework for infrared and visible light image fusion proposed in Embodiment 1 of the present invention; combined with Figure 1 and Figure 2 The process of implementing this invention will be explained together.

[0026] In step S1, infrared and visible light images are acquired by the vehicle-mounted image acquisition device, and geometric alignment processing is performed on the infrared and visible light images to obtain infrared-aligned images and visible light-aligned images. By simultaneously capturing images of the scene ahead of the vehicle while it is in motion using an infrared camera and a visible light camera installed at the same location on the vehicle, raw infrared images are obtained. and the original visible light image .

[0027] To ensure time consistency in subsequent fusion processing, the infrared and visible light cameras employ hardware-triggered frame-level synchronization, ensuring that both images capture the same scene at the same time. The infrared camera captures the thermal radiation information of objects in the scene, enabling clear imaging under low-light or adverse weather conditions such as nighttime, fog, rain, and snow; the visible light camera captures the texture details and color information of objects in the scene, providing rich structural information under sufficient lighting conditions.

[0028] Because infrared and visible light cameras cannot be perfectly aligned physically, there are fixed translational and rotational deviations between them, resulting in differences in the pixel positions of the same target in the two images, i.e., physical parallax. Furthermore, vibrations during vehicle movement can cause instantaneous camera displacement, leading to non-rigid deformation of the image. Therefore, geometric alignment processing is required for the original infrared and visible light images to eliminate physical parallax and non-rigid deformation, ensuring pixel-by-pixel correspondence between the two images in the same coordinate system.

[0029] Geometric alignment processing is divided into two stages: coarse alignment and fine alignment. In the coarse alignment stage, feature points are extracted from the infrared and visible light images and feature matching is performed. Based on the matched feature point pairs, the homography transformation matrix between the two images is calculated. This matrix is ​​then used to perform a projection transformation on one image, mapping it to the coordinate system of the other image. This compensates for the fixed physical parallax between the infrared and visible light cameras, achieving global-scale spatial alignment.

[0030] In the fine alignment stage, a spatial transformation network is used to refine the coarse alignment result. The spatial transformation network is a differentiable spatial transformation module that learns the local deformation field of the input image and performs non-rigid spatial transformations. The coarsely aligned infrared and visible light images are input into the spatial transformation network. This network regresses the spatial transformation parameters through a localization network, a mesh generator generates a sampling mesh based on the transformation parameters, and a sampler performs bilinear interpolation sampling on the input image, outputting the spatially transformed image. Through end-to-end training, the spatial transformation network learns to adjust the coarsely aligned image to a precise spatial correspondence, effectively compensating for the non-rigid deformations generated during vehicle movement.

[0031] After two stages of coarse and fine alignment, the physical parallax and non-rigid deformation between the infrared and visible light images are effectively compensated, and the output infrared and visible light aligned images are spatially precisely aligned, providing spatially consistent input for subsequent spatial frequency separation and multimodal feature fusion.

[0032] In step S2, spatial frequency separation is performed on the infrared aligned image and the visible light aligned image respectively. Low-frequency components and high-frequency components are extracted for each scale, and the low-frequency components and high-frequency components are downsampled by average pooling to enter the next scale. The above process is repeated until the low-frequency components of the preset scale and the high-frequency components of the corresponding scale are obtained. The low-frequency components of each scale are arranged in scale order to form a low-frequency feature pyramid, and the high-frequency components of each scale are arranged in the same scale order to form a high-frequency feature pyramid. The purpose of spatial frequency separation is to decompose an image into different frequency components. The low-frequency component represents the image's basic structural intensity information, reflecting the overall brightness and contrast distribution, and includes the main energy of thermal radiation. The high-frequency component represents the image's edge texture details, reflecting local abrupt changes and fine structure, including object edges, textures, and other detailed features. By decoupling infrared and visible light images into low-frequency and high-frequency bands respectively, fusion strategies can be designed for the physical characteristics of different frequency bands in subsequent processing, avoiding mutual interference between different modal features. Figure 3 This is a framework diagram of spatial frequency separation proposed in Embodiment 1 of the present invention; The specific process includes: for each alignment image in the infrared alignment image and the visible light alignment image respectively. , in scale Below, a Gaussian blur kernel is used. For the aligned image Perform convolutional filtering, and then perform average pooling downsampling on the filtered result to obtain the scale. low-frequency components , is represented as:

[0033] The aligned image With the low-frequency components at the current scale Subtracting pixel by pixel yields the scale. high frequency components , is represented as:

[0034] in, This represents the convolution operation; This indicates an average pooling downsampling operation; Indicates the total number of preset scales; Indicates the first Low-frequency components of size; Indicates the first High-frequency components of size.

[0035] Since infrared and visible light images are processed independently, and the operations of Gaussian blurring and residual subtraction are deterministic calculations, the high-frequency and low-frequency components obtained from the infrared and visible light images respectively completely retain the frequency domain information of the original images.

[0036] Obtain the low-frequency components at the current scale and high frequency components Then, the input image at the next scale is the low-frequency component of the current scale. That is, at the next scale In the middle, the low-frequency components at the current scale This process is repeated as a new aligned image. The above operations of Gaussian blur filtering, residual subtraction to obtain high-frequency components, and average pooling downsampling are performed. Through this iterative approach, each scale further extracts lower-frequency structural information and corresponding high-frequency details based on the low-frequency components of the previous scale, thus forming a multi-scale frequency domain feature pyramid.

[0037] In the above iteration process, the Gaussian blur kernel parameters at different scales can be the same, or they can be adaptively adjusted according to the resolution of different scales. In specific implementations, it is preferable to use the same Gaussian blur kernel parameters for each scale to maintain the consistency of frequency decomposition. Average pooling halves the spatial resolution of the feature map at each scale, while increasing the receptive field at each scale. This allows coarser scales to capture a wider range of structural information, while finer scales retain more refined local details. (Preset total number of scales) The preferred value is 3 to 5. When When the value is too small, multi-scale information is insufficient, affecting the fusion effect; when... When the value is too large, the feature map resolution is too low, resulting in the loss of effective information and increased computational overhead. In this embodiment, The preferred value is 3. The scope of protection of this invention is not limited to the values ​​listed in Example 1, and those skilled in the art can make reasonable adjustments according to the actual situation.

[0038] Through the above process, both the infrared aligned image and the visible light aligned image obtain a set of multi-scale low-frequency components. and corresponding high-frequency components .

[0039] The low-frequency components at each scale are arranged in scale order to form a low-frequency feature pyramid. Specifically, the low-frequency components obtained from the infrared aligned image at each scale are... According to the scale from coarse to fine (i.e., from scale 1 to scale 2) The order of scales forms the infrared low-frequency feature pyramid. Low-frequency components obtained at various scales from images aligned with visible light. Arranged in order of the same scale, they form a pyramid of low-frequency features of visible light. .

[0040] The high-frequency components at each scale are arranged in the same scale order to form a high-frequency feature pyramid. Specifically, this involves arranging the high-frequency components of the infrared-aligned image at each scale. According to the scale from coarse to fine (i.e., from scale 1 to scale 2) The order of scales constitutes the infrared high-frequency feature pyramid. High-frequency components obtained by aligning visible light images at various scales Arranged in order of the same scale, they form a pyramid of high-frequency features of visible light.

[0041] Thus, the infrared-aligned image and the visible-light-aligned image each form two independent feature pyramids: a low-frequency feature pyramid and a high-frequency feature pyramid, which respectively carry the basic structural information and edge detail information of the image. The features at each scale in the low-frequency feature pyramid reflect the main energy distribution of thermal radiation at different resolutions, while the features at each scale in the high-frequency feature pyramid reflect the edge texture details at different resolutions, providing input for the reversible low-frequency fusion and high-frequency element-wise addition in subsequent steps.

[0042] The purpose of this frequency separation step is not to directly fuse the images, but to provide a strictly decoupled basic energy matrix and edge detail matrix for the subsequent step S3, thereby reducing mutual interference between different modal features.

[0043] In step S3, the visible light low-frequency features and infrared low-frequency features in the low-frequency feature pyramid are subjected to lossless cross-modal feature fusion to obtain fused multi-scale low-frequency features; the visible light high-frequency features and infrared high-frequency features in the high-frequency feature pyramid are added element by element according to their corresponding scales to obtain fused multi-scale high-frequency features. Because low-frequency and high-frequency components carry different physical information in an image—low-frequency components represent the basic structural strength information of the image, reflecting the main energy distribution of thermal radiation; while high-frequency components represent the edge texture details of the image, reflecting the boundaries and texture features of objects—this invention designs differentiated fusion strategies for the characteristics of low-frequency and high-frequency components to achieve efficient integration of multimodal information.

[0044] The reversible fusion process of low-frequency features includes: for the low-frequency components in the low-frequency feature pyramid, a reversible interactive architecture is used to achieve lossless cross-modal feature fusion. Infrared low-frequency features contain the main energy distribution of thermal radiation, which can highlight targets with thermal signals in the scene; visible light low-frequency features contain the basic structural strength information of the scene, reflecting the brightness distribution of object surfaces. The two are complementary in the low-frequency domain, but direct fusion is prone to information suppression or texture masking due to modal differences. Therefore, this invention introduces a reversible transformation with theoretically lossless properties to perform deep fusion of low-frequency components.

[0045] For scale The visible light low-frequency components at this scale in the low-frequency feature pyramid and infrared low-frequency components As input, the forward interaction operation is: ; ; ; in, This represents the first learnable convolution operation, used to transform infrared low-frequency features and then additively inject them into the visible light branch; This represents the second learnable convolution operation, used to map the fused visible light branch features into a gated modulation signal; This represents the third learnable convolution operation, used to additively inject the infrared branch after transforming the fused visible light branch features; This represents the element-wise multiplication operation; This represents an exponentially gated modulation signal, which performs element-wise adaptive weighting on the low-frequency infrared features. Representing scale Low-frequency characteristics of the visible light branch after fusion; Representing scale The low-frequency characteristics of the infrared branch after fusion; scale Low-frequency characteristics after fusion.

[0046] In the formula middle, The infrared low-frequency features are transformed and then additively injected into the visible light branch. Through this operation, infrared thermal radiation information is introduced into the visible light feature path, achieving preliminary fusion of cross-modal information. Because an additive operation is used instead of a weighted average, this step does not compress or lose the original structural information of the visible light branch.

[0047] middle, The fused visible light branch features are mapped to a gated modulation signal, which is then exponentially transformed and multiplied element-wise with the original infrared low-frequency features. This exponential gating mechanism... This constitutes gated multiplicative modulation, the function of which is: for regions in the infrared features that are highly correlated with visible light structural information (usually corresponding to the background region), the gate value tends to 0, suppressing the transmission of infrared features; for regions in the infrared features that have significant thermal radiation targets (usually corresponding to pedestrians and vehicles, etc.), the gate value tends to 1, maintaining the complete transmission of infrared features. After transforming the fused visible light branch features, an additive injection into the infrared branch further supplements the visible light structural information. Through the above gating mechanism, salient targets in the infrared image are adaptively highlighted, while background clutter is effectively suppressed, achieving cross-modal adaptive fusion based on dynamic adjustment of scene content.

[0048] The output features of the two branches are additively merged to obtain the low-frequency features after fusion at this scale. .

[0049] The aforementioned reversible interactive architecture possesses theoretically lossless properties at the mathematical level. During forward propagation, the input information... Transformed step by step into output Furthermore, the determinant of the Jacobian matrix during the transformation process is not zero, indicating that the mapping is bijective, meaning that the input information does not undergo irreversible information compression or loss during the transformation. This characteristic ensures the integrity of multimodal information during cross-modal low-frequency feature fusion, avoiding the information loss problem in traditional fusion methods.

[0050] Each scale can share the same set of learnable convolution operation parameters, or each scale can have its parameters set independently. In practice, the shared parameter approach is preferred to reduce the number of model parameters and improve training efficiency. The low-frequency components from each scale are fused using the aforementioned reversible interactive architecture to obtain fused multi-scale low-frequency features. .

[0051] The process of element-wise addition and fusion of high-frequency features is as follows: For the high-frequency components in the high-frequency feature pyramid, due to the infrared high-frequency components... and visible light high-frequency components Both represent edge texture details of the image. They are highly correlated in spatial location and contain complementary edge information. The visible light high-frequency component provides rich texture details in well-lit areas, while the infrared high-frequency component provides thermal radiation edge information in low-light areas. Since the high-frequency components are sensitive to spatial location accuracy, element-wise addition is an effective way to preserve and merge complementary edge information, avoiding the introduction of additional phase shifts or nonlinear distortions.

[0052] For each scale The first visible light high-frequency feature pyramid High-frequency components of size , and the first in the infrared high-frequency feature pyramid High-frequency components of scale Element-wise addition is performed to obtain the high-frequency features after fusion at this scale. , represented as: ; in, Representing scale High-frequency characteristics after fusion.

[0053] By adding elements one by one, complementary edge information from visible and infrared images is merged, ensuring that clear textures and structures are fully preserved in the fusion result. The high-frequency features from each scale after fusion collectively constitute the multi-scale high-frequency features of the fused image. .

[0054] After step S3 is executed, the fused multi-scale low-frequency features are obtained. and the fused multi-scale high-frequency features .

[0055] In step S4, the fused multi-scale low-frequency features and the fused multi-scale high-frequency features at the same scale are recombined to merge low-frequency and high-frequency information and obtain reconstructed features at each scale; the reconstructed features at each scale are gradually decoded into the image space through multi-layer convolution operations to generate the final fused image.

[0056] In the aforementioned steps, the multi-scale low-frequency components of the infrared and visible light images achieve deep fusion through reversible interaction, while the high-frequency components achieve edge information merging through element-wise addition. At this point, a fused low-frequency feature exists at each scale. and a fused high-frequency feature Both carry basic structural information and edge detail information at that scale, respectively. The purpose of step S4 is to reorganize the low-frequency and high-frequency features at the same scale, and restore them to the original image resolution through stepwise decoding across scales, generating the final fused image.

[0057] scale Low-frequency characteristics after fusion With scale High-frequency features after fusion By adding elements one by one, the scale is obtained. Reconstruction features under ; indicates as: ; in, Representing scale The reconstructed features below.

[0058] The element-wise addition operation described above achieves the complete merging of low-frequency and high-frequency information at the same scale. After element-wise addition, the basic structural information and edge detail information at this scale are integrated into the same feature map, forming a complete frequency domain feature representation, providing complete input information for subsequent decoding.

[0059] Reconstruction features at various scales Together they constitute multi-scale reconstruction features, among which the coarser scale ( Smaller feature maps have smaller dimensions but larger receptive fields, containing global structural information; finer scale ( Larger feature maps are larger in size and contain local details.

[0060] Reconstructed features at each scale According to the scale from coarse to fine (i.e., from scale 1 to scale 2) The low-resolution feature maps are sequentially input into the multi-layer convolutional decoding module in the order of scale. This decoding process uses a combination of stepwise upsampling and convolution operations to gradually restore the low-resolution feature maps to the original image resolution.

[0061] The multi-layer convolutional decoding module includes at least one cascaded 3×3 convolutional layer and at least one 1×1 convolutional layer. The 3×3 convolutional layer refines the spatial texture of the input features by extracting correlations within the spatial neighborhood through local receptive fields, enhancing texture details and edge sharpness in the feature map. The 1×1 convolutional layer adjusts the number of feature channels to fit the image space, mapping high-dimensional feature channels to the target number of output channels (e.g., 1 channel for grayscale images and 3 channels for color images), while simultaneously achieving a linear combination of cross-channel information.

[0062] In the specific implementation process, the multi-layer convolutional decoding module operates according to the following steps: For the coarsest scale , will reconstruct features The input is fed into a multi-layer convolutional decoding module. First, the features are refined through a 3×3 convolutional layer. Then, an upsampling operation (such as bilinear interpolation or transposed convolution) is used to double the spatial resolution of the feature map, making it compatible with the next scale. Reconstruction features Consistent size; compare the upsampled features with Feature concatenation or element-wise addition is performed, followed by refinement through a 3×3 convolutional layer, and then upsampling to the next scale. This "upsampling-fusion-refinement" process is repeated until the feature map is restored to the original image resolution. In the final stage of the decoding path, a 1×1 convolutional layer maps the feature channel number to the target image channel number, generating the final fused image. .

[0063] In each decoding stage, the 3×3 convolutional layer is used in conjunction with batch normalization and activation functions (such as ReLU) to improve training stability and non-linear expressive power. At the same time, cross-scale feature fusion (combining the features upsampled from the previous stage with the features reconstructed at the current scale) enables global structural information to effectively guide the recovery of local details and avoids the loss of high-frequency information during upsampling.

[0064] Through the aforementioned progressive upsampling and convolution operations, the multi-frequency information contained in the reconstructed features at each scale is gradually converged and restored to the original image resolution. The final fused image output by the multi-layer convolutional decoding module is... At the pixel level, it achieves effective coordination between infrared thermal radiation contrast and visible light texture clarity: thermal targets (such as pedestrians and vehicles) in infrared images are presented with high contrast, and texture details (such as road markings and building outlines) in visible light images are clearly distinguishable without obvious blur artifacts or block effects.

[0065] Final fused image It retains both the thermal radiation contrast information of infrared images and the texture detail information of visible light images, and can be directly used for downstream vision tasks such as target detection and semantic segmentation in autonomous driving systems.

[0066] To verify the effectiveness of the cascaded deep learning framework proposed in this invention, this embodiment conducted a fusion experiment on a publicly available dataset of infrared and visible light images of autonomous driving scenarios, and made a qualitative and quantitative comparison between the method of this invention and existing mainstream fusion methods.

[0067] Figure 4 This is a visual comparison of the fusion results using this invention and existing technologies (DenseFuse, MemoryFusion, MMIF, RFN-Nest, SwinFusion, U2Fusion). In the figure, IR represents infrared source images, VIS represents visible light source images; DenseFuse is a densely connected fusion network, MemoryFusion is a memory fusion network, MMIF is a multimodal interactive fusion method, RFN-Nest is a recursive fusion nested network, SwinFusion is a Swin Transformer fusion network, and U2Fusion is a unified fusion network.

[0068] from Figure 4It can be seen that the infrared source image (IR column) can clearly highlight targets with thermal radiation in the scene (such as pedestrians and vehicles), but lacks background texture details and the overall vision is relatively blurry; the visible light source image (VIS column) contains rich texture details and structural information (such as road markings, building outlines, tree branches and leaves), but the target contrast is low in low light areas.

[0069] Among the comparative methods, while DenseFuse and MemoryFusion preserve the texture details of the visible light image, their contrast enhancement for infrared thermal targets is insufficient. Pedestrians and other thermal targets are not prominent enough in the fused image, making it difficult to meet the requirements for fast and accurate pedestrian and vehicle identification in autonomous driving scenarios. MMIF and RFN-Nest show some improvement in thermal target detection, but the preservation of visible light texture is insufficient, resulting in blurring in some background areas and overall insufficient clarity in the fused image. SwinFusion and U2Fusion offer relatively balanced overall visual effects, but the former still has room for improvement in edge sharpness, while the latter exhibits significant contrast reduction and information loss in low-light areas.

[0070] In comparison, the fusion results of the method of this invention (Ours) exhibit significant advantages in several aspects: Firstly, thermal targets (pedestrians, vehicles) in the infrared image are clearly presented with high contrast in the fused image, highlighting prominent targets, which is beneficial for downstream target detection tasks; secondly, texture details (road markings, building edges, tree outlines) in the visible light image are completely preserved in the fused image, with clear textures and sharp edges. The fused image has a natural overall visual appearance, without obvious blur artifacts or block effects, and is significantly superior to all comparison methods in terms of visual perception quality, achieving effective coordination between infrared thermal radiation contrast and visible light texture clarity.

[0071] This experiment employed four objective evaluation metrics to quantitatively assess the fusion results: Mutual Information (MI), which measures the amount of information retained by the fused image from the source images; Visual Information Fidelity (VIF), which evaluates the visual perception quality of the fused image; Structural Similarity (SSIM), which measures the structural similarity between the fused image and the source images; and Correlation Coefficient (CC), which reflects the degree of linear correlation between the fused image and the source images. These four metrics comprehensively evaluate the quality of the fused image from four dimensions: information retention, visual perception, structural fidelity, and correlation. Higher metric values ​​indicate better fusion results.

[0072] Table 1: Comparison of image fusion effects of the present invention and existing technologies without registration

[0073] As shown in Table 1, the method of the present invention achieved optimal results in all four evaluation indicators. Specifically: In terms of mutual information (MI) metric, this invention achieves 6.321, a 13.8% improvement over the second-best SwinFusion (5.553), indicating that the method of this invention can retain richer information from the source image. Mutual information reflects the degree to which the fused image inherits information from the source image, and the significant advantage of this invention demonstrates the effectiveness of the reversible interactive architecture in information preservation.

[0074] In terms of Visual Information Fidelity (VIF), this invention achieves a score of 1.305, significantly outperforming all comparative methods, indicating that the fused image generated by this invention has superior visual perception quality. The VIF metric measures the fidelity of the fused image at the human visual perception level, and the excellent performance of this invention is attributed to the synergistic effect of low-frequency reversible fusion and exponential gating mechanism.

[0075] In terms of structural similarity (SSIM) index, the present invention achieves 0.910, which is 8.7% higher than the second best DenseFuse (0.837), indicating that the method of the present invention has a significant advantage in terms of structural fidelity, and the fused image and the source image have the highest degree of similarity at the structural level.

[0076] In terms of correlation coefficient (CC), this invention achieves 0.958, which is also superior to all comparison methods, indicating that the linear correlation between the fused image and the source image is the strongest, and no obvious nonlinear distortion is introduced during the fusion process.

[0077] The quantitative results and subjective visual comparison results show good consistency, jointly verifying the superiority of the method of this invention in the task of infrared and visible light image fusion. First, by constructing a low-frequency and high-frequency dual-branch feature pyramid through spatial frequency separation, decoupling information from different frequency bands is achieved, reducing mutual interference between cross-modal features. Second, lossless fusion of low-frequency features is achieved through a reversible interactive architecture, mathematically ensuring the complete preservation of multimodal information. Third, an exponential gating mechanism is used to achieve adaptive highlighting of infrared salient targets and effective suppression of background clutter. The synergistic effect of these techniques enables the method of this invention to achieve optimal performance in all four dimensions: information preservation, visual perception, structural fidelity, and correlation.

[0078] Example 2 Based on the cascaded deep learning method for infrared and visible light image fusion proposed in Embodiment 1 of this invention, Embodiment 2 of this invention proposes a cascaded deep learning device for infrared and visible light image fusion. Figure 5 This is a schematic diagram of a cascaded deep learning device for fusing infrared and visible light images, as proposed in Embodiment 2 of the present invention.

[0079] At the hardware level, electronic device 500 includes a processor 510, and optionally, an internal bus 520, a network interface 530, and memory. The memory may include main memory, such as high-speed random-access memory (RAM), or it may also include non-volatile memory, such as at least one disk drive. Of course, the electronic device may also include other hardware required for other business operations.

[0080] The processor 510, network interface 530, and memory can be interconnected via an internal bus 520. This internal bus 520 can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. The bus can be categorized as an address bus, data bus, control bus, etc. For ease of illustration, only a single bidirectional arrow is used in this diagram, but this does not imply that there is only one bus or one type of bus. The memory is used to store programs. Specifically, the program can include program code, which includes computer operation instructions. The memory can include main memory 540 and non-volatile memory 550, and provides instructions and data to the processor 510.

[0081] Processor 510 reads the corresponding computer program from non-volatile memory 550 into memory 540 and then runs it, forming a cascaded deep learning method for infrared and visible light image fusion at the logical level. Processor 510 executes the program stored in memory and specifically performs the following: In step S1, infrared and visible light images are acquired by the vehicle-mounted image acquisition device, and geometric alignment processing is performed on the infrared and visible light images to obtain infrared-aligned images and visible light-aligned images. In step S2, spatial frequency separation is performed on the infrared aligned image and the visible light aligned image respectively. Low-frequency components and high-frequency components are extracted for each scale, and the low-frequency components and high-frequency components are downsampled by average pooling to enter the next scale. The above process is repeated until the low-frequency components of the preset scale and the high-frequency components of the corresponding scale are obtained. The low-frequency components of each scale are arranged in scale order to form a low-frequency feature pyramid, and the high-frequency components of each scale are arranged in the same scale order to form a high-frequency feature pyramid. In step S3, the visible light low-frequency features and infrared low-frequency features in the low-frequency feature pyramid are subjected to lossless cross-modal feature fusion to obtain fused multi-scale low-frequency features; the visible light high-frequency features and infrared high-frequency features in the high-frequency feature pyramid are added element by element according to their corresponding scales to obtain fused multi-scale high-frequency features. In step S4, the fused multi-scale low-frequency features and the fused multi-scale high-frequency features at the same scale are recombined to merge low-frequency and high-frequency information and obtain reconstructed features at each scale; the reconstructed features at each scale are gradually decoded into the image space through multi-layer convolution operations to generate the final fused image.

[0082] Figure 1 It can be applied to processor 510, or implemented by processor 510. The processor may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by the integrated logic circuit in the processor or by instructions in the form of software. The processor mentioned above can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this application. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in the embodiments of this application can be directly embodied as being executed by a hardware decoding processor, or executed by a combination of hardware and software modules in the decoding processor. The software module can reside in a mature storage medium in the field, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, or registers. This storage medium is located in memory, and the processor reads information from the memory and, in conjunction with its hardware, completes the steps of the above method.

[0083] Of course, in addition to software implementation, the electronic device of this application does not exclude other implementation methods, such as logic devices or a combination of hardware and software, etc. In other words, the execution subject of the following processing flow is not limited to each logic unit, but can also be hardware or logic devices.

[0084] The description of the cascaded deep learning device for infrared and visible light image fusion provided in Embodiment 2 of this application can be found in the detailed description of the corresponding part in the cascaded deep learning method for infrared and visible light image fusion provided in Embodiment 1 of this application, and will not be repeated here.

[0085] It should be noted that, in this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that the elements inherent in a process, method, article, or apparatus that includes a list of elements are included. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element. Additionally, portions of the technical solutions provided in the embodiments of this application that are consistent with the implementation principles of corresponding technical solutions in the prior art have not been described in detail to avoid excessive elaboration.

[0086] While specific embodiments of the present invention have been described above in conjunction with the accompanying drawings, this is not intended to limit the scope of protection of the present invention. Those skilled in the art can make other modifications or variations based on the above description. It is neither necessary nor possible to exhaustively describe all embodiments here. Various modifications or variations that can be made by those skilled in the art without creative effort based on the technical solutions of the present invention are still within the scope of protection of the present invention.

Claims

1. A cascaded deep learning method for fusing infrared and visible light images, characterized in that, Includes the following steps: The infrared and visible light images acquired by the vehicle-mounted image acquisition device are obtained, and the infrared and visible light images are geometrically aligned to obtain infrared-aligned images and visible light-aligned images. Spatial frequency separation is performed on the infrared aligned image and the visible light aligned image respectively. Low frequency components and high frequency components are extracted at each scale, and the low frequency components and high frequency components are downsampled by average pooling to enter the next scale. The above process is repeated until the low frequency components of the preset scale and the high frequency components of the corresponding scale are obtained. The low frequency components of each scale are arranged in scale order to form a low frequency feature pyramid, and the high frequency components of each scale are arranged in the same scale order to form a high frequency feature pyramid. The visible light low-frequency features and infrared low-frequency features in the low-frequency feature pyramid are subjected to lossless cross-modal feature fusion to obtain the fused multi-scale low-frequency features; the visible light high-frequency features and infrared high-frequency features in the high-frequency feature pyramid are added element by element according to the corresponding scale to obtain the fused multi-scale high-frequency features. The fused multi-scale low-frequency features and the fused multi-scale high-frequency features at the same scale are recombined to merge low-frequency and high-frequency information, resulting in reconstructed features at each scale. The reconstructed features at each scale are then decoded step by step into the image space through multi-layer convolution operations to generate the final fused image.

2. The method according to claim 1, characterized in that, To acquire infrared and visible light images from vehicle-mounted image acquisition equipment, specifically, infrared and visible light images are acquired simultaneously using an infrared camera and a visible light camera installed at the same location on the vehicle.

3. The method according to claim 1, characterized in that, Geometric alignment processing is performed on infrared and visible light images to obtain infrared-aligned and visible light-aligned images, specifically as follows: Feature points are extracted and matched from the infrared image and the visible light image, and the homography transformation matrix is ​​calculated to compensate for the physical parallax between the infrared camera and the visible light camera. The initial alignment results are finely adjusted using a spatial transformation network to compensate for the non-rigid deformations that occur during vehicle movement. Output spatially aligned infrared and visible light aligned images.

4. The method according to claim 1, characterized in that, The specific process of obtaining the low-frequency component at the preset scale and the high-frequency component at the corresponding scale includes: For each of the infrared alignment images and the visible light alignment images respectively , in scale Below, a Gaussian blur kernel is used. For the aligned image Perform convolutional filtering and then average pooling downsample the filtered result to obtain the scale. low-frequency components , represented as: The aligned image With the low-frequency components at the current scale Subtracting pixel by pixel yields the scale. high frequency components , represented as: in, This represents the convolution operation; This indicates an average pooling downsampling operation; Indicates the total number of preset scales; Indicates the first Low-frequency components of size; Indicates the first High-frequency components of size.

5. The method according to claim 1, characterized in that, The low-frequency components at each scale are arranged in scale order to form a low-frequency feature pyramid, and the high-frequency components at each scale are arranged in the same scale order to form a high-frequency feature pyramid. Specifically: The low-frequency components obtained from the infrared aligned image at each scale are arranged in order from coarse to fine to form an infrared low-frequency feature pyramid; the low-frequency components obtained from the visible light aligned image at each scale are arranged in the same order to form a visible light low-frequency feature pyramid. The high-frequency components obtained from the infrared aligned image at each scale are arranged in order from coarse to fine to form an infrared high-frequency feature pyramid; the high-frequency components obtained from the visible light aligned image at each scale are arranged in the same order to form a visible light high-frequency feature pyramid.

6. The method according to claim 4, characterized in that, The visible light low-frequency features and infrared low-frequency features in the low-frequency feature pyramid are subjected to lossless cross-modal feature fusion to obtain the fused multi-scale low-frequency features, specifically: The visible light low-frequency component at this scale in the low-frequency feature pyramid. and infrared low-frequency components As input, the forward interaction operation is: ； ； ； in, This represents the first learnable convolution operation, used to transform infrared low-frequency features and then additively inject them into the visible light branch; This represents the second learnable convolution operation, used to map the fused visible light branch features into a gated modulation signal; This represents the third learnable convolution operation, used to additively inject the infrared branch after transforming the fused visible light branch features; This represents the element-wise multiplication operation; This represents an exponentially gated modulation signal, which performs element-wise adaptive weighting on the low-frequency infrared features. Representing scale Low-frequency characteristics of the visible light branch after fusion; Representing scale The low-frequency characteristics of the infrared branch after fusion; scale Low-frequency characteristics after fusion.

7. The method according to claim 6, characterized in that, The visible light high-frequency features and infrared high-frequency features in the high-frequency feature pyramid are added element-wise according to their corresponding scales to obtain the fused multi-scale high-frequency features, specifically: For each scale The first part of the visible light high-frequency feature pyramid High-frequency components of size , and the first in the infrared high-frequency feature pyramid High-frequency components of scale Element-wise addition is performed to obtain the high-frequency features after fusion at this scale. , represented as: ； in, Representing scale High-frequency characteristics after fusion.

8. The method according to claim 7, characterized in that, The fused multi-scale low-frequency features at the same scale are recombined with the fused multi-scale high-frequency features, and the low-frequency and high-frequency information are merged to obtain the reconstructed features at each scale, specifically as follows: scale Low-frequency characteristics after fusion With scale High-frequency features after fusion By adding elements one by one, the scale is obtained. Reconstruction features under ; indicates as: ； in, Representing scale The reconstructed features below.

9. The method according to claim 8, characterized in that, The reconstructed features at each scale are progressively decoded into the image space through multi-layer convolution operations to generate the final fused image; specifically: scale Reconstruction features under The input features are sequentially fed into the multi-layer convolutional decoding module in order of scale from coarse to fine. The multi-layer convolutional decoding module includes at least one cascaded 3×3 convolutional layer and at least one 1×1 convolutional layer. The 3×3 convolutional layer is used to refine the spatial texture of the input features, and the 1×1 convolutional layer is used to adjust the number of feature channels to fit the image space. The multi-layer convolutional decoding module performs upsampling and convolution operations on the reconstructed features at each scale step by step to restore the original image resolution, and decodes the high-dimensional hybrid features to the pixel-level image space, outputting the final fused image. .

10. A cascaded deep learning device for fusing infrared and visible light images, including: At least one processor and a memory; the memory stores a computer program, characterized in that the computer program, when executed by the at least one processor, implements the cascaded deep learning method for infrared and visible light image fusion as described in any one of claims 1 to 9.

Images