A method and system for fusing infrared and visible light images

Abstract

The present application relates to a kind of for infrared image and visible light image fusion method and system, method includes: extracting shallow feature from infrared and visible light image;Obtain multiple levels of infrared and visible light features by multilevel residual coding;Obtain spatial domain and frequency domain fusion features by spatial domain fusion branch and frequency domain fusion branch;Carry out double-domain serial coupling processing, obtain spatial-frequency domain coupling fusion feature;Unified fusion guide feature is obtained by cross-layer convergence;Adaptive modulation obtains infrared and visible light high-level feature;Carry out high-level re-aggregation and reconstruction output to obtain high-level re-aggregation feature, and generate final fusion image according to high-level re-aggregation feature.Multiple levels of feature extraction simultaneously utilize spatial domain cross-modal complementary information and frequency domain amplitude-phase complementary information, through double-domain serial coupling mechanism realizes the interaction compensation and modulation of spatial information and frequency domain information, improves the target saliency of fusion image, structure definition and detail retention ability.

Description

Technical Field

[0001] This invention belongs to the field of image fusion and computer vision technology, and particularly relates to a method and system for fusing infrared images and visible light images. Background Technology

[0002] Infrared images can reflect the difference in thermal radiation between a target and the background, and are capable of highlighting thermal targets in complex environments such as low light, smoke, and nighttime. Visible light images provide rich texture, edge, and color information, which is beneficial for expressing scene structure and details. Infrared and visible light image fusion aims to combine the complementary information from the two types of sensors to generate a fused image that retains both the saliency of infrared targets and the structural texture of visible light. This technology is widely used in fields such as intelligent surveillance, autonomous driving, security patrol, target detection, and robot perception.

[0003] Traditional infrared-visible image fusion methods typically rely on manually designed transform domain rules, saliency weights, or multi-scale decomposition models. While these methods can achieve certain fusion effects in specific scenarios, they are prone to problems such as weakened infrared targets, overly smoothed visible light textures, and blurred edge details under complex lighting, low contrast, or highly coupled target and background conditions.

[0004] In recent years, deep learning-based image fusion methods have effectively improved fusion quality by achieving end-to-end feature extraction and reconstruction through convolutional neural networks, attention mechanisms, or autoencoder structures. However, existing methods still have several shortcomings: First, most methods mainly perform convolution or attention fusion in the spatial domain, failing to fully utilize frequency domain amplitude and phase information; second, although some methods introduce frequency domain information, they usually adopt simple concatenation or weighting strategies, lacking effective interaction between spatial and frequency domain features; third, the guiding relationship between multi-level features is insufficient, making it difficult for shallow details, mid-level structures, and contextual enhancement responses to jointly participate in the final reconstruction. Summary of the Invention

[0005] To address the aforementioned technical problems, this invention provides a method and system for fusing infrared and visible light images. Based on multi-level feature extraction, it simultaneously utilizes spatial domain cross-modal complementary information and frequency domain amplitude complementary information, and achieves interactive compensation and modulation of spatial and frequency domain information through a dual-domain serial coupling mechanism, thereby improving the target saliency, structural clarity, and detail preservation capability of the fused image.

[0006] The technical solution adopted in this invention is: Firstly, a method for fusing infrared and visible light images is provided, comprising: S1, acquire the infrared image and visible light image to be fused, and perform shallow feature mapping on the infrared image and visible light image respectively to obtain infrared shallow features and visible light shallow features; S2, performs multi-level residual coding on the shallow infrared features and the shallow visible light features respectively, to obtain infrared features and visible light features at multiple levels; S3. For each level of infrared and visible light features, a spatial domain fusion branch and a frequency domain fusion branch are constructed respectively. The infrared and visible light features of the same level are fused through the spatial domain fusion branch and the frequency domain fusion branch respectively to obtain the spatial domain fusion feature and the frequency domain fusion feature. S4. Perform dual-domain serial coupling processing on the spatial domain fusion features and frequency domain fusion features at the same level to obtain spatial-frequency domain coupled fusion features. S5, cross-layer convergence and channel compression of spatial-frequency domain coupled fusion features at multiple levels, to obtain unified fusion guidance features; S6. Using the unified fusion guidance feature, the infrared and visible light features at the highest level of multiple layers are adaptively modulated to obtain infrared high-level fusion features and visible light high-level fusion features. The infrared high-level fusion features and visible light high-level fusion features are then re-aggregated and reconstructed to obtain high-level re-aggregated features. Finally, the fused image is generated based on the high-level re-aggregated features.

[0007] Furthermore, multi-level residual coding includes ordinary convolutional branches and dilated convolutional branches; S2 performs multi-level residual coding on the shallow infrared and visible light features respectively, resulting in multiple levels of infrared and visible light features, including: S2.1, the infrared shallow features are input into the ordinary convolution branch and the dilated convolution branch. The outputs of the ordinary convolution branch and the dilated convolution branch are then processed by channel fusion and coordinate attention enhancement to obtain the infrared features. S2.2, the visible light shallow layer features are input into the ordinary convolution branch and the dilated convolution branch. The outputs of the ordinary convolution branch and the dilated convolution branch are then processed by channel fusion and coordinate attention enhancement to obtain the visible light features.

[0008] Furthermore, in S3, for each level of infrared and visible light features, a spatial domain fusion branch and a frequency domain fusion branch are constructed respectively. The infrared and visible light features of the same level are fused through the spatial domain fusion branch and the frequency domain fusion branch respectively to obtain the spatial domain fusion feature and the frequency domain fusion feature.

[0009] S3.1, construct a spatial domain fusion branch for infrared and visible light features at each level; S3.2, By fusing infrared and visible light features at the same level through the spatial domain fusion branch, spatial domain fusion features are obtained; S3.3, construct a frequency domain fusion branch for the infrared and visible light features of each level; S3.4, through the frequency domain fusion branch, the infrared features and visible light features at the same level are fused to obtain the frequency domain fused features.

[0010] Furthermore, in S3.2, infrared and visible light features at the same level are fused through a spatial domain fusion branch to obtain spatial domain fused features, including: S3.2.1, will the first Hierarchical visible light characteristics Serialized visible light window features are obtained through block embedding mapping. , will the Hierarchical infrared signature Serialized infrared window features are obtained through block embedding mapping. ; To preset visible light block embedding parameters, Preset infrared block embedding parameters; S3.2.2, based on the characteristics of the visible light window Obtain visible light query features , , To pre-set query weights; based on infrared window characteristics Infrared bond features were obtained and infrared value characteristics , , , Preset key weights, Preset weights; S3.2.3, Based on visible light query characteristics and infrared bond features The first attention response in the infrared to visible light direction was calculated. , ,in, Infrared bond features transpose, This represents the relative position offset matrix. Characterizing visible light query features Location and infrared bond features The relative spatial relationship between positions Indicates the preset scaling factor; S3.2.4, based on infrared window characteristics Obtain infrared query features , Based on the characteristics of the visible light window Obtain visible light bond features and visible light value characteristics , ,

[0011] S3.2.5, Based on infrared query features and visible light bond characteristics The second attention response from visible light to infrared direction was calculated. , ; S3.2.6, based on the first attention response Second attention response Perform fusion computing spatial domain fusion features , , , , For the preset antiblock recovery function, and Preset weights.

[0012] Furthermore, in S3.4, infrared and visible light features at the same level are fused through a frequency domain fusion branch to obtain frequency domain fused features, including: S3.4.1, for the first Hierarchical visible light characteristics and infrared features Perform two-dimensional Fourier transforms to obtain the visible light frequency domain features. and infrared frequency domain characteristics ; S3.4.2, extract infrared frequency domain features respectively and visible light frequency domain characteristics The amplitude component is obtained by obtaining the infrared amplitude component. and visible light amplitude component Infrared frequency domain features were extracted respectively. and visible light frequency domain characteristics The phase components are obtained to obtain the infrared phase components. and visible light phase component ; S3.4.3, convert the infrared amplitude component and visible light amplitude component By performing splicing and channel compression, the initial features of the fused amplitude are obtained. , ; Infrared phase component and visible light phase component By performing splicing and channel compression, the initial features of the fused phase are obtained. ; S3.4.4, Initial characteristics of fusion amplitude and initial features of fusion phase A nonlinear mapping is performed to obtain amplitude enhancement features. and phase enhancement features , , , This represents the amplitude nonlinear mapping function. This represents the first-phase nonlinear mapping function; S3.4.5, based on the initial characteristics of the fusion amplitude With amplitude enhancement features amplitude residuals between Generate a spatial-channel joint weight graph , Represents the Sigmoid function; S3.4.6, using the spatial-channel joint weight graph Amplitude enhancement features Adaptive modulation is performed, and the amplitude mapping function is applied. The fusion amplitude feature is obtained by connecting the residuals. Amplitude mapping function Depend on Convolution, nonlinear activation functions and Convolutional structure; S3.4.7, fuse the initial phase features Phase enhancement features The data is spliced ​​together, and the fused phase features are obtained through phase fusion mapping. , This represents the second-phase nonlinear mapping function; S3.4.8, based on the fusion amplitude characteristics and fusion phase features Reconstructing complex frequency domain features , Represent the imaginary unit; represent the frequency domain characteristics of complex numbers. Inverse Fourier transform is performed to obtain frequency domain fusion features. .

[0013] Furthermore, in S3.4.5, based on the initial characteristics of the fusion amplitude... With amplitude enhancement features amplitude residuals between Generate a spatial-channel joint weight graph ,include: Calculate the initial features of the fusion amplitude With amplitude enhancement features The absolute difference between them yields the amplitude residual. amplitude residual Used to characterize the spectral energy changes of amplitude-dependent features during nonlinear enhancement; For amplitude residuals Perform global average pooling to obtain a description of channel-level amplitude changes. ; Description of channel-level amplitude variation Perform Softmax normalization to obtain the channel weights. ; Using channel weights For amplitude residuals Element-wise modulation is performed to obtain amplitude-guided features. ; This represents element-wise multiplication; Amplitude-guided features Input a 1×1 convolution and a sigmoid function to generate a spatial-channel joint weight map. .

[0014] Furthermore, in S4, the spatial domain fusion features and frequency domain fusion features at the same level are subjected to dual-domain serial coupling processing to obtain spatial-frequency domain coupled fusion features, including: S4.1, according to the first Hierarchical spatial domain fusion features Generate frequency domain fusion features Frequency domain compensation term , ; S4.2, based on the frequency domain compensation term Obtain updated frequency domain fusion features , ; S4.3, based on the updated frequency domain fusion characteristics Generate scale terms and offset terms Spatial domain fusion characteristics Multiplicative scaling and additive enhancement are performed to obtain the updated spatial domain fusion features. ; S4.4, based on the updated frequency domain fusion characteristics and updated spatial domain fusion features Obtain spatial-frequency domain coupled fusion features .

[0015] Furthermore, in step S6, the highest-level infrared and visible light features from multiple layers are adaptively modulated using unified fusion guidance features to obtain infrared high-level fusion features and visible light high-level fusion features. These features are then re-aggregated and reconstructed to produce high-level re-aggregated features. Finally, the fused image is generated based on these high-level re-aggregated features, including: S6.1 Select the highest-level infrared and visible light features from multiple levels as infrared high-level features and visible light high-level features; S6.2, using the unified fusion guidance feature, the infrared high-level feature and the visible light high-level feature are modulated element by element to obtain the modulated infrared high-level feature and the modulated visible light high-level feature. S6.3, the modulated infrared high-level features and the modulated visible light high-level features are spliced ​​together in the channel dimension, and cross-channel coupling and dimensionality reduction are performed by quaternion convolution. Then, multiple quaternion convolution branches with different kernel sizes are used to extract the high-level coupling response at different scales. S6.4, stitches and fuses the high-level coupling responses at all scales to obtain high-level re-aggregation features; S6.5 generates the final fused image based on the high-level re-aggregation features.

[0016] Secondly, a system for fusing infrared and visible light images is provided, comprising: The image feature extraction module is used to acquire the infrared image and the visible light image to be fused, and to perform shallow feature mapping on the infrared image and the visible light image respectively to obtain infrared shallow features and visible light shallow features. The multi-level residual coding module is used to perform multi-level residual coding on infrared shallow features and visible light shallow features respectively, to obtain infrared features and visible light features at multiple levels. The feature fusion module is used to construct spatial domain fusion branches and frequency domain fusion branches for infrared and visible light features at each level, respectively. The infrared and visible light features at the same level are fused through the spatial domain fusion branches and frequency domain fusion branches to obtain spatial domain fused features and frequency domain fused features. The dual-domain serial coupling module is used to perform dual-domain serial coupling processing on spatial domain fusion features and frequency domain fusion features at the same level to obtain spatial-frequency domain coupled fusion features. The multi-level feature unification and fusion module is used to converge and compress spatial-frequency domain coupled features from multiple levels to obtain unified fusion guidance features. The high-level re-aggregation and image reconstruction module is used to adaptively modulate the infrared and visible light features at the highest level among multiple layers using unified fusion guidance features, to obtain infrared high-level fusion features and visible light high-level fusion features. The module then performs high-level re-aggregation and reconstruction on the infrared and visible light high-level fusion features to obtain high-level re-aggregated features, and finally generates the fused image based on the high-level re-aggregated features.

[0017] The beneficial effects achieved by this invention are as follows: S1. Acquire the infrared and visible light images to be fused, and perform shallow feature mapping on the infrared and visible light images respectively to obtain infrared shallow features and visible light shallow features; S2. Perform multi-level residual coding on the infrared shallow features and visible light shallow features respectively to obtain infrared features and visible light features at multiple levels; S3. For each level of infrared and visible light features, construct spatial domain fusion branches and frequency domain fusion branches respectively, and fuse the infrared features and visible light features of the same level through the spatial domain fusion branches and frequency domain fusion branches respectively to obtain spatial domain fused features and frequency domain fused features; S4. For the same... The spatial domain fusion features and frequency domain fusion features of the multiple levels are subjected to dual-domain serial coupling processing to obtain spatial-frequency domain coupled fusion features; S5, the spatial-frequency domain coupled fusion features of multiple levels are converged across layers and channel compressed to obtain unified fusion guiding features; S6, the unified fusion guiding features are used to adaptively modulate the infrared and visible light features of the highest level in multiple levels to obtain infrared high-level fusion features and visible light high-level fusion features. The infrared high-level fusion features and visible light high-level fusion features are then re-aggregated and reconstructed to obtain high-level re-aggregated features, and the final fused image is generated based on the high-level re-aggregated features. Based on multi-level feature extraction, spatial domain cross-modal complementary information and frequency domain amplitude complementary information are used simultaneously, and the interaction compensation and modulation of spatial and frequency domain information are realized through a dual-domain serial coupling mechanism, thereby improving the target saliency, structural clarity and detail preservation ability of the fused image. Attached Figure Description

[0018] Figure 1 This is a flowchart of the method for fusing infrared and visible light images according to the present invention; Figure 2 This is a structural diagram of the infrared and visible light image fusion system of the present invention. Detailed Implementation

[0019] The present invention will be further described below with reference to the accompanying drawings. The following embodiments are only used to more clearly illustrate the technical solution of the present invention, and should not be used to limit the scope of protection of the present invention.

[0020] like Figure 1As shown, this embodiment of the invention provides a method for fusing infrared images and visible light images, including: S1, acquire the infrared image and visible light image to be fused, and perform shallow feature mapping on the infrared image and visible light image respectively to obtain infrared shallow features and visible light shallow features; In this embodiment, infrared image and visible light images First, the shallow features are mapped to the same feature dimension through 3×3 convolution to obtain the infrared shallow features. and visible light shallow features .

[0021] S2, performs multi-level residual coding on the shallow infrared features and the shallow visible light features respectively, to obtain infrared features and visible light features at multiple levels; In this embodiment, multi-level residual coding includes ordinary convolutional branches. and dilated convolution branches In this embodiment, we assume that there are three multi-level layers, with each layer having 32, 64, and 128 channels respectively.

[0022] Regular convolution branches are used to extract local texture features; dilated convolution branches are used to expand the receptive field and extract contextual structural features. S2.1, Infrared shallow layer features The input consists of a regular convolution branch and a dilated convolution branch. The outputs of these two branches are then processed through channel fusion and coordinate attention enhancement to obtain the infrared features. ;exist At that time, the number of channels in the first level was 32. It is the first-level infrared signature; in At that time, the number of channels in the second level was 64. It is a second-level infrared signature; in At that time, the number of channels in the third level was 128. It is the infrared signature of the third level; according to the number of channels in the level, It is the highest level of infrared signature; S2.2, Visible light shallow layer characteristics The input consists of a regular convolution branch and a dilated convolution branch. The outputs of these two branches are then processed through channel fusion and coordinate attention enhancement to obtain the visible light features. .

[0023] S3. For each level of infrared and visible light features, a spatial domain fusion branch and a frequency domain fusion branch are constructed respectively. The infrared and visible light features of the same level are fused through the spatial domain fusion branch and the frequency domain fusion branch respectively to obtain the spatial domain fusion feature and the frequency domain fusion feature. In this embodiment, infrared and visible light features of each level are fused in both the spatial and frequency domains. The fusion process is as follows: S3.1, construct a spatial domain fusion branch for infrared and visible light features at each level; The spatial domain fusion branch has a cross-window interaction structure with heterogeneous dual-path directions. Through local receptive domains in different directions, it performs complementary modeling of cross-modal spatial structures, enhancing the ability to extract the correlation between edges, contours and region layouts in different directions. The spatial domain fusion branch includes a block embedding mapping unit, a cross-window cross attention unit, and an anti-block embedding recovery unit. The block embedding mapping unit is used to convert input features into serialized window features. The cross-window cross attention unit adopts a dual-branch directional heterogeneous form to establish a bidirectional interaction relationship between infrared features and visible light features within a local window. The anti-block embedding recovery unit is used to recover the serialized features into a spatial feature map.

[0024] S3.2, By fusing infrared and visible light features at the same level through the spatial domain fusion branch, spatial domain fusion features are obtained; The specific steps for merging the spatial domain fusion branches are as follows: S3.2.1, will the first Hierarchical visible light characteristics Serialized visible light window features are obtained through block embedding mapping. , will the Hierarchical infrared signature Serialized infrared window features are obtained through block embedding mapping. ; To preset visible light block embedding parameters, Preset infrared block embedding parameters; S3.2.2, based on the characteristics of the visible light window Obtain visible light query features , , To pre-set query weights; based on infrared window characteristics Infrared bond features were obtained and infrared value characteristics , , , Preset key weights, Preset weights; S3.2.3, Based on visible light query characteristics and infrared bond features The first attention response in the infrared to visible light direction was calculated. , ,in, Infrared bond features transpose, This represents the relative position offset matrix. Characterizing visible light query features Location and infrared bond features The relative spatial relationship between positions, and the relative position offset matrix This enables attention computation to simultaneously consider cross-modal content similarity and local spatial location relationships. Indicates the preset scaling factor; S3.2.4, based on infrared window characteristics Obtain infrared query features , Based on the characteristics of the visible light window Obtain visible light bond features and visible light value characteristics , , ; S3.2.5, Based on infrared query features and visible light bond characteristics The second attention response from visible light to infrared direction was calculated. , ; S3.2.6, based on the first attention response Second attention response Perform fusion computing spatial domain fusion features , , , , For the preset antiblock recovery function, and Preset weights.

[0025] S3.3, construct a frequency domain fusion branch for the infrared and visible light features of each level; S3.4, through the frequency domain fusion branch, the infrared features and visible light features at the same level are fused to obtain the frequency domain fused features.

[0026] The specific steps for fusing the frequency domain fusion branch are as follows: S3.4.1, for the first Hierarchical visible light characteristics and infrared features Perform two-dimensional Fourier transforms to obtain the visible light frequency domain features. and infrared frequency domain characteristics ; S3.4.2, extract infrared frequency domain features respectively and visible light frequency domain characteristics The amplitude component is obtained by obtaining the infrared amplitude component. and visible light amplitude component Infrared frequency domain features were extracted respectively. and visible light frequency domain characteristics The phase components are obtained to obtain the infrared phase components. and visible light phase component ; S3.4.3, convert the infrared amplitude component and visible light amplitude component By performing splicing and channel compression, the initial features of the fused amplitude are obtained. , ; Infrared phase component and visible light phase component By performing splicing and channel compression, the initial features of the fused phase are obtained. ; S3.4.4, Initial characteristics of fusion amplitude and initial features of fusion phase A nonlinear mapping is performed to obtain amplitude enhancement features. and phase enhancement features , , , This represents the amplitude nonlinear mapping function. This represents the first-phase nonlinear mapping function; S3.4.5, based on the initial characteristics of the fusion amplitude With amplitude enhancement features amplitude residuals between Generate a spatial-channel joint weight graph , Represents the Sigmoid function; The specific implementation details of this step are as follows: Calculate the initial features of the fusion amplitude With amplitude enhancement features The absolute difference between them yields the amplitude residual. amplitude residual Used to characterize the spectral energy changes of amplitude-dependent features during nonlinear enhancement; For amplitude residuals Perform global average pooling to obtain a description of channel-level amplitude changes. ; Description of channel-level amplitude variation Perform Softmax normalization to obtain the channel weights. ; Using channel weights For amplitude residuals Element-wise modulation is performed to obtain amplitude-guided features. ; This represents element-wise multiplication; Amplitude-guided features Input a 1×1 convolution and a sigmoid function to generate a spatial-channel joint weight map. .

[0027] S3.4.6, using the spatial-channel joint weight graph Amplitude enhancement features Adaptive modulation is performed, and the amplitude mapping function is applied. The fusion amplitude feature is obtained by connecting the residuals. Amplitude mapping function Depend on Convolution, nonlinear activation functions and Convolutional structure; S3.4.7, fuse the initial phase features Phase enhancement features The data is spliced ​​together, and the fused phase features are obtained through phase fusion mapping. , This represents the second-phase nonlinear mapping function; S3.4.8, based on the fusion amplitude characteristics and fusion phase features Reconstructing complex frequency domain features , Represent the imaginary unit; represent the frequency domain characteristics of complex numbers. Inverse Fourier transform is performed to obtain frequency domain fusion features. .

[0028] S4. Perform dual-domain serial coupling processing on the spatial domain fusion features and frequency domain fusion features at the same level to obtain spatial-frequency domain coupled fusion features. In this embodiment, the spatial domain fusion features and frequency domain fusion features are not simply concatenated, but are input into the dual-domain serial coupling module for interaction. The specific steps of the dual-domain serial coupling processing are as follows: S4.1, according to the first Hierarchical spatial domain fusion features Generate frequency domain fusion features Frequency domain compensation term , ; S4.2, based on the frequency domain compensation term Obtain updated frequency domain fusion features , ; S4.3, based on the updated frequency domain fusion characteristics Generate scale terms and offset terms Spatial domain fusion characteristics Multiplicative scaling and additive enhancement are performed to obtain the updated spatial domain fusion features. ; Scale item This represents the compensation mapping of the spatial branch to the frequency domain branch; S4.4, based on the updated frequency domain fusion characteristics and updated spatial domain fusion features Obtain spatial-frequency domain coupled fusion features .

[0029] S5, cross-layer convergence and channel compression of spatial-frequency domain coupled fusion features at multiple levels, to obtain unified fusion guidance features; In this embodiment, the spatial-frequency domain coupled fusion features of the three levels are concatenated into channels, and then compressed to 128 channels using 1×1 convolution to obtain a unified fusion guidance feature. ; Unified integration guidance features It also includes different levels of texture, structure, and contextual response, thus enabling adaptive modulation of infrared high-level features and visible light high-level features.

[0030] S6. Using the unified fusion guidance feature, the infrared and visible light features at the highest level of multiple layers are adaptively modulated to obtain infrared high-level fusion features and visible light high-level fusion features. The infrared high-level fusion features and visible light high-level fusion features are then re-aggregated and reconstructed to obtain high-level re-aggregated features. Finally, the fused image is generated based on the high-level re-aggregated features.

[0031] In this embodiment, taking three levels as an example, in conjunction with step S2, multiple levels are used. and These are the first-level infrared and visible light characteristics; and These are the second-level infrared and visible light characteristics; and It consists of third-level infrared and visible light characteristics.

[0032] S6.1 Select the highest-level infrared and visible light features from multiple levels as infrared high-level features and visible light high-level features; Of the three levels, the third level is the highest, so the infrared signature... and visible light characteristics As infrared high-level features and visible light high-level features.

[0033] S6.2, using the unified fusion guidance feature, the infrared high-level feature and the visible light high-level feature are modulated element by element to obtain the modulated infrared high-level feature and the modulated visible light high-level feature. Leveraging unified integration to guide features infrared high-altitude features Element-wise modulation was performed to obtain modulated infrared high-level features. ; Leveraging unified integration to guide features Visible light high-level features Element-wise modulation was performed to obtain modulated visible light high-level features. .

[0034] S6.3, the modulated infrared high-level features and the modulated visible light high-level features are spliced ​​together in the channel dimension, and cross-channel coupling and dimensionality reduction are performed by quaternion convolution. Then, multiple quaternion convolution branches with different kernel sizes are used to extract the high-level coupling response at different scales. Modulate infrared high-level features and modulation of visible light high-level features The concatenation is performed along the channel dimension, and cross-channel coupling and dimensionality reduction are achieved through quaternion convolution: ; Then, multiple quaternion convolution branches with different kernel sizes are used to extract the high-level coupling response at different scales. In this embodiment, three quaternion convolution branches with different kernel sizes are specifically used. express Quaternion convolution branches with varying kernel sizes express Quaternion convolution branches with varying kernel sizes express Quaternion convolution branch with kernel size; the expression for the high-level coupling response is... , , .

[0035] S6.4, stitches and fuses the high-level coupling responses at all scales to obtain high-level re-aggregation features; In step S6.3 , and To splice together, and through By fusing convolution and nonlinear activation, we obtain ; Then through quaternions Convolution outputs high-level re-aggregated features .

[0036] S6.5 generates the final fused image based on the high-level re-aggregation features.

[0037] Targeting high-level re-aggregation characteristics The multi-scale feature fusion unit (IFP) at each scale can be abstracted as follows: ; in, This represents the input to IFP. This indicates the output of IFP. Its function is to compress eigenvalues ​​into an interval. ; The final fused image can be represented as: .

[0038] The beneficial effects achieved by the embodiments of the present invention are as follows: S1. Acquire the infrared and visible light images to be fused, and perform shallow feature mapping on the infrared and visible light images respectively to obtain infrared shallow features and visible light shallow features; S2. Perform multi-level residual coding on the infrared shallow features and visible light shallow features respectively to obtain infrared features and visible light features at multiple levels; S3. For each level of infrared and visible light features, construct spatial domain fusion branches and frequency domain fusion branches respectively, and fuse the infrared features and visible light features of the same level through the spatial domain fusion branches and frequency domain fusion branches respectively to obtain spatial domain fused features and frequency domain fused features; S4. For the same... The spatial domain fusion features and frequency domain fusion features of the multiple levels are subjected to dual-domain serial coupling processing to obtain spatial-frequency domain coupled fusion features; S5, the spatial-frequency domain coupled fusion features of multiple levels are converged across layers and channel compressed to obtain unified fusion guiding features; S6, the unified fusion guiding features are used to adaptively modulate the infrared and visible light features of the highest level in multiple levels to obtain infrared high-level fusion features and visible light high-level fusion features. The infrared high-level fusion features and visible light high-level fusion features are then re-aggregated and reconstructed to obtain high-level re-aggregated features, and the final fused image is generated based on the high-level re-aggregated features. Based on multi-level feature extraction, spatial domain cross-modal complementary information and frequency domain amplitude complementary information are used simultaneously, and the interaction compensation and modulation of spatial and frequency domain information are realized through a dual-domain serial coupling mechanism, thereby improving the target saliency, structural clarity and detail preservation ability of the fused image.

[0039] Based on the fusion method for infrared and visible light images described in the above embodiments, the fusion system for infrared and visible light images will be described below through embodiments.

[0040] like Figure 2 As shown, this embodiment of the invention provides a system for fusing infrared and visible light images, comprising: The image feature extraction module 201 is used to acquire the infrared image and the visible light image to be fused, and to perform shallow feature mapping on the infrared image and the visible light image respectively to obtain infrared shallow features and visible light shallow features. The multi-level residual coding module 202 is used to perform multi-level residual coding on the infrared shallow features and the visible light shallow features respectively, so as to obtain infrared features and visible light features at multiple levels. The feature fusion module 203 is used to construct a spatial domain fusion branch and a frequency domain fusion branch for each level of infrared and visible light features, respectively, and fuse the infrared and visible light features of the same level through the spatial domain fusion branch and the frequency domain fusion branch to obtain spatial domain fused features and frequency domain fused features. The dual-domain serial coupling module 204 is used to perform dual-domain serial coupling processing on spatial domain fusion features and frequency domain fusion features at the same level to obtain spatial-frequency domain coupled fusion features. The multi-level feature unification and fusion module 205 is used to perform cross-layer convergence and channel compression of spatial-frequency domain coupled fusion features from multiple levels to obtain unified fusion guidance features. The high-level re-aggregation and image reconstruction module 206 is used to adaptively modulate the infrared and visible light features at the highest level among multiple layers using unified fusion guidance features to obtain infrared high-level fusion features and visible light high-level fusion features. The high-level re-aggregation and reconstruction output of the infrared high-level fusion features and visible light high-level fusion features are then performed to obtain high-level re-aggregated features, and the final fused image is generated based on the high-level re-aggregated features.

[0041] The method and system for fusing infrared and visible light images described above can simultaneously utilize spatial domain cross-modal complementary information and frequency domain amplitude complementary information based on multi-level feature extraction, and achieve interactive compensation and modulation of spatial and frequency domain information through a dual-domain serial coupling mechanism, thereby improving the target saliency, structural clarity and detail preservation ability of the fused image.

[0042] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0043] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0044] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0045] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0046] The above are merely embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention are included within the scope of the claims of the present invention pending approval.

Claims

1. A method for fusing infrared images and visible light images, characterized in that, include: S1, acquire the infrared image and visible light image to be fused, and perform shallow feature mapping on the infrared image and visible light image respectively to obtain infrared shallow features and visible light shallow features; S2, performs multi-level residual coding on the shallow infrared features and the shallow visible light features respectively, to obtain infrared features and visible light features at multiple levels; S3. For each level of infrared and visible light features, a spatial domain fusion branch and a frequency domain fusion branch are constructed respectively. The infrared and visible light features of the same level are fused through the spatial domain fusion branch and the frequency domain fusion branch respectively to obtain the spatial domain fusion feature and the frequency domain fusion feature. S4. Perform dual-domain serial coupling processing on the spatial domain fusion features and frequency domain fusion features at the same level to obtain spatial-frequency domain coupled fusion features. S5, cross-layer convergence and channel compression of spatial-frequency domain coupled fusion features at multiple levels, to obtain unified fusion guidance features; S6. Using the unified fusion guidance feature, the infrared and visible light features at the highest level of multiple layers are adaptively modulated to obtain infrared high-level fusion features and visible light high-level fusion features. The infrared high-level fusion features and visible light high-level fusion features are then re-aggregated and reconstructed to obtain high-level re-aggregated features. Finally, the fused image is generated based on the high-level re-aggregated features.

2. The method for fusing infrared and visible light images according to claim 1, characterized in that, Multi-level residual coding includes ordinary convolution branches and dilated convolution branches; S2 performs multi-level residual coding on the shallow infrared and visible light features respectively, resulting in multiple levels of infrared and visible light features, including: S2.1, the infrared shallow features are input into the ordinary convolution branch and the dilated convolution branch. The outputs of the ordinary convolution branch and the dilated convolution branch are then processed by channel fusion and coordinate attention enhancement to obtain the infrared features. S2.2, the visible light shallow layer features are input into the ordinary convolution branch and the dilated convolution branch. The outputs of the ordinary convolution branch and the dilated convolution branch are then processed by channel fusion and coordinate attention enhancement to obtain the visible light features.

3. The method for fusing infrared and visible light images according to claim 1, characterized in that, S3, for each level of infrared and visible light features, construct spatial domain fusion branches and frequency domain fusion branches respectively. Then, fuse the infrared and visible light features of the same level through these branches to obtain spatial domain fused features and frequency domain fused features, including: S3.1, construct a spatial domain fusion branch for infrared and visible light features at each level; S3.2, By fusing infrared and visible light features at the same level through the spatial domain fusion branch, spatial domain fusion features are obtained; S3.3, construct a frequency domain fusion branch for the infrared and visible light features of each level; S3.4, through the frequency domain fusion branch, the infrared features and visible light features at the same level are fused to obtain the frequency domain fused features.

4. The method for fusing infrared and visible light images according to claim 3, characterized in that, S3.2, by fusing infrared and visible light features at the same level through a spatial domain fusion branch, spatial domain fused features are obtained, including: S3.2.1, will the first Hierarchical visible light characteristics Serialized visible light window features are obtained through block embedding mapping. , will the Hierarchical infrared signature Serialized infrared window features are obtained through block embedding mapping. ; To preset visible light block embedding parameters, Preset infrared block embedding parameters; S3.2.2, based on the characteristics of the visible light window Obtain visible light query features , , To pre-set query weights; based on infrared window characteristics Infrared bond features were obtained and infrared value characteristics , , , Preset key weights, Preset weights; S3.2.3, Based on visible light query characteristics and infrared bond features The first attention response in the infrared to visible light direction was calculated. , ,in, Infrared bond features transpose, This represents the relative position offset matrix. Characterizing visible light query features Location and infrared bond features The relative spatial relationship between positions Indicates the preset scaling factor; S3.2.4, based on infrared window characteristics Obtain infrared query features , Based on the characteristics of the visible light window Obtain visible light bond features and visible light value characteristics , , ; S3.2.5, Based on infrared query features and visible light bond features The second attention response from visible light to infrared direction was calculated. , ; S3.2.6, based on the first attention response Second attention response Perform fusion computing spatial domain fusion features , , , , For the preset antiblock recovery function, and Preset weights.

5. The method for fusing infrared and visible light images according to claim 3, characterized in that, S3.4, through the frequency domain fusion branch, the infrared and visible light features at the same level are fused to obtain the frequency domain fused features, including: S3.4.1, for the first Hierarchical visible light characteristics and infrared features Perform two-dimensional Fourier transforms to obtain the visible light frequency domain features. and infrared frequency domain characteristics ; S3.4.2, extract infrared frequency domain features respectively and visible light frequency domain characteristics The amplitude component is obtained by obtaining the infrared amplitude component. and visible light amplitude component Infrared frequency domain features were extracted respectively. and visible light frequency domain characteristics The phase components are obtained to obtain the infrared phase components. and visible light phase component ; S3.4.3, convert the infrared amplitude component and visible light amplitude component By performing splicing and channel compression, the initial features of the fused amplitude are obtained. , ; Infrared phase component and visible light phase component By performing splicing and channel compression, the initial features of the fused phase are obtained. ; S3.4.4, Initial characteristics of fusion amplitude and initial features of fusion phase A nonlinear mapping is performed to obtain amplitude enhancement features. and phase enhancement features , , , This represents the amplitude nonlinear mapping function. This represents the first-phase nonlinear mapping function; S3.4.5, based on the initial characteristics of the fusion amplitude With amplitude enhancement features amplitude residuals between Generate a spatial-channel joint weight graph , Represents the Sigmoid function; S3.4.6, using the spatial-channel joint weight graph Amplitude enhancement features Adaptive modulation is performed, and the amplitude mapping function is applied. The fusion amplitude feature is obtained by connecting the residuals. Amplitude mapping function Depend on Convolution, nonlinear activation functions and Convolutional structure; S3.4.7, fuse the initial phase features Phase enhancement features The data is spliced ​​together, and the fused phase features are obtained through phase fusion mapping. , This represents the second-phase nonlinear mapping function; S3.4.8, based on the fusion amplitude characteristics and fusion phase features Reconstructing complex frequency domain features , Represent the imaginary unit; represent the frequency domain characteristics of complex numbers. Inverse Fourier transform is performed to obtain frequency domain fusion features. .

6. The method for fusing infrared and visible light images according to claim 5, characterized in that, S3.4.5, based on the initial characteristics of the fusion amplitude With amplitude enhancement features amplitude residuals between Generate a spatial-channel joint weight graph ,include: Calculate the initial features of the fusion amplitude With amplitude enhancement features The absolute difference between them yields the amplitude residual. amplitude residual Used to characterize the spectral energy changes of amplitude-dependent features during nonlinear enhancement; For amplitude residuals Perform global average pooling to obtain a description of channel-level amplitude changes. ; Description of channel-level amplitude variation Perform Softmax normalization to obtain the channel weights. ; Using channel weights For amplitude residuals Element-wise modulation is performed to obtain amplitude-guided features. ; This represents element-wise multiplication; Amplitude-guided features Input a 1×1 convolution and a sigmoid function to generate a spatial-channel joint weight map. .

7. The method for fusing infrared and visible light images according to claim 1, characterized in that, S4, perform dual-domain serial coupling processing on the spatial domain fusion features and frequency domain fusion features at the same level to obtain spatial-frequency domain coupled fusion features, including: S4.1, according to the first Hierarchical spatial domain fusion features Generate frequency domain fusion features Frequency domain compensation term , ; S4.2, based on the frequency domain compensation term Obtain updated frequency domain fusion features , ; S4.3, based on the updated frequency domain fusion features Generate scale terms and offset terms Spatial domain fusion characteristics Multiplicative scaling and additive enhancement are performed to obtain the updated spatial domain fusion features. ; S4.4, based on the updated frequency domain fusion characteristics and updated spatial domain fusion features Obtain spatial-frequency domain coupled fusion features .

8. The method for fusing infrared and visible light images according to claim 1, characterized in that, S6, using unified fusion guided features, adaptively modulates the highest-level infrared and visible light features across multiple layers to obtain infrared high-level fusion features and visible light high-level fusion features. High-level re-aggregation and reconstruction are then performed on these infrared and visible light high-level fusion features to obtain high-level re-aggregated features. Finally, the fused image is generated based on these high-level re-aggregated features, including: S6.1 Select the highest-level infrared and visible light features from multiple levels as infrared high-level features and visible light high-level features; S6.2, using the unified fusion guidance feature, the infrared high-level feature and the visible light high-level feature are modulated element by element to obtain the modulated infrared high-level feature and the modulated visible light high-level feature. S6.3, the modulated infrared high-level features and the modulated visible light high-level features are spliced ​​together in the channel dimension, and cross-channel coupling and dimensionality reduction are performed by quaternion convolution. Then, multiple quaternion convolution branches with different kernel sizes are used to extract the high-level coupling response at different scales. S6.4, stitches and fuses the high-level coupling responses at all scales to obtain high-level re-aggregation features; S6.5 generates the final fused image based on the high-level re-aggregation features.

9. A system for fusing infrared and visible light images, characterized in that, include: The image feature extraction module is used to acquire the infrared image and the visible light image to be fused, and to perform shallow feature mapping on the infrared image and the visible light image respectively to obtain infrared shallow features and visible light shallow features. The multi-level residual coding module is used to perform multi-level residual coding on infrared shallow features and visible light shallow features respectively, to obtain infrared features and visible light features at multiple levels. The feature fusion module is used to construct spatial domain fusion branches and frequency domain fusion branches for infrared and visible light features at each level, respectively. The infrared and visible light features at the same level are fused through the spatial domain fusion branches and frequency domain fusion branches to obtain spatial domain fused features and frequency domain fused features. The dual-domain serial coupling module is used to perform dual-domain serial coupling processing on spatial domain fusion features and frequency domain fusion features at the same level to obtain spatial-frequency domain coupled fusion features. The multi-level feature unification and fusion module is used to converge and compress spatial-frequency domain coupled features from multiple levels to obtain unified fusion guidance features. The high-level re-aggregation and image reconstruction module is used to adaptively modulate the infrared and visible light features at the highest level among multiple layers using unified fusion guidance features, to obtain infrared high-level fusion features and visible light high-level fusion features. The module then performs high-level re-aggregation and reconstruction on the infrared and visible light high-level fusion features to obtain high-level re-aggregated features, and finally generates the fused image based on the high-level re-aggregated features.

Images