Infrared structure prior-driven space-frequency state space collaborative image defogging method

Abstract

The application discloses an infrared structure prior-driven space-frequency state space collaborative image defogging method, and belongs to the technical field of digital image processing and computer vision. The method comprises the following steps: acquiring a registered foggy RGB image and an infrared image, and generating visible light features and infrared features; the two are spliced to obtain initial features of a fusion branch; single-mode features are jointly enhanced in a spatial domain and a frequency domain; a cross-guided attention module (CGA) is used to realize complementary guidance between the RGB and infrared features; a bidirectional cross-modal Mamba fusion module (BICM) is used to model an insert-type joint sequence of the RGB, infrared and fusion features; and finally, a clear RGB image after defogging is output through a convolution reconstruction head. The application can effectively improve the problems of reduced contrast, missing texture, blurred structure and color distortion in foggy visible light images, has strong structure recovery capability, good detail preservation effect and fully utilized cross-modal information.

Description

Technical Field

[0001] This invention belongs to the field of digital image processing and computer vision technology, and more specifically, it relates to an infrared structure prior-driven spatial frequency state-based collaborative image dehazing method. Background Technology

[0002] Image dehazing is a crucial issue in fields such as image restoration, intelligent transportation, nighttime surveillance, autonomous driving, security patrol, and remote sensing. In hazy weather, visible light images are significantly affected by atmospheric scattering, typically exhibiting problems such as decreased contrast, color shift, blurred distant targets, loss of texture details, and degraded edge structures. These issues consequently impact the reliability of subsequent target detection, semantic understanding, and visual perception tasks.

[0003] Existing single-image visible light dehazing methods mainly rely on the color, brightness, and texture cues of the RGB image itself for degradation modeling. While these methods can achieve some restoration results in synthetic data or lightly foggy scenes, they still have significant shortcomings in real-world, complex foggy scenes: First, fog degradation exhibits spatial non-uniformity, making it difficult for a single visible light branch to accurately distinguish between real object textures and fog occlusion; second, high-frequency details in distant and low-contrast regions are severely suppressed, making stable restoration difficult using only spatial domain convolution or ordinary attention; third, common fusion methods often only emphasize visual enhancement, without specifically modeling color consistency, structure preservation, and cross-modal complementary information utilization in dehazing tasks.

[0004] Infrared images are less affected by visible light scattering and can provide relatively stable target outlines, thermal radiation responses, and structural boundary information in foggy scenes. Therefore, introducing infrared modes as auxiliary information into the visible light dehazing process can help compensate for the structural loss of RGB images in heavily foggy regions. However, infrared images lack color information and have different modal distributions than visible light images. Simple stitching or direct addition can easily introduce artifacts, structural mismatches, or color distortion. Summary of the Invention

[0005] This invention addresses the technical problems existing in the prior art by providing a spatially coordinated image dehazing method based on infrared structure prior driving spatial frequency state.

[0006] To solve the above-mentioned technical problems, the present invention provides the following technical solution: A spatially coordinated image dehazing method driven by infrared structure priors, comprising the following steps: S1. Obtain the registered visible light hazy image and infrared image and perform shallow feature extraction. Shallow feature extraction is performed on the visible light hazy image and the infrared image respectively to obtain the corresponding visible light shallow features and infrared shallow features. The visible light shallow features and infrared shallow features are concatenated along the channel dimension and then processed by channel alignment mapping to obtain the initial fusion features for use by the subsequent multi-level enhancement network. The initial fusion features integrate the color texture information of visible light and the structural information of infrared image. S2. Constructing Spatial Frequency Domain Mamba Block SFMB Enhanced Single-Mode Features After normalizing the input features, multi-scale spatial enhancement and frequency domain amplitude and phase enhancement are obtained respectively. Then, learnable coefficients are used to superimpose the two enhanced features back into the original features in the form of residuals. S3. Calibrate the amplitude and phase spectra using the frequency domain amplitude and phase block FAB. The frequency domain amplitude-phase block (FAB) transforms the input features to the frequency domain, separates the amplitude spectrum and the phase spectrum, performs residual modulation on the amplitude spectrum to enhance the texture and residual correction on the phase spectrum to preserve structural information, reconstructs the spectrum and performs inverse Fourier transform to return to the spatial domain, and then performs residual fusion and normalization to output the result, thereby achieving high-frequency texture restoration while maintaining spatial positioning. S4. Modeling long-range spatial dependencies using adaptive multi-scale Mamba blocks (AMMB) Adaptive multi-scale Mamba block AMMB generates gated features to regulate the information flow. Local, mesoscale, and global spatial features are extracted through multiple parallel scale branches. Multi-scale features are adaptively fused using learnable weights, and then output through gated modulation and residual connection to achieve simultaneous capture of local texture, mesoscale structure, and global fog distribution. S5. Achieve mutual guidance between RGB and infrared features through the Cross-Guided Attention Module (CGA). The Cross-Guided Attention (CGA) module is set after each level of encoding to achieve bidirectional cross-modal guidance between RGB and infrared. The guidance weight is generated by the counterpart modality to modulate the local feature, and then the updated dual-modal feature is output after cross-channel information integration. Multi-level guidance reduces feature mismatch caused by modal differences. S6. Deep joint modeling using the bidirectional cross-modal Mamba fusion module BICM BICM interleaves RGB, infrared, and the previous layer fused features into a one-dimensional sequence according to their spatial positions, uses bidirectional Mamba to model long-range dependencies, learns modality fusion weights, processes them, and then switches back to image features, outputting new fused features from the residuals, thus realizing explicit long-range information interaction between multiple modalities. S7. Construct a multi-level infrared-assisted defogging network and output a clear image. The network consists of multi-level stacked modules for single-modal enhancement, cross-modal mutual guidance, and deep fusion. Each level sequentially performs bi-branch single-modal enhancement, cross-modal bi-directional guidance, and deep fusion of the fusion branch. After multi-level iterations, the fused deep features are added to the residuals of the RGB deep features, and the dehazed image is output after reconstruction mapping.

[0007] Preferably, S1 includes: acquiring a foggy RGB image. With infrared images ,in, , H and W represent the image height and width, respectively; and Spatially register images so that pixels or features at the same spatial location have a corresponding relationship; The foggy RGB image and infrared image are respectively input into the shallow feature extraction module consisting of two convolutional layers to obtain the visible light shallow features. and infrared shallow features :

[0008] In the formula, and These represent the shallow convolution mappings for the RGB branch and the infrared branch, respectively.

[0009] Preferably, and Splicing is performed according to channel dimensions, and through... Convolution obtains initial features of the fusion branch :

[0010] In the formula, [·,·] represents the channel splicing operation. The channel alignment mapping represents the merge branch; Color texture information, infrared structure information, and fusion priors are simultaneously introduced into subsequent multi-level enhancement networks.

[0011] Preferably, S2 includes: in the RGB branch, infrared branch and fusion branch, spatial frequency domain Mamba block (SFMB) is used to perform single-modal enhancement on the input features; Let the input features at any level be... First, perform layer normalization, then input adaptive multi-scale Mamba block (AMMB) and frequency domain amplitude-phase block (FAB) respectively:

[0012]

[0013] In the formula, Indicates spatial domain enhancement features, Indicates frequency domain enhancement features; Then, the two types of features are fused using a residual method to obtain the SFMB output:

[0014] In the formula, α and β are learnable fusion coefficients used to adaptively adjust the contributions of spatial domain enhancement and frequency domain enhancement to the final features.

[0015] Preferably, S3 includes: processing input features Performing a two-dimensional Fourier transform yields the complex spectrum representation:

[0016] In the formula, A = , representing the amplitude spectrum, P = , represents the phase spectrum, and j represents the imaginary unit; the amplitude spectrum mainly reflects the energy intensity of different frequency components and is related to texture, contrast and detail recovery; the phase spectrum can preserve spatial structure, edge contours and geometric position information; For the amplitude branch, input A with a 1×1 convolution, ReLU activation, and a 3×3 depthwise separable convolution, and generate amplitude modulation weights using a sigmoid function. : ; Then, the calibrated amplitude spectrum A′ is obtained by residual modulation: ; For the phase branch, P is input to a 3×3 convolution, ReLU activation, and a 3×3 depthwise separable convolution, and the phase residual is generated using Tanh. : ; Then, the calibrated phase spectrum is obtained using residual correction. :

[0017] In the formula, η is the phase perturbation coefficient, which can be set as a learnable parameter; Then, the complex spectrum is reconstructed based on A′ and P′, and the spatial domain frequency enhancement features are recovered through inverse Fourier transform: ; Finally, Residual fusion with input features and layer normalization output: .

[0018] Preferably, S4 includes: Let the input feature be... AMMB first constructs gated branches to control the throughput of multi-scale spatial information:

[0019] In the formula, σ represents a linear mapping, and σ represents the Sigmoid function; Then, three-scale spatial modeling branches are constructed. The first scale maintains the original resolution and uses linear mapping and... Local structure modeling is performed using depthwise convolution; the second and third scales are downsampled by the encoder, then modeled using 3×3 depthwise convolution, 2D-SSM, and layer normalization to model long-range spatial dependencies, and finally restored to the original scale by the decoder; this process can be abstractly represented as:

[0020]

[0021]

[0022] In the formula, and These represent the encoding and decoding mappings at the k-th scale, respectively. This represents a two-dimensional selective state-space modeling layer.

[0023] Preferably, an adaptive fusion method is used to aggregate multi-scale spatial features:

[0024] In the formula, θ is the learnable scale weight; Finally, the fused multi-scale features and gated features are modulated element-wise, and the AMMB output is obtained through linear mapping and residual connection:

[0025] In the formula, λ is the residual scaling parameter. This is a linear mapping for the output.

[0026] Preferably, S5 includes: after each encoding stage, processing the RGB branch features and infrared branching features The input cross-guided attention module (CGA) enables infrared structural information to guide the enhancement of visible light features, and inversely modulates infrared features with visible light texture and color-related information. First, global max pooling and global average pooling are performed on any modal feature to obtain cross-modal guidance statistics; the infrared guidance RGB channel-spatial guidance weights can be expressed as:

[0027] In the formula, MaxP and AvgP represent max pooling and average pooling, respectively. Represents a convolutional mapping; Then, guided weights are used to modulate the RGB features: ; It can also be generated from RGB features. And modulate infrared features ; Finally, the modulated two-mode features are spliced, segmented, and sequentially scanned, and then input into S6Blocks and Merge units for cross-channel information integration to obtain updated RGB and infrared features: .

[0028] Preferably, S6 includes: Let the i-th level input include RGB features Infrared features and fusion branch features ; First, the three features are organized into an insertion joint sequence according to their spatial location:

[0029] In the formula, Insert means interleaving the RGB token, infrared token, and fused token at the same spatial location to form a joint sequence; Then, average pooling and max pooling are performed on the joint sequence, and cross-modal fusion weights are generated by convolution and Sigmoid; simultaneously, the joint sequence is flattened and input into forward S6Blocks and backward S6Blocks respectively: ; The bidirectional sequence modeling results are merged, and the spatial features corresponding to each modality or fusion branch are restored through a segmentation operation. Then, element-wise weighted fusion and residual fusion are performed with the input features to obtain the current level fusion output: .

[0030] Preferably, S7 includes: for the i-th level network, the RGB branch and the infrared branch are first enhanced by SFMB, and then cross-modal mutual guidance is performed by CGA; The fusion branch updates its own features through SFMB and performs deep fusion with the current-level RGB and infrared features using BICM; the process can be represented as follows: ; ; ; After N iterations, the final RGB deep features are obtained. Infrared deep features and fusion features ; The fused features are added to the residuals of the RGB deep features and then input into the reconstruction convolution head to output a clear, dehazed image J.

[0031] In the formula, ψ represents the image reconstruction mapping, and the output J∈ .

[0032] Compared with the prior art, the present invention has the following beneficial effects: This invention provides a spatial collaborative image dehazing method driven by infrared structure priors. The method constructs a three-way collaborative framework consisting of an RGB branch, an infrared branch, and a fusion branch. First, shallow feature extraction is performed on both the hazy visible light image and the infrared image, and initial features for the fusion branch are generated through channel stitching. Second, a spatial frequency domain Mamba block (SFMB) is designed. In the spatial domain, an adaptive multi-scale Mamba block (AMMB) captures local structure and long-range dependencies, while in the frequency domain, a frequency domain amplitude-phase block (FAB) performs collaborative calibration of the amplitude and phase spectra. Then, a cross-guided attention module (CGA) is designed to allow infrared structure information and visible light texture and color information to interact progressively at each encoding stage. A bidirectional cross-modal Mamba fusion module (BICM) is designed to organize RGB, infrared, and fusion features into an interpolated joint sequence, and cross-modal long-range fusion is achieved through forward and backward S6 blocks. Finally, a reconstruction module outputs a clear, dehazed visible light image. This invention utilizes stable structural information provided by infrared images to assist visible light defogging, enhancing the contour recovery capability of heavily foggy areas. It employs dual-path collaborative modeling in both the spatial and frequency domains, simultaneously considering global fog distribution, local edge structures, and high-frequency texture details. Furthermore, it achieves lightweight cross-modal guidance and deep cross-modal fusion through CGA and BICM respectively, avoiding information mixing caused by simple stitching. The S6 / Mamba mechanism is used to model long-range dependencies with low computational complexity, making it suitable for complex outdoor fog and haze scenarios. Attached Figure Description

[0033] To more clearly illustrate the technical solutions in the embodiments of this application, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0034] Figure 1This is a schematic diagram of the overall framework structure of a multimodal infrared-assisted defogging network according to one embodiment of the present invention; Figure 2 This is a schematic diagram of the frequency domain amplitude-phase block (FAB) structure according to one embodiment of the present invention; Figure 3 This is a schematic diagram of an adaptive multi-scale Mamba block (AMMB) structure according to one embodiment of the present invention; Figure 4 This is a schematic diagram of the Cross-Guided Attention Module (CGA) structure according to one embodiment of the present invention; Figure 5 This is a schematic diagram of the bidirectional cross-modal Mamba fusion module (BICM) structure according to one embodiment of the present invention; Figure 6 This is a comparison of the input and output defogging effects in one embodiment of the present invention. Figure 1 ; Figure 7 This is a comparison of the input and output defogging effects in one embodiment of the present invention. Figure 2 ; Figure 8 This is a comparison of the input and output defogging effects in one embodiment of the present invention. Figure 3 . Detailed Implementation

[0035] To make the technical problems, technical solutions, and beneficial effects to be solved by this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and are not intended to limit the scope of this application.

[0036] Example 1 Please see Figure 1 This invention provides a spatial cooperative image dehazing method based on infrared structure prior driving spatial frequency state, comprising the following steps: S1. Obtain the registered visible light hazy image and infrared image and perform shallow feature extraction. Shallow feature extraction is performed on the visible light hazy image and the infrared image respectively to obtain the corresponding visible light shallow features and infrared shallow features. The visible light shallow features and infrared shallow features are concatenated along the channel dimension and then processed by channel alignment mapping to obtain the initial fusion features for use by the subsequent multi-level enhancement network. The initial fusion features integrate the color texture information of visible light and the structural information of infrared image. S2. Constructing Spatial Frequency Domain Mamba Block SFMB Enhanced Single-Mode Features After normalizing the input features, multi-scale spatial enhancement and frequency domain amplitude and phase enhancement are obtained respectively. Then, learnable coefficients are used to superimpose the two enhanced features back into the original features in the form of residuals. S3. Calibrate the amplitude and phase spectra using the frequency domain amplitude and phase block FAB. The frequency domain amplitude-phase block (FAB) transforms the input features to the frequency domain, separates the amplitude spectrum and the phase spectrum, performs residual modulation on the amplitude spectrum to enhance the texture and residual correction on the phase spectrum to preserve structural information, reconstructs the spectrum and performs inverse Fourier transform to return to the spatial domain, and then performs residual fusion and normalization to output the result, thereby achieving high-frequency texture restoration while maintaining spatial positioning. S4. Modeling long-range spatial dependencies using adaptive multi-scale Mamba blocks (AMMB) Adaptive multi-scale Mamba block AMMB generates gated features to regulate the information flow. Local, mesoscale, and global spatial features are extracted through multiple parallel scale branches. Multi-scale features are adaptively fused using learnable weights, and then output through gated modulation and residual connection to achieve simultaneous capture of local texture, mesoscale structure, and global fog distribution. S5. Achieve mutual guidance between RGB and infrared features through the Cross-Guided Attention Module (CGA). The Cross-Guided Attention (CGA) module is set after each level of encoding to achieve bidirectional cross-modal guidance between RGB and infrared. The guidance weight is generated by the counterpart modality to modulate the local feature, and then the updated dual-modal feature is output after cross-channel information integration. Multi-level guidance reduces feature mismatch caused by modal differences. S6. Deep joint modeling using the bidirectional cross-modal Mamba fusion module BICM BICM interleaves RGB, infrared, and the previous layer fused features into a one-dimensional sequence according to their spatial positions, uses bidirectional Mamba to model long-range dependencies, learns modality fusion weights, processes them, and then switches back to image features, outputting new fused features from the residuals, thus realizing explicit long-range information interaction between multiple modalities. S7. Construct a multi-level infrared-assisted defogging network and output a clear image. The network consists of multi-level stacked modules for single-modal enhancement, cross-modal mutual guidance, and deep fusion. Each level sequentially performs bi-branch single-modal enhancement, cross-modal bi-directional guidance, and deep fusion of the fusion branch. After multi-level iterations, the fused deep features are added to the residuals of the RGB deep features, and the dehazed image is output after reconstruction mapping.

[0037] This invention provides a spatial collaborative image dehazing method driven by infrared structure priors. The method constructs a three-way collaborative framework consisting of an RGB branch, an infrared branch, and a fusion branch. First, shallow feature extraction is performed on both the hazy visible light image and the infrared image, and initial features for the fusion branch are generated through channel stitching. Second, a spatial frequency domain Mamba block (SFMB) is designed. In the spatial domain, an adaptive multi-scale Mamba block (AMMB) captures local structure and long-range dependencies, while in the frequency domain, a frequency domain amplitude-phase block (FAB) performs collaborative calibration of the amplitude and phase spectra. Then, a cross-guided attention module (CGA) is designed to allow infrared structure information and visible light texture and color information to interact progressively at each encoding stage. A bidirectional cross-modal Mamba fusion module (BICM) is designed to organize RGB, infrared, and fusion features into an interpolated joint sequence, and cross-modal long-range fusion is achieved through forward and backward S6 blocks. Finally, a reconstruction module outputs a clear, dehazed visible light image. This invention utilizes stable structural information provided by infrared images to assist visible light defogging, enhancing the contour recovery capability of heavily foggy areas. It employs dual-path collaborative modeling in both the spatial and frequency domains, simultaneously considering global fog distribution, local edge structures, and high-frequency texture details. Furthermore, it achieves lightweight cross-modal guidance and deep cross-modal fusion through CGA and BICM respectively, avoiding information mixing caused by simple stitching. The S6 / Mamba mechanism is used to model long-range dependencies with low computational complexity, making it suitable for complex outdoor fog and haze scenarios.

[0038] Example 2 Based on the above embodiments, this invention provides a spatial cooperative image dehazing method driven by infrared structure prior, specifically including the following steps: S1. Obtain the registered visible light hazy image and infrared image and perform shallow feature extraction.

[0039] Obtain a foggy RGB image With infrared images ,in, , H and W represent the image height and width, respectively; and Spatially registered images ensure that pixels or features at the same spatial location have a corresponding relationship.

[0040] The foggy RGB image and infrared image are respectively input into the shallow feature extraction module consisting of two convolutional layers to obtain the visible light shallow features. and infrared shallow features :

[0041] In the formula, and These represent the shallow convolution mappings for the RGB branch and the infrared branch, respectively.

[0042] Furthermore, and Splicing is performed according to channel dimensions, and through... Convolution obtains initial features of the fusion branch :

[0043] In the formula, [·,·] represents the channel splicing operation. This represents the channel alignment mapping of the merged branch.

[0044] This step allows color texture information, infrared structural information, and fusion priors to be simultaneously introduced into subsequent multi-level enhancement networks.

[0045] S2. Construct a spatial frequency domain Mamba block (SFMB) to achieve single-modal feature enhancement.

[0046] In the RGB branch, infrared branch, and fusion branch, spatial frequency domain Mamba block (SFMB) is used to perform single-modal enhancement on the input features.

[0047] Let the input features at any level be... First, perform layer normalization, then input adaptive multi-scale Mamba block (AMMB) and frequency domain amplitude-phase block (FAB) respectively:

[0048]

[0049] In the formula, Indicates spatial domain enhancement features, This indicates frequency domain enhancement features.

[0050] Then, the two types of features are fused using a residual method to obtain the SFMB output:

[0051] In the formula, α and β are learnable fusion coefficients used to adaptively adjust the contributions of spatial domain enhancement and frequency domain enhancement to the final features.

[0052] S3. Calibrate the amplitude and phase spectra using the frequency domain amplitude and phase block FAB.

[0053] To recover suppressed high-frequency textures and preserve structural positioning information in fog images, this embodiment designs a frequency domain amplitude-phase block (FAB). A schematic diagram of the frequency domain amplitude-phase block (FAB) structure is shown below. Figure 2 As shown.

[0054] First, the input features Performing a two-dimensional Fourier transform yields the complex spectrum representation:

[0055] In the formula, A = , representing the amplitude spectrum, P = , represents the phase spectrum, and j represents the imaginary unit; the amplitude spectrum mainly reflects the energy intensity of different frequency components and is closely related to texture, contrast and detail recovery; the phase spectrum can preserve spatial structure, edge contour and geometric position information.

[0056] For the amplitude branch, input A with a 1×1 convolution, ReLU activation, and a 3×3 depthwise separable convolution, and generate amplitude modulation weights using a sigmoid function. : ; Then, the calibrated amplitude spectrum A′ is obtained by residual modulation: ; For the phase branch, P is input to a 3×3 convolution, ReLU activation, and a 3×3 depthwise separable convolution, and the phase residual is generated using Tanh. : .

[0057] Then, the calibrated phase spectrum is obtained using residual correction. :

[0058] In the formula, η is the phase perturbation coefficient, which can be set as a learnable parameter.

[0059] Then, the complex spectrum is reconstructed based on A′ and P′, and the spatial domain frequency enhancement features are recovered through inverse Fourier transform: .

[0060] Finally, Residual fusion with input features and layer normalization output: .

[0061] S4. Spatial long-range dependencies are modeled using adaptive multi-scale Mamba blocks (AMMB).

[0062] To simultaneously capture local texture, mesoscale structure, and global fog distribution, this embodiment designs an adaptive multi-scale Mamba block (AMMB). The structure of the AMMB is as follows: Figure 3 As shown.

[0063] Let the input features be AMMB first constructs gated branches to control the throughput of multi-scale spatial information:

[0064] In the formula, σ represents a linear mapping, and σ represents the Sigmoid function.

[0065] Then, three-scale spatial modeling branches are constructed. The first scale maintains the original resolution and uses linear mapping and... Local structure modeling is performed using depthwise convolution; the second and third scales are downsampled by the encoder, then modeled using 3×3 depthwise convolution, 2D-SSM, and layer normalization to model long-range spatial dependencies, and finally restored to the original scale by the decoder; this process can be abstractly represented as:

[0066]

[0067]

[0068] In the formula, and These represent the encoding and decoding mappings at the k-th scale, respectively. This represents a two-dimensional selective state-space modeling layer.

[0069] Furthermore, an adaptive fusion method is used to aggregate multi-scale spatial features:

[0070] In the formula, θ is the learnable scale weight; Finally, the fused multi-scale features and gated features are modulated element-wise, and the AMMB output is obtained through linear mapping and residual connection:

[0071] In the formula, λ is the residual scaling parameter. This is a linear mapping for the output.

[0072] S5. The cross-guided attention module (CGA) enables mutual guidance between RGB and infrared features.

[0073] After each encoding stage, the RGB branch features are... and infrared branching features The input cross-guided attention module (CGA) enables infrared structural information to guide the enhancement of visible light features, and inversely modulates infrared features with visible light texture and color-related information.

[0074] First, global max pooling and global average pooling are performed on any modal feature to obtain cross-modal guidance statistics; the infrared guidance RGB channel-spatial guidance weights can be expressed as:

[0075] In the formula, MaxP and AvgP represent max pooling and average pooling, respectively. This represents a convolutional mapping.

[0076] Then, guided weights are used to modulate the RGB features: ; Similarly, it can be generated from RGB features. And modulate infrared features .

[0077] Finally, the modulated two-mode features are spliced, segmented, and sequentially scanned, and then input into S6Blocks and Merge units for cross-channel information integration to obtain updated RGB and infrared features: .

[0078] CGA enables multi-level lightweight cross-modal guidance before deep fusion, reducing feature mismatch caused by modal differences. The structure diagram of CGA is shown below. Figure 4 As shown.

[0079] S6. Deep joint modeling is performed using the bidirectional cross-modal Mamba fusion module BICM.

[0080] To achieve explicit long-range interaction between RGB, infrared, and fusion branches, this embodiment designs a bidirectional cross-modal Mamba fusion module (BICM). The structure diagram of the BICM is shown below. Figure 5 As shown.

[0081] Let the i-th level input include RGB features Infrared features and fusion branch features .

[0082] First, the three features are organized into an insertion joint sequence according to their spatial location:

[0083] In the formula, Insert means interleaving the RGB token, infrared token, and fused token at the same spatial location to form a joint sequence; for example J i = [ r 1, t 1 m 1, r2, t 2, m 2, ..., r L , t L , m L ], L= H × W .

[0084] Then, average pooling and max pooling are performed on the joint sequence, and cross-modal fusion weights are generated by convolution and Sigmoid; simultaneously, the joint sequence is flattened and input into forward S6Blocks and backward S6Blocks respectively: ; The bidirectional sequence modeling results are merged, and the spatial features corresponding to each modality or fusion branch are restored through a segmentation operation. Then, element-wise weighted fusion and residual fusion are performed with the input features to obtain the current level fusion output: .

[0085] Through BICM, the model can simultaneously capture long-range dependencies between RGB color texture, infrared structural boundaries, and fused features on the joint sequence, thereby improving structural recovery and color consistency in heavily foggy regions.

[0086] S7. Construct a multi-level infrared-assisted defogging network and output a clear image.

[0087] The above SFMB, CGA and BICM are stacked in a multi-level structure.

[0088] For the i-th level network, the RGB branch and the infrared branch are first enhanced by SFMB, and then cross-modal mutual guidance is performed by CGA. The fusion branch updates its own features through SFMB and performs deep fusion with the current-level RGB and infrared features using BICM; the process can be represented as follows: ; ; .

[0089] After N iterations, the final RGB deep features are obtained. Infrared deep features and fusion features .

[0090] The fused features are added to the residuals of the RGB deep features and then input into the reconstruction convolution head to output a clear, dehazed image J.

[0091] In the formula, ψ represents the image reconstruction mapping, and the output J∈ .

[0092] By applying the methods described in the above embodiments to actual dehazed image output, a comparison of the input and output dehazing effects can be found in the following figures. Figures 6-8 .

[0093] This invention provides an infrared structure prior-driven spatial-frequency state-space collaborative image dehazing method that can simultaneously model infrared-visible complementary relationships, spatial domain long-range dependence, and frequency domain amplitude-phase degradation information. This method is applicable to registered foggy visible and infrared images, utilizing spatial-frequency domain modeling, cross-modal attention, and bidirectional selective state-space modeling to achieve image dehazing and restoration. It can effectively improve problems such as reduced contrast, missing texture, structural blurring, and color distortion in foggy visible images, and has advantages such as strong structural restoration capability, good detail preservation, and full utilization of cross-modal information.

[0094] This invention constructs a three-way collaborative framework consisting of an RGB branch, an infrared branch, and a fusion branch. First, shallow feature extraction is performed on both the hazy visible light image and the infrared image, and initial features for the fusion branch are generated through channel stitching. Second, a spatial-frequency domain Mamba block (SFMB) is designed. In the spatial domain, an adaptive multi-scale Mamba block (AMMB) captures local structure and long-range dependencies, while in the frequency domain, a frequency-domain amplitude-phase block (FAB) performs collaborative calibration of the amplitude and phase spectra. Then, a cross-guided attention module (CGA) is designed to allow infrared structural information and visible light texture and color information to interact progressively at each encoding stage. A bidirectional cross-modal Mamba fusion module (BICM) is designed to organize RGB, infrared, and fusion features into an interpolated joint sequence, and cross-modal long-range fusion is achieved through forward and backward S6 blocks. Finally, a clear, dehazed visible light image is output through a reconstruction module. Compared with existing methods, this invention has the following advantages: First, it utilizes stable structural information provided by infrared images to assist visible light dehazing, enhancing the contour recovery capability of heavily foggy areas; second, it uses dual-path collaborative modeling in the spatial and frequency domains to simultaneously consider global fog distribution, local edge structure, and high-frequency texture details; third, it achieves lightweight cross-modal guidance and deep cross-modal fusion through CGA and BICM respectively, avoiding information mixing caused by simple stitching; and fourth, it utilizes the S6 / Mamba mechanism to model long-range dependencies with low computational complexity, making it suitable for complex outdoor fog and haze scenarios.

[0095] In the description of this invention, it should be understood that terms such as "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of this application, "a plurality of" means two or more, unless otherwise explicitly specified.

[0096] The above description is merely a preferred embodiment of this application and is not intended to limit this application. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this application should be included within the protection scope of this application.

Claims

1. A spatial cooperative image dehazing method based on infrared structure prior-driven spatial frequency state, characterized in that, Includes the following steps: S1. Obtain the registered visible light hazy image and infrared image and perform shallow feature extraction. Shallow feature extraction was performed on the visible light hazy image and the infrared image respectively to obtain the corresponding visible light shallow features and infrared shallow features; The visible light shallow features and infrared shallow features are concatenated along the channel dimension and then processed by channel alignment mapping to obtain the initial fusion features for use by subsequent multi-level enhancement networks. The initial fusion features integrate the color texture information of visible light and the structural information of infrared images. S2. Constructing Spatial Frequency Domain Mamba Block SFMB Enhanced Single-Mode Features After normalizing the input features, multi-scale spatial enhancement and frequency domain amplitude and phase enhancement are obtained respectively. Then, learnable coefficients are used to superimpose the two enhanced features back into the original features in the form of residuals. S3. Calibrate the amplitude and phase spectra using the frequency domain amplitude and phase block FAB. The frequency domain amplitude-phase block (FAB) transforms the input features to the frequency domain, separates the amplitude spectrum and the phase spectrum, performs residual modulation on the amplitude spectrum to enhance the texture and residual correction on the phase spectrum to preserve structural information, reconstructs the spectrum and performs inverse Fourier transform to return to the spatial domain, and then performs residual fusion and normalization to output the result, thereby achieving high-frequency texture restoration while maintaining spatial positioning. S4. Modeling long-range spatial dependencies using adaptive multi-scale Mamba blocks (AMMB) Adaptive multi-scale Mamba block AMMB generates gated features to regulate the information flow. Local, mesoscale, and global spatial features are extracted through multiple parallel scale branches. Multi-scale features are adaptively fused using learnable weights, and then output through gated modulation and residual connection to achieve simultaneous capture of local texture, mesoscale structure, and global fog distribution. S5. Achieve mutual guidance between RGB and infrared features through the Cross-Guided Attention Module (CGA). The Cross-Guided Attention (CGA) module is set after each level of encoding to achieve bidirectional cross-modal guidance between RGB and infrared. The guidance weight is generated by the counterpart modality to modulate the local feature, and then the updated dual-modal feature is output after cross-channel information integration. Multi-level guidance reduces feature mismatch caused by modal differences. S6. Deep joint modeling using the bidirectional cross-modal Mamba fusion module BICM BICM interleaves RGB, infrared, and the previous layer fused features into a one-dimensional sequence according to their spatial positions, uses bidirectional Mamba to model long-range dependencies, learns modality fusion weights, processes them, and then switches back to image features, outputting new fused features from the residuals, thus realizing explicit long-range information interaction between multiple modalities. S7. Construct a multi-level infrared-assisted defogging network and output a clear image. The network consists of multi-level stacked modules for single-modal enhancement, cross-modal mutual guidance, and deep fusion. Each level sequentially performs bi-branch single-modal enhancement, cross-modal bi-directional guidance, and deep fusion of the fusion branch. After multi-level iterations, the fused deep features are added to the residuals of the RGB deep features, and the dehazed image is output after reconstruction mapping.

2. The spatial cooperative image dehazing method based on infrared structure prior-driven spatial frequency state according to claim 1, characterized in that, S1 includes: Obtain a foggy RGB image With infrared images ,in, , H and W represent the image height and width, respectively; and Spatially register images so that pixels or features at the same spatial location have a corresponding relationship; The hazy RGB image and infrared image are respectively input into a shallow feature extraction module consisting of two convolutional layers to obtain visible light shallow features. and infrared shallow features : In the formula, and These represent the shallow convolution mappings for the RGB branch and the infrared branch, respectively.

3. The spatial cooperative image dehazing method based on infrared structure prior-driven spatial frequency state according to claim 2, characterized in that, Will and Splicing is performed according to channel dimensions, and through... Convolution obtains initial features of the fusion branch : In the formula, [·,·] represents the channel splicing operation. The channel alignment mapping represents the merge branch; Color texture information, infrared structure information, and fusion priors are simultaneously introduced into subsequent multi-level enhancement networks.

4. The spatial cooperative image dehazing method based on infrared structure prior-driven spatial frequency state according to claim 1, characterized in that, S2 includes: in the RGB branch, infrared branch and fusion branch, spatial frequency domain Mamba block (SFMB) is used to perform single-modal enhancement on the input features; Let the input features at any level be... First, perform layer normalization, then input adaptive multi-scale Mamba block (AMMB) and frequency domain amplitude-phase block (FAB) respectively: In the formula, Indicates spatial domain enhancement features, Indicates frequency domain enhancement features; Then, the two types of features are fused using a residual method to obtain the SFMB output: In the formula, α and β are learnable fusion coefficients used to adaptively adjust the contributions of spatial domain enhancement and frequency domain enhancement to the final features.

5. The spatial cooperative image dehazing method based on infrared structure prior-driven spatial frequency state according to claim 1, characterized in that, S3 includes: input features Performing a two-dimensional Fourier transform yields the complex spectrum representation: In the formula, A = , representing the amplitude spectrum, P = , represents the phase spectrum, and j represents the imaginary unit; the amplitude spectrum mainly reflects the energy intensity of different frequency components and is related to texture, contrast and detail recovery; the phase spectrum can preserve spatial structure, edge contours and geometric position information; For the amplitude branch, input A with a 1×1 convolution, ReLU activation, and a 3×3 depthwise separable convolution, and generate amplitude modulation weights using a sigmoid function. : ； Then, the calibrated amplitude spectrum A′ is obtained by residual modulation: ； For the phase branch, P is input to a 3×3 convolution, ReLU activation, and a 3×3 depthwise separable convolution, and the phase residual is generated using Tanh. : ； Then, the calibrated phase spectrum is obtained using residual correction. : In the formula, η is the phase perturbation coefficient, which can be set as a learnable parameter; Then, the complex spectrum is reconstructed based on A′ and P′, and the spatial domain frequency enhancement features are recovered through inverse Fourier transform: ； Finally, Residual fusion with input features and layer normalization output: 。 6. The infrared structure prior-driven spatial frequency state-based spatial cooperative image dehazing method according to claim 1, characterized in that, S4 Includes: Let the input features be AMMB first constructs gated branches to control the throughput of multi-scale spatial information: In the formula, σ represents a linear mapping, and σ represents the Sigmoid function; Then, three-scale spatial modeling branches are constructed. The first scale maintains the original resolution and uses linear mapping and... Local structure modeling is performed using depthwise convolution; the second and third scales are downsampled by the encoder, then modeled using 3×3 depthwise convolution, 2D-SSM, and layer normalization to model long-range spatial dependencies, and finally restored to the original scale by the decoder; this process can be abstractly represented as: In the formula, and These represent the encoding and decoding mappings at the k-th scale, respectively. This represents a two-dimensional selective state-space modeling layer.

7. The spatial cooperative image dehazing method based on infrared structure prior-driven spatial frequency state according to claim 6, characterized in that, Adaptive fusion method is used to aggregate multi-scale spatial features: In the formula, θ is the learnable scale weight; Finally, the fused multi-scale features and gated features are modulated element-wise, and the AMMB output is obtained through linear mapping and residual connection: In the formula, λ is the residual scaling parameter. This is a linear mapping for the output.

8. The spatial cooperative image dehazing method based on infrared structure prior-driven spatial frequency state according to claim 1, characterized in that, S5 includes: after each encoding stage, the RGB branch features are... and infrared branching features The input cross-guided attention module (CGA) enables infrared structural information to guide the enhancement of visible light features, and inversely modulates infrared features with visible light texture and color-related information. First, global max pooling and global average pooling are performed on any modal feature to obtain cross-modal guidance statistics; the infrared guidance RGB channel-spatial guidance weights can be expressed as: In the formula, MaxP and AvgP represent max pooling and average pooling, respectively. Represents a convolution mapping; Then, the guided weights are used to modulate the RGB features: ； Similarly, it can be generated from RGB features. And modulate infrared features ; Finally, the modulated two-mode features are spliced, segmented, and sequentially scanned, and then input into S6Blocks and Merge units for cross-channel information integration to obtain updated RGB and infrared features: 。 9. The infrared structure prior-driven spatial frequency state-based spatial cooperative image dehazing method according to claim 1, characterized in that, S6 include: Let the i-th level input include RGB features Infrared features and fusion branch features ; First, the three features are organized into an insertion joint sequence according to their spatial location: In the formula, Insert means interleaving the RGB token, infrared token, and fused token at the same spatial location to form a joint sequence; Then, average pooling and max pooling are performed on the joint sequence, and cross-modal fusion weights are generated by convolution and Sigmoid; simultaneously, the joint sequence is flattened and input into forward S6Blocks and backward S6Blocks respectively: ； The bidirectional sequence modeling results are merged, and the spatial features corresponding to each modality or fusion branch are restored through a segmentation operation. Then, element-wise weighted fusion and residual fusion are performed with the input features to obtain the current level fusion output: 。 10. The infrared structure prior-driven spatial frequency state-based spatial cooperative image dehazing method according to claim 1, characterized in that, S7 includes: For the i-th level network, the RGB branch and the infrared branch are first enhanced by SFMB, and then cross-modal mutual guidance is performed by CGA; The fusion branch updates its own features through SFMB and performs deep fusion with the current-level RGB and infrared features using BICM; the process can be represented as follows: ； ； ； After N iterations, the final RGB deep features are obtained. Infrared deep features and fusion features ; The fused features are added to the residuals of the RGB deep features and then input into the reconstruction convolution head to output a clear, dehazed image J. In the formula, ψ represents the image reconstruction mapping, and the output J∈ .

Images