Physical prior guided frequency-spatial mamba fusion network and image fusion method thereof

Abstract

This invention discloses a physical prior-guided frequency-space Mamba fusion network and its image fusion method. The physical prior-guided frequency-space Mamba fusion network constructs a global-local multi-scale encoder and embeds a frequency-domain dual-modal cross-guidance module, effectively suppressing noise artifacts in the polarization channel and significantly enhancing texture details in the infrared channel. An offset modulation fusion module based on state-space low-level interaction is designed, addressing the lack of deep interaction in rigid fusion mechanisms by deepening cross-modal interaction fusion. A physical prior loss function based on joint spatial mask is proposed to guide network training and optimization, effectively bridging the modal differences between infrared and polarization. This invention solves the problem of insufficient detail in thermal radiation targets in complex scenes, while also addressing the susceptibility of polarization images to high-frequency noise interference.

Description

Technical Field

[0001] This invention relates to the field of infrared polarization image fusion technology, and in particular to a physical prior-guided frequency-space Mamba fusion network (PDSFMamba) and its image fusion method. Background Technology

[0002] Multimodal image fusion aims to integrate complementary information from different sensors to generate high-quality images with richer scene representations, which is of great significance for improving the robustness of advanced vision tasks such as object detection. Infrared thermal imaging captures the thermal radiation of objects for imaging, has strong anti-interference capabilities, and tends to highlight the thermal distribution characteristics of the target outline and surrounding background. However, it is difficult to distinguish objects with similar temperatures but different materials and lacks surface texture details. In contrast, polarization imaging reveals the inherent polarization properties of light waves. It contains multi-dimensional physical information such as degree of polarization and polarization angle, and is highly sensitive to the surface roughness, material, and geometric edges of objects. However, due to the inherent characteristics of sensors and computational methods, polarization images are prone to amplifying background clutter and have the inherent defect of severe high-frequency noise.

[0003] By fusing infrared intensity and polarization images, the high-contrast target localization contours of infrared thermal radiation can be combined with detailed descriptions of surface physical textures and geometric edges from a stable background and polarization, effectively mitigating the inherent limitations of single-mode imaging. The raw signals acquired by an infrared focal plane camera can directly calculate infrared and polarization information, achieving natural alignment in physical space without complex image pre-registration. In scenarios where infrared imaging quality is limited by thermal crosstalk or severe background clutter, polarization features can significantly supplement the missing high-frequency structures in infrared imaging, generating visually significant and detailed high-quality scene representations. This complementary advantage makes infrared and polarization image fusion a promising field for autonomous driving, intelligent monitoring, and disaster relief.

[0004] Early image fusion research primarily relied on traditional methods such as multi-scale transformation, sparse representation, subspaces, and saliency. While these methods offer strong mathematical interpretability, they heavily depend on manually designed fusion rules, making it difficult to adaptively handle complex and dynamic nonlinear fusion scenarios. With the widespread development and application of deep learning, image fusion based on deep learning methods has become the mainstream research direction in this field. Existing deep learning architectures encompass numerous methods based on CNNs, transformers, generative adversarial networks (GANs), diffusion models, and Mamba, significantly improving fusion performance in complex scenarios. Despite the significant progress made by current deep learning fusion methods, three key technical challenges remain in the field of infrared polarization image fusion. Existing solutions exhibit the following problems:

[0005] (1) High-frequency noise and artifact interference from polarization. In complex lighting and harsh environments, sensors are susceptible to thermal drift and inherent noise interference. Furthermore, the analysis and calculation of polarization images can easily amplify background clutter and sensor noise pollution, resulting in severe high-frequency noise and artifacts.

[0006] (2) Insufficient deep interaction caused by rigid fusion mechanism. The salient regions of infrared and polarization information have significant differences in their respective physical semantic feature distributions. The simple fusion paradigm lacks the ability to explicitly model cross-modal nonlinear mapping relationships and feature space offsets, resulting in insufficient interaction of multimodal deep features with different distributions in the latent space, which limits the network's ability to generalize to complex scenes.

[0007] (3) Differences in physical modal representations and lack of reliable truth values. Infrared intensity and polarization are fundamentally different in physical imaging mechanisms, and polarization features exhibit high spatial sparsity compared to infrared intensity features. Furthermore, image fusion is essentially an unsupervised task lacking a unique optimal solution. This weakly supervised approach, lacking physical priors, struggles to cope with significant inconsistencies across modalities and fails to accurately guide the network to focus on low-noise, high-value target regions.

[0008] In view of this, the present invention is hereby proposed. Summary of the Invention

[0009] The purpose of this invention is to provide a Physically Prior Guided Frequency-Space Mamba Fusion Network (PDSFMamba) and its image fusion method, which solves the problem of lack of detail in thermal radiation targets in complex scenes, while also addressing the issue of polarization images being susceptible to high-frequency noise interference.

[0010] To achieve the above objectives, the technical solution adopted by this invention is: a physical prior-guided frequency-space Mamba fusion network, characterized in that the physical prior-guided frequency-space Mamba fusion network is constructed entirely based on the efficient visual state space module EVSS, including: a dual-modal data acquisition and preprocessing module, used to acquire infrared intensity images collected by a focal plane detector. With polarization image , serving as the dual-modal input of the frequency-space Mamba fusion network guided by physical priors;

[0011] The frequency-space collaborative feature extraction module is used to extract the infrared intensity image. With the polarization image The inputs are fed into the CNN-Mamba-based frequency-space co-operational multi-scale feature extraction encoder (SFCE) to perform multi-scale feature extraction and output deep infrared features. and deep polarization characteristics ;

[0012] Selective offset modulation fusion module, used to integrate the deep infrared features and the deep polarization characteristics The input is fed into the Selective Offset Modulation (SOMF) fusion module based on the state space layer for feature fusion to obtain fused features;

[0013] The feature reconstruction image reconstruction module is used to input the fused features into the cascaded decoder and reconstruct the output along with the infrared intensity image. Infrared polarization fusion images with consistent resolution.

[0014] An image fusion method based on a physically-prior-guided frequency-space Mamba fusion network is characterized by utilizing the aforementioned physically-prior-guided frequency-space Mamba fusion network, with the following specific steps.

[0015] S1. Using the dual-modal data acquisition and preprocessing module, acquire the infrared intensity image obtained by solving the raw signal collected by the focal plane detector. With polarization image , serving as the bimodal input of the network;

[0016] S2. Using the frequency-space coordinated feature extraction module, the infrared intensity image is... With the polarization image The inputs are fed into the CNN-Mamba-based frequency-space co-operational multi-scale feature extraction encoder SFCE. Non-weight-sharing twin branches are used for independent forward propagation of the infrared intensity and polarization degree branches. Within each branch, the inputs sequentially pass through a preliminary shallow feature extraction encoder, a wavelet Mamba cross-guided module (WMCG), and a two-stage iterative deep feature extraction encoder DS-IDFE to perform multi-scale feature extraction, outputting deep infrared features. and deep polarization characteristics ;

[0017] S3. Using a selective offset modulation fusion module, the deep infrared features are... and the deep polarization characteristics The input is fed into the Selective Offset Modulation Fusion (SOMF) module, which is based on the state space layer, for feature fusion. SOMF consists of two stages connected in series: a Feature Guided Modulation (FGM) module and a Bidirectional Deep Coupling (BDC) module, to obtain the fused features. ;

[0018] S4. Using the feature reconstruction image reconstruction module, the fused features are input into the cascaded decoder. Spatial upsampling is performed progressively through multiple cascaded convolutional layers to reconstruct the output, which is then compared to the infrared intensity image. Infrared polarization fusion images with consistent resolution.

[0019] Furthermore, the efficient visual state space module EVSS process includes: given the input two-dimensional feature tensor After layer normalization and linear projection, the data is divided into main feature branches along the channel dimension. With gated branches The input is fed into the efficient state-space operator ESSM; in the main characteristic branch of the efficient state-space operator ESSM, The sequence is sequentially processed through 3×3 convolution and SiLU activation function before entering the efficient two-dimensional selective scanning module ES2D. ES2D significantly reduces the computational complexity of global modeling by employing a skip sampling strategy, while effectively preserving long-range dependencies. The compressed sequence then enters the core S6 operator to dynamically generate discretized state parameters. , , The hidden state is updated to capture the global context; finally, the structure is restored to a two-dimensional space through efficient fusion and recombination, yielding the main branch output. In the gated branch, Spatial gating graphs are generated only after SiLU activation. Ultimately, the main branch features With gated features Element-wise multiplication yields ; The final feature is output after linear projection and residual connection. The complete formula is shown below:

[0020]

[0021]

[0022]

[0023]

[0024] in This is for element-wise multiplication.

[0025] Furthermore, in step S2, the preliminary shallow feature extraction encoder process includes: processing the infrared intensity image... With the polarization image The infrared intensity image is processed by performing two 3×3 convolutional layers with the LeakyReLU modified linear unit activation function. With the polarization image The shallow infrared intensity features are obtained by mapping them to a high-dimensional feature space with 16 channels. and shallow polarization characteristics .

[0026] Furthermore, in step S2, the wavelet Mamba cross-guided module (WMCG) process includes: processing the shallow infrared intensity features... and the shallow polarization characteristics Discrete Wavelet Transform (DWT) decomposition is performed on each band, resulting in four frequency sub-bands. The HH and LH bands are infrared-guided polarization, the HL band is polarization-guided infrared, and the LL band does not undergo cross-guidance. The guiding and guided branches corresponding to different frequency bands are input into the structurally identical Mamba cross-guidance module (MCG) for cross-guidance. The Mamba cross-guidance module (MCG) is designed based on the efficient visual state space module (EVSS) and target features. As input to the main state branch, guiding features The two branches are used as inputs to the gated explicit guidance branches; explicit physical perception guidance is achieved through element-wise multiplication, and finally mapped back to the original dimension and a residual connection is added to output the guided target features. The specific formula for the Mamba cross-boot module MCG is as follows:

[0027]

[0028]

[0029]

[0030] in, For target features, The target features are those that have passed through the main state branch. To guide features, This refers to the guiding characteristics of gated branches. The target characteristics after booting, as output by the Mamba cross-booting module;

[0031] After completing guidance in different frequency bands and achieving denoising and enhancement of each sub-band feature, the four frequency bands of infrared and polarization are reconstructed into spatial domain infrared guidance features through discrete wavelet inverse transform (IDWT). With polarization guidance characteristics .

[0032] Furthermore, in step S2, the two-stage iterative deep feature extraction encoder (DS-IDFE) process includes: firstly, guiding features from the spatial domain... That is, the spatial domain infrared guidance feature With the polarization guiding feature The inputs are fed into a single-stage iterative deep feature extraction encoder (IDFE), which consists of a high-efficiency visual state space module (EVSS) for global feature extraction and a spatially adaptive gated intensifier (SAGE) for local detail enhancement, connected in parallel. The outputs of the two parallel branches, local and global, are then added element-wise to obtain the single-stage IDFE infrared output. and single-stage IDFE polarization output Then, both branches are convolved and mapped to 32 channels, which are then input into the second-stage iterative deep feature extraction encoder (IDFE) with the same structure. The spatial domain-guided features are explicitly externalized to the entire two-stage iterative deep feature extraction encoder (DS-IDFE). All features are constructed by convolutional upsizing and aligning to 32 channels to build residual alignment paths, ultimately outputting deep features. That is, deep infrared features With deep polarization characteristics The two-stage iterative deep feature extraction encoder DS-IDFE is shown in the following formula:

[0033]

[0034]

[0035]

[0036] in, To guide features, For the output features of the global branch in a single-level IDFE, The output characteristics of local branches in a single-level IDFE For single-level IDFE output features, This represents the deep features of the final output.

[0037] Furthermore, in step S2, the spatial adaptive gating enhancer SAGE includes: guiding spatial domain features That is, the spatial domain infrared guidance feature With the polarization guiding feature First, dimensionality is increased through group normalization and 1×1 convolution, followed by a 7×7 large receptive field convolutional layer. Then, the two modality branch features are evenly divided into content branch features along the channel dimension. and gated branch features ,right Generate a mask using the Gaussian error linear unit activation function GELU. and Element-wise multiplication; after dimensionality reduction, a learnable channel scaling factor is applied. Output local branch features The specific formula is shown below:

[0038]

[0039]

[0040]

[0041] in, To guide features, For content branching features, This is a gated branch feature. Output features for content branches. For gated branch output characteristics, This is a learnable channel scaling factor. This refers to the output characteristics of local branches in a single-level IDFE.

[0042] Furthermore, in step S3, the Feature Guided Modulation (FGM) process includes: transferring the deep infrared features... and the aforementioned deep polarization characteristics Perform channel splicing to form basic joint features The deep infrared features mentioned and the aforementioned deep polarization characteristics The main state branches are processed separately by the High-Efficiency Vision State Space Module (EVSS) and before the High-Efficiency Two-Dimensional Selective Scan Module (ES2D). and Element-wise multiplication is performed to construct a joint salient representation of the multimodalities. ; in basic joint features In the main state branch, the flattened one-dimensional basic joint feature sequence and joint significant characterization sequences respectively through linear projection matrix and Projecting onto the same parameter dimension space, the mapped joint salient representation is then used as a dynamic offset and superimposed onto the mapped basic joint feature sequence to form the modulated parameter generation tensor. Then, the time-varying parameters of the core state-space model SSM are derived. , , Output stage one infrared feature and Phase 1 polarization characteristics Each channel has 32 channels, and the first stage is a significant modulation feature. Channel 64, the specific formula is as follows:

[0043]

[0044]

[0045]

[0046]

[0047] The bidirectional deep coupling module BDC process includes: converting the aforementioned stage one infrared features... and the aforementioned stage one polarization and Channel splicing yields specific complementary features The aforementioned stage-1 saliency modulation features Complementary features to the aforementioned specificity Each branch modulates the other's features as offsets in the underlying parameter space, outputting saliency-coupled features. and specific coupling characteristics Both outputs have 64 channels. The two outputs are concatenated to obtain the final output fusion feature of the Selective Offset Modulation (SOMF) fusion module. The specific formula is shown below:

[0048]

[0049]

[0050]

[0051] Furthermore, during the network training phase, the image fusion method based on the physical prior-guided frequency-space Mamba fusion network also includes:

[0052] S5. Utilizing a self-supervised loss function module guided by physical priors, a polarization-purified polarization representation is obtained by constructing a polarization mask based on physical priors. Infrared intensity information is then injected to obtain a cross-modal cooperative polarization representation, ultimately forming a nonlinear enhanced polarization representation. The cross-modal cooperative polarization representation and the nonlinear enhanced polarization representation will replace the aforementioned polarization degree image. Different loss functions are fed into the infrared intensity image. Jointly construct target fusion image Guiding the network training phase; total loss function Due to strength loss gradient loss and multi-scale weighted structural similarity loss Together they form the following formula:

[0053]

[0054]

[0055]

[0056] ;

[0057] The weights of the three sub-loss functions are set based on experience. , , .

[0058] Furthermore, in step S5, the nonlinear enhanced polarization characterization process includes: inputting an infrared intensity image. Polarization image Polarization angle image Based on the physical characteristics of the aforementioned images, such as infrared intensity, degree of polarization, and polarization angle, a joint spatial mask is first constructed to match the degree of polarization. Purification is carried out using polarization degree With polarization angle Accurate solution of normalized horizontal-vertical linear polarization components Characterizing horizontal polarization properties using a soft-truncation gated activation function with a scaling factor. Extract the positive value of horizontal polarization and characterize the smoothness using the inverse ratio of infrared local gradient. Construct target surface mask And supplemented by polarization gradient to construct a line mask. Then, the maximum value of the two is taken to obtain the polarization mask based on physical priors. The specific formula is shown below:

[0059]

[0060]

[0061]

[0062] in, For the Sobel gradient operator, It is a pixel-level maximum value selection function;

[0063] Using physical prior polarization masks For the original polarization degree image Feature purification is performed to form a mask-purified polarization characterization. ,

[0064]

[0065] Introducing infrared intensity as background spatial compensation to form a cross-modal cooperative polarization characterization :

[0066]

[0067] Based on experience, and Set them to 1.0 and 0.5 respectively;

[0068] Characterization of cross-modal cooperative polarization in intensity loss and gradient loss Gamma nonlinear correction was performed to obtain a nonlinear enhanced polarization characterization. :

[0069]

[0070] Based on experience, The value is 0.5. Minimum value to prevent gradient explosion.

[0071] In view of the above technical features, the present invention has the following beneficial effects:

[0072] 1. This invention presents a frequency-space Mamba fusion network and its image fusion method based on physical prior guidance. It constructs a global-local multi-scale encoder and embeds a frequency-domain dual-modal cross-guidance module, effectively suppressing noise artifacts in the polarization channel and significantly enhancing texture details in the infrared channel. An offset modulation fusion module based on state-space low-level interaction is designed, addressing the lack of deep interaction in rigid fusion mechanisms by deepening cross-modal interaction fusion. A physical prior loss function based on joint spatial mask is proposed to guide network training optimization, effectively bridging the modal differences between infrared and polarization. This solves the problem of insufficient detail in thermal radiation targets in complex scenes, while also addressing the issue of polarization images being susceptible to high-frequency noise interference.

[0073] 2. The present invention provides a frequency-space Mamba fusion network and its image fusion method based on physical prior guidance, which can suppress polarization high-frequency noise and artifact interference. Specifically, a frequency-space coordinated multi-scale feature extraction encoder is designed, and a wavelet Mamba cross-guided module (WMCG) is embedded between shallow and deep features. In addition, a two-level iterative deep feature extraction encoder (DS-IDFE) is constructed to block the evolution of polarization noise into the deep layer and achieve high-fidelity extraction of cross-modal features.

[0074] 2. The present invention provides a frequency-space Mamba fusion network and its image fusion method based on physical prior guidance, which can fully interact and fuse deep bimodal features. Specifically, it constructs a selective offset modulation fusion module (SOMF), a feature-guided modulation module (FGM), and a bidirectional deep coupling module (BDC). By injecting cross-modal offsets before the selective scanning operator, it achieves dynamic alignment and deep coupling of cross-modal features in deep space.

[0075] 3. The present invention provides a frequency-space Mamba fusion network and its image fusion method based on physical prior guidance, which can supplement reliable fusion ground truth and make up for the difference in physical modality representation. Specifically, it proposes a self-supervised loss function based on physical prior guidance, explicitly constructs a physical mask that takes into account both noise suppression and salient feature preservation, reconstructs the loss function, and effectively solves the target-targeting optimization problem in scenarios without fusion ground truth.

[0076] 4. The present invention provides a frequency-space Mamba fusion network and its image fusion method based on physical prior guidance, which can guarantee the quality of infrared polarization fusion. That is, the qualitative and quantitative results prove the effectiveness of the present invention. Compared with existing advanced fusion methods, the present invention performs better in infrared polarization image fusion tasks. Attached Figure Description

[0077] Figure 1 This is a flowchart of a frequency-space Mamba fusion network based on physical prior guidance provided in Embodiment 1 of the present invention;

[0078] Figure 2 This is a specific framework diagram of the frequency-space Mamba fusion network based on physical prior guidance provided in Embodiment 1 of the present invention, wherein, Figure 2 (a) is a frequency-space Mamba fusion network framework based on physical prior guidance. Figure 2 (b) is the high-efficiency visual state space module EVSS. Figure 2 (c) is the efficient state-space operator ESSM;

[0079] Figure 3 This is a detailed framework diagram of the wavelet Mamba cross-booting module provided in Embodiment 1 of the present invention, wherein, Figure 3 (a) is the wavelet Mamba cross-booting module WMCG. Figure 3 (b) is the Mamba cross-boot module MCG;

[0080] Figure 4 This is a specific framework diagram of a single-stage iterative deep feature extraction encoder provided in Embodiment 1 of the present invention, wherein, Figure 4 (a) is a single-stage iterative deep feature extraction encoder (IDFE). Figure 4 (b) is the spatially adaptive gated enhancer SAGE;

[0081] Figure 5 This is a detailed framework diagram of the selective offset modulation fusion module provided in Embodiment 1 of the present invention, wherein, Figure 5 (a) is the Feature-Guided Modulation Module (FGM). Figure 5 (b) is the bidirectional deeply coupled module BDC;

[0082] Figure 6The qualitative comparison results between the method of the present invention and the comparative method provided in Embodiment 1 of the present invention are shown. Detailed Implementation

[0083] The present invention will be further described below with reference to specific embodiments. It should be understood that these embodiments are for illustrative purposes only and are not intended to limit the scope of the invention. Furthermore, it should be understood that after reading the teachings of this invention, those skilled in the art can make various alterations or modifications to the invention, and these equivalent forms also fall within the scope defined by the appended claims.

[0084] See Figures 1 to 6 Specific embodiment 1 provides a physical prior-guided frequency-space Mamba fusion network. The network is characterized by being constructed entirely based on the efficient visual state space module EVSS, including: a dual-modal data acquisition and preprocessing module for acquiring infrared intensity images collected by a focal plane detector. With polarization image , serving as the dual-modal input of the frequency-space Mamba fusion network guided by physical priors;

[0085] The frequency-space collaborative feature extraction module is used to extract the infrared intensity image. With the polarization image The inputs are fed into the CNN-Mamba-based frequency-space co-operational multi-scale feature extraction encoder (SFCE) to perform multi-scale feature extraction and output deep infrared features. and deep polarization characteristics ;

[0086] Selective offset modulation fusion module, used to integrate the deep infrared features and the deep polarization characteristics The input is fed into the state-space based selective offset modulation fusion module (SOMF) for feature fusion to obtain the fused features. ;

[0087] The feature reconstruction image reconstruction module is used to input the fused features into the cascaded decoder and reconstruct the output along with the infrared intensity image. Infrared polarization fusion images with consistent resolution;

[0088] A self-supervised loss function module guided by physical priors is used to purify polarization representations by constructing a polarization mask based on physical priors, injecting infrared intensity information to obtain cross-modal cooperative polarization representations, and finally forming a nonlinear enhanced polarization representation. The cross-modal cooperative polarization representation and the nonlinear enhanced polarization representation replace the polarization degree image. Input into the loss function and compare with the infrared intensity image Jointly construct target fusion image Provide guidance during the network training phase.

[0089] In this embodiment 1, Figure 1 This is a flowchart of a frequency-space Mamba fusion network guided by physical priors. Figure 2 In order to be in Figure 1 The following is a detailed framework diagram of a space-frequency fusion network based on physical prior guidance, which is developed from the above. Figure 2 (a) is a frequency-space Mamba fusion network framework based on physical prior guidance. Figure 2 (b) is the high-efficiency visual state space module EVSS. Figure 2 (c) is the efficient state-space operator ESSM, which obtains the infrared polarization fused image after fusing the infrared intensity image and the polarization degree image through a fusion network.

[0090] This embodiment 1 uses the High-Efficiency Visual State Space Module (EVSS) (see...). Figure 2 (b) The basic Mamba modules of the network, namely the modules in the physically prior-guided frequency-space Mamba fusion network of Embodiment 1 (such as the Mamba cross-guided module MCG, the single-level iterative deep feature extraction encoder IDFE, and the state-space-based selective offset modulation fusion module SOMF, etc.), are all innovative variants of the efficient visual state space module EVSS. The EVSS module (i.e., the efficient visual state space module) is mainly based on the given two-dimensional feature tensor of the input. After layer normalization and linear projection, the data is divided into main feature branches along the channel dimension. With gated branches Feed it into the efficient state-space operator ESSM (see Figure 2 (c)). In the principal characteristic branch of the efficient state-space operator ESSM, The sequence is sequentially processed through 3×3 convolution and SiLU activation function before entering the efficient 2D selective scanning module ES2D. ES2D significantly reduces the computational complexity of global modeling by employing a skip sampling strategy, while effectively preserving long-range dependencies. The compressed sequence then enters the core S6 operator to dynamically generate discretized state parameters. , , It also updates the hidden state to capture the global context. Finally, it restores the structure to a two-dimensional space through efficient fusion and recombination, yielding the main branch output. In the gated branch, Spatial gating graphs are generated only after SiLU activation. Ultimately, the main branch features With gated features Element-wise multiplication is performed to suppress background noise and amplify the response in significant regions. . The final feature is output after linear projection and residual connection. The complete formula is shown below.

[0091]

[0092]

[0093]

[0094]

[0095] in This is for element-wise multiplication.

[0096] An image fusion method based on a physically-prior-guided frequency-space Mamba fusion network is presented. The specific steps are as follows:

[0097] S1. Using the dual-modal data acquisition and preprocessing module, the infrared intensity obtained by solving the raw signal collected by the DoFP detector is acquired. )image With polarization degree ( )image , serving as the bimodal input of the network;

[0098] S2. Using the frequency-space coordinated feature extraction module, the infrared intensity image is... With the polarization image The data are input into the CNN-Mamba-based frequency-space co-processing multi-scale feature extraction encoder (SFCE), and non-weight-sharing twin branches are used to process the infrared intensity images. and polarization image Each branch performs independent forward propagation to independently extract unique multi-scale features of infrared and polarization. During feature extraction, within each branch, multi-scale feature extraction is carried out sequentially through a preliminary shallow feature extraction encoder, a wavelet Mamba cross-guided module (WMCG), and a two-stage iterative deep feature extraction encoder (DS-IDFE), outputting deep infrared features. and deep polarization characteristics ;

[0099] Specifically, the process of the S21 preliminary shallow feature extraction encoder includes: First, the input infrared intensity image is processed by a frequency-space Mamba fusion network guided by physical priors. With polarization image Shallow feature extraction is performed, which involves two 3×3 convolutional layers combined with a LeakyReLU modified linear unit activation function to process the single-channel source image (i.e., the infrared intensity image). With the polarization image The shallow infrared intensity features are mapped to a high-dimensional feature space with 16 channels respectively. and shallow polarization characteristics The sample is then fed into the subsequent wavelet Mamba cross-guided module (i.e., WMCG module) for feature guidance and purification.

[0100] S22, Wavelet Mamba Cross-Booting Module WMCG Procedure (see...) Figure 3 Shallow infrared intensity characteristics and shallow polarization characteristics The data is fed into a WMCG. High-frequency speckle denoising guided by the infrared high signal-to-noise ratio structure is used, and infrared target enhancement is guided by high-frequency polarization edge details. This not only ensures the dual-mode physical structure but also blocks the evolution of polarization physical noise into deep pseudo-semantics, enhancing the saliency expression of infrared targets and performing frequency domain polarization noise suppression and infrared texture enhancement. Specifically, this includes: processing the shallow infrared intensity features... and the shallow polarization characteristics Discrete Wavelet Transform (DWT) decomposition is performed on each frequency band, resulting in four frequency sub-bands. The HH and LH bands are infrared-guided polarization, the HL band is polarization-guided infrared, and the LL band does not undergo cross-guidance. The guiding and guided branches corresponding to different frequency bands are input into the structurally identical Mamba cross-guidance module (MCG) for cross-guidance. The Mamba cross-guidance module (MCG) is designed based on the aforementioned efficient visual state space module (EVSS), and its core mechanism lies in target features. As input to the main state branch, guiding features The two branches are used as inputs to the gated explicit guidance branches; explicit physical perception guidance is achieved through element-wise multiplication, and finally mapped back to the original dimension and a residual connection is added to output the guided target features. The specific formula for the Mamba cross-boot module MCG is as follows:

[0101]

[0102]

[0103]

[0104] in, For target features, The target features are those that have passed through the main state branch. To guide features, This refers to the guiding characteristics of gated branches. The target characteristics after booting, as output by the Mamba cross-booting module;

[0105] After completing guidance in different frequency bands and achieving denoising and enhancement of each sub-band feature, the four frequency bands of infrared and polarization are reconstructed into spatial domain infrared guidance features through discrete wavelet inverse transform (IDWT). With polarization guidance characteristics .

[0106] Compared to existing fusion networks that mostly provide indiscriminate guidance across the entire frequency band or simply distinguish between high and low frequencies, ignoring the physical characteristics and noise distribution differences of infrared and polarization imaging at different frequency bands, this step S22 utilizes the rigorous spatial frequency local resolution capability of Discrete Wavelet Transform (DWT) during the WMCG process of the wavelet Mamba cross-guidance module to decompose shallow features into four frequency band sub-bands.

[0107] In step S22, the guiding and guided branches for different frequency bands are determined: Based on the physical properties of each frequency band, the guiding and guided branches for infrared and polarization features are determined: The HH band utilizes the high-frequency structure of the infrared as a gate to accurately filter out high-frequency speckle noise in polarization. The LH band explicitly uses infrared to suppress incoherent fringe noise in polarization features, preserving the true vertical structure. The HL band explicitly injects polarization-specific texture information into the infrared branch. The LL band is an approximate representation of the spatial domain and has a relatively high signal-to-noise ratio; therefore, no cross-guiding operation is performed to maintain the integrity of its respective spatial structure.

[0108] In step S22, the Mamba cross-booting module MCG: the boot and booted branches corresponding to different frequency bands are input into the Mamba cross-booting module MCG with the same structure for cross-booting.

[0109] S23, Two-stage iterative deep feature extraction encoder DS-IDFE (see...) Figure 4 Spatial domain guidance characteristics after frequency domain cross-guidance (Space domain infrared guidance features) With polarization guidance characteristics The input is then fed into a two-stage iterative deep feature extraction encoder (DS-IDFE) to further mine multi-scale semantic information in the high-dimensional feature space. The core challenge of deep feature extraction lies in the trade-off between global context and local detail preservation, while avoiding the erroneous amplification of residual polarization noise during deep forward propagation. The two-stage iterative deep feature extraction encoder (DS-IDFE) consists of two single-stage iterative deep feature extraction encoders (IDFE), convolutional dimensionality increase, and residual connections.

[0110] The specific process includes: a single-level iterative deep feature extraction encoder (IDFE) process, which first guides the spatial domain features. That is, the spatial domain infrared guidance feature With the polarization guiding feature (16 channels) are input to a single-stage iterative deep feature extraction encoder (IDFE), which consists of a high-efficiency visual state space module (EVSS) for global feature extraction and a spatially adaptive gated intensifier (SAGE) for local detail enhancement, resulting in a single-stage IDFE infrared output. and single-stage IDFE polarization output The Spatial Adaptive Gated Augmenter (SAGE) borrows from large-kernel convolution and nonlinear feature filtering mechanisms. SAGE can accurately distinguish between stripe noise and true edges. The SAGE module process includes: guiding spatial domain features... That is, the spatial domain infrared guidance feature With the polarization guiding feature First, dimensionality is increased through group normalization and 1×1 convolution. Then, a 7×7 large receptive field convolutional layer is used to extract local spatial context. Finally, the two modal branch features are evenly divided into content branch features in the channel dimension. and gated branch features ,right Generate a mask using the Gaussian error linear unit activation function GELU. and Element-wise multiplication; after dimensionality reduction, a learnable channel scaling factor is applied. This improves the stability of the module and outputs local branch features. The specific formula is shown below:

[0111]

[0112]

[0113]

[0114]

[0115] The second-level iterative deep feature extraction encoder (IDFE) is connected to the residual: the EVSS global feature extraction branch is then connected. SAGE Local Detail Enhancement Branch Output Element-wise addition is performed (with residual connections integrated internally by EVSS) to achieve deep feature extraction and enhancement, resulting in a single-level IDFE output. (Single-stage IDFE infrared output) and single-stage IDFE polarization output Then, both branches are convolved and mapped to 32 channels, which are then input into the second-stage iterative deep feature extraction encoder (IDFE) with the same structure. The spatial domain-guided features are explicitly externalized to the entire two-stage iterative deep feature extraction encoder (DS-IDFE). All features are constructed by convolutional upsizing and aligning to 32 channels to build residual alignment paths, ultimately outputting deep features. That is, deep infrared features With deep polarization characteristics The specific formula for the two-stage iterative deep feature extraction encoder DS-IDFE is shown below:

[0116]

[0117]

[0118]

[0119] S3. Utilize the Selective Offset Modulation Fusion (SOMF) module based on the state-space layer, see [link / reference]. Figure 5 The deep infrared features and the deep polarization characteristics The input is fed into the state-space based selective offset modulation fusion module (SOMF) for feature fusion. SOMF is guided by the feature-guided modulation module (FGM) (see [link]). Figure 5 (a) and the bidirectional deep coupling module BDC (see Figure 5 (b) The two stages are connected in series to obtain the fused features. Traditional rigid fusion paradigms such as channel stitching often ignore the semantic distribution differences between infrared and polarization features in the deep latent space, which can easily lead to artifacts or loss of key information in the fused image. In the Selective Offset Modulation Fusion Module (SOMF), simple explicit feature addition and subtraction are abandoned. Instead, a modulation mechanism is introduced into the underlying mathematical mechanism of the State Space Model (SSM) to achieve deep coupling of cross-modal features. It consists of two stages connected in series: the Feature Guided Modulation Module (FGM) and the Bidirectional Deep Coupling Module (BDC). The target region is focused on the basic joint features of the two modes. After obtaining the salient modulation features, they are bidirectionally cross-offset input with the specific complementary features. The series connection of the two stages constructs a progressive non-rigid fusion paradigm from joint salient guidance to parameter-level interactive fusion, which greatly improves the alignment accuracy and information richness of the network in complex scenes.

[0120] Specifically, S31, the Feature Guided Modulation (FGM) process includes, as described in the S31 step, applying the deep infrared features... and the aforementioned deep polarization characteristics Perform channel splicing to form basic joint features The deep infrared features mentioned and the aforementioned deep polarization characteristics Each modality is processed separately by the efficient Visual State Space (EVSS) module to preserve its unique semantics. The main state branches are then processed before the efficient 2D Selective Scanning (ES2D) module. and Element-wise multiplication is performed to construct a joint salient representation of the multimodalities. This characterization precisely locates regions in physical space that simultaneously possess both high infrared thermal radiation and strong polarization texture; based on the fundamental joint features In the main state branch, the flattened one-dimensional basic joint feature sequence and joint significant characterization sequences respectively through linear projection matrix and Projecting the two sequences onto the same parameter dimension space achieves non-rigid alignment of the two sequences in the underlying parameter channel dimension. Subsequently, the mapped joint salient representation is used as a dynamic offset and superimposed onto the mapped basic joint feature sequence to form the modulated parameter generation tensor. Then, the time-varying parameters of the core state-space model SSM are derived. , , This mechanism utilizes joint saliency representation to dynamically amplify the state update step size of the target region. This approach assigns higher state update weights to the state-space model when scanning the spatial location of salient targets, while attenuating background clutter and noise. This feature-guided modulation (FGM) does not perform physical addition or subtraction in the image space, but rather alters the selective scanning behavior of the state-space model at a mathematical level. While ensuring no feature loss, it achieves an efficient, non-rigid, physically-perceived feature alignment method, accurately extracting salient targets shared across modalities. The FGM output stage is infrared feature generation. and Phase 1 polarization characteristics Each channel has 32 channels, and the first stage is a significant modulation feature. Channel 64, the specific formula is as follows:

[0121]

[0122]

[0123]

[0124]

[0125] Specifically, the S32, a bidirectional deep coupling module (BDC), inputs the stage one output features into the stage two BDC for fusion. The BDC, based on the EVSS module, is designed with a macroscopically symmetrical, internally bidirectionally coupled parallel structure. The specific process includes: inputting the stage one infrared features... and the aforementioned stage-one polarization characteristics Channel splicing yields specific complementary features The aforementioned stage-1 saliency modulation features Complementary features to the aforementioned specificity Each branch modulates the other's features as offsets in the underlying parameter space. To achieve deep parameterized interaction and fusion of saliency modulation features and specific complementary features, both branches modulate the other's features as offsets in the underlying parameter space. This bidirectional cross-modulation method, on the one hand, uses specific complementary features containing complete high-frequency details to prevent excessive smoothing of edge textures, and on the other hand, uses saliency modulation features focused on saliency targets to weaken background clutter regions. This ensures that the final fused feature not only retains the jointly saliency common target features, but also preserves the deep, differentiated textures. The bidirectional deep coupling module (BDC) outputs saliency coupling features. and specific coupling characteristics With 64 channels each, the two outputs are concatenated and summarized to avoid semantic aliasing caused by element-wise addition. Orthogonal feature aggregation is achieved at the channel level, resulting in the final output fused features of the Selective Offset Modulation (SOMF) fusion module. The specific formula is shown below:

[0126]

[0127]

[0128]

[0129] S4. Using the feature reconstruction image reconstruction module, the fused features are input into the cascaded decoder. Spatial upsampling is performed progressively through multiple cascaded convolutional layers to reconstruct the output, which is then compared to the infrared intensity image. The infrared polarization fusion image with consistent resolution is a high-quality fusion image with the same resolution as the source image. During the network testing phase, the image fusion method based on the physically prior-guided frequency-space Mamba fusion network in Example 1 has now completed all the steps required for infrared polarization image fusion.

[0130] During the network training phase, image fusion methods based on physically prior-guided frequency-space Mamba fusion networks also include:

[0131] S5. Utilizing a self-supervised loss function module guided by physical priors, a polarization-purified polarization representation is obtained by constructing a polarization mask based on physical priors. Infrared intensity information is then injected to obtain a cross-modal cooperative polarization representation, ultimately forming a nonlinear enhanced polarization representation. This enables the network to retain the infrared background while effectively suppressing noise and significantly enhancing the polarization characteristics of targets such as vehicles in an unsupervised state. The cross-modal cooperative polarization representation and the nonlinear enhanced polarization representation will replace the aforementioned polarization degree image. Different loss functions are fed into the infrared intensity image. Jointly construct target fusion image Provide guidance during the network training phase. Total loss function. Due to strength loss gradient loss and multi-scale weighted structural similarity loss Together they form the following formula:

[0132]

[0133]

[0134]

[0135]

[0136]

[0137] The weights of the three sub-loss functions are set based on experience. , , .

[0138] The intensity loss aims to constrain the fused image at the pixel level, aligning it with the enhanced polarization features in saliently polarized regions while maintaining optimal infrared thermal radiation distribution in the background. The gradient loss aims to measure fine-grained texture information, forcing the fused image to include the most significant edge details from the source images. The multi-scale structural similarity loss aims to force the fused image to conform to the enhanced polarization features through multi-window evaluation. Simultaneously align infrared intensity images with structural information and cross-modal co-polarization characterization .

[0139] In step S5, the nonlinear enhanced polarization characterization process includes:

[0140] S51. Physically Prior Polarization Mask: For example, considering the physical law that vehicles (such as hoods and windshields) in real-world scenarios often exhibit significant horizontal polarization characteristics, this embodiment 1 inputs an infrared intensity image. Polarization image Polarization angle image Based on the physical characteristics of the aforementioned images, such as infrared intensity, degree of polarization, and polarization angle, a joint spatial mask is first constructed to purify the degree of polarization. Then, the degree of polarization is utilized... With polarization angle Accurate solution of normalized horizontal-vertical linear polarization components Characterizing horizontal polarization properties avoids directly reading the original data. This introduces quantization truncation error. When Since horizontal polarization is dominant, a soft-truncation gated activation function with a scaling factor is used. Extracting positive horizontal polarization values ​​(e.g., in the response region), according to Fresnel's law, the smoother the object's surface, the more significant its polarization characteristics. This embodiment 1 utilizes the inverse proportion of the infrared local gradient to characterize smoothness. Construct target surface mask And supplemented by polarization gradient to construct a line mask. Then, the maximum value of the two is taken to obtain the polarization mask based on physical priors. To balance macroscopic significance and edge sharpness, the specific formula is as follows:

[0141]

[0142]

[0143]

[0144] in For the Sobel gradient operator, It is a pixel-level maximum value selection function;

[0145] S52, Polarization characterization purified by mask: using a polarization mask based on physical priors. For the original polarization degree image Feature purification is performed to form a mask-purified polarization characterization. ,

[0146]

[0147] S53. Cross-modal cooperative polarization characterization: Since the polarization mask highly focuses on the target area and loses most of the background radiation, this embodiment 1 introduces infrared intensity as background spatial compensation to form a cross-modal cooperative polarization characterization. :

[0148]

[0149] Based on experience, and Set them to 1.0 and 0.5 respectively;

[0150] S54. Nonlinear Enhanced Polarization Characterization: To compress the dynamic range and achieve significant enhancement of low-value polarization characteristics, cross-modal cooperative polarization characterization is performed in both intensity loss and gradient loss. Gamma nonlinear correction was performed to obtain a nonlinear enhanced polarization characterization. :

[0151]

[0152] Based on experience, The value is 0.5. Minimum value to prevent gradient explosion.

[0153] To verify the effectiveness of the physical prior-guided frequency-space Mamba fusion network and its image fusion method in Embodiment 1, the following experimental setup and evaluation methods were used for effectiveness verification.

[0154] Experimental Setup: The network training used infrared intensity and polarization images from the publicly available LDDRS long-wave infrared polarization dataset for image fusion training and testing. This dataset contains 2113 images with a resolution of 640×512. The dataset covers various complex scenes, including vehicles and pedestrians on highways and urban roads during day and night cycles. In this Example 1, the dataset was randomly divided into a training set (1690 images), a validation set (211 images), and a test set (212 images) to verify the model's generalization performance.

[0155] All experiments were implemented using the PyTorch-based deep learning framework, with a single NVIDIA GeForce RTX 4090 DGPU and an Intel Core i7-12700 CPU. During training, the model was optimized using the ADAM optimizer, with an initial learning rate of 0.0001, a batch size of 4, and 20,000 training iterations.

[0156] Evaluation Methods: Currently, there are relatively few publicly available algorithms specifically for infrared polarization image fusion. Therefore, to comprehensively evaluate the fusion performance of the proposed method, this embodiment 1 also selects visible light polarization-infrared fusion, visible light-infrared fusion, and general image fusion algorithms as comparison algorithms. Based on different network architectures, these comparison methods can be divided into five categories: CNN-based methods SeAFusion, PIPFNet; transformer-based methods SwinFusion, TIPFNet, CPIFuse; GAN-based method FreqGAN; diffusion model-based method DDFM; and Mamba-based methods MambaDFuse, FusionMamba.

[0157] In this embodiment 1, seven indicators are used to quantitatively evaluate image quality: information entropy ( ), standard deviation ( Mutual information ), feature mutual information ( ), a quality metric based on phase consistency ( ), based on variance-weighted structural measures ( Nonlinear related information entropy ( These indicators are respectively based on information capacity and transmission (…). , , Margins and salience ( , , ), visual perception and structure ( The quality of infrared polarization fusion images is measured by three core dimensions. For all metrics, higher values ​​indicate better fusion performance.

[0158] Analysis of experimental results: Figure 6 This paper presents a detailed qualitative comparison of the image fusion method based on a physically prior-guided frequency-space Mamba fusion network (PDSFMamba) in Embodiment 1 with contrasting methods. Local red and blue boxes magnify key details such as fine wires, building textures, vehicles, and pedestrians. When faced with strong background clutter in polarized images, CNN-based SeAFusion, SwinFusion, and Transformer-based TIPFNet result in some disordered stripes in smooth areas of the sky and road surface. CPIFuse ensures background purity, but the result shows less representation of high-frequency polarization edges, resulting in slightly insufficient scene texture details. Generative methods (FreqGAN, DDFM) produce images with lower sharpness of car windows and building edges, but the overall image is smoother. Furthermore, PIPFNet and Mamba-based methods (FusionMamba and MambaDFuse) exhibit strong global contrast, but produce halo effects around strongly polarized targets. In contrast, the proposed method (Ours) demonstrates superior feature extraction and fusion capabilities. As shown in the magnified view, the network accurately extracts and injects the edges of building and vehicle structures, bright windows, and clear pedestrian outlines caused by polarization characteristics from the polarization degree image. It perfectly inherits the pure thermal radiation context of the infrared intensity image in the smooth background area, effectively eliminating clutter interference. This intuitively demonstrates that the proposed method strictly follows physical prior laws, achieving high-quality fusion of significantly polarized material features with a pure infrared background.

[0159] Table 1 shows the detailed quantitative comparison results between the method in Example 1 and the comparative method.

[0160] Table 1. Quantitative comparison results between the method and the comparative method in Example 1.

[0161] (Bold and underlined text indicate the optimal and suboptimal results, respectively)

[0162] 𝑀𝐼 SeAFusion <![CDATA[ 6.1176 ]]> 25.1766 3.4289 0.9310 <![CDATA[ 0.5687 ]]> 0.3513 0.8740 PIPFNet 6.1019 29.3358 <![CDATA[ 4.6674 ]]> <![CDATA[ 0.9346 ]]> 0.3560 0.3946 0.8765 SwinFusion 6.0129 24.4455 4.1146 0.9275 0.5118 0.4568 0.8728 TIPFNet 5.7284 26.2023 2.6136 0.9195 0.3458 0.3539 0.8789 CPIFuse 4.8605 12.7989 3.0729 0.9105 0.0952 0.2083 0.8771 FreqGAN 4.9020 11.7092 4.0228 0.9220 0.2940 0.2648 0.8768 DDFM 5.5247 20.4887 3.0188 0.9241 0.5093 0.4033 <![CDATA[ 0.8805 ]]> FusionMamba 5.9588 23.8453 2.6081 0.9166 0.1071 0.2764 0.8775 MambaDFuse 6.0130 24.4937 4.1611 0.9271 0.5320 <![CDATA[ 0.4737 ]]> 0.8721 Ours 6.3341 <![CDATA[ 29.3164 ]]> 6.6156 0.9352 0.6326 0.4753 0.8849

[0163] In this embodiment 1, the method achieved optimal values ​​for 6 out of 7 indicators, and a suboptimal value for 1. In the dimensions of information capacity and transmission, information entropy... mutual information Nonlinear related information entropy All reached their optimal values ​​(6.3341, 6.6156, 0.8849). This is especially true for the measurement of the total amount of information transmitted. The performance index achieved a 41.74% improvement compared to the suboptimal value (4.6674). This fully demonstrates that the algorithm in this paper not only extracts effective information from the source image to the greatest extent without loss, but also exhibits the strongest nonlinear correlation between the fused image and the source image. In terms of edge preservation and saliency, the performance index measures the degree of feature transfer and phase structure preservation. and Both reached their optimal values ​​(0.9352, 0.6326), among which This represents an 11.24% improvement compared to the suboptimal value (0.5687). It also reflects the overall contrast ratio. The suboptimal value of 29.3164 in the comparison algorithm lags behind the optimal algorithm by only 0.0194. This objectively reflects the advantages of this network in extracting and mapping high-frequency polarization edge features. In the visual perception and structural dimensions, a variance-weighted structural metric is used. The optimal value of 0.4753 was achieved. This indicates that the fused image in Example 1 best matches the perceptual quality of the human visual system, effectively avoiding structural distortion and artifacts.

[0164] Based on a comprehensive qualitative and quantitative analysis, the method in Example 1 outperforms existing advanced fusion algorithms in terms of noise suppression, texture preservation, and highlighting of salient targets, achieving the best performance in the field of infrared polarization image fusion.

[0165] The above description is merely a preferred embodiment of the present invention and does not limit the patent scope of the present invention. Any equivalent structural or procedural transformations made based on the content of the present invention's specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of the present invention.

Claims

1. A frequency-space Mamba fusion network based on physical prior guidance, characterized in that, The frequency-space Mamba fusion network based on physical prior guidance is constructed entirely based on the efficient visual state space module EVSS, including: a dual-modal data acquisition and preprocessing module, used to acquire infrared intensity images collected by a focal plane detector. With polarization image , serving as the dual-modal input of the frequency-space Mamba fusion network guided by physical priors; The frequency-space collaborative feature extraction module is used to extract the infrared intensity image. With the polarization image The inputs are fed into the CNN-Mamba-based frequency-space co-operational multi-scale feature extraction encoder (SFCE) to perform multi-scale feature extraction and output deep infrared features. and deep polarization characteristics ; Selective offset modulation fusion module, used to integrate the deep infrared features and the deep polarization characteristics The input is fed into the state-space based selective offset modulation fusion module (SOMF) for feature fusion to obtain the fused features. ; The feature reconstruction image reconstruction module is used to input the fused features into the cascaded decoder and reconstruct the output along with the infrared intensity image. Infrared polarization fusion images with consistent resolution.

2. An image fusion method based on a physically prior-guided frequency-space Mamba fusion network, characterized in that, The specific steps for using the physical prior-guided frequency-space Mamba fusion network as described in claim 1 are as follows: S1. Using the dual-modal data acquisition and preprocessing module, acquire the infrared intensity image obtained by solving the raw signal collected by the focal plane detector. With polarization image , serving as the bimodal input of the network; S2. Using the frequency-space coordinated feature extraction module, the infrared intensity image is... With the polarization image The inputs are fed into the CNN-Mamba-based frequency-space co-operational multi-scale feature extraction encoder SFCE. Non-weight-sharing twin branches are used for independent forward propagation of the infrared intensity and polarization degree branches. Within each branch, the inputs sequentially pass through a preliminary shallow feature extraction encoder, a wavelet Mamba cross-guided module (WMCG), and a two-stage iterative deep feature extraction encoder DS-IDFE to perform multi-scale feature extraction, outputting deep infrared features. and deep polarization characteristics ; S3. Using a selective offset modulation fusion module, the deep infrared features are... and the deep polarization characteristics The input is fed into the Selective Offset Modulation Fusion (SOMF) module, which is based on the state space layer, for feature fusion. SOMF consists of two stages connected in series: a Feature Guided Modulation (FGM) module and a Bidirectional Deep Coupling (BDC) module, to obtain the fused features. ; S4. Using the feature reconstruction image reconstruction module, the fused features are input into the cascaded decoder. Spatial upsampling is performed progressively through multiple cascaded convolutional layers to reconstruct the output, which is then compared to the infrared intensity image. Infrared polarization fusion images with consistent resolution.

3. The image fusion method based on a physically prior-guided frequency-space Mamba fusion network according to claim 2, characterized in that, The efficient visual state space module (EVSS) process includes: given a two-dimensional feature tensor as input... After layer normalization and linear projection, the data is divided into main feature branches along the channel dimension. With gated branches The input is fed into the efficient state-space operator ESSM; in the main characteristic branch of the efficient state-space operator ESSM, The sequence is sequentially processed through 3×3 convolution and SiLU activation function before entering the efficient two-dimensional selective scanning module ES2D. ES2D utilizes a skip sampling strategy to reduce the computational complexity of global modeling while effectively preserving long-range dependencies. The compressed sequence then enters the core S6 operator to dynamically generate discretized state parameters. , , The hidden state is updated to capture the global context; finally, the structure is restored to a two-dimensional space through efficient fusion and recombination, yielding the main branch output. In the gated branch, Spatial gating graphs are generated only after SiLU activation. Ultimately, the main branch features With gated features Element-wise multiplication yields ; The final feature is output after linear projection and residual connection. The complete formula is shown below: in This is for element-wise multiplication.

4. The image fusion method based on a physically prior-guided frequency-space Mamba fusion network according to claim 2, characterized in that, In step S2, the preliminary shallow feature extraction encoder process includes: processing the infrared intensity image With the polarization image The infrared intensity image is processed by performing two 3×3 convolutional layers with the LeakyReLU modified linear unit activation function. With the polarization image The shallow infrared intensity features are obtained by mapping them to a high-dimensional feature space with 16 channels. and shallow polarization characteristics .

5. The image fusion method based on a physically prior-guided frequency-space Mamba fusion network according to claim 4, characterized in that, In step S2, the wavelet Mamba cross-guided module (WMCG) process includes: processing the shallow infrared intensity features... and the shallow polarization characteristics Discrete Wavelet Transform (DWT) decomposition is performed on each band, resulting in four frequency sub-bands. The HH and LH bands are infrared-guided polarization, the HL band is polarization-guided infrared, and the LL band does not undergo cross-guidance. The guiding and guided branches corresponding to different frequency bands are input into the structurally identical Mamba cross-guidance module (MCG) for cross-guidance. The Mamba cross-guidance module (MCG) is designed based on the efficient visual state space module (EVSS) and target features. As input to the main state branch, guiding features The two branches are used as inputs for the gated explicit guidance branches; explicit physical perception guidance is achieved through element-wise multiplication, and finally mapped back to the original dimension and a residual connection is added to output the guided target features. The specific formula for the Mamba cross-boot module MCG is as follows: in, For target features, The target features are those that have passed through the main state branch. To guide features, This refers to the guiding characteristics of gated branches. The target characteristics after booting, as output by the Mamba cross-booting module; After completing guidance in different frequency bands and achieving denoising and enhancement of each sub-band feature, the four frequency bands of infrared and polarization are reconstructed into spatial domain infrared guidance features through discrete wavelet inverse transform (IDWT). With polarization guidance characteristics .

6. The image fusion method based on a physically prior-guided frequency-space Mamba fusion network according to claim 5, characterized in that, In step S2, the two-stage iterative deep feature extraction encoder (DS-IDFE) process includes: firstly, guiding features from the spatial domain... That is, the spatial domain infrared guidance feature With the polarization guiding feature The inputs are fed into a single-stage iterative deep feature extraction encoder (IDFE), which consists of a high-efficiency visual state space module (EVSS) for global feature extraction and a spatially adaptive gated intensifier (SAGE) for local detail enhancement, connected in parallel. The outputs of the two parallel branches, local and global, are then added element-wise to obtain the single-stage IDFE infrared output. and single-stage IDFE polarization output Then, both branches are convolved and mapped to 32 channels, which are then input into the second-stage iterative deep feature extraction encoder (IDFE) with the same structure. The spatial domain-guided features are explicitly externalized to the entire two-stage iterative deep feature extraction encoder (DS-IDFE). All features are constructed by convolutional upsizing and aligning to 32 channels to build residual alignment paths, ultimately outputting deep features. That is, deep infrared features With deep polarization characteristics The two-stage iterative deep feature extraction encoder DS-IDFE is shown in the following formula: in, To guide features, For the output features of the global branch in a single-level IDFE, The output characteristics of local branches in a single-level IDFE For single-level IDFE output features, This represents the deep features of the final output.

7. The image fusion method based on a physically prior-guided frequency-space Mamba fusion network according to claim 6, characterized in that, In step S2, the spatial adaptive gating enhancer SAGE includes: guiding spatial domain features That is, the spatial domain infrared guidance feature With the polarization guiding feature First, dimensionality is increased through group normalization and 1×1 convolution, followed by a 7×7 large receptive field convolutional layer. Then, the two modal branch features are evenly divided into content branch features along the channel dimension. and gated branch features ,right Generate a mask using the Gaussian error linear unit activation function GELU. and Element-wise multiplication; after dimensionality reduction, a learnable channel scaling factor is applied. Output local branch features The specific formula is as follows: in, To guide features, For content branching features, This is a gated branch feature. Output features for content branches. For gated branch output characteristics, This is a learnable channel scaling factor. This refers to the output characteristics of local branches in a single-level IDFE.

8. The image fusion method based on a physically prior-guided frequency-space Mamba fusion network according to claim 2, characterized in that, In step S3, the Feature Guided Modulation (FGM) process includes: transferring the deep infrared features... and the aforementioned deep polarization characteristics Perform channel splicing to form basic joint features The deep infrared features mentioned and the aforementioned deep polarization characteristics The main state branches are processed separately by the High-Efficiency Vision State Space Module (EVSS) and before the High-Efficiency Two-Dimensional Selective Scan Module (ES2D). and Element-wise multiplication is performed to construct a joint salient representation of the multimodalities. ; in basic joint features In the main state branch, the flattened one-dimensional basic joint feature sequence and joint significant characterization sequences respectively through linear projection matrix and Projecting onto the same parameter dimension space, the mapped joint salient representation is then used as a dynamic offset and superimposed onto the mapped basic joint feature sequence to form the modulated parameter generation tensor. Then, the time-varying parameters of the core state-space model SSM are derived. , , Output stage one infrared feature and Phase 1 polarization characteristics Each channel has 32 channels, and the first stage is a significant modulation feature. Channel 64, the specific formula is as follows: The bidirectional deep coupling module BDC process includes: converting the aforementioned stage one infrared features... and the aforementioned stage one polarization and Channel splicing yields specific complementary features The aforementioned stage-1 saliency modulation features Complementary features to the aforementioned specificity Each branch modulates the other's features as offsets in the underlying parameter space, outputting saliency-coupled features. and specific coupling characteristics Both outputs have 64 channels. The two outputs are concatenated to obtain the final output fusion feature of the Selective Offset Modulation (SOMF) fusion module. The specific formula is shown below:

9. The image fusion method based on a physically prior-guided frequency-space Mamba fusion network according to claim 2, characterized in that, During the network training phase, image fusion methods based on physically prior-guided frequency-space Mamba fusion networks also include: S5. Utilizing a self-supervised loss function module guided by physical priors, a polarization-purified polarization representation is obtained by constructing a polarization mask based on physical priors. Infrared intensity information is then injected to obtain a cross-modal cooperative polarization representation, ultimately forming a nonlinear enhanced polarization representation. The cross-modal cooperative polarization representation and the nonlinear enhanced polarization representation will replace the aforementioned polarization degree image. Different loss functions are fed into the infrared intensity image. Jointly construct target fusion image Guiding the network training phase; total loss function Due to strength loss gradient loss and multi-scale weighted structural similarity loss Together they form the following formula: 。 The weights of the three sub-loss functions are set based on experience. , , .

10. The image fusion method based on a physically prior-guided frequency-space Mamba fusion network according to claim 9, characterized in that, In step S5, the nonlinear enhanced polarization characterization process includes: inputting an infrared intensity image. Polarization image Polarization angle image Based on the physical characteristics of the aforementioned images, such as infrared intensity, degree of polarization, and polarization angle, a joint spatial mask is first constructed to match the degree of polarization. Purification is carried out using polarization degree With polarization angle Accurate solution of normalized horizontal-vertical linear polarization components Characterizing horizontal polarization properties using a soft-truncation gated activation function with a scaling factor. Extract the positive value of horizontal polarization and characterize the smoothness using the inverse ratio of infrared local gradient. Construct target surface mask And supplemented by polarization gradient to construct a line mask. Then, the maximum value of the two is taken to obtain the polarization mask based on physical priors. The specific formula is as follows: in For the Sobel gradient operator, It is a pixel-level maximum value selection function; Using physical prior polarization masks For the original polarization degree image Feature purification is performed to form a mask-purified polarization characterization. , Introducing infrared intensity as background spatial compensation to form a cross-modal cooperative polarization characterization : Based on experience, and Set them to 1.0 and 0.5 respectively; Characterization of cross-modal cooperative polarization in intensity loss and gradient loss Gamma nonlinear correction was performed to obtain a nonlinear enhanced polarization characterization. : Based on experience, The value is 0.

5. Minimum value to prevent gradient explosion.

Images