Low-visibility infrared and polarization image fusion method and system

Abstract

The application discloses a kind of low visibility infrared and polarized image fusion method and system, the method includes: obtaining four-way polarized image data and infrared image, and carrying out feature enhancement extraction and information interaction, obtain the infrared feature of interactive compensation and the linear polarization degree feature of interactive compensation;Based on polarized attention fusion mechanism, the infrared feature of interactive compensation and the linear polarization degree feature of interactive compensation are preliminarily fused, and inject infrared highlight area mask, obtain the fusion result of infrared information enhancement;Based on four-way polarized image data, generate polarized texture residual and inject to the fusion result of infrared information enhancement, obtain the final image fusion result.The application can give consideration to the reservation of infrared significant target information and the recovery of polarized texture microstructure, realize high-fidelity image fusion.The application is a kind of low visibility infrared and polarized image fusion method and system, can be widely applied in image fusion technical field.

Description

Technical Field

[0001] This invention relates to the field of image fusion technology, and in particular to a method and system for fusion of low-visibility infrared and polarized images. Background Technology

[0002] In critical applications such as modern national defense reconnaissance, unmanned aerial vehicles, maritime search and rescue, and industrial inspection, the performance of visual perception systems in extreme environments highly depends on their ability to fuse multimodal information. Traditional infrared-visible light image fusion methods mainly rely on visible light images to provide surface texture and color information. However, in extreme environments such as low light, dense smoke, strong glare, or nighttime, visible light imaging suffers from insufficient light flux or severe medium scattering, resulting in a significant loss of image detail. Infrared thermal imaging can provide significant thermal energy information through the temperature difference between the target and the background, but its spatial resolution is usually low, making it unable to fully capture the microscopic structural features of objects. Furthermore, the contrast of infrared images is close to zero in thermal equilibrium, rendering a single infrared image almost ineffective in extreme environments.

[0003] To overcome the aforementioned limitations, polarization imaging has been introduced as a supplementary method. Polarization images can provide physical information independent of light intensity and temperature, such as surface geometry, roughness, and material properties, effectively supplementing infrared information at the energy and structural levels. However, existing infrared-polarization image fusion methods still have significant limitations. Current methods primarily focus on global feature representation and aggregation, but lack protection mechanisms for high-energy infrared heat source regions. This results in bright infrared targets being easily smoothed or weakened by polarization textures during the fusion process, severely affecting the integrity of heat source information. In low-light or strong fog environments, the microscopic geometric structure and edge information signals in polarization images are weak, and existing methods lack effective enhancement strategies, making it difficult to ensure that microstructural features are clearly presented in the fusion result.

[0004] Furthermore, existing methods also suffer from shortcomings in terms of fusion efficiency and computational complexity. Traditional multi-scale analysis, sparse representation, or saliency detection methods rely on manually designed rules, making them difficult to adapt to diverse scenarios, resulting in low fusion efficiency and a lack of end-to-end optimization capabilities. While deep learning-based methods can extract nonlinear features, convolutional networks are limited by local receptive fields, making it difficult to model long-range cross-modal dependencies. Generative adversarial networks (GANs) suffer from training instability and mode collapse issues. Although Transformer networks possess global modeling capabilities, they still have limitations in handling high-frequency polarization abrupt changes and infrared energy balance. In summary, existing infrared-polarization image fusion methods still face significant technical bottlenecks in extreme environments: they struggle to simultaneously preserve both infrared bright targets and polarization microstructures, lack physical consistency modeling methods for low-light and thermal equilibrium conditions, and lack efficient computational frameworks. This limits the practical application value and reliability of infrared-polarization fusion technology in key application scenarios. Summary of the Invention

[0005] To address the aforementioned technical problems, the present invention aims to provide a method and system for fusing low-visibility infrared and polarized images, which can simultaneously preserve the information of prominent infrared targets and restore the microstructure of polarized textures, thereby achieving high-fidelity image fusion.

[0006] The first technical solution adopted in this invention is: a method for fusing low-visibility infrared and polarization images, comprising the following steps: Acquire four-directional polarization image data and infrared image, and perform feature enhancement extraction and information interaction to obtain interactively compensated infrared features and interactively compensated linear polarization degree features. Based on the polarization attention fusion mechanism, the interactively compensated infrared features and interactively compensated linear polarization features are initially fused, and an infrared high-brightness region mask is injected to obtain the fusion result with enhanced infrared information. The final image fusion result is obtained by generating polarization texture residuals based on four-directional polarization image data and injecting them into the fusion result enhanced by infrared information.

[0007] Furthermore, the step of acquiring four-directional polarization image data and infrared images, and performing feature enhancement extraction and information interaction to obtain interactively compensated infrared features and interactively compensated linear polarization degree features specifically includes: Acquire four-axis polarization image data and infrared images; Image preprocessing of four-axis polarization image data using the Stokes vector method yields a linear polarization image. Feature enhancement extraction and feature stitching are performed on linear polarization degree images and infrared images to obtain infrared deep feature maps and linear polarization degree deep feature maps; Information is exchanged between the infrared deep feature map and the linear polarization degree deep feature map to obtain the interactively compensated infrared features and the interactively compensated linear polarization degree features.

[0008] Furthermore, the step of performing information interaction between the infrared deep feature map and the linear polarization degree deep feature map to obtain the interactively compensated infrared features and the interactively compensated linear polarization degree features specifically includes: The infrared deep feature map and the linear polarization degree deep feature map are concatenated into a two-dimensional tensor and expanded into a sequence form to obtain the preprocessed infrared sequence features and the preprocessed linear polarization degree sequence features. Based on the linear state-space model mechanism, the preprocessed infrared sequence features and the preprocessed linear polarization degree sequence features are sequentially updated and recursively mapped to obtain the updated infrared sequence features and the updated linear polarization degree sequence features. The updated infrared sequence features and the updated linear polarization degree sequence features are sequentially divided and upsampled to obtain interactively compensated infrared features and interactively compensated linear polarization degree features.

[0009] Furthermore, the step of initially fusing the interactively compensated infrared features and the interactively compensated linear polarization degree features based on the polarization attention fusion mechanism, and injecting an infrared highlight region mask to obtain a fusion result with enhanced infrared information, specifically includes: Based on the polarization attention fusion mechanism, the interactively compensated infrared features and interactively compensated linear polarization degree features are initially fused to obtain preliminary image fusion results. Infrared highlight region masks are generated based on infrared images and injected into the preliminary image fusion results to obtain fusion results with enhanced infrared information.

[0010] Furthermore, the step of performing preliminary fusion of the interactively compensated infrared features and the interactively compensated linear polarization degree features based on the polarization attention fusion mechanism to obtain a preliminary image fusion result specifically includes: Local enhancement processing is performed on the interactively compensated infrared features and the interactively compensated linear polarization features, and they are mapped to the same latent spatial dimension for splicing and unfolding to obtain the infrared local enhancement feature sequence and the linear polarization local enhancement feature sequence. Based on the linear dynamic equation, the state of the infrared local enhancement feature sequence and the linear polarization degree local enhancement feature sequence is updated. Then, the enhanced feature recovery is transposed and upsampled to be mapped back to two-dimensional space to obtain the mapped infrared enhancement feature and the mapped linear polarization degree enhancement feature. A lightweight feature detection head is used to extract the display area from linear polarization images and infrared images to obtain infrared high-emissivity area masks and polarization geometric texture significant area masks. The mapped infrared enhancement features, mapped linear polarization enhancement features, infrared high-emissivity region mask, and polarization geometric texture significant region mask are concatenated to obtain preliminary image fusion results.

[0011] Furthermore, the step of generating an infrared highlight region mask based on the infrared image and injecting it into the preliminary image fusion result to obtain an infrared-enhanced fusion result specifically includes: The infrared image is normalized to obtain a normalized infrared image. By introducing brightness threshold and kurtosis threshold, infrared mask generation processing is performed on the normalized infrared image to obtain a continuous infrared mask. The infrared continuous mask is lightly smoothed, and a small-scale morphological opening operation is performed to obtain the infrared highlight area mask. By using an adaptive injection scaling function based on the squared brightness term, the infrared high-brightness region is masked and injected into the preliminary image fusion result, resulting in an infrared-enhanced fusion result.

[0012] Furthermore, the expression for the adaptive injection ratio function based on the brightness square term is as follows: ; In the above formula, This represents the adaptive injection scaling function based on the squared luminance term. This represents the mask for the infrared highlight area.

[0013] Furthermore, the step of generating polarization texture residuals based on four-directional polarization image data and injecting them into the fusion result enhanced by infrared information to obtain the final image fusion result specifically includes: The four-directional polarization image data is decomposed into four sub-bands by Haar wavelet, and then reconstructed by wavelet sub-band enhancement and inverse wavelet transform to obtain the reconstructed polarization data. The reconstructed polarization data is subjected to polarization contrast enhancement processing to obtain enhanced polarization texture data. Multi-scale convolution information is extracted based on the enhanced polarization texture data. Second-order gradient features are extracted from the four-directional polarization image data using the Laplacian operator. The multi-scale convolution information and the second-order gradient features are then concatenated to obtain the polarization texture residual. By introducing a mask control mechanism, the polarization texture residual is injected into the fusion result enhanced by infrared information to obtain the final image fusion result.

[0014] Furthermore, it also includes introducing a global fusion loss function, an infrared saliency loss function, and a polarization texture loss function to construct a total loss function, the specific expression of which is shown below: ; ; ; ; In the above formula, Represents the total loss function. Represents the global fusion loss function. Represents the infrared significance loss function. Represents the polarization texture loss function. , , Indicates the weighting coefficient. This indicates that the brightness distribution of the constrained fused image is balanced between the infrared and polarized images. This means ensuring that the gradient magnitude after fusion retains the strongest edge changes among all modes. This indicates a brightness-adaptive mask. Represents an infrared image. This represents the final image fusion result. Represents the linear polarization degree image. This indicates a normalization operation. Represents the second-order gradient feature. This indicates an adaptive mask for the dark area.

[0015] The second technical solution adopted in this invention is: a low-visibility infrared and polarization image fusion system, comprising: The first module is used to acquire four-directional polarization image data and infrared images, and to perform feature enhancement extraction and information interaction to obtain interactively compensated infrared features and interactively compensated linear polarization degree features. The second module is used to perform preliminary fusion of interactively compensated infrared features and interactively compensated linear polarization degree features based on the polarization attention fusion mechanism, and inject infrared high brightness region mask to obtain fusion result with enhanced infrared information. The third module is used to generate polarization texture residuals based on four-way polarization image data and inject them into the fusion result enhanced by infrared information to obtain the final image fusion result.

[0016] The beneficial effects of the method and system of this invention are as follows: This invention acquires four-directional polarization image data and infrared images, and performs feature enhancement extraction and information interaction to obtain interactively compensated infrared features and interactively compensated linear polarization degree features, which can take into account both local texture and global structural information. Subsequently, the multi-scale features are expanded into a sequential input linear selective state-space model, and global interaction between infrared energy information and polarization texture features is achieved through recursive update of the state vector, effectively modeling cross-modal long-range dependencies and ensuring the physical consistency of the fusion result and the structure-energy dual-domain fusion. Furthermore, based on the polarization attention fusion mechanism, the interactively compensated infrared features and interactively compensated linear polarization degree features are initially fused, and an infrared highlight region mask is injected to obtain a fusion result with enhanced infrared information. As a result, the infrared high-brightness injection module performs soft-threshold masking on the infrared high-brightness area and injects preliminary fusion features to achieve complete preservation of infrared salient targets and protection of high-brightness areas. Finally, polarization texture residuals are generated based on four-directional polarization image data and injected into the fusion result enhanced by infrared information to obtain the final image fusion result. Multi-scale enhancement of high-frequency textures in the polarization image is performed in the wavelet domain and gradient domain, and microstructural details are restored by adaptive contrast adjustment to achieve high-fidelity restoration of polarization textures. In order to ensure the reliability of the fusion result at both the visual and physical levels, it can completely preserve the information of infrared salient targets in extreme low visibility environments, while restoring polarization microstructural textures with high fidelity, significantly improving the accuracy, physical consistency and detail representation of multimodal image fusion. Attached Figure Description

[0017] Figure 1 This is a flowchart of the steps of a low-visibility infrared and polarization image fusion method of the present invention; Figure 2 This is a structural block diagram of a low-visibility infrared and polarization image fusion system according to the present invention; Figure 3 This is a schematic diagram of the image fusion process provided in a specific embodiment of the present invention; Figure 4 This is a schematic diagram of the polarization attention fusion mechanism module provided in a specific embodiment of the present invention; Figure 5 This is a schematic diagram of an infrared high-brightness injection module provided in a specific embodiment of the present invention; Figure 6 This is a schematic diagram of the polarization feature enhancement module provided in a specific embodiment of the present invention; Figure 7 This is a schematic diagram showing the comparison results between this embodiment and seven image fusion methods provided in a specific embodiment of the present invention. Detailed Implementation

[0018] The present invention will now be described in further detail with reference to the accompanying drawings and specific embodiments. The step numbers in the following embodiments are only for ease of explanation and do not limit the order of the steps. The execution order of each step in the embodiments can be adapted according to the understanding of those skilled in the art.

[0019] This embodiment addresses the problems of lost infrared bright target information, smoothed polarization microstructure texture, and lack of detail in existing infrared-polarization image fusion methods under extremely low visibility conditions. It proposes an infrared-polarization image fusion method based on a linear selective state-space model. The overall processing flow includes multi-scale feature extraction, cross-modal long-range dependency modeling, preliminary fusion, infrared bright target injection, and polarization feature enhancement. The entire model design aims to balance the preservation of infrared salient target information and the restoration of polarization texture microstructure, while maintaining physical consistency and high fidelity.

[0020] Reference Figure 1 This invention provides a method for fusing low-visibility infrared and polarized images, the method comprising the following steps: S100: Acquire four-directional polarization image data and infrared image, and perform feature enhancement extraction and information interaction to obtain interactively compensated infrared features and interactively compensated linear polarization degree features. S110, Acquire four-axis polarization image data and infrared image; S120. Image preprocessing of the four-axis polarization image data is performed using the Stokes vector method to obtain a linear polarization degree image. In this embodiment, image preprocessing uses the Stokes vector method to process the four-axis polarization image data. The process is performed to obtain a linear polarization degree (DOLP) image. The specific calculation is shown in the following formula: ; in, For Stokes vectors, This represents the total light intensity component. The difference between the linear polarization components in the 0° and 90° polarization directions. The difference between the linear polarization components in the 45° and 135° polarization directions. This is the difference between the right-hand circular polarization component and the left-hand circular polarization component. Image intensity at polarization direction 0° Image intensity at a polarization direction of 45°. Image intensity at a polarization direction of 90° Image intensity at a polarization direction of 135°. The intensity of right-handed circularly polarized light. This represents the intensity of left-handed circularly polarized light.

[0021] Then, the Stokes parameters are further normalized according to the normalized polarization parameter calculation formula to obtain the polarization parameters after removing light intensity interference. , , Its expression is: ; Then, using the normalized polarization parameters, the degree of linear polarization (DOLP) is calculated according to the formula for calculating the degree of linear polarization. DOLP is used to characterize the proportion of linearly polarized light in the total light intensity. Its expression is: ; S130. Perform feature enhancement extraction and feature stitching processing on the linear polarization degree image and the infrared image to obtain the infrared deep feature map and the linear polarization degree deep feature map. In this embodiment, feature enhancement extraction of infrared and polarization images is achieved. First, the infrared image is... and linear polarization image The infrared deep feature maps are obtained by entering the feature enhancement and extraction module respectively. Deep feature map of linear polarization The feature extraction process of this module is as follows: First, input the... and Features at different semantic levels are extracted through multiple layers of convolution and channel attention mechanisms. A total of three feature extraction paths are set up, corresponding to feature maps at original resolution, 2x downsampling, and 4x downsampling, respectively. The final layer uses channel attention to dynamically adjust the feature response and output deep features. and Secondly, due to and Significant differences exist in spatial texture distribution. To capture multi-level structural information, the input deep features... and Then use respectively , and Three different scales of convolution operations were followed by batch normalization. and nonlinear activation functions Then, the feature information obtained from different receptive fields is concatenated into Cat, as shown in the multi-scale feature extraction formula: ; ; ; ; This module ensures that the model can take into account details at different scales in complex scenes and output corresponding multi-scale fusion features. and .

[0022] S140. Perform information interaction between the infrared deep feature map and the linear polarization degree deep feature map to obtain interactively compensated infrared features and interactively compensated linear polarization degree features.

[0023] Specifically, the infrared deep feature map and the linear polarization degree deep feature map are concatenated into a two-dimensional tensor and expanded into a sequence form to obtain preprocessed infrared sequence features and preprocessed linear polarization degree sequence features. Based on the linear state-space model mechanism, the preprocessed infrared sequence features and preprocessed linear polarization degree sequence features are sequentially updated and recursively mapped to obtain updated infrared sequence features and updated linear polarization degree sequence features. The updated infrared sequence features and updated linear polarization degree sequence features are sequentially divided and upsampled to obtain interactively compensated infrared features and interactively compensated linear polarization degree features.

[0024] In this embodiment, cross-modal long-range dependency modeling is achieved, and the output is... and The input information interaction module enables cross-modal global dependency modeling, resulting in enhanced cross-modal features. The specific information interaction module is as follows: First of all and The components are concatenated into a two-dimensional tensor and expanded into a sequence. Linear recursive modeling of long-range spatial dependencies is achieved through the state update and state output formulas, as shown in the following expressions: ; ; in The state-space model represents the state-space model in the th... The hidden state vector at each position, Indicates the current input features. This indicates the corresponding output feature. , and It is a learnable parameter matrix used to implement a linear recursive mapping from input features to hidden states and output features.

[0025] Secondly, the obtained By dividing the data and upsampling it separately, we can obtain the result that achieves interactive compensation in the semantic space. and Unlike the self-attention mechanism of Transformer, Mamba's state transition process is linear, reducing computational complexity from O(N²) to O(N), significantly lowering memory overhead while maintaining global dependency capture capabilities. After passing through the information exchange module... and Mapped to implement interactive compensation and This provides a unified feature base for subsequent integration.

[0026] S200. Based on the polarization attention fusion mechanism, the interactively compensated infrared features and interactively compensated linear polarization features are initially fused, and an infrared high-brightness region mask is injected to obtain the fusion result with enhanced infrared information. S210. Based on the polarization attention fusion mechanism, the interactively compensated infrared features and the interactively compensated linear polarization degree features are initially fused to obtain preliminary image fusion results. Specifically, the interactively compensated infrared features and the interactively compensated linear polarization features are locally enhanced and mapped to the same latent spatial dimension for splicing and unfolding, resulting in an infrared local enhancement feature sequence and a linear polarization local enhancement feature sequence. Based on the linear dynamic equation, the infrared local enhancement feature sequence and the linear polarization local enhancement feature sequence are updated in state. After transposing and upsampling operations through enhancement feature recovery, they are mapped back to two-dimensional space, resulting in mapped infrared enhancement features and mapped linear polarization enhancement features. A lightweight feature detection head is used to extract the display area from the linear polarization image and the infrared image, resulting in an infrared high-emissivity region mask and a polarization geometric texture significant region mask. The mapped infrared enhancement features, the mapped linear polarization enhancement features, the infrared high-emissivity region mask, and the polarization geometric texture significant region mask are spliced ​​together to obtain the preliminary image fusion result.

[0027] In this embodiment, as Figure 4 As shown, the initial fusion of polarization-guided attention has achieved interactive compensation. and Preliminary fused features are obtained through the polarization attention fusion mechanism module. The specific polarization attention fusion mechanism module is as follows: First, use two convolution pairs. and Local feature enhancement is performed by mapping infrared energy features and polarization texture features to the same latent spatial dimension. Then, the enhanced infrared and polarization features are concatenated and unfolded into a sequence. The cross-spatial dependencies are modeled using linear dynamic equations, as shown in the state update and output formulas. Finally, based on the enhanced feature recovery formula, the sequence is transposed... Operation and upsampling Operation will Mapping back to two-dimensional space and upsampling to recover the enhanced features obtained after dynamic equation mapping The enhanced feature recovery formula is shown below: ; Then, lightweight feature detection heads were used to analyze the input. and The display area is extracted, and the formula for estimating the salient region is shown below: ; in For input First, perform a convolution operation, then activate the function using a non-linear ReLU function. Indicates first to First perform convolution, then map to a sigmoid function. The masks representing the automatically estimated infrared high-emissivity regions are obtained respectively. And a mask representing the salient region of the automatically estimated polarization geometry texture. .

[0028] Finally we will , and The weights are then concatenated and enhanced using a salient region estimation formula. And divided into and Finally, based on the preliminary fusion output formula, it is compared with... and The corresponding multiplication and addition yields the preliminary fusion result. The preliminary fusion output formula is shown below: ; S220. Generate an infrared highlight area mask based on the infrared image and inject it into the preliminary image fusion result to obtain an infrared information-enhanced fusion result.

[0029] Specifically, the infrared image is normalized to obtain a normalized infrared image; a brightness threshold and a kurtosis threshold are introduced, and an infrared mask generation process is performed on the normalized infrared image to obtain a continuous infrared mask; the continuous infrared mask is lightly smoothed, and a small-scale morphological opening operation is performed to obtain an infrared highlight region mask; the infrared highlight region mask is injected into the preliminary image fusion result through an adaptive injection ratio function based on the brightness square term to obtain a fusion result with enhanced infrared information.

[0030] In this embodiment, as Figure 5 As shown, the injection of infrared high-brightness information is achieved, and the preliminary fusion results are obtained. and Input the infrared highlight injection module to obtain the fusion result with enhanced infrared information. The infrared high-brightness injection module is shown below: First, we obtain the normalized infrared image according to the infrared normalization formula. The infrared normalization formula is shown below: ; Then, by continuously adjusting the brightness threshold during model training... and steepness threshold Generate a continuous mask according to the infrared mask generation formula. Finally, for According to the pooling kernel After performing a light smoothing operation, a small-scale morphological opening operation is performed to obtain the infrared highlight region mask. The formula for generating the infrared mask is as follows: ; Secondly, to prevent the infrared highlights from causing a "bleaching" effect on the background, this embodiment proposes an adaptive injection ratio function based on the square of the brightness term. By combining linear and squared terms, infrared injection is enhanced in high-brightness areas, while energy compensation is automatically reduced in medium- and low-brightness areas, making the transition more natural. The expression for the adaptive injection ratio function based on the square of the brightness term is: ; Finally, the infrared enhancement fusion formula was applied to achieve... The fusion result obtained after injecting significant infrared information is based on this. The infrared enhancement fusion formula is shown below: ; In the above formula, This indicates the fusion result with enhanced infrared information.

[0031] S300: Based on the four-directional polarization image data, polarization texture residuals are generated and injected into the fusion result enhanced by infrared information to obtain the final image fusion result.

[0032] Specifically, the four-directional polarization image data is decomposed into four sub-bands using Haar wavelets, and then reconstructed using wavelet sub-band enhancement and inverse wavelet transform to obtain the reconstructed polarization data. Polarization contrast enhancement processing is then applied to the reconstructed polarization data to obtain enhanced polarization texture data. Multi-scale convolution information is extracted based on the enhanced polarization texture data, and second-order gradient features are extracted from the four-directional polarization image data using the Laplacian operator. The multi-scale convolution information and second-order gradient features are then concatenated to obtain the polarization texture residual. A masking mechanism is introduced to inject the polarization texture residual into the infrared information enhancement fusion result, resulting in the final image fusion result.

[0033] In this embodiment, as Figure 6 As shown, polarization texture enhancement is achieved by... , and The final fusion result is obtained by inputting the polarization feature enhancement module. The polarization feature enhancement module is detailed below: ; First, use Haar wavelets to... Decomposed into four sub-bands ( ),in , , , These represent the low-frequency sub-band, horizontal high-frequency sub-band, vertical high-frequency sub-band, and diagonal high-frequency sub-band of the polarization image after Haar wavelet decomposition, respectively. These four sub-bands are then sequentially enhanced using the wavelet sub-band enhancement formula, and finally reconstructed using inverse wavelet transform. This enables accurate reconstruction of high-frequency textures in the frequency domain.

[0034] Then, according to the polarization contrast enhancement formula, the following was obtained. The polarization contrast enhancement formula significantly improves the texture of low-contrast areas (such as shadows or diffuse areas) as shown below: ; in For average pooling operation, For max pooling operation, Indicates input First, perform a convolution operation, then activate using a non-linear ReLU function, followed by another convolution operation and then mapping to a Sigmoid function. .

[0035] Furthermore, to further supplement the feature details, the Laplacian operator was also used to... Extracting second-order gradient features And multi-scale convolutional information extracted for polarization texture details at different spatial scales. The four polarization texture features are then sequentially concatenated and fused through convolution to form the polarization texture residual. .

[0036] Finally, to prevent interference with target energy in the high-brightness infrared region, a mask control mechanism is introduced through the polarization residual injection formula. First, texture residuals are suppressed in the highlight areas, and texture weights are amplified in the background areas. Finally, polarization residuals are injected to obtain the final fusion result. The polarization residual injection formula is shown below: ; Finally, to ensure that the fused image maintains physical consistency in both the infrared energy domain and the polarization geometry domain, Polar-Mamba designed a multi-objective joint optimization mechanism, which mainly includes three core loss terms: , , These four aspects correspond to global consistency, preservation of infrared highlight targets, and enhancement of texture details, respectively, optimizing the model's cross-modal fusion performance from different levels. The total loss function formula is shown below: ; in This represents the weight coefficients corresponding to each learnable loss term. Indicates the global fusion loss. Indicates significant loss in infrared signature. This indicates the loss of polarization texture.

[0037] Firstly, the fusion stage needs to ensure that the output image closely matches the energy distribution of the infrared image and the texture information of the polarization image in terms of both brightness and structure. This section, based on the global fusion loss formula, constrains the physical rationality of the fusion output from two perspectives: energy transfer and structural consistency. The normalization process is performed according to the infrared normalization formula. This represents the first gradient of X. This involves constraining the brightness distribution of the fused image to achieve a balance between the infrared and polarized images, while This ensures that the gradient magnitude after fusion retains the strongest edge changes among all modes. Together, these two mechanisms achieve a synergistic balance between infrared energy transfer and polarization structure alignment at the physical level, forming the fundamental fusion constraint. The global fusion loss function is shown below: ; ; ; Secondly, infrared loss Primarily addressing the issue of weakened infrared saliency of targets in low visibility, a brightness-adaptive mask is introduced to distinguish infrared bright areas, thereby achieving infrared-dominated energy injection and polarization-dominated background control, as shown in the infrared saliency loss formula. Firstly, regarding... Normalization is performed, and then high-radiation regions are selected based on the threshold value greater than 0.8 to obtain a brightness-adaptive mask. The first item is for the infrared bright area. Apply strong constraints to ensure the complete preservation of heat source energy; the second item is in the background area. Adding polarization constraints prevents infrared highlight overflow from causing background bleaching. From a physical perspective, this mechanism achieves spatially adaptive energy constraints: preserving thermal radiation features in salient target areas and initially enhancing polarization texture features in insignificant areas, thus achieving preliminary compatibility between heavy infrared energy enhancement and weak structure enhancement. The infrared saliency loss formula is shown below: ; at last The discernibility of polarization textures after fusion is maintained through second-order Laplacian constraints and dark area masking mechanisms, with particular emphasis on the microstructure restoration of dark and weakly reflective regions. Firstly, Normalization is performed, and then regions with a threshold less than 0.4 are marked as dark areas to obtain an adaptive mask for dark areas. The first term ensures that the fused image retains the local texture of the polarized image at the detail level, preventing the loss of high-frequency details after infrared energy injection, and achieving high-fidelity reconstruction of surface microstructures. The second term enhances the preservation of polarization features in low-brightness areas, ensuring that shadow and dark surface details are not overwhelmed by the infrared signal. The specific expression for polarization texture loss is as follows: ; In the above formula, Represents the total loss function. Represents the global fusion loss function. Represents the infrared significance loss function. Represents the polarization texture loss function. , , Indicates the weighting coefficient. This indicates that the brightness distribution of the constrained fused image is balanced between the infrared and polarized images. This means ensuring that the gradient magnitude after fusion retains the strongest edge changes among all modes. This indicates a brightness-adaptive mask. Represents an infrared image. This represents the final image fusion result. Represents the linear polarization degree image. This indicates a normalization operation. Represents the second-order gradient feature. This indicates an adaptive mask for the dark area.

[0038] In summary, such as Figure 3 As shown, this embodiment aims to address the problems of traditional infrared-polarization image fusion in extreme environments such as low light, strong glare, thermal equilibrium, and nighttime, including loss of infrared bright target information, smoothing of polarization microstructure texture, and loss of image details. The method in this embodiment first performs multi-layer convolution processing on the infrared intensity image and the polarization intensity image, and extracts multi-scale features using a channel attention mechanism to balance local texture and global structural information. Then, the multi-scale features are unfolded into a sequential input linear selective state-space model. Global interaction between infrared energy information and polarization texture features is achieved through recursive updating of the state vector, effectively modeling cross-modal long-range dependencies and ensuring the physical consistency of the fusion result and the fusion of the structure-energy dual domains. Next, the polarization attention fusion mechanism module adaptively weights and fuses the infrared energy and polarization texture to generate preliminary fusion features. Simultaneously, the infrared brightness injection module further enhances the infrared... Highlighted areas are subjected to soft-threshold masking and injected with preliminary fusion features to achieve complete preservation of infrared salient targets and protection of bright areas. Subsequently, in the polarization feature enhancement module, high-frequency textures of the polarization image are enhanced at multiple scales in the wavelet and gradient domains, and microstructural details are restored by adaptive contrast adjustment, achieving high-fidelity restoration of polarization textures. To ensure the reliability of the fusion result at both the visual and physical levels, this invention designs a joint optimization training strategy. End-to-end training is performed using a multi-objective loss function that considers global brightness consistency, infrared saliency constraints, and polarization texture fidelity constraints, ensuring that the fusion result balances detail fidelity, infrared energy preservation, and texture enhancement. Through the above technical solutions, the method of this invention can completely preserve infrared salient target information in extremely low visibility environments, while simultaneously restoring polarization microstructure textures with high fidelity. This significantly improves the accuracy, physical consistency, and detail representation of multimodal image fusion, and can be widely applied to advanced vision tasks in complex environments such as national defense reconnaissance, unmanned aerial vehicles, maritime search and rescue, and industrial inspection.

[0039] Finally, to further illustrate the effectiveness of this method, this embodiment... Figure 7 The results of two sets of linear polarization images and infrared image fusion comparison methods are presented. In the first set of images, it can be seen that our method maintains a natural energy transition in both bright and dark areas: the wall details are intact, the infrared bright human shadow information is also clear and delicate, and the effective suppression of infrared redundant information can be clearly observed in the magnified area. In the second set of images, the human body area in our invention maintains a complete thermal energy contour, and the wall information is closest to the original linear polarization image, demonstrating the best synergistic performance of brightness and texture, thus demonstrating the effectiveness of the invention.

[0040] As can be seen in Table 1 regarding the specific quantitative indicators for the seven methods, the method in this embodiment... , AG, SF Both are optimal, firstly in terms of human visual perception. and The method achieved optimal results, reflecting the best coordination of the fused image in terms of overall structure, energy, and spatial alignment. Secondly, in terms of physical fidelity, the method achieved optimal results in both AG and SF indices, which directly reflect image sharpness and microtexture richness, further confirming the effectiveness and authenticity of the method. In summary, the method proposed in this paper achieves optimal results in balancing subjective vision and objective indicators.

[0041] Table 1 shows the average values ​​of various fusion indices obtained from six sets of linear polarization degree images and infrared image pairs. ; Reference Figure 2 A low-visibility infrared and polarization image fusion system, comprising: The first module 201 is used to acquire four-directional polarization image data and infrared image, and perform feature enhancement extraction and information interaction to obtain interactively compensated infrared features and interactively compensated linear polarization degree features. The second module 202 is used to perform preliminary fusion of interactively compensated infrared features and interactively compensated linear polarization degree features based on the polarization attention fusion mechanism, and inject infrared high brightness region mask to obtain fusion result with enhanced infrared information. The third module 203 is used to generate polarization texture residuals based on four-way polarization image data and inject them into the fusion result of infrared information enhancement to obtain the final image fusion result.

[0042] The content of the above method embodiments is applicable to this system embodiment. The specific functions implemented in this system embodiment are the same as those in the above method embodiments, and the beneficial effects achieved are also the same as those achieved in the above method embodiments.

[0043] The above is a detailed description of the preferred embodiments of the present invention. However, the present invention is not limited to the embodiments described. Those skilled in the art can make various equivalent modifications or substitutions without departing from the spirit of the present invention. All such equivalent modifications or substitutions are included within the scope defined by the claims of this application.

Claims

1. A method for fusing low-visibility infrared and polarization images, characterized in that, Includes the following steps: Acquire four-directional polarization image data and infrared image, and perform feature enhancement extraction and information interaction to obtain interactively compensated infrared features and interactively compensated linear polarization degree features. Based on the polarization attention fusion mechanism, the interactively compensated infrared features and interactively compensated linear polarization features are initially fused, and an infrared high-brightness region mask is injected to obtain the fusion result with enhanced infrared information. The final image fusion result is obtained by generating polarization texture residuals based on four-way polarization image data and injecting them into the fusion result enhanced by infrared information.

2. The low-visibility infrared and polarization image fusion method according to claim 1, characterized in that, The step of acquiring four-directional polarization image data and infrared images, and performing feature enhancement extraction and information interaction to obtain interactively compensated infrared features and interactively compensated linear polarization degree features specifically includes: Acquire four-axis polarization image data and infrared images; Image preprocessing of four-axis polarization image data using the Stokes vector method yields a linear polarization image. Feature enhancement extraction and feature stitching are performed on linear polarization degree images and infrared images to obtain infrared deep feature maps and linear polarization degree deep feature maps; Information is exchanged between the infrared deep feature map and the linear polarization degree deep feature map to obtain the interactively compensated infrared features and the interactively compensated linear polarization degree features.

3. The low-visibility infrared and polarization image fusion method according to claim 2, characterized in that, The step of exchanging information between the infrared deep feature map and the linear polarization degree deep feature map to obtain the interactively compensated infrared features and the interactively compensated linear polarization degree features specifically includes: The infrared deep feature map and the linear polarization degree deep feature map are concatenated into a two-dimensional tensor and expanded into a sequence form to obtain the preprocessed infrared sequence features and the preprocessed linear polarization degree sequence features. Based on the linear state-space model mechanism, the preprocessed infrared sequence features and the preprocessed linear polarization degree sequence features are sequentially updated and recursively mapped to obtain the updated infrared sequence features and the updated linear polarization degree sequence features. The updated infrared sequence features and the updated linear polarization degree sequence features are sequentially divided and upsampled to obtain interactively compensated infrared features and interactively compensated linear polarization degree features.

4. The low-visibility infrared and polarization image fusion method according to claim 3, characterized in that, The step of initially fusing the interactively compensated infrared features and the interactively compensated linear polarization degree features based on the polarization attention fusion mechanism, and injecting an infrared highlight region mask to obtain a fusion result with enhanced infrared information, specifically includes: Based on the polarization attention fusion mechanism, the interactively compensated infrared features and interactively compensated linear polarization degree features are initially fused to obtain preliminary image fusion results. Infrared highlight region masks are generated based on infrared images and injected into the preliminary image fusion results to obtain fusion results with enhanced infrared information.

5. The low-visibility infrared and polarization image fusion method according to claim 4, characterized in that, The step of performing preliminary fusion of interactively compensated infrared features and interactively compensated linear polarization degree features based on the polarization attention fusion mechanism to obtain preliminary image fusion results specifically includes: Local enhancement processing is performed on the interactively compensated infrared features and the interactively compensated linear polarization features, and they are mapped to the same latent spatial dimension for splicing and unfolding to obtain the infrared local enhancement feature sequence and the linear polarization local enhancement feature sequence. Based on the linear dynamic equation, the state of the infrared local enhancement feature sequence and the linear polarization degree local enhancement feature sequence is updated. Then, the enhanced feature recovery is transposed and upsampled to be mapped back to two-dimensional space to obtain the mapped infrared enhancement feature and the mapped linear polarization degree enhancement feature. A lightweight feature detection head is used to extract the display area from linear polarization images and infrared images to obtain infrared high-emissivity area masks and polarization geometric texture significant area masks. The mapped infrared enhancement features, mapped linear polarization enhancement features, infrared high-emissivity region mask, and polarization geometric texture significant region mask are concatenated to obtain preliminary image fusion results.

6. The low-visibility infrared and polarization image fusion method according to claim 5, characterized in that, The step of generating an infrared highlight region mask based on the infrared image and injecting it into the preliminary image fusion result to obtain an infrared-enhanced fusion result specifically includes: The infrared image is normalized to obtain a normalized infrared image. By introducing brightness threshold and kurtosis threshold, infrared mask generation processing is performed on the normalized infrared image to obtain a continuous infrared mask. The infrared continuous mask is lightly smoothed, and a small-scale morphological opening operation is performed to obtain the infrared highlight area mask. By using an adaptive injection scaling function based on the squared brightness term, the infrared high-brightness region is masked and injected into the preliminary image fusion result, resulting in an infrared-enhanced fusion result.

7. The low-visibility infrared and polarization image fusion method according to claim 6, characterized in that, The specific expression for the adaptive injection ratio function based on the squared luminance term is as follows: ； In the above formula, This represents the adaptive injection scaling function based on the squared luminance term. This represents the mask for the infrared highlight area.

8. The low-visibility infrared and polarization image fusion method according to claim 7, characterized in that, The step of generating polarization texture residuals based on four-directional polarization image data and injecting them into the fusion result enhanced by infrared information to obtain the final image fusion result specifically includes: The four-directional polarization image data is decomposed into four sub-bands by Haar wavelet, and then reconstructed by wavelet sub-band enhancement and inverse wavelet transform to obtain the reconstructed polarization data. The reconstructed polarization data is subjected to polarization contrast enhancement processing to obtain enhanced polarization texture data. Multi-scale convolution information is extracted based on the enhanced polarization texture data. Second-order gradient features are extracted from the four-directional polarization image data using the Laplacian operator. The multi-scale convolution information and the second-order gradient features are then concatenated to obtain the polarization texture residual. By introducing a mask control mechanism, the polarization texture residual is injected into the fusion result enhanced by infrared information to obtain the final image fusion result.

9. The low-visibility infrared and polarization image fusion method according to claim 8, characterized in that, It also includes introducing a global fusion loss function, an infrared saliency loss function, and a polarization texture loss function to construct a total loss function, the specific expression of which is shown below: ； ； ； ； In the above formula, Represents the total loss function. Represents the global fusion loss function. Represents the infrared significance loss function. Represents the polarization texture loss function. , , Indicates the weighting coefficient. This indicates that the brightness distribution of the constrained fused image is balanced between the infrared and polarized images. This means ensuring that the gradient magnitude after fusion retains the strongest edge changes among all modes. This indicates a brightness-adaptive mask. Represents an infrared image. This represents the final image fusion result. Represents the linear polarization degree image. This indicates a normalization operation. Represents the second-order gradient feature. This indicates an adaptive mask for the dark area.

10. A low-visibility infrared and polarization image fusion system, characterized in that, Includes the following modules: The first module is used to acquire four-directional polarization image data and infrared images, and to perform feature enhancement extraction and information interaction to obtain interactively compensated infrared features and interactively compensated linear polarization degree features. The second module is used to perform preliminary fusion of interactively compensated infrared features and interactively compensated linear polarization degree features based on the polarization attention fusion mechanism, and inject infrared high brightness region mask to obtain fusion result with enhanced infrared information. The third module is used to generate polarization texture residuals based on four-way polarization image data and inject them into the fusion result enhanced by infrared information to obtain the final image fusion result.

Images