Infrared and visible image fusion method and system based on modal adaptive conversion and co-visibility perception multi-scale registration

Abstract

The present application belongs to the technical field of infrared and visible light image fusion, and particularly relates to an infrared and visible light image fusion method and system based on modal adaptive conversion and co-visibility perception multi-scale registration. The method comprises: mapping a visible light image to a pseudo-infrared domain based on a modal adaptive transformation network to obtain a pseudo-infrared image; registering an infrared image and the pseudo-infrared image based on a co-visibility perception multi-scale registration network to obtain a registered infrared image; and fusing the registered infrared image and the visible light image based on a cross-modal interaction fusion module to obtain a fused image. The present application realizes effective integration of infrared salient target information and visible light texture details.

Description

Technical Field

[0001] This invention belongs to the field of infrared and visible light image fusion technology, specifically involving an infrared and visible light image fusion method and system based on modality adaptive conversion and co-visibility perception multi-scale registration. Background Technology

[0002] In infrared and visible light image fusion tasks, cross-modal differences and spatial misregistration are significant factors affecting fusion performance. Due to the different imaging principles of the two types of images, their feature distributions and structural responses are significantly inconsistent, making it difficult to establish stable correspondences between cross-modal features. Simultaneously, changes in viewpoint and dynamic scene effects often lead to various complex geometric mismatches between images, such as translation, rotation, and non-rigid deformation, further increasing the difficulty of alignment. Current methods mostly focus on cross-modal feature alignment or modal transformation. However, these methods have significant limitations. First, relying on a unified feature space or direct cross-modal matching is insufficient to effectively address the problem of modal distribution differences, often resulting in unstable matching and gradual error accumulation in complex scenes. Secondly, while modal transformation can reduce the differences between modalities to some extent, it often does not explicitly model the spatial geometric relationships and adaptable regions. Once there are local deformations or non-co-visual regions, it is easy to cause incorrect matching, which in turn restricts the overall registration accuracy. Moreover, many current methods treat registration and fusion separately and do not perform unified modeling for the collaborative expression of cross-modal information, so they cannot fully leverage the advantages of the two modalities complementing each other.

[0003] To address the aforementioned shortcomings, this invention proposes an infrared and visible light image fusion method based on modal adaptive transformation and co-vision perception multi-scale registration, addressing three aspects: modal difference mitigation, matching region modeling, and cross-modal fusion. First, modal adaptive transformation maps the visible light image to the pseudo-infrared domain, reducing cross-modal differences at the feature distribution level and transforming the cross-modal registration problem into an approximate single-modal matching problem. Second, a co-vision perception mechanism is introduced to explicitly model regions with stable correspondences in the cross-modal images, and combined with multi-scale feature representation and coarse-to-fine deformation field estimation, achieving fine alignment under complex deformation conditions. Finally, a cross-modal fusion module is constructed in a unified feature space to adaptively integrate infrared salient information and visible light texture details, thereby obtaining a fusion result that combines structural consistency and detail fidelity. Summary of the Invention

[0004] To address the problems existing in the prior art, this invention provides an infrared and visible light image fusion method and system based on modal adaptive conversion and co-visibility perception multi-scale registration. The aim is to achieve synergistic optimization from three levels: modal alignment, spatial registration, and feature fusion, to obtain a fusion result that combines structural consistency and detail fidelity.

[0005] To achieve the above objectives, the present invention provides the following solution: An infrared and visible light image fusion method based on modality adaptive transformation and co-visibility perception multi-scale registration, the method comprising: Based on a modality adaptive transformation network, a visible light image is mapped to the pseudo-infrared domain to obtain a pseudo-infrared image; Based on a common-view perception multi-scale registration network, the infrared image and the pseudo-infrared image are registered to obtain the registered infrared image. Based on the cross-modal interactive fusion module, the registered infrared image and the visible light image are fused to obtain a fused image.

[0006] Preferably, the method for obtaining a pseudo-infrared image by mapping a visible light image to the pseudo-infrared domain based on a modality adaptive transform network includes: A modality adaptive transformation network was built based on CycleGAN to realize the mapping of visible light images to pseudo infrared images; A modal adaptive statistical attention module is introduced into the generator of the modal adaptive transformation network to enhance the ability to express key features and suppress interference from irrelevant information, thereby obtaining a pseudo-infrared image.

[0007] Preferably, the method for registering the infrared image and the pseudo-infrared image based on a co-visibility sensing multi-scale registration network to obtain the registered infrared image includes: The common-view perception feature matching module based on the common-view perception multi-scale registration network explicitly models the matching reliability of different spatial locations in the infrared image and the pseudo-infrared image. By combining a multi-scale deformation field estimation strategy from coarse to fine, high-precision spatial registration is achieved, and the registered infrared image is obtained.

[0008] Preferably, the method for explicitly modeling the matching reliability of different spatial locations in the infrared image and the pseudo-infrared image using a common-view perception feature matching module based on a common-view perception multi-scale registration network includes: By using a lightweight convolutional mapping function, response modeling is performed on the features of the infrared image and the pseudo-infrared image at different scales to obtain the corresponding matchability weights: ; in, and Infrared images and pseudo-infrared images In the Features at various scales; Represents the convolution mapping function; This represents the Sigmoid activation function; and Infrared images With pseudo-infrared images In the Matchability weights at each scale; Constructing a shared-vision perceptual mask through element-by-element fusion: ; in, For co-visibility perception mask; Based on the co-visibility sensing mask, a residual feature modulation strategy is used to weight and enhance the features of the infrared image and the pseudo-infrared image at different scales to obtain the corresponding enhanced features: ; in, and Infrared images With pseudo-infrared images In the Enhanced features at various scales.

[0009] Preferably, methods for achieving high-precision spatial registration and obtaining registered infrared images by combining a multi-scale deformation field estimation strategy from coarse to fine include: At low resolution, global matching is performed using the corresponding enhanced features to obtain the coarse deformation field: ; in, For coarse deformation field; This is a global module; and Infrared images and pseudo-infrared images Enhanced features at low-resolution scales; Local feature modeling of coarse deformation field: ; in, For fine deformation field; To refine the modules; The fine deformation field obtained at a low resolution scale Upsampling is performed, and the data is mapped to a high-resolution scale to obtain the deformation field at that high-resolution scale. Acting on high-resolution infrared features This achieves initial alignment at the feature level: ; ; in, This is an upsampling operation; Preliminary alignment features; During the image resampling stage, the final deformation field is applied to the infrared image to be registered, and spatial mapping is achieved through bilinear interpolation to obtain the final registration result: ; in, The registered infrared image; For image resampling operations; This is the final deformation field.

[0010] Preferably, the method for fusing the registered infrared image and the visible light image based on the cross-modal interactive fusion module to obtain the fused image includes: The registered infrared image and the visible light image are respectively input into the dual-branch feature extraction module to obtain the corresponding modal features; A bidirectional cross-attention mechanism is introduced to construct a cross-modal interaction fusion module, which jointly models the corresponding modal features of the two paths to obtain the corresponding interaction features; A residual feature update strategy is adopted to fuse the corresponding interaction features and modal features of the two paths to obtain the corresponding enhanced features; The corresponding enhancement features of the two channels are concatenated along the channel dimension and input into the channel embedding module for deep fusion to obtain fused features, thereby obtaining a fused image.

[0011] The present invention also provides an infrared and visible light image fusion system based on modality adaptive conversion and co-visibility perception multi-scale registration. The system is used to implement the aforementioned method and includes: a mapping module, a registration module, and a fusion module. The mapping module is used to map a visible light image to the pseudo-infrared domain based on a modality adaptive transformation network to obtain a pseudo-infrared image. The registration module is used to register the infrared image and the pseudo-infrared image based on a common-view perception multi-scale registration network to obtain a registered infrared image. The fusion module is used to fuse the registered infrared image and the visible light image based on the cross-modal interactive fusion module to obtain a fused image.

[0012] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the aforementioned method.

[0013] The present invention also provides a computer-readable storage medium storing a computer program that, when executed, implements the aforementioned method.

[0014] Compared with the prior art, the beneficial effects of the present invention are as follows: This invention addresses the significant differences in feature distribution and spatial misalignment between infrared and visible light images under cross-modal conditions. It proposes a method and system for infrared and visible light image fusion based on modal adaptive transformation and co-vision perception multi-scale registration, synergistically optimizing three aspects: modal alignment, spatial registration, and feature fusion. First, an Adaptive Modal Transformation Network (AMTN) is constructed to reduce the differences in cross-modal feature distribution, transforming the cross-modal registration problem into an approximate single-modal registration task. Second, a Co-vision Perception Multi-Scale Registration Network (CAMR) is designed, introducing a Co-vision Perception Feature Matching Module (CAFMM) to model matching reliability and combining a coarse-to-fine deformation field estimation strategy to achieve high-precision spatial alignment. Finally, a dual-path feature encoding and cross-modal interactive fusion module (CMIF) is constructed in the fusion stage. Through cross-modal feature interaction and adaptive fusion, it effectively integrates infrared salient target information and visible light texture details. Experimental results show that the proposed network achieves significant improvements in both registration accuracy and fusion quality, validating the effectiveness of each module design and the superiority of the overall framework. Attached Figure Description

[0015] To more clearly illustrate the technical solution of the present invention, the drawings used in the embodiments are briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0016] Figure 1 This is a schematic diagram of the overall architecture of the infrared and visible light image fusion network according to an embodiment of the present invention; Figure 2 This is a schematic diagram of the CycleGAN network structure according to an embodiment of the present invention; Figure 3 This is a schematic diagram of the generator structure according to an embodiment of the present invention; Figure 4 This is a schematic diagram of the modality adaptive statistical attention module structure according to an embodiment of the present invention; Figure 5 This is a schematic diagram of the architecture of the common vision perception feature matching module in an embodiment of the present invention; Figure 6 This is a schematic diagram of the cross-modal interaction fusion module structure according to an embodiment of the present invention; Figure 7 This is a schematic diagram comparing infrared images and distorted infrared images according to an embodiment of the present invention; Figure 8 This is a schematic diagram showing the qualitative comparison results of the RoadScene and TNO fusion methods in an embodiment of the present invention. Detailed Implementation

[0017] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0018] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0019] Example 1 This invention provides a method for fusion of infrared and visible light images based on modality adaptive conversion and co-visibility perception multi-scale registration, the method comprising: Based on a modality adaptive transformation network, a visible light image is mapped to the pseudo-infrared domain to obtain a pseudo-infrared image; Based on a common-view perception multi-scale registration network, the infrared image and the pseudo-infrared image are registered to obtain the registered infrared image. Based on the cross-modal interactive fusion module, the registered infrared image and the visible light image are fused to obtain a fused image.

[0020] like Figures 1-6 As shown, the specific implementation process of the present invention is as follows: This invention proposes an infrared and visible light image fusion network based on modality adaptive transformation and co-visibility perception multi-scale registration. The overall architecture is as follows: Figure 1As shown. The Adaptive Modality Transformation Network (AMTN) maps visible light images to the pseudo-infrared domain, reducing cross-modal differences from a feature distribution perspective and providing a more consistent feature representation for subsequent registration. The Covisibility-Aware Multi-scale Registration (CAMR) network takes infrared and pseudo-infrared images as input, performing hierarchical modeling in a multi-scale feature space. It also utilizes a Covisibility-Aware Feature Matching Module (CAFMM) to model and weight the reliability of fit at different spatial locations, prompting the network to pay attention to high-confidence regions. Based on this, a coarse-to-fine deformation field estimation strategy is used to infer the displacement field between the two images, achieving accurate alignment in near-single-modal cases. In the fusion stage, a framework is formed that integrates dual-path feature encoding and cross-modal interactive fusion. Each branch performs feature acquisition on the registered infrared and visible light images, comprehensively preserving information unique to each modality. Finally, a Cross-Modal Interaction Fusion module is used. The CMIF (Commonly Module for Infrared Targets) model shapes bidirectional feature interactions within a unified feature space. By explicitly modeling cross-modal dependencies, it adaptively selects and fuses complementary information from different modalities, thereby achieving effective integration of infrared salient target information and visible light texture details. Specifically, this includes: 1. Adaptive Modality Transformation Network (AMTN) 1.1 CycleGAN Network Architecture In cross-modal image transfer tasks, achieving stable and effective mapping relationships in the absence of paired samples is a key issue affecting model performance. Traditional style transfer methods based on generative adversarial networks generally employ a supervised learning paradigm, completing image transfer by constructing a one-way mapping from the source domain to the target domain. However, this method requires a large amount of carefully paired training data. In infrared and visible light image scenarios, due to the significant differences in sensor imaging mechanisms and limited acquisition conditions, it is difficult to obtain high-quality, strictly aligned cross-modal image pairs, which greatly limits its practical application. One-way mapping models often fail to effectively constrain the structure of the input content, easily leading to structural distortion or semantic shifts during cross-modal transfer, making it difficult to meet the structural consistency requirements of subsequent accurate registration and fusion tasks.

[0021] To address the aforementioned issues, CycleGAN introduces cycle consistency constraints and a bidirectional adversarial learning mechanism to achieve effective modeling of cross-domain mapping relationships under unsupervised conditions. For example... Figure 2 As shown, the model contains two generators and two discriminators: generators Responsible for the source domain Image mapping to target domain Discriminator Used to distinguish generated images from real images; at the same time, the generator Source domain Image mapping to target domain Discriminator This is used to distinguish between generated and real images. Through joint optimization of adversarial learning and bidirectional mapping mechanisms, CycleGAN can learn cross-modal mapping relationships without pairwise supervision, effectively mitigating the content distortion problem.

[0022] In terms of structural design, generators often adopt a three-stage architecture of encoding-conversion-decoding, just like... Figure 3 As shown, firstly, in the encoding stage, features of the input image are obtained through multi-layer convolution operations and progressive downsampling, and mapped to a low-dimensional latent feature space, thus obtaining a feature representation with strong semantic expressive power; secondly, in the transformation stage, a large number of residual blocks (ResNet Blocks) are used to perform deep nonlinear mapping on the latent features, relying on the residual combination mechanism to preserve the original structural information and achieve cross-domain feature transformation, enabling the network to effectively model complex textures and high-level semantic information; thirdly, in the decoding stage, the spatial resolution of the features is gradually restored by upsampling or deconvolution operations, thereby reconstructing the image of the target domain. This structure not only retains the ability to transform across modes, but also effectively enhances the ability to preserve image structural information. Discriminators often employ the PatchGAN structure, which discriminates images by starting with local receptive fields. It first divides the input image into many overlapping small regions, and then performs a true / false judgment for each small region. In this way, it pays attention to the high-frequency details and texture features of the image. With this local discrimination method, it can effectively improve the realism of the generated image in terms of edge structure and detail representation, reduce the difficulty of modeling the global structure, and help stabilize the adversarial training process. Compared with the discriminators that rely on global discrimination, PatchGAN has advantages in terms of the number of parameters and computational complexity. It can maintain the unity of the overall framework and optimize the visual effect of local regions, thereby further improving the level of detail reproduction and sensory quality of the generated results.

[0023] In this invention, CycleGAN is introduced as the basic framework of the Adaptive Modality Transformation Network (AMTN). By converting visible light images into pseudo-infrared representations, it reduces the distribution differences between the two modalities, providing a consistent and stable feature representation for subsequent registration and fusion. However, the standard CycleGAN still has shortcomings in preserving details in key regions and expressing cross-modal saliency, which can easily lead to the weakening of important structural information or interference from redundant information. To address this, this invention introduces a Modality-Adaptive Statistical Attention (MASA) module to jointly model and adaptively modulate features in both the channel and spatial dimensions. In the channel dimension, by learning the importance weights of different feature channels, it strengthens the discriminative semantic response; in the spatial dimension, by fusing multiple statistical features, it dynamically focuses on key regions, effectively suppressing the interference of background noise and irrelevant information. Through these improvements, AMTN significantly enhances the detail representation and regional saliency of the modality transformation results while maintaining overall structural consistency, thus providing a more reliable input basis for subsequent multi-scale registration and fusion.

[0024] 1.2 Modality Adaptive Statistical Attention Module In the registration process between infrared and visible light images, cross-modal feature matching is a core issue affecting registration accuracy. The registration effect essentially depends on the reliability of the feature correspondence between the two images. However, due to the fundamental differences in the imaging mechanisms of infrared and visible light sensors, they exhibit significant inconsistencies in grayscale distribution, texture structure, and statistical characteristics. Furthermore, during actual acquisition, factors such as motion blur and noise interference can further exacerbate the instability of feature representation. These factors collectively make it difficult to establish a stable and accurate correspondence between cross-modal features, thus limiting the improvement of registration accuracy.

[0025] To address the aforementioned issues, this invention introduces adaptive modality transformation into the image registration process as a crucial preprocessing step. Through cross-modal transformation, visible light images are converted to a feature space approximating the infrared distribution, thereby reducing matching difficulty and transforming the cross-modal problem into approximately single-modal matching, thus improving the stability and accuracy of feature matching. To this end, an Adaptive Modality Transformation Network (AMTN) is constructed, based on the CycleGAN framework, to achieve the mapping from visible light images to pseudo-infrared images. While maintaining the structural and semantic consistency of the original images, effective alignment and distribution transfer of cross-modal features are achieved.

[0026] The original CycleGAN primarily relies on convolutional operations for feature extraction. However, its ability to model key structural information is limited during cross-modal transformation, and the generated results are easily affected by visible light background information (such as incorrect mapping of shadows or reflective areas). To further improve the quality of generated pseudo-infrared images, this invention introduces a Modality-Adaptive Statistical Attention (MASA) module into the generator to enhance the representation of key features and suppress interference from irrelevant information.

[0027] like Figure 4 As shown, the modality adaptive statistical attention module first models the input features, and then adaptively weights the features in both the channel and spatial dimensions to enhance key features and suppress redundant information. Specifically, for the input features... First, attention weights are constructed along the channel dimension. The calculation process of the channel attention module is shown in equations (1) and (2). Input features Channel description vectors are obtained by performing global average pooling and global max pooling operations respectively. Then, the two statistical features are input into a shared mapping function for feature transformation, and fused by element-wise addition. Finally, the channel attention weights are obtained by normalization using the Sigmoid function. Applying this weight to the original feature yields channel-enhanced features. .

[0028] (1) (2) In the formula, and These represent global average pooling and max pooling operations, respectively. For channel mapping functions, This represents the Sigmoid activation function. This represents element-wise multiplication. Through this process, channels in the feature that are beneficial for cross-modal transitions are enhanced, while irrelevant channels are suppressed.

[0029] Building upon channel attention, channel-enhanced features are added. The input is further processed by the spatial attention module. The calculation process of the spatial attention module is shown in equations (3) and (4). First, the mean, maximum, minimum and cumulative response (summation) of the features are calculated in the channel dimension to characterize the spatial distribution information of the features under different statistical modes. Then, the above statistical features are concatenated in the channel dimension and spatial attention weights are generated through convolution mapping. Finally, the signal is normalized using the Sigmoid function and applied to the channel enhancement features. Spatial augmentation features are obtained. .

[0030] (3) (4) In the formula, , , and These represent the mean, maximum, minimum, and summation operations along the channel dimension, respectively. Indicates channel splicing. This is the convolution mapping function.

[0031] This invention introduces minimum and cumulative responses to supplement the inadequacies of mean and maximum values ​​in describing the feature space distribution. The minimum value characterizes low-response regions, while the cumulative response characterizes the overall response intensity of each spatial location across all channels. The combined use of these four statistical information sets describes the feature distribution from multiple perspectives, including average, strong, weak, and overall responses, providing more complete statistical information for spatial attention weight generation. By fusing various statistical measures, the spatial attention module can more comprehensively describe the feature distribution, thereby improving the response capability of key regions.

[0032] Finally, the results of channel attention and spatial attention are fused to obtain the final output features. The calculation process is shown in formula (5): (5) Through the aforementioned mechanism, MASA can enhance discriminative feature responses in the channel dimension while focusing on key regions in the spatial dimension, thereby effectively suppressing background noise and redundant information, improving the ability to preserve structural information and the saliency of the target region during cross-modal conversion. Furthermore, since infrared images are primarily composed of low-frequency information, complex nonlinear transformations can easily introduce redundant representations and increase training difficulty. This invention simplifies the generator structure by reducing the number of residual blocks to lower model complexity and improve training stability. Compared to the original CycleGAN, this invention, through the introduction of the MASA module and structural optimization, effectively improves the conversion quality from visible light images to pseudo-infrared images. The generated results, while retaining the original structural information, can more accurately represent the infrared feature distribution, making the pseudo-infrared image closer to the real infrared image, thus significantly reducing cross-modal differences and providing more reliable input data for subsequent image registration.

[0033] 1.3 Perceptual Style Transfer Loss In cross-modal image conversion, relying solely on pixel-level or adversarial losses is insufficient to adequately constrain the high-level semantic consistency and texture representation capabilities of the generated results. Therefore, this invention introduces Perceptual Style Transfer Loss (PST) to constrain the generated image from the feature space, thereby improving the structure preservation and style consistency of pseudo-infrared images. This loss consists of two parts: perceptual loss and style loss.

[0034] First, the difference in high-level feature representations between the input image and the recurrently reconstructed image is evaluated using perceptual loss. Specifically, feature representations of different layers are extracted using a pre-trained VGG-19 network, and constraints are imposed on the recurrent mapping process, as defined in equation (6): (6) in, This represents the input image in modal domain A, i.e., the visible light image; This represents the input image in modal domain B, i.e., the infrared image; This indicates the VGG-19 network's... The feature mapping of the layer, through this constraint, can effectively ensure that the structural information and semantic content of the image do not shift significantly during cross-modal conversion.

[0035] Based on this, a style loss is introduced to further constrain the texture distribution of the generated image. This loss characterizes the correlation between features by calculating the Gram matrix of the feature maps, thereby describing the overall style information of the image, and its definition is shown in equation (7): (7) in, This indicates a Gram matrix calculation operation. These are the weight coefficients for different feature layers. This loss function can enhance the consistency of the generated image in terms of texture structure while effectively mitigating artifact problems.

[0036] Combining the above two parts, the perceptual style transfer loss is defined as Equation (8): (8) in, and These are the weighting coefficients for perceptual loss and style loss, respectively. In this invention, we set... It is 1.0. The value is 100. By introducing PST loss, the model can constrain the cross-modal mapping process at the feature space level, which not only improves the realism and naturalness of the generated pseudo-infrared images, but also enhances structural consistency and texture expression capabilities, providing a more stable and reliable input for subsequent image registration.

[0037] 2. Covisibility-Aware Multi-scale Registration (CAMR) Network In the previous stage, visible light images were processed using a modality adaptive transformation network. Convert to pseudo-infrared image By aligning the two modalities at the feature distribution level, this method significantly alleviates the differences between infrared and visible light, transforming the registration problem from a cross-modal issue into an approximate same-modal registration between infrared and pseudo-infrared, thereby reducing matching uncertainty. However, it should be further noted that although the conversion process achieves modal alignment across the overall distribution, due to the limitations of the generative model's expressive power in local regions, pseudo-infrared images inevitably suffer from issues such as missing structural details, insufficient texture representation, and background pseudo-responses. These problems often exhibit significant spatial non-uniformity, leading to substantial differences in the reliability of feature matching across different regions.

[0038] 2.1 Co-visualization feature matching module Traditional registration methods typically assume that all spatial locations have equal importance during the matching process, meaning that a globally consistent feature interaction and matching model is applied to the entire image. However, in actual registration of infrared and pseudo-infrared images, this "globally equal weight" matching strategy often introduces a large amount of unreliable matching information. For example, in low-texture regions or uniform background regions, ambiguous matches are easily generated due to weak feature discriminative power; in regions with large pseudo-infrared generation errors, their structural information deviates from that of the real infrared image, which also leads to the accumulation of incorrect matches; furthermore, in the presence of local occlusion or blurred boundaries, it is difficult to accurately establish the correspondence between different images. These factors combined make the deformation field estimation process susceptible to noise interference, thus affecting the final registration accuracy.

[0039] To address the aforementioned problems, this invention proposes a Covisibility-Aware Feature Matching Module (CAFMM), the overall architecture of which is as follows: Figure 5As shown in the diagram. The core idea of ​​this module is to introduce a co-visibility perception mechanism during multi-scale feature matching, explicitly modeling the matching reliability at different spatial locations. This allows for dynamic filtering of effective regions and suppression of interference from invalid regions during the feature interaction stage. Specifically, by constructing a co-visibility perception mask, regions with high matching confidence in infrared and pseudo-infrared images are enhanced, guiding the network to focus more on establishing true correspondences while reducing interference from potentially non-corresponding regions. Furthermore, by combining a coarse-to-fine multi-scale deformation field estimation strategy, the model can achieve a balance between global structural alignment and local detail optimization, thereby achieving high-precision spatial registration.

[0040] In the feature modeling stage, the infrared images are first analyzed separately. With pseudo-infrared images Multi-scale feature extraction is performed to obtain feature representations at different scales. Let the first... The features at each scale are respectively and To characterize the matching reliability at different spatial locations, this invention uses a lightweight convolutional mapping function to model the response of features, obtaining the corresponding matchability weights, defined as shown in equation (9): (9) in, Represents the convolution mapping function, This represents the Sigmoid activation function. This weight characterizes the stability and discriminative power of spatial locations at the feature level, and its magnitude reflects the reliability of the region's participation in matching. Through this process, the model can adaptively learn which regions are more suitable for establishing correspondences without additional supervision.

[0041] After obtaining the matching responses of the two images, a co-visibility perceptual mask is constructed by element-wise fusion, as defined in equation (10): (10) This mask, based on bi-branch consistency, only significantly enhances its value when a spatial location exhibits a high response in both the infrared and pseudo-infrared images, effectively filtering out potential reliable matching regions. Compared to unilateral feature response modeling, this strategy effectively avoids matching bias caused by unilateral misjudgment, allowing the model to focus more on regions with genuine correspondences. Furthermore, this mechanism can be viewed to some extent as an approximate model of "co-observable regions," emphasizing regions with stable observations in both images, thereby improving the consistency and reliability of the matching.

[0042] To further enhance the feature representation capability of reliable regions while avoiding information loss that may result from direct screening, this invention employs a residual feature modulation strategy to weight and enhance the original features, as defined in equation (11): (11) in, and Infrared images and pseudo-infrared images In the Enhanced features at various scales.

[0043] In this approach, the features of high-response regions are further enhanced during the matching process, while low-response regions retain their basic information, thus suppressing noise interference while avoiding the loss of effective information. This soft-weighting strategy has better stability than traditional hard-screening methods, helping to improve the convergence performance of the network training process.

[0044] After feature enhancement, deformation field estimation is performed sequentially at different scales, from coarse to fine. At the low-resolution scale, global matching is performed using the enhanced features to obtain the coarse deformation field, which is defined as shown in equation (12): (12) in, For coarse deformation field; This is a global module; and Infrared images and pseudo-infrared images Enhanced features at low resolution scales.

[0045] This stage focuses on the consistency of the overall image structure, modeling the global deformation relationships in the image through a larger receptive field. Since low-resolution features have stronger semantic aggregation capabilities in space, they can effectively ignore local detail interference, thus more stably capturing large-scale displacements and overall structural differences. Therefore, the coarse deformation field estimated at this scale can provide reliable initial alignment results for the subsequent high-resolution stage, while significantly reducing the search space for the subsequent matching process. The fine deformation field is defined as shown in Equation (13): (13) Among them, the refinement module By analyzing the coarse deformation field Local feature modeling is performed, and its residual correction term is learned. This residual form can compensate for local errors while preserving the overall trend of the original deformation field, thereby improving the accuracy of the estimation. Compared with direct re-regression of the deformation field, the residual learning method has better convergence stability and can effectively avoid the instability caused by large offsets.

[0046] Next, the fine deformation field obtained at the low resolution scale will be... Upsampling is performed, and the data is mapped to a high-resolution scale to obtain the deformation field at that high-resolution scale. Acting on infrared images High-resolution infrared features extracted The initial alignment at the feature level is achieved as shown in equations (14) and (15): (14) (15) in, This is an upsampling operation; Preliminary alignment features; This represents a feature space transformation operation based on the deformation field.

[0047] This operation eliminates most of the offset caused by global deformation at a high-resolution scale, enabling preliminary spatial alignment between the two images. At this point, the focus of subsequent matching shifts from global structure to local details, significantly reducing matching difficulty and improving matching accuracy.

[0048] At high resolution, the enhanced features are further matched and optimized using a co-visibility perception mechanism to regress more refined local deformation relationships. Since the input features have already undergone initial alignment under the guidance of the low-scale deformation field at this stage, the network can focus more on modeling edge contours, texture details, and local structural differences, thereby gradually correcting residual registration errors and achieving high-precision alignment.

[0049] During the image resampling stage, the final deformation field Applied to the infrared image to be registered, spatial mapping is achieved through bilinear interpolation to obtain the final registration result, which is defined as shown in equation (16): (16) in, The registered infrared image; For image resampling operations; This is the final deformation field.

[0050] This process redistributes pixel positions from the original image to the target coordinate system through continuous coordinate mapping, achieving sub-pixel-level precise alignment while maintaining image structural continuity. Compared to discrete pixel-level transformation methods, bilinear interpolation effectively reduces aliasing and information loss during resampling, thereby improving the visual quality and structural consistency of the registration results. Finally, spatial consistency modeling between the infrared image and the pseudo-infrared image is completed, providing a reliable foundation for subsequent image fusion and target analysis.

[0051] 2.2 Registration Loss To further improve the accuracy and stability of image registration, this invention unifies the modeling of deformation field constraints, edge constraints, and bidirectional structural similarity constraints. This allows the model to optimize the registration results from multiple perspectives during training, thereby effectively suppressing local noise interference while ensuring overall spatial consistency, enhancing the matching ability of key structural regions, and ultimately improving the robustness and accuracy of image registration. (Registration loss) The expression is defined as shown in equation (17): (17) in, , and These are weighting coefficients used to balance the contributions of each loss term in the overall optimization process. By setting the weights appropriately, the model can focus more on the smooth estimation of the deformation field in the early stages of training, and gradually strengthen the constraints on edge details and structural consistency in the later stages, thereby obtaining a more stable registration effect.

[0052] Deformation field loss This is used to constrain the continuity and smoothness of the deformation field in space. In image registration tasks, without regularization constraints on the deformation field, the model is prone to excessive displacement changes in local regions, leading to discontinuities, unevenness, or even local distortions in the deformation field. This phenomenon not only reduces the visual naturalness of the registration results but also affects the reliability of subsequent image fusion and target analysis tasks. To address this, this invention constrains the local changes of the deformation field by calculating its gradients in the horizontal and vertical directions, as defined in equation (18): (18) in, This represents the final estimated deformation field. and These represent the partial derivatives of the deformation field in the horizontal and vertical directions, respectively. By minimizing this loss term, drastic fluctuations in displacement between adjacent pixels can be effectively suppressed, making the estimated deformation field more spatially smooth and continuous, thereby improving the stability of the registration process.

[0053] To further enhance the alignment accuracy of structural boundary regions in images, this invention introduces edge loss. Constraints are imposed on the registration results. Since edge regions typically correspond to target contours, locations of structural abrupt changes, and significant geometric boundary information, these regions are of high importance in image registration. If only overall intensity or structural similarity is considered while edge information is ignored, subtle but crucial misalignments can easily occur in local contour regions. To address this, this invention models the consistency of edge information by comparing the differences between the registered infrared image and the pseudo-infrared image in gradient space. The mathematical expression of this model is shown in formula (19): (19) in, Represents the image gradient operator. This represents the registered infrared image obtained by deformation field resampling. This represents a pseudo-infrared image. This loss term effectively emphasizes high-frequency structural information, enabling the model to focus more on the accurate alignment of edges and contour regions during optimization, thereby improving the registration quality of local details.

[0054] To ensure the overall consistency of the registration results, this invention introduces structural similarity loss. Considering that unidirectional constraints can easily lead to a shift in the registration result in one direction, thus introducing asymmetric errors, this invention adopts a bidirectional consistency strategy, simultaneously constraining both the forward and reverse registration processes to enhance the symmetry and stability of the registration results. Specifically, on the one hand, it constrains the registered infrared image... With pseudo-infrared images The difference between them; on the other hand, utilizing the reverse deformation field Resample the pseudo-infrared image to make it approximate the original infrared image. This further reduces the potential bias caused by one-way registration. Its definition is shown in formula (20): (20) in, and The weight parameters are for the bidirectional consistency constraint. This loss, by simultaneously constraining the forward and backward mapping relationships, ensures that the model, during optimization, not only focuses on the closeness between the registration result and the target image, but also takes into account the symmetry and consistency of the transformation process, thereby improving the overall accuracy of structural alignment.

[0055] 3. Cross-modal interaction fusion 3.1 Cross-modal interaction fusion module After achieving accurate registration between infrared and pseudo-infrared images, effectively fusing information from the two modalities becomes a key issue for further improving overall performance. Although the two images are structurally largely identical after spatial alignment, significant differences remain in their feature representation: the infrared image primarily reflects the target's thermal radiation information and has strong target saliency, while the visible light image possesses rich texture and structural details. Therefore, the core challenge in image fusion is how to fully utilize detailed information while maintaining target saliency, achieving collaborative representation of cross-modal information.

[0056] To address the aforementioned issues, this invention proposes a Cross-Modal Interaction Fusion Module (CMIF). The core idea of ​​this method is to construct a bidirectional feature interaction mechanism between the registered infrared and visible light images. Through explicit cross-modal association modeling, it adaptively selects effective information from different modalities and achieves collaborative fusion in a unified feature space. Specifically, this invention first employs a dual-branch feature extraction structure to encode the registered infrared and visible light images separately. Then, it utilizes the cross-modal interaction mechanism to establish the correspondence between infrared and visible light features in different feature subspaces, thereby enhancing the complementary expression of target saliency information and texture detail information. Finally, through feature embedding and fusion mapping, the final fused features are obtained, providing a more compact and discriminative feature foundation for subsequent image reconstruction.

[0057] In the feature extraction stage, the registered infrared images are first... With visible light images The corresponding modal features are obtained by inputting them into the dual-branch feature extraction module, and their definitions are shown in formula (21): (twenty one) in This indicates the feature extraction module. Its parameters are set. Through the dual-branch structure, the characteristic information of the two modes can be retained respectively, so that the target response information in the infrared features and the texture detail information in the visible light features can be fully encoded.

[0058] After obtaining the dual-modal features, to further enhance the effective interaction between infrared and visible light features, this invention constructs a cross-modal interaction fusion module to jointly model the two modal features, such as... Figure 6As shown. Considering the differences in contribution levels of different modalities at different spatial locations, this invention introduces a bidirectional cross-attention mechanism to explicitly model the correlation between the two features, thereby achieving full interaction and fusion of cross-modal information. Specifically, infrared features and visible light features are used as queries, and another modal feature is used as the key and value, respectively, to establish cross-modal information flow from two directions. The calculation process is shown in formulas (22) and (23): (twenty two) (twenty three) in, , and These represent queries, key-value mappings, and key-value mappings, respectively. This represents the feature dimension. Through this bidirectional interaction mechanism, the model can establish a dependency relationship between infrared and visible light features globally, enabling each modal feature to fully integrate supplementary information from the other modality during the update process, thereby improving the expressive and discriminative power of the fused features.

[0059] After completing the cross-modal attention interaction, in order to avoid the information disturbance problem caused by directly replacing the original features, this invention further adopts a residual feature update strategy to fuse the interaction features with the original features to obtain the enhanced features. The calculation process is shown in formula (24): (twenty four) in, This represents a linear mapping and normalization operation. This residual update form can introduce supplementary representations from another modality while maintaining the original modality's discriminativeness, thereby effectively mitigating the feature shift problem caused by strong interaction modeling and improving the stability of the training process.

[0060] Building upon this, to further achieve cross-channel information recombination and fusion compression, the two enhanced features are spliced ​​together along the channel dimension and input into the channel embedding module for deep fusion to obtain fused features. Its expression is shown in formula (25): (25) in, This indicates a channel splicing operation. Through this fusion mapping process, saliency information from infrared features and detail information from visible light features can be uniformly embedded into the same feature space, thereby achieving deep integration of cross-modal information. This is achieved through the fusion features... Feature decoding and reconstruction are performed to obtain the final fused image. .

[0061] 3.2 Fusion Loss To further improve the visual quality and information representation capability of fused images, this invention provides a unified modeling approach for the fusion results, focusing on three aspects: structural consistency, texture detail preservation, and salient target enhancement. This allows the model to fully utilize the complementary information between infrared and visible light images during training, ensuring reasonable overall brightness distribution and structural representation while effectively enhancing the representation of edge details and key regions, thereby improving the clarity and target discernibility of the fused image. Fusion Loss As shown in formula (26): (26) in, , and is a weighting coefficient used to balance the contribution of each loss term in the overall optimization process.

[0062] Multiscale structural similarity loss Used to constrain fused images Consistency in overall structure and intensity distribution. In the process of image fusion, if there is a lack of effective modeling of structural information, it is easy to cause problems such as brightness imbalance or structural distortion in the fusion result, which will affect the visual quality. To this end, the present invention constrains the structural similarity between the fused image and the registered infrared image and the visible light image at the same time, so that the fusion result maintains a stable structural expression at different scales, which is defined as shown in formula (27): (27) in, The Structural Similarity Index Measure function measures the similarity between two images in terms of brightness, contrast, and structural information. The higher the value, the greater the structural similarity between the two images.

[0063] By minimizing this loss term, the fused image can simultaneously approximate the two source images in terms of brightness, contrast, and structural information, thereby improving overall visual consistency.

[0064] Joint gradient loss This is used to enhance the texture details and edge structure information in the fused image. During the fusion process, a lack of constraints on high-frequency information can easily lead to blurred details or weakened edges, especially in complex scenes where it can significantly reduce image clarity. Therefore, this invention constructs an upper bound constraint on the gradient, making the gradient response of the fused image approximate the maximum gradient information in the infrared and visible light images, as shown in formula (28): (28) in, This represents the image gradient operator. This loss effectively highlights high-frequency structural information, enabling the model to focus more on texture details and edge regions during optimization, thereby improving the clarity and detail representation of the fused image.

[0065] Significance Preservation Loss This invention constrains the ability of the fused image to represent key target regions. In infrared and visible light images, salient regions typically correspond to target objects or important structural information. If these regions are not effectively preserved during the fusion process, the target information may be weakened or even lost. Therefore, this invention constructs an adaptive weight allocation mechanism based on saliency response. By modeling the saliency of infrared and visible light images, corresponding weight coefficients are obtained. and And the weight is used to perform weighted fusion of the two modal information, as defined in formulas (29) and (30): (29) (30) in, and These represent the saliency response maps of the infrared and visible light images, respectively. Through this loss term, the fused image can better preserve infrared information in salient regions while fully utilizing the texture details of the visible light image in background regions, thus achieving an adaptive balance in information representation.

[0066] 4. Experiment and Analysis 4.1 Dataset To verify the effectiveness of the method proposed in this invention for cross-modal image registration and fusion tasks, experiments were conducted on two commonly used infrared and visible light image datasets: the TNO dataset and the RoadScene dataset. Both datasets are widely used in multimodal image fusion and registration research, providing good representativeness and a basis for comparison.

[0067] The TNO dataset, provided by the TNO organization, contains multiple pairs of infrared and visible light images, covering complex outdoor scenes such as pedestrians, vehicles, and various background environments. The images in this dataset have typically undergone precise manual registration, exhibiting high spatial consistency, making it suitable as a benchmark set for model performance evaluation. Due to its relatively small size, this invention only uses it in the testing phase to objectively evaluate the model's generalization ability in different scenarios. The RoadScene dataset mainly contains pairs of infrared and visible light images of road traffic scenes, covering various lighting conditions such as day and night, totaling 221 pairs. Compared to the TNO dataset, this dataset has richer scene variations and target distributions, and therefore was used in the model training phase. In specific experiments, this invention employs an unsupervised training strategy, using the entire RoadScene dataset for training and randomly allocating approximately 55% of the images for testing to further verify the model's performance on unseen samples.

[0068] The original images in both datasets mentioned above are already registered. However, the focus of this invention is to address the non-rigid registration problem of cross-modal images. Therefore, to construct a more challenging experimental setup, this invention introduces artificial deformation perturbations into the infrared images during the training phase. A deformation field is constructed by randomly generating affine and elastic deformations and applied to the infrared images to create spatially displaced data. The experimental task is to register visible light and distort infrared images. Figure 7 It demonstrates the comparison effect of real infrared images before and after transformation.

[0069] 4.2 Experimental Environment and Details (1) Experimental Environment All experiments in this invention use the same experimental environment, and the specific hardware and software environment configuration is shown in Table 1.

[0070] Table 1 2) Experimental details During model training, this invention employs a random sampling strategy to construct training batches from the training dataset. Specifically, in each iteration, eight 256×256 image patches are randomly cropped as input samples to form a training batch. Regarding the optimization strategy, this invention uses the Adam optimizer to update the network parameters, where the momentum parameter is set to... The initial learning rate of the model is set to... The parameters were kept constant throughout the training process. The model was trained for 300 epochs to ensure sufficient convergence of the network parameters. Furthermore, to ensure the reproducibility and fairness of the experimental results, all experiments were conducted under the same training configuration, including a unified data preprocessing procedure and parameter initialization method. During training, input images were normalized, and random cropping was used to enhance data diversity, thereby improving the model's adaptability to different scenes and deformation conditions.

[0071] 4.3 Image Registration and Comparison Experiment To systematically evaluate the performance of the proposed method in cross-modal image registration tasks, this invention conducted image registration comparison experiments and selected mean square error (MSE), normalized cross-correlation (NCC), and mutual information (MI) as quantitative evaluation indicators to comprehensively analyze the registration effects of different methods. The relevant experimental results are shown in Table 2.

[0072] The experimental results show that the proposed method achieves optimal performance across all three evaluation metrics. Specifically, the significant reduction in the MSE metric indicates that pixel-level differences in the registered images are effectively suppressed. Meanwhile, the improvements in the NCC and MI metrics further demonstrate the proposed method's stronger expressive power in structural consistency modeling and cross-modal information alignment. This indicates that the proposed method can accurately establish spatial correspondences under complex modal differences, thereby achieving higher-precision image registration. Further analysis shows that the modality adaptive transformation network effectively reduces the distribution difference between infrared and visible light by remapping the feature space of the visible light image, thereby reducing the uncertainty in the cross-modal matching process. On this basis, the co-vision perception multi-scale registration network suppresses low-confidence regions by explicitly modeling the matching reliability of spatial positions, and gradually optimizes the deformation field estimation in a multi-scale framework, so that the model achieves a good balance between global structure alignment and local detail correction.

[0073] Table 2 4.4 Image Fusion Comparison Experiment (1) Quantitative comparison To further verify the effectiveness of the proposed method in image fusion tasks, quantitative comparative experiments were conducted on the TNO and RoadScene datasets, and cross-correlation (CC), visual information fidelity (VIF), and structural similarity (SSIM) were selected as evaluation metrics. The relevant experimental results are shown in Tables 3 and 4.

[0074] Experimental results show that the method of this invention outperforms existing mainstream infrared and visible light image fusion methods in all three evaluation metrics, achieving the best performance in most metrics. Specifically, the improved CC and SSIM metrics indicate better performance in terms of structural consistency and overall contrast, while the significant improvement in the VIF metric demonstrates the proposed method's clear advantages in information preservation and detail representation. In particular, compared with better-performing fusion methods in recent years, the method of this invention achieves further improvement in the VIF metric, fully demonstrating that the proposed cross-modal collaborative fusion mechanism can more effectively integrate salient target information from infrared images with texture detail information from visible light images, thereby generating a fusion result with richer information and clearer structure.

[0075] Table 3 Table 4 (2) Qualitative comparison To further and more intuitively evaluate the fusion effect of different methods, this invention selects two typical scenarios from the TNO dataset and the RoadScene dataset for visual comparative analysis. The relevant results are as follows: Figure 8 As shown, by observing the fusion results and their magnified local areas, the performance differences of each method in different scenarios can be analyzed more clearly.

[0076] In the TNO dataset, due to the complex contrast relationship between the target and the background, some contrast methods are prone to insufficient target information representation or blurred edges during the fusion process, especially in low-texture areas, resulting in a certain degree of information loss. In the RoadScene dataset, due to significant illumination variations and rich texture information, some methods struggle to effectively coordinate infrared and visible light information, leading to uneven contrast or detail degradation in the fusion result. In contrast, the method of this invention demonstrates superior visual effects in both datasets. In the TNO scene, the fusion result more clearly highlights salient targets in the infrared image while maintaining the structural integrity of the background area; in the RoadScene scene, the method of this invention effectively preserves texture details in the visible light image, making the fused image clearer and more layered in edge areas. Furthermore, magnified local areas reveal that the method of this invention is more accurate in representing target contours and detailed structures, effectively avoiding artifacts and information aliasing.

[0077] 4.5 Ablation Test To systematically evaluate the role of each key module in cross-modal image fusion tasks, this invention conducts ablation experiments on the TNO and RoadScene datasets, analyzing the contributions of the Modal Adaptive Transform Network (AMTN), the Common Vision Aware Feature Matching Module (CAFMM), and the Cross-Modal Interactive Fusion Module (CMIF). The experimental results are shown in Table 5.

[0078] The experimental results show that, while the model possesses basic cross-modal feature integration capabilities when only CMIF is used for fusion (without introducing AMTN and CAMMF), its overall performance is still significantly limited, particularly in terms of structure preservation (SSIM) and information fidelity (VIF). This indicates that, without effective spatial alignment and modal distribution constraints, direct feature fusion struggles to fully exploit the complementary information between infrared and visible light images, and is easily affected by feature misalignment and distribution inconsistencies.

[0079] Building upon this foundation, the introduction of CAMMF for cross-modal registration significantly improved all evaluation metrics. For example, on the TNO dataset, CC, SSIM, and VIF improved from 0.441, 0.374, and 0.766 to 0.472, 0.457, and 0.885, respectively; a similar trend was observed on the RoadScene dataset. These results demonstrate that the co-visibility sensing mechanism can explicitly model the matching reliability of spatial locations, effectively suppressing the interference of low-confidence regions on deformation field estimation, improving the spatial consistency of cross-modal features, and providing more accurate alignment for subsequent fusion.

[0080] Furthermore, the model performance reaches its optimal state with the complete model configuration (AMTN + CAMMF + CMIF). Specifically, the VIF on the TNO dataset improves to 1.021, and on the RoadScene dataset to 0.896, demonstrating further enhancements in the information representation capability and structural consistency of the fusion results. This phenomenon indicates that AMTN, through modal adaptive mapping of visible light images, effectively reduces cross-modal feature distribution differences, thereby lowering matching uncertainty and further improving the collaborative optimization capability of the registration and fusion processes.

[0081] Table 5 In summary, this invention addresses the significant differences in feature distribution and spatial misalignment between infrared and visible light images under cross-modal conditions. It proposes an image fusion network based on modal adaptive transformation and co-vision perception registration, synergistically optimizing modal alignment, spatial registration, and feature fusion. First, an Adaptive Modal Transformation Network (AMTN) is constructed to reduce the differences in cross-modal feature distribution, transforming the cross-modal registration problem into an approximate single-modal registration task. Second, a Co-vision Perception Multi-Scale Registration Network (CAMR) is designed. This network models matching reliability by introducing a Co-vision Perception Feature Matching Module (CAFMM) and combines a coarse-to-fine deformation field estimation strategy to achieve high-precision spatial alignment. Finally, a Dual-Path Feature Encoding and Cross-Modal Interactive Fusion Module (CMIF) is constructed in the fusion stage. Through cross-modal feature interaction and adaptive fusion, saliency information and texture details are effectively integrated. Experimental results demonstrate that the proposed network achieves significant improvements in both registration accuracy and fusion quality, validating the effectiveness of each module design and the superiority of the overall framework.

[0082] Example 2 Based on the same inventive concept, the present invention also provides an infrared and visible light image fusion system based on modality adaptive conversion and co-visibility perception multi-scale registration. The system is used to implement the method described in the foregoing embodiments. The system includes: a mapping module, a registration module, and a fusion module. The mapping module is used to map a visible light image to the pseudo-infrared domain based on a modality adaptive transformation network to obtain a pseudo-infrared image. The registration module is used to register the infrared image and the pseudo-infrared image based on a common-view perception multi-scale registration network to obtain a registered infrared image. The fusion module is used to fuse the registered infrared image and the visible light image based on the cross-modal interactive fusion module to obtain a fused image.

[0083] Furthermore, in this embodiment, the mapping module maps the visible light image to the pseudo-infrared domain based on a modality adaptive transformation network. The process of obtaining the pseudo-infrared image includes: A modality adaptive transformation network was built based on CycleGAN to realize the mapping of visible light images to pseudo infrared images; A modal adaptive statistical attention module is introduced into the generator of the modal adaptive transformation network to enhance the ability to express key features and suppress interference from irrelevant information, thereby obtaining a pseudo-infrared image.

[0084] Furthermore, in this embodiment, the registration module registers the infrared image and the pseudo-infrared image based on a co-visibility sensing multi-scale registration network, and the process of obtaining the registered infrared image includes: The common-view perception feature matching module based on the common-view perception multi-scale registration network explicitly models the matching reliability of different spatial locations in the infrared image and the pseudo-infrared image. By combining a multi-scale deformation field estimation strategy from coarse to fine, high-precision spatial registration is achieved, and the registered infrared image is obtained.

[0085] Furthermore, in this embodiment, the process of explicitly modeling the matching reliability of different spatial locations in the infrared image and the pseudo-infrared image by the common-view perception feature matching module based on the common-view perception multi-scale registration network includes: By using a lightweight convolutional mapping function, response modeling is performed on the features of the infrared image and the pseudo-infrared image at different scales to obtain the corresponding matchability weights: ; in, and Infrared images With pseudo-infrared images In the Features at various scales; Represents the convolution mapping function; This represents the Sigmoid activation function; and Infrared images With pseudo-infrared images In the Matchability weights at each scale; Constructing a shared-vision perceptual mask through element-by-element fusion: ; in, For co-visibility perception mask; Based on the co-visibility sensing mask, a residual feature modulation strategy is used to weight and enhance the features of the infrared image and the pseudo-infrared image at different scales to obtain the corresponding enhanced features: ; in, and Infrared images and pseudo-infrared images In the Enhanced features at various scales.

[0086] Furthermore, in this embodiment, a high-precision spatial registration is achieved by combining a coarse-to-fine multi-scale deformation field estimation strategy. The process of obtaining the registered infrared image includes: At low resolution, global matching is performed using the corresponding enhanced features to obtain the coarse deformation field: ; in, For coarse deformation field; This is a global module; and Infrared images and pseudo-infrared images Enhanced features at low-resolution scales; Local feature modeling of coarse deformation field: ; in, For fine deformation field; To refine the modules; The fine deformation field obtained at a low resolution scale Upsampling is performed, and the data is mapped to a high-resolution scale to obtain the deformation field at that high-resolution scale. Acting on high-resolution infrared features This achieves initial alignment at the feature level: ; ; in, This is an upsampling operation; Preliminary alignment features; During the image resampling stage, the final deformation field is applied to the infrared image to be registered, and spatial mapping is achieved through bilinear interpolation to obtain the final registration result: ; in, The registered infrared image; For image resampling operations; This is the final deformation field.

[0087] Furthermore, in this embodiment, the fusion module, based on the cross-modal interactive fusion module, fuses the registered infrared image and the visible light image to obtain the fused image. The process includes: The registered infrared image and the visible light image are respectively input into the dual-branch feature extraction module to obtain the corresponding modal features; A bidirectional cross-attention mechanism is introduced to construct a cross-modal interaction fusion module, which jointly models the corresponding modal features of the two paths to obtain the corresponding interaction features; A residual feature update strategy is adopted to fuse the corresponding interaction features and modal features of the two paths to obtain the corresponding enhanced features; The corresponding enhancement features of the two channels are concatenated along the channel dimension and input into the channel embedding module for deep fusion to obtain fused features, thereby obtaining a fused image.

[0088] Example 3 Based on the same inventive concept, the present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the methods described in the foregoing embodiments.

[0089] Example 4 Based on the same inventive concept, the present invention also provides a computer-readable storage medium storing a computer program that, when executed, implements the methods described in the foregoing embodiments.

[0090] The embodiments described above are merely preferred embodiments of the present invention and are not intended to limit the scope of the present invention. Any modifications and improvements made to the technical solutions of the present invention by those skilled in the art without departing from the spirit of the present invention should fall within the protection scope defined by the claims of the present invention.

Claims

1. A method for fusion of infrared and visible light images based on modality adaptive conversion and co-visibility perception multi-scale registration, characterized in that, The method includes: Based on a modality adaptive transformation network, a visible light image is mapped to the pseudo-infrared domain to obtain a pseudo-infrared image; Based on a common-view perception multi-scale registration network, the infrared image and the pseudo-infrared image are registered to obtain the registered infrared image. Based on the cross-modal interactive fusion module, the registered infrared image and the visible light image are fused to obtain a fused image.

2. The method according to claim 1, characterized in that, Methods for obtaining pseudo-infrared images by mapping visible light images to the pseudo-infrared domain based on modality adaptive transformation networks include: A modality adaptive transformation network was built based on CycleGAN to realize the mapping of visible light images to pseudo infrared images; A modal adaptive statistical attention module is introduced into the generator of the modal adaptive transformation network to enhance the ability to express key features and suppress interference from irrelevant information, thereby obtaining a pseudo-infrared image.

3. The method according to claim 1, characterized in that, The method for registering an infrared image and the pseudo-infrared image based on a co-visibility sensing multi-scale registration network to obtain a registered infrared image includes: The common-view perception feature matching module based on the common-view perception multi-scale registration network explicitly models the matching reliability of different spatial locations in the infrared image and the pseudo-infrared image. By combining a multi-scale deformation field estimation strategy from coarse to fine, high-precision spatial registration is achieved, and the registered infrared image is obtained.

4. The method according to claim 3, characterized in that, The common-view perceptual feature matching module based on a common-view perceptual multi-scale registration network includes a method for explicitly modeling the matching reliability of different spatial locations in the infrared image and the pseudo-infrared image, comprising: By using a lightweight convolutional mapping function, response modeling is performed on the features of the infrared image and the pseudo-infrared image at different scales to obtain the corresponding matchability weights: ； in, and Infrared images With pseudo-infrared images In the Features at various scales; Represents the convolution mapping function; This represents the Sigmoid activation function; and Infrared images With pseudo-infrared images In the Matchability weights at each scale; Constructing a shared-vision perceptual mask through element-by-element fusion: ； in, For co-visibility perception mask; Based on the co-visibility sensing mask, a residual feature modulation strategy is used to weight and enhance the features of the infrared image and the pseudo-infrared image at different scales to obtain the corresponding enhanced features: ； in, and Infrared images and pseudo-infrared images In the Enhanced features at various scales.

5. The method according to claim 4, characterized in that, Methods for achieving high-precision spatial registration and obtaining registered infrared images by combining a coarse-to-fine multi-scale deformation field estimation strategy include: At low resolution, global matching is performed using the corresponding enhanced features to obtain the coarse deformation field: ； in, For coarse deformation field; This is a global module; and Infrared images and pseudo-infrared images Enhanced features at low-resolution scales; Local feature modeling of coarse deformation field: ； in, For fine deformation field; To refine the modules; The fine deformation field obtained at a low resolution scale Upsampling is performed, and the data is mapped to a high-resolution scale to obtain the deformation field at that high-resolution scale. Acting on high-resolution infrared features This achieves initial alignment at the feature level: ； ； in, This is an upsampling operation; Preliminary alignment features; During the image resampling stage, the final deformation field is applied to the infrared image to be registered, and spatial mapping is achieved through bilinear interpolation to obtain the final registration result: ； in, The registered infrared image; For image resampling operations; This is the final deformation field.

6. The method according to claim 1, characterized in that, The method for fusing the registered infrared image and the visible light image based on the cross-modal interactive fusion module to obtain the fused image includes: The registered infrared image and the visible light image are respectively input into the dual-branch feature extraction module to obtain the corresponding modal features; A bidirectional cross-attention mechanism is introduced to construct a cross-modal interaction fusion module, which jointly models the corresponding modal features of the two paths to obtain the corresponding interaction features; A residual feature update strategy is adopted to fuse the corresponding interaction features and modal features of the two paths to obtain the corresponding enhanced features; The corresponding enhancement features of the two channels are concatenated along the channel dimension and input into the channel embedding module for deep fusion to obtain fused features, thereby obtaining a fused image.

7. An infrared and visible light image fusion system based on modality adaptive conversion and co-visibility perception multi-scale registration, the system being used to implement the method described in any one of claims 1-6, characterized in that, The system includes: a mapping module, a registration module, and a fusion module; The mapping module is used to map a visible light image to the pseudo-infrared domain based on a modality adaptive transformation network to obtain a pseudo-infrared image. The registration module is used to register the infrared image and the pseudo-infrared image based on a common-view perception multi-scale registration network to obtain a registered infrared image. The fusion module is used to fuse the registered infrared image and the visible light image based on the cross-modal interactive fusion module to obtain a fused image.

8. An electronic device, characterized in that, It includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the program, implements the method according to any one of claims 1-6.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed, implements the method described in any one of claims 1-6.

Images