An infrared and visible light image fusion method and system based on bidirectional semantic-space alignment and differential perception

Abstract

The application provides an infrared and visible light image fusion method and system based on bidirectional semantic-space alignment and differential perception, and relates to the technical field of image processing. In order to solve the problem that the expression capacity is insufficient when the existing method processes complex cross-modal semantic interaction. The technical points of the application include: S1, collecting infrared images and visible light images; S2, fusing infrared and visible light images based on a semantic fusion network, wherein the semantic fusion network is based on a PSFusion network model, the SDFM module of the PSFusion network model is replaced by a differential perception complementary module DACM, the PSFM module is replaced by a deep semantic fusion block DSFB, and a space semantic alignment module SSAM is added after the deep semantic fusion block and the differential perception complementary module; S3, image reconstruction and supervision are performed on the aligned features. The method provided by the application shows superior potential compared with the current advanced image fusion algorithm.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and more specifically, to an infrared and visible light image fusion method and system based on bidirectional semantic-spatial alignment and differential sensing. Background Technology

[0002] Multimodal imaging sensors can describe scene information based on different physical principles. Visible light sensors generate images by recording the intensity of reflected light, providing clear texture and color details under sufficient lighting conditions. However, their image quality is severely limited in low-light, smoke, or obstructed environments. In contrast, infrared sensors image by detecting the thermal radiation of objects, are less affected by lighting conditions, and can effectively locate targets with thermal features. However, due to the physical characteristics of the sensor array, infrared images typically have low resolution and lack texture details. Given the complementary characteristics of these two sensors, infrared and visible light image fusion technology aims to generate a single image containing complete scene information to compensate for the shortcomings of single-sensor imaging.

[0003] Driven by the rapid development of computer vision technology, image fusion has become an important research direction in the field of multimodal visual perception. It has wide applications in remote sensing, medical imaging, autonomous driving, target detection, and surveillance. In these tasks, infrared-visible image fusion (IVIF) is particularly crucial in challenging environments such as low light, nighttime scenes, and adverse weather conditions. Infrared images capture thermal radiation, highlighting targets invisible in low light, while visible light images provide richer texture and spatial details. Fusion of these two modalities integrates thermal and structural information, thereby enhancing the performance of downstream tasks such as target detection, action recognition, and segmentation.

[0004] Existing fusion algorithms are generally divided into signal processing-based methods and deep learning-based methods. However, existing methods have engineering design limitations when handling cross-scale features. One issue is feature space misalignment. Most methods use simple upsampling to concatenate low-resolution deep semantic features with high-resolution shallow texture features. This operation ignores the spatial bias caused by scale changes, leading to artifacts or semantic blurring at object edges in the fused image. Another issue is the lack of physical perception. The aggregation of shallow features often relies on statistical operations such as maximum or average values, failing to fully consider the physical differences between infrared thermal radiation and visible light reflectivity, resulting in the inability to effectively preserve thermal target information under illumination interference. Furthermore, in deep feature interactions, existing methods often face a trade-off between model complexity and performance. While most self-attention modules can capture global contextual information, their large number of parameters and secondary computational complexity limit their deployment potential in resource-constrained systems. Meanwhile, simple MLP gating mechanisms are insufficient in their expressive power when handling complex cross-modal semantic interactions. Summary of the Invention

[0005] The technical problem to be solved by this invention is:

[0006] Existing methods have insufficient expressive power when dealing with complex cross-modal semantic interactions.

[0007] The technical solution adopted by the present invention to solve the above-mentioned technical problems is as follows:

[0008] This invention provides an infrared-visible image fusion method based on bidirectional semantic-spatial alignment and differential sensing, comprising the following steps:

[0009] S1. Acquire infrared and visible light images;

[0010] S2. Infrared and visible light images are fused based on a semantic fusion network. The semantic fusion network is based on the PSFusion network model. Its SDFM module is replaced with a differential sensing complementary module DAM, the PSFM module is replaced with a deep semantic fusion module DSFB, and a spatial semantic alignment module SSAM is added after the deep semantic fusion block and the differential sensing complementary module.

[0011] The DAM module is used to calibrate infrared and visible light features, and captures infrared and visible light calibration features separately through differential sensing pooling operations. , and The dependencies between dimensions ultimately output the aggregated features of the three branches;

[0012] The DSFB module is used to enhance the features extracted by the backbone network and convert the enhanced features into keys and values. Then, a neutral initial state is created and concatenated with the features of each modality. The shared context encoder uses convolutional layers with shared weights to extract context query values ​​containing interaction information, calculates cross-modal attention maps, and obtains interaction features with global context. The concatenated interaction features are then fed into a dual-gating mechanism, which preserves the original amplitude information and gradient flow of the features through a linear path and captures complex semantic dependencies through a non-linear path, finally obtaining fused features.

[0013] The SSAM module is used to upsample each deep fusion semantic feature to the same spatial resolution as the selected anchor point, and then project the features onto a unified alignment space. Each layer introduces a globally learnable scalar to concatenate and integrate all aligned features. The integrated features are concatenated and compressed with the projected features of the original anchor point, and residual connections are introduced to generate aligned semantic features.

[0014] S3. Perform image reconstruction and supervision on alignment features.

[0015] Furthermore, the DAM module described in S2 is used to calibrate infrared and visible light features, specifically as follows:

[0016] The input infrared and visible light features are concatenated along the channel dimension to generate joint features. Then the joint features A self-calibration unit containing convolution and global pooling operations is fed in to generate calibration weights; subsequently, these weights are applied to the joint features by element-wise multiplication. The feature representation is further enhanced by using residual connections; finally, a feature aggregation layer is used to compress the channel dimension and extract the joint feature representation. And re-segmented into calibrated visible light features and infrared light characteristics .

[0017] Furthermore, as described in S2, infrared and visible light calibration features are captured respectively through differential sensing pooling operations. , and The dependencies between dimensions are as follows:

[0018] First, calculate the pixel-level modal difference map. Subsequently, the feature tensor is compressed into three descriptor channels, explicitly preserving texture intensity, background statistics, and modal conflict information:

[0019]

[0020] in It is the input tensor. Is with Dimensionally aligned difference plots This indicates a compression operation performed along the 0th dimension;

[0021] Then let the input features ,exist and In the two branches, given two input tensors Input tensor respectively along shaft and Rotate the axis 90° counterclockwise to obtain the transformed tensor. and The rotated tensor is compressed through a DAP layer and then... Standard convolutional layers and sigmoid activation functions generate attention weights and ; Restore the original shape by reverse rotation; In Within the branches, the input is directly from the original input. DAP and convolution operations are applied to capture spatial dependencies and generate weights. The output tensor is obtained by aggregating the features of the three branches. Specifically:

[0022]

[0023] in This indicates element-wise multiplication;

[0024] Finally Weights are generated by two layers of convolution and non-linear activation, mapping high-dimensional attention features to a single-channel fusion mask. Then, a mask is used to model the feature pairs obtained in the pre-calibration stage. Perform adaptive weighting to obtain the final output. .

[0025] Furthermore, the functional implementation process of the DSFB module is as follows:

[0026] First, dense connection layers are used to enhance the features extracted by the backbone network, followed by a... After a convolutional layer and a ReLU activation function, the output is an enhanced deep feature pair. Then, a projection function employing shared convolution and reshaping operations is deployed to transform the enhanced features into keys and values.

[0027] Construct a neutral initial state. The initial state is concatenated with the features of each modality. Then, a shared context encoder is used to extract the context query value containing interaction information through a convolutional layer with shared weights. The specific formula is as follows:

[0028]

[0029] in , Represents the ReLU activation function;

[0030] Calculate cross-modal attention maps and obtain interaction features with global context based on the attention maps:

[0031]

[0032] in, Indicates another mode;

[0033] By calculating the scaled dot product attention, we obtain the interaction feature pairs. ;

[0034] Features of splicing The dual-channel gating mechanism generates a gating diagram. :

[0035]

[0036] in This represents the Sigmoid function. and These represent the learnable projection matrices for nonlinear and linear paths, respectively.

[0037] Finally, using gating graphs Adaptive weighted fusion to obtain fused features:

[0038] .

[0039] Furthermore, the functional implementation process of the SSAM module is as follows:

[0040] Select the last layer feature output by the DCAM module. For space anchor points For each deep fusion semantic feature Perform the following calibration operations on the anchor points:

[0041]

[0042] in, Indicates the calibrated number Layer features, This indicates a bilinear upsampling operation. It is a globally learnable scalar. The projection function is defined as follows: ;

[0043] calibrated features The features are concatenated and integrated to generate aligned semantic features. Specifically:

[0044]

[0045]

[0046] in, For standard projection units, This is a grouped convolutional projection.

[0047] Further, step S3 includes the following process:

[0048] First, a semantic awareness module (SPM) is constructed. The SPM module consists of consecutive convolutional blocks, each containing a kernel of size [missing information]. Convolutional layers, batch normalization, and ReLU activation functions are used to perform deep decoding of features and output high-level semantic features. Subsequently, boundary segmentation results are generated through three parallel projection paths. Semantic segmentation results and binary segmentation results :

[0049]

[0050]

[0051]

[0052] in, , and These represent the learnable convolutional kernel parameters for boundary prediction, semantic segmentation, and binary foreground mask paths, respectively.

[0053] A cascaded feature injection module (FIM) is constructed to progressively recover the spatial resolution of shallow features; specifically: firstly, ... As a conditional feature, it is injected into the first FIM. Subsequently, the fused features are used as new conditions and injected into the highest resolution via a second FIM. In the middle, the initially aggregated features are enhanced with finer-grained details to output texture features. ;

[0054] semantic features Injected into texture features Specifically:

[0055]

[0056]

[0057]

[0058] in, Indicates instance normalization, This indicates element-wise addition;

[0059] The semantically enhanced features are smoothed and refined to obtain the features. ; through a Image reconstruction head composed of convolutional layers and Tanh activation function synthesizes and fuses images. A source consistency reconstruction path is constructed, consisting of a modality-specific mask, convolutional layers, and a Tanh activation function, used to reconstruct the source consistency path from the source consistency path. Reconstructing the source image yields an infrared image. and visible light images .

[0060] Furthermore, the semantic fusion network is trained under supervised conditions using the following fusion loss:

[0061]

[0062] in, , and It's a hyperparameter; For semantic segmentation loss, For binary segmentation loss, For boundary segmentation loss;

[0063] Furthermore,

[0064]

[0065] in For balance coefficient, For weighted intensity loss, For texture loss, This is the correlation loss;

[0066]

[0067] in, express Norm, For the Sigmoid function, For target masking;

[0068]

[0069] in, This represents the Sobel gradient operator. Indicates absolute operation. Indicates the maximum selection at the element level;

[0070]

[0071] in, This indicates the calculation of the correlation between two images;

[0072]

[0073] in , Image reconstructed for the SCR branch.

[0074] This invention provides an infrared and visible light image fusion system based on bidirectional semantic-spatial alignment and differential sensing. The system has a program module corresponding to the steps of any of the methods described in the above technical solutions, and executes the steps in the infrared and visible light image fusion method based on bidirectional semantic-spatial alignment and differential sensing during runtime.

[0075] This invention provides a computer-readable storage medium storing a computer program configured to, when invoked by a processor, implement the steps in the infrared-visible light image fusion method based on bidirectional semantic-spatial alignment and differential sensing as described in any of the above technical solutions.

[0076] Compared with the prior art, the beneficial effects of the present invention are:

[0077] 1. A bidirectional semantic-spatial interaction framework is proposed, which systematically solves the feature misalignment problem in semantic-driven fusion by establishing a bidirectional path that guides deep semantics from shallow details and then guides texture reconstruction by alignment semantics.

[0078] 2. A Differential Sensing Complementary Module (DACM) was developed. It extracts physical complementary features by calculating the calibrated modal difference map and generates high-resolution spatial anchor points. By introducing calibrated differential priors, the network's physical sensing capability for thermal radiation targets in extreme environments is significantly enhanced.

[0079] 3. A spatial-semantic alignment module (SSAM) was designed, which uses shallow spatial anchors to calibrate deep semantic features layer by layer to eliminate edge bias introduced by upsampling. By using a learnable spatial anchor guidance mechanism, pixel-level cross-scale feature alignment is achieved, effectively suppressing boundary artifacts in the fusion result.

[0080] 4. In deep interaction, a deep semantic fusion block (DSFB) with embedded Parametric Efficient Context Interaction (PECI) captures the global semantic context while maintaining low computational cost and injects it back into the image reconstruction path. This mechanism effectively reduces computational redundancy while ensuring global context modeling capabilities through shared weights and a dual-path gating design. Attached Figure Description

[0081] Figure 1 This is an overall framework diagram of SASFusion in an embodiment of the present invention;

[0082] Figure 2 This is an overall framework diagram of DAM in an embodiment of the present invention;

[0083] Figure 3 This is an overall framework diagram of the DSFB in an embodiment of the present invention;

[0084] Figure 4 This is an overall framework diagram of SSAM in an embodiment of the present invention;

[0085] Figure 5 This is a qualitative comparison chart of SASFusion and seven other methods on the M3FD dataset in this embodiment of the invention.

[0086] Figure 6 This is a qualitative comparison chart of SASFusion and seven other methods on the RoadScene dataset in this embodiment of the invention.

[0087] Figure 7 This is a qualitative comparison chart of SASFusion and seven other methods on the TNO dataset in this embodiment of the invention.

[0088] Figure 8 The images show the segmentation results of each fusion algorithm in this embodiment of the invention on the FMB dataset. Detailed Implementation

[0089] To enable those skilled in the art to better understand the present invention, exemplary embodiments or examples of the present invention will be described below in conjunction with the accompanying drawings. Obviously, the described embodiments or examples are merely some, not all, of the embodiments or examples of the present invention. All other embodiments or examples obtained by those skilled in the art based on the embodiments or examples of the present invention without inventive effort should fall within the scope of protection of the present invention.

[0090] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

[0091] Specific Implementation Scheme 1: This invention provides an infrared-visible light image fusion method based on bidirectional semantic-spatial alignment and differential sensing, comprising the following steps:

[0092] S1. Acquire infrared and visible light images;

[0093] S2. Infrared and visible light images are fused based on a semantic fusion network. The semantic fusion network is based on the PSFusion network model, with its SDFM module replaced by a differential sensing complementary module DCAM, its PSFM module replaced by a deep semantic fusion module DSFB, and a spatial semantic alignment module SSAM added after the deep semantic fusion block.

[0094] The DAM module is used to calibrate infrared and visible light features, and captures infrared and visible light calibration features separately through differential sensing pooling operations. , and The dependencies between dimensions ultimately output the aggregated features of the three branches;

[0095] The DSFB module is used to enhance the features extracted by the backbone network and convert the enhanced features into keys and values. Then, a neutral initial state is created and concatenated with the features of each modality. The shared context encoder uses convolutional layers with shared weights to extract context query values ​​containing interaction information, calculates cross-modal attention maps, and obtains interaction features with global context. The concatenated interaction features are then fed into a dual-gating mechanism, which preserves the original amplitude information and gradient flow of the features through a linear path and captures complex semantic dependencies through a non-linear path, finally obtaining fused features.

[0096] The SSAM module is used to upsample each deep fusion semantic feature to the same spatial resolution as the selected anchor point, and then project the features onto a unified alignment space. Each layer introduces a globally learnable scalar to concatenate and integrate all aligned features. The integrated features are concatenated and compressed with the projected features of the original anchor point, and residual connections are introduced to generate aligned semantic features.

[0097] S3. Perform image reconstruction and supervision on alignment features.

[0098] This invention proposes a spatial anchor-guided semantic fusion network, abbreviated as SASFusion. It introduces a bidirectional semantic-spatial interaction (BSSI) strategy to address the cross-scale feature misalignment problem through explicit spatial anchor guidance. For example... Figure 1As shown, the proposed method comprises a Main Fusion Branch and an Auxiliary Semantic Branch. The Main Fusion Branch includes a main image reconstruction path and an auxiliary source consistency path. These two paths share a Feature Injection Module (FIM) and a Dense Feature Aggregation Block (DFAB), aiming to seamlessly integrate semantic information into texture features. The Auxiliary Semantic Branch consists of a Spatial-Semantic Alignment Module (SSAM), a Semantic Awareness Module (SPM), and a Semantic Segmentation Head (SDH). The SDH contains three task-specific headers for generating different segmentation results.

[0099] Given a pair of registered infrared images and visible images The goal of the main fusion branch is to synthesize fused images using the Image Reconstruction Head (IRH). The source image is reconstructed using an auxiliary source consistency reconstruction head. and The semantic auxiliary branch uses SSAM for feature alignment and then predicts the boundary segmentation results. Semantic segmentation results and binary segmentation results .

[0100] To ensure the accurate injection of semantic features from the semantic auxiliary branch into the main fusion branch, it is desirable to share a feature extraction network between these two branches. However, the main fusion branch needs to preserve high-resolution features to maintain fine scene texture, while advanced vision tasks require downsampling to extract sufficient semantic features and capture global structure. This conflict between these needs, along with the inherent spatial ambiguity of deep features, hinders the direct application of existing backbone networks. Therefore, as... Figure 1 As shown, ResNet is used. As the basic feature extraction network, two shallow feature blocks (SFBs) were designed to replace the first two layers (Stage 1 and Stage 2) of ResNet. This design preserves the original texture details to the greatest extent in the initial stage. Subsequently, the features are fed into the Differential Aware Complementary Module (DACM) and the Deep Semantic Fusion Block (DSFB) for processing. The feature extraction process is described as follows:

[0101]

[0102]

[0103] in, Represents the input source image. SFB and ResB represent custom shallow feature blocks and standard residual blocks, respectively. Specifically, when... when, It is a shallow physical perception feature (PPC feature), which mainly includes detailed textures and physical properties; hour These are Global Semantic Features (GSC Features), which contain abstract contextual semantics. To address the differences in the physical properties of these two types of features, DAM and DSFB were designed for processing, respectively.

[0104] Specific implementation scheme two: The DAM module mentioned in S2 is used to calibrate infrared and visible light features, specifically as follows:

[0105] Considering shallow features It incorporates the most basic physical properties, such as infrared thermal radiation intensity and visible light reflectance texture, while traditional "modal-blind" fusion operations like simple Sum or Max-Pooling often ignore the physical complementarity between modalities. To address this issue, Differential Aware Complementarity Module (DACM) is proposed. The architecture of DAM is as follows: Figure 2 As shown, it includes two stages: feature pre-calibration and differential perception attention.

[0106] Feature pre-calibration stage:

[0107] Directly calculating the differences in the original features is susceptible to sensor noise and feature distribution misalignment. Therefore, a self-calibration unit is introduced to clean the features before interaction. Specifically, the input infrared and visible light features are concatenated along the channel dimension to generate joint features. Then the joint features A self-calibration unit containing convolution and global pooling operations is fed in to generate calibration weights; subsequently, these weights are applied to the joint features by element-wise multiplication. The feature representation is further enhanced by using residual connections; finally, a feature aggregation layer is used to compress the channel dimension and extract the joint feature representation. The calibration process is as follows:

[0108]

[0109]

[0110]

[0111] in, This represents element-wise addition. This indicates element-wise multiplication. This represents the Sigmoid function. Indicates ReLU activation. This indicates global average pooling. This is used for subsequent calculations of physical differences.

[0112] Will Re-segmented into calibrated visible light features and infrared light characteristics .

[0113] Specific implementation scheme three: As described in S2, infrared and visible light calibration features are captured separately through differential sensing pooling operations. , and The dependencies between dimensions are as follows:

[0114] In acquiring calibration features and aggregation features Subsequently, in order to capture cross-dimensional physical complementarity, a three-branch rotational architecture was designed, each responsible for capturing... , and Dependencies between dimensions.

[0115] Unlike existing attention modules that only focus on the intensity statistics of features (Max / Avg), this invention proposes a Differential-Aware Pooling (DAP) operation to replace traditional Z-Pooling. Specifically,

[0116] First, calculate the pixel-level modal difference map. Subsequently, the feature tensor is compressed into three descriptor channels, explicitly preserving texture intensity, background statistics, and modal conflict information:

[0117]

[0118] in It is the input tensor. Is with Dimensionally aligned difference plots This indicates a compression operation performed along the 0th dimension;

[0119] Differential-Prior Attention (DPA) is defined as a three-branch module, and then the input features are... ,exist and In the two branches, to establish the interaction between the channel and the space, two input tensors are given. Input tensor respectively along shaft and Rotate the axis 90° counterclockwise to obtain the transformed tensor. and The rotated tensor is compressed through a DAP layer and then... Standard convolutional layers and sigmoid activation functions generate attention weights and The generated weights are used to modulate the corresponding rotation features; the original shape is recovered through reverse rotation; in Within the branches, the input is directly from the original input. DAP and convolution operations are applied to capture spatial dependencies and generate weights. ; Output tensor of DPA The refined features from these three branches are aggregated, and the process is described as follows:

[0120]

[0121] in This represents the sigmoid activation function; This represents a standard two-dimensional convolutional layer with a kernel size of 7 in the three branches of DPA. This represents a 90° clockwise rotation; through this mechanism, DPA uses differential priors to guide the network to focus on highly complementary regions, thereby highlighting salient targets while preserving details. Simplified to:

[0122]

[0123] in This indicates element-wise multiplication;

[0124] Finally Weights are generated by two layers of convolution and non-linear activation, mapping high-dimensional attention features to a single-channel fusion mask. Then, a mask is used to model the feature pairs obtained in the pre-calibration stage. Adaptive weighting is performed to obtain the final output of DAM. The process of fusing shallow features is described as follows:

[0125]

[0126]

[0127] in, Responsible for further extracting local contextual information, Used for channel dimensionality reduction and compression. This indicates element-wise multiplication.

[0128] Specific Implementation Plan Four: The functional implementation process of the DSFB module is as follows:

[0129] Given that high-level vision tasks typically require rich contextual information for comprehensive understanding, this invention designs a deep semantic fusion module (DSFB) based on embedded parameter-efficient contextual interaction to fuse deep features. For example... Figure 3 As shown, DSFB first utilizes densely connected layers to enhance the features extracted by the backbone network, and then passes through a... After a convolutional layer and a ReLU activation function, the output is an enhanced deep feature pair. Then, a projection function employing shared convolution and reshaping operations is deployed to transform the enhanced features into keys and values; as shown below:

[0130]

[0131] in Represents mode, Indicates key, Indicates the value. and These correspond to a convolutional layer with a kernel size of 1×1 and a reshaping operation, respectively. It is worth noting that... The parameters are fully shared between the two modalities, which helps to build a unified semantic space. , and These refer to the input features respectively. Height, width and passageway.

[0132] Construct a neutral initial state. To eliminate modal bias, the initial state is concatenated with the features of each modality. Then, a shared context encoder (SCE) is used to extract the context query value containing interaction information through convolutional layers with shared weights. The specific formula is as follows:

[0133]

[0134] in , This represents the ReLU activation function; and the parameters of SCE are completely shared between the two modes.

[0135] Calculate the cross-modal attention map and multiply the value matrix by the attention map to obtain interaction features with global context:

[0136]

[0137] in, Indicates another mode;

[0138] By calculating scaled dot product attention, cross-modal global information exchange is achieved, resulting in interactive feature pairs. ;

[0139] To generate accurate decision weights from interaction features, the concatenated features are... The input is fed into a dual-path gating mechanism (DPG). Unlike traditional single-path MLP gating, DPG contains two parallel feature activation paths: one is a linear path designed to preserve the original amplitude information and gradient flow of features; the other is a nonlinear path that introduces SiLU activation to capture complex semantic dependencies. This dual-path design ensures that the gating mechanism can respond to both large-amplitude linear features and finely tune nonlinear semantics.

[0140] Gating diagram The generation and weighted fusion process is as follows:

[0141]

[0142] in This represents the Sigmoid function. and These represent the learnable projection matrices for nonlinear and linear paths, respectively.

[0143] Finally, using gating graphs Adaptive weighted fusion to obtain fused features:

[0144] .

[0145] Specific Implementation Plan Five: The functional implementation process of the SSAM module is as follows:

[0146] Although the DAM and DSFB modules were used to obtain the fused shallow detail features and deep semantic features respectively, fusing these two types of features faces the challenge of cross-scale spatial misalignment. Figure 4 As shown, due to the deep fusion feature It is formed by fusing features at low resolution, and the semantic information it contains is presented as blurred "blobs". If the existing simple upsampling and stitching strategy is used, this blurred semantic boundary will not be able to match the real physical edge of the image, resulting in "semantic overflow" or texture distortion in the fusion result. To solve this problem, this invention proposes a Spatial-Semantic Alignment Module (SSAM), which aims to establish a "modulation flow" and use high-resolution shallow fusion features as spatial anchors to structurally calibrate and constrain the coarse deep fusion semantics.

[0147] Select the last layer feature output by the DCAM module. For space anchor points This is because Located at the end of the shallow flow, it retains the full spatial resolution and fine physical boundaries (textures, edges), and integrates differential complementary information of infrared and visible light through DAM, enabling it to provide structured guidance for the entire scene.

[0148] For each deep fusion semantic feature First, bilinear interpolation is used to upsample to the same spatial resolution as the anchor point. Then, batch normalization, ReLU activation function, and... Convolution projects features into a uniform alignment space. For anchor points... Perform the same projection operation as well. Finally, for each layer... Introducing a globally learnable scalar This endows SSAM with scale-aware characteristics, allowing the network to automatically adjust through backpropagation. Finding the optimal balance between "semantic abstraction" and "spatial constraints." Specifically, this means finding the optimal balance between spatially close hierarchies (such as...). The network may automatically assign larger weights to strengthen boundary alignment, while reducing the injection strength for the highest-level abstract semantics to maintain semantic purity. The specific process is as follows:

[0149]

[0150] in, Indicates the calibrated number Layer features, This indicates a bilinear upsampling operation. It is the Sigmoid function, representing the calibrated th... Layer features, It is a globally learnable scalar. The projection function is defined as follows: ;

[0151] calibrated features The features are then stitched together and fed into a grouping projection module for feature integration. To further smooth the gradient flow, the integrated features are compared with the original anchor points. The projected features are concatenated and compressed, and residual connections are introduced to generate the final aligned semantic features. :

[0152]

[0153]

[0154] in, For standard projection units, This is a grouped convolutional projection.

[0155] The spatial anchor guidance mechanism in this implementation scheme effectively eliminates edge artifacts caused by upsampling, ensuring that the features injected into subsequent reconstruction branches have both advanced semantic understanding and pixel-level spatial accuracy.

[0156] Specific implementation plan six: Step S3 includes the following process:

[0157] After precise calibration with SSAM, alignment features that combine global semantic understanding with pixel-level spatial accuracy were obtained. In order to effectively The information is converted into visual information to guide the final image generation. The following section will focus on image reconstruction and supervision.

[0158] First, in order to align the features Mapping to a specific semantic category space, a semantic awareness module (SPM) is constructed. The SPM module consists of consecutive convolutional blocks, each containing a kernel of size [missing information]. Convolutional layers, batch normalization, and ReLU activation functions are used to perform deep decoding of features and output high-level semantic features. ;

[0159] A lightweight semantic decoding head (SDH) was constructed. The SDH consists of three parallel projection paths, each comprising a convolutional layer that modifies the channel dimensions and a subsequent bilinear upsampling operation. These paths generate boundary segmentation results. Semantic segmentation results and binary segmentation results :

[0160]

[0161]

[0162]

[0163] in, This indicates bilinear upsampling to the original image resolution. , and These represent the learnable convolutional kernel parameters for boundary prediction, semantic segmentation, and binary foreground mask paths, respectively.

[0164] To ensure the main fusion branch can fully utilize the fine semantic information provided by the auxiliary branches, a cascaded feature injection module (FIM) is constructed to progressively restore the spatial resolution of shallow features; specifically: firstly, the output of the last DAM layer is... As a conditional feature, it is injected into the output of the second-layer DAM through the first FIM. Subsequently, the fused features are used as new conditions and injected into the output of the highest-resolution DAM first layer through a second FIM. The initially aggregated features are fed into a dense feature aggregation block (DFAB), which consists of convolutional blocks and dense connections to enhance fine-grained details and output aggregated texture features. ;

[0165] Semantic features generated by SPM are processed through FIM. Injected into texture features Specifically:

[0166]

[0167]

[0168]

[0169] in, Indicates instance normalization, This indicates element-wise addition;

[0170] Another DFAB is used to smooth and refine the semantically enhanced features, eliminating artifacts that may occur during the injection process, and obtaining the features. ; through a Image reconstruction head (IRH) composed of convolutional layers and Tanh activation function synthesizes and fuses images. A source consistency reconstruction (SCR) path is constructed, consisting of a modality-specific mask, convolutional layers, and a Tanh activation function, used to reconstruct the source consistency reconstruction (SCR) path from the source consistency reconstruction path. The source image is reconstructed from the image; therefore, SCR can constrain... Containing complete information, an infrared image is reconstructed. and visible light images This self-supervised constraint ensures that the fusion results maintain the integrity of physical information while pursuing semantic consistency.

[0171] Specific Implementation Scheme Seven: The semantic fusion network is trained with the following fusion loss:

[0172]

[0173] in, , and It's a hyperparameter; For semantic segmentation loss, For binary segmentation loss, For boundary segmentation loss;

[0174] Furthermore,

[0175]

[0176] in For balance coefficient, For weighted intensity loss, For texture loss, This is the correlation loss;

[0177]

[0178] in, express Norm, The sigmoid function is used to limit the range of two combined masks to a certain value. Inside, For target masking;

[0179] Texture loss This allows the merged image to contain rich texture information:

[0180]

[0181] in, This represents the Sobel gradient operator. Indicates absolute operation. Indicates the maximum selection at the element level;

[0182] Correlation loss Used to enhance the correlation between the fused image and the source image:

[0183]

[0184] in, This indicates the calculation of the correlation between two images;

[0185] Source consistency loss, ensuring that the SCR branch can achieve this goal, consists of intensity and texture terms:

[0186]

[0187] in , Image reconstructed for the SCR branch.

[0188] The infrared and visible light image fusion method (algorithm) based on bidirectional semantic-spatial alignment and differential perception proposed in this invention is the underlying technical core of this invention, and various products can be derived based on the algorithm.

[0189] Based on the method proposed in this invention, an infrared and visible light image fusion system based on bidirectional semantic-spatial alignment and differential sensing is developed using a programming language. This system has program modules corresponding to the steps of the above-mentioned technical solution, and executes the steps in the above-mentioned infrared and visible light image fusion method based on bidirectional semantic-spatial alignment and differential sensing during runtime.

[0190] The developed system (software) computer program is stored on a computer-readable storage medium. This computer program is configured to implement the steps of the aforementioned infrared-visible light image fusion method based on bidirectional semantic-spatial alignment and differential sensing when invoked by a processor. In other words, the invention is materialized on a carrier, becoming a computer program product.

[0191] Various implementations of the systems and techniques described herein can be implemented in digital electronic circuit systems, integrated circuit systems, application-specific integrated circuits (ASICs), computer hardware, firmware, software, and / or combinations thereof. These various implementations may include: implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0192] The computational programs (also referred to as programs, software, software applications, or code) of this invention include machine instructions of a programmable processor and can be implemented using high-level procedural and / or object-oriented programming languages, and / or assembly / machine languages. As used herein, the terms "machine-readable medium" and "computer-readable medium" refer to any computer program product, device, and / or apparatus (e.g., disk, optical disk, memory, programmable logic device PLD) for providing machine instructions and / or data to a programmable processor, including machine-readable media that receive machine instructions as machine-readable signals. The term "machine-readable signal" refers to any signal for providing machine instructions and / or data to a programmable processor.

[0193] The beneficial effects of the present invention will be described below with reference to specific embodiments.

[0194] Example 1

[0195] In the infrared-visible image fusion experiments, the proposed method was first trained on the MSRS dataset, which contains 1,083 pairs of long-wave infrared and visible light images. Subsequently, the trained model was evaluated on three other datasets: TNO (57 pairs), RoadScene (221 pairs), and M3FD (300 pairs). These four benchmark datasets cover a variety of real-world scenarios, including urban roads, nighttime environments, military targets, and multispectral detection, providing comprehensive testing of the method's robustness and generalization ability. Furthermore, to specifically evaluate the facilitation effect of the fusion results on downstream tasks, semantic segmentation performance was tested on the FMB dataset (280 pairs).

[0196] All experiments were conducted on a workstation equipped with an 18-core AMD EPYC 9754 CPU and an NVIDIA GeForce RTX 4090D GPU. End-to-end joint training was performed on the main fusion branch and the auxiliary semantic branch within a single framework. Hyperparameters controlling various loss term tradeoffs were empirically set to... = 10、 = 5、 = 10 and =0.1. Model optimization uses a stochastic gradient descent (SGD) optimizer with a batch size of 16. Momentum is set to 0.9, and weight decay is set to 0.0005. The initial learning rate is set to 0.001, and a multinomial learning rate decay strategy is used. The model is trained for 1000 epochs to fully extract semantic information. All images are normalized to [0, 1] before being input into the network.

[0197] The fusion results are quantitatively evaluated using six metrics: entropy (EN), standard deviation (SD), spatial frequency (SF), average gradient (AG), Qabf, and sum of correlation differences (SCD). Higher values ​​for these metrics indicate better fusion. EN assesses the information content of the fused image, SD measures grayscale contrast, SF reflects the activity of texture details, AG assesses overall sharpness and contrast intensity, Qabf assesses edge preservation, and SCD measures the difference in information relevance between the fused image and the source image.

[0198] To comprehensively evaluate the effectiveness of SASFusion, extensive qualitative and quantitative comparisons were performed against six state-of-the-art image fusion methods. The selected baseline methods included U2Fusion. ,BDLFusion DDFM LRRNet CAF TIMFusion MaeFuse To ensure openness and consistency in the evaluation, publicly available implementations of these methods were used, along with the same experimental setup, including hyperparameters and input preprocessing procedures.

[0199] The visual results of various fusion algorithms on the M3FD, TNO, and RoadScene datasets are shown in [the respective tables]. Figure 5 , 6 And 7. For example Figure 5 As shown in the first row, under smoke-covered conditions, BDLFusion, LRRNet, and TIMFusion failed to effectively penetrate the smoke, making the containers behind them almost invisible. Although U2Fusion, DDFM, CAF, and MaeFuse could barely render the container outlines, the overall clarity was poor, and they failed to highlight the thermal radiation characteristics of pedestrians in the smoke. In contrast, the SASFusion of this invention not only clearly displayed the structural details of the containers but also highlighted pedestrians with the highest salience, achieving a perfect balance between de-occlusion and target prominence.

[0200] exist Figure 5 In the second row of nighttime scenes, the pedestrians in the fusion results of U2Fusion, DDFM, LRRNet, MaeFuse, and CAF are generally dark and lack contrast. While BDLFusion and TIMFusion improve brightness, they produce obvious edge blurring in the lower body area of ​​the pedestrians. In contrast, the image of this invention not only maintains sufficient thermal target intensity but also presents a sharp outline at the leg edges without halos or clipping.

[0201] Furthermore, the visual effects on TNO and RoadScene also demonstrate the superiority of the method of this invention. From Figure 6 It can be noted that the SASFusion of this invention not only preserves the semantic markings on the sign, but also generates the clearest support rods, completely eliminating the edge blurring or ghosting phenomena common in other methods. Figure 7 As shown, only the method of this invention perfectly penetrates the smoke, clearly displaying the background tree trunks and soldiers while cleverly preserving the shape and outline of the smoke. Extensive qualitative comparisons and analyses fully demonstrate the superior fusion performance of SASFusion from a visual perception perspective. In particular, the proposed method can effectively cope with harsh environments such as nighttime, foggy days, and obstructions, providing semantically clear and physically structurally complete fused images.

[0202] The quantitative comparison results are shown in Tables 1-3. For the M3FD dataset, which has independent training and test sets, the various metrics of different algorithms were evaluated on the test set. For the TNO and RoadScene datasets, which do not have predefined data partitions, 57 and 221 pairs of images, respectively, were selected for quantitative comparison. The experimental results show that SASFusion exhibits a significant performance advantage on all three datasets. Particularly on the M3FD dataset, the method of this invention ranks first in all six metrics, fully demonstrating its comprehensive superiority in complex scenes. On the TNO and RoadScene datasets, it also achieves the best results on the vast majority of metrics, only ranking second by a slight margin on a few metrics (such as SF or SD).

[0203] SASFusion achieved the best results on the EN metric, indicating that the generated fused image contains sufficient and rich information; its best performance on the SCD metric demonstrates the highest correlation between the fused result and the source image. This advantage is attributed to the SCR branch designed in this invention, which, through self-supervised constraints, ensures that the fusion network does not lose physical information of the source image during deep feature interactions.

[0204] Furthermore, SASFusion's overall lead in AG metrics and its excellent performance in SF metrics (first in M3FD, top two in TNO / RoadScene) indicate that the fusion results contain rich texture details, consistent with the qualitative analysis results. It is worth noting that although SF is slightly lower than the comparison methods on some datasets, this is generally because SASFusion suppresses high-frequency noise in the source images, thus generating visually cleaner images.

[0205] Meanwhile, the method of this invention achieved an extremely high score on the SD metric (first in M3FD / RoadScene), indicating that the fused image has optimal contrast. This is mainly attributed to the contrast mask introduced by the DAM module, which guides the network to adaptively preserve the high intensity of infrared light. Finally, the excellent performance on the Qabf metric, which measures edge information (first in M3FD / TNO), shows that the fused image generated by SASFusion successfully preserves the edge structure and contour integrity of the source image. This result strongly confirms that the SSAM module achieves pixel-level precise feature alignment through spatial anchor point guidance.

[0206] In summary, both qualitative and quantitative experimental results fully demonstrate the superior performance of the proposed method in image fusion tasks.

[0207] Table 1. Quantitative comparison results of SASFusion and 7 other methods on 300 image pairs in the M3FD dataset.

[0208]

[0209] Note: The best result is highlighted in bold, while the second best result is highlighted in underline.

[0210] Table 2. Quantitative comparison results of SASFusion and 7 other methods on 57 image pairs in the TNO dataset.

[0211]

[0212] Note: The best result is highlighted in bold, while the second best result is highlighted in underline.

[0213] Table 3. Quantitative comparison results of SASFusion and 7 other methods on 221 image pairs in the RoadScene dataset.

[0214]

[0215] Note: The best result is highlighted in bold, while the second best result is highlighted in underline.

[0216] Segmentation, comparison, and analysis:

[0217] To quantitatively evaluate the semantic quality of fused images in high-level vision tasks, this embodiment conducts semantic segmentation experiments on the FMB dataset. For each fusion method, fused versions of the official FMB training and test sets are first generated. Then, SegFormer-B3 is selected. As a unified segmentation baseline model, each model was trained on its own fused training set and evaluated on its corresponding fused test set. All models were trained for 100 epochs with a batch size of 8, supervised by the cross-entropy loss function, and optimized using AdamW with an initial learning rate of 1e-4, decaying through cosine annealing. Segmentation performance was measured by mean intersection-union (mIoU) and mean accuracy (mAcc).

[0218] Table 4 shows the segmentation performance of different fusion methods on the FMB dataset. SASFusion achieved the best results in both mIoU and mAcc, significantly outperforming the comparison methods.

[0219] Specifically, in categories with fine structures such as poles and signs, the fusion results of this invention achieved the best ACC and IoU, strongly demonstrating that SSAM achieves pixel-level precise alignment through spatial anchor point guidance, effectively preventing structural breaks or ghosting of slender objects during the fusion process. Furthermore, in categories with significant thermal radiation characteristics such as cars, buses, and trucks, SASFusion maintains a leading IoU metric. Thanks to DAM's keen capture of physically complementary information, the method of this invention still completely preserves the thermal radiation contours of vehicles under complex lighting conditions. It is worth noting that for large-scale background regions such as roads and buildings, although the ACC of this invention is comparable to or even slightly lower than the comparison methods, it achieves the highest IoU. In semantic segmentation, high ACC and low IoU usually indicate a tendency for the model to 'oversegment'. However, this invention does the opposite, indicating that the fused image generated by SASFusion has sharper and more convergent edges at the boundaries of different regions. This precise segmentation characteristic effectively reduces edge pixel confusion and overflow errors.

[0220] Table 4. Segmentation results of SASFusion and 7 other methods on the FMB dataset.

[0221]

[0222] Note: The best result is highlighted in bold, while the second best result is highlighted in underline.

[0223] The visualization of the segmentation results is shown in Figure 8. In the first smoke scene, the semantic segmentation task was significantly interfered with due to the strong scattering of thermal radiation in the high-density smoke. Most contrastive methods failed to extract effective discriminative features from the low signal-to-noise ratio source image, resulting in complete false negatives. Although CAF was able to locate the pedestrian on the left side of the smoke, its generated segmentation mask suffered from severe structural truncation, barely covering only a small portion of the lower limb region. In contrast, the SASFusion method of this invention exhibits stronger feature recovery capabilities. By capturing subtle differences in thermal radiation using DAM, the method of this invention can reconstruct a larger and structurally coherent torso region under strong occlusion conditions. Although perfect segmentation is difficult to achieve under extreme occlusion, SASFusion demonstrates a more reasonable trade-off between maintaining semantic continuity and target coverage integrity compared to existing methods.

[0224] Secondly, in the second complex road scenario, existing methods often exhibit fragmented or region-blurring problems in predicting the boundary between the sidewalk and the background. Only the method of this invention successfully depicted a continuous and structurally complete sidewalk region. This advantage is mainly attributed to the pixel-level spatial guidance provided by SSAM, which enables the segmentation network to accurately define the topological structure of different semantic regions using aligned sharp texture boundaries.

[0225] The quantitative and qualitative comparison results fully verify the superior potential of the proposed method compared with the current state-of-the-art image fusion algorithms in high-level vision tasks.

[0226] While the present invention has been disclosed above, its scope of protection is not limited thereto. Those skilled in the art can make various changes and modifications without departing from the spirit and scope of the present invention, and all such changes and modifications will fall within the scope of protection of the present invention.

[0227] The documents cited in this invention include:

[0228] K. He, X. Zhang, S. Ren, J. Sun, Deep residual learning for imagerecognition, in: Proceedings of the IEEE Conference on Computer Vision andPattern Recognition, 2016: pp. 770–778.

[0229] H. Xu, J. Ma, J. Jiang, X. Guo, H. Ling, U2Fusion: A UnifiedUnsupervised Image Fusion Network, (n.d.).

[0230] Z. Liu, J. Liu, G. Wu, L. Ma, X. Fan, R. Liu, Bi-level DynamicLearning for Jointly Multi-modality Image Fusion and Beyond, (2023). https: / / doi.org / 10.48550 / arXiv.2305.06720.

[0231] Z. Zhao, H. Bai, Y. Zhu, J. Zhang, S. Xu, Y. Zhang, K. Zhang, D.Meng, R. Timofte, L. Van Gool, DDFM: Denoising Diffusion Model for Multi-Modality Image Fusion, in: 2023: pp. 8082–8093.

[0232] H. Li, T. Xu, X.-J. Wu, J. Lu, J. Kittler, LRRNet: A NovelRepresentation Learning Guided Fusion Network for Infrared and VisibleImages, IEEE Transactions on Pattern Analysis and Machine Intelligence 45(2023) 11040–11052

[0233] J. Liu, G. Wu, Z. Liu, L. Ma, R. Liu, X. Fan, Where elegancemeets precision: Towards a compact, automatic, and flexible framework formulti-modality image fusion and applications, in: Proceedings of the Thirty-Third International Joint Conference on Artificial Intelligence, 2024: pp.1110–1118.

[0234] R. Liu, Z. Liu, J. Liu, X. Fan, Z. Luo, A Task-Guided,Implicitly-Searched and Meta-Initialized Deep Model for Image Fusion, IEEETransactions on Pattern Analysis and Machine Intelligence 46 (2024) 6594–6609.

[0235] J. Li, J. Jiang, P. Liang, J. Ma, L. Nie, MaeFuse: TransferringOmni Features with Pretrained Masked Autoencoders for Infrared and VisibleImage Fusion via Guided Training, (2025).

[0236] E. Xie, W. Wang, Z. Yu, A. Anandkumar, J.M. Alvarez, P. Luo,SegFormer: Simple and Efficient Design for Semantic Segmentation withTransformers, in: Advances in Neural Information Processing Systems, CurranAssociates, Inc., 2021: pp. 12077–12090。

Claims

1. A method for infrared and visible light image fusion based on bidirectional semantic-spatial alignment and differential sensing, characterized in that, Includes the following steps: S1. Acquire infrared and visible light images; S2. Infrared and visible light images are fused based on a semantic fusion network. The semantic fusion network is based on the PSFusion network model. Its SDFM module is replaced with a differential sensing complementary module DAM, the PSFM module is replaced with a deep semantic fusion module DSFB, and a spatial semantic alignment module SSAM is added after the deep semantic fusion block and the differential sensing complementary module. The DAM module is used to calibrate infrared and visible light features, and captures infrared and visible light calibration features separately through differential sensing pooling operations. , and The dependencies between dimensions ultimately output the aggregated features of the three branches; The DSFB module is used to enhance the features extracted by the backbone network and convert the enhanced features into keys and values. Then, a neutral initial state is created and concatenated with the features of each modality. The shared context encoder uses convolutional layers with shared weights to extract context query values ​​containing interaction information, calculates cross-modal attention maps, and obtains interaction features with global context. The concatenated interaction features are then fed into a dual-gating mechanism, which preserves the original amplitude information and gradient flow of the features through a linear path and captures complex semantic dependencies through a non-linear path, finally obtaining fused features. The SSAM module is used to upsample each deep fusion semantic feature to the same spatial resolution as the selected anchor point, and then project the features onto a unified alignment space. Each layer introduces a globally learnable scalar to concatenate and integrate all aligned features. The integrated features are concatenated and compressed with the projected features of the original anchor point, and residual connections are introduced to generate aligned semantic features. S3. Perform image reconstruction and supervision on alignment features.

2. The method according to claim 1, characterized in that, The DAM module described in S2 is used to calibrate infrared and visible light features, specifically: The input infrared and visible light features are concatenated along the channel dimension to generate joint features. Then the joint features A self-calibration unit containing convolution and global pooling operations is fed in to generate calibration weights; subsequently, these weights are applied to the joint features by element-wise multiplication. The feature representation is further enhanced by using residual connections; finally, a feature aggregation layer is used to compress the channel dimension and extract the joint feature representation. And re-segmented into calibrated visible light features and infrared light characteristics .

3. The method according to claim 2, characterized in that, As described in S2, infrared and visible light calibration features are captured through differential sensing pooling operations, respectively. , and The dependencies between dimensions are as follows: First, calculate the pixel-level modal difference map. Subsequently, the feature tensor is compressed into three descriptor channels, explicitly preserving texture intensity, background statistics, and modal conflict information: in It is the input tensor. Is with Dimensionally aligned difference plots This indicates a compression operation performed along the 0th dimension; Then let the input features ,exist and In the two branches, given two input tensors Input tensor respectively along shaft and Rotate the axis 90° counterclockwise to obtain the transformed tensor. and ; The rotated tensor is compressed through a DAP layer, and then... Standard convolutional layers and sigmoid activation functions generate attention weights and ; The original shape is restored by reverse rotation; Within the branches, the input is directly from the original input. DAP and convolution operations are applied to capture spatial dependencies and generate weights. The output tensor is obtained by aggregating the features of the three branches. Specifically: in This indicates element-wise multiplication; Finally Weights are generated by two layers of convolution and non-linear activation, mapping high-dimensional attention features to a single-channel fusion mask. Then, a mask is used to model the feature pairs obtained in the pre-calibration stage. Perform adaptive weighting to obtain the final output. .

4. The method according to claim 3, characterized in that, The functional implementation process of the DSFB module is as follows: First, dense connection layers are used to enhance the features extracted by the backbone network, followed by a... After a convolutional layer and a ReLU activation function, the output is an enhanced deep feature pair. Then, a projection function employing shared convolution and reshaping operations is deployed to transform the enhanced features into keys and values. Construct a neutral initial state. The initial state is concatenated with the features of each modality. Then, a shared context encoder is used to extract the context query value containing interaction information through a convolutional layer with shared weights. The specific formula is as follows: in , Represents the ReLU activation function; Calculate cross-modal attention maps and obtain interaction features with global context based on the attention maps: in, Indicates another mode; By calculating the scaled dot product attention, we obtain the interaction feature pairs. ; Features of splicing The dual-channel gating mechanism generates a gating diagram. : in This represents the Sigmoid function. and These represent the learnable projection matrices for nonlinear and linear paths, respectively. Finally, using gating graphs Adaptive weighted fusion to obtain fused features: 。 5. The method according to claim 4, characterized in that, The functional implementation process of the SSAM module is as follows: Select the last layer feature output by the DCAM module. For space anchor points For each deep fusion semantic feature Perform the following calibration operations on the anchor points: in, Indicates the calibrated number Layer features, This indicates a bilinear upsampling operation. It is a globally learnable scalar. The projection function is defined as follows: ; calibrated features The features are concatenated and integrated to generate aligned semantic features. Specifically: in, For standard projection units, This is a grouped convolutional projection.

6. The method according to claim 5, characterized in that, Step S3 includes the following process: First, a semantic awareness module (SPM) is constructed. The SPM module consists of consecutive convolutional blocks, each containing a kernel of size [missing information]. Convolutional layers, batch normalization, and ReLU activation functions are used to perform deep decoding of features and output high-level semantic features. Subsequently, boundary segmentation results are generated through three parallel projection paths. Semantic segmentation results and binary segmentation results : in, , and These represent the learnable convolutional kernel parameters for boundary prediction, semantic segmentation, and binary foreground mask paths, respectively. A cascaded feature injection module (FIM) is constructed to progressively recover the spatial resolution of shallow features; specifically: firstly, ... As a conditional feature, it is injected into the first FIM. Subsequently, the fused features are used as new conditions and injected into the highest resolution via a second FIM. In the middle, the initially aggregated features are enhanced with finer-grained details to output texture features. ; semantic features Injected into texture features Specifically: in, Indicates instance normalization, This indicates element-wise addition; The semantically enhanced features are smoothed and refined to obtain the features. ; through a Image reconstruction head composed of convolutional layers and Tanh activation function synthesizes and fuses images. A source consistency reconstruction path is constructed, consisting of a modality-specific mask, convolutional layers, and a Tanh activation function, used to reconstruct the source consistency path from the source consistency path. Reconstructing the source image yields an infrared image. and visible light images .

7. The method according to claim 6, characterized in that, The semantic fusion network is trained under supervision using the following fusion loss: in, , and It's a hyperparameter; For semantic segmentation loss, For binary segmentation loss, For boundary segmentation loss; Furthermore, in For balance coefficient, For weighted intensity loss, For texture loss, This is the correlation loss; in, express Norm, For the Sigmoid function, For target masking; in, This represents the Sobel gradient operator. Indicates absolute operation. Indicates the maximum selection at the element level; in, This indicates the calculation of the correlation between two images; in , Image reconstructed for the SCR branch.

8. An infrared-visible light image fusion system based on bidirectional semantic-spatial alignment and differential sensing, characterized in that, The system has a program module corresponding to the steps of the method described in any one of claims 1 to 7, and executes the steps in the infrared and visible light image fusion method based on bidirectional semantic-spatial alignment and differential perception when it is run.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program configured to, when invoked by a processor, implement the steps of the infrared-visible image fusion method based on bidirectional semantic-spatial alignment and differential sensing as described in any one of claims 1 to 7.

Images