Infrared and visible image fusion method and system combined with self-supervised feature alignment

Abstract

The application discloses an infrared and visible light image fusion method and system combining self-supervised feature alignment, and belongs to the technical field of image fusion. The infrared and visible light image fusion method combining self-supervised feature alignment comprises the following steps: acquiring an infrared image and a visible light image; performing feature extraction on the infrared image and the visible light image based on a hierarchical double-flow heterogeneous model, and generating appearance features and geometric features of the infrared image and the visible light image based on a cross-modal projection model; aligning the appearance features and the geometric features based on a double-space hierarchical self-supervised alignment model to generate aligned features of the infrared image and the visible light image; performing feature fusion on the aligned features of the infrared image and the visible light image based on a spectrum-graph neural joint fusion model to obtain fusion features; and reconstructing a fusion image based on the fusion features and outputting the fusion image. The application can effectively improve the definition, detail integrity and target saliency of the fusion image.

Description

Technical Field

[0001] This invention belongs to the field of image fusion technology, specifically relating to an infrared and visible light image fusion method and system that combines self-supervised feature alignment. Background Technology

[0002] The statements herein provide only background information in relation to this invention and do not necessarily constitute prior art.

[0003] Infrared and visible light image fusion technology aims to comprehensively utilize the thermal radiation characteristics of infrared images and the texture details of visible light images to obtain high-quality images that contain both thermal information and clear details. This technology has wide applications in fields such as night vision surveillance, target detection, medical imaging, and intelligent transportation.

[0004] However, the inventors discovered that existing infrared and visible light image fusion methods have shortcomings in feature extraction, alignment accuracy, data dependence, fusion balance, and computational efficiency. There is an urgent need for a new method with better fusion results, lower data dependence, and the ability to effectively solve alignment problems. Summary of the Invention

[0005] The purpose of this invention is to overcome the shortcomings of the prior art and provide an infrared and visible light image fusion method and system that combines self-supervised feature alignment, which can effectively improve the clarity, detail integrity and target saliency of the fused image.

[0006] To achieve the above objectives, the present invention is implemented through the following technical solution:

[0007] On one hand, the technical solution of the present invention provides a method for fusion of infrared and visible light images by combining self-supervised feature alignment, including:

[0008] Acquire infrared and visible light images;

[0009] Feature extraction is performed on infrared and visible light images based on a hierarchical dual-stream heterogeneous model, and appearance and geometric features of infrared and visible light images are generated based on a cross-modal projection model.

[0010] Based on the dual-space hierarchical self-supervised alignment model, appearance features and geometric features are aligned to generate alignment features for infrared and visible light images, respectively.

[0011] The alignment features of infrared and visible light images are fused using a spectrum-graph neural joint fusion model to obtain fused features.

[0012] The fused image is reconstructed and output based on the fusion features.

[0013] In at least one embodiment, after acquiring infrared and visible light images, the acquired images are preprocessed, including size alignment and intensity normalization. For infrared images, temperature statistical feature extraction is also performed, and for visible light images, texture enhancement is also performed.

[0014] In at least one embodiment, the hierarchical dual-stream heterogeneous model includes three branches: an infrared branch, a visible light branch, and a shared geometry branch.

[0015] The infrared branch employs a lightweight multi-scale convolutional neural network and introduces a spectral attention submodule to preserve the local intensity and shape boundaries of the thermal spectrum. The visible light branch uses a hybrid convolutional neural network—Transformer—to extract local textures, while a small Transformer at the back end extracts global textures and color distributions. The shared geometry branch uses a shallow convolutional neural network and introduces a small learnable offset module to capture the common geometry of the two modes.

[0016] In at least one embodiment, the process of generating appearance features and geometric features specifically includes:

[0017] Using preprocessed infrared and visible light images as input, low-level features are extracted based on a hierarchical dual-stream heterogeneous model to obtain the low-level features of the infrared image and the visible light image.

[0018] Using the extracted low-level features of infrared images and visible light images as input, the low-level features are modally mapped and dimensionally adjusted based on the first cross-modal projection module to obtain infrared deep semantic features and visible light deep semantic features.

[0019] Based on the second cross-modal projection model, infrared deep semantic features and visible light deep semantic features are mapped to appearance subspace and geometric subspace respectively, generating appearance and geometric features of infrared images, as well as appearance and geometric features of visible light images.

[0020] In at least one embodiment, the dual-space hierarchical self-supervised alignment model employs a dual-space hierarchical self-supervised feature alignment framework and a dynamic weight multi-task joint optimization mechanism to achieve consistent representation by minimizing cross-modal feature differences.

[0021] In at least one embodiment, appearance features are aligned with geometric features based on a dual-space hierarchical self-supervised alignment model, specifically including:

[0022] The appearance and geometric features of infrared and visible light are L2 normalized and then embedded into the alignment space through a linear projection layer.

[0023] Appearance spatial alignment based on local patch and spatially weighted InfoNCE;

[0024] Based on geometric alignment branch prediction of cross-modal displacement field and confidence map, spatial alignment of geometric features of infrared and visible light is performed;

[0025] The corrected geometric coordinates are mapped and synchronized to the appearance features, so that the infrared and visible light features are spatially aligned, generating aligned features for the infrared and visible light images respectively.

[0026] In at least one embodiment, the dynamic weighted multi-task joint optimization mechanism includes reconstruction loss, perception loss, spatially aware InfoNCE loss, geometrical alignment loss, structural similarity loss, and gradient preservation loss.

[0027] In at least one embodiment, three dynamic mechanisms—uncertainty weighting, gradient balancing-based dynamic adjustment, and meta-learning controller—are used to dynamically adjust the various loss weights in the dynamic weight multi-task joint optimization mechanism.

[0028] In at least one embodiment, the spectrum-graph neural joint fusion model adopts a spectrum-graph neural dual-branch structure. The low-frequency component is used to construct the spatial graph. On the spatial graph, a multi-layer graph attention network is used to extract topologically consistent structures to obtain the structure-enhanced low-frequency components, which are then fused across modally to obtain low-frequency fusion features. The high-frequency components are enhanced in appearance and texture and geometric edges through spectrum attention to obtain appearance texture enhanced high-frequency components and geometric edge enhanced high-frequency components, which are then fused across modally to obtain high-frequency fusion features. The low-frequency fusion features and high-frequency fusion features are inversely transformed to obtain low-frequency enhanced features and high-frequency enhanced features, which are then fused in the spatial domain through cross-attention to obtain fusion features.

[0029] On the other hand, the technical solution of the present invention also provides an infrared and visible light image fusion system combining self-supervised feature alignment, comprising:

[0030] The image acquisition module is configured to acquire infrared images and visible light images;

[0031] The feature extraction module is configured to: extract features from infrared and visible light images based on a hierarchical dual-stream heterogeneous model, and generate appearance and geometric features of infrared and visible light images based on a cross-modal projection model.

[0032] The feature alignment module is configured to align appearance features with geometric features based on a dual-space hierarchical self-supervised alignment model, generating alignment features for infrared and visible light images respectively.

[0033] The feature fusion module is configured to: fuse the alignment features of infrared and visible light images based on a spectrum-graph neural joint fusion model to obtain fused features;

[0034] The reconstruction and output module is configured to: reconstruct and output the fused image based on the fusion features.

[0035] The beneficial effects of the above-described technical solution of the present invention are as follows:

[0036] 1) The infrared and visible light image fusion method of the present invention, which combines self-supervised feature alignment, mines complementary information between infrared and visible light through a hierarchical dual-stream heterogeneous model and a cross-modal projection model to obtain more discriminative feature representations. The dual-space hierarchical self-supervised alignment model effectively reduces geometric misalignment problems and ensures accurate position of details in the fused image. The spectrum-graph neural joint fusion model balances the fusion effect, so that while maintaining the salient targets in infrared, it retains the rich texture details in visible light, achieving information complementarity, thereby improving the clarity, detail integrity and target salientity of the fused image.

[0037] 2) The infrared and visible light image fusion method combined with self-supervised feature alignment of the present invention adopts a self-supervised training method, which reduces the dependence on large-scale labeled samples, improves the generalizability of the method, and adopts an efficient structural design in the feature alignment and fusion process, which reduces the computational complexity and is more suitable for embedded and real-time application scenarios.

[0038] 3) This invention can be applied to various scenarios such as night vision surveillance, intelligent transportation, medical imaging, and military reconnaissance. It has a wide range of applications and high application value. Attached Figure Description

[0039] The accompanying drawings, which form part of this invention, are used to provide a further understanding of the invention. The illustrative embodiments of the invention and their descriptions are used to explain the invention and do not constitute an improper limitation of the invention.

[0040] Figure 1 This is a schematic diagram of the infrared and visible light image fusion method combining self-supervised feature alignment disclosed in Embodiment 1 of the present invention;

[0041] Figure 2 This is a flowchart of the infrared and visible light image fusion method combining self-supervised feature alignment disclosed in Embodiment 1 of the present invention;

[0042] Figure 3 This is a schematic diagram of an infrared and visible light image fusion system combining self-supervised feature alignment disclosed in Embodiment 2 of the present invention. Detailed Implementation

[0043] It should be noted that the following detailed description is illustrative and intended to provide further explanation of the invention. Unless otherwise specified, all technical and scientific terms used in this invention have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains.

[0044] As introduced in the background section, existing infrared and visible light image fusion methods have shortcomings in terms of feature extraction, alignment accuracy, data dependency, fusion balance, and computational efficiency.

[0045] (1) Insufficient feature extraction: Traditional methods rely heavily on manually designed features (such as wavelet transform, sparse representation, etc.), which makes it difficult to fully capture the complementary information between infrared and visible light images, resulting in limited detail representation of the fused image.

[0046] (2) Inaccurate alignment: In multimodal image fusion, infrared images and visible light images often have geometric mismatch, scale differences or viewpoint shifts. Most existing methods align them through simple interpolation or registration, but misalignment is still likely to occur in complex scenes, thus affecting the accuracy of the fusion results.

[0047] (3) Strong dependence on supervised learning: Although some deep learning methods improve the fusion effect, they often rely on large-scale labeled data for supervised training. Due to the lack of high-quality labeled samples in practical applications, the generalization ability of these methods is insufficient, which limits their application scope.

[0048] (4) Poor balance of fusion results: Some algorithms tend to lose visible light details when enhancing infrared salient targets; while when emphasizing visible light texture, they weaken infrared thermal information, making it difficult for the fusion results to balance clarity and target salientity.

[0049] (5) High computational complexity: Some fusion methods based on multi-scale or complex deep networks have a large amount of computation, making it difficult to meet real-time requirements and limiting their application in embedded devices or edge computing scenarios.

[0050] To overcome the shortcomings of the prior art, in a typical embodiment of the present invention, such as... Figure 1 and Figure 2 As shown, this embodiment discloses a method for fusing infrared and visible light images by combining self-supervised feature alignment, specifically including the following steps:

[0051] S1. Acquire infrared and visible light images;

[0052] S2. Based on the hierarchical dual-stream heterogeneous model, feature extraction is performed on infrared and visible light images respectively, and appearance and geometric features of infrared and visible light images are generated based on the cross-modal projection model respectively;

[0053] S3. Based on the dual-space hierarchical self-supervised alignment model, the appearance features and geometric features are aligned to generate alignment features for infrared and visible light images respectively;

[0054] S4. Based on the spectrum-graph neural joint fusion model, the alignment features of infrared images and visible light images are fused to obtain fused features;

[0055] S5. Reconstruct and output the fused image based on the fusion features.

[0056] The following describes in detail the infrared and visible light image fusion method that combines self-supervised feature alignment with specific implementation methods.

[0057] S1. Acquire infrared and visible light images.

[0058] In this step, infrared and visible light images are first acquired, and then preprocessed. The specific process is as follows:

[0059] S11. Acquire infrared and visible light images.

[0060] In this step, the acquisition of infrared and visible light image pairs employs a dual-sensor synchronous acquisition system. The system comprises an infrared imaging unit and a visible light imaging unit. A trigger signal enables time-synchronized exposure of the two imaging units, acquiring spatiotemporally consistent infrared images of the same scene. With visible light images This results in the formation of spatiotemporally consistent cross-modal original image pairs.

[0061] S12. Image preprocessing to obtain preprocessed infrared and visible light images.

[0062] In this step, it is necessary to process the acquired infrared image. With visible light images Preprocessing is performed, including size alignment and intensity normalization, for infrared images. Temperature statistical feature extraction is also performed on visible light images. Texture enhancement processing is also performed.

[0063] S121. Size alignment.

[0064] In this step, cv2.resize is used to resize the infrared image. With visible light images The image is scaled proportionally to 512×512 (or the required size) using bilinear interpolation to achieve smooth scaling and avoid pixel mapping issues during image resizing. Bilinear interpolation can be specifically expressed as:

[0065] .

[0066] S122. Intensity normalization.

[0067] In this step, the infrared image is first processed. and visible light images Percentile cropping is performed, including the visible light image. use Cropping area, infrared image use The clipping interval is then mapped to a Min-Max area using the Min-Max method. .

[0068] Specifically, visible light images Percentile clipping can be represented as:

[0069] , ;

[0070] Infrared images Percentile clipping can be represented as:

[0071] , ;

[0072] Min-Max to Specifically, it can be expressed as:

[0073] .

[0074] S123. Generate gradient map.

[0075] In this step, a gradient map is generated by calculating the gradient magnitude of the Gaussian-smoothed image, specifically as follows:

[0076] ;

[0077] This step generates more modally consistent intermediate representations, aiding in multimodal registration. Gradient maps are not only used for structural enhancement in the preprocessing stage but also continuously participate in computation during subsequent feature alignment and fusion stages. By introducing gradient channels in the feature extraction stage (HDH module), the network can learn salient features of edges and contours early on; in the self-supervised alignment stage, gradient maps are used to construct spatial weighting factors. This approach makes cross-modal feature alignment more focused on structural boundaries; in the geometric alignment and fusion stage, gradient information is used to guide displacement field prediction and graph topological weight calculation, thereby maintaining geometric consistency and edge sharpness; in the image reconstruction stage, gradient consistency loss is used... Further enhancing the detail fidelity of the fused images. Multi-stage involvement of gradient features significantly improves the model's structure awareness and fusion quality.

[0078] S124. Targeted image enhancement.

[0079] To further improve the stability and saliency of infrared images, this step involves processing the infrared images. Temperature statistical enhancement (Temp-hist) is performed, generating additional thermally salient feature channels by calculating local means, variances, and quantiles; for visible light images... Adaptive histogram equalization (CLAHE) is applied to CLAHE to improve texture clarity.

[0080] The two images undergo preprocessing steps S121, S122, S123, and S124 to form an input tensor containing multiple channels, which serves as the final input. Specifically, it is represented as follows:

[0081]

[0082]

[0083] In the formula, and This represents infrared and visible light images after percentile cropping and normalization. and Represents the image gradient map; This indicates the temperature statistics layer.

[0084] S2. Based on the hierarchical dual-stream heterogeneous model, feature extraction is performed on infrared and visible light images respectively, and appearance and geometric features of infrared and visible light images are generated based on the cross-modal projection model respectively.

[0085] In this step, infrared images are first extracted using a hierarchical dual-stream heterogeneous model. With visible light images The underlying features are extracted, then deep feature extraction is performed on the extracted underlying features, and the obtained deep feature maps are generated into infrared images through a cross-modal projection model. appearance features and geometric features and visible light images appearance features and geometric features .

[0086] S21. Extract low-level features.

[0087] In this step, the hierarchical dual-stream heterogeneous model (HDH) includes three branches that enable cross-modal interaction: the infrared (IR) path, the visible light (VI) path, and the shared geometry (Geo) path. The IR path focuses on preserving the local intensity and shape boundaries of the thermal spectrum. It employs a lightweight multi-scale convolutional neural network (CNN), primarily using Depthwise-Separable Conv, to replace traditional CNNs for improved efficiency. Simultaneously, it introduces a Spectral Attention (Spectral-Attn) submodule to map local features to the frequency domain and decomposes them into high-frequency and low-frequency components using FFT. The high-frequency components enhance boundary and texture details, while the low-frequency components preserve the thermal intensity distribution. The visible light branch (VI-path) employs a hybrid CNN-Transformer, a complementary fusion structure of local convolution and global attention. This structure supports adaptive adjustment of features between modalities. It utilizes an early-CNN to extract local texture, while a small backend Transformer model models long-range color and brightness distributions to extract global texture and color distributions. A local residual fusion layer connects the outputs of the early-CNN and Transformer, avoiding modal feature fragmentation and effectively improving the cross-domain transferability of texture details. The shared geometry branch (Geo-path) captures common geometric structures (such as edges / contours) between the two modalities. It uses a shallow CNN and introduces a small, deformable learnable offset module. This module learns only micro-displacement deviations, with a limited offset range, effectively reducing computational cost. It achieves cross-modal space self-correction at the low-level feature level, outputting geometrically consistent joint features to provide initial alignment input for the FFN aligner. During training, Geo-path shares some parameters with the infrared and visible light branches to enhance geometric consistency.

[0088] Specifically, the visible light image after preprocessing in step S1 and temperature statistical layer Infrared images As the original image view, the visible light branch (VI-path) and the infrared branch (IR-path) of the hierarchical two-stream heterogeneous model (HDH) are respectively input, and the visible light image is extracted through the visible light branch (VI-path). The infrared image is extracted using the infrared branch IR-path to capture low-level features such as texture details and brightness distribution. The local thermal intensity, shape contour, and boundary features are obtained, and mode-independent geometric structure features are extracted through the shared geometric branch Geo-path. Visible light bottom-layer features are obtained through these three paths. Infrared bottom layer features and geometric shared features .

[0089] S22. Extract deep features.

[0090] In this step, firstly, the infrared bottom-layer features extracted in step S21 are... Visible light underlying features and geometric shared features As input, a first cross-modal projection module is used to perform modal space mapping and dimensionality adjustment on the low-level features. This first cross-modal projection module employs a lightweight neural network structure, consisting of a 1×1 Conv, BN+ReLU, and a 3×3 Depthwise Conv. The specific process of its modal space mapping and dimensionality adjustment of the low-level features can be represented as follows:

[0091]

[0092]

[0093]

[0094] In the formula, , Represents the infrared and visible light features mapped onto the appearance space. It represents the geometrically shared features mapped to the geometric space; the projection parameters W and b are learnable and used for feature normalization and spatial transformation.

[0095] Then, the infrared features, visible light features, and geometrically shared features are spliced ​​and fused, which can be specifically represented as:

[0096]

[0097] In the formula, This indicates the features after splicing and merging.

[0098] Subsequently, the spliced ​​and merged features As input, a deep encoder is used to extract deep features. This deep encoder consists of convolutional residual blocks, a cross-modal attention module, and a small Transformer, performing high-level modeling of the input features, specifically represented as follows:

[0099]

[0100] In the formula, Indicates the output infrared deep semantic features Or visible light deep semantic features .

[0101] S23. Generate appearance features and geometric features.

[0102] In this step, a second cross-modal projection model is used to generate appearance and geometric features from infrared deep semantic features and visible light deep semantic features. This second cross-modal projection model comprises a dual-branch structure with appearance and geometry branches. The appearance branch includes 1×1 convolutions and MLPs to highlight texture and detail representations; the geometry branch includes 3×3 convolutions, deformable convolutions, and graph convolutional layers to maintain consistency in edges, contours, and spatial topology. Through the projection mechanism of the second cross-modal projection model, the decoupling of appearance and geometric features is effectively achieved, providing independent and complementary feature representations for subsequent cross-modal alignment and fusion.

[0103] Specifically, infrared deep semantic features and visible light deep semantic features Infrared images are generated by mapping the images to the appearance subspace and the geometry subspace using a second transmodal projection model. appearance features and geometric features and visible light images appearance features and geometric features .

[0104] S3. Based on the dual-space hierarchical self-supervised alignment model, the appearance features and geometric features are aligned to generate alignment features for infrared and visible light images, respectively.

[0105] In this step, a self-supervised learning framework based on the dual-space hierarchical self-supervised alignment model is used for feature alignment. A dual-space hierarchical self-supervised feature alignment framework (HDSA) and a dynamic weight multi-task joint optimization mechanism are proposed to achieve consistent representation by minimizing cross-modal feature differences.

[0106] S31. Feature standardization and embedding.

[0107] In this step, the appearance features are... , and geometric features , L2 normalization is performed, and the features are embedded into the alignment space through a linear projection layer. Specifically, the appearance feature mapping can be represented as:

[0108] ;

[0109] Geometric feature mapping can be specifically represented as:

[0110] ;

[0111] In the formula, This represents the appearance branch mapping function; This represents the geometric branch mapping function.

[0112] S32. Construct positive and negative sample pairs.

[0113] Infrared features paired with visible light features from the same location are considered positive sample pairs, while cross-modal features from different locations and cross-modal features from other images are considered negative samples. A memory queue (refer to MoCo) can be used to store historical negative samples.

[0114] S33. Appearance alignment.

[0115] This step proposes an InfoNCE based on local patches and with spatial weights. During appearance spatial alignment, local patches are used as positive samples, combined with learnable offsets. For spatial matching, the loss function is specifically expressed as:

[0116]

[0117] In the formula, The significance weights are calculated based on the gradient plot. This indicates an adaptive temperature, typically ranging from 0.05 to 0.2. Represents the set of negative samples; Indicates the location of the infrared appearance feature map. Appearance characteristics; This indicates the spatial location index, i.e., the center of the patch; This indicates a negative sample.

[0118] S34. Geometric Alignment.

[0119] In this step, spatial correction for infrared and visible light is performed using a geometric alignment branch. This geometric alignment branch employs an improved Feature Flow Network (FFN), extending the original optical flow FFN to include a local correlation volume building layer, a deformable convolutional prediction layer, a confidence guidance layer, and a cycle consistency constraint module. This network takes the geometric features of infrared and visible light as input and predicts cross-modal micro-displacement fields. and confidence plot Achieve spatial alignment of local geometric structures.

[0120] Specifically, FFN uses a lightweight U-Net (encoder-decoder) + deformable convolution to predict the offset of each spatial location. (Shape 2×H×W), achieving differentiable geometric registration at the feature level; introducing a self-cyclic consistency constraint to ensure... This avoids learning ineffective distortions.

[0121] The geometric alignment loss is specifically expressed as:

[0122]

[0123] In the formula, Indicates the position of the infrared geometric feature map eigenvectors; This represents the predicted displacement in the visible light geometric feature map. The corresponding feature vector; This represents the regularization weight parameter, with a range of... .

[0124] S35. Generate alignment features.

[0125] In this step, the displacement field output by the Feature Flow Network (FFN) It is used not only for geometric feature alignment but also to guide the spatial alignment of appearance features. Specifically, a differentiable spatial transformation module (STN) is used to align infrared appearance features. According to displacement field Resampling is performed to map the infrared and visible light feature coordinates, thereby achieving spatial alignment of the infrared and visible light appearance features, and outputting the aligned and optimized appearance features. By constructing positive and negative sample pairs using S32 and minimizing cross-modal appearance differences using an appearance loss function based on S33, a self-supervised contrastive learning framework is employed to output appearance consistency features. Geometric alignment is predicted using FFN. And warp infrared geometric features This aligns infrared and visible light in spatial coordinates, yielding geometrically aligned features. The spatial difference is minimized using the geometric spatial alignment loss function of S34, outputting the aligned and optimized geometric features. In bidirectional aligned FFN, it is possible to access from the reverse ( )predict Obtain the aligned and optimized geometric features In this embodiment, the geometric alignment feature using the vi coordinate system is selected. , It is only used for cyclic consistency constraints and does not enter S4 feature fusion.

[0126] S36. The system weights are automatically adjusted based on a dynamic weight multi-task joint optimization mechanism.

[0127] In this step, the dual-space hierarchical self-supervised alignment model uses a multi-task loss function to adaptively weight each loss term. Based on the loss value and training state, the weights of each loss term are dynamically generated using uncertainty estimation, gradient normalization, or a meta-controller mechanism, which effectively improves the stability of training and the overall quality of the fused image.

[0128] Specifically, the dynamic weighted multi-task joint optimization mechanism includes reconstruction loss, perceptual loss, spatially perceptual InfoNCE loss, geometrical alignment loss, structural similarity loss, and gradient preservation loss. Its overall training objective is specifically expressed as follows:

[0129]

[0130] In the formula, Indicates the losses incurred during reconstruction; Indicates perceived loss; This represents the spatial perception InfoNCE loss; Indicates the geometric alignment loss; Represents structural similarity loss; This represents the gradient preservation loss.

[0131] First, set the weights for each loss. The initial values ​​are as follows: .

[0132] Then, the weights of each loss are adjusted using three dynamic mechanisms: uncertainty-weighted adjustment, gradient-balanced dynamic adjustment (GradNorm improvement), and meta-learning controller. Dynamic adjustments are made. Specifically, a meta-learning controller is preferentially used to dynamically adjust the weights through a lightweight control network. Input the current loss value and downstream performance metrics, and output the optimal loss weight. Furthermore, the downstream detection / segmentation performance is maximized through meta-gradient optimization.

[0133] In the training objective representation:

[0134] Reconstruction losses Represented as:

[0135]

[0136] In the formula, Represents a fused image; A reference mode (or the reconstruction target of the input mode) can be selected. Indicates the number of pixels.

[0137] Perceived loss Represented as:

[0138]

[0139] In the formula, Indicates the perceptron in the first... Feature maps extracted from layers; The number of floors.

[0140] Spatial perception InfoNCE loss Represented as:

[0141]

[0142] In the formula, Indicates the position of infrared and visible light Appearance characteristics; Indicates the position of visible light Appearance characteristics; This indicates the offset predicted by Feature-Flow-Net; Indicates weights based on significance or marginality; Indicates temperature parameter; This represents the set of negative samples.

[0143] Geometric alignment loss Ensuring geometric consistency is represented as:

[0144]

[0145] In the formula, Indicates geometric feature alignment error; This represents a cycle consistency constraint.

[0146] Structural similarity loss Represented as:

[0147]

[0148] In the formula, Measure the consistency of structure, brightness, and contrast.

[0149] Gradient Preservation Loss To maintain sharp edges, it is represented as:

[0150]

[0151] In the formula, This represents an image gradient operator (such as Sobel or Scharr).

[0152] S4. Based on the spectrum-graph neural joint fusion model, the alignment features of infrared images and visible light images are fused to obtain fused features.

[0153] S41. Frequency domain decomposition.

[0154] In this step, the S35 is aligned with the optimized appearance features. and geometric features Perform frequency domain decomposition (FFT) and use low-pass / high-pass masks to divide the spectrum into low-frequency and high-frequency components.

[0155] First, optimize the appearance features after alignment. and geometric features Channel dimensionality reduction can be specifically represented as:

[0156]

[0157]

[0158] In the formula, This represents the appearance characteristics after dimensionality reduction; This represents the geometric features after dimensionality reduction.

[0159] Then, a frequency domain transformation is performed on the reduced appearance and geometric features. Specifically, a two-dimensional fast Fourier transform is performed on each channel c, as follows:

[0160]

[0161]

[0162] In the formula, (u,v) are frequency coordinates; For FFT transform operator, The amplitude spectrum A and the phase spectrum P are obtained.

[0163] Subsequently, spectral mask segmentation is performed. Specifically, the mask is defined based on the radius r of the frequency coordinate distance from the center:

[0164]

[0165]

[0166] In the formula, Indicates a low-frequency mask; Represents a high-frequency mask; It is the frequency domain boundary threshold, which is usually taken as 1 / 4 to 1 / 3 of the length of the image diagonal.

[0167] Then, the low-frequency and high-frequency components are separated and represented as follows:

[0168]

[0169]

[0170]

[0171]

[0172] In the formula, Indicates the low-frequency components of the appearance features; Represents the high-frequency components of appearance features; Represents the low-frequency components of geometric features; It represents the high-frequency components of geometric features.

[0173] S42. Feature fusion.

[0174] In this step, the spectrogram-graph neural joint fusion model adopts a spectrogram-graph neural dual-branch structure, with low-frequency components as the appearance features. and low-frequency components of geometric features To construct a spatial graph G (nodes are 8×8 patches, edges are built based on spatial proximity and spectral similarity), a multi-layer graph attention network (GAT) (3 layers, 128 hidden nodes) is used on this graph to extract topologically consistent structures, resulting in structure-enhanced low-frequency components. .

[0175] High-frequency components of appearance characteristics and geometric high-frequency components Enhancement of appearance texture and geometric edges is performed using Spectral-Attn, and the spectral energy intensity is calculated and expressed as:

[0176]

[0177] Then, weights are assigned using spectral attention:

[0178]

[0179]

[0180] The two types of enhanced high frequencies are obtained, including the appearance texture enhancement high frequency component. and geometric edge enhancement of high frequency components , respectively represented as:

[0181]

[0182]

[0183] Finally, enhanced cross-modal fusion is performed, including low-frequency fusion and high-frequency fusion. Low-frequency fusion is represented as follows:

[0184]

[0185] In the formula, Indicates low-frequency fusion characteristics; and Determined adaptively based on modal confidence.

[0186] High-frequency fusion is represented as:

[0187]

[0188] In the formula, This indicates high-frequency fusion characteristics.

[0189] Subsequently, an inverse transform was performed to obtain the low-frequency enhancement features. and high frequency enhancement features , respectively represented as:

[0190]

[0191]

[0192] Then the low-frequency enhancement features generated by the bi-branch High-frequency enhancement features In the spatial domain, cross-attention is used for fusion to obtain fused features. .

[0193] Specifically, high-frequency enhancement features As input for the query, Query(Q), Low-frequency enhancement features Input Key(K) and Value(V) as keys and values. , , , The final fused feature, obtained by calculating the cross-modal attention weights (where is the feature dimension), is represented as follows:

[0194] .

[0195] S5. Reconstruct and output the fused image based on the fusion features.

[0196] In this step, Conv Decoder is used to extract the enhanced fusion features. Restoring to a high-quality image can be specifically represented as:

[0197]

[0198] In the formula, This indicates the enhanced fusion features.

[0199] At the same time, pixel-level loss is used to ensure reconstruction quality, specifically expressed as follows:

[0200]

[0201] In the formula, For reference only.

[0202] Finally, Output the final fused image, or provide feature vectors to downstream tasks.

[0203] Example 2

[0204] In a typical embodiment of the present invention, such as Figure 3 As shown, this embodiment discloses an infrared and visible light image fusion system combining self-supervised feature alignment, including:

[0205] The image acquisition module is configured to acquire infrared images and visible light images;

[0206] The feature extraction module is configured to: extract features from infrared and visible light images based on a hierarchical dual-stream heterogeneous model, and generate appearance and geometric features of infrared and visible light images based on a cross-modal projection model.

[0207] The feature alignment module is configured to align appearance features with geometric features based on a dual-space hierarchical self-supervised alignment model, generating alignment features for infrared and visible light images respectively.

[0208] The feature fusion module is configured to: fuse the alignment features of infrared and visible light images based on a spectrum-graph neural joint fusion model to obtain fused features;

[0209] The reconstruction and output module is configured to: reconstruct and output the fused image based on the fusion features.

[0210] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for fusing infrared and visible light images by combining self-supervised feature alignment, characterized in that, include: Acquire infrared and visible light images; Feature extraction is performed on infrared and visible light images based on a hierarchical dual-stream heterogeneous model, and appearance and geometric features of infrared and visible light images are generated based on a cross-modal projection model. The hierarchical dual-stream heterogeneous model comprises three branches: an infrared branch, a visible light branch, and a shared geometry branch. The infrared branch employs a lightweight multi-scale convolutional neural network with a spectral attention submodule to preserve local intensity and shape boundaries of the thermal spectrum. The visible light branch uses a hybrid CNN-Transformer, leveraging a convolutional neural network to extract local textures while a small backend Transformer extracts global textures and color distributions. The shared geometry branch uses a shallow convolutional neural network with a small learnable offset module to capture the shared geometry between the two modalities. Based on a dual-space hierarchical self-supervised alignment model, appearance features and geometric features are aligned to generate alignment features for infrared and visible light images, respectively. Specifically, these features include: Appearance characteristics in infrared and visible light , and geometric features , L2 normalization is performed, and the feature maps are embedded into the alignment space through a linear projection layer; where the appearance feature maps are represented as: ； Geometric feature mapping is represented as: ； In the formula, This represents the appearance branch mapping function; Represents the geometric branch mapping function; Appearance spatial alignment is performed based on InfoNCE with local patches and spatial weights; during appearance spatial alignment, local patches are used as positive samples, combined with learnable offsets. Spatial matching is performed, and the loss function is: In the formula, The significance weights are calculated based on the gradient plot. Indicates adaptive temperature; Represents the set of negative samples; Indicates the location of the infrared appearance feature map. Appearance characteristics; This represents the spatial location index, which is the center of the patch; Indicates a negative sample; The geometric alignment branch employs an improved feature flow network (FFN) to predict the transmodal displacement field. With confidence plot Spatial alignment of geometric features for infrared and visible light; The geometric alignment loss is specifically expressed as: In the formula, Indicates the position of the infrared geometric feature map eigenvectors; This represents the predicted displacement in the visible light geometric feature map. The corresponding feature vector; This represents the regularization weight parameter; infrared appearance features According to displacement field Resampling is performed to map the infrared and visible light features to the visible light feature coordinate system, so that the infrared and visible light features are spatially aligned. The alignment features of infrared and visible light images are fused using a spectrum-graph neural joint fusion model to obtain fused features. The model employs a spectrum-graph neural dual-branch structure, where low-frequency components are used to construct a spatial graph. A multi-layer graph attention network is then used to extract topologically consistent structures from the spatial graph, resulting in structure-enhanced low-frequency components that are fused across modally to obtain low-frequency fused features. High-frequency components are enhanced for appearance texture and geometric edges through spectrum attention, resulting in high-frequency components with enhanced appearance texture and geometric edges, which are then fused across modally to obtain high-frequency fused features. The low-frequency and high-frequency fused features are then inversely transformed to obtain low-frequency enhanced features and high-frequency enhanced features, which are then fused in the spatial domain through cross-attention to obtain the final fused features. The fused image is reconstructed and output based on the fusion features.

2. The infrared and visible light image fusion method combining self-supervised feature alignment as described in claim 1, characterized in that, After acquiring infrared and visible light images, the acquired images are preprocessed, including size alignment and intensity normalization. Temperature statistical feature extraction is also performed on the infrared images, and texture enhancement is performed on the visible light images.

3. The infrared and visible light image fusion method combining self-supervised feature alignment as described in claim 1, characterized in that, The generation process of appearance features and geometric features specifically includes: Using preprocessed infrared and visible light images as input, low-level features are extracted based on a hierarchical dual-stream heterogeneous model to obtain the low-level features of the infrared image and the visible light image. Using the extracted low-level features of infrared images and visible light images as input, the low-level features are modally mapped and dimensionally adjusted based on the first cross-modal projection module to obtain infrared deep semantic features and visible light deep semantic features. Based on the second cross-modal projection model, infrared deep semantic features and visible light deep semantic features are mapped to appearance subspace and geometric subspace respectively, generating appearance and geometric features of infrared images, as well as appearance and geometric features of visible light images.

4. The infrared and visible light image fusion method combining self-supervised feature alignment as described in claim 1, characterized in that, The dual-space hierarchical self-supervised alignment model adopts a dual-space hierarchical self-supervised feature alignment framework and a dynamic weight multi-task joint optimization mechanism to achieve consistent representation by minimizing cross-modal feature differences.

5. The infrared and visible light image fusion method combining self-supervised feature alignment as described in claim 1, characterized in that, The dynamic weighted multi-task joint optimization mechanism includes reconstruction loss, perception loss, spatially aware InfoNCE loss, geometrical alignment loss, structural similarity loss, and gradient preservation loss.

6. The infrared and visible light image fusion method combining self-supervised feature alignment as described in claim 1, characterized in that, Three dynamic mechanisms—uncertainty-based weighting, gradient-balance-based dynamic adjustment, and meta-learning controller—are used to dynamically adjust the various loss weights in the dynamic weight multi-task joint optimization mechanism.

7. An infrared and visible light image fusion system combining self-supervised feature alignment, employing the infrared and visible light image fusion method combining self-supervised feature alignment as described in claim 1, characterized in that... include: The image acquisition module is configured to acquire infrared images and visible light images; The feature extraction module is configured to: extract features from infrared and visible light images based on a hierarchical dual-stream heterogeneous model, and generate appearance and geometric features of infrared and visible light images based on a cross-modal projection model. The feature alignment module is configured to align appearance features with geometric features based on a dual-space hierarchical self-supervised alignment model, generating alignment features for infrared and visible light images respectively. The feature fusion module is configured to: fuse the alignment features of infrared and visible light images based on a spectrum-graph neural joint fusion model to obtain fused features; The reconstruction and output module is configured to: reconstruct and output the fused image based on the fusion features.

Images