Task-driven equivariant consistency image fusion method and system

Abstract

This invention provides a task-driven, equivariant-consistent image fusion method and system. It performs feature decomposition on the input infrared and visible light images to obtain their respective basic and detailed features. In the subsequent fusion process, geometric transformations that satisfy equivariant-consistent constraints are applied to ensure the stability of feature transformations. The fusion model is semantically constrained using the supervision information of the semantic segmentation task. By collaboratively optimizing the equivariant-consistent constraints and the semantic loss through the loss function, the invention effectively balances the preservation of local details with the enhancement of global semantic information while maintaining robustness to geometric transformations. This produces high-quality fused images and improves the performance of downstream high-level visual tasks.

Description

Technical Field

[0001] The present invention relates to the field of infrared and visible light fusion technology, and in particular to a task-driven equivariant consistency image fusion method and system for the fusion of infrared and visible light images. The method aims to improve image quality and semantic information by fusing images of the two modalities to support advanced visual tasks. Background Art

[0002] Infrared and visible light image fusion technology aims to integrate information captured by different sensors to produce a single image that contains more comprehensive scene information. This is crucial for improving situational awareness in complex environments, such as low light and smoke. Fusion leverages the infrared image's ability to detect targets in adverse conditions while combining it with the rich texture and color details provided by visible light images. Currently, the use of deep neural network models to handle image fusion tasks has become a research hotspot.

[0003] For example, Chinese patent application number 202411048544.5 discloses a method, device, electronic device, and storage medium for fusing visible light imaging and infrared imaging. This method directly extracts semantic information from visible light imaging, presets task information, and obtains task prompts through a large language model. The task prompts are then used as input to predict infrared images, and the target fused image is obtained by fusing the current frame visible light image with the predicted infrared image. However, this task-guided method only focuses on global semantic information and lacks detailed modeling of local structures and details. As a result, while the fused image improves the performance of downstream tasks, it often sacrifices some of the details and structural information of the original image. In addition, relying on task prompts makes it difficult to fully explore and utilize multi-level and multi-scale semantic features, and the fusion results have limited adaptability to complex scenes.

[0004] Therefore, the existing infrared and visible light image fusion methods still have obvious shortcomings in terms of structure restoration, detail preservation, semantic enhancement, and adaptability to downstream tasks. How to more effectively combine transformation consistency decomposition with task guidance mechanism and realize active perception and enhancement of multi-level information is a technical problem that needs to be solved urgently in this field. Summary of the Invention

[0005] To address these shortcomings, the present invention aims to provide a task-driven equivariant consistency image fusion method and system. This method, by incorporating a divide-and-conquer approach, hierarchically models structural information (basic features), detail information (detail features), and semantic information during the fusion process. Multiple loss constraints are then constructed. Feature decomposition loss and structural similarity loss effectively enhance the fused image's structural restoration and detail preservation capabilities, while semantic loss enhances support for downstream tasks. This overcomes the drawbacks of existing loss functions, which suffer from a single design and difficulty balancing multi-level information expression. This addresses the technical issue of existing fusion technologies, which struggle to maintain robustness to geometric transformations while effectively balancing texture details and global semantic information.

[0006] To achieve the above object, the technical solution of the present invention is achieved as follows:

[0007] In a first aspect, the present invention provides a task-driven equivariant consistency image fusion method, the method comprising the following steps:

[0008] Acquire infrared images and visible light images;

[0009] Constructing a fusion model, wherein the fusion model includes an encoder, a decoder, and a geometric transformation operation;

[0010] Decomposing the infrared image and the visible light image in an encoder to obtain basic features and detailed features of each;

[0011] The basic features of the infrared image and the basic features of the visible light image are added element by element to obtain a unified basic feature; the detail features of the infrared image and the detail features of the visible light image are added element by element to obtain a unified detail feature;

[0012] Applying geometric transformation operations to the unified basic features and the unified detail features to obtain transformed basic features and transformed detail features respectively;

[0013] At the same time, the unified basic features and the unified detail features are input into the decoder to obtain a fused image;

[0014] Applying a geometric transformation operation to the fused image to obtain a transformed fused image, and re-inputting the transformed fused image into the encoder and the decoder to obtain a reconstructed fused image;

[0015] Among them, the geometric transformation operation satisfies the equivariant consistency constraint;

[0016] During the training process, the fused image is simultaneously processed by the semantic segmentation model to obtain a predicted segmentation map, and the semantic loss between the predicted segmentation map and the corresponding true segmentation label map is calculated; the semantic loss participates in the total loss calculation and backpropagation of the entire model, guiding the parameter update of the fusion model, and the total loss is obtained by weighting the fusion loss and the semantic loss;

[0017] The trained fusion model is used to fuse infrared images and visible light images.

[0018] Furthermore, the fusion loss includes structural similarity loss, feature decomposition loss, intensity loss and gradient loss.

[0019] Furthermore, the structural similarity loss is a weighted sum of the similarity between the transformed basic features and the reconstructed basic features, the similarity between the transformed detail features and the reconstructed detail features, and the similarity between the fused images before and after reconstruction;

[0020] The feature decomposition loss is obtained by the correlation coefficient between the basic features of the infrared image and the visible light image, the correlation coefficient between the detail features, the correlation coefficient between the transformed detail features and the reconstructed detail features, and the correlation coefficient between the transformed basic features and the reconstructed basic features;

[0021] The intensity loss is used to calculate the difference in pixel intensity between the transformed fused image and the reconstructed fused image;

[0022] The gradient loss is used to calculate the gradient loss of the transformed fused image and the reconstructed fused image.

[0023] Furthermore, the semantic loss is a cross entropy loss.

[0024] Furthermore, the encoder includes a shared feature encoder, a basic feature encoder, and a detail feature encoder. The infrared image and the visible light image are respectively input into the shared feature encoder and then processed by the basic feature encoder and the detail feature encoder to obtain their respective basic features and detail features.

[0025] The shared feature encoder is implemented using a Restormer module, the basic feature encoder is implemented using a Transformer module, and the detail feature encoder is implemented using a convolutional neural network module.

[0026] Furthermore, the geometric transformation operation includes a translation operation, a rotation operation, and a flip operation; and is set to perform a translation of a random distance, a rotation of 10 degrees, a left mirror flip, and a right mirror flip once each;

[0027] The result of splicing the transformed basic features and the transformed detail features is equivalent to the result of splicing the unified basic features and the unified detail features and then applying the same transformation; and the transformed fused image and the reconstructed fused image remain consistent.

[0028] In a second aspect, the present invention provides a task-driven equivariant consistency image fusion system, the system comprising:

[0029] Image acquisition module: used to acquire paired infrared images and visible light images;

[0030] Fusion model: used for fusing infrared and visible light images; the fusion model includes an encoder, an equivariant consistency constraint module, and a decoder; the encoder is used to perform feature decomposition on the input infrared and visible light images to obtain their respective basic features and detail features; the equivariant consistency constraint module is configured to impose equivariant consistency constraints based on geometric transformation operations;

[0031] a semantic guidance module, comprising a semantic segmentation model associated with a preset downstream visual task, configured to generate a semantic constraint signal based on supervision information of the downstream visual task;

[0032] The decoder receives constraints from the equivariant consistency constraint module and the semantic guidance module, is used to fuse the basic features and detail features input into the decoder, and decodes to generate a fused image.

[0033] The present invention also protects a computer-readable storage medium having a computer program stored thereon, which can implement the steps of the method when executed by a processor.

[0034] Compared with the prior art, the present invention has the following beneficial effects:

[0035] (1) This invention organically combines equivariant consistency constraints and downstream segmentation semantic guidance into the fusion model for the first time, and constrains the equivariant detail features and basic features, as well as the reconstructed detail features and basic features, through feature decomposition loss, achieving a synergistic improvement in the fused image in terms of geometric transformation robustness, detail preservation, and semantic enhancement. This overcomes the problem that existing technologies, in practical applications, often only focus on global consistency and lack effective fusion of local high-frequency details and semantic information, resulting in deficiencies in the fused image in terms of detail representation and semantic expression.

[0036] (2) The present invention introduces equivariant consistency constraints, which not only improves the robustness of the fused image to geometric transformations such as rotation, translation, and folding, but also preserves local detail information. The equivariant consistency constraints can preserve more detailed information of the image.

[0037] (3) The present invention performs well in downstream tasks and has higher application potential. By introducing semantic segmentation tasks and corresponding semantic loss terms, the fusion model can actively enhance the semantic feature expression that is beneficial to subsequent visual tasks. Compared with existing methods that only rely on pixel-level or perceptual-level losses, the generated fusion image shows better performance in downstream tasks such as target detection and semantic segmentation. The specific effects are as follows: Figure 7 and Figure 8 shown.

[0038] (4) The present invention maximizes the preservation of the transformed basic features and detailed features, as well as the reconstructed basic features and detailed features, through feature decomposition loss, thereby achieving a balance in the overall performance of the model. In addition, by introducing a reasonable semantic loss term and adjusting its weight in the total loss, an optimal compromise between detail clarity and semantic richness is achieved in the fused image. Experiments show that when the semantic loss weight is set to 0.2, all evaluation indicators reach the optimal state. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] The accompanying drawings, which constitute a part of the present invention, are used to provide a further understanding of the present invention. The exemplary embodiments and descriptions thereof are used to explain the present invention and do not constitute improper limitations on the present invention.

[0040] Figure 1 A schematic diagram of the network structure of an embodiment of the task-driven equivariant consistency image fusion method of the present invention;

[0041] Figure 2 A schematic diagram showing a visual comparison of basic features and detail features with and without equivariant consistency constraints in an embodiment of the present invention;

[0042] Figure 3 This is a comparison chart of the experimental results of the method of the present invention and the existing method on the M3FD daytime dataset;

[0043] Figure 4 This is a comparison chart of the experimental results of the method of the present invention and the existing method on the M3FD night dataset;

[0044] Figure 5 This is a comparison chart of the experimental results of the method of the present invention and the existing method on the TNO dataset;

[0045] Figure 6 This is a comparison chart of the experimental results of the method of the present invention and the existing method on the DroneVehicle dataset;

[0046] Figure 7 This is a performance comparison chart of the method of Example 1 of the present invention and the existing method when applied to downstream target detection tasks;

[0047] Figure 8This is a performance comparison chart of the method in Example 1 of the present invention and the existing method when applied to downstream semantic segmentation tasks. DETAILED DESCRIPTION

[0048] It should be noted that, in the absence of conflict, the embodiments of the present invention and the features in the embodiments may be combined with each other.

[0049] This paper is based on a task-driven equivariant consistency image fusion method, which uses the supervised information of semantic segmentation to guide the fusion process, so that the generated fused image contains richer and more accurate semantic information. It aims to generate high-quality fused images that not only retain local texture details but are also rich in global semantic information and robust to geometric transformations, so as to improve the performance of downstream advanced vision tasks.

[0050] The equivariant image fusion paradigm solves the problem that existing fusion methods are sensitive to geometric transformations (such as translation, rotation, and folding) and are prone to losing spatial details. This paper introduces the equivariant consistency principle. Its core idea is to ensure that the fusion process F is equivariant to the transformations in the transformation group, that is, to satisfy The property of , I represents the input image. This ensures that applying the transformation to the input image before fusion is equivalent to first fusion and then applying the same transformation to the result.

[0051] Equivariant consistency constraint: This invention constructs a structural similarity loss function to constrain the consistency between the transformed fused image and the transformed detail features, the transformed basic features and the reconstructed fused image and the reconstructed detail features, the reconstructed basic features, thereby introducing a strong prior mechanism to enable the model to maintain good robustness in the face of geometric transformations such as rotation, translation, and folding.

[0052] Apply a preset geometric transformation operation to the unified basic features and the unified detail features to obtain the transformed basic features and detail features. The result of splicing the transformed basic features and the transformed detail features is equivalent to the result of splicing the unified basic features and the unified detail features and then applying the same transformation; the definition is as follows:

[0053]

[0054] in, Represents the translation, rotation and flipping operations applied to the image; Represents unified detail features, Represents the unified basic characteristics, Indicates the fusion process.

[0055] Semantic segmentation guidance: In response to the shortcomings of existing image fusion models in semantic information extraction, which in turn affects their applicability in downstream visual tasks, the fusion model of the present invention is guided by a semantic segmentation model to guide the fusion process to focus more on the retention and enhancement of semantic features. A joint training method of the semantic segmentation model and the fusion model is adopted to achieve effective collaboration between the two through shared features and multi-task optimization, and determine the optimal collaboration strategy between the fusion model and the semantic segmentation model. A semantic loss term is introduced, and its weight is regulated to achieve a good balance between image clarity and semantic expression ability, thereby significantly improving the application performance of the fused image in downstream tasks such as target detection and semantic segmentation. After the training is completed, the semantic segmentation model is only used as an auxiliary module during training to guide the fusion model to learn more accurate semantic features. It is no longer retained in the inference stage, thereby ensuring the inference efficiency of the model.

[0056] The encoder and decoder structures in the present invention can be implemented using conventional technologies in the art.

[0057] In order to make the purpose, technical solutions and advantages of the present invention more clear, the technical solutions of the present invention will be described in detail below in conjunction with specific embodiments.

[0058] Example 1:

[0059] This embodiment is based on a task-driven equivariant consistency image fusion method, including the following steps:

[0060] Acquire infrared images and visible light images;

[0061] Construct a fusion model, which includes an encoder, a decoder, and geometric transformation operations; the overall framework of the fusion model is shown in Figure 1 ,

[0062] The encoder is used to receive paired infrared images and visible light images and output their respective basic features and detail features. It consists of three main parts:

[0063] Shared feature encoder: The shared feature encoder in this embodiment is implemented using the Restormer module, whose main function is to extract and encode features of the input infrared and visible light images, and can effectively capture the global and local features in the image.

[0064] Basic feature encoder: The Transformer module is used to further process the shallow features and extract the low-frequency basic features. The basic features extracted from the infrared image are recorded as , the basic features extracted from the visible light image are recorded as The Transformer module is good at capturing global structural information and helps to ensure semantic consistency.

[0065] Detail feature encoder: The convolutional neural network module is used to process deep features and extract high-frequency detail features. The detail features extracted from the infrared image are recorded as , the detail features extracted from the visible light image are recorded as .

[0066] The basic features of infrared images Basic features of visible light images Add the elements to get a unified basic feature . The detailed features of the infrared image Detailed features of visible light images Add elements to obtain unified detail features .

[0067] Basic characteristics of unity and unified detail features Apply geometric transformation operations separately , that is, translate, rotate, and flip once each to obtain the transformed basic features and the transformed detail features .

[0068] At the same time, the unified basic features and unified detail features are input into the decoder to obtain a fused image ;

[0069] For fused images Apply the same geometric transformation operation , and get the transformed fused image .

[0070] The transformed fused image Re-input into the encoder to obtain the reconstructed basic features and the reconstructed detail features . Then reconstruct the basic features and the reconstructed detail features Input decoder to generate reconstructed fused image .

[0071] At the same time, the fused image Input into the semantic segmentation model to obtain the predicted segmentation map , with segmentation labels on it. In this embodiment, the semantic segmentation model uses the BiSeNet semantic segmentation network. The main feature of BiSeNet is that it can take into account both the spatial details and global semantic information of the segmentation results through parallel spatial path and context path structures, thereby achieving accurate segmentation of different semantic regions in the image. The specific formula for the semantic segmentation model processing can be expressed as:

[0072]

[0073] Compute predicted segmentation map And the corresponding real segmentation label map Semantic loss between , the true segmentation labels come from the annotations in the training dataset.

[0074] The semantic loss is involved in the total loss calculation and back propagation of the entire model, thereby guiding the parameter update of the fusion model, so that the fusion image generated by the fusion model not only has good visual effects, but also is conducive to subsequent semantic segmentation tasks.

[0075] The total loss function of this embodiment By fusion loss and semantic loss Weighted composition,aims to optimize both image fidelity and semantic consistency,simultaneously.

[0076] The fusion loss includes structural similarity loss, feature decomposition loss, strength and gradient loss:

[0077] Structural similarity loss : Based on the SSIM metric, it is used to quantify and ensure the preservation of image information after transformation and reconstruction. It mainly calculates the similarity between the following three:

[0078]

[0079] in, Represents the basic feature visualization image after transformation, Represents the reconstructed basic feature visualization image, Represents the transformed detail feature visualization image, Represents the reconstructed detail feature visualization image, represents the transformed fused image, represents the reconstructed fused image; Represents the similarity between the basic features before and after the transformation; Represents the similarity between the detail features before and after the transformation; Represents the similarity between the fused images before and after reconstruction; is a weight parameter used to control the balance between the fusion image loss and the decomposition image loss. In this embodiment, it is set to 0.1.

[0080] Eigendecomposition loss : It is used to ensure that basic features and detailed features are effectively separated and maintain consistency. It calculates the correlation between the original decomposition features (the decomposition features include basic features and detailed features) and the transformed and reconstructed decomposition features to ensure that the basic features and detailed features can still be preserved after undergoing equivariant consistency and semantic tasks. It is expressed as:

[0081]

[0082] in, is the correlation coefficient operator, Represents the detailed features of the infrared image, Represents the detail features of visible light images, Represents the basic features of infrared images, Represents the basic features of visible light images, represents the basic features after transformation, Represents the detailed features after transformation, represents the basic features after reconstruction, Represents the detailed features after reconstruction; is a constant. In order to prevent the denominator from being 0, it is set here is 0.01.

[0083] Strength and Gradient Loss : Strength loss Calculate the difference in pixel intensity between the transformed fused image and the reconstructed fused image to maintain the overall brightness and contrast information; gradient loss Calculate the gradient loss of the transformed fused image and the reconstructed fused image to preserve edge and texture details; the intensity and gradient loss are expressed as:

[0084]

[0085] Where λ is a parameter used to control the weight of gradient loss.

[0086] Semantic loss : Use the cross entropy loss function to calculate the predicted segmentation map output by the semantic segmentation model And the corresponding real segmentation label map The semantic loss is used to improve the semantic expression ability of the fused image and its adaptability to downstream tasks. The specific formula is:

[0087]

[0088] in, represents the cross entropy function.

[0089] Therefore, the total loss can be expressed as:

[0090] Total loss:

[0091] in, is the fusion loss, is the semantic loss, It is a hyperparameter used to balance the importance between fusion fidelity and semantic information enhancement. Set to 0.2.

[0092] To validate the effectiveness and superiority of the method described in this embodiment, the performance of the method was compared with several representative existing fusion methods (e.g., SwinFusion, SDNet, LRRNet, SeAFusion, EMMA, DDFM, and CDDFuse) on the publicly available M3FD and RoadScene infrared and visible light image datasets using commonly used objective evaluation metrics in the field, including information entropy (EN), standard deviation (SD), spatial frequency (SF), sum of correlation differences (SCD), visual information fidelity (VIF), and gradient-based fusion performance index (Qabf). The objective evaluation results, shown in Tables 1 and 2, clearly demonstrate that the method described in this embodiment exhibits significant advantages across multiple key metrics: achieving the highest values among all compared methods in each metric. This indicates that the fused images generated by the method described in this embodiment are richer in information, have higher contrast, and sharper texture details, and more effectively preserve the complementary information of the source images. Overall, these quantitative data strongly demonstrate the significant superiority and beneficial effects of the method described in this embodiment over existing techniques in generating high-quality, information-rich, detailed, and well-preserved infrared and visible light fusion images.

[0093]

[0094]

[0095] Figure 2 This is a visual comparison diagram of the basic features and detail features with and without equivariant consistency constraints in an embodiment of the present invention. Figure 2 The first row shows the original infrared image and visible light image of the front of a car under high exposure conditions. The second row shows the visualization of the unified basic features and unified detail features obtained after the geometric transformation operation without setting the equivariant consistency constraint is directly processed and spliced by the encoder. The third row shows the visualization of the transformed basic features and transformed detail features with the equivariant consistency constraint set. Figure 2As can be seen from the figure, the equivariant consistency constraint effectively improves the model's ability to preserve structural and texture information. The feature decomposition loss function imposes a correlation constraint on the transformed and reconstructed detail features, further enhancing the model's performance in preserving detail information.

[0096] from Figures 3 to 6 As can be seen from the figure, compared to other models, the fused image produced by our method contains more texture detail and structural information, with clearer edge contours and more prominent target features. Furthermore, our method effectively preserves the thermal radiation information in infrared images and the rich color details in visible light images, achieving a complementary fusion of information. The overall visual effect is more natural, with a richer level of detail. Figure 7 It shows that the downstream monitoring task is a target detection task, and the model used for the target detection task is the YOLOv5 model. After the infrared image and the visible light image are fused using the fusion model trained by the embodiment of the present invention and other existing fusion models, the YOLOv5 model is used to detect vehicles and personnel. The detection accuracy of the method of the present invention is significantly improved compared with other models, indicating that the method of the present invention has advantages in enhancing the expression of target features, can more accurately identify and locate targets, and improves the overall performance and reliability of detection. Figure 8 It shows that the monitoring downstream task is a semantic segmentation task, and the model used for the semantic segmentation task is the BiSNet semantic segmentation network. After the infrared image and the visible light image are fused using the fusion model trained by the embodiment of the present invention and other existing fusion models, the BiSNet semantic segmentation network is used for semantic segmentation. The object structure in the fused image obtained by the method of the present invention is clear and the boundaries are distinct, and the number of segmented targets is significantly more than that of other comparison methods, which reflects the advantages of this method in retaining and expressing semantic information.

[0097] To verify the impact of the semantic loss term weight on overall model performance in the semantic segmentation model, this example sets the weight of this loss term to 0.1, 0.2, and 0.3, respectively, while keeping other parameters unchanged. Comparative experiments were conducted to explore the optimal weight configuration. The experimental results, shown in Table 3, show that when the weight is set to 0.2, optimal or near-optimal performance is achieved across multiple evaluation metrics, including EN, SD, SF, MI, VIF, and Qabf. This demonstrates that this setting strikes a good balance between image clarity and semantic expressiveness, validating the effectiveness of semantic loss in improving the overall quality of fused images.

[0098]

[0099] The present invention can effectively fuse infrared and visible light images to generate fused images with excellent visual quality, retain fine details, be robust to geometric transformations, and be rich in semantic information, thereby significantly improving the performance of downstream advanced visual tasks such as target detection and semantic segmentation.

[0100] Example 2:

[0101] This embodiment is based on a task-driven equivariant consistency image fusion system, including:

[0102] Image acquisition module: used to acquire paired infrared images and visible light images;

[0103] Fusion model: used for fusing infrared and visible light images; the fusion model includes an encoder, an equivariant consistency constraint module, and a decoder; the encoder is used to perform feature decomposition on the input infrared and visible light images to obtain their respective basic features and detail features; the equivariant consistency constraint module is configured to impose equivariant consistency constraints based on geometric transformation operations;

[0104] a semantic guidance module, comprising a semantic segmentation model associated with a preset downstream visual task, configured to generate a semantic constraint signal based on supervision information of the downstream visual task;

[0105] The decoder receives constraints from the equivariant consistency constraint module and the semantic guidance module to fuse the basic features and detail features input into the decoder and decode to generate a fused image.

[0106] The geometric transformation operation in this embodiment includes a combination of a translation operation, a rotation operation, and a flipping operation, which is set to translate a random distance, rotate 10 degrees, flip to the left, and flip to the right once each.

[0107] The present invention imposes equivariant consistency constraints based on geometric transformation operations, which can ensure the robustness of the fusion process to geometric transformations; the semantic guidance module can impose semantic constraints on the fusion process, perform fusion reconstruction based on equivariant consistency constraints and semantic constraints, and obtain the final fused image.

[0108] Example 3:

[0109] In the system of this embodiment, the encoder includes a shared feature encoder, a basic feature encoder, and a detail feature encoder; the basic feature encoder is used to extract basic information, and the detail feature encoder is used to extract detail information. The decoder includes a splicing operation and a Restormer module. It should be noted that the Restormer module used in the encoder and decoder has similar structures, but its functions and parameter settings are different, determined through training to adapt them to encoding and decoding tasks respectively. The two inputs of the decoder are processed through the splicing operation and then enter the Restormer module for decoding.

[0110] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

[0111] Any matters not described in the present invention are applicable to the prior art.

Claims

1. A task-driven equivariant consistency image fusion method, characterized by: The method comprises the following steps: Acquire infrared images and visible light images; Constructing a fusion model, wherein the fusion model includes an encoder, a decoder, and a geometric transformation operation; Decomposing the infrared image and the visible light image in an encoder to obtain basic features and detailed features of each; The basic features of the infrared image and the basic features of the visible light image are added element by element to obtain a unified basic feature; the detail features of the infrared image and the detail features of the visible light image are added element by element to obtain a unified detail feature; Applying geometric transformation operations to the unified basic features and the unified detail features to obtain transformed basic features and transformed detail features respectively; At the same time, the unified basic features and the unified detail features are input into the decoder to obtain a fused image; Applying a geometric transformation operation to the fused image to obtain a transformed fused image, and re-inputting the transformed fused image into the encoder and the decoder to obtain a reconstructed fused image; Among them, the geometric transformation operation satisfies the equivariant consistency constraint; During the training process, the fused image is simultaneously processed by the semantic segmentation model to obtain a predicted segmentation map, and the semantic loss between the predicted segmentation map and the corresponding true segmentation label map is calculated; the semantic loss participates in the total loss calculation and backpropagation of the entire model, guiding the parameter update of the fusion model, and the total loss is obtained by weighting the fusion loss and the semantic loss; The trained fusion model is used to fuse infrared images and visible light images.

2. The task-driven equivariant consistency image fusion method according to claim 1, characterized in that: The fusion loss includes structural similarity loss, feature decomposition loss, intensity loss and gradient loss.

3. The task-driven equivariant consistency image fusion method according to claim 2, characterized in that: The structural similarity loss is a weighted sum of the similarity between the transformed basic features and the reconstructed basic features, the similarity between the transformed detail features and the reconstructed detail features, and the similarity between the fused images before and after reconstruction; The feature decomposition loss is obtained by the correlation coefficient between the basic features of the infrared image and the visible light image, the correlation coefficient between the detail features, the correlation coefficient between the transformed detail features and the reconstructed detail features, and the correlation coefficient between the transformed basic features and the reconstructed basic features; The intensity loss is used to calculate the difference in pixel intensity between the transformed fused image and the reconstructed fused image; The gradient loss is used to calculate the gradient loss of the transformed fused image and the reconstructed fused image.

4. The task-driven equivariant consistency image fusion method according to claim 1, characterized in that: The semantic loss is cross entropy loss.

5. The task-driven equivariant consistency image fusion method according to claim 1, characterized in that: The encoder includes a shared feature encoder, a basic feature encoder and a detail feature encoder. The infrared image and the visible light image are respectively input into the shared feature encoder and then processed by the basic feature encoder and the detail feature encoder to obtain their respective basic features and detail features. The shared feature encoder is implemented using a Restormer module, the basic feature encoder is implemented using a Transformer module, and the detail feature encoder is implemented using a convolutional neural network module.

6. The task-driven equivariant consistency image fusion method according to claim 1, characterized in that: The geometric transformation operation includes translation, rotation and flipping operations; the operation is set to translate a random distance, rotate 10 degrees, flip to the left and flip to the right once each.

7. A task-driven equivariant consistency image fusion system, characterized by: The system comprises: Image acquisition module: used to acquire paired infrared images and visible light images; Fusion model: used for fusing infrared and visible light images; the fusion model includes an encoder, an equivariant consistency constraint module, and a decoder; the encoder is used to perform feature decomposition on the input infrared and visible light images to obtain their respective basic features and detail features; the equivariant consistency constraint module is configured to impose equivariant consistency constraints based on geometric transformation operations; a semantic guidance module, comprising a semantic segmentation model associated with a preset downstream visual task, configured to generate a semantic constraint signal based on supervision information of the downstream visual task; The decoder receives constraints from the equivariant consistency constraint module and the semantic guidance module, is used to fuse the basic features and detail features input into the decoder, and decodes to generate a fused image.

Images