Downstream task driven learning based missing infrared modality image fusion method

Abstract

The present application relates to a missing infrared modal image fusion method based on downstream task-driven learning, belonging to the technical field of infrared and visible light image fusion. It comprises: obtaining a dictionary encoder and a decoder by training through paired infrared and visible light images; obtaining pseudo infrared encoding coefficients; realizing multi-weight adaptive coefficient fusion of the pseudo infrared encoding coefficients and visible light encoding coefficients in the coefficient domain; in the fusion process, the pseudo infrared encoding coefficient reasoning network is trained together with the encoding coefficient adaptive fusion network, and the final fused image is reconstructed; in the reconstruction process, the fused image is input into the downstream task, the pseudo infrared encoding coefficient reasoning network and the encoding coefficient adaptive fusion network are simultaneously updated through the loss, and a fused image more suitable for the downstream task is obtained. The present application solves how to make the coefficient reasoning and fusion network more suitable for the downstream task through the guidance of the downstream task under the condition of missing infrared modal, and obtains a fused image meeting the requirements of the downstream task through a single visible light image.

Description

Technical Field

[0001] This invention relates to a missing infrared modality image fusion method based on downstream task-driven learning, belonging to the field of infrared and visible light image fusion technology. Background Technology

[0002] In practical applications of infrared-visible light fusion, infrared modalities provide more stable saliency information for hot targets and low-light scenes, while visible light modalities offer richer texture details and semantic structures. Traditional infrared-visible light fusion methods typically assume that both modalities are available simultaneously and use visual quality or statistical indicators as the primary optimization objective to generate a fusion result with "better visual appeal." However, in real-world deployments, infrared sensors are constrained by cost, energy consumption, size, maintenance conditions, and imaging environment interference, often resulting in problems such as missing infrared modalities, severe signal quality degradation, or asynchronous acquisition. This leads to the instability of traditional fusion models that rely on dual-modal input. To address the missing infrared problem, existing research has proposed strategies based on modal completion / modal conversion, i.e., using visible light to estimate pseudo-infrared before fusion, or using generative models to achieve cross-modal mapping to fill in missing information. However, these methods often rely on pixel-level reconstruction consistency as the core supervision, which can easily produce pseudo-infrared representations that appear realistic but are task-irrelevant, leading to instability in the fused image regarding target boundaries, saliency of hot targets, or detection of small targets. On the other hand, the "downstream task-oriented fusion" concept that has emerged in recent years attempts to directly input the fusion results into the detection / segmentation network and use the task loss to constrain the fusion network in reverse, making the fusion more in line with the discrimination requirements; however, most existing task-driven fusion still rely on real infrared input and lack a collaborative optimization mechanism for modality generation and fusion in "infrared missing" scenarios. Summary of the Invention

[0003] The technical problem solved by this invention is that infrared modalities are easily missing, degraded, or unavailable in practical applications, causing existing infrared-visible light fusion methods to be highly dependent on real infrared input and unable to work stably under conditions of missing infrared. This invention proposes a missing infrared modal image fusion method based on downstream task-driven learning. This invention enables the fusion results to be directly used for downstream tasks, and uses the downstream task loss to inversely constrain the modality transformation and fusion process. It realizes end-to-end collaborative learning of modality completion, fusion representation generation, and task optimization under conditions of missing infrared modalities, thereby effectively improving the robustness, discriminativeness, and application value of the fusion results in practical tasks.

[0004] The technical solution of this invention is: a missing infrared modality image fusion method based on downstream task-driven learning, the method comprising:

[0005] Step 1: Obtain paired infrared and visible light image training datasets;

[0006] Step 2: Obtain the dictionary encoder and decoder by training paired infrared and visible light images;

[0007] Step 3: The visible light image is fed into the encoder to decompose it into visible light coding coefficients. The visible light coding coefficients are then passed through a coarse- and fine-grained pseudo-infrared coding coefficient inference network to obtain pseudo-infrared coding coefficients.

[0008] Step 4: Implement multi-weight adaptive coefficient fusion of pseudo-infrared coding coefficients and visible light coding coefficients in the coefficient domain. During the fusion process, the pseudo-infrared coding coefficient inference network and the coding coefficient adaptive fusion network are trained together to finally reconstruct the final fused image.

[0009] Step 5: During the reconstruction of the fused image, the fused image is input into the downstream task. By simultaneously updating the pseudo-infrared coding coefficient inference network and the coding coefficient adaptive fusion network through loss, a fused image more suitable for the downstream task is obtained.

[0010] Furthermore, in Step 1, each pair of infrared and visible light images in the dataset used for dictionary training consists of a set of corresponding infrared and visible light images; the resolution of each image is 512×512.

[0011] The dataset of infrared and visible light images is preprocessed by randomly cropping the data into 256×256 pixels and normalizing the processed data to the range [0,1].

[0012] Furthermore, Step 2 includes:

[0013] Paired infrared and visible light images and Simultaneously, the encoded coefficients are input into the initialized multi-layer dictionary encoder. The dictionary encoder includes shared dictionary parameters and a coefficient calculation module. The two modal images and encoded coefficients are fed into each layer for calculation and updating, finally yielding the corresponding encoded coefficients, denoted as... and At this point, the bimodal coefficients are simultaneously in a unified coefficient domain space, and a mapping between the two modal coefficients is established through a deep network. Subsequently, the bimodal coefficients are simultaneously fed into a multi-layer dictionary decoder, and after multi-layer computation and updates, the reconstructed infrared and visible light images are obtained. The quality is ensured by an explicit reconstruction loss, defined as:

[0014] ;

[0015] in, and These represent the visible light image and the infrared image reconstructed by the dictionary decoder, respectively. After multiple training sessions, a dictionary encoder and decoder capable of simultaneously decomposing and reconstructing both visible light and infrared modal images are obtained.

[0016] Furthermore, Step 3 includes:

[0017] In the pseudo-infrared coding coefficient inference network, visible light images are... The input is fed into the dictionary encoder and decomposed into coding coefficients for visible light modes. At this point, the visible light coding coefficients and the corresponding infrared coding coefficients can exist in a unified coefficient space, establishing the possibility of mode conversion between coefficients; subsequently, the visible light coding coefficients... The input is processed by a coarse- or fine-grained coefficient mode conversion network; the mode conversion task is decoupled into two stages: global infrared mapping and detailed infrared correction, each handled by a coarse conversion network. A detail correction network To complete;

[0018] The coarse transformation network is accomplished using a U-Net that combines convolutional and attention network layers, as shown below:

[0019] ;

[0020] Obtain the coarse infrared coding coefficients of the initial conversion It is used to learn the global mapping relationship of coefficients from visible light to infrared, model the overall correspondence across modes, and obtain the low-frequency components and global structure of infrared images;

[0021] A multi-expert detail correction network was introduced:

[0022] ;

[0023] The detail correction network is used to predict high-frequency detail residuals and compensate for image details that were not converted in the coarse conversion network. The pseudo-infrared coding coefficients after mode conversion are obtained by summing the expert outputs with different weights. pseudo-infrared coding coefficients after mode conversion The pseudo-infrared image is obtained after decoding. .

[0024] Furthermore, Step 3 also includes:

[0025] During the training mode conversion process, the correction amplitude is limited by dual infrared intensity gradient constraints, which are defined as follows:

[0026] ;

[0027] ;

[0028] Then through heat loss Normalized heat weighting plot Strengthening thermal intensity level alignment, defined as follows:

[0029] ;

[0030] in, It is an infrared image. Coarse infrared coding coefficients The pseudo-infrared image is obtained after decoding. This represents the gradient operator.

[0031] Furthermore, Step 4 includes:

[0032] In the multi-weight adaptive coefficient fusion network, visible light coding coefficients and pseudo-infrared coding coefficients are compared and weighted according to different input contents. The results are fed into the fusion module MAFN to achieve multi-weight adaptive fusion in the coefficient domain. The fusion coefficients are obtained by comparing the visible light structure with the predicted infrared thermal semantic output. :

[0033] ;

[0034] in, For visible light coding coefficients, The pseudo-infrared coding coefficients after mode conversion are used. During the fusion process, MAFN adaptively generates weight values ​​based on the influence of different inputs on the target fusion result. and For different inputs;

[0035] Then the fusion coefficient The shared dictionary D is fed into a trained dictionary decoder for consistency reconstruction, resulting in an infrared-visible fused image with missing infrared modes. .

[0036] Furthermore, Step 5 includes:

[0037] Obtain the fused image Next, the images will be merged. It is fed as a single modality into the pre-trained and frozen downstream task network. middle:

[0038] ;

[0039] in, Indicates the task prediction result. For downstream task network parameters; the fusion network is viewed as a task-oriented intermediate representation, connecting different downstream tasks. As a tag proxy, it outputs the fusion results of different downstream tasks, utilizing the tags of the downstream tasks. Constraints are imposed on the fusion results to ensure that the fusion quality is measured by the performance of downstream tasks;

[0040] Furthermore, during the training of the adaptive fusion network for coding coefficients, the coarse transformation network is frozen to ensure the stability of the global mode transformation, while the detail correction network participates in the training. Guided by the backpropagation of the loss from the downstream task, the pseudo-infrared details are corrected to varying degrees, making the pseudo-infrared coding coefficients output by the network suitable for the downstream task. This is then combined with the subsequent adaptive fusion network for coding coefficients for further adjustments. The secondary training process of inference and fusion works together to affect the fusion output, thereby obtaining fusion results with varying degrees of correction.

[0041] Furthermore, Step 5 also includes:

[0042] Utilizing downstream task losses The detail correction network and the adaptive fusion network of the coding coefficients in the pseudo-infrared coding coefficient inference network of the preceding term are constrained simultaneously, and the constraint loss corresponding to each downstream task is different. The network model weights for each task are also different. The loss corresponding to the semantic segmentation task is defined as follows:

[0043] ;

[0044] in, This represents the cross-entropy loss, emphasizing pixel-by-pixel classification accuracy; This represents the Dice loss, which maximizes the overall overlap between the predicted segmented region and the ground truth region. Indicates weight, This represents the downstream task label; while the loss for the object detection task is defined as:

[0045] ;

[0046] in, Represents classification loss. This represents the bounding box regression loss. The target loss is represented; the gradient of the task loss simultaneously reacts to the pseudo-infrared inference and coefficient fusion processes, thereby guiding the fusion network to generate a fused image that is more conducive to downstream discrimination.

[0047] Based on task loss, intensity consistency loss is adopted. and gradient consistency loss :

[0048] ;

[0049] Where max represents taking the larger value point by point across the pixel / gradient magnitude, prompting... It simultaneously inherits the peak thermal intensity of infrared light and the sharp edges of visible light; the final fusion loss is:

[0050] .

[0051] The beneficial effects of this invention are:

[0052] 1. This invention introduces a downstream task-driven joint learning mechanism, so that the fusion result is no longer optimized solely by visual quality or statistical indicators, but directly by the performance of actual tasks such as object detection and semantic segmentation as the final evaluation criteria, thereby significantly improving the discrimination effectiveness and practical value of the fused image in real application scenarios.

[0053] 2. Under the condition of missing infrared modes, this invention unifies mode completion, fusion representation generation and downstream task optimization into the same learning framework, realizes the reverse constraint of downstream task loss on the mode conversion and fusion process, effectively avoids the problem of generation error accumulating step by step and being difficult to correct in traditional staged methods, and improves the overall stability and robustness of the system.

[0054] 3. This invention utilizes visible light information to perform task-related implicit reasoning on missing infrared modes, rather than simply pursuing pixel-level infrared reconstruction consistency. This makes the generated infrared representation more in line with the discrimination requirements of downstream tasks, significantly enhancing the saliency of thermal targets, the separability of small targets, and the perception capability in complex backgrounds.

[0055] 4. This invention introduces the fusion result directly into the downstream task network and uses the losses of downstream tasks such as target detection and semantic segmentation as the core supervision signals to jointly constrain the modality conversion and fusion process. This enables the generated fusion representation to highlight the regions and feature information that are most critical to task discrimination, thus avoiding the problem of unstable performance in actual tasks of traditional fusion methods that only optimize visual quality.

[0056] 5. The method proposed in this invention does not rely on additional infrared sensors or complex hardware configurations. It can complete infrared information reasoning and fusion representation construction using only visible light input, which reduces system cost and deployment difficulty. It has good versatility and scalability and is suitable for various application scenarios such as night monitoring, autonomous driving, security patrol and medical image-assisted analysis. Attached Figure Description

[0057] Figure 1 This is a schematic diagram of the process structure of the present invention;

[0058] Figure 2This is a schematic diagram of the pseudo-infrared coding coefficient inference module of the present invention;

[0059] Figure 3 This is a schematic diagram of the adaptive fusion module for coding coefficients of the present invention;

[0060] Figure 4 This is a schematic diagram of the downstream task gradient backpropagation of the present invention;

[0061] Figure 5 Experimental results of some existing methods Figure 1 ;

[0062] Figure 6 Experimental results of some existing methods Figure 2 ;

[0063] Figure 7 The figures show the experimental results of the method of the present invention and some existing methods. Detailed Implementation

[0064] Example 1: As Figures 1-7 As shown, a missing infrared modality image fusion method based on downstream task-driven learning is described, the method comprising:

[0065] Step 1: Obtain the paired infrared and visible light image training dataset; each pair of infrared and visible light images in the dataset used for dictionary training consists of a set of corresponding infrared and visible light images; the resolution of each image is 512×512;

[0066] The dataset of infrared and visible light images is preprocessed by randomly cropping the data into 256×256 pixels and normalizing the processed data to the range [0,1].

[0067] Step 2: Obtain the dictionary encoder and decoder by training paired infrared and visible light images;

[0068] In order to establish a modal mapping relationship between infrared and visible light modes, a pair of encoders and decoders are first trained to map the images of the two modes to the same domain space, which can simultaneously decompose and reconstruct the images of the two modes, serving as the basis for subsequent modal reasoning and fusion, and ensuring the interpretability and accuracy of the model.

[0069] Furthermore, Step 2 includes:

[0070] Paired infrared and visible light images and Simultaneously, the encoded coefficients are input into the initialized multi-layer dictionary encoder. The dictionary encoder includes shared dictionary parameters and a coefficient calculation module. The two modal images and encoded coefficients are fed into each layer for calculation and updating, finally yielding the corresponding encoded coefficients, denoted as... and At this point, the bimodal coefficients are simultaneously in a unified coefficient domain space, and a mapping between the two modal coefficients is established through a deep network. To ensure that the encoded coefficients can still reconstruct the original infrared and visible light images, the bimodal coefficients are then simultaneously fed into a multi-layer dictionary decoder. After multi-layer calculation and updates, the reconstructed infrared and visible light images are obtained. To ensure that the final reconstructed infrared and visible light images are consistent with the original images, an explicit reconstruction loss is used to guarantee its effectiveness. The loss is defined as:

[0071] ;

[0072] in, and These represent the visible light image and the infrared image reconstructed by the dictionary decoder, respectively. After multiple training sessions, a dictionary encoder and decoder capable of simultaneously decomposing and reconstructing both visible light and infrared modal images are obtained.

[0073] Step 3: The visible light image is fed into the encoder to decompose it into visible light coding coefficients. The visible light coding coefficients are then passed through a coarse- and fine-grained pseudo-infrared coding coefficient inference network to obtain pseudo-infrared coding coefficients.

[0074] This includes: decomposing a single visible light image into coding coefficients using a learned dictionary encoder; establishing a two-layer coarse- and fine-grained mode conversion relationship from visible light to infrared in the coefficient domain; establishing a global mapping through a coarse-grained network; and fine-tuning the target details through a detail correction network. Unlike a one-stage transform, the combined coarse- and fine-grained approach is more beneficial to the accuracy of the mapping. Specific operations are as follows: Figure 2 , Figure 2 This is a schematic diagram of the pseudo-infrared coding coefficient inference module of the present invention;

[0075] Furthermore, Step 3 includes:

[0076] In the pseudo-infrared coding coefficient inference network, visible light images are... The input is fed into the dictionary encoder and decomposed into coding coefficients for visible light modes. At this point, the visible light coding coefficients and the corresponding infrared coding coefficients can exist in a unified coefficient space, establishing the possibility of mode conversion between coefficients; subsequently, the visible light coding coefficients... The input is fed into a coarse- or fine-grained coefficient mode conversion network for processing. Given the significant differences in origin and scale between low-frequency structures (overall thermal distribution, target contours) and high-frequency details (boundaries, local thermal gradients) in infrared images, using the same network for modeling often leads to mutual interference. Therefore, the mode conversion task is decoupled into two stages: global infrared mapping and detail infrared correction, each handled by a coarse conversion network. A detail correction network To complete;

[0077] The coarse transformation network is accomplished using a U-Net that combines convolutional and attention network layers, as shown below:

[0078] ;

[0079] Obtain the coarse infrared coding coefficients of the initial conversion It is used to learn the global mapping relationship of coefficients from visible light to infrared, model the overall correspondence across modes, and obtain the low-frequency components and global structure of infrared images;

[0080] To constrain and ensure clear boundaries and local thermal gradients in infrared images, a multi-expert detail correction network is introduced:

[0081] ;

[0082] The detail correction network is used to predict high-frequency detail residuals and compensate for image details that were not converted in the coarse conversion network. The pseudo-infrared coding coefficients after mode conversion are obtained by summing the expert outputs with different weights. pseudo-infrared coding coefficients after mode conversion The pseudo-infrared image is obtained after decoding. .

[0083] Furthermore, Step 3 also includes:

[0084] During mode conversion training, to ensure that the detail correction network does not excessively affect the coarse conversion network and thus disrupt the global structure, a dual infrared intensity gradient constraint is used to limit the correction magnitude, defined as follows:

[0085] ;

[0086] ;

[0087] Then through heat loss Normalized heat weighting plot Strengthening thermal intensity level alignment, defined as follows:

[0088] ;

[0089] in, It is an infrared image. Coarse infrared coding coefficients The pseudo-infrared image is obtained after decoding. This represents the gradient operator.

[0090] Step 4: Implement multi-weight adaptive coefficient fusion of pseudo-infrared coding coefficients and visible light coding coefficients in the coefficient domain. During the fusion process, the pseudo-infrared coding coefficient inference network and the coding coefficient adaptive fusion network are trained together to finally reconstruct the final fused image.

[0091] This includes: concatenating visible light coding coefficients with pseudo-infrared coding coefficients obtained from coarse-grained and fine-grained inference, performing multi-weight adaptive fusion in the coefficient domain, and then decoding and reconstructing the fused coefficients using a dictionary decoder to obtain the fused image under conditions lacking infrared light; specific operations are as follows... Figure 3 , Figure 3 This is a schematic diagram of the adaptive fusion module for coding coefficients of the present invention;

[0092] Furthermore, Step 4 includes:

[0093] In the multi-weight adaptive coefficient fusion network, visible light coding coefficients and pseudo-infrared coding coefficients are compared and weighted according to different input contents. The results are fed into the fusion module MAFN to achieve multi-weight adaptive fusion in the coefficient domain. The fusion coefficients are obtained by comparing the visible light structure with the predicted infrared thermal semantic output. :

[0094] ;

[0095] in, For visible light coding coefficients, The pseudo-infrared coding coefficients after mode conversion are represented by MAFN, which consists of two concatenated convolutional-attention fusion modules. During the fusion process, MAFN adaptively generates weight values ​​based on the influence of different inputs on the target fusion result. and For different inputs; atoms associated with structural edges are usually more inherited. Atoms related to thermal semantics inherit more Thus, within a unified atomic space, Assign more weight to the edges of the result, More weight is allocated to the hot semantic information to achieve fine fusion of the coefficient domain.

[0096] Then the fusion coefficient The shared dictionary D is fed into a trained dictionary decoder for consistency reconstruction, resulting in an infrared-visible fused image with missing infrared modes. .

[0097] Step 5: During the reconstruction of the fused image, the fused image is input into the downstream task. By simultaneously updating the pseudo-infrared coding coefficient inference network and the coding coefficient adaptive fusion network through loss, a fused image more suitable for the downstream task is obtained.

[0098] This includes: to apply the fusion results to downstream tasks, feeding the reconstruction results into the downstream task network to obtain prediction results, calculating the loss between the prediction results and the true labels, and simultaneously updating the coefficient inference and fusion networks using gradients. Specific operations are as follows: Figure 4 As shown, Figure 4 This is a schematic diagram of the downstream task gradient backpropagation of the present invention;

[0099] Furthermore, Step 5 includes:

[0100] Obtain the fused image Subsequently, to be suitable for downstream tasks such as object detection, semantic segmentation, and depth estimation, the fused images will be... It is fed as a single modality into the pre-trained and frozen downstream task network. middle:

[0101] ;

[0102] in, Indicates the task prediction result. For downstream task network parameters; the fusion network is viewed as a task-oriented intermediate representation, connecting different downstream tasks. As a tag proxy, it outputs the fusion results of different downstream tasks, utilizing the tags of the downstream tasks. Constraints are imposed on the fusion results to ensure that the fusion quality is measured by the performance of downstream tasks;

[0103] Furthermore, during the training of the adaptive fusion network for coding coefficients, the coarse transformation network is frozen to ensure the stability of the global mode transformation, while the detail correction network participates in the training. Guided by the backpropagation of the loss from the downstream task, the pseudo-infrared details are corrected to varying degrees, making the pseudo-infrared coding coefficients output by the network suitable for the downstream task. This is then combined with the subsequent adaptive fusion network for coding coefficients for further adjustments. The secondary training process of inference and fusion works together to affect the fusion output, thereby obtaining fusion results with varying degrees of correction.

[0104] Furthermore, the preceding network training is updated using specific task losses for different downstream tasks to ensure different pseudo-infrared inference and prediction for different tasks, thereby achieving higher task adaptability; Step 5 also includes:

[0105] To ensure that the fusion result is shifted to downstream tasks as expected, the loss from downstream tasks is utilized. The detail correction network and the adaptive fusion network of the coding coefficients in the pseudo-infrared coding coefficient inference network of the preceding term are constrained simultaneously, and the constraint loss corresponding to each downstream task is different. The network model weights for each task are also different. The loss corresponding to the semantic segmentation task is defined as follows:

[0106] ;

[0107] in, This represents the cross-entropy loss, emphasizing pixel-by-pixel classification accuracy; This represents the Dice loss, which maximizes the overall overlap between the predicted segmented region and the ground truth region. Indicates weight, This represents the downstream task label; while the loss for the object detection task is defined as:

[0108] ;

[0109] in, Represents classification loss. This represents the bounding box regression loss. The target loss is represented; the gradient of the task loss simultaneously reacts to the pseudo-infrared inference and coefficient fusion processes, thereby guiding the fusion network to generate a fused image that is more conducive to downstream discrimination.

[0110] Based on task loss, to simultaneously highlight the salience of thermal radiation and maintain the visible light structure, intensity uniformity loss is adopted. and gradient consistency loss :

[0111] ;

[0112] Where max represents taking the larger value point by point across the pixel / gradient magnitude, prompting... It simultaneously inherits the peak thermal intensity of infrared light and the sharp edges of visible light; the final fusion loss is:

[0113] .

[0114] To verify the effectiveness of the method, the performance was evaluated on three publicly available datasets: FLIR, KAIST, and MSRS. For training, 5445, 8310, and 536 pairs of infrared-visible light images were selected from these three datasets, respectively; for testing, 765, 1246, and 179 pairs of infrared-visible light images were selected, respectively. The proposed model was trained on a server platform equipped with an RTX4090 and a PyTorch environment. During training, the number of epochs was set to 20, the learning rate to 1e-5, the batch size to 24, and the AdamW optimizer was used to update the network parameters.

[0115] Furthermore, the visual effects of this invention were compared with existing infrared-visible image fusion methods such as U2Fusion, TarDAL, CDDFuse, LRRNet, IVFWSR, CoCoNet, EMMA, TIMFusion, DCEvo, and SAGE. Downstream task detection methods using Segformer and Yolov5 were also employed. The comparison results of the fusion results of this method with those of U2Fusion, TarDAL, CDDFuse, LRRNet, IVFWSR, CoCoNet, EMMA, TIMFusion, DCEvo, and SAGE in downstream tasks are shown below. Figure 5 , Figure 6 , Figure 7 As shown, where, Figure 5 , Figure 6 , Figure 7 The bottom color block represents the region with different semantic information in the segmented image obtained after the fusion result is fed into the Segformer segmentation method. This shows that the method proposed in this invention can maintain a good fusion visual effect even when the infrared modality is missing, and exhibits a similar level of performance in downstream tasks as the fusion result without the missing infrared modality.

[0116] This invention addresses the problem of missing infrared modes by performing infrared mode inference and fusion in a unified coding coefficient space. Pseudo-infrared coding coefficients are obtained through coarse- and fine-grained coefficient domain mode inference, and feature-level multi-weight adaptive fusion is implemented at the dictionary atomic level. At the same time, downstream task loss is introduced to jointly constrain the mode inference and fusion process, making the fusion result more in line with the discrimination requirements of downstream tasks and improving the fusion effect and task adaptability under missing infrared conditions. Different deployment environments are used in the inference stage according to the downstream task scenarios.

[0117] The specific embodiments of the present invention have been described in detail above with reference to the accompanying drawings. However, the present invention is not limited to the above embodiments. Within the scope of knowledge possessed by those skilled in the art, various changes can be made without departing from the spirit of the present invention.

Claims

1. A missing infrared modality image fusion method based on downstream task-driven learning, characterized in that: The method includes: Step 1: Obtain paired infrared and visible light image training datasets; Step 2: Obtain the dictionary encoder and decoder by training paired infrared and visible light images; Step 3: The visible light image is fed into the encoder to decompose it into visible light coding coefficients. The visible light coding coefficients are then passed through a coarse- and fine-grained pseudo-infrared coding coefficient inference network to obtain pseudo-infrared coding coefficients. Step 4: Implement multi-weight adaptive coefficient fusion of pseudo-infrared coding coefficients and visible light coding coefficients in the coefficient domain. During the fusion process, the pseudo-infrared coding coefficient inference network and the coding coefficient adaptive fusion network are trained together to finally reconstruct the final fused image. Step 5: During the reconstruction of the fused image, the fused image is input into the downstream task. The pseudo-infrared coding coefficient inference network and the coding coefficient adaptive fusion network are updated simultaneously through loss to obtain a fused image that is more suitable for the downstream task. Step 3 includes: In the pseudo-infrared coding coefficient inference network, visible light images are... The input is fed into the dictionary encoder and decomposed into coding coefficients for visible light modes. At this point, the visible light coding coefficients and the corresponding infrared coding coefficients can exist in a unified coefficient space, establishing the possibility of mode conversion between coefficients; subsequently, the visible light coding coefficients... The input is processed by a coarse- or fine-grained coefficient mode conversion network; the mode conversion task is decoupled into two stages: global infrared mapping and detailed infrared correction, each handled by a coarse conversion network. A detail correction network To complete; The coarse transformation network is accomplished using a U-Net that combines convolutional and attention network layers, as shown below: ； Obtain the coarse infrared coding coefficients of the initial conversion It is used to learn the global mapping relationship of coefficients from visible light to infrared, model the overall correspondence across modes, and obtain the low-frequency components and global structure of infrared images; A multi-expert detail correction network was introduced: ； The detail correction network is used to predict high-frequency detail residuals and compensate for image details that were not converted in the coarse conversion network. The pseudo-infrared coding coefficients after mode conversion are obtained by summing the expert outputs with different weights. pseudo-infrared coding coefficients after mode conversion The pseudo-infrared image is obtained after decoding. .

2. The missing infrared modality image fusion method based on downstream task-driven learning according to claim 1, characterized in that: In Step 1, each pair of infrared and visible light images in the dataset used for dictionary training consists of a set of corresponding infrared and visible light images; the resolution of each image is 512×512. The dataset of infrared and visible light images is preprocessed by randomly cropping the data into 256×256 pixels and normalizing the processed data to the range [0,1].

3. The missing infrared modality image fusion method based on downstream task-driven learning according to claim 1, characterized in that: Step 2 includes: Paired infrared and visible light images and Simultaneously, the encoded coefficients are input into the initialized multi-layer dictionary encoder. The dictionary encoder includes shared dictionary parameters and a coefficient calculation module. The two modal images and encoded coefficients are fed into each layer for calculation and updating, finally yielding the corresponding encoded coefficients, denoted as... and At this point, the bimodal coefficients are simultaneously in a unified coefficient domain space, and a mapping between the two modal coefficients is established through a deep network. Subsequently, the bimodal coefficients are simultaneously fed into a multi-layer dictionary decoder, and after multi-layer computation and updates, the reconstructed infrared and visible light images are obtained. The quality is ensured by an explicit reconstruction loss, defined as: ； in, and These represent the visible light image and the infrared image reconstructed by the dictionary decoder, respectively. After multiple training sessions, a dictionary encoder and decoder capable of simultaneously decomposing and reconstructing both visible light and infrared modal images are obtained.

4. The missing infrared modality image fusion method based on downstream task-driven learning according to claim 1, characterized in that: Step 3 also includes: During the training mode conversion process, the correction amplitude is limited by dual infrared intensity gradient constraints, which are defined as follows: ； ； Then through heat loss Normalized heat weighting plot Strengthening thermal intensity level alignment, defined as follows: ； in, It is an infrared image. Coarse infrared coding coefficients The pseudo-infrared image is obtained after decoding. This represents the gradient operator.

5. The missing infrared modality image fusion method based on downstream task-driven learning according to claim 1, characterized in that: Step 4 includes: In the multi-weight adaptive coefficient fusion network, visible light coding coefficients and pseudo-infrared coding coefficients are compared and weighted according to different input contents. The results are fed into the fusion module MAFN to achieve multi-weight adaptive fusion in the coefficient domain. The fusion coefficients are obtained by comparing the visible light structure with the predicted infrared thermal semantic output. : ； in, For visible light coding coefficients, The pseudo-infrared coding coefficients after mode conversion are used. During the fusion process, MAFN adaptively generates weight values ​​based on the influence of different inputs on the target fusion result. and For different inputs; Then the fusion coefficient With shared dictionaries The image is fed into a trained dictionary decoder for consistency reconstruction to obtain an infrared-visible fusion image with missing infrared modes. .

6. The missing infrared modality image fusion method based on downstream task-driven learning according to claim 1, characterized in that: Step 5 includes: Obtain the fused image Next, the images will be merged. It is fed as a single modality into the pre-trained and frozen downstream task network. middle: ； in, Indicates the task prediction result. For downstream task network parameters; the fusion network is viewed as a task-oriented intermediate representation, connecting different downstream tasks. As a tag proxy, it outputs the fusion results of different downstream tasks, utilizing the tags of the downstream tasks. Constraints are imposed on the fusion results to ensure that the fusion quality is measured by the performance of downstream tasks; Furthermore, during the training of the adaptive fusion network for coding coefficients, the coarse transformation network is frozen to ensure the stability of the global mode transformation, while the detail correction network participates in the training. Guided by the backpropagation of the loss from the downstream task, the pseudo-infrared details are corrected to varying degrees, making the pseudo-infrared coding coefficients output by the network suitable for the downstream task. This is then combined with the subsequent adaptive fusion network for coding coefficients for further adjustments. The secondary training process of inference and fusion works together to affect the fusion output, thereby obtaining fusion results with varying degrees of correction.

7. The missing infrared modality image fusion method based on downstream task-driven learning according to claim 6, characterized in that: Step 5 also includes: Utilizing downstream task losses The detail correction network and the adaptive fusion network of the coding coefficients in the pseudo-infrared coding coefficient inference network of the preceding term are constrained simultaneously, and the constraint loss corresponding to each downstream task is different, and the network model weights of each task are different. The loss corresponding to the semantic segmentation task is defined as follows: ； in, This represents the cross-entropy loss, emphasizing pixel-by-pixel classification accuracy; This represents the Dice loss, which maximizes the overall overlap between the predicted segmented region and the ground truth region. Indicates weight, This represents the downstream task label; while the loss for the object detection task is defined as: ； in, Represents classification loss. This represents the bounding box regression loss. The target loss is represented; the gradient of the task loss simultaneously reacts to the pseudo-infrared inference and coefficient fusion processes, thereby guiding the fusion network to generate a fused image that is more conducive to downstream discrimination. Based on task loss, intensity consistency loss is adopted. and gradient consistency loss : ； Where max represents taking the larger value point by point across the pixel / gradient magnitude, prompting... It simultaneously inherits the peak thermal intensity of infrared light and the sharp edges of visible light; the final fusion loss is: .

Images