Memory-enhanced deep unfolding multimodal image fusion method with enhanced downstream tasks

Abstract

The application discloses a memory reinforcement deep unfolding multi-modal image fusion method with downstream task enhancement, which comprises the following steps: collecting infrared images and visible light images, and dividing training set and test set; establishing an optimization target and solving, to obtain an iterative formula; using a neural network to replace a proximal operator in the iterative formula, to obtain a neural network structure; sending the training set into the neural network to obtain a fusion picture, and calculating a total loss according to the fusion picture, the infrared image and the visible light image; updating parameters of the neural network according to the total loss to obtain an updated neural network; inputting the infrared image and the visible light image of the test set into the updated neural network to obtain a fusion image. The application makes the fused image have characteristics easy to be distinguished by a downstream task network, and can realize the best performance on a data set, while performance and interpretability are taken into account, and the application has rationality and applicability.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of image processing, and particularly relates to a memory reinforcement deep unfolding multi-modal image fusion method with downstream task enhancement. BACKGROUND

[0002] Image fusion obtains a fusion image which inherits all the advantages and information of all source images by integrating the imaging of the same scene by different sensors. Image fusion extracts effective information from images of different modalities, eliminates the imaging defects of a single sensor, and generates high-quality pictures, which is also robust to disturbances, thereby facilitating identification and tracking. Infrared and visible light image fusion is the most representative topic in multi-modal image fusion. The combination of thermal radiation information in infrared images and texture information in visible light images is widely used in automatic driving, face recognition, fire control, security, environmental monitoring and other fields. Infrared images and visible light images have different wavelengths and contain different information. Infrared images can capture the heat of objects and are basically not affected by lighting conditions, can penetrate smoke, clouds and other objects, and ensure the prominent position of objects in the environment, but it is difficult to capture the texture information of the scene and is easily disturbed by noise. In contrast, visible light images can reflect the brightness, color and texture details of objects, provide more intuitive and realistic image performance, and are very important for human observation, analysis and decision-making, but they are easily affected by lighting conditions and occlusions, so it is impossible to distinguish objects from the background. The fusion image generated by infrared and visible light image fusion has both thermal radiation information and texture details, enhancing the completeness and visibility of image information and improving the recognition and positioning ability of the image.

[0003] In recent years, many image fusion techniques have been proposed, which can be mainly divided into traditional methods and deep learning-based methods. Traditional methods include multi-scale transformation methods, sparse representation methods, and subspace methods. Most traditional methods require manual design of fusion criteria, which limits performance. Neural networks have strong feature extraction and learning capabilities, so many high-performance methods based on deep learning have been proposed, mainly including convolutional neural network-based methods, autoencoder-based methods, generative adversarial network-based methods, and algorithm unfolding models.

[0004] Since the fused image has rich information that cannot be completely captured by a single modality, multi-modal image fusion is also used as a preprocessing module for advanced visual tasks such as semantic segmentation, object detection, object tracking, and pedestrian recognition. Although existing deep learning-based methods can successfully generate high-quality fusion images, most methods only consider evaluation indicators or visual quality, which do not absolutely help advanced visual tasks. There are also some works that hope to guide fusion from the perspective of semantic information, such as manually designing semantic richness or introducing perceptual loss, but such methods cannot directly help advanced visual tasks. SUMMARY

[0005] To overcome the problems in the prior art, the purpose of the present application is to provide a downstream task enhanced memory reinforcement deep unfolding multi-modal image fusion method.

[0006] To achieve the above purpose, the technical scheme adopted by the present application is:

[0007] The downstream task enhanced memory reinforcement deep unfolding multi-modal image fusion method comprises the following steps:

[0008] 1) Collecting infrared images and visible light images; and dividing the infrared images and the visible light images into a training set and a test set;

[0009] 2) Establishing an optimization objective, solving the optimization objective by using a gradient projection method to obtain an iterative formula; using a neural network to replace the proximal operator in the iterative formula to obtain a neural network structure;

[0010] 3) Sending the training set into the neural network to obtain a fusion picture, calculating a total loss of the fusion picture, the infrared image and the visible light image; updating the parameters of the neural network according to the total loss by using a gradient backpropagation algorithm to obtain an updated neural network;

[0011] 4) Inputting the infrared image and the visible light image of the test set into the updated neural network to obtain a fusion image.

[0012] Further, the optimization objective is as follows:

[0013]

[0014] Wherein, DS(·) is a score item related to a downstream task, h(·) is a prior item, λ is a first weight parameter, η is a second weight parameter, I f is a fusion image.

[0015] Further, the iterative formula is as follows:

[0016]

[0017] Wherein, ρ k is a learnable step size, prox h (·) is a proximal operator corresponding to the prior item h(·), represents the gradient of the intermediate result of the k-1 stage;

[0018] The gradient of the intermediate result of the k-1 stage is calculated by the following formula:

[0019]

[0020] Further, the proximal operator prox h (·) is composed of a convolution layer, an attribution attention module, and a memory enhancement module.

[0021] The convolution layer is used for convolution.

[0022] The attribution attention module is used for processing the attribution map and the result after convolution to obtain a fusion result.

[0023] The memory enhancement module is used to enhance the information interaction between different stages.

[0024] Further, the fusion picture of the kth stage is obtained through the following process:

[0025] The input k-1 stage fusion result I f (k-1) is updated by one step of gradient to obtain a first intermediate feature The first intermediate feature is extracted by the convolution layer to obtain a second intermediate feature The second intermediate feature is concatenated with the short-term memory of the k-1 stage and the long-term memory and processed by convolution to obtain a third intermediate feature

[0026] The attribution map att-map is operated by a Sigmoid function and then multiplied element by element with the third intermediate feature after one layer of convolution, and the product is added to the third intermediate feature to obtain a fourth intermediate feature The fourth intermediate feature is compressed in channel by one layer of convolution to obtain the fusion picture of the kth stage

[0027] Further, the ith pixel of the attribution map att-map is calculated by the following formula,

[0028]

[0029] DS(I) = Score SS (I) + βScore OD (I) (5)

[0030] where β is a weight parameter, Score SS (I) and Score oD (I) are the scores of semantic segmentation and object detection respectively, and I is the input.

[0031] Further, the score Score of semantic segmentation SS (I) is calculated by the following formula:

[0032]

[0033] The score Score of object detection OD (I) is calculated by the following formula:

[0034]

[0035] wherein, represents the discriminant score of the k-th class of the semantic segmentation network of the (i, j) pixel in the input I, i represents the horizontal position coordinate, and j represents the vertical position coordinate.

[0036] Further, the short-term memory of the k-1 stage is calculated by the following formula:

[0037]

[0038] wherein, is the fourth intermediate feature of the k-1 stage.

[0039] Further, the long-term memory of the k-1 stage is calculated by the following formula:

[0040]

[0041] wherein, z (k-1) is the feature after the intermediate features of the k-th stage of the neural network are spliced together, h (k-1) is the output of the k-th stage of conv-lstm, h (k-2) is the output of the k-1 stage of conv-lstm, c (k-1) is the cell state of the k-th stage of conv-lstm, c (k-2) is the cell state of the k-1 stage of conv-lstm, is the second intermediate feature of the k-1 stage, is the third intermediate feature of the k-1 stage, is the fourth intermediate feature of the k-1 stage.

[0042] Further, the total loss is calculated by the following formula:

[0043]

[0044] wherein, is the intensity loss, is the gradient loss, where represents Sobel gradient operator, |·| is an absolute value operator, and max(·) represents a pixel-level maximum operation.

[0045] Compared with the prior art, the present application has the beneficial effects of:

[0046] A novel image fusion network is proposed in the present application, which starts from an optimization problem containing downstream task items, establishes an optimization target, uses gradient projection method to solve the optimization target, and obtains an iterative formula; neural networks are used to replace the proximal operator in the iterative formula to obtain a neural network structure, so that the fused image has features that are easy to be distinguished by downstream task networks, and can achieve the best performance on the data set while taking into account the performance and interpretability. The application of the fused image in high-level visual tasks proves the rationality and applicability of the method of the present application.

[0047] Further, since the optimization target of the present application includes sub-items related to downstream tasks, and through iterative updating, the fused image has features suitable for downstream task networks.

[0048] Further, the items related to downstream tasks are added to the optimization target to force the fused image to contain features that enhance downstream tasks, and a deep neural network is constructed to enhance the interpretability of the model; the attribution analysis method is used to determine the regions and pixels in the fused image that contribute to the downstream tasks.

[0049] Further, since the memory enhancement module is used, the memory enhancement is used to solve the information loss problem in the general algorithm unfolding network and enhance the information flow and interaction between different stages.

[0050] Further, in the present application, the attribution attention module is used to find the pixels and regions in the fused image that promote the downstream tasks, i.e., the pixels and regions that are most helpful to high-level visual tasks, and make the convolutional neural network pay more attention to the region. BRIEF DESCRIPTION OF DRAWINGS

[0051] Figure 1 Flowchart of the downstream task-enhanced memory reinforcement deep unfolding multi-modal image fusion method of the present application;

[0052] Figure 2 Process diagram for stage k;

[0053] Figure 3are the fusion results of "00718N" of MSRS dataset under different methods, wherein (a) is an infrared picture, (b) is a visible light picture, (c) is a DIDFuse fusion result, (d) is a U2Fusion fusion result, (e) is an SDNet fusion result, (f) is a TarDAL fusion result, (g) is a DeFusion fusion result, (h) is a ReCoNet fusion result, (i) is a UMFusion fusion result, and (j) is a fusion result of the present application;

[0054] Figure 4 are the fusion results of "00718N" of MSRS dataset under different methods, wherein (a) is an infrared picture, (b) is a visible light picture, (c) is a DIDFuse fusion result, (d) is a U2Fusion fusion result, (e) is an SDNet fusion result, (f) is a TarDAL fusion result, (g) is a DeFusion fusion result, (h) is a ReCoNet fusion result, (i) is a UMFusion fusion result, and (j) is a fusion result of the present application;

[0055] Figure 5 are the fusion results of "FLIR 06430" of RoadScene dataset under different methods, wherein (a) is an infrared picture, (b) is a visible light picture, (c) is a DIDFuse fusion result, (d) is a U2Fusion fusion result, (e) is an SDNet fusion result, (f) is a TarDAL fusion result, (g) is a DeFusion fusion result, (h) is a ReCoNet fusion result, (i) is a UMFusion fusion result, and (j) is a fusion result of the present application;

[0056] Figure 6 are the fusion results of "00232" of M3FD dataset under different methods, wherein (a) is an infrared picture, (b) is a visible light picture, (c) is a DIDFuse fusion result, (d) is a U2Fusion fusion result, (e) is an SDNet fusion result, (f) is a TarDAL fusion result, (g) is a DeFusion fusion result, (h) is a ReCoNet fusion result, (i) is a UMFusion fusion result, and (j) is a fusion result of the present application. DETAILED DESCRIPTION

[0057] In order to facilitate the understanding of the present application, the present application will be described more fully below with reference to the accompanying drawings. The preferred embodiments of the present application are shown in the drawings. However, the present application can be realized in many different forms and is not limited to the embodiments described herein. On the contrary, these embodiments are provided so that the disclosure of the present application can be more thoroughly and completely understood.

[0058] Referring to Figure 1 , the downstream task enhanced memory reinforcement deep unfolding multi-modal image fusion method of the present application comprises the following steps:

[0059] 1) Collecting infrared image I ir and visible light image I vi ; I ir ∈R H×W and I vi ∈R H×W , H is height, W is width, I f ∈R H×W represents the corresponding fusion image, and R is a real number;

[0060] 2) Since the fused image needs to preserve the information of the source image as much as possible and be suitable for downstream tasks, the present application first establishes an optimization objective according to the initial value The gradient projection method is used to solve the optimization objective to obtain an iterative formula; the neural network is used to replace the prox h (·) of the prior term h(·) in the iterative formula to obtain a neural network structure; that is, the PM block in Figure 1 One step iteration is converted into one operation of the PM block. The output result of the kth stage is the fusion picture of the kth stage Repeat iteration K times, and the result of K times of iteration is converted into the fusion result of K layers of PM blocks K is the total number of iterations, and k is the specific iteration number; K in the present application is determined according to actual needs, preferably, K = 20, and k is 1, 2, …, K; as shown in Figure 1 Each layer (stage) of the neural network is regarded as one iteration of the optimization algorithm.

[0061] Wherein, the optimization objective is as follows:

[0062]

[0063] In the formula, and are the fidelity terms of the image, DS(·) is the score term related to the downstream task, h(·) is the prior term, λ is the first weight parameter, η is the second weight parameter, and I f is the fusion picture.

[0064]

[0065] Wherein, is the intermediate result of the kth stage, and ρ k is a learnable step size, and prox h(·) is the proximal operator corresponding to the prior term h(·), express The gradient, when k is 0, the initial value The infrared image I is extracted by using the convolution layer ir and visible light image I vi The features are concatenated and convolved again.

[0066] The proximal operator prox corresponding to the prior term h(·) h (·) It mainly consists of convolutional layers, attribution attention modules, and memory enhancement modules.

[0067] Taking the kth stage of convolutional neural network as an example, Figure 2 The detailed implementation process of the iteration is described. The specific iteration can be carried out through the following steps to obtain the fusion image of the kth stage Among them F i (k) ,i=0,1,2…, represents the intermediate features;

[0068]

[0069] The k-1 stage fusion result of the input I f (k-1) First, perform a gradient update to obtain the first intermediate feature of stage k First intermediate feature of stage k After extracting features through the convolution layer, the second intermediate features of stage k are obtained Second intermediate feature with k-1 stages of short-term memory and long-term memory Splicing and processing using convolution layer to obtain the third intermediate feature of stage k

[0070] The attribution attention module uses attribution analysis to find the area of ​​the fusion image that promotes the downstream task and makes the neural network pay more attention to this area. The specific process is: first, the attribution map att-map is operated by Sigmoid function and then the third intermediate feature of the k stage after a layer of convolution is obtained. Perform element-by-element multiplication, and then multiply the product with the third intermediate feature of the k stage Add together to complete the attention task and obtain the fourth intermediate feature of the k stage The fourth intermediate characteristic of the k stage After another layer of convolution compression channel, the fusion image of the kth stage is obtained

[0071] The i-th pixel of the attribution map att-map is calculated as follows:

[0072]

[0073] wherein

[0074] DS(I) = Score SS (I) + βScore OD (I) (5)

[0075] wherein β is a weight parameter, Score SS (I) and Score OD (I) are the scores of semantic segmentation and object detection respectively, and I is the input;

[0076] The score Score SS (I) of semantic segmentation is calculated by the following formula:

[0077]

[0078] The score Score OD (I) of object detection is calculated by the following formula:

[0079]

[0080] The task score is designed as a function that best represents the network output. For the semantic segmentation task, it is the maximum class score of each pixel, and the subscripts i and j represent the horizontal and vertical coordinates of the position, respectively. represents the discriminant score of the k-th class of the semantic segmentation network of the (i, j) pixel in the input I. And for the object detection task, N represents the total number of potential bounding boxes, and obj n and respectively represent the objectness and class probability of the n-th bounding box. The objectness obj n of the n-th bounding box represents whether the network predicts to contain an object, and the class probability of the n-th bounding box represents the class to which the object may belong. The semantic segmentation network and the object detection network are pre-trained, and only inference is required during training of the convolutional neural network without training.

[0081] The memory enhancement module is used to enhance the information interaction between different stages to solve the problem of information loss in traditional deep unfolding networks.

[0082] Let the feature before the last layer of convolution of each stage be the short-term memory, which is directly input into the first convolutional layer of the next stage; the short-term memory of the l-1 stage is calculated by the following formula:

[0083] ​

[0084] And long-term memory is modeled by Conv-LSTM, which can balance the information of past and current state. The first convolutional layer of each stage also accepts the long-term memory of all previous stages and sends the intermediate features of each stage into Conv-LSTM for processing as long-term memory.

[0085] Long-term memory of k-1 stage It is calculated by the following formula:

[0086]

[0087] Where, z (k-1) is the feature after the intermediate features of the neural network k-1 stage are spliced together, h (k-1) is the output of the Conv-LSTM of the k-1 stage, h (k-2) is the output of the Conv-LSTM of the k-2 stage, c (k-1) is the cell state of the Conv-LSTM of the k-1 stage, c (k-2) is the cell state of the Conv-LSTM of the k-2 stage, is the second intermediate feature of the k-1 stage, is the third intermediate feature of the k-1 stage, is the fourth intermediate feature of the k-1 stage.

[0088] 3) In the training stage, the collected infrared image and visible light image are divided into training set and test set, and the training set is sent into the neural network structure to obtain the fusion picture. The fusion picture, infrared image and visible light image are calculated by formula (10) to calculate the total loss

[0089]

[0090] Where, is the intensity loss, is the gradient loss, represents the Sobel gradient operator, |·| is the absolute value operator, max(·) represents the maximum operation at the pixel level, μ is a parameter, which is used to balance the loss.

[0091] In order to make the fused image more close to the source image in intensity and gradient, and update the parameters of the neural network using the gradient backpropagation algorithm according to the total loss, the updated neural network is obtained.

[0092] 4) In the test stage, the infrared image and visible light image of the test set are input into the updated neural network to obtain the fusion image.

[0093] Example 1

[0094] Experiments are implemented on four datasets: TNO, RoadScene, MSRS and M3FD. The network is trained on the MSRS training set (1083 pairs), comparative experiments are conducted on the test set (361 pairs), and generalization experiments are conducted on the remaining three datasets. The parameters λ, η, β, μ are set to 1, 1, 1, 10 respectively.

[0095] The fusion results are comprehensively compared in multiple aspects by using 6 indicators, including entropy (EN), spatial frequency (SF), average gradient (AG), mutual information (MI), visual information fidelity (VIF) and QAB / F. The present application is compared with the most advanced methods, including DIDFuse, U2Fusion, SDNet, TarDAL, DeFusion, ReCoNet and UMFusion. The auxiliary semantic segmentation network is trained using the infrared and visible images of MSRS, and the data set is divided according to the original article. The auxiliary object detection network is trained on the infrared and visible images of M3FD, and the training set and test set are divided by 9:1. The segmentation and detection networks both use MoblienetV2 as the backbone, with segmentation heads and detection heads. All experiments are conducted on a PC with a single NVIDIA GeForce RTX 3090 GPU. The experimental results are shown in Table 1 and Table 2. Figure 3 and Table 1.

[0096] Table 1 Infrared and visible image fusion experiment

[0097]

[0098] Figure 3 (a)-(j) in Table 1, Figure 4 (a)-(j) in Table 1, Figure 5 (a)-(j) in Table 1 and Figure 6Figs. (a)-(j) show the comparison of fusion results of different scenarios, which are “00191D,00718N” in MSRS, “FLIR 06430” in RoadScene and “00232” in M3FD, respectively. “00191D” is a daytime scene, the thermal radiation information from infrared image is used to supplement the visible light information and maintain the texture information between the fused images. As can be seen from the figure, only DeFusion and the method of the present application can well retain the thermal radiation information, such as the two people on the left side of the image, while the method of the present application retains better texture and details. “00718N” is a night scene with less than ideal lighting conditions, in which case the visible light image can basically see no target, while the combination of infrared image and visible light image can help better understand the scene. As shown in the figure, the fusion result of the present application has more prominent target highlighting, and has better texture details in low light conditions, such as the two highlighted parts in the image. “FLIR 06430” and “00232” demonstrate the generalization performance on daytime and nighttime scenes, respectively. Due to the superior texture retention performance of the method of the present application, in “FLIR 06430”, the present application has the clearest bus stop sign, while in “00232”, the present application fuses the image with clear car texture. In addition, the method of the present application has obvious advantages in highlighting small targets, which is reflected in both “FLIR 06430” and “00232”. In Table 1, it can be seen that the six indicators of the method of the present application show obvious advantages in the four data sets. The highest EN indicates that the fused image of the present application has the most information, the highest SF and AG indicate that the generated image contains more texture and details. The most information is transferred from the source image pair, so the method of the present application shows the highest MI, in addition, the highest QAB / F proves that the method of the present application retains the most edge information. At the same time, the method of the present application achieves the highest VIF, proving that the fused image of the present application is more compatible with the human visual system.

[0099] In addition, the present application also explores the promotion of image fusion to downstream tasks, including multi-modal semantic segmentation and multi-modal object detection, taking Deeplabv3+ and Yolov5 as benchmark models, respectively. The images fused by different fusion methods are used to train the semantic segmentation network and the target detection network, and their mIOU and mAP are compared, the results of which are shown in Tables 2 and 3.

[0100] Table 2 Multi-modal semantic segmentation

[0101]

[0102] Table 3 Multi-modal object detection

[0103]

[0104] From the experimental results, it can be seen that the method of the present application achieves the highest score in most categories and mIOU and mAP, proving that the method of the present application can indeed help downstream tasks.

[0105] The present application designs a novel image fusion network through deep unfolding, and each layer of the image fusion network is constructed through single-step iteration of an optimization algorithm. Starting from the optimization objective considering the downstream task related clauses, the proximal operator of each layer completes the task of obtaining and updating the memory unit from the memory enhancement module. In addition, adjustment of attention belonging to the downstream task is introduced to assist the network to pay more attention to the area that is helpful to the downstream task. The source image is simply initialized and sent into the network model, and finally the fusion result is obtained through layer-by-layer modules. The loss function is set to let the network retain the maximum intensity and gradient information of the source image, so that the fused image has target highlighting and texture details. The qualitative and quantitative experiments on the four data sets of TNO, RoadScene, MSRS and M3FD show that the present application produces high-quality fused images. The experiments of applying the fused image to the downstream task prove the adaptability of the method of the present application to the downstream task. In the future, it is hoped that the algorithm will be applied to more downstream tasks, and the present application will be extended to more image fusion, image generation or image reconstruction tasks.

[0106] The above only describes the best embodiments of the present application, but cannot be understood as limiting the claims. The present application is not limited to the above embodiments, and the specific structure allows changes. Any changes made within the protection scope of the independent claims of the present application are within the protection scope of the present application.

[0107] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which the present application belongs. The terms used in the specification of the present application herein are only for the purpose of describing specific embodiments and are not intended to limit the present application. The term "and / or" as used herein includes any and all combinations of one or more of the associated listed items.

Claims

1. A memory-enhanced deep unfolding multimodal image fusion method with enhanced downstream tasks, characterized by: The following steps are involved: 1) Collect infrared images and visible light images; and divide the infrared images and visible light images into training sets and test sets; 2) Establish the optimization target, use the gradient projection method to solve the optimization target, and obtain the iterative formula; The neural network is used to replace the proximal operator in the iterative formula to obtain the neural network structure; 3) Feed the training set into the neural network to obtain a fused image, and calculate the total loss of the fused image, infrared image, and visible light image; According to the total loss, the parameters of the neural network are updated using the gradient back propagation algorithm to obtain the updated neural network; 4) Input the infrared image and visible light image of the test set into the updated neural network to obtain a fused image; The optimization objectives are as follows: Among them, DS(·) is the score term related to the downstream task, h(·) is the prior term, λ is the first weight parameter, η is the second weight parameter, and I f is the fused image.

2. The memory-enhanced deep unfolding multimodal image fusion method with downstream task enhancement according to claim 1, characterized in that: The iteration formula is as follows: Among them, ρ k is the learnable step size, prox h (·) is the proximal operator corresponding to the prior term h(·), Indicates the intermediate result of k-1 stage gradient; Intermediate results of k-1 stages The gradient of is calculated as follows:

3. The memory-enhanced deep unfolding multimodal image fusion method with downstream task enhancement according to claim 1, characterized in that: The proximal operator prox corresponding to the prior term h(·) h (·) It consists of a convolutional layer, an attribution attention module, and a memory enhancement module; Convolutional layer, used for convolution; The attribution attention module is used to process the attribution graph and the convolution result to obtain the fusion result; The memory enhancement module is used to strengthen the information interaction between different stages of iteration.

4. The memory-enhanced deep unfolding multimodal image fusion method with downstream task enhancement according to claim 1, characterized in that: The fusion image of the kth stage Obtained through the following process: The k-1 stage fusion result of the input Perform a gradient update to obtain the first intermediate feature First intermediate feature After the convolution layer extracts features, the second intermediate feature is obtained Second intermediate feature with k-1 stages of short-term memory and long-term memory Splicing and processing using convolution layer to obtain the third intermediate feature The third intermediate feature after the attribution map att-map is operated by the Sigmoid function and after a layer of convolution Perform element-by-element multiplication, and then multiply the product with the third intermediate feature Add together to obtain the fourth intermediate feature The fourth intermediate feature After another layer of convolution compression channel, the fusion image of the kth stage is obtained 5. The memory-enhanced deep unfolding multimodal image fusion method with downstream task enhancement according to claim 4, characterized in that: The i-th pixel of the attribution map att-map is calculated as follows: DS(I)=Score SS (I)+βScore OD (I) (5) Among them, β is the weight parameter, Score SS (I) and Score OD (I) are the scores of semantic segmentation and object detection respectively, and I is the input.

6. The memory-enhanced deep unfolding multimodal image fusion method with downstream task enhancement according to claim 5, characterized in that: Semantic segmentation score SS (I) is calculated by the following formula: Object detection score OD (I) is calculated by the following formula: in, Represents the discriminant score of the kth class of the semantic segmentation network of the (i, j) pixel in the input I, where i represents the horizontal coordinate of the position and j represents the vertical coordinate of the position.

7. The memory-enhanced deep unfolding multimodal image fusion method with downstream task enhancement according to claim 4, characterized in that: k-1 stage short-term memory Calculated by the following formula: in, It is the fourth intermediate feature of the k-1 stage.

8. The memory-enhanced deep unfolding multimodal image fusion method with downstream task enhancement according to claim 4, characterized in that: k-1 stage long-term memory Calculated by the following formula: Among them, z (k-1) h is the feature after the intermediate features of the k-1th stage of the neural network are spliced ​​together, (k-1) is the output of the Conv-LSTM at the k-1th stage, h (k-2) is the output of the Conv-LSTM at stage k-2, c (k-1) is the cell state of the Conv-LSTM in the k-1th stage, c (k-2) is the cell state of the Conv-LSTM at the k-2 stage, is the second intermediate feature of the k-1 stage, is the third intermediate feature of the k-1 stage, It is the fourth intermediate feature of the k-1 stage.

9. The memory-enhanced deep unfolding multimodal image fusion method with downstream task enhancement according to claim 1, characterized in that: Total loss Calculated by the following formula: in, is the strength loss, is the gradient loss, represents the Sobel gradient operator, |·| is the absolute value operator, and max(·) represents the maximum operation at the pixel level.

Images