An image fusion method and device based on dynamic relative enhancement

Abstract

The application discloses a kind of based on dynamic relative enhancement's image fusion method and device, comprising: the feature of multiple-source image is input to fusion network, generates fusion image;Calculate fusion loss and perform back propagation, train fusion base network, obtain fusion base network parameter;Freeze fusion base network parameter, insert cross-modal enhancement module based on hint into fusion base network, obtain image fusion network based on dynamic relative enhancement;Degraded visible light image is input to image fusion network based on dynamic relative enhancement, obtain the feature of visible light image and infrared image, input feature to cross-modal enhancement module based on hint;After obtaining fusion image, respectively with clear image that has not been degraded loss calculation is carried out, and back propagation is carried out;The trained image fusion network based on dynamic relative enhancement is deployed, and the image fusion network based on dynamic relative enhancement completes degradation repair and image fusion task.The device comprises: processor and memory.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of image fusion, in particular to an image fusion method and device based on dynamic relative enhancement. BACKGROUND

[0002] Image fusion aims to integrate multi-source image information acquired by different sensors to generate a comprehensive single image. Multi-modal (visible light and infrared) image fusion has been widely used in automatic driving, unmanned aerial vehicles, forest fire monitoring and other fields. Images captured by different imaging sensors often have different characteristics. Infrared images can adapt to different lighting conditions, but usually lack detailed texture; while visible light images can capture rich colors and details in sufficient light, but often have quality degradation in complex environments. Image fusion takes advantage of the complementary information between modalities, retaining the advantages of visible light and infrared images while minimizing their limitations. However, in real-world environments, visible light and infrared images are often affected by noise and blurring (e.g., Gaussian noise and JPEG noise) due to sensor failure or external interference, resulting in degraded image quality.

[0003] To enhance low-quality images, some existing methods have achieved significant results by learning the distribution of the data set. Most existing methods treat image fusion and image restoration as two independent tasks. Recently, some research attempts to handle both tasks in one framework, as they have similar requirements in extracting effective information. From the perspective of multi-modal complementarity, image fusion can extract dominant information from different modalities, further improving the effect of image restoration. Therefore, in real-world scenarios, it is necessary to integrate these two tasks to capture their inherent consistency. SUMMARY

[0004] The present application provides an image fusion method and device based on dynamic relative enhancement, and introduces a dynamic relative enhancement image fusion framework to jointly improve the performance of image fusion and enhancement. The natural multi-modal complementarity of image fusion is introduced to enhance the quality of cross-modal images, especially in the disadvantaged areas of each modality. A simple and easily integrated module, called a cross-modal enhancement module based on prompts, is designed. This module captures the relative advantages of each modality, thereby promoting cross-modal enhancement in relatively weak quality areas, generating enhanced multi-modal features. Based on the mutual guidance between modalities, the degradation problem and the image fusion problem can be handled simultaneously, as described in detail below.

[0005] In a first aspect, a method of image fusion based on dynamic relative enhancement, the method comprising the following steps:

[0006] The visible light image and the infrared image are input into a double-flow backbone network; the features of the multi-source images are input into a fusion network to generate a fusion image; a fusion loss is calculated to perform back propagation, and the fusion base network is trained to obtain fusion base network parameters;

[0007] The fusion base network parameters are frozen, a prompt-based cross-modal enhancement module is inserted into the fusion base network to obtain an image fusion network based on dynamic relative enhancement; the degraded visible light image is input into the image fusion network based on dynamic relative enhancement to obtain features of the visible light image and the infrared image, and the features are input into the prompt-based cross-modal enhancement module;

[0008] After obtaining the fusion image, loss calculation is performed with the clear image without degradation, and back propagation is performed; the trained image fusion network based on dynamic relative enhancement is deployed, and the image fusion network based on dynamic relative enhancement completes the degradation repair and image fusion task.

[0009] The prompt-based cross-modal enhancement module includes two components: a relative dominance component and a cross-modal enhancement component.

[0010] The relative dominance component is calculated by integrating the features of multiple modalities into a gating network;

[0011] Given a visible light feature F v and an infrared image feature F I , the relative dominance RD i is calculated by a relative dominance gate G:

[0012] RD i =G([F i ;F v ]

[0013] Where G(x) = σ(Conv(x)), σ is an activation function, given the relative dominance RD m , the modal prompt information p m containing the relative dominance prompt is obtained F m ·RD m .

[0014] The cross-modal enhancement component is: the prompt information p m is transmitted to two convolutional layers and a Sigmoid layer, and the output will be used as an attention map The attention map represents the dominant information of another modality, guiding the reconstruction process, as a prompt information to enhance the performance of the disadvantaged modality.

[0015] The method further includes perception and repair of unknown degradation types.

[0016] Based on the current characteristics, generate a self-improving characteristic: SEF i =Cov(Re(∑ c z·σ(∑ H ∑ W (F i )))), where z is a learnable parameter, σ is the activation function, and Re represents the bilinear upsampling operation; As a cross-modal cue, it assists in feature enhancement and repair, obtaining relatively enhanced features. SA stands for Self Attention Module, and FFN stands for Linear Layer.

[0017] A second aspect is an apparatus for image fusion based on dynamic relativity enhancement, the apparatus comprising: a processor and a memory, the memory storing program instructions, the processor invoking the program instructions stored in the memory to cause the apparatus to perform the method described in any of the first aspects.

[0018] Third aspect, a computer-readable storage medium storing a computer program, the computer program including program instructions that, when executed by a processor, cause the processor to perform the method described in any one of the first aspects.

[0019] The beneficial effects of the technical solution provided by this invention are:

[0020] 1. This invention utilizes the principle of complementarity between modes to design an image fusion method based on dynamic relativity enhancement. This method integrates image fusion and image inpainting into a single network, using the relative dominance generated by image fusion to aid image inpainting, and enhancing the robustness of image fusion through image inpainting.

[0021] 2. This invention designs a prompt-based cross-modal enhancement module for modal feature enhancement and repair, which is plug-and-play. The module captures the relatively dominant region of each mode to specifically enhance defective regions in other modalities. The module can be easily and flexibly integrated into a general fusion model, thereby effectively improving the fusion performance of degraded images.

[0022] 3. This invention enables the model to perceive and repair unknown degradation types through learnable parameters, thereby achieving the ability of one model to repair multiple degradation types, thus enhancing the practical application capability of image fusion.

[0023] 4. This invention designs an image-based scalable inference method. Based on the Transformer mechanism, during the inference process, any image size can be input into the network for inference, which increases the usability and applicable scenarios of this method in practical applications. Attached Figure Description

[0024] Figure 1 This is a flowchart of the image fusion method based on dynamic relativity enhancement proposed in this invention;

[0025] Figure 2 This is a schematic diagram of the cross-modal enhancement module of the present invention;

[0026] Figure 3 This is a schematic diagram of the image fusion network structure based on dynamic relativity enhancement proposed in this invention. Detailed Implementation

[0027] To make the objectives, technical solutions, and advantages of the present invention clearer, the embodiments of the present invention will be described in further detail below.

[0028] To address the technical problems existing in the background art, this invention proposes an image fusion network based on dynamic relative enhancement. This network enhances and repairs weak areas of the image by dynamically perceiving the relative dominance between different modalities during the image fusion process, thereby further improving the model's performance. This network can perform image fusion and image inpainting through a single model; image fusion guides image inpainting, and image inpainting enhances the robustness of image fusion. This network structure effectively solves the technical problems existing in the background art, better combining these two tasks, thereby improving the performance and robustness of image fusion.

[0029] like Figure 1 As shown, an image fusion network based on dynamic relativity enhancement consists of two parts, and the functions of each part are as follows:

[0030] In the image fusion foundation network, this embodiment of the invention includes a dual-stream backbone network and a fusion network. The dual-stream backbone network consists of multiple Transformers, including an encoder and a decoder. The encoder extracts features from the multi-source images, and the decoder restores the features.

[0031] Furthermore, feature extraction from multi-source images can be represented as: Given a visible light image and infrared images Its features for each modality can be expressed by the following formula:

[0032]

[0033] in, Represents the corresponding features of a given mode. Indicates decoder, denotes the encoder, i denotes the infrared modality, v denotes the visible light modality, m denotes one of the set of infrared-visible light modalities, F l denotes the feature obtained by the encoder, F h denotes the feature obtained by the decoder, H and W denote the height and width of the image.

[0034] The target of the embodiment of the present application is to fuse the complementary information of two original images to generate a fused image I f Therefore, the fusion network can be represented as Where [;] denotes the concatenation operation, denotes the fusion network, F i denotes the feature of the infrared modality, F v denotes the feature of the visible light modality.

[0035] Further, the prompt-based cross-modal enhancement includes two components: a relative dominance component and a cross-modal enhancement component. Wherein, the relative dominance of each modality can be calculated by integrating the features of multiple modalities into the input of a gating network. In practical applications, taking infrared as an example, given a visible light feature F v and an infrared image feature F I , the relative dominance RD i of the two can be calculated by a relative dominance gate G: RD i = G([F i ; F v ], where G(x)=σ(Conv(x)), σ is an activation function. Given the relative dominance RD m , further, the modality prompt information p m containing the relative dominance prompt can be obtained: p m =F m ·RD m . Where · is a dot product operation.

[0036] Further, the cross-modal image enhancement is as shown in Figure 2 The prompt information p m is passed to two convolutional layers and a Sigmoid layer. The output thereof will be used as an attention map This attention map represents the dominant information of another modality, and points out the insufficient part that needs to be enhanced. It can guide the reconstruction process and serve as a prompt information to enhance the performance of the disadvantaged modality.

[0037] Further, as shown in Figure 2 , the perception and repair specific process of unknown degradation type is: according to the current feature, a self-enhanced feature SEF i =Cov(Re(∑ c z·σ(∑ H ∑W (F i )))),where z is a learnable parameter, is an activation function, and Re denotes a bilinear up-sampling operation. As a cross-modal prompt to assist in the enhancement and repair of features, the disadvantaged features can be improved to obtain relatively enhanced features where SA is a self-attention module, and FFN is a linear layer.

[0038] Embodiment 1

[0039] The embodiment of the application provides an image fusion method based on dynamic relative enhancement, which comprises the following steps:

[0040] 101: The image fusion base network is composed of a double-flow backbone network and a fusion network;

[0041] The double-flow backbone network is composed of multiple Transformers, including an encoder and a decoder. The structure of each flow is the same, but the parameters are not shared. The fusion network is composed of multiple Transformer layers and a convolution layer.

[0042] Specifically, the encoder in the double-flow backbone network is responsible for feature extraction of multi-source images, and the decoder is responsible for feature representation restoration. The fusion network is used to fuse the features extracted by the double-flow backbone network to finally generate a fusion image.

[0043] 102: input the clear visible light image and the infrared image into the double-flow backbone network;

[0044] The double-flow backbone network encodes the multi-source images through the encoder to obtain encoded features, and then inputs the encoded features into the decoder to obtain decoded features.

[0045] 103: input the features of the multi-source images into the fusion network to generate a fusion image;

[0046] After the features of the multi-source images are connected, they are input into the fusion network. After passing through multiple TransformerBlocks (Transformer modules), the features are finally reconstructed into a 3-channel fusion image through a convolution layer.

[0047] 104: after obtaining the fusion image, calculate the fusion loss, perform back propagation, and then train the fusion base network to obtain the fusion base network parameters;

[0048] The fusion loss function used when calculating the fusion loss includes: maximum pixel loss (Max-Pixel), maximum gradient loss (Max-Grad), structural consistency loss (SSIM), and color consistency loss (Color).

[0049] 105: Freeze the fusion base network parameters, insert the prompt-based cross-modal enhancement module into the fusion base network, and obtain an image fusion network based on dynamic relative enhancement.

[0050] Specifically, a prompt-based cross-modal enhancement module is added to each layer of the decoder.

[0051] 106: Randomly degrade clear visible light and infrared images to obtain degraded visible light images and degraded infrared images.

[0052] The random degradation type includes one of Gaussian noise, Poisson noise, speckle noise, JEPG compression, down-sampling compression, and dynamic blur.

[0053] 107: Input the degraded visible light image into the image fusion network based on dynamic relative enhancement to obtain features of the visible light image and the infrared image, input the features into the prompt-based cross-modal enhancement module, and perform feature enhancement.

[0054] The number of network layers of the decoder is the same as the number of prompt-based cross-modal enhancement modules. Therefore, the network includes a plurality of steps 105-107. Finally, the enhanced features are input into the fusion base network.

[0055] 108: After obtaining the fused image, loss calculation is performed with the clear image without degradation, and back propagation is performed.

[0056] Specifically, the loss calculation is consistent with step 104.

[0057] 109: Deploy the trained image fusion network based on dynamic relative enhancement. The image fusion network based on dynamic relative enhancement can complete the degradation repair and image fusion task. By using the learnable parameters to perceive the degradation type, the image fusion network based on dynamic relative enhancement can repair multiple degradation types without knowing the specific degradation type. Repair and fusion can be efficiently integrated into one model, saving memory and computing overhead, and promoting the development of the image fusion field and the image repair field.

[0058] In summary, the embodiment of the present application gradually forms a network with repair capability by two parts of the image fusion base network and the prompt-based image prompt module, generates a fused image; uses the dominance of image fusion to generate a dominant prompt area to guide image enhancement; uses image enhancement to improve the result and robustness of image fusion; uses prompt learning to enable the model to perceive multiple types of degradation without knowing the degradation type in advance. By combining image fusion and image enhancement, one method can complete two tasks, improve task performance, and save computing time and computing overhead.

[0059] Embodiment 2

[0060] The scheme in Embodiment 1 is further introduced in combination with specific examples and calculation formulas, and details are described below:

[0061] I. Data preparation

[0062] The experiments of the embodiment of the present application are performed on two publicly available datasets, LLVIP and MSRS.

[0063] LLVIP is a visible-infrared paired dataset for low-light vision tasks. It contains 15,488 pairs of data, most of which are taken in very dark scenes. All images are strictly aligned in time and space. Pedestrians in the dataset are labeled. The dataset is fine and standardized, and can be used to estimate and improve the performance of existing image fusion algorithms, low-light human body detection and image translation methods.

[0064] MSRS is a multispectral dataset for infrared and visible image fusion, containing 1,444 pairs of high-quality, aligned infrared and visible images. The dataset is derived from the MFNet dataset, and after removing unaligned image pairs, it contains 715 pairs of daytime images and 729 pairs of nighttime images.

[0065] II. Image fusion network structure based on dynamic relative enhancement

[0066] The image fusion network process based on dynamic relative enhancement is shown in Figure 1 , which consists of two parts, and the specific network details are shown in Figure 3 . In the image fusion base network, the embodiment of the present application includes a dual-stream backbone network and a fusion network. The dual-stream backbone network is composed of multiple Transformers, which include an encoder and a decoder. The encoder extracts features from multi-source images, and the decoder restores the features.

[0067] Further, the feature extraction of the multi-source image can be represented as Given a visible light image and an infrared image The features for each modality can be expressed by the following formula:

[0068]

[0069] wherein, denotes the corresponding feature of a given modality. The goal of the embodiment of the present application is to fuse the complementary information of the two original images to generate a fused image I f Therefore, the fusion network can be represented as where [;] denotes concatenation operation.

[0070] Further, the prompt-based cross-modal enhancement includes two components: relative dominance and cross-modal enhancement. The extraction of relative dominance of each modality is achieved by integrating the features of multiple modalities into the input of the gating network. Specifically, taking infrared as an example, given a visible light feature F v and an infrared image feature F I , the relative dominance RD i of the two modalities can be calculated by the relative dominance gating G: RD i = G ([F i ; F v ], where G (x) = σ (Conv (x)), and σ is an activation function. Given the relative dominance RD m , the modality prompt information p m containing the relative dominance prompt can be further obtained: p m = F m · RD m , where · is the dot product operation.

[0071] Further, the cross-modal image enhancement is shown in Figure 2 The prompt information p m is passed to two convolutional layers and a sigmoid layer. The output will be used as the attention map This attention map represents the dominant information of the other modality and points out the insufficient part that needs to be enhanced. It can guide the reconstruction process and serve as a prompt information to enhance the performance of the disadvantaged modality.

[0072] Further, as shown in Figure 2 , the perception and repair of the specific process of unknown degradation type is: according to the current feature, a self-enhanced feature SEF i = Cov (Re (∑ c z·σ (∑ H ∑ W (F i ))), where z is a learnable parameter, σ is an activation function, and Re represents a bilinear up-sampling operation. The is used as a cross-modal prompt to assist the enhancement and repair of the feature, and the performance of the disadvantaged feature is improved, and the relatively enhanced feature can be obtained, where SA is a self-attention module, and FFN is a linear layer.

[0073] III. Evaluation index and protocol

[0074] The embodiments of the present application are used to verify the effectiveness of the method with 5 indicators. Entropy (EN), gradient-based metric Qabf, structural similarity (SSIM), mutual information (MI), and fused visual information fidelity (VIFF).

[0075] IV. Details of the use of the model

[0076] 1. Data augmentation: Due to limited computing resources, the embodiments of the present application use random flipping and cropping methods to improve the diversity of data. Specifically: first, the visible light and infrared images are randomly cropped to 96x96 size, and then randomly flipped horizontally and vertically with a probability of 50% for visible light and infrared images.

[0077] 2. Model optimization: the batch size during training in the embodiments of the present application is 12, and the AdamW optimization algorithm with β1=0.9, β2=0.999 is used, and the learning rate is reduced from the initial 1x10 -6 In the middle, the learning rate is reduced to 1x10 -8 .

[0078] 3. Hyperparameter setting: the number of cross-modal enhancement modules based on hints in the embodiments of the present application is 4.

[0079] 4. Loss setting: the loss in the embodiments of the present application is an unsupervised loss. It is composed of maximum pixel loss (Max-Pixel), maximum gradient loss (Max-Grad), structural consistency loss (SSIM) and color consistency loss (Color), and the final loss is obtained by weighted sum of the four in the ratio of 4:1:1:10.

[0080] The embodiments of the present application provide a dynamic relative enhancement image fusion framework, aiming to jointly improve the performance of image fusion and enhancement. The embodiments of the present application introduce the natural multi-modal complementarity of image fusion to enhance the quality of cross-modal images, especially in the disadvantaged areas of each modality. The embodiments of the present application design a simple and easy-to-integrate module called cross-modal promotion enhancement. This module captures the relative advantages of each modality, and then promotes cross-modal enhancement in the relatively weak quality area, thereby generating enhanced multi-modal features. The embodiments of the present application can simultaneously handle degradation and image fusion problems based on mutual guidance between modalities.

[0081] The embodiments of the present application have the following three key points:

[0082] I. An image fusion method based on dynamic relative enhancement is proposed

[0083] Technical effects: This method integrates image fusion and image inpainting into one network, uses the relative dominance generated by image fusion to help image inpainting, and enhances the robustness of image fusion through image inpainting.

[0084] II. A prompt-based cross-modal enhancement module is proposed

[0085] Technical effects: This module is used for prompt cross-modal enhancement and has the characteristics of plug and play. The module captures the relatively dominant area of each mode to specifically enhance the defect area in other modalities. The design makes the module flexible and easy to integrate into a general image fusion model, thereby effectively improving the fusion performance of degraded images.

[0086] III. A blind inpainting method is proposed

[0087] Technical effects: This method can perceive and repair unknown degradation types through learnable parameters, so that a model can perform blind inpainting of multiple degradation types, thereby enhancing the practical application ability of image fusion.

[0088] IV. An image scalable inference method is proposed

[0089] Technical effects: For the mechanism of Transformer, small block images are used for training during the training process, and during the inference process, any size of image can be input into the network for inference, increasing the usability and practical application significance of the actual use scene.

[0090] Embodiment 3

[0091] The method proposed in the embodiments of the present application is compared with a plurality of prior art methods on a multi-modal data set to verify the performance of image fusion.

[0092] I. Comparison of multi-modal data sets without degradation.

[0093] As shown in Table 1, the embodiments of the present application are evaluated on the LLVIP and MSRS datasets through five evaluation indicators. On the LLVIP dataset, the method of the embodiments of the present application outperforms other comparative methods on all five indicators, showing significant advantages, especially on the Qabf, SSIM, MI and VIFF indicators. Specifically, the highest EN and MI scores indicate that the present method can retain the most information, benefiting from its ability to jointly enhance non-dominant regions. Higher Qabf scores reflect better alignment of local gradients and intensities between the source images and the fused image, indicating that the present method can better retain valuable information from multiple modalities. In addition, higher VIFF scores indicate that the fused image retains a large amount of visual information in the source images, indicating superior fusion quality. The SSIM score measures structural similarity, indicating that the present method can retain more structural details. These qualitative results demonstrate that the present method achieves excellent fusion performance in dynamic relative enhancement of image quality.

[0094] Table 1

[0095]

[0096] II. Degraded multi-modal dataset comparison.

[0097] For degraded data, the experimental results are shown in Table 2. The results show that even without a dedicated restoration model, the present method can still achieve competitive performance. Specifically, EN, SSIM and VIFF are close to the best values, while Qabf and MI reach the best scores. Higher MI and VIFF scores indicate that the present method can effectively retain mutual information between source images, ensuring that necessary details and contextual information are retained in the final output while maintaining subjective visual quality. This performance highlights the ability of the present method to effectively maintain image authenticity and enhance degraded features without relying on a dedicated restoration model.

[0098] Table 2

[0099]

[0100] Embodiment 4

[0101] An apparatus for image fusion based on dynamic relative enhancement, the apparatus comprising: a processor and a memory, the memory storing program instructions, the processor invoking the program instructions stored in the memory to cause the apparatus to perform the following method steps in Embodiment 1:

[0102] inputting the visible light image and the infrared image into the dual-stream backbone network; inputting the features of the multi-source images into the fusion network to generate a fused image; calculating a fusion loss to perform backpropagation and train the fusion base network to obtain fusion base network parameters;

[0103] Freeze the fusion base network parameters, insert the prompt-based cross-modal enhancement module into the fusion base network to obtain an image fusion network based on dynamic relative enhancement; input the degraded visible light image into the image fusion network based on dynamic relative enhancement to obtain the features of the visible light image and the infrared image, and input the features into the prompt-based cross-modal enhancement module;

[0104] After obtaining the fusion image, loss calculation is performed with the clear image without degradation, and back propagation is performed; the trained image fusion network based on dynamic relative enhancement is deployed, and the image fusion network based on dynamic relative enhancement completes the degradation repair and image fusion task.

[0105] The prompt-based cross-modal enhancement module includes two components: a relative dominance component and a cross-modal enhancement component.

[0106] The relative dominance component is calculated by integrating the features of multiple modalities into a gating network;

[0107] Given a visible light feature F v and an infrared image feature F I , the relative dominance RD i is calculated by a relative dominance gate G:

[0108] RD i = G([F i ; F v ]

[0109] Where G(x) = σ(Conv(x)), σ is an activation function, given the relative dominance RD m , the modal prompt information p m containing the relative dominance prompt is obtained F m · RD m .

[0110] The cross-modal enhancement component is: the prompt information p m is passed to two convolutional layers and a Sigmoid layer, and the output will be used as an attention map The attention map represents the dominant information of another modality, guiding the reconstruction process, as a prompt information to enhance the performance of the disadvantaged modality.

[0111] The device further includes perception and repair of unknown degradation types.

[0112] According to the current feature, a self-improving feature SEF i is generated Cov(Re(∑ c z·σ(∑ H ∑W (F i )))),where z is a learnable parameter, σ is an activation function, Re denotes a bilinear up-sampling operation; the As a cross-modal prompt to assist in the enhancement and repair of features, the relatively enhanced features where SA is a self-attention module, and FFN is a linear layer.

[0113] It should be pointed out here that the device description in the above embodiments corresponds to the method description in the embodiments, and the embodiments of the application will not be repeated here.

[0114] The execution subject of the processor and the memory described above can be a computer, a single-chip microcomputer, a microcontroller, etc. having a computing function, and in a specific implementation, the embodiments of the application do not limit the execution subject, which is selected according to the actual application needs.

[0115] The memory and the processor transmit data signals through a bus, and the embodiments of the application will not be repeated here.

[0116] Based on the same inventive concept, the embodiments of the application also provide a computer-readable storage medium, which includes a stored program, and when the program runs, controls the device where the storage medium is located to execute the method steps in the above embodiments.

[0117] The computer-readable storage medium includes, but is not limited to, a flash memory, a hard disk, a solid-state disk, etc.

[0118] It should be pointed out here that the readable storage medium description in the above embodiments corresponds to the method description in the embodiments, and the embodiments of the application will not be repeated here.

[0119] In the above embodiments, all or part of them can be realized by software, hardware, firmware or any combination thereof. When realized by software, all or part of them can be realized in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions according to the embodiments of the application are generated.

[0120] The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions can be stored in or transmitted by a computer-readable storage medium. The computer-readable storage medium can be any available medium accessible by a computer or a data storage device such as a server, data center, etc. integrated with one or more available media sets. The available medium can be a magnetic medium or a semiconductor medium, etc.

[0121] The model of each device is not limited unless otherwise specified, and any device that can complete the above functions can be used.

[0122] Those skilled in the art can understand that the drawings are only schematic diagrams of a preferred embodiment, and the above embodiment numbers are only for description, not representing the advantages and disadvantages of the embodiments.

[0123] The above is only the preferred embodiment of the present application, and is not intended to limit the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principles of the present application shall be included in the protection scope of the present application.

Claims

1. A method of image fusion based on dynamic relative enhancement, characterized in that, The method comprises the following steps: inputting the visible light image and the infrared image into a double-flow backbone network; inputting the features of the multi-source image into a fusion network to generate a fused image; calculating a fusion loss to perform back propagation and train the fusion base network to obtain fusion base network parameters; freezing the fusion base network parameters, inserting a prompt-based cross-modal enhancement module into the fusion base network to obtain an image fusion network based on dynamic relative enhancement; inputting the degraded visible light image into the image fusion network based on dynamic relative enhancement to obtain features of the visible light image and the infrared image, and inputting the features into the prompt-based cross-modal enhancement module; after obtaining the fused image, loss calculation is performed with the clear image without degradation, and back propagation is performed; the image fusion network based on dynamic relative enhancement is deployed after training, and the image fusion network based on dynamic relative enhancement completes the degradation repair and image fusion task.

2. The method of image fusion based on dynamic relative enhancement according to claim 1, characterized in that, The prompt-based cross-modal enhancement module comprises two components: a relative dominance component and a cross-modal enhancement component.

3. The method of image fusion based on dynamic relative enhancement according to claim 2, characterized in that, The relative dominance component is calculated by integrating the features of multiple modalities into a gating network; Given a visible light feature and an infrared image feature their relative dominance is computed by the relative dominance gate Computes: ; wherein, , is an activation function, given the relevant dominance , obtaining modal cue information containing a relevant dominance cue .

4. The method of image fusion based on dynamic relative enhancement according to claim 2, characterized in that, The cross-modal enhancement component is: prompt information Pass to two convolutional layers and a sigmoid layer, the output will be as an attention map The attention map represents the dominant information of another modality, Guides the reconstruction process and enhances the performance of the disadvantaged modality as prompt information.

5. The method of image fusion based on dynamic relative enhancement according to claim 1, characterized in that, The method further comprises perception and repair of unknown degradation types. According to the current feature, a self-enhancing feature is generated: wherein is a learnable parameter, is an activation function, represents a bilinear up-sampling operation; and is obtained as a cross-modal prompt to assist the enhancement and repair of the feature, and the relatively enhanced feature wherein is a self-attention module, is a linear layer.

6. An apparatus for image fusion based on dynamic relative enhancement, the apparatus comprising: The device comprises a processor and a memory, and the memory stores program instructions; the processor invokes the program instructions stored in the memory to enable the device to execute the method of any one of claims 1-5.

7. A computer readable storage medium characterized in that, The computer readable storage medium stores a computer program, and the computer program comprises program instructions; when the program instructions are executed by the processor, the processor executes the method of any one of claims 1-5.

Images