Mask-guided image restoration and fusion model, its training method and usage

Abstract

This application belongs to the field of image processing technology and discloses a mask-guided image restoration and fusion model, its training method, and its usage method. The model uses an encoder to encode features of infrared and visible light images to obtain preliminary features and generate preliminary fusion features. Then, multiple cascaded mask-guided feature extraction modules dynamically generate masks based on the input features and perform cross-modal cross-attention calculations under the guidance of the masks. Finally, the final infrared, visible light, and fusion features are output through the first HTB module, the first residual module, and the output layer. This model overcomes the problems of high network optimization difficulty, significant error accumulation, and insufficient generalization ability caused by "pseudo-Ground Truth" constraint training in existing two-stage "restoration + fusion" schemes.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing, in particular to a mask-guided image restoration and fusion model and a training method and use method thereof. BACKGROUND

[0002] Multi-modal image fusion (MMIF) technology aims to generate a fused image containing richer scene details and stronger semantic expression ability by integrating image information from different modalities. For example, infrared and visible light image fusion, visible light images can provide rich texture information in the scene, while infrared images can highlight the thermal radiation information of the target. By effectively fusing this information, the performance in practical applications such as target detection, semantic segmentation, and depth estimation can be significantly improved.

[0003] However, in complex real-world scenarios (such as the presence of noise, bad weather, or blur, etc.), multi-modal image fusion faces great challenges. The existing technology often adopts a two-stage scheme of "restoration + fusion", i.e., first restoring the features of each modality image, and then performing fusion. This method has significant problems: first, the network optimization is difficult, the restoration network focuses on the restoration of intra-modality features, while the fusion network needs to learn cross-modality complementary features, and the inconsistent optimization goals lead to unstable training and convergence difficulties. Second, error accumulation is obvious, the error generated in the first stage will be passed to the fusion process, which may be amplified by the fusion network, resulting in the final fusion result being unable to accurately restore the scene information.

[0004] To solve the above optimization difficulties, some research introduces a "pseudo Ground Truth" constraint network training, i.e., using the fusion results obtained by existing fusion methods based on clean infrared and visible light source images as the network learning target. This method reduces the optimization complexity, enabling the network to more effectively learn the overall structure and detail information of the image, and performs more outstanding in color restoration and texture preservation. However, "pseudo Ground Truth" itself has information loss and modality bias, if the network directly learns "pseudo Ground Truth", it will tend to copy these features, causing the fusion network to shift its focus from multi-modal information interaction to feature copying, and only perform surface reconstruction of the target image. This learning method reduces the generalization ability of the network, making its excellent effect limited to complex scenes, and lacking modeling ability in ideal scenes.

[0005] In view of the above problems, the existing technology needs to be improved. SUMMARY

[0006] The purpose of the present application is to provide a mask-guided image restoration and fusion model and its training method and use method, aiming to solve the challenges faced by multi-modal image fusion in complex real-world scenarios, especially the problems of large network optimization difficulty, obvious error accumulation, and insufficient generalization ability caused by "pseudo Ground Truth" constraint training existing in the existing "restoration + fusion" two-stage scheme.

[0007] In a first aspect, the present application provides a mask-guided image restoration and fusion model, comprising a first input layer, a second input layer, two encoders, a plurality of serial mask-guided feature extraction modules, a first HTB module, two first residual modules, and an output layer.

[0008] The first input layer and the second input layer are respectively used to receive infrared source images and visible light source images, and are respectively input into two encoders for feature encoding to obtain preliminary infrared features and preliminary visible light features, the preliminary infrared features and the preliminary visible light features are added to obtain preliminary fusion features, and the preliminary infrared features, the preliminary visible light features and the preliminary fusion features are input into the first mask-guided feature extraction module.

[0009] Each mask-guided feature extraction module is used to generate a mask according to the input infrared features, input visible light features and input fusion features, and perform cross-attention calculation on the input infrared features, input visible light features and input fusion features under the guidance of the mask to obtain and output optimized output infrared features, output visible light features and output fusion features.

[0010] The output infrared features and the output visible light features output by the last mask-guided feature extraction module are respectively input into two first residual modules, and the output fusion features output by the last mask-guided feature extraction module are input into the first HTB module.

[0011] The two first residual modules and the first HTB module output final infrared features, final visible light features and final fusion features through the output layer respectively.

[0012] In a second aspect, the present application provides a training method of the mask-guided image restoration and fusion model as described above, comprising the steps of:

[0013] A1. Obtain an image dataset; the image dataset comprises a plurality of image groups, each image group comprising a degraded infrared source image and a degraded visible light source image that are mutually registered, and corresponding clean infrared source image and clean visible light source image;

[0014] A2. taking the degraded infrared source image and the degraded visible light source image in the image data set as model input, generating reference data by using a corresponding clean infrared source image and a clean visible light source image and a preset image restoration network, training the mask-guided image restoration and fusion model based on a total loss function, and obtaining a trained mask-guided image restoration and fusion model.

[0015] In a third aspect, the present application provides a method for using the mask-guided image restoration and fusion model as described above, comprising the steps of:

[0016] B1. obtaining an image to be fused, wherein the image to be fused comprises a visible light image and an infrared image that are mutually registered;

[0017] B2. inputting the image to be fused into the trained mask-guided image restoration and fusion model to obtain a final fusion feature output by the mask-guided image restoration and fusion model as a fused image.

[0018] Beneficial effects: the mask-guided image restoration and fusion model, the training method and the using method thereof provided by the present application overcome the problems of large network optimization difficulty and obvious error accumulation in the traditional "restoration + fusion" two-stage scheme, because the model adopts an end-to-end manner to closely combine the restoration and fusion processes, avoiding the transmission and amplification of intermediate errors; at the same time, through the mask-guided cross-modal cross-attention mechanism, the model can more effectively learn and integrate complementary information of different modalities, avoiding information loss and modal bias caused by "pseudo Ground Truth" constraint training, thereby improving the generalization ability and fusion effect of the model in complex scenes. Through this design, the model of the present application can generate a fused image containing richer scene details and stronger semantic expression ability, significantly improving the performance in practical applications such as target detection, semantic segmentation and depth estimation. BRIEF DESCRIPTION OF DRAWINGS

[0019] Figure 1 FIG. 1 is a structural schematic diagram of a mask-guided image restoration and fusion model provided by the present application.

[0020] Figure 2 FIG. 4 is a structural schematic diagram of an encoder.

[0021] Figure 3 FIG. 6 is a structural schematic diagram of a mask-guided feature extraction module.

[0022] Figure 4 FIG. 8 is a structural schematic diagram of a mask generation module.

[0023] Figure 5 FIG. 10 is a structural schematic diagram of a mask-guided cross-modal cross-attention module.

[0024] Figure 6 Flow chart of the training method provided in the present application.

[0025] Figure 7 Flow chart of the use method provided in the present application.

[0026] Figure 8 Comparison chart of the fusion results of multi-modal source images under rainy conditions.

[0027] Figure 9 Comparison chart of the fusion results of multi-modal source images under snowy conditions.

[0028] Figure 10 Comparison chart of the fusion results of multi-modal source images under haze conditions.

[0029] Label explanation: 1, first input layer; 2, second input layer; 3, encoder; 301, first convolutional layer; 302, third residual module; 4, mask guided feature extraction module; 401, second residual module; 402, second HTB module; 403, mask generation module; 404, mask guided cross-modal cross-attention module; 405, first operation module; 406, sigmoid activation function layer; 407, fourth residual module; 408, ReLU activation function layer; 409, fifth residual module; 410, first depth separable convolutional layer; 411, second depth separable convolutional layer; 412, third depth separable convolutional layer; 413, weighted operation module; 414, cross-attention module; 5, first HTB module; 6, first residual module; 7, output layer. DETAILED DESCRIPTION

[0030] The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments. The components of the embodiments of the present application described and shown in the drawings herein can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present application provided in the drawings is not intended to limit the scope of the claimed present application, but only represents selected embodiments of the present application. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without creative labor are within the scope of protection of the present application.

[0031] It should be noted that: similar labels and letters represent similar items in the following drawings, therefore, once an item is defined in one drawing, it does not need to be further defined and explained in subsequent drawings. Meanwhile, in the description of the present application, the terms "first", "second", etc. are only used for differentiation in description, and cannot be understood as indicating or implying relative importance.

[0032] Reference will be made to Figures 1-5 The application provides a mask-guided image restoration and fusion model, comprising a first input layer 1, a second input layer 2, two encoders 3, a plurality of mask-guided feature extraction modules 4 connected in series, a first HTB module 5 (HTB module, i.e., histogram Transformer module), two first residual modules 6 and an output layer 7.

[0033] The first input layer 1 and the second input layer 2 are respectively used for receiving an infrared source image and a visible light source image, and inputting the two encoders 3 for feature encoding to obtain preliminary infrared features and preliminary visible light features, and the preliminary infrared features and the preliminary visible light features are added to obtain preliminary fusion features, and the preliminary infrared features, the preliminary visible light features and the preliminary fusion features are input into the first mask-guided feature extraction module 4.

[0034] Each mask-guided feature extraction module 4 is used for generating a mask M according to the input infrared features IR, the input visible light features VI and the input fusion features F, and performing cross-attention calculation on the input infrared features, the input visible light features and the input fusion features under the guidance of the mask to obtain and output optimized output infrared features IR', output visible light features VI' and output fusion features F'. Figure 3 Figure 3 Figure 3 Figure 3 Figure 3 Figure 3 Figure 3

[0035] The output infrared features and the output visible light features output by the last mask-guided feature extraction module 4 are respectively input into the two first residual modules 6, and the output fusion features output by the last mask-guided feature extraction module 4 are input into the first HTB module 5.

[0036] The two first residual modules 6 and the first HTB module 5 respectively output final infrared features, final visible light features and final fusion features through the output layer 7.

[0037] ​​​​​​​The application proposes a mask-guided image restoration and fusion model. By introducing a mask-guiding mechanism, the model dynamically adjusts the weights of different modal information during feature extraction and fusion, effectively solving the optimization difficulties and error accumulation problems in traditional methods. The model receives infrared source images and visible light source images through the first input layer 1 and the second input layer 2, and uses the encoder 3 to perform preliminary feature encoding. Subsequently, multiple serial mask-guided feature extraction modules 4 generate masks according to the input features and perform cross-attention calculation under the guidance of the masks, thereby optimizing the infrared features, visible light features, and fusion features. Finally, the first HTB module 5, the first residual module 6, and the output layer 7 output the final infrared features, visible light features, and fusion features, effectively improving the performance of image restoration and fusion.

[0038] Among them, "infrared source image" and "visible light source image" respectively refer to the original image data obtained from infrared sensors and visible light sensors, which are the initial input of the model processing.

[0039] Among them, "preliminary infrared feature" and "preliminary visible light feature" refer to the feature representation obtained after the infrared source image and visible light source image are preliminarily processed by the encoder 3, which contains the basic semantic and texture information of the image.

[0040] Among them, "preliminary fusion feature" refers to the feature obtained by adding the preliminary infrared feature and the preliminary visible light feature, which represents the preliminary combination of the two modal information.

[0041] Among them, "mask" is a dynamically generated weight map used to guide the model to focus on specific areas or information in the image during feature extraction and fusion.

[0042] Among them, "cross-attention calculation" is an attention mechanism that allows the model to refer to the features of another modal when processing the features of one modal, thereby realizing cross-modal information interaction and fusion.

[0043] Among them, "final infrared feature", "final visible light feature", and "final fusion feature" refer to the final feature representation obtained after the entire model processing, which is the image restoration and fusion result of the model output.

[0044] The mask-guided image restoration and fusion model of the application is characterized by ingenious module design, which effectively restores and fuses infrared and visible light images. Specifically, the model includes a first input layer 1 and a second input layer 2, which are used to receive infrared source images and visible light source images, respectively. These two input layers can use standard data preprocessing techniques such as image normalization, size adjustment, etc. to ensure that the input data meets the requirements of the model.

[0045] The received infrared source image and visible light source image are then input into two encoders 3 for feature encoding. Encoder 3 can be any neural network structure capable of extracting image features. Through encoder 3, the infrared source image and visible light source image are converted into preliminary infrared features and preliminary visible light features. These preliminary features are abstract representations of the original images and contain key information about the images.

[0046] The preliminary infrared and preliminary visible light features are added together to obtain the preliminary fused features. This addition operation can be a simple element-wise addition or a more complex feature fusion strategy, such as a weighted summation using an additional convolutional layer. The preliminary fused features represent the combination of infrared and visible light information in the early stages. Subsequently, the preliminary infrared features, preliminary visible light features, and preliminary fused features are input into the first mask-guided feature extraction module 4.

[0047] Each mask-guided feature extraction module 4 is one of the core components of this application. It generates a mask based on the input infrared features, input visible light features, and input fusion features, and performs cross-attention calculations on these features under the guidance of the mask, obtaining and outputting optimized output infrared features, output visible light features, and output fusion features. The mask can be a two-dimensional matrix, where each element represents a weight at a corresponding position. Under the guidance of the mask, cross-attention calculations can weight features from different modalities, thereby highlighting important information and suppressing irrelevant information. For example, when processing infrared features, the mask can instruct the model to focus more on the thermal radiation region of the target in the infrared image; when processing visible light features, the mask can guide the model to focus on texture details in the visible light image.

[0048] Multiple mask-guided feature extraction modules 4 are connected in series. This means that the output infrared features, output visible light features, and output fused features of the previous module will serve as the input infrared features, input visible light features, and input fused features of the next module. This series structure allows the model to progressively optimize features at multiple levels, thereby achieving more refined image restoration and fusion.

[0049] After the final mask-guided feature extraction module 4 completes its processing, its output infrared and visible light features are fed into two first residual modules 6 for further feature refinement. Simultaneously, the output fused features are input into the first HTB module 5. The first HTB module 5 can segment and classify spatial features based on pixel intensity and selectively process features in different regions using a self-attention mechanism, while also focusing on similar degradation patterns in distant regions, thereby effectively capturing image degradation features caused by severe weather. Finally, through the output layer 7, these refined features are converted into final infrared features, final visible light features, and final fused features, thus restoring the original degraded image and generating a high-quality fused image.

[0050] The core innovation of this application lies in the introduction of a "mask-guided" mechanism. By dynamically generating masks in multiple cascaded mask-guided feature extraction modules 4, the model can intelligently adjust the weights of different modal information based on the real-time status of the input infrared features, input visible light features, and input fused features. This dynamic adjustment mechanism enables the model to selectively focus on more informative regions and modalities in the image during cross-attention calculation, thereby achieving more refined and effective fusion at the feature level.

[0051] Compared with traditional methods, the advantages of this application are:

[0052] First, through the mask-guided cross-attention mechanism, the model can directly perform deep intermodal interactions during feature extraction and fusion, avoiding the problem of inconsistent restoration and fusion goals in traditional two-stage methods, thereby improving the optimization efficiency and stability of the network.

[0053] Secondly, the dynamic generation and guidance of the mask enable the model to better adapt to image degradation in complex scenes. For example, when an infrared image has a high signal-to-noise ratio in a specific region, the mask can guide the model to utilize more infrared information; when a visible light image has rich texture details in another region, the mask can guide the model to utilize more visible light information. This adaptive fusion strategy effectively avoids the information loss and modal bias problems caused by "pseudo-Ground Truth," enabling the model to learn more realistic and comprehensive multimodal information.

[0054] Finally, through multiple cascaded mask-guided feature extraction modules 4, the model can progressively refine and optimize features in a multi-level structure, thereby capturing complex features of the image at different scales and levels of abstraction, and ultimately outputting high-quality restored and fused images.

[0055] In summary, the mask-guided image restoration and fusion model of this application significantly improves the restoration and fusion performance of multimodal images in complex scenes through its unique mask-guided mechanism and end-to-end optimization strategy, providing a more reliable and robust solution for related applications.

[0056] In some possible implementations, see Figure 2 The encoder 3 includes a first convolutional layer 301 and a third residual module 302 connected in series.

[0057] The first convolutional layer 301 is typically used for preliminary feature extraction of the input image data. It scans the image using convolutional kernels to capture local features such as edges and textures. Its purpose is to transform the raw image data into a more abstract, higher-dimensional feature representation, laying the foundation for subsequent deep feature learning. The first convolutional layer 301 can use a 3x3 convolutional kernel, but is not limited to this.

[0058] The third residual module 302 is configured after the first convolutional layer 301 to further deepen feature extraction and address the vanishing gradient problem in deep network training. Residual modules typically contain skip connections, allowing information to be directly passed from the previous layer to the next, thus effectively alleviating the difficulty of gradient propagation while increasing network depth and facilitating the learning of more complex feature representations. Through this front-to-back connection structure, encoder 3 can efficiently extract preliminary infrared and visible light features from infrared source images and visible light source images while maintaining network training stability.

[0059] Through the above technical solution, encoder 3 can perform feature encoding in a more efficient and stable manner. The first convolutional layer 301 provides basic feature perception capabilities, while the third residual module 302, through its unique structure, ensures effective learning of deep features and smooth propagation of gradient information. As a result, the model can obtain more expressive and robust preliminary infrared and visible light features, which is crucial for subsequent mask-guided feature extraction and image restoration and fusion processes, significantly improving the overall feature extraction capability of the model and the final image restoration and fusion effect.

[0060] In some implementations, see Figure 3 The mask-guided feature extraction module 4 includes two second residual modules 401, a second HTB module 402, a mask generation module 403, and a mask-guided cross-modal cross-attention module 404; the second residual module 401, the second HTB module 402, and the mask generation module 403 are all connected to the mask-guided cross-modal cross-attention module 404.

[0061] The two second residual modules 401 are used to extract optimized infrared features from the input infrared features and the input visible light features, respectively. Figure 3 IR_0 in the middle) and optimized visible light characteristics ( Figure 3 In VI_0), the second HTB module 402 is used to perform feature optimization on the input fusion features to obtain optimized fusion features ( Figure 3 (F_0 in the middle).

[0062] The mask generation module 403 is used to generate a mask using input infrared features, input visible light features and input fusion features;

[0063] The mask-guided cross-modal cross-attention module 404 is used to perform cross-attention calculations on optimized infrared features, optimized visible light features and optimized fusion features under the guidance of a mask, and to obtain and output optimized output infrared features, output visible light features and output fusion features.

[0064] The proposed solution refines the mask-guided feature extraction module 4 into multiple functional modules, enabling sophisticated processing of infrared, visible light, and fused features. Specifically, the input infrared and visible light features are first subjected to depth feature extraction by two second residual modules 401 to enhance their representational power and remove redundant information, resulting in optimized infrared and visible light features. Simultaneously, the input fused features are optimized by the second HTB module 402 to extract salient information from the image, effectively capturing image degradation features and further improving the quality of the fused features. Based on this, the mask generation module 403 comprehensively utilizes the complementary information of the input infrared, visible light, and fused features to generate a mask that highlights important regions or features in the image. Subsequently, the mask-guided cross-modal attention module 404 uses this mask as a guide to perform cross-attention calculations on the optimized infrared, visible light, and fused features. This guidance mechanism allows the attention mechanism to selectively focus on key information in the image and suppress irrelevant or noisy regions, thereby achieving more accurate and effective cross-modal feature fusion.

[0065] Through the above technical solutions, the mask-guided feature extraction module 4 can process multimodal image features in a more refined and controllable manner. By performing independent residual learning on infrared and visible light features, and by performing dedicated HTB optimization on the fused features, the quality of each modal feature before fusion is ensured. More importantly, the introduction of the mask generation module 403 and the mask-guided cross-modal attention module 404 makes the feature fusion process no longer a simple superposition or averaging, but rather an adaptive generation of guiding masks based on image content, and biased information interaction under the guidance of these masks. This significantly improves the model's ability to understand and fuse multimodal information in complex scenes, thereby obtaining higher quality output infrared features, output visible light features, and output fused features, and thus improving the overall performance of image restoration and fusion.

[0066] Furthermore, see Figure 4 The mask generation module 403 includes a first operation module 405 and a sigmoid activation function layer 406, a fourth residual module 407, a ReLU activation function layer 408 and a fifth residual module 409 connected in series.

[0067] The first calculation module 405 is used to calculate a preliminary mask using the input infrared features, input visible light features, and input fusion features. Figure 4 The initial mask is input into the sigmoid activation function layer 406 (M_1 in the model), and the initial mask is added to the output of the fifth residual module 409 to obtain the final mask. Figure 4 M in (the middle part).

[0068] Specifically, the first computation module 405 is responsible for preliminary processing of the input infrared features, input visible light features, and input fused features to extract key information for mask generation, and calculates a preliminary mask based on this information. This preliminary mask can be understood as a preliminary estimate of the importance or attention of different regions in the image. The preliminary mask is then input into the sigmoid activation function layer 406. The role of the sigmoid activation function layer 406 is to map the numerical range of the preliminary mask to between 0 and 1, giving it a probabilistic or weighted interpretation, thereby better representing the degree of attention of different regions. Following this, the mask processed by the sigmoid activation function layer 406 passes sequentially through the fourth residual module 407, the ReLU activation function layer 408, and the fifth residual module 409. The cascaded design of these modules aims to further refine the preliminary mask and learn its features. The residual modules (fourth residual module 407 and fifth residual module 409) effectively alleviate the vanishing gradient problem in deep network training and allow the network to learn more complex feature maps, thereby improving the expressive power and detail capture ability of the mask. The ReLU activation function layer 408 introduces non-linearity, enhancing the model's expressive power. Finally, the initial mask is element-wise added to the output of the fifth residual module 409 to obtain the final mask. This skip connection design, where the initial mask is added to the refined features, helps retain the original global information while incorporating local details and optimized features. This results in a final mask that contains both initial global guidance information and the local details and robustness refined by deep learning.

[0069] The proposed solution employs a multi-layer residual module (fourth residual module 407 and fifth residual module 409) and activation function layers (sigmoid activation function layer 406 and ReLU activation function layer 408) to perform deep learning and refinement of the initial mask. This enables the extraction of richer and more discriminative feature information from input infrared features, input visible light features, and input fused features. In particular, the design of adding the initial mask to the output of the fifth residual module 409 forms a residual connection. This allows the network to retain the original initial mask information while learning the refined mask, avoiding information loss and facilitating gradient propagation. Consequently, the generated mask more accurately reflects the saliency and importance of different regions in the image, providing more effective guidance for subsequent cross-modal cross-attention calculations.

[0070] Through the above technical solution, the mask generation module 403 can generate masks with higher quality and richer details. This refined mask can more accurately capture complementary information and salient regions in infrared and visible light images, thus providing more precise guidance in mask-guided cross-modal attention calculations. As a result, the model can more effectively fuse features from different modalities, improving the performance of image restoration and fusion, especially when processing complex scenes and images with rich details, resulting in clearer and more complete fused images.

[0071] Preferably, the first calculation module 405 generates a preliminary mask by performing the following calculations:

[0072] (1);

[0073] Where M_1 is the initial mask, F is the input fusion feature, IR is the input infrared feature, VI is the input visible light feature, and ε is the error parameter.

[0074] In fact, the core objective of multimodal image fusion is to suppress redundancy between modalities and effectively extract complementary information through sparse modeling strategies, thereby achieving a more comprehensive fusion representation that combines global semantics and local details. Without considering feature enhancement and restoration, F can be expressed as...

[0075] (2);

[0076] When the three components F, VI, and IR are known, we can obtain the expression for M_1.

[0077] (3);

[0078] Since VI and IR each have modality-specific features, direct subtraction may result in negative numbers and zero terms, making the training process unstable. Therefore, we rewrite the expression of M_1 as Equation (1), adding F to the denominator to avoid negative or zero denominators that could lead to training instability. Using the mask M obtained based on M_1, we can obtain how different fusion algorithms express different modal information. Based on this, by guiding the network to learn the distribution rules of multimodal information in the fusion result within the "pseudo-Ground Truth," we can dynamically determine the feature contributions of different modalities, thereby avoiding surface fitting of the overall scene.

[0079] Preferably, see Figure 5 The mask-guided cross-modal attention module 404 includes a first depthwise separable convolutional layer 410, a second depthwise separable convolutional layer 411, a third depthwise separable convolutional layer 412, a weighted operation module 413, and a cross-attention module 414.

[0080] The first depth separable convolutional layer 410, the second depth separable convolutional layer 411, and the third depth separable convolutional layer 412 are used to perform convolution processing on optimized infrared features, optimized visible light features, and optimized fused features, respectively, to obtain convolutional infrared features. Figure 5 IR_1 in the convolution, visible light features after convolution ( Figure 5 VI_1 in the middle) and post-convolution fusion features ( Figure 5 (F_1 in the middle).

[0081] The weighted operation module 413 is used to perform weighted processing on the convolved infrared features and the convolved visible light features using a mask to obtain weighted infrared features. Figure 5 IR_2) and weighted visible light features ( Figure 5 (VI_2 in the text)

[0082] The cross-attention module 414 is used to perform cross-attention calculation with weighted infrared features and weighted visible light features as queries and convolutional fused features as keys and values, to obtain and output optimized output infrared features, output visible light features and output fused features.

[0083] Specifically, the mask, as a guiding mechanism, reflects the importance or salience of different regions in the image. Through the weighted operation module 413, the mask is applied to the convolutional infrared features and convolutional visible light features, enabling the model to selectively enhance or suppress feature information of specific regions according to the mask's indication, thereby achieving refined guidance and adjustment of infrared and visible light features.

[0084] The cross-attention mechanism allows the model to focus on and integrate relevant information from other modalities while processing features from one modality. Here, weighted infrared and weighted visible light features are used as queries, representing information sources that need to be focused on and integrated; while the convolutionally fused features are used as keys and values, representing reference information that provides context and content. In this way, the cross-attention module 414 can effectively fuse information from infrared, visible light, and fused features to generate output infrared features, output visible light features, and output fused features with stronger expressive power.

[0085] The proposed solution refines the mask-guided cross-modal attention module 404 into multiple functional units, achieving refined processing and efficient fusion of infrared, visible light, and fused features. First, the first depthwise separable convolutional layer 410, the second depthwise separable convolutional layer 411, and the third depthwise separable convolutional layer 412 perform preliminary feature extraction and dimensionality reduction on the input optimized infrared, visible light, and fused features, providing a structured feature representation for subsequent attention calculations. Then, the weighting module 413 uses a pre-generated mask to weight the convolutional infrared and visible light features, enabling the model to selectively adjust the weights of infrared and visible light features based on the saliency or importance regions indicated by the mask, thus achieving the guiding role of the mask at the feature level. Finally, the cross-attention module 414 uses the mask-weighted infrared and visible light features as queries and the fused features as keys and values ​​to perform cross-modal attention calculations. This mechanism enables the model to dynamically capture the interdependencies between different modal features and, guided by the mask, selectively integrate key information from infrared and visible light features within the context of feature fusion, thereby generating more robust and informative output features.

[0086] Through the above technical solution, the internal structure and workflow of the mask-guided cross-modal attention module 404 are clearly defined, enabling the model to process multimodal image features in a more efficient and refined manner. The introduction of depthwise separable convolutional layers significantly reduces computational complexity and the number of model parameters, improving the model's operating efficiency. The weighted operation module 413, in conjunction with the mask, weights infrared and visible light features, ensuring that the attention mechanism can accurately focus on important regions in the image, enhancing the model's ability to perceive key information. Furthermore, by using the weighted infrared and visible light features as queries and the fused features as keys and values ​​for cross-attention calculation, this solution effectively integrates the advantageous information of different modalities, overcoming the limitations of a single modality, thereby generating high-quality fused features and providing a solid foundation for subsequent image restoration and fusion tasks.

[0087] In some possible implementations, the weighting module 413 performs the following operations to weight the convolved infrared features and the convolved visible light features:

[0088] ;

[0089] ;

[0090] Wherein, VI_2 is the weighted visible light feature, VI_1 is the convolutional visible light feature, M is the mask, IR_2 is the weighted infrared feature, and IR_1 is the convolutional infrared feature.

[0091] Specifically, the weighted visible light feature VI_2 is obtained by element-wise multiplying the convolved visible light feature VI_1 with the mask M. Similarly, the weighted infrared feature IR_2 is obtained by element-wise multiplying the convolved infrared feature IR_1 with the complement of the mask M (i.e., 1-M). The mask M is a numerical matrix between 0 and 1, and its values ​​reflect the salience or importance of different regions in the image.

[0092] The scheme in this application achieves adaptive weighting of features from different modalities through the aforementioned calculations. Specifically, when the value of mask M is high in a certain region, it indicates that the region may have higher saliency or information content in the visible light modality. In this case, the weight of visible light feature VI_1 in that region is amplified after convolution, while the weight of infrared feature IR_1 in that region is correspondingly reduced. Conversely, when the value of mask M is low in a certain region, it indicates that the region may have higher saliency or information content in the infrared modality. In this case, the weight of infrared feature IR_1 in that region is amplified after convolution, while the weight of visible light feature VI_1 in that region is correspondingly reduced. This complementary weighting mechanism allows the model to dynamically adjust the contribution of infrared and visible light features based on the image content information captured by mask M, thereby more effectively fusing key information from different modalities in subsequent cross-attention calculations.

[0093] Through the above technical solution, the weighted operation module 413 can weight infrared and visible light features in a precise and interpretable manner, guided by the mask M. This weighting process based on complementary weights (M and 1-M) ensures that features of different modalities are reasonably allocated according to their importance in specific regions during the fusion process, avoiding the problem of single modal information being too strong or too weak. As a result, the model can more effectively integrate the thermal radiation information of the infrared image and the texture detail information of the visible light image, improving the quality and information integrity of the fused image. Especially when processing images with complex scenes and variable lighting conditions, it can generate fusion results with better visual effects and richer information.

[0094] refer to Figure 6 This application provides a training method for a mask-guided image restoration and fusion model as described above, including the following steps:

[0095] A1. Obtain the image dataset; the image dataset includes multiple image groups, each image group including mutually registered degraded infrared source images and degraded visible light source images, as well as corresponding clean infrared source images and clean visible light source images;

[0096] A2. Using degraded infrared source images and degraded visible light source images from the image dataset as model inputs, reference data is generated using corresponding clean infrared source images and clean visible light source images, as well as a preset image restoration network. Based on the total loss function, the mask-guided image restoration and fusion model is trained to obtain the trained mask-guided image restoration and fusion model.

[0097] Specifically, an image dataset refers to a collection of image samples used to train a machine learning model. This image dataset can be acquired in various ways, such as downloading from a public image database or capturing images using specialized image acquisition equipment. The composition of each image set is crucial. It includes degraded infrared source images and degraded visible light source images to simulate low-quality inputs that might be encountered in real-world applications (such as images containing degraded information like rain, snow, or fog). The corresponding clean infrared source images and clean visible light source images serve as true labels or references during training (where clean infrared source images are the images without degraded information corresponding to degraded infrared source images, and clean visible light source images are the images without degraded information corresponding to degraded visible light source images), guiding the model's learning. Mutual registration refers to the spatial alignment of the infrared source images and visible light source images to ensure they depict the same scene, which is essential for subsequent feature fusion and reconstruction.

[0098] In this context, model input refers to the process of feeding the image data to be processed into the model. In this step, degraded infrared source images and degraded visible light source images are input into the mask-guided image restoration and fusion model to be trained. The pre-set image restoration network can be a standalone, pre-trained network (e.g., the Restormer network), whose role is to assist in generating reference data needed for training, such as for calculating certain loss terms. The total loss function is a comprehensive metric that measures the difference between the model output and the true label. It is usually composed of multiple sub-loss functions, designed to evaluate the model's performance from different dimensions. Training the model based on the total loss function means continuously adjusting the model's internal parameters through backpropagation algorithms and optimizers (e.g., gradient descent and its variants) to minimize the value of the total loss function. When the total loss function converges or reaches the preset number of training epochs, the trained mask-guided image restoration and fusion model is obtained, which can effectively perform image restoration and fusion tasks.

[0099] This application's solution effectively addresses the problem that mask-guided image restoration and fusion models cannot achieve their expected performance without training by introducing a structured training method. Specifically, step A1 obtains a paired dataset containing degraded and clean images, providing necessary input and supervision signals for the model's learning. The degraded images simulate real-world problems such as noise, blurring, or low contrast, while the clean images provide the learning target for the model. In step A2, the degraded images are input into the model to be trained, and reference data is generated using the clean images and a pre-defined image restoration network. These reference data are compared with the model's output, and the difference between them is quantified using a total loss function. The design of the total loss function is crucial, as it comprehensively considers performance metrics such as image restoration, fusion quality, and color consistency. By minimizing this total loss function, the model's internal parameters are iteratively adjusted, enabling the model to progressively learn how to recover clear infrared and visible light features from the degraded input and effectively fuse them, ultimately generating a high-quality fused image. This supervised learning-based training mechanism enables the model to extract useful features and patterns from a large amount of image data, thus allowing it to output optimized restoration and fusion results even when faced with new and unseen degraded images.

[0100] Through the aforementioned technical solution, this application provides a systematic and efficient training method that transforms a mask-guided image restoration and fusion model from its original, untrained state into a model with powerful image processing capabilities. This training method, through a carefully constructed image dataset and an optimization process based on the total loss function, ensures the model's performance in image restoration and multimodal image fusion tasks. Specifically, this method enables the model to learn how to effectively remove degradative factors from images while preserving and enhancing key information from different modalities (infrared and visible light), ultimately generating visually high-quality, information-rich fused images. This training mechanism significantly improves the model's practicality and generalization ability, enabling it to adapt to various complex image processing scenarios, thus playing a crucial role in fields such as security monitoring, medical imaging, and autonomous driving.

[0101] Preferably, step A2 may include:

[0102] A201. Select an image group from the image dataset, input the degraded infrared source image and the degraded visible light source image of the image group into the mask-guided image restoration and fusion model, and obtain the final infrared features, final visible light features and final fusion features output by the mask-guided image restoration and fusion model;

[0103] A202. Using the clean infrared source image and the clean visible light source image of the selected image group, calculate the reference infrared feature and the reference visible light feature based on the following formula (both the reference infrared feature and the reference visible light feature are reference data):

[0104] ;

[0105] ;

[0106] ;

[0107] in, For reference to visible light characteristics, For reference infrared features, For clean, visible light source images, For clean infrared source images, This is a reference mask (one of the reference data). To and The reference fusion features obtained through fusion (can be obtained by fusing existing multimodal fusion algorithms, i.e.) In fact, it is a "pseudo-ground truth", which is also one of the reference data. ε is an error parameter (one of the optimizable model parameters. When optimizing model parameters in step A208, it includes optimizing ε).

[0108] A203. Based on the final infrared features, final visible light features, reference infrared features, and reference visible light features, calculate the mask-guided learning policy loss using the following formula:

[0109] ;

[0110] in, The loss is defined as the mask-guided learning strategy loss, where H and W represent the pixel length and pixel width of the source image, respectively. For the final visible light characteristics, For the final infrared signature, Represents the 1-norm;

[0111] A204. Extract the mask generated by the last mask-guided feature extraction module 4 in this iteration, and multiply it element-wise with the final fused feature to obtain the visible light distribution map in the final fused feature;

[0112] A205. Use a pre-defined image restoration network to restore the visible light distribution map, obtain the restored image, and calculate the task-coupled degradation perception loss according to the following formula:

[0113] ;

[0114] in, Loss perception due to task coupling degradation. This is a visible light distribution map. This is a reconstruction image;

[0115] A206. Calculate the gradient maps of the final fused features, the clean infrared source image, and the clean visible light source image, and calculate the gradient loss according to the following formula:

[0116] ;

[0117] in, For gradient loss, The gradient map of the final fused features. The gradient map of a clean, visible light source image. Gradient plot of a clean infrared source image;

[0118] A207. Calculate the color consistency loss using the following formula:

[0119] ;

[0120] in, For color consistency loss, For the final fusion feature, This represents a function that converts an image space to the CbCr color space.

[0121] A208. Calculate the total loss function according to the following formula, and optimize the model parameters of the mask-guided image restoration and fusion model based on the total loss function:

[0122] ;

[0123] in, This is the total loss function;

[0124] A209. Iterate through steps A201-A208 until the total loss function is lower than the preset loss threshold or the number of iterations reaches the preset number threshold, to obtain the trained mask-guided image restoration and fusion model.

[0125] Specifically, in step A201, an image group is first selected from the pre-acquired image dataset. This image group contains mutually registered degraded infrared source images and degraded visible light source images, as well as corresponding clean infrared source images and clean visible light source images. Then, the degraded infrared source images and degraded visible light source images in this image group are used as input to the mask-guided image restoration and fusion model to be trained, to obtain the final infrared features, final visible light features, and final fused features output by the model in the current state. When selecting image groups, priority is given to image groups that have not been selected before or those selected the least.

[0126] In step A202, reference infrared features and reference visible light features are calculated using specific formulas from the selected image set, utilizing clean infrared source images and clean visible light source images. These reference features serve as ground truth or targets during the training process to guide model learning. The calculation of the reference mask involves the pixel value differences between the clean visible light source images and the clean infrared source images, and an error parameter ε is introduced to enhance the robustness of the calculation. The reference fusion features are obtained by fusing the reference infrared features and the reference visible light features.

[0127] In step A203, the mask-guided learning strategy loss is calculated based on the final infrared and visible light features output by the model, as well as the reference infrared and reference visible light features calculated in step A202. This loss function aims to measure the accuracy of the model in feature extraction and mask guidance. By minimizing the difference between the model output features and the reference features, it encourages the model to learn more effective feature representations and mask generation strategies, and constrains the network to learn the multimodal information allocation rules in the "pseudo-ground truth".

[0128] In step A204, the mask generated by the last mask-guided feature extraction module 4 in this iteration is extracted. This mask is designed to guide the feature recovery and fusion process. Subsequently, the mask is multiplied element-wise with the final fused feature to obtain the visible light distribution map in the final fused feature. This visible light distribution map reflects the distribution and weight of visible light information in the fused feature.

[0129] In step A205, the visible light distribution map obtained in step A204 is restored using a pre-defined image restoration network to obtain a restored image (this is existing technology, for example, using the existing Restormer network to restore the visible light distribution map). Next, the task-coupled degradation perception loss is calculated based on the visible light distribution map and the restored image. This loss function aims to evaluate the model's ability to handle image degradation, prompting the model to better perceive and recover degradation information in the image by comparing the original distribution map with the restored image.

[0130] In step A206, gradient maps of the final fused features, the clean infrared source image, and the clean visible light source image are calculated (the calculation method for the gradient maps is existing technology and will not be detailed here). Gradient maps reflect the edge and texture information of the image. Based on these gradient maps, gradient loss is calculated. The purpose of gradient loss is to preserve the structural and detail information of the fused image, ensuring that the fused image maintains consistency with the original clean image in terms of edges and texture.

[0131] In step A207, the color consistency loss is calculated. This loss function is performed by converting the final fused features to the CbCr color space and comparing them with the representations of clean infrared source images and clean visible light source images in that color space, to ensure that the color information of the fused image remains consistent with the original image and to avoid color distortion.

[0132] In step A208, the mask-guided learning policy loss, task coupling degradation perception loss, gradient loss, and color consistency loss calculated above are summed (in practice, a weighted summation can also be used, i.e., assigning a corresponding weight to each loss and then calculating the sum of the products of each loss and its corresponding weight, as the total loss function), to obtain the total loss function. The total loss function comprehensively considers the model's performance in multiple aspects such as feature learning, degradation perception, structure preservation, and color consistency. Subsequently, based on this total loss function, the model parameters of the mask-guided image restoration and fusion model are optimized using an optimization algorithm (such as gradient descent) to minimize the total loss.

[0133] In step A209, the iterative process from steps A201 to A208 is repeated. This iterative training continues until the total loss function value is lower than a preset loss threshold, or the number of iterations reaches a preset threshold. Through this iterative optimization, the model parameters are gradually adjusted, resulting in a well-trained and high-performance mask-guided image restoration and fusion model.

[0134] This application's solution effectively addresses the inefficiencies and poor restoration and fusion results of traditional training methods by introducing multiple carefully designed loss functions to synergistically optimize the mask-guided image restoration and fusion model. Specifically, the mask-guided learning strategy loss ensures the model accurately captures and represents image information, particularly complementary features of infrared and visible light modes, by directly comparing the model's output features with reference features. The task-coupled degradation perception loss focuses on improving the model's ability to perceive and restore image degradation by restoring and comparing the visible light distribution map, prompting the model to learn how to effectively remove noise and artifacts. The introduction of gradient loss aims to maintain the structural integrity and edge sharpness of the fused image, preventing blurring or loss of detail during the fusion process. The color consistency loss ensures that the fused image maintains a high degree of consistency with the original clean image in the color space, preventing color drift or distortion. By weighting and combining these loss functions to form a total loss function and employing an iterative optimization strategy, the model can self-adjust and improve across multiple dimensions, achieving superior performance and robustness in image restoration and fusion tasks.

[0135] Through the above technical solution, this application enables efficient and robust training of a mask-guided image restoration and fusion model. Specifically, the synergistic effect of multiple loss functions allows the model to not only learn high-quality feature representations but also effectively perceive and restore degradation information in the image while maintaining the consistency of image structure, detail, and color. This comprehensive training strategy significantly improves the model's ability to restore and fuse infrared and visible light images in complex scenes, resulting in a fused image with higher clarity, richer detail, and a more natural visual effect. This overcomes the performance bottlenecks and insufficient generalization capabilities that may occur when training with only a single or simple loss function.

[0136] refer to Figure 7 This application provides a method for using the mask-guided image restoration and fusion model as described above, including the following steps:

[0137] B1. Acquire the images to be fused, which include mutually registered visible light and infrared images;

[0138] B2. Input the image to be fused into the trained mask-guided image restoration and fusion model, and obtain the final fusion features output by the mask-guided image restoration and fusion model as the fused image.

[0139] Specifically, in step B1, the infrared image and the visible light image are registered to each other. This means that the two images are spatially aligned, i.e., the same physical points in the images have the same pixel coordinates in both images or have been corrected through geometric transformation to eliminate geometric distortions caused by differences in sensor position, viewing angle, etc. This registration ensures the accuracy and effectiveness of subsequent fusion processing.

[0140] Further, in step B2, the image to be fused is input into the trained mask-guided image restoration and fusion model. Here, the "trained" model refers to a model whose parameters have been optimized using the aforementioned training method, enabling it to recover and fuse information from degraded infrared and visible light source images. After receiving the registered infrared and visible light images, the model performs a series of processes according to its internal structure and trained parameters, including feature extraction, mask generation, and cross-attention calculation, ultimately outputting the final fused feature. This final fused feature is the result of the model's deep fusion and restoration of the input infrared and visible light images. It contains the thermal radiation information of the infrared image and the texture details of the visible light image, and potential degradation effects have been removed through the model's restoration mechanism. This final fused feature can be directly output as the fused image for subsequent image analysis, target recognition, or display applications.

[0141] Figures 8-10 This paper presents a comparison between the fusion results of this application and existing methods for multimodal source images in severe weather scenarios. Specifically, Figure 8 This is a comparison image of the fusion results of multimodal source images under rainy conditions; Figure 9 This is a comparison image of the fusion results of multimodal source images under snowy conditions; Figure 10 This image shows a comparison of the fusion results of multimodal source images under hazy conditions. In the image, a represents a visible light source image, b represents an infrared source image, c represents the fusion result of the comparison method in the prior art, and d represents the fusion result of this application. From the source images, severe weather significantly obscures details in the visible light image, causing useful textures to mix with degradation factors, resulting in overexposure or decreased contrast. The infrared image, however, remains relatively stable, showing only a slight degradation in contrast. The fusion results demonstrate that this application not only achieves full interaction of multimodal features and restores the original clean features of the scene under severe weather conditions, but also effectively preserves subtle texture information that is not easily noticeable in visible light, resulting in a higher quality fused image.

[0142] In summary, this application has at least the following advantages:

[0143] 1. A unified framework for multimodal image fusion in complex scenarios is proposed, which can realize feature restoration and modal interaction within a single module. With the guidance of external priors, the proposed algorithm effectively reduces the optimization difficulty of multimodal image fusion tasks under complex conditions.

[0144] 2. A mask-guided cross-modal attention module 404 was designed to reconstruct the fusion result in a simple and effective way. At the same time, a mask-guided network learning scheme was proposed (the learning process introduces a mask-guided learning strategy loss), which effectively solves the problem of ignoring multimodal dynamic interaction modeling due to overfitting static features.

[0145] 3. A task-coupled degradation-aware learning strategy is proposed (the learning process introduces task-coupled degradation-aware loss), which effectively enhances the degradation-awareness capability of the fusion process and provides a more meaningful supervision signal for the feature fusion process by utilizing the restoration task;

[0146] 4. It demonstrates significant effectiveness in adverse weather conditions such as rain, fog, and snow, and has the potential for practical application.

[0147] The above description is merely an embodiment of this application and is not intended to limit the scope of protection of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of protection of this application.

Claims

1. A mask-guided image restoration and fusion model, characterized in that, The method comprises a first input layer, a second input layer, two encoders, a plurality of mask-guided feature extraction modules connected in series, a first HTB module, two first residual modules and an output layer. The first input layer and the second input layer are respectively used for receiving infrared source images and visible light source images, and inputting the two encoders to obtain preliminary infrared features and preliminary visible light features, and the preliminary infrared features and the preliminary visible light features are added to obtain preliminary fusion features, and the preliminary infrared features, the preliminary visible light features and the preliminary fusion features are input into the first mask-guided feature extraction module. Each mask-guided feature extraction module is used for generating a mask according to input infrared features, input visible light features and input fusion features, and performing cross-attention calculation on the input infrared features, the input visible light features and the input fusion features under the guidance of the mask to obtain and output optimized output infrared features, output visible light features and output fusion features. The output infrared features and the output visible light features output by the last mask-guided feature extraction module are respectively input into the two first residual modules, and the output fusion features output by the last mask-guided feature extraction module are input into the first HTB module. The two first residual modules and the first HTB module output final infrared features, final visible light features and final fusion features through the output layer.

2. The mask-guided image restoration and fusion model of claim 1, wherein, The encoder comprises a first convolutional layer and a third residual module connected in front and back.

3. The mask-guided image restoration and fusion model of claim 1, wherein, The mask-guided feature extraction module comprises two second residual modules, a second HTB module, a mask generation module and a mask-guided cross-modal cross-attention module; the second residual module, the second HTB module and the mask generation module are connected with the mask-guided cross-modal cross-attention module; The two second residual modules are respectively used for extracting features of the input infrared features and the input visible light features to obtain optimized infrared features and optimized visible light features, and the second HTB module is used for optimizing features of the input fusion features to obtain optimized fusion features. The mask generation module is used for generating a mask by using the input infrared features, the input visible light features and the input fusion features. The mask-guided cross-modal cross-attention module is used for performing cross-attention calculation on the optimized infrared features, the optimized visible light features and the optimized fusion features under the guidance of the mask to obtain and output optimized output infrared features, output visible light features and output fusion features.

4. The mask-guided image restoration and fusion model of claim 3, wherein, The mask generation module comprises a first operation module and a sigmoid activation function layer, a fourth residual module, a ReLU activation function layer and a fifth residual module connected in series. The first operation module is used for calculating a preliminary mask by using the input infrared features, the input visible light features and the input fusion features, and inputting the preliminary mask into the sigmoid activation function layer; and the preliminary mask and the output of the fifth residual module are added to obtain the final mask.

5. The mask-guided image restoration and fusion model of claim 4, wherein, The first operation module generates the preliminary mask by performing the following operation: ； Wherein, M_1 is the preliminary mask, F is the input fusion feature, IR is the input infrared feature, VI is the input visible light feature, and ε is an error parameter.

6. The mask-guided image restoration and fusion model of claim 3, wherein, The mask-guided cross-modal cross-attention module includes a first depth separable convolution layer, a second depth separable convolution layer, a third depth separable convolution layer, a weighting operation module, and a cross-attention module. The first depth separable convolution layer, the second depth separable convolution layer, and the third depth separable convolution layer are respectively used for convolution processing on the optimized infrared feature, the optimized visible light feature, and the optimized fusion feature to obtain a convolution-processed infrared feature, a convolution-processed visible light feature, and a convolution-processed fusion feature. The weighting operation module is used for weighting processing on the convolution-processed infrared feature and the convolution-processed visible light feature by using the mask to obtain a weighted infrared feature and a weighted visible light feature. The cross-attention module is used for cross-attention calculation by taking the weighted infrared feature and the weighted visible light feature as queries, and taking the convolution-processed fusion feature as keys and values to obtain and output an optimized output infrared feature, an optimized output visible light feature, and an optimized output fusion feature.

7. The mask-guided image restoration and fusion model of claim 6, wherein, The weighting operation module realizes the weighting processing on the convolution-processed infrared feature and the convolution-processed visible light feature by performing the following operation: ； ； Wherein, VI_2 is the weighted visible light feature, VI_1 is the convolution-processed visible light feature, M is the mask, IR_2 is the weighted infrared feature, and IR_1 is the convolution-processed infrared feature.

8. The method of training a mask-guided image restoration and fusion model according to any one of claims 1-7, wherein, The method comprises the following steps: A1. Obtain an image data set; the image data set comprises a plurality of image groups, each image group comprising mutually registered degraded infrared source images and degraded visible light source images, and corresponding clean infrared source images and clean visible light source images; A2. Take the degraded infrared source images and the degraded visible light source images in the image data set as model inputs, generate reference data by using the corresponding clean infrared source images and clean visible light source images and a preset image restoration network, train the mask-guided image restoration and fusion model based on a total loss function, and obtain a trained mask-guided image restoration and fusion model.

9. The training method of claim 8, wherein, Step A2 comprises: A201. Select an image group from the image data set, input the degraded infrared source images and the degraded visible light source images of the image group into the mask-guided image restoration and fusion model, and obtain final infrared features, final visible light features, and final fusion features output by the mask-guided image restoration and fusion model; A202. Calculate reference infrared features and reference visible light features based on the following formula by using the clean infrared source images and the clean visible light source images of the selected image group: ； ； ； wherein, is the reference visible feature, is the reference infrared feature, is the clean visible source image, is the clean infrared source image, is a reference mask, is a reference fusion feature obtained by fusing and with an error parameter ε. A203. Calculate a mask-guided learning strategy loss based on the following formula according to the final infrared features, the final visible light features, the reference infrared features, and the reference visible light features: ； wherein, is the mask-guided learning policy loss, H and W are the pixel length and pixel width of the source image, respectively, is the final visible light feature, is the final infrared feature, denotes the 1-norm; A204. Extracting the mask generated by the last mask-guided feature extraction module in this iteration, and performing element-wise multiplication with the final fusion feature to obtain a visible light distribution map in the final fusion feature; A205. Restoring the visible light distribution map using a pre-set image restoration network to obtain a restored map, and calculating a task-coupled degradation perception loss according to the following formula: ； wherein, a task-coupled degenerated perception loss for the task, a visible light assignment map, a restoration map; A206. Calculating gradient maps of the final fusion feature, the clean infrared source image, and the clean visible light source image, and calculating a gradient loss according to the following formula: ； wherein, is the gradient loss for the gradient map, is the gradient map of the final fused feature, is the gradient map of the clean visible light source image, is the gradient map of the clean infrared source image; A207. Calculating a color consistency loss according to the following formula: ； wherein, is the color consistency loss, is the final fused feature, denotes a function that converts the image space to the CbCr color space; A208. Calculating a total loss function according to the following formula, and optimizing the model parameters of the mask-guided image restoration and fusion model according to the total loss function: ； wherein, is the total loss function; A209. Iteratively performing steps A201-A208 until the total loss function is lower than a pre-set loss threshold or the number of iterations reaches a pre-set number threshold, to obtain a trained mask-guided image restoration and fusion model.

10. A method of using the mask-guided image restoration and fusion model of any one of claims 1-7, wherein, Comprising the steps of: B1. Obtaining to-be-fused images, the to-be-fused images comprising mutually registered visible light images and infrared images; B2. Inputting the to-be-fused images into the trained mask-guided image restoration and fusion model to obtain a final fusion feature output by the mask-guided image restoration and fusion model as a fused image.

Images