Multimodal image fusion and deblurring model and its training method and usage method

Abstract

The present application belongs to the field of image processing technology, and discloses a multimodal image fusion and deblurring model, a training method thereof, and a usage method thereof. By constructing an integrated multimodal image fusion and deblurring model and adopting a specific texture enhancement and feature interaction fusion mechanism, the integrated processing of multimodal image fusion and deblurring is realized, the efficiency is improved and the error accumulation is reduced, and the texture of the visible light image is enhanced. The application has the advantages of realizing the integrated processing of multimodal image fusion and deblurring, improving the efficiency and reducing the error accumulation, and improving the image fusion quality.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to a multimodal image fusion and deblurring model and its training method and usage method. Background Art

[0002] In real-world scenarios, different sensors are often required to detect and perceive the target scene. For example, visible light sensors can provide high-definition and high-resolution image information in favorable environments, but their disadvantage is weak anti-interference capabilities. Infrared sensors, on the other hand, can identify objects through thermal radiation, making them effective in detecting thermal targets in complex environments. However, the spatial resolution of the images they provide is low, and the imaged scene details are limited. Therefore, multimodal image fusion techniques can be used to extract effective information from source images captured by different sensors and eliminate redundant information to generate a fused image that provides a richer interpretation of the scene, thereby enhancing understanding of the real scene. In this way, infrared and visible light image fusion methods have the ability to assist advanced visual tasks such as target tracking, unmanned driving, and military reconnaissance, and have been widely studied and applied.

[0003] In real-world photography, motion blur often occurs in images due to camera shake or rapid object movement. Existing multimodal image fusion algorithms typically address this problem using a two-stage approach: first, deblurring the source image, then fusing the deblurred source image. This separate processing approach clearly presents two problems. The first is low processing efficiency, and the second is error accumulation. Deblurring a degraded image often results in blurred edges in the output, impacting subsequent fusion performance.

[0004] On the other hand, in actual shooting, especially in road scenes, more local motion blur problems are often encountered. This motion blur mainly comes from the rapid movement of vehicles or people, resulting in some targets in the captured image being blurred. Even though the current global deblurring algorithm has achieved great success, there are still huge challenges in using it to solve this local small area motion blur problem. Because the characteristic of local blur is that the motion blur of small targets appears randomly under the condition of a clear background, if the global blurring algorithm is directly used, not only will the algorithm's motion blur removal effect be greatly reduced, but the clear texture in the background will also be smoothed out. Therefore, it is challenging to fuse infrared and visible light images in the presence of motion blur, restore the blurred detail information, and obtain high-quality fusion results.

[0005] In view of the above problems, the existing technology is in urgent need of improvement. Summary of the Invention

[0006] The purpose of this application is to provide a multimodal image fusion and deblurring model and its training method and usage method, which has the advantages of realizing integrated processing of multimodal image fusion and deblurring, improving efficiency and reducing error accumulation, while improving image fusion quality.

[0007] In a first aspect, the present application provides a multimodal image fusion and deblurring model, comprising a first input layer, a second input layer, a first encoder, a second encoder, multiple visible light texture enhancement modules, multiple efficient feature interaction fusion modules, a decoder, and a first output layer;

[0008] The first input layer and the second input layer are respectively used to input a visible light image and an infrared image to be fused and registered with each other; the first encoder and the second encoder are both multi-layer encoders, and the decoder is a multi-layer decoder; the first encoder is used to sequentially extract visible light feature maps of different scales from the visible light image, wherein the visible light feature maps include high-frequency visible light feature maps and low-frequency visible light feature maps; the second encoder is used to sequentially extract infrared feature maps of different scales from the infrared image, wherein the infrared feature maps include high-frequency infrared feature maps and low-frequency infrared feature maps;

[0009] Each of the visible light texture enhancement modules and each of the efficient feature interaction fusion modules corresponds to each scale one-to-one; at each scale, the visible light texture enhancement module is used to perform texture enhancement processing on the corresponding high-frequency visible light feature map using the corresponding high-frequency infrared feature map to obtain an enhanced high-frequency visible light feature map of the corresponding scale; at each scale, the high-frequency infrared feature map and the low-frequency infrared feature map are added, and the enhanced high-frequency visible light feature map and the low-frequency visible light feature map are added, and then input together into the corresponding efficient feature interaction fusion module for fusion;

[0010] The decoder is used to perform layer-by-layer upsampling and convolution operations on the output fusion features of each of the efficient feature interactive fusion modules to obtain a final fusion image and output it from the first output layer.

[0011] Preferably, the first encoder includes multiple first encoding layers, the second encoder includes multiple second encoding layers, the decoder includes multiple decoding layers, and the number of the first encoding layers, the second encoding layers, the decoding layers, the visible light texture enhancement module, and the efficient feature interaction fusion module is the same;

[0012] The first encoder is used to sequentially extract visible light feature maps of different scales from the visible light image through each first encoding layer; the second encoder is used to sequentially extract infrared feature maps of different scales from the infrared image through each second encoding layer;

[0013] The decoder performs upsampling and convolution operations layer by layer from the decoding layer of the last layer to the decoding layer of the first layer, and the decoding layer of the first layer is connected to the first output layer; the output fusion features of the efficient feature interactive fusion module corresponding to the last two coding layers are input into the decoding layer of the last layer, and from back to front, the output fusion features of the efficient feature interactive fusion module corresponding to other coding layers are input into the decoding layers from the second to last layer to the second layer in sequence.

[0014] Preferably, the visible light texture enhancement module includes a third input layer, a fourth input layer, a first convolutional layer, an LDC layer, a Sigmoid activation function layer, a second convolutional layer and a second output layer;

[0015] The third input layer is used to input a high-frequency visible light feature map, and the fourth input layer is used to input a high-frequency infrared feature map. The third input layer inputs the high-frequency visible light feature map into the first convolution layer, and the fourth input layer inputs the high-frequency infrared feature map into the LDC layer. The high-frequency visible light feature map is subtracted from the high-frequency infrared feature map and then input into the Sigmoid activation function layer. The output of the first convolution layer is multiplied by the output of the Sigmoid activation function layer and then merged with the output of the LDC layer. The merged result is input into the second convolution layer, and the output end of the second convolution layer is connected to the second output layer.

[0016] Preferably, the LDC layer is used to perform the following operations:

[0017] ;

[0018] in, is the pixel value of the (i, j) pixel in the output of the LDC layer, (i, j) is the pixel coordinate, is a common convolution kernel, for The (i,j) element of is the pixel value of the (i, j) pixel in the input of the LDC layer, is a learnable parameter, is a 3×3 all-ones matrix, is a learnable description operator, represents the convolution operation, Represents element-wise multiplication.

[0019] Preferably, the efficient feature interaction fusion module includes a fifth input layer, a sixth input layer, an interactive spatial attention block, an interactive channel attention block, a third convolutional layer and a third output layer;

[0020] The fifth input layer is used to input a visible light fusion image, which is a feature map obtained by adding the enhanced high-frequency visible light feature map and the low-frequency visible light feature map; the sixth input layer is used to input an infrared fusion image, which is a feature map obtained by adding the high-frequency infrared feature map and the low-frequency infrared feature map;

[0021] The visible light fusion image and the infrared fusion image are both input into the interactive space attention block and the interactive channel attention block. The output of the interactive space attention block and the output of the interactive channel attention block are respectively multiplied element-by-element with the average feature map and then input into the third convolutional layer; the average feature map is the feature map obtained by performing the average feature operation on the visible light fusion image and the infrared fusion image; the output of the third convolutional layer is added element-by-element to the average feature map and then output to the third output layer.

[0022] Preferably, the interactive spatial attention block is configured to perform the following operations:

[0023] ;

[0024] The interactive channel attention block is used to perform the following operations:

[0025] ;

[0026] in, is the spatial interaction result, is the channel interaction result, is the visible light fusion image, is the infrared fusion image, for The transpose of for The transpose of represents the adaptive pooling operation, represents the convolution operation, is the Sigmoid activation function, is the Softmax activation function.

[0027] In a second aspect, the present application provides a method for training a multimodal image fusion and deblurring model as described above, comprising the steps of:

[0028] A1. Obtain a first image dataset and convert the visible light images in the first image dataset from the RGB color domain to the YCrCb color domain to obtain a first training dataset; the first image dataset includes multiple sets of mutually registered and unambiguous visible light images and infrared images;

[0029] A2. Performing local motion blur on the visible light image and infrared image in the first training data set using the mask image to obtain a second training data set;

[0030] A3. Construct a first model and a second model; the second model is the multimodal image fusion and deblurring model described above. The first model is obtained by removing the visible light texture enhancement module from the multimodal image fusion and deblurring model and adding the high-frequency visible light feature map and the low-frequency visible light feature map, which are then input into the corresponding efficient feature interaction fusion module;

[0031] A4. Using the first training dataset and a first total loss function, train the first model to obtain a trained first model;

[0032] A5. Use the second training data set and the trained first model to train the second model based on the second total loss function to obtain a trained multimodal image fusion and deblurring model.

[0033] Preferably, the first total loss function is:

[0034] ;

[0035] in, is the first total loss function value, 、 、 is a hyperparameter, For color loss, For detail loss, is the pixel intensity loss, and:

[0036] ;

[0037] ;

[0038] ;

[0039] in, is the output fused image of the first model, For The corresponding unblurred visible light image in the input image, For The corresponding unblurred infrared image in the input image, It means converting the image from RGB color domain to YCrCb color domain and then extracting CB and CR channels for calculation. represent The Laplace pyramid decomposition result is, represent The Laplace pyramid decomposition result is, represent The Laplace pyramid decomposition result is, 、 are the pixel height and pixel width of the image, respectively.

[0040] Preferably, the second total loss function is:

[0041] ;

[0042] in, is the second total loss function value, 、 、 、 is a hyperparameter, is the label mask and blurred area detection map Mean squared error between, label mask map The mask map used by the second module for local motion blur processing of the current input image, the blurred area detection map The detection image is obtained by performing fuzzy area detection on the current input image of the second module using the salient target detection method. is the perceptual loss of the mask area, is the supervision loss between the efficient feature interaction fusion modules of the first and second models, is the fusion result deviation between the first model and the second model, and:

[0043] ;

[0044] ;

[0045] ;

[0046] ;

[0047] in, is the output fused image of the second model, The output fused image of the trained first model is obtained by inputting the unblurred image corresponding to the current input image of the second model into the trained first model. Indicates When inputting the VGG model, the features extracted by the VGG model in the inference phase, Indicates When inputting the VGG model, the features extracted by the VGG model in the inference phase, is the number of efficient feature interaction fusion modules in the second model, is the output feature of the i-th efficient feature interaction fusion module of the second model, When the unblurred image corresponding to the current input image of the second model is input into the trained first model, the output feature of the i-th efficient feature interaction fusion module of the trained first model is: is a constant.

[0048] In a third aspect, the present application provides a method for using the multimodal image fusion and deblurring model as described above, comprising the steps of:

[0049] B1. Acquire an image to be fused, wherein the image to be fused includes a visible light source image and an infrared source image that are registered with each other;

[0050] B2. Converting the visible light source image from the RGB color gamut to the YCrCb color gamut to obtain a converted visible light source image;

[0051] B3. Input the converted visible light source image and the infrared source image into the trained multimodal image fusion and deblurring model to obtain a fused image output by the trained multimodal image fusion and deblurring model.

[0052] Beneficial effects: The multimodal image fusion and deblurring model provided in this application, as well as its training method and usage method, realizes the integrated processing of multimodal image fusion and deblurring by constructing an integrated multimodal image fusion and deblurring model and adopting a specific texture enhancement and feature interaction fusion mechanism, thereby improving efficiency and reducing error accumulation, and enhancing the texture of visible light images. It has the advantages of realizing the integrated processing of multimodal image fusion and deblurring, improving efficiency and reducing error accumulation, and improving image fusion quality. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] Figure 1 A schematic diagram of the structure of the multimodal image fusion and deblurring model provided in an embodiment of the present application.

[0054] Figure 2 Schematic diagram of the structure of the visible light texture enhancement module.

[0055] Figure 3 Schematic diagram of the structure of the efficient feature interaction fusion module.

[0056] Figure 4 A flowchart of the training method provided in an embodiment of the present application.

[0057] Figure 5 This is a structural diagram of the first model.

[0058] Figure 6 A flowchart of the method of use provided in an embodiment of the present application.

[0059] Figure 7 Comparison chart of fusion results of different fusion methods.

[0060] Explanation of numbers: 1. First input layer; 2. Second input layer; 3. First encoder; 301. First coding layer; 4. Second encoder; 401. Second coding layer; 5. Visible light texture enhancement module; 501. Third input layer; 502. Fourth input layer; 503. First convolutional layer; 504. LDC layer; 505. Sigmoid activation function layer; 506. Second convolutional layer; 507. Second output layer; 6. Efficient feature interaction fusion module; 601. Fifth input layer; 602. Sixth input layer; 603. Interactive spatial attention block; 604. Interactive channel attention block; 605. Third convolutional layer; 606. Third output layer; 7. Decoder; 701. Decoding layer; 8. First output layer. DETAILED DESCRIPTION

[0061] The technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all of the embodiments. The components of the embodiments of the present application generally described and shown in the drawings here can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present application provided in the drawings is not intended to limit the scope of the application for protection, but merely represents the selected embodiments of the present application. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without making creative work fall within the scope of protection of the present application.

[0062] It should be noted that similar reference numerals and letters represent similar items in the following drawings. Therefore, once an item is defined in one drawing, it does not need to be further defined or explained in subsequent drawings. At the same time, in the description of this application, the terms "first", "second", etc. are only used to distinguish the description and should not be understood as indicating or implying relative importance.

[0063] Please refer to Figure 1-Figure 3 , the present application provides a multimodal image fusion and deblurring model, comprising a first input layer 1, a second input layer 2, a first encoder 3, a second encoder 4, multiple visible light texture enhancement modules 5, multiple efficient feature interaction fusion modules 6, a decoder 7 and a first output layer 8;

[0064] The first input layer 1 and the second input layer 2 are respectively used to input the visible light image and the infrared image to be fused and registered with each other; the first encoder 3 and the second encoder 4 are both multi-layer encoders, and the decoder 7 is a multi-layer decoder; the first encoder 3 is used to sequentially extract visible light feature maps of different scales from the visible light image, and the visible light feature maps include high-frequency visible light feature maps and low-frequency visible light feature maps; the second encoder 4 is used to sequentially extract infrared feature maps of different scales from the infrared image, and the infrared feature maps include high-frequency infrared feature maps and low-frequency infrared feature maps;

[0065] Each visible light texture enhancement module 5 and each efficient feature interaction fusion module 6 corresponds to each scale one-to-one (i.e., corresponds to each coding layer of the first encoder 3 and the second encoder 4); at each scale, the visible light texture enhancement module 5 is used to perform texture enhancement processing on the corresponding high-frequency visible light feature map using the corresponding high-frequency infrared feature map (such as Figure 1 As shown, the high-frequency visible light feature map extracted by the first coding layer 301 and the high-frequency infrared feature map extracted by the second coding layer 401 at the same level are input into the corresponding visible light texture enhancement module 5) to obtain the enhanced high-frequency visible light feature map of the corresponding scale; at each scale, the high-frequency infrared feature map and the low-frequency infrared feature map are added, and the enhanced high-frequency visible light feature map and the low-frequency visible light feature map are added, and then input into the corresponding efficient feature interactive fusion module 6 for fusion;

[0066] The decoder 7 is used to perform layer-by-layer upsampling and convolution operations on the output fusion features of each efficient feature interaction fusion module 6 to obtain the final fusion image and output it from the first output layer 8.

[0067] Among them, the multimodal image fusion and deblurring model refers to a technical system that can simultaneously process images from different sensors (such as visible light and infrared) and reduce or eliminate the effects of motion blur in the images.

[0068] Among them, the encoder refers to a neural network structure used to extract multi-scale features from the input image. It usually contains multiple processing layers (i.e., encoding layers), which can capture the information of the image at different resolutions and decompose the features into a high-frequency part containing details and a low-frequency part containing structural information.

[0069] The visible light texture enhancement module 5 refers to a processing unit for enhancing the texture details of the visible light image using the characteristic information of the infrared image, with the purpose of restoring the detail information lost in the visible light image due to blur.

[0070] Among them, the efficient feature interaction fusion module 6 refers to a processing unit for effectively integrating feature information from visible light and infrared modalities to generate fusion features containing complementary information. Its design aims to improve the efficiency and effectiveness of the fusion process.

[0071] Among them, the decoder 7 refers to a neural network structure used to gradually restore the fused feature map to the image resolution, which converts the abstract features into the final visual image through upsampling and convolution operations.

[0072] The core innovation of this application lies in constructing an end-to-end multimodal image fusion and deblurring model, introducing a visible light texture enhancement module 5 at different scales, using the high-frequency information of the infrared image to guide the high-frequency texture restoration of the visible light image, and integrating multi-scale features through an efficient feature interaction fusion module 6, thereby realizing image fusion and visible light deblurring in a single processing process, effectively solving the efficiency and error accumulation problems of the existing two-stage method, and improving the ability to restore details in blurred areas.

[0073] Specifically, the visible light image and infrared image to be fused enter the model through the corresponding input layers respectively. The first encoder 3 extracts visible light feature maps of different scales from the visible light image layer by layer, including high-frequency and low-frequency components. At the same time, the second encoder 4 extracts infrared feature maps of corresponding scales from the infrared image layer by layer, also including high-frequency and low-frequency components. At each corresponding scale, the visible light texture enhancement module 5 receives the high-frequency visible light feature map and high-frequency infrared feature map of the scale, and uses the high-frequency infrared feature map to perform texture enhancement processing on the high-frequency visible light feature map to obtain an enhanced high-frequency visible light feature map. Subsequently, at this scale, the high-frequency and low-frequency feature maps of the infrared are combined, and the enhanced high-frequency visible light feature map and low-frequency visible light feature map are also combined. These two groups of combined features Figure 1 The high-efficiency interactive feature fusion module 6 corresponding to the input scale performs fusion processing to generate fused features at that scale. The decoder 7 receives the fused features output by the high-efficiency interactive feature fusion module 6 at each scale and, starting from the deepest layer, performs upsampling and convolution operations layer by layer to gradually restore the image resolution. Finally, it generates a fused image and outputs it through the first output layer. This entire process is completed within a unified network framework, achieving the coordinated processing of fusion and deblurring.

[0074] Through the above scheme, the present application avoids the inefficiency and error accumulation problems brought about by traditional two-stage processing by integrating image fusion and visible light deblurring in an end-to-end model. Taking advantage of the sensitivity of infrared images to thermal targets and the relatively stable characteristics of their high-frequency information, the visible light texture enhancement module 5 effectively guides and restores the texture details lost in the visible light image due to motion blur, especially with good processing capabilities for local motion blur. The efficient feature interaction fusion module 6 ensures that the complementary information from the two modalities can be fully utilized and integrated during the fusion process. The final generated fused image not only contains the thermal target information of the infrared image, but also restores the clear texture details of the visible light image, improving the image quality and the ability to interpret the scene.

[0075] Preferably, see Figure 1 , the first encoder 3 includes multiple first encoding layers 301, the second encoder 4 includes multiple second encoding layers 401, the decoder 7 includes multiple decoding layers 701, and the number of first encoding layers 301, second encoding layers 401, decoding layers 701, visible light texture enhancement modules 5 and efficient feature interaction fusion modules 6 is the same;

[0076] The first encoder 3 is used to sequentially extract visible light feature maps of different scales from the visible light image through each first encoding layer 301; the second encoder 4 is used to sequentially extract infrared feature maps of different scales from the infrared image through each second encoding layer 401;

[0077] The decoder 7 performs layer-by-layer upsampling and convolution operations from the decoding layer 701 of the last layer to the decoding layer 701 of the first layer. The decoding layer 701 of the first layer is connected to the first output layer 8; the output fusion features of the efficient feature interactive fusion module 6 corresponding to the last two coding layers (the last two layers of the first coding layer 301 and the second coding layer 401) are input into the decoding layer 701 of the last layer. From back to front, the output fusion features of the efficient feature interactive fusion module 6 corresponding to other coding layers are input into the decoding layers 701 from the second to the last layer in turn.

[0078] Among them, the first encoding layer 301, the second encoding layer 401 and the decoding layer 701 refer to the basic processing units that constitute the encoder and decoder 7, which usually include one or more convolution operations, activation functions, and downsampling or upsampling operations for transforming and abstracting the input feature maps.

[0079] Building on the above model, this solution further clarifies the structural correspondence and data flow connection between key components within the model. By setting the number of first encoding layer 301, second encoding layer 401, decoding layer 701, visible light texture enhancement module 5, and efficient feature interaction fusion module 6 to be equal, this ensures that at each scale level of the model, there is a corresponding encoding layer responsible for feature extraction, a corresponding intermediate module responsible for feature processing and fusion, and a corresponding decoding layer 701 responsible for feature recovery. This strict numerical correspondence creates a symmetrical and hierarchical processing architecture, providing structural guarantees for the subsequent precise interaction and fusion of features at different scales. Through their multiple encoding layers, the first encoder 3 and the second encoder 4 are able to gradually extract and abstract feature representations at different scales from visible light and infrared images. These feature maps contain rich content, from low-level edge texture to high-level semantic information. The decoder 7 then gradually restores the spatial resolution of the feature maps through layer-by-layer upsampling and convolution operations from deep to shallow layers, ultimately generating a high-resolution fused image. Crucially, this solution precisely defines how the output fused features of the efficient feature interaction fusion module 6 are input into the decoder 7. Specifically, the outputs of the efficient feature interaction fusion module 6 from the last two layers (typically those with the richest semantic information and the lowest spatial resolution) are fed into the last layer of the decoder 7 (the layer where upsampling begins). This allows the decoder 7 to utilize the most abstract and robust fused features at the beginning of resolution restoration. Subsequently, the outputs of the efficient feature interaction fusion modules 6 from the remaining layers (from deep to shallow, containing increasingly rich detail information) are sequentially fed into the second to last layer of the decoder 7. This connection ensures that as the decoder 7 performs layer-by-layer upsampling to restore resolution, it continuously fuses texture-enhanced and efficiently interactive feature information from the encoder at corresponding scales. By precisely injecting fused features of different scales into the corresponding scale layers of the decoder 7, the decoder 7 can fully utilize multi-scale contextual information, particularly the detail information in shallow features and the semantic information in deep features, when restoring image details and suppressing blur, significantly improving the quality of the fused image and the deblurring effect.

[0080] This structural design and data flow arrangement enables the model to more effectively integrate and utilize multimodal and multi-scale features, overcoming the problems of detail loss and residual blur that may result from simple upsampling, and achieving more refined and robust image fusion and deblurring.

[0081] The specific number of the first coding layer 301, the second coding layer 401, the decoding layer 701, the visible light texture enhancement module 5 and the efficient feature interaction fusion module 6 can be adjusted according to actual needs. Figure 1In

[15] , the number is 4, which has a good deblurring and fusion effect, and will not make the model structure too complicated due to too many numbers, affecting the fusion efficiency.

[0082] In some embodiments, see Figure 2 , the visible light texture enhancement module 5 includes a third input layer 501, a fourth input layer 502, a first convolutional layer 503, an LDC layer 504 (a learnable description convolutional layer), a Sigmoid activation function layer 505, a second convolutional layer 506 and a second output layer 507;

[0083] The third input layer 501 is used to input a high-frequency visible light feature map, and the fourth input layer 502 is used to input a high-frequency infrared feature map. The third input layer 501 inputs the high-frequency visible light feature map into the first convolutional layer 503, and the fourth input layer 502 inputs the high-frequency infrared feature map into the LDC layer 504. The high-frequency visible light feature map is subtracted from the high-frequency infrared feature map and then input into the Sigmoid activation function layer 505. The output of the first convolutional layer 503 is multiplied by the output of the Sigmoid activation function layer 505 and then merged with the output of the LDC layer 504. The merged result is input into the second convolutional layer 506, and the output end of the second convolutional layer 506 is connected to the second output layer 507.

[0084] Specifically, the third input layer 501 and the fourth input layer 502 serve as data inputs, receiving the high-frequency visible light feature map and the high-frequency infrared feature map to be processed, respectively. The first convolutional layer 503 performs a convolution operation on the input high-frequency visible light feature map to extract visible light features. The LDC layer 504 receives the high-frequency infrared feature map and extracts features with local descriptive capabilities from the infrared feature map. These features contain local information related to texture or salient objects in the infrared image and can be used to guide visible light texture enhancement. The sigmoid activation function layer 505 receives the result of subtracting the high-frequency visible light feature map from the high-frequency infrared feature map and maps the difference between 0 and 1, which serves as a weight or gating signal reflecting the degree of difference in high-frequency information between visible light and infrared. The second convolutional layer 506 performs further convolution on the merged features to integrate the information and generate an enhanced feature map. The second output layer 507 serves as a data output, outputting the final enhanced high-frequency visible light feature map.

[0085] The visible light texture enhancement module 5 of the present application achieves texture enhancement by processing high-frequency visible light feature maps and high-frequency infrared feature maps in parallel. The high-frequency visible light feature map first undergoes preliminary feature extraction through the first convolutional layer 503. Simultaneously, the high-frequency infrared feature map is fed into the LDC layer 504, which captures local structure and texture information in the infrared image. Furthermore, the difference between the high-frequency visible light feature map and the high-frequency infrared feature map is calculated and normalized through the Sigmoid activation function layer 505 to generate a weight map. This weight map reflects the degree of similarity or difference in high-frequency information between visible light and infrared. Regions with larger differences (for example, areas where visible light is blurry but infrared is clear) receive higher weights. Subsequently, the output of the first convolutional layer 503 is element-wise multiplied by the output of the Sigmoid activation function layer 505. This means that the visible light features are weighted according to the difference between the high-frequency features of visible light and infrared, thereby enhancing the visible light features in a targeted manner. The weighted visible light features are merged with the infrared local description features extracted by the LDC layer 504, resulting in an enhanced feature map that incorporates both the weighted visible light information and the infrared local description information. Finally, the merged result is fed into the second convolutional layer 506 for further processing and integration, resulting in the final enhanced high-frequency visible light feature map, which is then output via the second output layer 507.

[0086] As a result, the module can effectively utilize infrared high-frequency information, especially its local descriptive features, to specifically enhance the texture of the visible light feature map, which is particularly helpful in restoring details lost due to local blur in the visible light image. The enhanced high-frequency visible light feature map is added to the low-frequency visible light feature map, and the result, along with the sum of the high-frequency infrared feature map and the low-frequency infrared feature map, is input into the efficient feature interaction fusion module 6 for fusion. This can fully utilize the clear texture information of the infrared image to compensate for the blur of the visible light image, thereby generating a higher quality fused image in the subsequent fusion process, effectively solving the problem of multimodal image fusion in the presence of local blur in the visible light image.

[0087] Specifically, the LDC layer 504 is used to perform the following operations:

[0088] ;

[0089] in, is the pixel value of the (i, j) pixel in the output of the LDC layer 504, (i, j) is the pixel coordinate, is a common convolution kernel, for The (i,j) element of is the pixel value of the (i, j) pixel in the input of the LDC layer 504 (i.e., the high-frequency infrared feature map), is a learnable parameter, is a 3×3 all-ones matrix, is a learnable description operator, represents the convolution operation, Represents element-wise multiplication.

[0090] A common convolution kernel is a standard convolution kernel whose weights are learned and determined during training. It is used to perform linear filtering on the input feature map and extract local features. A learnable parameter is a parameter whose value is automatically adjusted by an optimization algorithm during model training. It participates in the operation as a scaling factor or bias term, providing the model with the ability to adjust feature contributions. A learnable description operator is one or a group of convolution kernels or filters learned and determined during training. It is used to extract specific, task-specific local patterns or descriptive information from the input feature map. Compared to common convolution, learnable description convolution (LDC) uses a learnable convolution kernel. By adjusting the convolution kernel weights, it emphasizes the texture information of infrared images and enhances the model's texture feature perception.

[0091] In some embodiments, see Figure 3 , the efficient feature interaction fusion module 6 includes a fifth input layer 601, a sixth input layer 602, an interactive spatial attention block 603, an interactive channel attention block 604, a third convolutional layer 605 and a third output layer 606;

[0092] The fifth input layer 601 is used to input a visible light fusion image, where the visible light fusion image is a feature image obtained by adding the enhanced high-frequency visible light feature image and the low-frequency visible light feature image. The sixth input layer 602 is used to input an infrared fusion image, where the infrared fusion image is a feature image obtained by adding the high-frequency infrared feature image and the low-frequency infrared feature image.

[0093] The visible light fusion image and the infrared fusion image are both input into the interactive spatial attention block 603 and the interactive channel attention block 604. The output of the interactive spatial attention block 603 and the output of the interactive channel attention block 604 are respectively multiplied element-wise with the average feature map and then input into the third convolution layer 605; the average feature map is the average feature operation of the visible light fusion image and the infrared fusion image ( Figure 3 , symbol A represents the feature map obtained by the average feature operation); the output of the third convolutional layer 605 is added element-by-element to the average feature map and then output to the third output layer 606.

[0094] The various components of this module work together. Specifically, the fifth input layer 601 and the sixth input layer 602 receive the preliminarily fused visible light feature map and infrared feature map, respectively. These feature maps are simultaneously fed into the interactive spatial attention block 603 and the interactive channel attention block 604. The interactive spatial attention block 603 learns and generates a spatial weight map, which indicates which regions in the spatial dimension are more important for fusion. This learning process takes into account the relationship between visible light and infrared features. The interactive channel attention block 604 learns and generates a channel weight vector, which indicates which feature channels are more important for fusion in the channel dimension. This learning process also takes into account the interaction between features of different modalities at the channel level. Simultaneously, an average feature operation is performed on the input visible light fusion map and infrared fusion map to produce an average feature map, which can be considered a basic representation of the features of both modalities. The outputs of the interactive spatial attention block 603 (the spatial weight map) and the outputs of the interactive channel attention block 604 (the channel weight vector) are respectively element-wise multiplied with this average feature map. This multiplication operation applies the learned spatial and channel attention weights to the basic average feature representation, thereby retaining the basic information of the two modalities while highlighting the key feature areas and channels that are more beneficial to the fusion task. The attention-weighted features are then input to the third convolutional layer 605 for further feature extraction, integration, and transformation. The output of the third convolutional layer 605 is added element-by-element to the original average feature map to form a residual connection. This residual connection structure helps the model learn the residual information of the features, that is, the incremental changes based on the average features, which not only helps gradient propagation, but also better preserves the original feature information and enhances the details and robustness of the fusion results. Finally, the fused features obtained through the residual connection are output through the third output layer 606 as the fusion result at the current scale for subsequent decoding or further processing.

[0095] Through the above structure and processing flow, the module achieves deep interaction and effective fusion of visible and infrared features, generating a fused feature map containing rich complementary information. This design fully utilizes the rich texture details of visible light images and the robustness of infrared images to thermal targets. Through an interactive attention mechanism, it adaptively adjusts the contributions of different modal features, thereby generating high-quality fused features at different scales, laying the foundation for the ultimate generation of clear and information-rich fused images.

[0096] Specifically, the interactive spatial attention block 603 is used to perform the following operations:

[0097] ;

[0098] The interactive channel attention block 604 is used to perform the following operations:

[0099] ;

[0100] in, is the spatial interaction result, is the channel interaction result, is the visible light fusion image, For infrared fusion image, for The transpose of for The transpose of represents the adaptive pooling operation, represents the convolution operation, is the Sigmoid activation function, is the Softmax activation function.

[0101] Through the above-mentioned specific spatial and channel attention operation methods, this solution can achieve deep interaction between visible light and infrared features in the spatial and channel dimensions. This interaction mechanism enables the model to guide and enhance the features of the visible light image based on the robust information of the infrared image, especially its perception of thermal targets, especially when the visible light image has local motion blur resulting in loss of details. The clear features of the infrared image can help the model restore or reconstruct the detailed information of the blurred area of ​​the visible light image. At the same time, the channel attention mechanism is used to optimize the fusion weights of different channel information, further improving the quality and clarity of the fused image. Therefore, this solution can effectively solve the challenge of fusing infrared and visible light images in the presence of local motion blur, restore the blurred detail information and obtain high-quality fusion results.

[0102] refer to Figure 4 , the present application provides a training method for a multimodal image fusion and deblurring model as described above, comprising the steps of:

[0103] A1. Obtain a first image dataset and convert the visible light images in the first image dataset from the RGB color domain to the YCrCb color domain to obtain a first training dataset; the first image dataset includes multiple sets of mutually registered and unambiguous visible light images and infrared images;

[0104] A2. Using the mask image, perform local motion blur on the visible light image and infrared image in the first training dataset to obtain a second training dataset;

[0105] A3. Constructing a first model and a second model; the second model is the multimodal image fusion and deblurring model mentioned above. The first model removes the visible light texture enhancement module 5 in the multimodal image fusion and deblurring model, and adds the high-frequency visible light feature map and the low-frequency visible light feature map to the corresponding efficient feature interaction fusion module 6 (e.g. Figure 5 shown);

[0106] A4. Using the first training dataset and the first total loss function, train the first model to obtain a trained first model;

[0107] A5. Use the second training dataset and the trained first model to train the second model based on the second total loss function to obtain a trained multimodal image fusion and deblurring model.

[0108] Among them, in step A1, obtaining the first image dataset refers to obtaining the original image set used for model training, which contains multiple groups of mutually aligned and unambiguous visible light images and infrared images, which can be collected from specific scenes or obtained using existing public datasets.

[0109] Among them, in step A2, using the mask image to perform local motion blur processing on the visible light image and infrared image in the first training data set means simulating the local motion blur effect by applying a motion blur kernel to a specific area of ​​the image. The mask image is used to specify the area where the blur is applied, and the motion blur kernel can be generated according to the simulated motion trajectory.

[0110] Among them, in step A3, constructing the first model and the second model refers to configuring a neural network structure for training, the second model is the complete image fusion and deblurring model to be trained, and the first model is a simplified version of the complete model, which is obtained by removing some modules and adjusting the connection method.

[0111] Among them, in step A4, based on the first total loss function, training the first model means using the first total loss function as an optimization target to adjust the internal parameters of the first model so that it achieves the expected performance on the unambiguous data.

[0112] In which, in step A5, based on the second total loss function, training the second model means using the second total loss function as the optimization target to adjust the internal parameters of the second model so that it achieves the expected performance on data with local ambiguity. The second total loss function may include supervision items for the output or intermediate features of the first model.

[0113] The overall working principle of this method is as follows. Through a staged training approach, this method enables the multimodal image fusion and deblurring model to be trained to effectively learn to perform both image fusion and deblurring tasks, particularly in the presence of local motion blur. First, in step A1, a dataset of unblurred visible and infrared images is acquired. The visible images are converted to the YCrCb color domain to separate luminance and color information for easier processing. Next, in step A4, a simplified model (the first model) is trained using this unblurred data and a first total loss function. This simplified model lacks full texture enhancement capabilities and aims to learn to perform multimodal image fusion under unblurred conditions, laying the foundation for subsequent training of the full model. Subsequently, in step A2, the original unblurred data is locally motion blurred using a mask image to generate training data with local blur. This local blur simulates common conditions in real scenes, making the training data more realistic. Finally, in step A5, this locally blurred data and the trained simplified model are used to train the full model (the second model) using a second total loss function. The second total loss function can be designed to guide the complete model to process fuzzy data while its fusion results or intermediate features are consistent with the results or features of the simplified model when processing the corresponding unfuzzy data, so that the complete model can inherit and utilize the prior fusion knowledge obtained by the simplified model while learning the defuzzification ability.

[0114] This training strategy avoids the difficulty of directly training complex models end-to-end. By first learning basic blur-free fusion and then learning joint tasks in blurry conditions, it effectively improves the performance of the model in complex scenes, especially the ability to handle local motion blur, enabling the model to better restore details in blurred areas and generate high-quality fused images.

[0115] The method is further described below with reference to a specific embodiment. In one embodiment, step A1 may specifically include: obtaining a dataset containing multiple sets of mutually registered and unblurred visible light images and infrared images, such as an M3FD dataset. The visible light images in the dataset are converted from the RGB color domain to the YCrCb color domain using a standard color space conversion algorithm to obtain a first training dataset. Step A2 may specifically include: for each set of images in the first training dataset, identifying movable objects (such as vehicles, pedestrians, etc.) in the image, and generating a mask image based on the identification results. The mask image specifies the areas in the image that require local blurring (i.e., the areas where the movable objects are located). A motion blur kernel that simulates the motion trajectory is generated, and the visible light image and infrared image are convolved with the corresponding motion blur kernel within the area specified by the mask image, thereby obtaining a second training dataset with local motion blur. Step A3 may specifically include: constructing a second model whose structure is consistent with the multimodal image fusion and deblurring model to be trained. Construct a first model, whose structure is based on the second model, but removes all visible light texture enhancement modules 5, and directly inputs the corresponding high-frequency visible light feature map and low-frequency visible light feature map of the corresponding scale into the corresponding efficient feature interaction fusion module 6 after adding them. Step A4 can be specifically as follows: using the first training data set, based on the first total loss function, using the Adam optimizer to train the first model, and training until convergence or reaching a preset training round. Step A5 can be specifically as follows: using the second training data set, based on the second total loss function, using the Adam optimizer to train the second model. During the training process, the unblurred image corresponding to the second training data set is input into the trained first model, and its output and intermediate features are obtained for calculating the supervision term in the second total loss function. The training is carried out until convergence or reaching a preset training round, and finally the trained multimodal image fusion and deblurring model is obtained.

[0116] By adopting the above-mentioned training method, the present application can effectively train a multimodal image fusion and deblurring model, so that it can perform image fusion and deblurring tasks simultaneously. This phased training strategy, especially using unblurred data to train a simplified model as a basis, and using data with local motion blur to train a complete model, can overcome the difficulties of directly training complex models end-to-end. This training method enables the model to better learn to deal with local motion blur that is common in real scenes, and improves the model's detail recovery ability and fusion performance in blurred situations. Ultimately, the trained model can generate fused images with richer details and higher quality in the presence of motion blur, which improves the practicality of the model in complex environments.

[0117] Preferably, the first total loss function may be:

[0118] ;

[0119] in, is the first total loss function value, 、 、 is a hyperparameter, For color loss, For detail loss, is the pixel intensity loss, and:

[0120] ;

[0121] ;

[0122] ;

[0123] in, is the output fusion image of the first model, For The corresponding unblurred visible light image in the input image, For The corresponding unblurred infrared image in the input image, It means converting the image from RGB color domain to YCrCb color domain and then extracting CB and CR channels for calculation. represent The Laplace pyramid decomposition result is, represent The Laplace pyramid decomposition result is, represent The Laplace pyramid decomposition result is, 、 are the pixel height and pixel width of the image, respectively.

[0124] The purpose of introducing color loss is to retain the color information in the visible light image as much as possible; in order to force the fused image to obtain more significant details in the infrared image and visible light image, the maximum pixel value function max is used for calculation in the detail loss; and in order to ensure the target saliency in the fused image, the maximum pixel value function max is used for calculation in the pixel intensity loss.

[0125] By adopting the above-mentioned loss function for training, this scheme can achieve the following technical effects. By adopting a combined loss function including color loss, detail loss and pixel intensity loss as the first total loss function, this scheme can comprehensively measure the quality of the fused image generated by the first model on the unblurred data. Color loss ensures that the fused image can effectively retain the color information of the visible light image and avoid color distortion. Detail loss helps the fused image retain high-frequency details such as texture and edges in the source image, thereby improving the clarity of the fusion result. Pixel intensity loss enables the fused image to retain important target information and contrast characteristics in the source image. This comprehensive loss function design can guide the first model to learn high-quality fusion mapping on unblurred data, effectively retaining the key information of the source image, thereby laying a solid foundation for the subsequent training of a complete fusion and deblurring model on blurred data, and improving the final deblurring fusion performance.

[0126] Preferably, the second total loss function may be:

[0127] ;

[0128] in, is the second total loss function value, 、 、 、 is a hyperparameter, is the label mask and blurred area detection map Mean squared error between, label mask map The mask map used by the second module for local motion blur processing of the current input image, the blurred area detection map The detection image is obtained by performing fuzzy area detection on the current input image of the second module using the salient target detection method. is the perceptual loss of the mask area, is the supervision loss between the efficient feature interaction fusion module 6 of the first model and the second model, is the fusion result deviation between the first model and the second model, and:

[0129] ;

[0130] ;

[0131] ;

[0132] ;

[0133] in, is the output fusion image of the second model, The output fused image of the trained first model is obtained by inputting the unblurred image corresponding to the current input image of the second model into the trained first model. Indicates When inputting the VGG model, the features extracted by the VGG model in the inference phase, Indicates When inputting the VGG model, the features extracted by the VGG model in the inference phase, is the number of efficient feature interaction fusion modules 6 in the second model, is the output feature of the i-th efficient feature interaction fusion module 6 of the second model, When the unblurred image corresponding to the current input image of the second model is input into the trained first model, the output feature of the i-th efficient feature interaction fusion module of the trained first model is: is a constant.

[0134] Among them, the label mask map refers to a binary image used to specify which areas need to be blurred when performing local motion blur processing on the original unblurred image. It can be achieved by manually drawing, generating based on target detection results, or generating according to specific rules. The blurred area detection map refers to the analysis of the input image through an algorithm to identify and mark the blurred areas in the image. It can be achieved by methods such as salient target detection based on gradient analysis, frequency analysis, or deep learning models. The salient target detection method refers to a technology that aims to identify the most eye-catching or visually important areas in the image. It can be achieved by methods based on contrast, frequency, prior knowledge, or deep learning models.

[0135] Perceptual loss is a loss function that measures the perceptual similarity between two images. It is typically implemented by comparing features extracted from two images at different layers of a pre-trained deep neural network (such as the VGG model). This can be achieved by calculating the Euclidean distance or cosine similarity between feature maps. The VGG model is a classic convolutional neural network architecture known for its deep layers and small convolution kernels. It is commonly used for image feature extraction tasks and can be implemented using the VGG-16 or VGG-19 models.

[0136] Among them, supervision loss refers to the loss calculated by comparing the difference between the model output and the preset supervision signal (such as label, reference image or reference feature) during the training process. It is used to guide model learning and can be implemented using mean square error, cross entropy or other custom loss functions.

[0137] The fusion result deviation refers to the difference measure between the fused image processed by the model and a reference fused image, which is used to evaluate the quality and consistency of the fusion result. It can be calculated using pixel-level differences, structural similarity, or perceptual differences.

[0138] The second total loss function provided by this solution is used in the second stage of the training method. Its working principle is to comprehensively guide the learning process of the second model when processing locally motion blurred images through the synergistic effect of multiple loss components. Specifically, the mean square error term between the label mask map and the blurred area detection map prompts the model to learn to accurately identify the blurred areas in the image, providing precise regional positioning information for subsequent deblurring processing. The perceptual loss term of the mask area focuses on the blurred area. By comparing the output of the second model in the blurred area with the perceptual features of the unblurred image in the same area after processing by the first model, it guides the model to learn to restore the details and texture of the blurred area, so that the visual effect after deblurring is close to the ideal unblurred state. At the same time, the supervised loss term between the efficient feature interaction fusion module 6 of the first model and the second model constrains the second model at the feature level so that when processing locally blurred images, its internal feature representation can be consistent or similar to the features learned by the first model on the unblurred data. This helps to enhance the model's robustness to blurred input and ensure the effectiveness of the fusion process. Finally, the deviation term of the fusion results between the first model and the second model is directly supervised at the final output level, requiring that the fused deblurred image obtained after the second model processes the locally blurred image be as consistent as possible with the result of the unblurred image processed by the first model. This ensures that the model does not damage the overall fusion quality while deblurring and maintains the stability of the results.

[0139] By combining these loss components, the second total loss function comprehensively guides the training of the second model, enabling it to not only effectively deblur local motion blurred areas, but also achieve high-quality multimodal image fusion across the entire image, ensuring that the results of processing blurred images are highly consistent with those of processing unblurred images. This multifaceted constraint enables the model to effectively address the challenges posed by local motion blur, strike a balance between deblurring and fusion tasks, and thus generate high-quality fused deblurred images.

[0140] refer to Figure 6 , the present application provides a method for using the multimodal image fusion and deblurring model as described above, comprising the steps of:

[0141] B1. Acquire the image to be fused, which includes the mutually registered visible light source image and infrared source image;

[0142] B2. Convert the visible light source image from the RGB color domain to the YCrCb color domain to obtain a converted visible light source image;

[0143] B3. Input the converted visible light source image and infrared source image into the trained multimodal image fusion and deblurring model to obtain a fused image output by the trained multimodal image fusion and deblurring model.

[0144] The image to be fused refers to the actual image data that needs to be fused and deblurred. It can be acquired using various visible light cameras and infrared cameras. Mutually registered visible light and infrared source images refer to image registration techniques that ensure that the same scene points in the two images have corresponding pixel locations on the image plane. This can be achieved using registration methods based on feature point matching, region matching, or deep learning.

[0145] Among them, the conversion of RGB color domain to YCrCb color domain refers to converting the visible light image from the three primary colors of red, green and blue to the brightness (Y) and two chrominance (Cr, Cb) components. It can be implemented using standard color space conversion formulas or related image processing library functions.

[0146] Among them, the trained multimodal image fusion and deblurring model refers to a model whose internal parameters have been adjusted through the training process and has the ability to fuse and deblur the input image. It can be trained using the training method mentioned above.

[0147] This method aims to input the actual image to be fused into a trained model to obtain a fused and deblurred image. First, obtaining mutually registered visible and infrared images is the foundation of multimodal image fusion, ensuring accurate correspondence and integration of information from different sensors. Next, the visible image is converted to the YCrCb color domain, specifically separating the luminance information (Y channel) from the color information (CrCb channels). This helps the subsequent fusion process better utilize the texture and thermal information of the infrared image while preserving the color characteristics of the visible image. This conversion aligns with the color space processing methods that may be used during model training, helping the model leverage its trained performance. Finally, the preprocessed (color space converted) visible image and the original infrared image are fed as input to the model optimized by the training method. Based on its internal structure and learned parameters, the trained model extracts, enhances, interactively fuses, and decodes the input visible and infrared features, ultimately generating an image that fuses rich visible light details with infrared thermal target information and removes local motion blur. By feeding input images in a specific format into a trained model, the model's functionality is realized in real-world applications, solving the problem of efficiently fusing and deblurring images containing localized motion blur. This method, combined with the trained model, enables the model to process real-world input data, thereby achieving its designed functionality.

[0148] Through the above method, the actually collected visible light and infrared images containing local motion blur can be effectively input into the trained multimodal image fusion and deblurring model, thereby obtaining a fused image that combines the rich details of visible light and infrared thermal target information and effectively removes local motion blur. This solves the problem of how the trained model receives and processes input data in practical applications, enabling the model to exert its capabilities in fusion and deblurring.

[0149] Below, the trained multimodal image fusion and deblurring model provided by this application is compared with five infrared and visible light image fusion methods in the prior art. The comparison of the fusion results is shown in the figure below. Figure 7 As shown in the figure, IR represents the infrared source image to be fused; VI represents the visible light source image to be fused; “Deblurring + CoCoNet” means that the source image is first globally deblurred, and then the CoCoNet model is used to fuse the deblurred source image to obtain the fused image; “Deblurring + MFEIF” means that the source image is first globally deblurred, and then the MFEIF model is used to fuse the deblurred source image to obtain the fused image; “Deblurring + ReCoNet” means that the source image is first globally deblurred, and then the ReCoNet model is used to fuse the deblurred source image to obtain the fused image. The blurred source images are fused to obtain a fused image; "Deblurring + SHIP" means that the source images are first globally deblurred, and then the deblurred source images are fused using the SHIP model to obtain a fused image; "Deblurring + TarDAL" means that the source images are first globally deblurred, and then the deblurred source images are fused using the TarDAL model to obtain a fused image; "this method" refers to the fused image obtained using the trained multimodal image fusion and deblurring model provided in this application; wherein, a local enlarged image of the vehicle license plate is displayed in the lower left corner of each image.

[0150] from Figure 7 As can be seen, the fusion results of each comparison method (except for the methods in this application) exhibit varying degrees of detail loss. This is because global blurring affects the entire image uniformly, resulting in a smooth transition overall, while local motion blur manifests as abrupt blurring near the boundaries of moving objects. Therefore, global deblurring algorithms perform poorly when dealing with local motion blur, and they may also adversely affect the clarity of background details. Furthermore, the two-stage process also introduces error accumulation. This is evident in the fusion results of "deblurring + CoCoNet," where detailed information is severely lost.

[0151] In addition, the performance indicators of these methods are compared in Table 1:

[0152] Table 1

[0153]

[0154] As can be seen from Table 1, this application (Quality Metric based on Edge Preservation, an indicator used to evaluate the quality of the fused image), (edge ​​strength), Chen-Blum metric is a perceptual quality evaluation index based on the human visual system model , PSNR (peak signal-to-noise ratio) and MS-SSIM (multi-scale structural similarity), showing that this application performs better than the five comparison methods in terms of robustness, human visual effects, edge information, structural information and information volume.

[0155] The above description is merely an embodiment of the present application and is not intended to limit the scope of protection of the present application. For those skilled in the art, various modifications and variations of the present application are possible. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present application shall be included in the scope of protection of the present application.

Claims

1. A multimodal image fusion and deblurring model, characterized in that: It includes a first input layer, a second input layer, a first encoder, a second encoder, multiple visible light texture enhancement modules, multiple efficient feature interaction fusion modules, a decoder and a first output layer; The first input layer and the second input layer are respectively used to input a visible light image and an infrared image to be fused and registered with each other; the first encoder and the second encoder are both multi-layer encoders, and the decoder is a multi-layer decoder; the first encoder is used to sequentially extract visible light feature maps of different scales from the visible light image, wherein the visible light feature maps include high-frequency visible light feature maps and low-frequency visible light feature maps; the second encoder is used to sequentially extract infrared feature maps of different scales from the infrared image, wherein the infrared feature maps include high-frequency infrared feature maps and low-frequency infrared feature maps; Each of the visible light texture enhancement modules and each of the efficient feature interaction fusion modules corresponds to each scale one-to-one; at each scale, the visible light texture enhancement module is used to perform texture enhancement processing on the corresponding high-frequency visible light feature map using the corresponding high-frequency infrared feature map to obtain an enhanced high-frequency visible light feature map of the corresponding scale; at each scale, the high-frequency infrared feature map and the low-frequency infrared feature map are added, and the enhanced high-frequency visible light feature map and the low-frequency visible light feature map are added, and then input together into the corresponding efficient feature interaction fusion module for fusion; The decoder is used to perform layer-by-layer upsampling and convolution operations on the output fusion features of each of the efficient feature interactive fusion modules to obtain a final fusion image and output it from the first output layer; The visible light texture enhancement module includes a third input layer, a fourth input layer, a first convolutional layer, an LDC layer, a Sigmoid activation function layer, a second convolutional layer and a second output layer; The third input layer is used to input a high-frequency visible light feature map, and the fourth input layer is used to input a high-frequency infrared feature map. The third input layer inputs the high-frequency visible light feature map into the first convolutional layer, and the fourth input layer inputs the high-frequency infrared feature map into the LDC layer. The high-frequency visible light feature map is subtracted from the high-frequency infrared feature map and then input into the Sigmoid activation function layer. The output of the first convolutional layer is multiplied by the output of the Sigmoid activation function layer and then merged with the output of the LDC layer. The merged result is input into the second convolutional layer, and the output end of the second convolutional layer is connected to the second output layer. The LDC layer is used to perform the following operations: ； in, is the pixel value of the (i, j) pixel in the output of the LDC layer, (i, j) is the pixel coordinate, is a common convolution kernel, for The (i,j) element of is the pixel value of the (i, j) pixel in the input of the LDC layer, is a learnable parameter, is a 3×3 all-ones matrix, is a learnable description operator, represents the convolution operation, Represents element-wise multiplication.

2. The multimodal image fusion and deblurring model according to claim 1, characterized in that: The first encoder includes multiple first encoding layers, the second encoder includes multiple second encoding layers, the decoder includes multiple decoding layers, and the number of the first encoding layers, the second encoding layers, the decoding layers, the visible light texture enhancement module, and the efficient feature interaction fusion module is the same; The first encoder is used to sequentially extract visible light feature maps of different scales from the visible light image through each first encoding layer; the second encoder is used to sequentially extract infrared feature maps of different scales from the infrared image through each second encoding layer; The decoder performs upsampling and convolution operations layer by layer from the decoding layer of the last layer to the decoding layer of the first layer, and the decoding layer of the first layer is connected to the first output layer; the output fusion features of the efficient feature interactive fusion module corresponding to the last two coding layers are input into the decoding layer of the last layer, and from back to front, the output fusion features of the efficient feature interactive fusion module corresponding to other coding layers are input into the decoding layers from the second to last layer to the second layer in sequence.

3. The multimodal image fusion and deblurring model according to claim 1, wherein: The efficient feature interaction fusion module includes a fifth input layer, a sixth input layer, an interactive spatial attention block, an interactive channel attention block, a third convolutional layer and a third output layer; The fifth input layer is used to input a visible light fusion image, which is a feature map obtained by adding the enhanced high-frequency visible light feature map and the low-frequency visible light feature map; the sixth input layer is used to input an infrared fusion image, which is a feature map obtained by adding the high-frequency infrared feature map and the low-frequency infrared feature map; The visible light fusion image and the infrared fusion image are both input into the interactive spatial attention block and the interactive channel attention block. The outputs of the interactive spatial attention block and the interactive channel attention block are respectively multiplied element-by-element with the average feature map and then input into the third convolutional layer. The average feature map is a feature map obtained by performing an average feature operation on the visible light fusion map and the infrared fusion map; the output of the third convolutional layer is added element-by-element to the average feature map and then output to the third output layer.

4. The multimodal image fusion and deblurring model according to claim 3, wherein: The interactive spatial attention block is used to perform the following operations: ； The interactive channel attention block is used to perform the following operations: ； in, is the spatial interaction result, is the channel interaction result, is the visible light fusion image, is the infrared fusion image, for The transpose of for The transpose of represents the adaptive pooling operation, represents the convolution operation, is the Sigmoid activation function, is the Softmax activation function.

5. A method for training a multimodal image fusion and deblurring model according to any one of claims 1 to 4, characterized in that: Including steps: A1. Obtain a first image dataset and convert the visible light images in the first image dataset from the RGB color domain to the YCrCb color domain to obtain a first training dataset; the first image dataset includes multiple sets of mutually registered and unambiguous visible light images and infrared images; A2. Performing local motion blur on the visible light image and infrared image in the first training data set using the mask image to obtain a second training data set; A3. Construct a first model and a second model; the second model is the multimodal image fusion and deblurring model. The first model is obtained by removing the visible light texture enhancement module from the multimodal image fusion and deblurring model and adding the high-frequency visible light feature map and the low-frequency visible light feature map, which are then input into the corresponding efficient feature interaction fusion module; A4. Using the first training dataset and a first total loss function, train the first model to obtain a trained first model; A5. Use the second training data set and the trained first model to train the second model based on the second total loss function to obtain a trained multimodal image fusion and deblurring model.

6. The training method according to claim 5, characterized in that The first total loss function is: ； in, is the first total loss function value, 、 、 is a hyperparameter, For color loss, For detail loss, is the pixel intensity loss, and: ； ； ； in, is the output fused image of the first model, For The corresponding unblurred visible light image in the input image, For The corresponding unblurred infrared image in the input image, It means converting the image from RGB color domain to YCrCb color domain and then extracting CB and CR channels for calculation. represent The Laplace pyramid decomposition result is, represent The Laplace pyramid decomposition result is, represent The Laplace pyramid decomposition result is, 、 are the pixel height and pixel width of the image, respectively.

7. The training method according to claim 5, characterized in that The second total loss function is: ； in, is the second total loss function value, 、 、 、 is a hyperparameter, is the label mask and blurred area detection map Mean squared error between, label mask map The mask map used by the second module for local motion blur processing of the current input image, the blurred area detection map The detection image is obtained by performing fuzzy area detection on the current input image of the second module using the salient target detection method. is the perceptual loss of the mask area, is the supervision loss between the efficient feature interaction fusion modules of the first and second models, is the fusion result deviation between the first model and the second model, and: ； ； ; ； in, is the output fused image of the second model, The output fused image of the trained first model is obtained by inputting the unblurred image corresponding to the current input image of the second model into the trained first model. Indicates When inputting the VGG model, the features extracted by the VGG model in the inference phase, Indicates When inputting the VGG model, the features extracted by the VGG model in the inference phase, is the number of efficient feature interaction fusion modules in the second model, is the output feature of the i-th efficient feature interaction fusion module of the second model, When the unblurred image corresponding to the current input image of the second model is input into the trained first model, the output feature of the i-th efficient feature interaction fusion module of the trained first model is: is a constant.

8. A method for using the multimodal image fusion and deblurring model according to any one of claims 1 to 4, characterized in that: Including steps: B1. Acquire an image to be fused, wherein the image to be fused includes a visible light source image and an infrared source image that are registered with each other; B2. Converting the visible light source image from the RGB color gamut to the YCrCb color gamut to obtain a converted visible light source image; B3. Input the converted visible light source image and the infrared source image into the trained multimodal image fusion and deblurring model to obtain a fused image output by the trained multimodal image fusion and deblurring model.

Images