A training method of an image fusion model, an image fusion method and device

Abstract

The application discloses a kind of training method of image fusion model, image fusion method and device, relate to image fusion field, the training method of image fusion model, its steps mainly include: image preprocessing, feature extraction, feature fusion, image reconstruction, loss calculation and iterative training, segmentation test and retraining, finally obtain the trained image fusion model.The image fusion model training method, image fusion method and device provided by the present application can effectively capture the global information of the image, adapt to the diversified image characteristics, and improve the performance of the fused image in advanced visual tasks.

Description

Technical Field

[0001] This invention relates to the field of computer vision image processing technology, and more specifically, to infrared and visible light image fusion technology, proposing a training method for an image fusion model, an image fusion method, and an apparatus. Background Technology

[0002] Image fusion is a key research area in computer vision, aiming to generate a new image by combining information from two or more different images, thereby providing richer information and better visual representation. With the rapid development of deep learning technology, deep learning-based image fusion methods have become a research hotspot. These methods, leveraging their powerful feature extraction and generalization capabilities, have greatly promoted the progress of image fusion technology.

[0003] However, traditional deep learning-based image fusion methods, such as those based on Convolutional Neural Networks (CNNs), Auto Encoders (AEs), or Generative Adversarial Networks (GANs), while achieving significant progress in image fusion quality, still have some limitations. For example, CNN methods often struggle to capture global information in an image, while AE and GAN methods also face challenges in training and feature fusion strategies. These limitations affect the performance of image fusion algorithms, especially in high-level vision tasks involving global context and long-range dependencies.

[0004] Furthermore, with the rise of Vision Transformer (ViT), it has shown great potential in handling long-range dependencies, providing a new perspective for the field of image fusion. Transformer models handle long-distance dependencies through self-attention mechanisms, making them well-suited for tasks requiring global context integration, such as image fusion. However, current Transformer-based image fusion methods still have shortcomings in adapting to the inherent characteristics of source images and adjusting the Transformer structure based on the correlation between source images. To improve the practicality of these algorithms, some researchers have begun to combine image fusion tasks with high-level vision tasks to enhance image fusion performance in subsequent high-level vision tasks. While these methods recognize the importance of subsequent vision tasks, there is still room for improvement in the design of joint training strategies, as they cannot effectively combine subsequent enhancement modules and fusion networks. Summary of the Invention

[0005] The purpose of this invention is to provide a training method, an image fusion method, and an apparatus for an image fusion model, which can effectively capture global information of an image, adapt to diverse image characteristics, and improve the performance of fused images in advanced vision tasks.

[0006] The technical solution adopted by this invention to solve its technical problem is to construct a training method for an image fusion model. The image fusion model includes an image acquisition module, a contrast residual module, a texture enhancement module, a cross-feature attention module, and an image reconstruction module. The image acquisition module is used to acquire and preprocess infrared and visible light images. The contrast residual module is used to extract infrared output features. The texture enhancement module is used to extract visible light output features. The cross-feature attention module is used to obtain fusion features. The image reconstruction module is used to generate a fused image. The training method includes the following steps: S1: Acquire infrared and visible light images, align the infrared and visible light images temporally and spatially, and convert them into tensor formats to obtain infrared and visible light tensors; S2: Use the contrast residual module to perform feature extraction processing on the infrared tensor to obtain infrared output features; Use the texture enhancement module... S3: Extract features from the visible light tensor to obtain visible light output features; S4: Use a cross-feature attention module to fuse the infrared output features and visible light output features to obtain fused features; S5: Use an image reconstruction module to reconstruct the fused features to obtain a fused image; Construct a content loss function to calculate the content loss of the image fusion model; S6: Obtain a preset value for the content loss. If the content loss function is not less than the preset value, return to step S1 and continue training the image fusion model; If the content loss function is less than the preset value, stop training and obtain a pre-trained image fusion model; S7: Obtain infrared and visible light images, and use the pre-trained image fusion model to generate a fused image; Obtain a semantic segmentation model, and further train the pre-trained image fusion model based on the semantic segmentation model and the fused image to obtain a trained image fusion model.

[0007] Furthermore, step S2 of the training method for the above image fusion model includes the following steps: using the contrast residual module to perform feature extraction processing on the infrared tensor to obtain the infrared output features, as shown in the formula: ,in Infrared output characteristics To compare the residual modules, Infrared input characteristics, It is a 3×3 convolutional layer. It is a 1×1 convolutional layer. This is an element-wise summation operation; the visible light tensor is processed using a texture enhancement module to extract features, resulting in visible light output features, as shown in the formula: ,in Visible light output characteristics, For texture enhancement module, Δ represents the visible light input feature, and Δ is the Laplacian operator.

[0008] Furthermore, the aforementioned comparative residual module includes a first 3×3 convolutional layer, a second 3×3 convolutional layer, an activation function, a 1×1 convolutional layer, and a summer; the first 3×3 convolutional layer, the activation function, and the second 3×3 convolutional layer are connected in series, the input of the first 3×3 convolutional layer is connected to the input of the 1×1 convolutional layer, the output of the first 3×3 convolutional layer is connected to the first input of the summer, and the output of the 1×1 convolutional layer is connected to the second input of the summer; the first and second 3×3 convolutional layers are used to extract 3×3 image features, the activation function is used to activate the first 3×3 convolutional layer, the 1×1 convolutional layer is used to extract 1×1 image features, and the summer is used for element-wise summation.

[0009] Furthermore, the texture enhancement module includes a first 3×3 convolutional layer, a second 3×3 convolutional layer, an activation function, a Laplacian operator, a 1×1 convolutional layer, and a summer. The first 3×3 convolutional layer, the activation function, and the second 3×3 convolutional layer are connected in series. The input of the first 3×3 convolutional layer is connected to the input of the Laplacian operator, the output of the Laplacian operator is connected to the input of the 1×1 convolutional layer, the output of the first 3×3 convolutional layer is connected to the first input of the summer, and the output of the 1×1 convolutional layer is connected to the second input of the summer. The Laplacian operator is used to sharpen the image, the first and second 3×3 convolutional layers are used to extract 3×3 image features, the activation function is used to activate the first 3×3 convolutional layer, the 1×1 convolutional layer is used to extract 1×1 image features, and the summer is used for element-wise summation.

[0010] Furthermore, the aforementioned cross-feature attention module includes an infrared attention module and a visible light attention module; furthermore, the training method step S3 of the aforementioned image fusion model includes the following steps: using the infrared attention module to perform feature extraction processing on the infrared output features to obtain the output of the infrared attention module, as shown in the formula: ,in, For the input of the infrared attention module, For the output of the infrared attention module, For the infrared attention module, `Trans()` is a transformer operation used for feature transformation. The visible light attention module extracts features from the visible light output, yielding the output of the visible light attention module, as shown in the formula: ,in, This serves as the input for the visible light attention module. This is the output of the visible light attention module. For the visible light attention module, the outputs of the infrared attention module and the visible light attention module are concatenated to obtain the fused features, as shown in the formula: ,in, As a feature of fusion, This is a concatenation operation used to combine depth features extracted from different types of images.

[0011] Furthermore, step S4 of the training method for the above image fusion model includes the following steps: using the image reconstruction module to perform image reconstruction processing on the fusion features to obtain the fused image, as shown in the formula: ,in, This is an image reconstruction module used to reconstruct fused features. As a feature of fusion, To fuse the images, a content loss function is constructed, as shown in the formula:

[0012]

[0013]

[0014]

[0015] in, For content loss function, For strength loss, For structural loss, For the loss of details, The first adjustment constant is... This is the second adjustment constant, used to adjust the weights of strength loss, structural loss, and detail loss. The height of the image. The width of the image. I ir Infrared image, I vi Visible light image, It is a 1-norm. The maximum value of the pixels in the two images. and Each represents one of two images. and Represents constant coefficients. and These represent the mean and standard deviation, respectively. express and Covariance between For the Laplace operator, used to extract from and High-frequency information is extracted along the axial direction. This indicates the absolute value operation.

[0016] Furthermore, step S6 of the training method for the above image fusion model includes the following steps: S61: Acquire infrared and visible light images, and generate a fused image using the pre-trained image fusion model; S62: Acquire a semantic segmentation model, and segment the fused image using the semantic segmentation model to obtain a segmented image; S63: Calculate the segmentation loss based on the fused image and the segmented image, as shown in the formula:

[0017]

[0018]

[0019] in, To divide the loss, Main segmentation loss, For additional segmentation loss, To balance the constant between the main segmentation loss and the additional segmentation loss, This is the vector obtained from the segmentation labels. The predicted segmentation result is given, where H is the image height, W is the image width, and C is the number of image channels. For the segmentation result obtained using deeper feature maps; S64: Obtain the preset value of segmentation loss. If the segmentation loss is not less than the preset value of segmentation loss, return to step S61 to further train the pre-trained image fusion model; if the segmentation loss is less than the preset value of segmentation loss, stop training and obtain the trained image fusion model.

[0020] This invention also provides a training device for an image fusion model, wherein the image fusion model includes an image acquisition module, a contrast residual module, a texture enhancement module, a cross-feature attention module, and an image reconstruction module; the image acquisition module is used to acquire and preprocess infrared and visible light images; the contrast residual module is used to extract infrared output features; the texture enhancement module is used to extract visible light output features; the cross-feature attention module is used to obtain fusion features; and the image reconstruction module is used to generate a fused image; the device includes the following modules: an image preprocessing module, used to: acquire infrared and visible light images, align the infrared and visible light images temporally and spatially and convert them into tensor formats to obtain infrared and visible light tensors; a feature extraction module, used to: perform feature extraction processing on the infrared tensor using the contrast residual module to obtain infrared output features; and perform feature extraction processing on the visible light tensor using the texture enhancement module to obtain visible light output features. The system comprises the following modules: Feature fusion module, which uses the cross-feature attention module to fuse infrared and visible light output features to obtain fused features; Image reconstruction and content loss module, which uses the image reconstruction module to reconstruct the fused features to obtain a fused image; Constructing a content loss function to calculate the content loss of the image fusion model; Content loss and iteration module, which obtains a preset value for content loss, and when the content loss function is not less than the preset value, returns to step S1 to continue training the image fusion model; When the content loss function is less than the preset value, training stops, and a pre-trained image fusion model is obtained; Segmentation testing and retraining module, which obtains infrared and visible light images, generates a fused image using the pre-trained image fusion model; Obtains a semantic segmentation model, and further trains the pre-trained image fusion model based on the semantic segmentation model and the fused image to obtain a trained image fusion model.

[0021] The present invention also provides an image fusion method, comprising the following steps: acquiring an infrared image and a visible light image to be fused; acquiring an image fusion model, training the image fusion model using the above-mentioned image fusion model training method to obtain a trained image fusion model; and using the trained image fusion model to perform image fusion processing on the infrared image and the visible light image to be fused to obtain a fused image.

[0022] The present invention also provides an image fusion apparatus, the apparatus comprising the following modules: an image acquisition module, used for: acquiring an infrared image and a visible light image to be fused; a model training module, used for: acquiring an image fusion model, and training the image fusion model using the above-mentioned image fusion model training method to obtain a trained image fusion model; and a fusion module, used for: using the trained image fusion model to perform image fusion processing on the infrared image and the visible light image to be fused to obtain a fused image.

[0023] The training method, image fusion method, and apparatus for the image fusion model provided by this invention have the following beneficial effects: This invention overcomes the limitations of existing deep learning methods in image fusion by employing a unique attention mechanism and innovative training strategy, establishing an attention-based image fusion method for advanced visual tasks. The fundamental principle is to extract features from both infrared and visible light images and generate a fused image with stronger representational capabilities. First, the invention acquires the infrared and visible light images to be fused and performs necessary preprocessing to ensure the effectiveness and accuracy of the fusion. Then, it extracts features from both modalities using a contrastive residual module and a texture enhancement module. The contrastive residual module extracts salient features from the infrared image, while the texture enhancement module focuses on the detailed information of the visible light image. Next, the system utilizes a cross-feature attention module to analyze and fuse the extracted depth features. This module strengthens the complementarity and correlation between different image features through an attention mechanism, thereby selecting the most suitable fusion features. This step effectively combines the thermal imaging advantages of infrared images with the rich detail of visible light images. Then, through an image reconstruction module, the fusion network converts the concatenated features into a fused image. Finally, the training strategy employed in this invention further optimizes the performance of the fused image on advanced visual tasks. First, the fusion network is pre-trained on a large-scale dataset to improve its generalization ability. Second, the pre-trained network is used to generate fused images, which serve as the basis for subsequent training. Finally, feedback training from the semantic segmentation model further enhances the performance of the fused images in advanced vision tasks.

[0024] (1) It can handle infrared and visible light images in complex environments, and improves the fusion effect of infrared and visible light images.

[0025] (2) It can address the problem of the weak relationship between infrared and visible light images, and strengthen the complementarity and correlation between different image features.

[0026] (3) It enhances the application value of fused images in advanced vision tasks. Attached Figure Description

[0027] The present invention will be further described below with reference to the accompanying drawings and embodiments. In the accompanying drawings: Figure 1 This is a flowchart of the training method, image fusion method and apparatus for the image fusion model provided by the present invention; Figure 2 This is a schematic diagram of the fused image provided by the present invention; Where (a) is the input infrared image, (b) is the input visible light image, and (c) is the output fused image; Figure 3This is a schematic diagram of the overall structure provided by the present invention; Figure 4 This is a schematic diagram of the fusion network provided by the present invention; Figure 5 This is a schematic diagram of the structure of the contrast residual module and the texture enhancement module provided by the present invention; (a) is a schematic diagram of the contrast residual module structure, and (b) is a schematic diagram of the texture enhancement module structure. Figure 6 This is a schematic diagram of the cross-feature attention module structure provided by the present invention; Among them, (a) is the infrared attention module, and (b) is the cross-feature attention module; Figure 7 This is a schematic diagram of the three-stage training strategy provided by the present invention; Figure 8 This is a qualitative comparison of the present invention with other fusion methods; Figure 9 This is a comparison of the segmentation results between the present invention and other fusion methods; Figure 10 This is a comparison of pedestrian detection results between the present invention and other fusion methods; Figure 11 This is a structural diagram of an attention-based image fusion device for advanced visual tasks provided by the present invention. Detailed Implementation

[0028] To provide a clearer understanding of the technical features, objectives, and effects of the present invention, specific embodiments of the present invention will now be described in detail with reference to the accompanying drawings.

[0029] Figure 1This diagram illustrates one implementation of the model training method of this embodiment. In this embodiment, the image fusion model includes an image acquisition module, a contrast residual module, a texture enhancement module, a cross-feature attention module, and an image reconstruction module. The image acquisition module is used to acquire and preprocess infrared and visible light images. The contrast residual module is used to extract infrared output features. The texture enhancement module is used to extract visible light output features. The cross-feature attention module is used to obtain fusion features. The image reconstruction module is used to generate a fused image. The training method includes the following steps: S1: Acquire infrared and visible light images, align the infrared and visible light images in time and space, and convert them into tensor formats to obtain infrared and visible light tensors; S2: Use the contrast residual module to perform feature extraction processing on the infrared tensor to obtain infrared output features; use the texture enhancement module to perform feature extraction processing on the visible light tensor to obtain... S3: Use the cross-feature attention module to fuse the infrared output features and visible light output features to obtain fused features; S4: Use the image reconstruction module to reconstruct the fused features to obtain a fused image; Construct a content loss function to calculate the content loss of the image fusion model; S5: Obtain a preset value for the content loss. If the content loss function is not less than the preset value, return to step S1 and continue training the image fusion model; If the content loss function is less than the preset value, stop training and obtain a pre-trained image fusion model; S6: Obtain infrared and visible light images, and use the pre-trained image fusion model to generate a fused image; Obtain a semantic segmentation model, and further train the pre-trained image fusion model based on the semantic segmentation model and the fused image to obtain a trained image fusion model.

[0030] Specifically, the training method step S2 of the above image fusion model includes the following steps: using the contrast residual module to perform feature extraction processing on the infrared tensor to obtain the infrared output features, as shown in the formula: ,in Infrared output characteristics To compare the residual modules, Infrared input characteristics, It is a 3×3 convolutional layer. It is a 1×1 convolutional layer. This is an element-wise summation operation; the visible light tensor is processed using a texture enhancement module to extract features, resulting in visible light output features, as shown in the formula: ,in Visible light output characteristics, For texture enhancement module, Δ represents the visible light input feature, and Δ is the Laplacian operator.

[0031] Specifically, the aforementioned comparison residual module includes a first 3×3 convolutional layer, a second 3×3 convolutional layer, an activation function, a 1×1 convolutional layer, and a summer. The first 3×3 convolutional layer, the activation function, and the second 3×3 convolutional layer are connected in series. The input of the first 3×3 convolutional layer is connected to the input of the 1×1 convolutional layer, the output of the first 3×3 convolutional layer is connected to the first input of the summer, and the output of the 1×1 convolutional layer is connected to the second input of the summer. The first and second 3×3 convolutional layers are used to extract 3×3 image features, the activation function is used to activate the first 3×3 convolutional layer, the 1×1 convolutional layer is used to extract 1×1 image features, and the summer is used for element-wise summation.

[0032] Specifically, the texture enhancement module includes a first 3×3 convolutional layer, a second 3×3 convolutional layer, an activation function, a Laplacian operator, a 1×1 convolutional layer, and a summer. The first 3×3 convolutional layer, the activation function, and the second 3×3 convolutional layer are connected in series. The input of the first 3×3 convolutional layer is connected to the input of the Laplacian operator, the output of the Laplacian operator is connected to the input of the 1×1 convolutional layer, the output of the first 3×3 convolutional layer is connected to the first input of the summer, and the output of the 1×1 convolutional layer is connected to the second input of the summer. The Laplacian operator is used to sharpen the image, the first and second 3×3 convolutional layers are used to extract 3×3 image features, the activation function is used to activate the first 3×3 convolutional layer, the 1×1 convolutional layer is used to extract 1×1 image features, and the summer is used for element-wise summation.

[0033] Specifically, the aforementioned cross-feature attention module includes an infrared attention module and a visible light attention module; further, the training method step S3 of the aforementioned image fusion model includes the following steps: using the infrared attention module to perform feature extraction processing on the infrared output features to obtain the output of the infrared attention module, as shown in the formula: ,in, For the input of the infrared attention module, For the output of the infrared attention module, For the infrared attention module, `Trans()` is a transformer operation used for feature transformation. The visible light attention module extracts features from the visible light output, yielding the output of the visible light attention module, as shown in the formula: ,in, This serves as the input for the visible light attention module. This is the output of the visible light attention module. For the visible light attention module, the outputs of the infrared attention module and the visible light attention module are concatenated to obtain the fused features, as shown in the formula: ,in, As a feature of fusion, This is a concatenation operation used to combine depth features extracted from different types of images.

[0034] Specifically, step S4 of the training method for the above image fusion model includes the following steps: using the image reconstruction module to perform image reconstruction processing on the fusion features to obtain the fused image, as shown in the formula: ,in, This is an image reconstruction module used to reconstruct fused features. As a feature of fusion, To fuse the images, a content loss function is constructed, as shown in the formula:

[0035]

[0036]

[0037]

[0038] in, For content loss function, For strength loss, For structural loss, For the loss of details, The first adjustment constant is... This is the second adjustment constant, used to adjust the weights of strength loss, structural loss, and detail loss. The height of the image. The width of the image. I ir Infrared image, I vi Visible light image, It is a 1-norm. The maximum value of the pixels in the two images. and Each represents one of two images. and Represents constant coefficients. and These represent the mean and standard deviation, respectively. express and Covariance between For the Laplace operator, used to extract from and High-frequency information is extracted along the axial direction. This indicates the absolute value operation.

[0039] Specifically, the training method step S6 of the above image fusion model includes the following steps: S61: Acquire infrared and visible light images, and generate a fused image using the pre-trained image fusion model; S62: Acquire a semantic segmentation model, and segment the fused image using the semantic segmentation model to obtain a segmented image; S63: Calculate the segmentation loss based on the fused image and the segmented image, as shown in the formula:

[0040]

[0041]

[0042] in, To divide the loss, Main segmentation loss, For additional segmentation loss, To balance the constant between the main segmentation loss and the additional segmentation loss, This is the vector obtained from the segmentation labels. The predicted segmentation result is given, where H is the image height, W is the image width, and C is the number of image channels. For the segmentation result obtained using deeper feature maps; S64: Obtain the preset value of segmentation loss. If the segmentation loss is not less than the preset value of segmentation loss, return to step S61 to further train the pre-trained image fusion model; if the segmentation loss is less than the preset value of segmentation loss, stop training and obtain the trained image fusion model.

[0043] This embodiment provides a training device for an image fusion model, wherein the image fusion model includes an image acquisition module, a contrast residual module, a texture enhancement module, a cross-feature attention module, and an image reconstruction module; the image acquisition module is used to acquire and preprocess infrared images and visible light images; the contrast residual module is used to extract infrared output features; the texture enhancement module is used to extract visible light output features; the cross-feature attention module is used to obtain fusion features; and the image reconstruction module is used to generate a fused image; the device includes the following modules: an image preprocessing module, used to: acquire infrared images and visible light images, align the infrared images and visible light images in time and space and convert them into tensor format to obtain infrared tensors and visible light tensors; a feature extraction module, used to: perform feature extraction processing on the infrared tensor using the contrast residual module to obtain infrared output features; and perform feature extraction processing on the visible light tensor using the texture enhancement module to obtain visible light output features. The system comprises the following modules: Feature fusion module, which uses the cross-feature attention module to fuse infrared and visible light output features to obtain fused features; Image reconstruction and content loss module, which uses the image reconstruction module to reconstruct the fused features to obtain a fused image; Constructing a content loss function to calculate the content loss of the image fusion model; Content loss and iteration module, which obtains a preset value for content loss, and when the content loss function is not less than the preset value, returns to step S1 to continue training the image fusion model; When the content loss function is less than the preset value, training stops, and a pre-trained image fusion model is obtained; Segmentation testing and retraining module, which obtains infrared and visible light images, generates a fused image using the pre-trained image fusion model; Obtains a semantic segmentation model, and further trains the pre-trained image fusion model based on the semantic segmentation model and the fused image to obtain a trained image fusion model.

[0044] This embodiment provides an image fusion method, including the following steps: acquiring an infrared image and a visible light image to be fused; acquiring an image fusion model, training the image fusion model using the above-mentioned image fusion model training method to obtain a trained image fusion model; and using the trained image fusion model to perform image fusion processing on the infrared image and the visible light image to be fused to obtain a fused image.

[0045] This embodiment provides an image fusion device, which includes the following modules: an image acquisition module, used to acquire an infrared image and a visible light image to be fused; a model training module, used to acquire an image fusion model and train the image fusion model using the above-mentioned image fusion model training method to obtain a trained image fusion model; and a fusion module, used to perform image fusion processing on the infrared image and the visible light image to be fused using the trained image fusion model to obtain a fused image.

[0046] In some embodiments, the first adjustment constant is 1 and the second adjustment constant is 10 in the training method of the above image fusion model.

[0047] In some embodiments, the preset values ​​for content loss and segmentation loss in the training method of the above image fusion model are 0.0015 and 0.02, respectively.

[0048] In some embodiments, the image reconstruction module in the training method of the above image fusion model can be obtained by splicing together the information of each pixel in the fusion features, or it can be achieved by weighted averaging and taking the largest value.

[0049] In some embodiments, the image fusion method described above mainly includes the following steps: Acquiring the infrared and visible light images to be fused; specifically, in this embodiment, the infrared and visible light images to be fused need to be of the same size and spatially aligned. The source image and the fusion result are as follows: Figure 2 As shown below, the datasets used in the experiments are examples. MSRS Multimodal Image Dataset: The MSRS dataset is a new dataset built upon the MFNet dataset, containing 1444 pairs of high-quality, temporally and spatially aligned infrared and visible light images with semantic segmentation labels. These images are suitable for training and testing infrared and visible light image fusion networks. LLVIP Multimodal Image Dataset: The LLVIP dataset contains 30976 images, or 15488 pairs of infrared and visible light images, focusing on low-light environments. The images are strictly aligned temporally and spatially, suitable for evaluating algorithms such as image fusion and pedestrian detection in low-light vision applications. These datasets demonstrate the applicability of this embodiment under different environments and conditions, from high-quality image fusion to performance evaluation in advanced vision tasks, ensuring the broad applicability and effectiveness of the method.

[0050] Feature extraction is performed using a contrast residual module and a texture enhancement module; specifically, this includes: firstly, the fusion network in this embodiment aims to preserve the infrared image as much as possible. Significant information and visible light images The rich detail information in the image is then used to construct a new fused image. The overall structure diagram of this embodiment is as follows: Figure 3 As shown. This step requires no manual operation. The problem this embodiment aims to solve is how to implement the three stages of feature extraction, feature aggregation, and image reconstruction in an end-to-end manner. Given the lack of real labels in image fusion tasks, this embodiment proposes an unsupervised image fusion network, such as... Figure 4As shown. Furthermore, this embodiment designs a loss function to obtain a more ideal fusion result, including content loss and segmentation loss. Secondly, existing methods do not specifically design feature extractors in the feature extraction stage, ignoring the connections between images of different modalities. This embodiment specifically designs different feature extraction modules to extract and utilize complementary information from the source images based on the different characteristics of infrared and visible light images, and strengthen the correlation between images of different modalities. Three densely connected residual modules first obtain a set of features from the source images. In the infrared and visible light branches, these two types of modules are cascaded together and can be considered as the feature extraction modules of the image fusion network in this embodiment. and , can be represented as: (1) in and These represent infrared image features and visible light image features, respectively. This feature extraction module extracts depth features to improve the performance of the fusion results.

[0051] Using the comparative residual module Infrared input features Perform the transformation to obtain the output features. The contrast residual module of the infrared branch is represented as follows: Figure 5 As shown in (a): (2) Similarly, the texture enhancement module can be obtained. Visible light input characteristics Processing is performed to obtain , represented as: (3) in, Representing the Laplacian operator, this is a special convolution operation in the network of this embodiment, used to extract high-frequency and detail information from the image. The network structure is as follows: Figure 5 As shown in (b). Furthermore, the element-wise summation operation utilizing features... This approach integrates features and effectively combines them with the concept of residual networks. The contrast residual module and texture enhancement module integrate salient and detailed information from the source image, providing a foundation for subsequent operations.

[0052] The analysis and fusion of deep features are combined using a cross-feature attention module; specifically including: First, the output features of the residual module are fed into the attention module for processing to enhance the long-range dependencies of the image. In the attention module, a self-attention mechanism weighs the relationship between each input and other inputs, optimizing the image fusion task that requires global information. Compared to other attention-based fusion methods, the method in this embodiment considers the relationships between different modalities and can handle images of different sizes. Assume the input features of the infrared branch can be represented as... The output is Its branch attention module can be represented as, for example Figure 6 As shown in (a): (4) in, This represents the infrared attention module. Similarly, the input features of the visible light branch can be represented as... The output is Its branch attention module is a specific manifestation of the cross-feature attention module, namely: (5) Representing the visible light attention module, such as Figure 6 As shown in (b).

[0053] Next, this embodiment uses The output features of the two branches are connected along the channel dimension to combine the depth features extracted from different types of images. This process can be represented as: (6) in, This represents the characteristics after two branches are connected.

[0054] In the cross-feature attention module, this embodiment argues that existing fusion methods fail to fully utilize the cross-feature spatial dependencies between source images, as well as long-range details in the process of mixing texture and saliency information. Although existing deep neural networks have successfully extracted and fused features from source images to some extent, they do not adjust each pixel based on the global information of the features, but rather use convolutional kernels to adjust the values ​​of spatial regions together. Therefore, this embodiment extracts the cross-feature spatial dependencies between infrared and visible light features through the cross-feature attention module and finds more advanced textures for infrared features before fusion. This method effectively enables the fusion network of this embodiment to capture the cross-feature spatial dependencies and long-range details of the source images, thereby ensuring that the fusion result contains texture detail information from the visible image and salient information from the infrared image. Figure 6 As shown in (b), this embodiment uses infrared features, visible light features, and visible light features obtained from the feature extractor as the query (Q), key (K), and value (V) in the attention mechanism.

[0055] In this attention-based architecture, the embedding vector size is set to 256, which limits the input size of the processing network. To accommodate this, this embodiment crops and modifies the size of the input image during training to match the network size. Alignment is performed as an integer multiple of the input size. During testing, it's more common to encounter inputs of different sizes. This embodiment first captures the original size of the input image and then pads the image to fit. The result is an integer multiple of the original size. After obtaining the fusion result, this embodiment cropps the result back to its original size, enabling the network of this embodiment to process images of different sizes.

[0056] The image reconstruction module is used to generate a fusion result from the obtained features; specifically, this includes: Image reconstruction module Fusion features Reconstruction processing is performed to obtain the fusion result. The process is as follows: (7) The image reconstruction module contains four A convolutional layer and a The convolutional layers, except for the last one, use the Leaky ReLU activation function. Furthermore, this embodiment uses reflection padding to ensure that the feature map size remains unchanged after each convolution operation, thus better preserving spatial information. To enhance visual perception and improve quantitative metrics, while adapting the fusion network of this embodiment to advanced visual tasks, a loss function is designed to address these two aspects. First, this embodiment constructs a conventional fusion loss function, namely content loss, to help the fusion network generate a fusion result with good complementarity. In addition, to enable the fusion network of this embodiment to be combined with advanced visual tasks, a segmentation loss is designed. This embodiment combines these two loss functions to ensure that the fused image contains as much salient information as possible from the infrared image and texture detail information from the visible image, enhancing the fusion network's orientation towards advanced visual tasks. The purpose of content loss is to integrate complementary information from the source images. Therefore, content loss is designed to better utilize the extracted complementary information, helping the fusion network generate a fusion result with good complementarity. Content loss includes three main parts: intensity loss, structure loss, and detail loss. Detailed formulas are as follows: (8) in, and Used for balance The three components, in this embodiment, =1, It is 10. The constrained fusion results show an intensity distribution similar to the source image. Preserve the structural information of the source image. Texture detail information is preserved in the fusion result. Intensity loss is defined as: (9) in, and These represent the height and width of the image, respectively. express Norm, This represents the maximum pixel value in the two images. This part of the loss function actually employs a maximum selection strategy, in which this embodiment finds a more reasonable intensity distribution for the fused image. However, using only the intensity distribution to constrain the fusion result is insufficient; therefore, this embodiment uses... and , The definition is as follows: (10) SSIM is used to measure the structural similarity between images. and These represent two images respectively. and Represents constant coefficients. and These represent the mean and standard deviation, respectively. express and The covariance between them. Constrain the structural similarity between the source image and the fused image. Besides preserving structural information, detail information is equally important. Defined as: (11) in Represents the Laplace operator, used to extract from... and Extracting high-frequency information along the axis improves the fusion results for better performance in subsequent advanced vision tasks. This represents the absolute value operation. The above three components together constitute the content loss, which, in collaboration with the residual module and attention module, enables the fusion network to perform well in both visual perception and quantitative metrics. This embodiment also designs a segmentation loss to address the issue that existing fusion methods focus on visual effects and quantitative parameter comparisons, neglecting the importance of subsequent tasks. The segmentation loss consists of two parts... and Compared to other similar methods' loss functions, it has the advantage of using multiple segmentation results to guide network training. This method improves the network's adaptability to different scenarios, making it more generalizable. Specifically, The backbone network learns the semantic information of the entire fusion result, thereby optimizing the segmentation result of the entire image. assist Optimize segmentation results for specific regions or objects. The final segmentation result is obtained after predicting the fusion result at a certain stage by constructing a segmentation network. Then, the segmentation loss is backpropagated to the fusion network to positively influence the fusion network in this embodiment. and It reflects the semantic information of the fusion result from different aspects. Main segmentation loss. It captures the global semantic information of the entire fusion result. The formula is: (12) in From the split label The vector obtained after transformation. This represents the predicted segmentation result. On the other hand, there's the additional segmentation loss. It focuses on refining the segmentation of specific regions. It uses deep features to obtain improved segmentation results for complex regions. Its expression is: (13) in, This indicates the segmentation result obtained using deeper feature maps. This helps improve the segmentation accuracy of more challenging regions in the fused image. This can be expressed as: (14) in, The constant representing the balance between the main segmentation loss and the additional segmentation loss is set to 0.1 in this embodiment. In summary, the entire loss function can be formulated as follows: (15) in This is a coefficient that balances content loss and segmentation loss; in this embodiment, it is set to 1. The entire loss function combines image fusion with high-level visual tasks. The method in this embodiment ensures optimal fusion results by emphasizing the visual perception of the source images and their robustness to subsequent high-level visual tasks. This design not only improves the visual quality of the fused images but also enhances the fusion network's ability to perform high-level visual tasks, making it more adaptable and accurate when processing images in complex scenes and under different conditions. Through this method, the network in this embodiment can effectively integrate and utilize the high-quality visual characteristics of the source images to improve the performance of subsequent tasks, such as object detection, scene understanding, and image segmentation, while maintaining these characteristics. This comprehensive strategy, considering both visual perception and task orientation, makes the fusion method of this invention innovative and practical in the field of image fusion.

[0057] In some embodiments, a three-stage training strategy is introduced to improve the performance of advanced vision tasks; specifically, it includes: First stage: Independent training of the image fusion network. In this stage, this embodiment focuses on the independent training of the image fusion network. Unlike conventional methods that only use labeled training sets, this embodiment expands the training set by adding 1000 pairs of image pairs containing day and night scenes to improve the generalization ability of the fusion network; Second stage: Generating fused images using the pre-trained fusion network, which will serve as the base data for the remaining stages; Third stage: Training the segmentation network and backpropagating the segmentation loss. In this stage, the segmentation network is first trained to improve its segmentation ability on the fused images. As training progresses, the performance of the segmentation network gradually improves, thereby more effectively guiding the fusion network to improve the performance of advanced vision tasks. Then, the segmentation network is used to predict the segmentation results, and the segmentation loss is backpropagated to the fusion network to guide its performance improvement. The implementation of the above steps ensures that the image fusion method not only excels in the quality of the fusion results, but also performs well in advanced vision tasks such as object recognition. Figure 7 This diagram illustrates the three-stage training strategy. Through this strategy, the image fusion network can better adapt to different advanced visual tasks, thus demonstrating superior performance in diverse application scenarios.

[0058] To test the performance and efficiency of the method, qualitative and quantitative comparative experiments were conducted on four infrared and visible light datasets, totaling 108 pairs of infrared and visible light images. These four datasets are: MFNet, RoadScene, LLVIP, and TNO.

[0059] The qualitative comparison results of this embodiment are as follows: Figure 8 As shown. To demonstrate the generalization ability of the method in this embodiment and the visual effect of the fused image, this embodiment selects a set of comparative results for display, focusing on comparing salient and detailed regions. Since image fusion mainly integrates complementary information from multiple images into one image, the more meaningful information from the source images is included, the better the fusion result. The comparative results show that GTF, DenseFuse, FusionGAN, and SDNet methods failed to preserve the texture details of visible light images well. TGFuse, U2Fusion, SeAFusion, and DATFuse failed to preserve the salient information of the source images well. MUFusion did not preserve the information of the visible light image well and added noise to the fused image.

[0060] The method of this invention preserves not only the detailed information of the visible light image but also salient information to enhance the infrared target. Furthermore, similar phenomena were observed in the comparison results of other groups. In summary, the fusion method of this invention benefits from the residual module and the attention module, thereby obtaining a high-quality fused image.

[0061] Table 1 shows the quantitative comparison results of this embodiment, comparing it with other mainstream methods on the LLVIP, MFNet, RoadScene, and TNO datasets. The testing method involved randomly selecting 25 pairs of images from each dataset, obtaining the fusion results, and then measuring the quantitative metrics of each method on these datasets. First, it is evident that the method of this embodiment performs best on the MFNet and TNO datasets. Second, the method of this embodiment has the highest VIF metric across all four datasets, indicating that the fusion results of this embodiment are highly consistent with the human visual system. Finally, the method of this embodiment also performs well on metrics such as SCD and MI.

[0062] Table 1

[0063] The above quantitative results are attributed to the contrast residual module, texture enhancement module, cross-feature attention module, and three-stage training strategy of the present invention.

[0064] The fusion result of this embodiment performs as follows in the semantic segmentation task: Figure 9 As shown. To demonstrate the advantages of the method in this embodiment for advanced vision tasks, this embodiment compares the segmentation results with those of other methods.

[0065] As can be seen from the figure, the segmentation model can generate more accurate segmentation results on the fused image using the method of this invention. For example, in the pedestrian walkway portion within the frame, the segmentation result of the method of this invention is significantly better than that of other fusion methods.

[0066] The fusion results in this embodiment perform as follows in the pedestrian detection task: Figure 10 As shown in the figure, since thermal radiation information in infrared images is often found on people, this embodiment uses pedestrian detection as a comparison task for the fusion results. The figure clearly shows that the method in this embodiment outperforms the average detection accuracy in the pedestrian detection task. FusionGAN fails to preserve the detailed texture information of visible light images well, and therefore does not detect pedestrians. GTF, U2Fusion, and MUFusion fail to preserve the brightness information of visible light images, while SDNet and TGFuse are affected by negative information and fail to properly display salient targets.

[0067] This invention preserves texture details without reducing brightness and effectively displays prominent targets.

[0068] refer to Figure 11 In some embodiments, an attention-based image fusion apparatus for advanced visual tasks is also provided, comprising the following modules: an image acquisition module 1 for acquiring two source images to be fused; an infrared residual module 2 for selectively extracting salient information from the infrared image; a texture enhancement module 3 for selectively extracting detail information from the visible light image; an infrared attention module 4 for enhancing the long-range dependencies of the infrared image; a cross-feature attention module 5 for enhancing the relationship between the long-range dependencies of the visible light image and the infrared image; an image reconstruction module 6 for generating a fused image based on depth features; and a semantic segmentation module 7 for enhancing the performance of the fusion result on advanced visual tasks.

[0069] This invention overcomes the limitations of existing deep learning methods in image fusion by employing a unique attention mechanism and an innovative three-stage training strategy, establishing an attention-based image fusion method for advanced visual tasks. The fundamental principle is to extract features from both infrared and visible light images and generate a fused image with stronger representational capabilities. First, the invention acquires the infrared and visible light images to be fused and performs necessary preprocessing to ensure the effectiveness and accuracy of the fusion. Then, it extracts features from both modalities using a contrastive residual module and a texture enhancement module. The contrastive residual module extracts salient features from the infrared image, while the texture enhancement module focuses on the detailed information of the visible light image. Next, the system utilizes a cross-feature attention module to analyze and fuse the extracted depth features. This module strengthens the complementarity and correlation between different image features through an attention mechanism, thereby selecting the most suitable fusion features. This step effectively combines the thermal imaging advantages of infrared images with the rich detail of visible light images. Then, through an image reconstruction module, the fusion network transforms the concatenated features into the fusion result. Finally, the three-stage training strategy employed in this invention further optimizes the performance of the fusion result on advanced visual tasks. First, the fusion network is pre-trained on a large-scale dataset to improve its generalization ability. Second, the pre-trained network is used to generate fused images, which serve as the basis for subsequent training. Finally, feedback training through a semantic segmentation network further enhances the performance of the fused images in advanced vision tasks.

[0070] In some embodiments, the image fusion method described above can also be implemented in the following ways.

[0071] Since this embodiment is based on deep learning, it is divided into training and testing phases. Training involves learning and optimizing the model's parameters using a large dataset, enabling the model to learn the features and patterns in the data. Testing involves using the trained model to predict new and unseen data, which is equivalent to applying the model to actual production.

[0072] The algorithm in this embodiment is developed using the PyTorch deep learning framework. After the training phase is completed, a model file with the suffix "fusion network.pt" can be obtained. During the testing phase, by loading this "fusion network.pt" model file and then inputting infrared and visible light images, the fusion result can be obtained.

[0073] Detailed training steps: 1. Data Preprocessing. Infrared and visible light image data are read by an algorithm, the image size is adjusted, and then converted into a tensor format suitable for deep learning.

[0074] 2. The fusion network – the contrast residual module and the detail enhancement module – feed the infrared and visible light image tensors into the infrared branch respectively. Figure 4 Upper branch) and visible light branch ( Figure 4 (Lower branch). The infrared tensor first passes through a convolutional layer with a 1×1 kernel, then through a Leaky ReLU activation function; next, it is processed by three densely connected contrast residual modules; finally, the infrared features are output. The visible light tensor first passes through a convolutional layer with a 1×1 kernel, then through a Leaky ReLU activation function; next, it is processed by three densely connected detail enhancement modules; finally, the visible light features are output.

[0075] 3. Fusion Network - Infrared Attention Module and Visible Light Attention Module: The infrared and visible light features output from the previous step are input into the infrared attention module and visible light attention module respectively for processing, resulting in processed infrared and visible light features. These two feature sets are then concatenated, ready to be sent to the next module for processing.

[0076] 4. Fusion Network - Image Reconstruction Module: The features obtained in the previous step are fed into the reconstruction module to obtain the fusion result.

[0077] 5. Fusion Network - Loss Calculation: The fused result and the input images (input infrared and visible light images) are subjected to a content loss function to train the entire model. After the model is trained, a "fusion network.pt" file is obtained.

[0078] 6. Generate fusion results using the fusion network, and update the parameters of the model "fusion network.pt" using the segmentation network.

[0079] The fusion network loads the "fusion network.pt" file obtained in the previous step, then uses this weight to generate the fusion result, and finally uses this fusion result for training. Figure 3 The segmentation network in the middle, and then the segmentation network propagates the segmentation loss to the training process. Figure 3 The fusion network updates its weights based on this segmentation loss, enabling the generated fusion result to perform well in high-level vision tasks. After training, the updated "fusion network.pt" file is saved for practical applications.

[0080] This embodiment uses the well-known DeepLabV3+ model for segmentation, which is specifically designed for image segmentation tasks. It combines dilated convolution, encoder-decoder architecture, and ASPP technology.

[0081] Detailed testing steps: Load the trained model file "fusion network.pt", input infrared and visible light images, and generate fused images for advanced vision tasks.

[0082] The embodiments of the present invention have been described above with reference to the accompanying drawings. However, the present invention is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of the present invention without departing from the spirit and scope of the claims. All of these forms are within the protection scope of the present invention.

Claims

1. A training method for an image fusion model, characterized in that, The training method includes the following steps: S1: Acquire infrared and visible light images, expand the training set by adding 1000 pairs of image pairs containing day and night scenes, align the infrared and visible light images in time and space and convert them into tensor format to obtain infrared tensors and visible light tensors. S2: The infrared tensor is processed by the contrast residual module to extract features and obtain infrared output features; the visible light tensor is processed by the texture enhancement module to extract features and obtain visible light output features. S3: Use the infrared attention module to extract features from the infrared output, and obtain the output of the infrared attention module. ,in, For the input of the infrared attention module, For the infrared attention module, `Trans()` is a transformer operation used for feature transformation. The visible light attention module is then used to extract features from the visible light output, yielding the output of the visible light attention module. ,in, This is the input for the visible light attention module. For the visible light attention module, the outputs of the infrared attention module and the visible light attention module are concatenated to obtain the fused features. ,in, This is a connection operation used to combine depth features extracted from different types of images; S4: Use the image reconstruction module to perform image reconstruction processing on the fusion features to obtain a fused image; construct a content loss function to calculate the content loss of the image fusion model; S5: Obtain a preset value for content loss. If the content loss function is not less than the preset value for content loss, return to step S1 and continue training the image fusion model. If the content loss function is less than the preset value for content loss, stop training and obtain the pre-trained image fusion model. S6: Acquire infrared and visible light images, and generate a fused image using the pre-trained image fusion model; train a semantic segmentation model using the fused image to obtain a pre-trained semantic segmentation model; obtain a segmentation result and calculate the segmentation loss based on the pre-trained semantic segmentation model and the fused image; when the segmentation loss is not less than a preset value, feed the segmentation loss back to the pre-trained image fusion model to further train the pre-trained image fusion model; when the segmentation loss is less than the preset value, stop training and obtain a trained image fusion model.

2. The training method for the image fusion model according to claim 1, characterized in that, Step S2 includes the following steps: The infrared tensor is processed by a contrast residual module to extract features, resulting in infrared output features, as shown in the formula: ,in Infrared output characteristics To compare the residual modules, Infrared input characteristics, It is a 3×3 convolutional layer. It is a 1×1 convolutional layer. This is an element-wise summation operation; The visible light tensor is processed by a texture enhancement module to extract features, resulting in visible light output features, as shown in the formula: ,in Visible light output characteristics, For texture enhancement module, Δ represents the visible light input feature, and Δ is the Laplacian operator.

3. The training method for the image fusion model according to claim 2, characterized in that, The contrast residual module includes a first 3×3 convolutional layer, a second 3×3 convolutional layer, an activation function, a 1×1 convolutional layer, and a summer. The first 3×3 convolutional layer, the activation function, and the second 3×3 convolutional layer are connected in series. The input of the first 3×3 convolutional layer is connected to the input of the 1×1 convolutional layer, the output of the first 3×3 convolutional layer is connected to the first input of the summer, and the output of the 1×1 convolutional layer is connected to the second input of the summer. The first and second 3×3 convolutional layers are used to extract 3×3 image features, the activation function is used to activate the first 3×3 convolutional layer, the 1×1 convolutional layer is used to extract 1×1 image features, and the summer is used for element-wise summation.

4. The training method for the image fusion model according to claim 2, characterized in that, The texture enhancement module includes a first 3×3 convolutional layer, a second 3×3 convolutional layer, an activation function, a Laplacian operator, a 1×1 convolutional layer, and a summer. The first 3×3 convolutional layer, the activation function, and the second 3×3 convolutional layer are connected in series. The input of the first 3×3 convolutional layer is connected to the input of the Laplacian operator, the output of the Laplacian operator is connected to the input of the 1×1 convolutional layer, the output of the first 3×3 convolutional layer is connected to the first input of the summer, and the output of the 1×1 convolutional layer is connected to the second input of the summer. The Laplacian operator is used to sharpen the image, the first and second 3×3 convolutional layers are used to extract 3×3 image features, the activation function is used to activate the first 3×3 convolutional layer, the 1×1 convolutional layer is used to extract 1×1 image features, and the summer is used for element-wise summation.

5. The training method for the image fusion model according to claim 1, characterized in that, Step S4 includes the following steps: The image reconstruction module is used to perform image reconstruction processing on the fusion features to obtain the fused image, as shown in the formula: ,in, This is an image reconstruction module used to reconstruct fused features. As a feature of fusion, To merge images; Construct a content loss function, as shown in the formula: in, For content loss function, For strength loss, For structural loss, For the loss of details, The first adjustment constant, This is the second adjustment constant, used to adjust the weights of strength loss, structural loss, and detail loss. The height of the image. The width of the image. I ir Infrared image, I vi Visible light image, It is a 1-norm. The maximum value of the pixels in the two images. and Each represents one of two images. and Represents constant coefficients. and These represent the mean and standard deviation, respectively. express and Covariance between For the Laplace operator, used to extract from and High-frequency information is extracted along the axial direction. This indicates the absolute value operation.

6. The training method for the image fusion model according to claim 5, characterized in that, Step S6 includes the following steps: S61: Acquire infrared and visible light images, and generate a fused image using the pre-trained image fusion model; S62 acquires a semantic segmentation model, and uses the semantic segmentation model to segment the fused image to obtain a segmented image; S63: Calculate the segmentation loss based on the fused image and the segmented image, as shown in the formula: in, To divide the loss, Main segmentation loss, For additional segmentation loss, To balance the constant between the main segmentation loss and the additional segmentation loss, This is the vector obtained from the segmentation labels. The predicted segmentation result is given, where H is the image height, W is the image width, and C is the number of image channels. The segmentation result obtained using deeper feature maps; S64: Obtain a preset value for segmentation loss. If the segmentation loss is not less than the preset value, return to step S61 to further train the pre-trained image fusion model. If the segmentation loss is less than the preset value, stop training and obtain the trained image fusion model.

7. A training device for an image fusion model, characterized in that, The device includes the following modules: The image preprocessing module is used to: acquire infrared images and visible light images; expand the training set by adding 1000 pairs of image pairs containing day and night scenes; align the infrared images and visible light images in time and space and convert them into tensor format to obtain infrared tensors and visible light tensors. The feature extraction module is used to: perform feature extraction processing on the infrared tensor using the contrast residual module to obtain infrared output features; The visible light tensor is processed by a texture enhancement module to extract features and obtain visible light output features. The feature fusion module is used to: extract features from the infrared output features using the infrared attention module, and obtain the output of the infrared attention module. ,in, For the input of the infrared attention module, For the infrared attention module, `Trans()` is a transformer operation used for feature transformation. The visible light attention module is then used to extract features from the visible light output, yielding the output of the visible light attention module. ,in, This is the input for the visible light attention module. For the visible light attention module, the outputs of the infrared attention module and the visible light attention module are concatenated to obtain the fused features. ,in, This is a connection operation used to combine depth features extracted from different types of images; The image reconstruction and content loss module is used to: perform image reconstruction processing on the fusion features using the image reconstruction module to obtain a fused image; and construct a content loss function to calculate the content loss of the image fusion model. The content loss and iteration module is used to: obtain a preset value for content loss; when the content loss function is not less than the preset value for content loss, return to the image preprocessing module to continue training the image fusion model; when the content loss function is less than the preset value for content loss, stop training and obtain a pre-trained image fusion model. The segmentation testing and retraining module is used to: acquire infrared and visible light images; generate a fused image using the pre-trained image fusion model; train a semantic segmentation model using the fused image to obtain a pre-trained semantic segmentation model; obtain a segmentation result and calculate a segmentation loss based on the pre-trained semantic segmentation model and the fused image; when the segmentation loss is not less than a preset value, feed the segmentation loss back to the pre-trained image fusion model to further train the pre-trained image fusion model; when the segmentation loss is less than the preset value, stop training and obtain a trained image fusion model.

8. An image fusion method, characterized in that, Includes the following steps: Acquire the infrared and visible light images to be fused; Obtain an image fusion model, and train the image fusion model using the training method of any one of claims 1-6 to obtain a trained image fusion model; The trained image fusion model is used to perform image fusion processing on the infrared image and the visible light image to be fused, resulting in a fused image.

9. An image fusion apparatus, characterized in that, The device includes the following modules: The image acquisition module is used to acquire the infrared image and the visible light image to be fused. The model training module is used to: acquire an image fusion model, train the image fusion model using the training method of the image fusion model as described in any one of claims 1-6, and obtain a trained image fusion model; The fusion module is used to: perform image fusion processing on the infrared image and the visible light image to be fused using the trained image fusion model to obtain a fused image.

Images