A Method and System for Infrared and Visible Image Fusion Based on Semantic Segmentation and Multi-Scale Feature Interaction

Abstract

This invention discloses an infrared and visible light image fusion method and system based on semantic segmentation and multi-scale feature interaction, relating to the fields of artificial intelligence and image processing technology. The method includes the following steps: feature extraction from the original image to obtain multi-scale feature maps; combining a feature fusion block with an attention mechanism to fuse the multi-scale feature maps; introducing a multi-scale feature interaction module, a Transformer block, and a feature reorganization module for interactive processing and integration to obtain an optimized fused image; introducing a semantic segmentation module to obtain a semantically fused image; and introducing a joint loss function to obtain the processed image. This invention aims to capture and retain feature information at different scales in infrared and visible light image fusion, promote the interactive utilization of information at different scales, increase semantic information in the fused image, and effectively preserve key feature information and global dependencies in the image.

Description

Technical Field

[0001] This invention relates to the fields of artificial intelligence and image processing technology, and in particular to an infrared and visible light image fusion method and system based on semantic segmentation and multi-scale feature interaction. Background Technology

[0002] Image fusion is the process of integrating information from multiple image sensors into a single fused image, aiming to improve image quality and maintain the integrity of important features. In the field of image fusion, infrared and visible light image fusion is one of the most widely studied key areas. Infrared images perform well in dark environments and in adverse weather conditions, and can clearly show heat source targets, but they have low resolution, high noise, and lack texture details. Visible light images have high resolution and rich texture details under good lighting conditions, but perform poorly in dark or adverse weather conditions.

[0003] Therefore, this paper proposes a deep learning-based method for the fusion of infrared and visible light images. The infrared and visible light image fusion technology integrates the complementary information of the two spectra to generate a fused image that combines the thermal radiation information of the infrared image and the texture details of the visible light image, thereby achieving a more complete presentation of image information.

[0004] However, in actual research, it has been found that deep learning fusion methods tend to weaken or lose important features at different scales, resulting in the inability to effectively integrate key information and thus affecting the quality of the fused image. Secondly, existing methods pay little attention to the effective interaction between multi-scale information, making it difficult to achieve effective information coordination and thus failing to fully utilize feature information at different scales. Finally, the global contextual relationships in the image and the semantic relationships between different objects or regions are also important factors that need to be considered in the research.

[0005] To address the aforementioned issues, this application proposes a solution for fusing infrared and visible light images based on semantic segmentation and multi-scale feature interaction. Summary of the Invention

[0006] Purpose of the invention: To propose an infrared and visible light image fusion method and system based on semantic segmentation and multi-scale feature interaction, in order to solve the above-mentioned problems existing in the prior art.

[0007] Technical solution: An infrared and visible light image fusion method based on semantic segmentation and multi-scale feature interaction, comprising the following steps:

[0008] S1. Obtain the infrared image and the visible light image of the original image respectively, and perform feature extraction on them respectively to obtain the infrared image multi-scale feature map and the corresponding visible light multi-scale feature map.

[0009] S2. By combining pre-constructed feature fusion blocks with an attention mechanism, a deep feature fusion module is obtained, which fuses the multi-scale feature maps of infrared images and the corresponding multi-scale feature maps of visible light to obtain deep fused features.

[0010] S3. Process deep fusion features through a pre-built multi-scale feature interaction module to obtain information interaction fusion features;

[0011] S4. By pre-constructing and optimizing the Transformer block, the information interaction fusion features are processed to obtain the global context information of the information interaction fusion features;

[0012] S5. The feature recombination module recombines and optimizes the global context information and the corresponding information interaction fusion features obtained after step 4 to obtain an optimized fusion image.

[0013] S6. Using pre-constructed semantic segmentation, extract and retain semantic information from the optimized fusion image obtained in step S5, and output the semantic fusion image.

[0014] S7. Using a pre-constructed joint loss function, input the semantic fusion image obtained in step S6 and the optimized fusion image obtained in step S5, and output the processed image.

[0015] S8. Set the number of iterations for the processing in steps 1 to 7, and output the image after iterative processing as a fused infrared and visible light image.

[0016] Furthermore, step S1 further comprises:

[0017] Two parallel scale feature extraction modules are used to process the infrared image and the corresponding visible light image, respectively.

[0018] The scale feature extraction module uses at least 5 convolutional blocks, and the number of channels in each convolutional block is set to 32. The first, second, and fifth convolutional blocks maintain the original resolution. The second convolutional block continues to maintain the original resolution, while the third and fourth convolutional blocks are downsampled by a factor of 2.

[0019] Furthermore, step S2 further comprises:

[0020] S21. Set the number of feature fusion modules, connect them in a preset order, and set the corresponding convolution kernel size;

[0021] S23. Input the infrared image multi-scale feature map and the corresponding visible light multi-scale feature map, perform at least 3 iterations, and output the corresponding output tensor pairs.

[0022] S24. Add the tensor pairs obtained in step S23, output the feature representation, and obtain the deep fusion feature.

[0023] Furthermore, the expressions for the deep feature fusion module and the feature fusion module are as follows:

[0024]

[0025] Among them, F S and F C Let F, S, and C represent spatial features, spatial features, and channel features, respectively. FF represents feature fusion, C represents concatenation, SA(·) represents spatial attention, CA(·) represents channel attention, and Conv1(·) represents convolution with a kernel size of 1. The expressions for SA(·) and CA(·) are as follows:

[0026] SA(x)=Sig(Conv3(C{Max(x),Avg(x)}))×x

[0027] CA(y)=Sig(FC(AP(y))+FC(MP(y))×y

[0028] Where x represents the input of spatial attention, y represents the input of channel attention, Max(·) represents the maximum value operation, Avg(·) represents the average value operation, Conv3(·) represents the convolution operation with a kernel size of 3, Sig(·) represents the sigmoid activation function operation, AP(·) represents the average pooling operation, MP(·) represents the max pooling operation, and FC(·) represents the fully connected operation.

[0029] and These represent the multi-scale features of infrared and visible light images obtained in the five stages of the multi-scale feature extraction block, respectively. and It is the feature obtained after j iterations of the multi-scale feature acquired in the i-th stage. It is the final output of the deep feature fusion module.

[0030] Furthermore, step S3 further comprises:

[0031] Three multi-scale feature interaction modules were used to process the deep fusion features. Perform preliminary processing and interactive results. The fourth multi-scale feature interaction fusion module is used to process the interacted features. Further processing yields interactive features. The corresponding function expression is:

[0032]

[0033] Wherein, MFI represents multi-scale feature interaction operation, and its functional expression is:

[0034] MFI{x,y,z}=Conv1(C({DC(Down(x)),DC(y),DC(Up(z))}))

[0035] Where Down(·) represents a downsampling operation, Up(·) represents an upsampling operation, and DC(·) represents a dense linking operation.

[0036] Furthermore, step S4 further involves optimizing the feature processing method based on the Transformer block and introducing a convolution-based local feature extraction channel to achieve more comprehensive feature extraction. The function expression of the optimized Transformer block is as follows:

[0037]

[0038] Where T(·) represents the Transformer operation, F T This indicates the characteristics that have passed through the Transformer block.

[0039] Furthermore, the further content of steps S6 to S7 is as follows:

[0040] A semantic segmentation framework is introduced. Within this framework, the segmentation loss function and the fusion loss function are combined to obtain a joint loss function for training. The expression for the joint loss function is as follows:

[0041] L J =L F +αL S

[0042] Among them, L J L F and L S These represent the joint loss function, fusion loss function, and segmentation loss function, respectively, with α representing a hyperparameter used to control L. F and L S The ratio between them; L F It combines pixel intensity loss L P Loss of detail L D The design and its corresponding function expression are as follows:

[0043] L F =L P +βL D

[0044] Where β represents a hyperparameter used to control L P and LD The ratio between them; L P and L D These are used to measure the difference between the generated image and the real image at the pixel level, and to measure the preservation of detail and texture information in the image, respectively. Their function expressions are as follows:

[0045]

[0046] Where H represents the height of the image, W represents the width of the image, and max(·) represents the operation of calculating the maximum value of the maximum value of the elements;

[0047] L S It is used to add semantic information to the fused image, and its function body expression is as follows:

[0048] L S =L Main +λL Aux

[0049] Where λ is a hyperparameter, L Main L represents the principal segmentation loss. Aux The auxiliary segmentation loss is expressed as follows:

[0050]

[0051] Where C represents the number of channels in the image, I S Indicates main output, I Sa Indicates auxiliary output, I So This represents a one-hot encoded vector.

[0052] An infrared and visible light image fusion system based on semantic segmentation and multi-scale feature interaction includes: at least one processor; and a memory communicatively connected to at least one of the processors; wherein the memory stores instructions executable by the processor to implement the infrared and visible light image fusion method based on semantic segmentation and multi-scale feature interaction as described above.

[0053] Compared with the prior art, this application has the following beneficial effects:

[0054] 1. A deep feature fusion module combining spatial and channel attention mechanisms is proposed, which can expand the channel dimension of multi-scale features and combine feature maps multiple times in the spatial dimension, thereby achieving adaptive fusion of important information.

[0055] 2. A multi-scale feature interaction module is proposed to capture feature information at different scales and realize information interaction between features at different scales. This interaction mechanism can more fully express features and enhance the details and information integrity of the fused image.

[0056] 3. By combining the improved Transformer block and semantic segmentation network, and leveraging its powerful global context information capture capabilities, the semantic understanding and contextual information expression of fused images are enhanced.

[0057] 4. Compared with existing algorithms, this method performs well on multiple evaluation metrics, effectively captures significant targets, and presents clear texture details. The ablation experiment results further verify the effectiveness of the proposed module. Attached Figure Description

[0058] Figure 1 This is a schematic diagram of the overall network framework structure in one embodiment of this application;

[0059] Figure 2 This is a schematic diagram of the deep feature fusion module in one embodiment of this application;

[0060] Figure 3 This is a schematic diagram of a multi-scale feature interaction module in one embodiment of this application;

[0061] Figure 4 This is a schematic diagram of a feature recombination module in one embodiment of this application;

[0062] Figure 5 This is the fusion result of one embodiment of this application and six typical image methods on the MSRS dataset;

[0063] Figure 6 This is the fusion result of one embodiment of this application and six typical image methods on the TNO dataset. Detailed Implementation

[0064] The invention will now be further described with reference to the accompanying drawings.

[0065] Based on the problems mentioned in the background technology, this application proposes a deep feature fusion module in infrared and visible light image fusion schemes, and designs a novel attention combination mechanism to extract key information from multi-scale feature maps. Building upon this, a multi-scale feature interaction module is also proposed, integrating complementary information and details from multiple different scales to create a multi-level feature map, enabling interactive utilization between multi-scale features, thereby more fully expressing the features.

[0066] This application is based on a semantic segmentation-driven network, combined with an improved Transformer block, to enhance the semantic and contextual information of the fused image, further improving the semantic understanding of the fused image. It aims to capture and preserve feature information at different scales, promote the interactive utilization of information at different scales, increase semantic information in the fused image, and effectively preserve key feature information and global dependencies in the image.

[0067] The technical solution of this invention is as follows:

[0068] Step 1: Infrared and visible light images are first processed through multiple convolutional blocks for feature extraction, generating corresponding multi-scale feature maps;

[0069] Step 2: The extracted multi-scale feature maps are fed into the designed deep feature fusion module. This module combines channel and spatial information, enabling adaptive feature fusion.

[0070] Step 3: The fused features are processed by the multi-scale feature interaction module, which integrates complementary information and details from different scales to achieve effective interaction between multi-scale features and realize a more comprehensive expression of features.

[0071] Step 4: The interactive features enter the Transformer block to obtain global context information.

[0072] Step 5: The features are then processed by the feature recombination module to generate the fused image;

[0073] Step 6: The fused images undergo further processing by a semantic segmentation network to extract semantic information, thereby more effectively promoting the fusion process of the fusion network and preserving and enhancing high-level semantic features.

[0074] Step 7: Iterate according to steps 1 to 6 until stopping, and output the final fused image.

[0075] The training of the fusion network and segmentation network involves iterative training, first training the fusion network, then the segmentation network, and so on. Fusion is performed only on infrared and visible light images. The purpose of iteration is to add semantic information to the fused image, so iteration is necessary. The termination condition is manually set; here, the maximum number of iterations is set to 5. In this application, the iterative training of the fusion network and segmentation network is not described in detail. Existing datasets can be used to introduce neural networks for effective training, and the trained network output can then be used.

[0076] This involves designing a multi-scale feature interaction module to capture feature information at different scales and enable information interaction between features at each scale, thereby more fully expressing the features. The multi-scale feature interaction module and the deep feature fusion module are then integrated into the semantic segmentation network to increase the semantic information in the fused image and preserve the global dependencies in the image.

[0077] The specific details of each of the above steps are as follows:

[0078] The overall network structure is as follows Figure 1As shown, it mainly consists of four parts: Multi-Scale Feature Extraction (MSFE), Deep Feature Fusion (DFF), Multi-Scale Feature Reorganization (MSFR), and Semantic Segmentation (SS).

[0079] In the image fusion network, infrared and visible light images are first input into the multi-scale feature extraction block to obtain the corresponding multi-scale features; then, the deep feature fusion module performs the fusion.

[0080] During the fusion process, the deep feature fusion module focuses on key information in features at different scales, highlighting important features to obtain more effective feature representations.

[0081] Ultimately, these fused features are processed using multi-scale feature interactions.

[0082] In this application, the multi-scale feature interaction consists of a multi-scale feature interaction module, a Transformer block, and a feature reorganization module, which work together to generate the final fused image. Specifically, the multi-scale feature interaction module is used for cross-scale interactive utilization of multi-scale features, the Transformer block is used to capture long-distance dependencies, and the feature reorganization module is responsible for further optimizing and integrating features.

[0083] like Figure 1 As shown, the final semantic segmentation is mainly aimed at enhancing the semantic information of the fused image by calculating the semantic information in the fused image during the iterative training process of the fusion network and the semantic segmentation network.

[0084] The multi-scale feature extraction module in step 1 mainly uses a convolution-based approach to extract features from the source image. For example... Figure 1 As shown, multi-scale feature extraction consists of multiple multi-scale feature extraction blocks, and the size of the feature maps is also displayed in the figure. Each multi-scale feature extraction block employs two parallel branches, processing the infrared and visible light images in five stages to extract multi-scale features. Each branch contains five convolutional blocks. The first convolutional block maintains the original resolution and increases the number of channels to 32, while subsequent convolutional blocks maintain the same number of channels. The second convolutional block continues to maintain the original resolution, while the third and fourth convolutional blocks perform a 2x downsampling, and the last convolutional block maintains the same resolution.

[0085] Compared to traditional feature fusion, this application designs a deep feature fusion module to enhance the fusion capability of key features in images. This module is based on an attention mechanism, concatenating channel information and adding channel attention, while simultaneously adding spatial information and adding spatial attention. This processing method combines channel and spatial information, enabling adaptive feature fusion.

[0086] like Figure 2 As shown, the deep feature fusion module mainly consists of three sequentially connected feature fusion modules. Here, 3×3Conv represents a convolution with a kernel size of 3, and 1×1Conv represents a convolution with a kernel size of 1.

[0087] First, the feature fusion module performs three iterations on the input infrared and visible light image feature tensor pairs, generating corresponding output tensor pairs. Finally, these output tensor pairs are summed to obtain the final feature representation.

[0088] set up and These represent the multi-scale features of infrared and visible light images obtained in the five stages of the multi-scale feature extraction block, respectively. and It is the feature obtained after j iterations of the multi-scale feature acquired in the i-th stage. This is the final output of the deep feature fusion module. The specific formulas for the deep feature fusion module and the feature fusion module are as follows:

[0089]

[0090] Where FF represents the feature fusion operation, its expression is:

[0091]

[0092] Where C represents the concatenation operation, SA(·) represents spatial attention, CA(·) represents channel attention, and Conv1(·) represents a convolution operation with a kernel size of 1. The specific expressions for SA(·) and CA(·) are as follows:

[0093] SA(x)=Sig(Conv3(C{Max(x),Avg(x)}))×x

[0094] CA(y)=Sig(FC(AP(y))+FC(MP(y))×y

[0095] In this context, Max(·) represents the maximum value operation, Avg(·) represents the average value operation, Conv3(·) represents the convolution operation with a kernel size of 3, Sig(·) represents the sigmoid activation function operation, AP(·) represents the average pooling operation, MP(·) represents the max pooling operation, and FC(·) represents the fully connected operation.

[0096] To effectively combine detailed and global information and comprehensively utilize multi-scale features, this application designs a multi-scale feature interaction module. This module mainly consists of three parallel branches, each employing dense connections to further preserve and reuse feature information. In this way, the module can integrate complementary information and details from different scales, thereby achieving effective interaction between multi-scale features and ultimately realizing a more comprehensive feature representation. Figure 1 As shown, the MSFR section includes four multi-scale feature interaction fusion modules. The first three modules perform preliminary processing on the deep fusion features, and the fourth module further processes the features after the preliminary interaction. The specific structure of the multi-scale feature interaction fusion module is as follows: Figure 3 As shown, the first three multi-scale feature interaction modules first perform deep fusion features. After initial processing, the fourth multi-scale feature interaction fusion module performs initial processing on the interacted features. Further processing yields interactive features. Where BN represents batch normalization, and ReLU represents the modified linear unit activation function. The specific operation is shown in the following formula:

[0097]

[0098] Wherein, MFI represents multi-scale feature interaction operation, and its specific expression is:

[0099] MFI{x,y,z}=Conv1(C({DC(Down(x)),DC(y),DC(Up(z))}))

[0100] Where Down(·) represents a downsampling operation, Up(·) represents an upsampling operation, and DC(·) represents a dense linking operation.

[0101] In this application, the improved Transformer optimizes the feature processing method based on the traditional Transformer and introduces a convolution-based local feature extraction channel to achieve more comprehensive feature extraction. This design effectively overcomes the redundancy problem in the self-attention mechanism computation of the traditional Transformer, thus enabling the model to capture global information in the image more easily while maintaining performance. The Transformer block is used to capture the global context information of the fused image, and its specific operation is as follows:

[0102]

[0103] Where T(·) represents the Transformer operation, F T This represents the features processed by the Transformer block. Finally, the feature reorganization module is responsible for generating the final fused image, the specific structure of which is as follows: Figure 4 As shown, the numbers 32, 24, 16, 8, and 1 represent the number of feature channels, Tanh represents the hyperbolic tangent activation function, and I... F This represents the output fused image.

[0104] To better utilize semantic information to facilitate the image fusion process, this application proposes combining low-level and high-level visual tasks. For example... Figure 1 As shown, a semantic segmentation framework is introduced, using a segmentation loss function to compute the semantic information of the fused image to guide the fusion framework. Here, CB represents a convolutional block, BA represents bilateral attention, OM represents the output module, and GT represents the ground truth label. The segmentation loss function and the fusion loss are combined into a joint loss function to optimize the fusion result. Under the constraints of the joint loss function, the fused image not only retains the content of the source image but also enhances semantic consistency, thereby effectively preserving the semantic information of the source image and further improving the quality of the fused image.

[0105] To improve the quality and semantic richness of the fused images, this application employs a joint loss function, consisting of a fusion loss function and a segmentation loss function, to train the network. The expression for the joint loss function is as follows:

[0106] L J =L F +αL S

[0107] Where L J L F and L S These represent the joint loss function, fusion loss function, and segmentation loss function, respectively, with α representing a hyperparameter used to control L. F and L S The ratio between them. L F It combines pixel intensity loss L P Loss of detail L D The designs, and their respective formulas, are as follows:

[0108] L F =L P +βL D

[0109] Where β represents a hyperparameter used to control L P and L DThe ratio between them. L P and L D These are used to measure the difference between the generated image and the real image at the pixel level, and to measure the preservation of detail and texture information in the image, respectively. Their specific expressions are as follows:

[0110]

[0111] Where H represents the height of the image, W represents the width of the image, and max(·) represents the operation of calculating the maximum value of the maximum value of the elements.

[0112] Among them, L S The specific expression used to add semantic information to the fused image is as follows:

[0113] L S =L Main +λL Aux

[0114] Where λ is a hyperparameter, whose value is gradually increased during training according to the training strategy to enhance the semantic information in the fused image, L Main L represents the principal segmentation loss. Aux The auxiliary segmentation loss is expressed as follows:

[0115]

[0116] Where C represents the number of channels in the image, I S Indicates main output, I Sa Indicates auxiliary output, I So This represents the one-hot encoded vector. In the above function, h, w, and c also correspond to the image height, image width, and number of image channels, respectively. These are standard function representations and will not be elaborated further.

[0117] To better illustrate the solution and practical effects of this application, specific embodiments are provided to explain the implementation of the invention. Both training and testing of this invention are conducted on public datasets to ensure the reliability and fairness of the experimental results. Specifically, two datasets were used: the MSRS dataset and the TNO dataset. The MSRS dataset is specifically designed for spectral road scenes, containing 1444 pairs of infrared and visible light images, covering various types of target objects such as cars, pedestrians, and bicycles. This dataset has been pre-divided into training and testing sets, with the training set containing 1083 image pairs and the testing set containing 361 image pairs. To verify the model's generalization ability, experiments were conducted on the TNO dataset. A total of 226 image pairs were used for testing in the TNO dataset. This dataset covers a wide range of scene types and can comprehensively reflect the model's performance in processing various complex scene images.

[0118] The training process is divided into two iterative optimization phases. In the first phase, the Adam optimizer is used to train the fusion framework with an initial learning rate of 0.001, a batch size of 2, and 60 complete iterations to update the framework parameters.

[0119] In the second stage, this invention continues to use the Adam optimizer to train the semantic segmentation network, with the initial learning rate adjusted to 0.01, the batch size set to 16, and the maximum number of iterations set to 8000. The experiments were conducted using the PyTorch framework, with a computing environment configured with an NVIDIA RTX A4000 graphics card and an Intel Core i7-13700K CPU. The parameters of the fusion graph on the MSRS dataset are shown in Table 1.

[0120] Table 1. Mean values ​​of various indicators for different methods on the MSRS dataset.

[0121]

[0122]

[0123] Quantitative analysis was performed on 361 images on the MSRS test set. Table 1 shows the average values ​​of various evaluation metrics, with the best results highlighted in bold. According to the data in Table 1, the proposed method performs best in EN, SF, SD, MI, VIF, and QAB / F metrics. This indicates that the algorithm has good performance in preserving image details, contrast, and information content. EN, MI, and QAB / F reflect the information content and similarity of the fused image, while SF and SD reflect the sharpness and contrast of the fused image. VIF is a comprehensive evaluation metric; data shows that the method proposed in this invention can generate fused images that are highly similar to the original image in terms of information content.

[0124] To further illustrate the fusion performance of this method, this embodiment conducts a generalization experiment on 226 pairs of images in the TNO dataset. The parameters of the fused image on the TNO dataset are shown in Table 2.

[0125] Table 2. Mean values ​​of various indicators for different methods on the TNO dataset.

[0126] Network EN↑ SF↑ SD↑ MI↑ VIF↑ Qabf↑ U2Fusion 6.9013 0.0487 9.7672 2.6686 0.6623 0.4667 RFN-Nest 6.9957 0.0308 10.0824 2.7469 0.7248 0.2982 FusionGAN 7.0729 0.0343 9.9864 2.7557 0.5785 0.2579 GANMcC 7.2438 0.0365 10.0427 2.7626 0.6804 0.3355 TUFusion 6.8852 0.0266 9.6775 2.6942 0.6699 0.2728 YDTR 6.9148 0.0431 10.2568 2.9491 0.7147 0.4297 Ours 7.3113 0.0742 10.8140 2.9032 0.7347 0.4394

[0127] As shown in Table 2, this embodiment achieved the best performance in EN, SF, VIF, and QAB / F metrics, indicating that its fused image outperforms other methods in terms of information content, sharpness, contrast, and similarity. Although not optimal in SD and MI metrics, it is still close to the best results. This indicates a certain advantage in maintaining the overall structure and information content of the original image, but there is still room for improvement. The advantages of this invention in performance metrics EN, SF, VIF, and QAB / F are more obvious, indicating that the method has a strong advantage in overall fusion quality. Overall, this invention performs excellently in fusion performance, verifying its good fusion capability and generalization ability.

[0128] Although this application has been described in conjunction with specific features and embodiments, it is obvious that various modifications and combinations can be made thereto without departing from the spirit and scope of this application. Accordingly, this specification and drawings are merely illustrative descriptions of the application as defined by the appended claims, and are considered to cover any and all modifications, variations, combinations, or equivalents within the scope of this application. Clearly, those skilled in the art can make various alterations and modifications to this application without departing from its scope. Thus, if such modifications and modifications fall within the scope of the claims and their equivalents, this application is also intended to include such modifications and modifications.

[0129] The above are merely preferred embodiments of the present invention. It should be noted that those skilled in the art can make various improvements and modifications without departing from the principle of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A method for fusion of infrared and visible light images based on semantic segmentation and multi-scale feature interaction, characterized in that, Includes the following steps: S1. Obtain the infrared image and the visible light image of the original image respectively, and perform feature extraction on them respectively to obtain the infrared image multi-scale feature map and the corresponding visible light multi-scale feature map. S2. A deep feature fusion module, obtained by combining a pre-built feature fusion module with an attention mechanism, fuses multi-scale feature maps of infrared images and corresponding multi-scale feature maps of visible light to obtain deep fused features; specifically including the following: S21. Set the number of feature fusion modules, connect them in a preset order, and set the corresponding convolution kernel size; S23. Input the infrared image multi-scale feature map and the corresponding visible light multi-scale feature map, perform at least 3 iterations, and output the corresponding output tensor pairs. S24. Add the tensor pairs obtained in step S23, output the feature representation, and obtain the deep fusion feature; The expressions for the deep feature fusion module and the feature fusion module are as follows: Among them, F S and F C Let F, S, and C represent spatial features, spatial features, and channel features, respectively. FF represents feature fusion, C represents concatenation, SA(·) represents spatial attention, CA(·) represents channel attention, and Conv1(·) represents convolution with a kernel size of 1. The expressions for SA(·) and CA(·) are as follows: SA(x)=Sig(Conv3(C{Max(x),Avg(x)}))×x CA(y)=Sig(FC(AP(y))+FC(MP(y))×y Where x represents the input of spatial attention, y represents the input of channel attention, Max(·) represents the maximum value operation, Avg(·) represents the average value operation, Conv3(·) represents the convolution operation with a kernel size of 3, Sig(·) represents the sigmoid activation function operation, AP(·) represents the average pooling operation, MP(·) represents the max pooling operation, and FC(·) represents the fully connected operation. and These represent the multi-scale features of infrared and visible light images obtained in the five stages of the multi-scale feature extraction block, respectively. and It is the feature obtained after j iterations of the multi-scale feature acquired in the i-th stage. It is the final output of the deep feature fusion module; S3. Through a pre-built multi-scale feature interaction module, deep fusion features are processed to obtain information interaction fusion features; specifically, the following content is included: Three multi-scale feature interaction modules were used to process the deep fusion features. Perform preliminary processing and interactive results. The fourth multi-scale feature interaction fusion module is used to process the interacted features. Further processing yields interactive features. The corresponding function expression is: Wherein, MFI represents multi-scale feature interaction operation, and its functional expression is: MFI{x,y,z}=Conv1(C({DC(Down(x)),DC(y),DC(Up(z))})) Where Down(·) represents a downsampling operation, Up(·) represents an upsampling operation, and DC(·) represents a dense linking operation; S4. By pre-building and optimizing the Transformer module, the information interaction fusion features are processed to obtain the global context information of the information interaction fusion features; specifically, this includes the following: The feature processing method is optimized based on the Transformer block, and a convolution-based local feature extraction channel is introduced to achieve more comprehensive feature extraction. The function expression of the optimized Transformer block is as follows: Where T(·) represents the Transformer operation, F T Indicates the features that have passed through the Transformer block; S5. The feature recombination module recombines and optimizes the global context information and the corresponding information interaction fusion features obtained after step S4 to obtain an optimized fusion image. S6. Using pre-constructed semantic segmentation, extract and retain semantic information from the optimized fusion image obtained in step S5, and output the semantic fusion image. S7. Using a pre-constructed joint loss function, input the semantic fusion image obtained in step S6 and the optimized fusion image obtained in step S5, and output the processed image. S8. Set the number of iterations for the processing of steps S1 to S7, and output the image after iterative processing as an infrared and visible light fusion image.

2. The infrared and visible light image fusion method based on semantic segmentation and multi-scale feature interaction as described in claim 1, characterized in that, Step S1 further includes: Two parallel scale feature extraction modules are used to process the infrared image and the corresponding visible light image, respectively. The scale feature extraction module uses at least 5 convolutional blocks, and the number of channels of the convolutional blocks is set to 32. The first, second and fifth convolutional blocks maintain the original resolution. The second convolutional block continues to maintain the original resolution, and the third and fourth convolutional blocks are downsampled by 2 times.

3. The infrared and visible light image fusion method based on semantic segmentation and multi-scale feature interaction as described in claim 1, characterized in that, The further details of steps S6 and S7 are as follows: A semantic segmentation framework is introduced. Within this framework, the segmentation loss function and the fusion loss function are combined to obtain a joint loss function for training. The expression for the joint loss function is as follows: L J =L F +αL S Among them, L J L F and L S These represent the joint loss function, fusion loss function, and segmentation loss function, respectively, with α representing a hyperparameter used to control L. F and L S The ratio between them; L F It combines pixel intensity loss L P Loss of detail L D The design and its corresponding function expression are as follows: L F =L P +βL D Where β represents a hyperparameter used to control L P and L D The ratio between them; L P and L D These are used to measure the difference between the generated image and the real image at the pixel level, and to measure the preservation of detail and texture information in the image, respectively. Their function expressions are as follows: Where H represents the height of the image, W represents the width of the image, and max(·) represents the operation of calculating the maximum value of the elements; L S It is used to add semantic information to the fused image, and its function body expression is as follows: THE S =L Main +λL Aux Where λ is a hyperparameter, L Main L represents the principal segmentation loss. Aux The auxiliary segmentation loss is expressed as follows: Where C represents the number of channels in the image, I S Indicates main output, I Sa Indicates auxiliary output, I So This represents a one-hot encoded vector.

4. An infrared and visible light image fusion system based on semantic segmentation and multi-scale feature interaction, characterized in that, include: At least one processor; as well as, A memory communicatively connected to at least one of the processors; wherein, The memory stores instructions that can be executed by the processor to implement the infrared and visible light image fusion method based on semantic segmentation and multi-scale feature interaction as described in any one of claims 1 to 3.

Images