Infrared and visible image fusion method and system based on primary and secondary feature separation

Abstract

The application discloses an infrared and visible light image fusion method and system based on primary and secondary feature separation, which comprises the following steps: using a SIFT algorithm to pre-process all images in a data set to obtain a primary feature image set and a background feature image set; constructing two dense residual neural networks DetailFoucs and BackgroundFoucs with different lengths, designing a loss function to pre-train the model, reserving model parameters, inputting the pre-processed training set into the dense residual neural network for unsupervised training, obtaining an optimized neural network, supplementing an image edge fusion module, and designing an overall network architecture; and inputting the primary and secondary image segmentation and classification into the trained neural network to obtain a fusion image. I vis and I ir The primary and secondary features of the source image are separated, the main features of the source image are retained to the maximum extent, the processing application range is improved, and resources are saved.

Description

Technical Field

[0001] This invention belongs to the field of computer software, and in particular relates to a method and system for infrared and visible light image fusion based on the separation of primary and secondary features. Background Technology

[0002] Image fusion is an image enhancement technique that overcomes the limitations of a single sensor or optical imaging device in fully representing key information about a scene. It generates a composite image by integrating complementary information from multiple original images according to predetermined fusion rules. Infrared images offer more complete contours but are limited by their imaging characteristics, resulting in poor representation of local details. Visible light images, due to their ability to capture rich gradient-varying reflected light information, possess finer texture information and higher spatial resolution, but suffer from significant loss of detail in highlights and shadows. The goal of image fusion is to preserve local details in visible light images while using infrared images to supplement missing details, thereby obtaining an image with clearer textures and more complete contours.

[0003] Currently, in the field of visible light and infrared image fusion, research focuses on deep learning-based image fusion methods, which typically operate in an end-to-end manner. The neural networks within these models are self-contained, allowing them to maintain generalization to input images. In CNN (Convolutional Neural Networks)-based methods, complex network architectures and customized loss functions are used for feature extraction, feature fusion, and image reconstruction. CNNs can learn the most valuable features from a large number of training images. DenseNet, as a neural network structure that effectively learns image features, employs dense connections, directly passing feature maps from all previous layers to the current layer. This allows subsequent layers to better utilize information from earlier layers, effectively mitigating the vanishing gradient problem and improving network training efficiency. In image fusion, DenseNet is primarily used to generate high-quality fused images. Some researchers use DenseNet as a base model, adjusting the network structure and parameters to fuse multiple input images into a single high-quality output image. Furthermore, some researchers combine DenseNet with other neural network structures to further improve image fusion performance.

[0004] In existing infrared and visible light fusion methods based on dense residual networks, some researchers construct networks at multiple scales, continuously extending the network length. This results in a small receptive field at shallow scales, enabling the extraction of richer detailed features at the same size, but the extracted features have weak semantic meaning. As the scale of the image passing through the neural network increases, the receptive field enlarges, enhancing the ability to represent spatial features and thus improving semantic meaning. However, an excessively large receptive field dilutes the image resolution, blurring the feature images and limiting their applicability. While current image fusion networks have achieved good fusion results, they still suffer from drawbacks such as wasted resources, narrow application scope, loss of some key features, and low network efficiency and quality.

[0005] In the Chinese patent "A Method for Fusion of Infrared and Visible Light Images" (application number CN 202010457106.X, publication number CN 111709903B), the decomposed detail signal is refined by extending the two-channel UMW method to a three-channel UMW method. This expands the original second-order detail signal into two first-order detail signals, making the detail signal fusion more targeted. This patent uses a traditional image fusion method. Although the three-channel UMW method emphasizes the main details of the source image, the manual customization of the fusion rules limits its application scope. Summary of the Invention

[0006] This invention addresses the shortcomings of existing technologies by providing an infrared and visible light image fusion method based on the separation of primary and secondary features. By preprocessing all images in the dataset, training a neural network model, supplementing the image edge fusion module, and designing an overall network architecture, this method can better emphasize the main features in the fused image and improve the recognition probability in target detection, outperforming similar models based on traditional and deep learning methods.

[0007] To achieve the above functions, the present invention adopts the following technical solution:

[0008] An infrared and visible light image fusion method based on primary and secondary feature separation includes:

[0009] S1. Preprocess all images in the dataset to obtain the main feature image set and the background feature image set.

[0010] S2. Construct two dense residual neural networks, DetailFoucs and BackgroundFoucs, with different lengths, and design a loss function to pre-train the model. Preserve the model parameters and design the overall network architecture based on the trained neural network.

[0011] S3, for the source image I vis and I irAfter performing primary and secondary image segmentation and classification, the images are fed into the neural network trained in step S2 to obtain the fused image.

[0012] Furthermore, in step S1, the specific steps for obtaining the image set are as follows:

[0013] S101. Use the SIFT algorithm to annotate key points in all infrared and visible light images in the dataset.

[0014] S102. Traverse the keypoints and segment the image into primary feature images and background feature images using rectangles. The segmentation rules are as follows: Define a 20x20 block and initialize a list of rectangles. Loop through each keypoint, calculate the coordinates of its surrounding smaller blocks, extract the block from the image, calculate the average intensity of the pixels within the block, adjust the size of the rectangle based on the average intensity, and calculate the coordinates of the rectangle. After the traversal is complete, the image contained within the rectangle is used as the primary feature image, and the remaining parts are used as secondary feature images.

[0015] Furthermore, in step S2, the specific steps for designing the overall network architecture are as follows:

[0016] S201. Construct a dense residual neural network with varying residual block lengths according to the network functional requirements: DetailFoucs for fusing primary features and BackgroundFoucs for fusing secondary features.

[0017] DetailFoucs consists of common layer1, common layer2, common layer3, common layer4, common layer5, Block1, Block2, Block3, and Block4. Common layer 1 has 4 input channels and 48 output channels; common layer 2 has 240 input channels and 240 output channels; common layer 3 has 240 input channels and 128 output channels; common layer 4 has 128 input channels and 64 output channels; common layer 5 has 64 input channels and 1 output channel; Block 1 consists of Conv1 with 48 input channels and 48 output channels and Conv2 with 48 input channels and 48 output channels; Block 2 consists of Conv1 with 96 input channels and 48 output channels and Conv2 with 48 input channels and 48 output channels; Block 3 consists of Conv1 with 144 input channels and 48 output channels and Conv2 with 48 input channels and 48 output channels; Block 4 consists of Conv1 with 192 input channels and 48 output channels and Conv2 with 48 input channels and 48 output channels.

[0018] BackgroundFormat consists of common layer1, common layer2, common layer3, common layer4, and Block. Common layer1 has 4 input channels and 48 output channels; common layer2 has 240 input channels and 128 output channels; common layer3 has 128 input channels and 64 output channels; common layer4 has 64 input channels and 1 output channel; Block consists of Conv1 with 48 input channels and 48 output channels, and Conv2 with 48 input channels and 48 output channels.

[0019] S202. Design different loss functions for DetailFoucs (used for fusing primary features) and BackgroundFoucs (used for fusing secondary features), respectively, so that the fusion network tends to preserve detailed textures or maintain overall image quality. The loss function for DetailFoucs is L... details (I ir ,I vis The BackgroundFocus loss function is composed of SSIM (Structural Similarity), which measures the similarity between two images based on pixel count, brightness, and overall structure. background (I ir ,I vis Image quality is measured by the MSE (Mean Squared Error) of the image, which measures the overall distribution of pixels.

[0020] The specific formula for the loss function is as follows:

[0021] L details (I ir ,I vis ) = SSIM(I F ,I VIS )+SSIM(I F ,I ir )

[0022]

[0023]

[0024]

[0025] Where μ represents the mean, σ represents the root mean square error, C1 and C2 represent the hyperparameters of the stability measure, H and W represent the height and width of the image, M represents the input image, and IF Indicates the fused image, I ir Indicating an infrared image, I vis Let S(i,j) and I represent the visible light image. F In (i,j), i and j represent the pixels in the horizontal and vertical directions of the image, || 2 This represents the L2 norm.

[0026] L details (I ir ,I vis This helps train the DetailFoucs fusion network to be suitable for loss functions in image fusion tasks with complex textures;

[0027] L background (I ir ,I vis The loss function helps train the BackgroundFoucs fusion network to be suitable for image fusion tasks with flat backgrounds.

[0028] S203. Input the preprocessed training set into the dense residual neural network for unsupervised training. The training optimizer used is Adamw, the learning rate is 1e-3, the training batch size is 200, and the batch size is 10. After training, the optimized neural network is obtained.

[0029] S204. The main feature images of infrared and visible light are concatenated on the channel and fed into the dense residual neural network DetailFoucs to obtain the fusion result. The fusion result is a grayscale fusion image of size (1, 1, H, W).

[0030] The secondary feature images of infrared and visible light are concatenated on the channel and fed into the dense residual neural network BackgroundFoucs to obtain the fusion result, which is a grayscale fused image of size (1, 1, H, W).

[0031] The fusion results of DetailFoucs and BackgroundFoucs are sorted according to their positions on the source image to obtain the image to be stitched together.

[0032] S205. Add an image edge fusion module to the back end of the model, and use Baisong fusion to stitch the fused segmented images to obtain the fusion result.

[0033] Furthermore, this invention also proposes an infrared and visible light image fusion system based on primary and secondary feature separation, comprising:

[0034] The feature image set extraction module is used to preprocess all images in the dataset to obtain the main feature image set and the background feature image set.

[0035] The overall network architecture design module is used to construct two dense residual neural networks of different lengths, design a loss function to pre-train the model, retain the model parameters, and design the overall network architecture based on the trained neural network.

[0036] The fused image training module is used to perform primary and secondary image segmentation and classification on the source image and then feed it into the trained neural network to obtain the fused image.

[0037] Furthermore, the specific steps for obtaining the feature image set in the feature image set extraction module are as follows:

[0038] Step 1: Use the SIFT algorithm to annotate key points in all infrared and visible light images in the dataset.

[0039] Step 2: Traverse the keypoints and segment the image into primary feature images and background feature images using rectangles. The segmentation rules are as follows: Define a 20x20 block and initialize a list of rectangles. Loop through each keypoint, calculate the coordinates of its surrounding smaller blocks, extract the block from the image, calculate the average intensity of the pixels within the block, adjust the size of the rectangle based on the average intensity, and calculate the coordinates of the rectangle. After the traversal is complete, the image contained within the rectangle is used as the primary feature image, and the remaining parts are used as secondary feature images.

[0040] Furthermore, in the overall network architecture design module, the specific steps for designing the overall network architecture are as follows:

[0041] Step 1: Construct a dense residual neural network with varying residual block lengths according to the network functional requirements: DetailFoucs for fusing primary features and BackgroundFoucs for fusing secondary features.

[0042] The DetailFoucs consists of common layer1, common layer2, common layer3, common layer4, common layer5, Block1, Block2, Block3, and Block4. Common layer1 has 4 input channels and 48 output channels; common layer2 has 240 input channels and 240 output channels; common layer3 has 240 input channels and 128 output channels; and common layer4 has 128 input channels and 64 output channels. Layer 5 has 64 input channels and 1 output channel; Block 1 consists of Conv1 with 48 input channels and 48 output channels and Conv2 with 48 input channels and 48 output channels; Block 2 consists of Conv1 with 96 input channels and 48 output channels and Conv2 with 48 input channels and 48 output channels; Block 3 consists of Conv1 with 144 input channels and 48 output channels and Conv2 with 48 input channels and 48 output channels; Block 4 consists of Conv1 with 192 input channels and 48 output channels and Conv2 with 48 input channels and 48 output channels.

[0043] BackgroundFoucs consists of common layer1, common layer2, common layer3, common layer4, and Block; common layer1 has 4 input channels and 48 output channels; common layer2 has 240 input channels and 128 output channels; common layer3 has 128 input channels and 64 output channels; common layer4 has 64 input channels and 1 output channel; Block consists of Conv1 with 48 input channels and 48 output channels and Conv2 with 48 input channels and 48 output channels.

[0044] Step 2: Design different loss functions for DetailFoucs and BackgroundFoucs respectively, so that the fusion network tends to preserve detailed textures or maintain overall image quality. The loss function for DetailFoucs is L... details (I ir ,I vis The BackgroundFocus loss function is composed of the Structural Similarity Index (SSIM), which measures the similarity between two images based on pixel count, brightness, and overall structure. background (I ir ,Ivis Image quality is measured by the mean square error (MSE) of the image, which measures the overall distribution of pixels.

[0045] The specific formula for the loss function is as follows:

[0046] L details (I ir ,I vis ) = SSIM(I F ,I VIS )+SSIM(I F ,I ir )

[0047]

[0048]

[0049]

[0050] Where μ represents the mean, σ represents the root mean square error, C1 and C2 represent the hyperparameters of the stability measure, H and W represent the height and width of the image, M represents the input image, and I F Indicates the fused image, I ir Indicating an infrared image, I vis Let S(i,j) and I represent the visible light image. F In (i,j), i and j represent the pixels in the horizontal and vertical directions of the image, || 2 This represents the L2 norm.

[0051] L details (I ir ,I vis This helps train the DetailFoucs fusion network to be suitable for loss functions in image fusion tasks with complex textures;

[0052] L background (I ir ,I vis The loss function helps train the BackgroundFoucs fusion network to be suitable for image fusion tasks with flat backgrounds.

[0053] Step 3: Input the preprocessed training set into the dense residual neural network for unsupervised training. The training optimizer is Adamw, the learning rate is 1e-3, the training batch size is 200, and the batch size is 10. After training, the optimized neural network is obtained.

[0054] Step 4: Concatenate the main feature images of infrared and visible light on the channel and feed them into the dense residual neural network DetailFoucs to obtain the fusion result. The fusion result is a grayscale fused image of size (1, 1, H, W).

[0055] The secondary feature images of infrared and visible light are concatenated on the channel and fed into the dense residual neural network BackgroundFoucs to obtain the fusion result, which is a grayscale fused image of size (1, 1, H, W).

[0056] The fusion results of DetailFoucs and BackgroundFoucs are sorted according to their positions on the source image to obtain the image to be stitched together.

[0057] Step 5: Add an image edge fusion module to the back end of the model, and use Baisong fusion to stitch the fused segmented images to obtain the fusion result.

[0058] The present invention adopts the above technical solution, and its significant technical effects compared with the prior art are as follows:

[0059] (1) No need to customize fusion rules: The fusion network is designed using a deep learning-based method to perform fusion in an end-to-end manner. The model can adaptively change the fusion rules, avoiding the drawbacks of existing methods. For example, misaligned source image edges and changes in image modality may lead to the loss of details in the fusion result.

[0060] (2) Separation of primary and secondary features: This invention separates the primary and secondary features of the source image to obtain primary features and secondary features. It designs a dense residual image fusion neural network that tends to preserve details and textures or maintain the overall image quality for the information-rich primary feature image and the information-poor background feature. This avoids the disadvantages of a unified fusion network losing some primary features, wasting resources, and having a narrow application scope. It preserves the primary features of the source image to the greatest extent, ensuring that the features have clear outlines and rich textures, which promotes the success rate of target detection.

[0061] (3) Fusion network design: Different fusion networks and loss functions are designed according to the imaging characteristics of the main features and secondary features, so as to avoid the shortcomings of existing fusion networks that cause the fusion results to lose some main features. This makes the network as a whole exhibit high efficiency and high quality.

[0062] (4) Concentrated application resources: Image fusion is mainly used in the field of target detection, and the main requirement is to detect key features. This invention does not pursue the overall image fusion quality but concentrates resources on the main features, which makes the fusion network use resources efficiently. Attached Figure Description

[0063] Figure 1 This is a flowchart illustrating the overall steps of the present invention.

[0064] Figure 2 This is a schematic diagram of the DetailFoucs content construction of the dense residual fusion network of the present invention.

[0065] Figure 3 This is a schematic diagram of the content construction of the dense residual fusion network BackgroundFoucs of this invention.

[0066] Figure 4 This is an input visible light image, which is an embodiment of the present invention.

[0067] Figure 5 This is an input infrared image according to an embodiment of the present invention.

[0068] Figure 6 This is an infrared key point annotation image of an embodiment of the present invention.

[0069] Figure 7 This is a visible light key point annotation image of an embodiment of the present invention.

[0070] Figure 8 These are the main feature images of visible light and infrared images in embodiments of the present invention.

[0071] Figure 9 This is a secondary feature image of a visible light image according to an embodiment of the present invention.

[0072] Figure 10 This is a secondary feature image of the infrared image in an embodiment of the present invention.

[0073] Figure 11 This is a diagram showing the final fusion result of an embodiment of the present invention.

[0074] Figure 12 This is a comparison of the present invention with the most representative method on six sets of data.

[0075] Figure 13 This is a comparison of the results of this invention with the most representative method in target detection. Detailed Implementation

[0076] The specific implementation of the present invention will be further described below with reference to the accompanying drawings:

[0077] An infrared and visible light image fusion method based on primary and secondary feature separation, such as Figure 1 As shown, it includes:

[0078] S1. Preprocess all images in the dataset to obtain the main feature image set and the background feature image set. The specific steps are as follows:

[0079] S101. Use the SIFT algorithm to annotate key points in all infrared and visible light images in the dataset.

[0080] S102. Traverse the keypoints and segment the image into primary feature images and background feature images using rectangles. The segmentation rules are as follows: Define a 20x20 block and initialize a list of rectangles. Loop through each keypoint, calculate the coordinates of its surrounding smaller blocks, extract the block from the image, calculate the average intensity of the pixels within the block, adjust the size of the rectangle based on the average intensity, and calculate the coordinates of the rectangle. After the traversal is complete, the image contained within the rectangle is used as the primary feature image, and the remaining parts are used as secondary feature images.

[0081] S2. Construct two dense residual neural networks of different lengths, design a loss function to pre-train the model, retain the model parameters, and design the overall network architecture based on the trained neural network. The specific steps are as follows:

[0082] S201. Construct a dense residual neural network with varying residual block lengths according to the network functional requirements: DetailFoucs for fusing primary features and BackgroundFoucs for fusing secondary features.

[0083] Among them, such as Figure 2 As shown, DetailFoucs consists of common layer1, common layer2, common layer3, common layer4, common layer5, Block1, Block2, Block3, and Block4. Common layer 1 has 4 input channels and 48 output channels; common layer 2 has 240 input channels and 240 output channels; common layer 3 has 240 input channels and 128 output channels; common layer 4 has 128 input channels and 64 output channels; common layer 5 has 64 input channels and 1 output channel; Block 1 consists of Conv1 with 48 input channels and 48 output channels and Conv2 with 48 input channels and 48 output channels; Block 2 consists of Conv1 with 96 input channels and 48 output channels and Conv2 with 48 input channels and 48 output channels; Block 3 consists of Conv1 with 144 input channels and 48 output channels and Conv2 with 48 input channels and 48 output channels; Block 4 consists of Conv1 with 192 input channels and 48 output channels and Conv2 with 48 input channels and 48 output channels.

[0084] like Figure 3As shown, BackgroundFormat consists of common layer1, common layer2, common layer3, common layer4, and Block. Common layer1 has 4 input channels and 48 output channels; common layer2 has 240 input channels and 128 output channels; common layer3 has 128 input channels and 64 output channels; common layer4 has 64 input channels and 1 output channel; Block consists of Conv1 with 48 input channels and 48 output channels, and Conv2 with 48 input channels and 48 output channels.

[0085] S202. Different loss functions are designed for DetailFoucs (used for fusing primary features) and BackgroundFoucs (used for fusing secondary features), so that the fusion network tends to preserve detailed textures or maintain overall image quality. The loss function L for DetailFoucs is... details (I ir ,I vis The BackgroundFocus loss function is composed of the Structural Similarity Index (SSIM), which measures the similarity between two images based on pixel count, brightness, and overall structure. background (I ir ,I vis Image quality is measured by the mean square error (MSE) of the image, which measures the overall distribution of pixels.

[0086] The specific formula for the loss function is as follows:

[0087] L details (I ir ,I vis ) = SSIM(I F ,I VIS )+SSIM(I F ,I ir )

[0088]

[0089]

[0090]

[0091] Where μ represents the mean, σ represents the root mean square error, C1 and C2 represent the hyperparameters of the stability measure, H and W represent the height and width of the image, M represents the input image, and I F Indicates the fused image, I ir Indicating an infrared image, I visRepresents the visible light image. S(i,j) and I F In (i,j), i and j represent the pixels in the horizontal and vertical directions of the image. 2 This represents the L2 norm.

[0092] L details (I ir ,I vis This helps train the DetailFoucs fusion network to be suitable for loss functions in image fusion tasks with complex textures;

[0093] L background (I ir ,I vis The loss function helps train the BackgroundFoucs fusion network to be suitable for image fusion tasks with flat backgrounds.

[0094] S203. Input the preprocessed training set into the dense residual neural network for unsupervised training. The training optimizer is Adamw, the learning rate is 1e-3, the training batch size is 200, and the batch size is 10. After training, the optimized neural network is obtained.

[0095] S204. The main feature images of infrared and visible light are concatenated on the channel and fed into the dense residual neural network DetailFoucs to obtain the fusion result. The fusion result is a grayscale fusion image of size (1, 1, H, W).

[0096] The secondary feature images of infrared and visible light are concatenated on the channel and fed into the dense residual neural network BackgroundFoucs to obtain the fusion result, which is a grayscale fused image of size (1, 1, H, W).

[0097] The fusion results of DetailFoucs and BackgroundFoucs are sorted according to their positions on the source image to obtain the image to be stitched together.

[0098] S205. Add an image edge fusion module to the back end of the model, and use Baisong fusion to stitch the fused segmented images to obtain the fusion result.

[0099] S3, for the source image I vis and I ir After performing primary and secondary image segmentation and classification, the images are fed into a trained neural network to obtain a fused image.

[0100] right Figure 4 The visible light image input into the input and Figure 5The input infrared image is annotated with key points. The key points are then traversed, and rectangles are used to segment the image into primary feature images and background feature images. The segmentation rules are as follows: Define a 20x20 block and initialize a list of rectangles. Iterate through each key point, calculate the coordinates of its surrounding smaller blocks, extract the smaller blocks from the image, calculate the average intensity of the pixels within the block, adjust the size of the rectangle based on the average intensity, and calculate the coordinates of the rectangle. After the traversal, the image contained within the rectangle is used as the primary feature image, and the remaining parts are used as secondary feature images. This results in the following: Figure 6 The infrared key point annotation image shown and such Figure 7 The visible light keypoint annotation image shown has dots representing keypoints and rectangles representing the main feature regions. By observing the annotation areas within the rectangles, it is clear that the important information contained in the image has been marked. Based on these rectangles, the input visible light image and the input infrared image are segmented to obtain the following... Figure 8 The main feature images shown are as follows: Figure 9 The secondary feature images of the visible light image shown and such as Figure 10 The secondary feature images of the infrared image shown show that the information-rich areas have been extracted, and the segmented secondary feature images are characterized by background information, flat regions, and low information content.

[0101] Will Figure 8 Main feature images and Figure 9 , Figure 10 The secondary feature images are fed into a fusion network for block fusion, and finally, the final fusion result is obtained using Boson fusion, such as... Figure 11 As shown. Figure 11 The final fused image has prominent subject features and rich background features, and the overall image exhibits excellent readability, which can greatly improve the results of object detection.

[0102] In addition, such as Figure 12 Qualitative experimental results show that the method of this invention is compared with the most representative method in multiple sets of visible light and infrared image pairs. For example... Figure 13 The target detection results show that the method of this invention is compared with the most representative method in target detection experiments on multiple sets of visible light and infrared image pairs. Figure 12 It can be observed that the experimental results of this invention show that FocusNet outperforms other methods, through... Figure 13 This invention demonstrates that it enhances the confidence level of target detection.

[0103] This invention also proposes an infrared and visible light image fusion system based on primary and secondary feature separation, including a feature image set extraction module, an overall network architecture design module, a fused image training module, and a computer program that can run on a processor. It should be noted that each module in the above system corresponds to a specific step of the method provided in this invention embodiment, possessing the corresponding functional modules and beneficial effects for executing the method. Technical details not described in detail in this embodiment can be found in the method provided in this invention embodiment.

[0104] The above embodiments are merely illustrative of the technical concept of the present invention and should not be construed as limiting the scope of protection of the present invention. Any modifications made to the technical solution based on the technical concept proposed in this invention shall fall within the scope of protection of this invention.

Claims

1. An infrared and visible light image fusion method based on primary and secondary feature separation, characterized in that, include: S1. Preprocess all images in the dataset to obtain the main feature image set and the background feature image set; S2. Construct two dense residual neural networks of different lengths, design a loss function to pre-train the model, retain the model parameters, and design the overall network architecture based on the trained neural networks; specifically: S201. Construct a dense residual neural network with varying residual block lengths according to the network functional requirements: DetailFoucs for fusing main features and BackgroundFoucs for fusing secondary features; S202. Design different loss functions based on DetailFoucs and BackgroundFoucs respectively. The loss function for DetailFoucs is L. details (I ir ,I vis The loss function of BackgroundFocus consists of the structural similarity index SSIM. background (I ir ,I vis It consists of the mean square error (MSE) of the image; The specific formula for the loss function is as follows: L details (I ir ,I vis )=SSIM(I F ,I VIS )+SSIM(I F ,I ir ) Where μ represents the mean, σ represents the root mean square error, C1 and C2 represent the hyperparameters of the stability measure, H and W represent the height and width of the image, M represents the input image, and I F Indicates the fused image, I ir Indicating an infrared image, I vis Let S(i,j) and I represent the visible light image. F In (i,j), i and j represent the pixels in the horizontal and vertical directions of the image, || 2 Represents the L2 norm; S203. Input the preprocessed training set into the dense residual neural network for unsupervised training. The training optimizer is Adamw, the learning rate is 1e-3, the training batch size is 200, and the batch size is 10. After training, the optimized neural network is obtained. S204. The main feature images of infrared and visible light are concatenated on the channel and fed into the dense residual neural network DetailFoucs to obtain the fusion result. The fusion result is a grayscale fusion image of size (1, 1, H, W). The secondary feature images of infrared and visible light are concatenated on the channel and fed into the dense residual neural network BackgroundFoucs to obtain the fusion result, which is a grayscale fusion image of size (1, 1, H, W). The fusion results of DetailFoucs and BackgroundFoucs are sorted according to their positions on the source image to obtain the image to be stitched together; S205. Add an image edge fusion module to the back end of the model, use Baisong fusion to stitch and fuse the images, and stitch the fused segmented images to obtain the fusion result. S3, for the source image I vis and I ir After performing primary and secondary image segmentation and classification, the images are fed into a trained neural network to obtain a fused image.

2. The infrared and visible light image fusion method based on primary and secondary feature separation according to claim 1, characterized in that, In step S1, the specific steps for obtaining the image set are as follows: S101. Use the SIFT algorithm to annotate key points in all infrared and visible light images in the dataset; S102. Traverse the key points and use rectangles to segment the image into a primary feature image and a background feature image. The segmentation rules are as follows: Define a 20x20 block and initialize a list of rectangles. Iterate through each key point, calculate the coordinates of the surrounding small blocks, extract the small block from the image, calculate the average intensity of the pixels within the block, adjust the size of the rectangle according to the average intensity, and calculate the coordinates of the rectangle. After the traversal, the image contained in the rectangle is used as the primary feature image, and the rest is used as the secondary feature image.

3. The infrared and visible light image fusion method based on primary and secondary feature separation according to claim 1, characterized in that, In step S2, DetailFoucs consists of common layer1, common layer2, common layer3, common layer4, common layer5, Block1, Block2, Block3, and Block4; common layer1 has 4 input channels and 48 output channels; common layer2 has 240 input channels and 240 output channels; common layer3 has 240 input channels and 128 output channels; common layer4 has 128 input channels and 64 output channels; common layer5... Layer 5 has 64 input channels and 1 output channel; Block 1 consists of Conv1 with 48 input channels and 48 output channels, and Conv2 with 48 input channels and 48 output channels; Block 2 consists of Conv1 with 96 input channels and 48 output channels, and Conv2 with 48 input channels and 48 output channels; Block 3 consists of Conv1 with 144 input channels and 48 output channels, and Conv2 with 48 input channels and 48 output channels; Block 4 consists of Conv1 with 192 input channels and 48 output channels, and Conv2 with 48 input channels and 48 output channels. BackgroundFoucs consists of common layer1, common layer2, common layer3, common layer4, and Block; common layer1 has 4 input channels and 48 output channels; common layer2 has 240 input channels and 128 output channels; common layer3 has 128 input channels and 64 output channels; common layer4 has 64 input channels and 1 output channel; Block consists of Conv1 with 48 input channels and 48 output channels and Conv2 with 48 input channels and 48 output channels.

4. An infrared and visible light image fusion system based on primary and secondary feature separation, characterized in that, include: The feature image set extraction module is used to preprocess all images in the dataset to obtain the main feature image set and the background feature image set; The overall network architecture design module is used to construct two dense residual neural networks of different lengths, design a loss function to pre-train the model, retain the model parameters, and design the overall network architecture based on the trained neural networks; specifically: Step 1: Construct a dense residual neural network with varying residual block lengths according to the network functional requirements: DetailFoucs for fusing primary features and BackgroundFoucs for fusing secondary features; Step 2: Design different loss functions for DetailFoucs and BackgroundFoucs respectively. The loss function for DetailFoucs is L. details (I ir ,I vis The loss function of BackgroundFocus consists of the structural similarity index SSIM. background (I ir ,I vis It consists of the mean square error (MSE) of the image; The specific formula for the loss function is as follows: L details (I ir ,I vis )=SSIM(I F ,I VIS )+SSIM(I F ,I ir ) Where μ represents the mean, σ represents the root mean square error, C1 and C2 represent the hyperparameters of the stability measure, H and W represent the height and width of the image, M represents the input image, and I F Indicates the fused image, I ir Indicating an infrared image, I vis Let S(i,j) and I represent the visible light image. F In (i,j), i and j represent the pixels in the horizontal and vertical directions of the image, || 2 Represents the L2 norm; Step 3: Input the preprocessed training set into the dense residual neural network for unsupervised training. The training optimizer is Adamw, the learning rate is 1e-3, the training batch size is 200, and the batch size is 10. After training, the optimized neural network is obtained. Step 4: Concatenate the main feature images of infrared and visible light on the channel and feed them into the dense residual neural network DetailFoucs to obtain the fusion result. The fusion result is a grayscale fusion image of size (1, 1, H, W). The secondary feature images of infrared and visible light are concatenated on the channel and fed into the dense residual neural network BackgroundFoucs to obtain the fusion result, which is a grayscale fusion image of size (1, 1, H, W). The fusion results of DetailFoucs and BackgroundFoucs are sorted according to their positions on the source image to obtain the image to be stitched together; Step 5: Add an image edge fusion module to the back end of the model, use Baisong fusion to stitch and fuse the images, and stitch the fused segmented images to obtain the fusion result; The fused image training module is used to perform primary and secondary image segmentation and classification on the source image and then feed it into the trained neural network to obtain the fused image.

5. The infrared and visible light image fusion system based on primary and secondary feature separation according to claim 4, characterized in that, The specific steps for obtaining the feature image set in the feature image set extraction module are as follows: Step 1: Use the SIFT algorithm to annotate key points in all infrared and visible light images in the dataset; Step 2: Traverse the key points and use rectangles to segment the image into a primary feature image and a background feature image. The segmentation rules are as follows: Define a 20x20 block and initialize a list of rectangles. Iterate through each key point, calculate the coordinates of the surrounding small blocks, extract the small block from the image, calculate the average intensity of the pixels within the block, adjust the size of the rectangle according to the average intensity, and calculate the coordinates of the rectangle. After the traversal is complete, the image contained in the rectangle is used as the primary feature image, and the rest is used as the secondary feature image.

6. The infrared and visible light image fusion system based on primary and secondary feature separation according to claim 4, characterized in that, In the overall network architecture design module, DetailFoucs consists of common layer1, common layer2, common layer3, common layer4, common layer5, Block1, Block2, Block3, and Block4; common layer1 has 4 input channels and 48 output channels; common layer2 has 240 input channels and 240 output channels; common layer3 has 240 input channels and 128 output channels; common layer4 has 128 input channels and 64 output channels; common... Layer 5 has 64 input channels and 1 output channel; Block 1 consists of Conv1 with 48 input channels and 48 output channels, and Conv2 with 48 input channels and 48 output channels; Block 2 consists of Conv1 with 96 input channels and 48 output channels, and Conv2 with 48 input channels and 48 output channels; Block 3 consists of Conv1 with 144 input channels and 48 output channels, and Conv2 with 48 input channels and 48 output channels; Block 4 consists of Conv1 with 192 input channels and 48 output channels, and Conv2 with 48 input channels and 48 output channels. BackgroundFoucs consists of common layer1, common layer2, common layer3, common layer4, and Block; common layer1 has 4 input channels and 48 output channels; common layer2 has 240 input channels and 128 output channels; common layer3 has 128 input channels and 64 output channels; common layer4 has 64 input channels and 1 output channel; Block consists of Conv1 with 48 input channels and 48 output channels and Conv2 with 48 input channels and 48 output channels.

Images