Lightweight infrared and visible image fusion method based on vgg16 and target detection

Abstract

The application discloses a lightweight infrared and visible light image fusion method based on VGG16 and target detection, and is specifically implemented according to the following steps: step 1, designing an infrared and visible light image fusion network based on DenseNet; step 2, generating a final image fusion result through the infrared and visible light image fusion network; step 3, designing a loss function using a VGG16 network and a YOLO V5 network, and training an infrared and visible light image fusion algorithm based on visual enhancement; and step 4, designing an alternating cyclic learning strategy, and finally obtaining an optimal fusion image by using target detection loss to guide the fusion network after multiple alternating training. The application solves the problems of the prior art that a foreground target of a fusion image is not prominent and an image does not conform to human eye perception, and significantly improves the overall effect in advanced visual tasks.

Description

Technical Field

[0001] This invention belongs to the field of computer digital image processing technology, specifically relating to a lightweight infrared and visible light image fusion method based on VGG16 and target detection. Background Technology

[0002] The core objective of infrared and visible light image fusion technology is to generate high-quality composite images for downstream tasks such as target detection and image segmentation. However, current deep learning-based infrared and visible light image fusion algorithms neglect the impact of the fused image on downstream tasks, and simple loss functions cannot simulate human visual perception. Specifically, the fused images generated by these algorithms often fail to highlight foreground targets, and the L2 loss used does not match the image quality perceived by the human eye. To address this issue, there is an urgent need to develop an innovative infrared and visible light image fusion algorithm to improve the quality and applicability of the generated images. Summary of the Invention

[0003] The purpose of this invention is to provide a lightweight infrared and visible light image fusion method based on VGG16 and target detection, which solves the problems of indistinct foreground targets and inconsistent image perception in the fused images of the prior art, and significantly improves the overall performance in advanced vision tasks.

[0004] The technical solution adopted in this invention is a lightweight infrared and visible light image fusion method based on VGG16 and target detection, which is implemented according to the following steps:

[0005] Step 1: Design an infrared and visible light image fusion network based on DenseNet;

[0006] Step 2: Generate the final image fusion result using an infrared and visible light image fusion network;

[0007] Step 3: Design loss functions using VGG16 and YOLO v5 networks, and train an infrared and visible light image fusion algorithm based on visual enhancement.

[0008] Step 4: Design an alternating cyclic learning strategy. After multiple alternating training sessions, use object detection loss to guide the fusion network and finally obtain the optimal fused image.

[0009] The invention is further characterized in that,

[0010] Step 1 is implemented in the following steps:

[0011] Visible light and infrared image fusion networks are based on the DenseNet structure; the mainstream approach uses the Y channel of visible light images as input.

[0012] Step 1.1: Convert the RGB channels of the visible light image to CbCr channels, and concatenate the Y channel of the visible light image with the infrared image along the channel dimension to form a two-dimensional feature map. This process is represented as follows:

[0013] F c =concat(F vis ,F ir (1)

[0014] Among them, F c This represents the feature map after stitching visible light and infrared light images. `concat(,)` indicates stitching along the channel dimension. F vis The Y channel of a visible light image, F ir Represents an infrared image;

[0015] Step 1.2, F c First, a convolution operation and a batch normalization operation are performed, followed by a ReLU activation function to obtain shallow features. This process can be represented as follows:

[0016] F1 = ReLU(BN(Conv(F)) c (2)

[0017] F1 is the shallow feature obtained, ReLU is the non-linear activation function, BN is the batch normalization check, and Conv is a convolution operation with kernel_size (3,3) and stride (1,1).

[0018] Step 1.3: After the initial feature extraction in Step 1.2, the data is processed through a 3-layer DenseNet network. The results from each layer are spliced ​​to the next layer. The F1 score obtained earlier is spliced ​​to the next three layers along the channel dimension. The convolution operation is performed by grouped convolutions, as shown in the following process:

[0019] F2=ReLU(BN(Conv(F1))) (3)

[0020] F3=ReLU(BN(Conv(concat(F1,F2))) (4)

[0021] F4=ReLU(BN(Conv(concat(F1,F2,F3))) (5)

[0022] Where Conv represents a grouped convolution with kernel size (3,3), stride (1,1), and group size of 64, F2, F3, and F4 are the outputs of each layer. F2, F3, and F4 all contain information from the features of the previous layer. F2 is obtained by convolution, normalization, and activation of the previous F1 layer; F3 is obtained by convolution, normalization, and activation of the previous F1 and F2 layers; and F4 is obtained by convolution, normalization, and activation of the previous F1, F2, and F3 layers, so that the final output can contain both shallow detailed information and deep semantic information.

[0023] Step 1.4: After obtaining the F4 matrix with rich features, the features need to be fused to achieve the best results. After inputting the F4 matrix into the base convolution, the last layer uses the Tanh activation function. The process is represented as follows:

[0024] F5 = ReLU(Conv(F4)) (6)

[0025] F6 = ReLU(BN(Conv(F5))) (7)

[0026] F7 = ReLU(BN(Conv(F6))) (8)

[0027] F fuse =Tanh(Conv(F7)) (9)

[0028] F5, F6, and F7 are the outputs of the intermediate layers. F5, F6, and F7 use grouped convolutions with a kernel size of (3,3), a stride of (1,1), and a group size of 64. fuse The final result of the fusion of visible and infrared images uses a normal convolution with kernel_size of (3,3), stride of (1,1), and group size of 64.

[0029] Step 2 is implemented in the following steps:

[0030] First, prepare training data for infrared and visible light images. Infrared light has only one channel. Convert the RGB channel of the visible light image to a CbCr channel. Then, stitch the Y channel of the visible light image with the infrared image in the channel dimension. Input the stitching result into the fusion network constructed in step 1 to obtain the fused image. Construct the loss function in step 3 using the fused image.

[0031] Step 3 is implemented in the following steps:

[0032] Step 3.1: To ensure structural similarity between the fused image and the source image, a structural similarity loss function (SIMM) is introduced. The SIMM loss function is based on the perceptual characteristics of the human visual system and primarily evaluates image quality from three aspects: brightness, contrast, and structure. Specifically:

[0033] L SIMM =α1SIMM(F vis ,F fuse )+α2SIMM(F ir ,F fuse (10)

[0034]

[0035] Where x and y are the two images to be compared, and μ x ,μ y σ represents the mean of the image x and y. x ,σ y σ represents the variance of the image x and y. xy Let C1 and C2 represent the covariance of x and y, C1 and C2 are smoothing coefficients to prevent the denominator from being 0, and α1 and α2 are hyperparameters.

[0036] Step 3.2: The VGG16 algorithm has four downsampling operations. Through these four downsampling operations, the VGG16 is divided into four layers. The shallow layers have a good extraction effect on texture detail features, while the deep layers have a good extraction effect on semantic information of the target person, vehicle, and target. The initial input visible light image and the fused image obtained in step 2 are input into the pre-trained VGG16, and the first layer is used for L1 loss. The initial input infrared light image and the fused image obtained in step 2 are input into the pre-trained VGG16, and the third layer is used for L1 loss. Specifically:

[0037]

[0038] in, This represents the output of the m-th layer of the VGG16 network, which is input as a fused image. This represents the output of the m-th layer of a VGG16 network when a visible light image is input. This represents the output of the nth layer of the VGG16 network, which is the input of the fused image. This represents the output of the nth layer of the VGG16 network when an infrared light image is input; β1 and β2 are balancing parameters.

[0039] Step 3.3, Introduce object detection loss: A typical object detection network, YOLOv5, is represented as:

[0040]

[0041] st F fuse= Φ(F vis ,F ir ;ω f (14)

[0042] Among them, F fuse For the generated fused image, F ir Represents an infrared image, F vis This indicates that the light image ω can be represented f ω d Let Φ and ψ be the trainable parameters in the fusion network and the object detection network, respectively, and L be the parameters of the fusion network and the object detection network. d This is the target detection loss, and ω is optimized using the target detection loss. f ω d To obtain the optimal parameters;

[0043] Step 3.4, the total loss is expressed as:

[0044] L total =aL SIMM +bL v +cL d (15)

[0045] Where a, b, and c are equilibrium parameters, and L total L represents the total loss. SIMM Represents structural loss L v Indicates VGG16 loss, L d This indicates the target detection loss.

[0046] Step 4 is implemented in the following steps:

[0047] Step 4.1 In the first round of training, the fusion network from Step 1 is initially trained. VGG16 is not trained because gradient truncation is performed. Formulas (10) and (12) are used to ensure the rapid convergence of the fusion network training.

[0048] Step 4.2: In the subsequent training, the fusion network and the object detection network are trained simultaneously using formula (15), which includes object detection loss, to train the fusion network of step 1. Formula (13) is used to train the object detection network introduced in step 3.2, so as to obtain a better object detection network to guide the training of the fusion network.

[0049] Step 4.3: Through multiple iterations of training, step 4.1 executes 50 epochs, and step 4.2 executes 50 epochs of alternating training. Through multiple alternating training, the fusion network and the object detection network are optimized. Finally, the optimal fusion network is used to obtain the fused image.

[0050] The beneficial effects of this invention are that the lightweight infrared and visible light image fusion method based on VGG16 and object detection employs an alternating learning strategy. Through multiple alternating training iterations, the fusion network and the object detection network successfully reach their optimal state, facilitating the object detection network to guide the fusion network to focus more on foreground objects. In specific implementation, the algorithm adds grouped convolutions to the DenseNet network to reduce the amount of data. Object detection loss is used to guide the fusion network to focus on foreground objects, making the fused image more prominent for the foreground objects. VGG16 loss is used to make the generated fused image closer to human visual perception, enriching semantic information. Attached Figure Description

[0051] Figure 1 This is the overall design concept of the lightweight infrared and visible light image fusion method based on VGG16 and target detection in this invention;

[0052] Figure 2 This is a schematic diagram of the framework of the lightweight infrared and visible light image fusion method based on VGG16 and target detection of the present invention;

[0053] Figure 3 This is a schematic diagram of the VGG16 loss in the lightweight infrared and visible light image fusion method based on VGG16 and target detection of the present invention.

[0054] Figure 4 These are fusion image renderings of the lightweight infrared and visible light image fusion method based on VGG16 and object detection in the LLVIP dataset under different scenarios. Detailed Implementation

[0055] The present invention will now be described in detail with reference to the accompanying drawings and specific embodiments.

[0056] like Figure 1 As shown, firstly, the Y channel of the visible light image is extracted and concatenated with the infrared light image in a dimensional manner. This concatenation is then input into the constructed DenseNet network, which extracts details at different levels while avoiding information loss. Grouped convolutions are used to address the excessive computational cost caused by the large number of feature maps. Secondly, VGG16 loss and object detection loss are constructed. The VGG16 loss generates a fused image that is closer to human perception and enriches semantic information; the object detection loss helps the fusion network focus on foreground objects, making the fused image more prominent for foreground objects. Finally, an alternating learning strategy is constructed to achieve mutual cyclical promotion between the image fusion and object detection networks.

[0057] This invention presents a lightweight infrared and visible light image fusion method based on VGG16 and target detection, which is implemented according to the following steps:

[0058] Step 1: Design an infrared and visible light image fusion network based on DenseNet, where DenseNet uses grouped convolutions to reduce the number of parameters. DenseNet is used to obtain feature combinations at different levels of the image, and the final fused image is obtained through several convolutional layers.

[0059] Step 1 is implemented in the following steps:

[0060] Visible light and infrared image fusion networks are based on the DenseNet structure; the mainstream approach uses the Y channel of visible light images as input.

[0061] Step 1.1: Convert the RGB channels of the visible light image to CbCr channels, and concatenate the Y channel of the visible light image with the infrared image along the channel dimension to form a two-dimensional feature map. This process is represented as follows:

[0062] F c =concat(F vis ,F ir (1)

[0063] Among them, F c This represents the feature map after stitching visible light and infrared light images. `concat(,)` indicates stitching along the channel dimension. F vis The Y channel of a visible light image, F ir Represents an infrared image;

[0064] Step 1.2, F c First, a convolution operation and a batch normalization operation are performed, followed by a ReLU activation function to obtain shallow features. This process can be represented as follows:

[0065] F1 = ReLU(BN(Conv(F)) c (2)

[0066] F1 is the shallow feature obtained, ReLU is the non-linear activation function, BN is the batch normalization check, and Conv is a convolution operation with kernel_size (3,3) and stride (1,1).

[0067] Step 1.3: After the initial feature extraction in Step 1.2, the data is processed through a 3-layer DenseNet network. The results from each layer are spliced ​​to subsequent layers to avoid information loss during convolution and to ensure that earlier shallow features are incorporated into later deep features, resulting in richer features. The F1 score obtained earlier is then spliced ​​to the next three layers along the channel dimension. The convolution operation is performed using grouped convolutions to reduce the number of parameters. The process is represented as follows:

[0068] F2=ReLU(BN(Conv(F1))) (3)

[0069] F3=ReLU(BN(Conv(concat(F1,F2))) (4)

[0070] F4=ReLU(BN(Conv(concat(F1,F2,F3))) (5)

[0071] Where Conv represents a grouped convolution with kernel size (3,3), stride (1,1), and group size of 64, F2, F3, and F4 are the outputs of each layer. F2, F3, and F4 all contain information from the features of the previous layer. F2 is obtained by convolution, normalization, and activation of the previous F1 layer; F3 is obtained by convolution, normalization, and activation of the previous F1 and F2 layers; and F4 is obtained by convolution, normalization, and activation of the previous F1, F2, and F3 layers, so that the final output can contain both shallow detailed information and deep semantic information.

[0072] Step 1.4: After obtaining the F4 matrix with rich features, the features need to be fused to achieve the best results. After inputting the F4 matrix into the base convolution, the last layer uses the Tanh activation function. The process is represented as follows:

[0073] F5 = ReLU(Conv(F4)) (6)

[0074] F6 = ReLU(BN(Conv(F5))) (7)

[0075] F7 = ReLU(BN(Conv(F6))) (8)

[0076] F fuse =Tanh(Conv(F7)) (9)

[0077] F5, F6, and F7 are the outputs of the intermediate layers. F5, F6, and F7 use grouped convolutions with a kernel size of (3,3), a stride of (1,1), and a group size of 64. fuse The final result of the fusion of visible and infrared images uses a normal convolution with kernel_size of (3,3), stride of (1,1), and group size of 64.

[0078] Step 2: Generate the final image fusion result using an infrared and visible light image fusion network;

[0079] Step 2 is implemented in the following steps:

[0080] First, prepare training data for infrared and visible light images. Infrared light has only one channel. Convert the RGB channel of the visible light image to a CbCr channel. Then, stitch the Y channel of the visible light image with the infrared image in the channel dimension. Input the stitching result into the fusion network constructed in step 1 to obtain the fused image. Construct the loss function in step 3 using the fused image.

[0081] Step 3: Design loss functions using VGG16 and YOLO v5 networks, and train an infrared and visible light image fusion algorithm based on visual enhancement.

[0082] Step 3 is implemented in the following steps:

[0083] Step 3.1: To ensure structural similarity between the fused image and the source image, a structural similarity loss function (SIMM) is introduced. The SIMM loss function is based on the perceptual characteristics of the human visual system and primarily evaluates image quality from three aspects: brightness, contrast, and structure. Specifically:

[0084] L SIMM =α1SIMM(F vis ,F fuse )+α2SIMM(F ir ,F fuse (10)

[0085]

[0086] Where x and y are the two images to be compared, and μ x ,μ y σ represents the mean of the image x and y. x ,σ y σ represents the variance of the image x and y. xy The covariance of x and y is represented by C1 and C2, which are smoothing coefficients to prevent the denominator from being zero, and α1 and α2 are hyperparameters to balance the two source images.

[0087] Step 3.2: To address the mismatch between L2 loss and the image quality perceived by the human eye, VGG16 loss is introduced, such as... Figure 3 The VGG16 algorithm has four downsampling operations, which divide it into four layers. The shallow layers effectively extract texture details, while the deeper layers effectively extract semantic information about objects, vehicles, and targets. The initial visible light image and the fused image obtained in step 2 are input into the pre-trained VGG16 algorithm. The first layer is used for L1 loss to retain more detail. The initial infrared image and the fused image obtained in step 2 are input into the pre-trained VGG16 algorithm. The third layer is used for L1 loss to retain more saliency information. Specifically:

[0088]

[0089] in, This represents the output of the m-th layer of the VGG16 network, which is input as a fused image. This represents the output of the m-th layer of the VGG16 network when a visible light image is input; the default value is 1. This represents the output of the nth layer of the VGG16 network, which is the input of the fused image. This represents the output of the nth layer of the VGG16 network when an infrared image is input; the default value is 3. β1 and β2 are balancing parameters.

[0090] Step 3.3: To address the issue of insufficient foreground prominence in fused images and facilitate downstream advanced tasks, an object detection loss is introduced: A typical object detection network, YOLOv5, is represented as:

[0091]

[0092] st F fuse = Φ(F vis ,F ir ;ω f (14)

[0093] Among them, F fuse For the generated fused image, F ir Represents an infrared image, F vis This indicates that the light image ω can be represented f ω d Let Φ and ψ be the trainable parameters in the fusion network and the object detection network, respectively, and L be the parameters of the fusion network and the object detection network. d This is the target detection loss, and ω is optimized using the target detection loss. f ω d To obtain the optimal parameters;

[0094] Step 3.4, the total loss is expressed as:

[0095] L total =aL SIMM +bL v +cL d (15)

[0096] Where a, b, and c are equilibrium parameters, and L total L represents the total loss. SIMM Represents structural loss L v Indicates VGG16 loss, L d This indicates the target detection loss.

[0097] Step 4: Design an alternating cyclic learning strategy. After multiple alternating training sessions, a better YOLO v5 network is obtained. The object detection loss is used to guide the fusion network, and finally the optimal fusion network is obtained.

[0098] Step 4 is implemented in the following steps:

[0099] Step 4.1: In the first round of training, the fusion network from Step 1 is initially trained to provide better data for the subsequent training of the object detection network. VGG16 is used for gradient truncation and is not trained; formulas (10) and (12) are used to ensure rapid convergence of the fusion network training.

[0100] Step 4.2: In the subsequent training, the fusion network and the object detection network are trained simultaneously using formula (15), which includes object detection loss, to train the fusion network of step 1. Formula (13) is used to train the object detection network introduced in step 3.2, so as to obtain a better object detection network to guide the training of the fusion network.

[0101] Step 4.3: Through multiple iterations of training, step 4.1 executes 50 epochs, and step 4.2 executes 50 epochs of alternating training. Through multiple alternating training, the fusion network and the object detection network are optimized. Finally, the optimal fusion network is used to obtain the fused image.

[0102] Example 1

[0103] This invention presents a lightweight infrared and visible light image fusion method based on VGG16 and target detection, which is implemented according to the following steps:

[0104] Step 1: Design an infrared and visible light image fusion network based on DenseNet, where DenseNet uses grouped convolutions to reduce the number of parameters. DenseNet is used to obtain feature combinations at different levels of the image, and the final fused image is obtained through several convolutional layers.

[0105] Step 2: Generate the final image fusion result using an infrared and visible light image fusion network;

[0106] Step 3: Design loss functions using VGG16 and YOLO v5 networks, and train an infrared and visible light image fusion algorithm based on visual enhancement.

[0107] Step 4: Design an alternating cyclic learning strategy. After multiple alternating training sessions, a better YOLO v5 network is obtained. The object detection loss is used to guide the fusion network, and finally the optimal fusion network is obtained.

[0108] Example 2

[0109] This invention presents a lightweight infrared and visible light image fusion method based on VGG16 and target detection, which is implemented according to the following steps:

[0110] Step 1: Design an infrared and visible light image fusion network based on DenseNet, where DenseNet uses grouped convolutions to reduce the number of parameters. DenseNet is used to obtain feature combinations at different levels of the image, and the final fused image is obtained through several convolutional layers.

[0111] Step 1 is implemented in the following steps:

[0112] Visible light and infrared image fusion networks are based on the DenseNet structure; the mainstream approach uses the Y channel of visible light images as input.

[0113] Step 1.1: Convert the RGB channels of the visible light image to CbCr channels, and concatenate the Y channel of the visible light image with the infrared image along the channel dimension to form a two-dimensional feature map. This process is represented as follows:

[0114] F c =concat(F vis ,F ir (1)

[0115] Among them, F c This represents the feature map after stitching visible light and infrared light images. `concat(,)` indicates stitching along the channel dimension. F vis The Y channel of a visible light image, F ir Represents an infrared image;

[0116] Step 1.2, F c First, a convolution operation and a batch normalization operation are performed, followed by a ReLU activation function to obtain shallow features. This process can be represented as follows:

[0117] F1 = ReLU(BN(Conv(F)) c (2)

[0118] F1 is the shallow feature obtained, ReLU is the non-linear activation function, BN is the batch normalization check, and Conv is a convolution operation with kernel_size (3,3) and stride (1,1).

[0119] Step 1.3: After the initial feature extraction in Step 1.2, the data is processed through a 3-layer DenseNet network. The results from each layer are spliced ​​to subsequent layers to avoid information loss during convolution and to ensure that earlier shallow features are incorporated into later deep features, resulting in richer features. The F1 score obtained earlier is then spliced ​​to the next three layers along the channel dimension. The convolution operation is performed using grouped convolutions to reduce the number of parameters. The process is represented as follows:

[0120] F2=ReLU(BN(Conv(F1))) (3)

[0121] F3=ReLU(BN(Conv(concat(F1,F2))) (4)

[0122] F4=ReLU(BN(Conv(concat(F1,F2,F3))) (5)

[0123] Where Conv represents a grouped convolution with kernel size (3,3), stride (1,1), and group size of 64, F2, F3, and F4 are the outputs of each layer. F2, F3, and F4 all contain information from the features of the previous layer. F2 is obtained by convolution, normalization, and activation of the previous F1 layer; F3 is obtained by convolution, normalization, and activation of the previous F1 and F2 layers; and F4 is obtained by convolution, normalization, and activation of the previous F1, F2, and F3 layers, so that the final output can contain both shallow detailed information and deep semantic information.

[0124] Step 1.4: After obtaining the F4 matrix with rich features, the features need to be fused to achieve the best results. After inputting the F4 matrix into the base convolution, the last layer uses the Tanh activation function. The process is represented as follows:

[0125] F5 = ReLU(Conv(F4)) (6)

[0126] F6 = ReLU(BN(Conv(F5))) (7)

[0127] F7 = ReLU(BN(Conv(F6))) (8)

[0128] F fuse =Tanh(Conv(F7)) (9)

[0129] F5, F6, and F7 are the outputs of the intermediate layers. F5, F6, and F7 use grouped convolutions with a kernel size of (3,3), a stride of (1,1), and a group size of 64. fuseThe final result of the fusion of visible and infrared images uses a normal convolution with kernel_size of (3,3), stride of (1,1), and group size of 64.

[0130] Step 2: Generate the final image fusion result using an infrared and visible light image fusion network;

[0131] Step 2 is implemented in the following steps:

[0132] First, prepare training data for infrared and visible light images. Infrared light has only one channel. Convert the RGB channel of the visible light image to a CbCr channel. Then, stitch the Y channel of the visible light image with the infrared image in the channel dimension. Input the stitching result into the fusion network constructed in step 1 to obtain the fused image. Construct the loss function in step 3 using the fused image.

[0133] Step 3: Design loss functions using VGG16 and YOLO v5 networks, and train an infrared and visible light image fusion algorithm based on visual enhancement.

[0134] Step 3 is implemented in the following steps:

[0135] Step 3.1: To ensure structural similarity between the fused image and the source image, a structural similarity loss function (SIMM) is introduced. The SIMM loss function is based on the perceptual characteristics of the human visual system and primarily evaluates image quality from three aspects: brightness, contrast, and structure. Specifically:

[0136] L SIMM =α1SIMM(F vis ,F fuse )+α2SIMM(F ir ,F fuse (10)

[0137]

[0138] Where x and y are the two images to be compared, and μ x ,μ y σ represents the mean of the image x and y. x ,σ y σ represents the variance of the image x and y. xy The covariance of x and y is represented by C1 and C2, which are smoothing coefficients to prevent the denominator from being zero, and α1 and α2 are hyperparameters to balance the two source images.

[0139] Step 3.2: To address the mismatch between L2 loss and the image quality perceived by the human eye, VGG16 loss is introduced, such as... Figure 3The VGG16 algorithm has four downsampling operations, which divide it into four layers. The shallow layers effectively extract texture details, while the deeper layers effectively extract semantic information about objects, vehicles, and targets. The initial visible light image and the fused image obtained in step 2 are input into the pre-trained VGG16 algorithm. The first layer is used for L1 loss to retain more detail. The initial infrared image and the fused image obtained in step 2 are input into the pre-trained VGG16 algorithm. The third layer is used for L1 loss to retain more saliency information. Specifically:

[0140]

[0141] in, This represents the output of the m-th layer of the VGG16 network, which is input as a fused image. This represents the output of the m-th layer of the VGG16 network when a visible light image is input; the default value is 1. This represents the output of the nth layer of the VGG16 network, which is the input of the fused image. This represents the output of the nth layer of the VGG16 network when an infrared image is input; the default value is 3. β1 and β2 are balancing parameters.

[0142] Step 3.3: To address the issue of insufficient foreground prominence in fused images and facilitate downstream advanced tasks, an object detection loss is introduced: A typical object detection network, YOLOv5, is represented as:

[0143]

[0144] st F fuse = Φ(F vis ,F ir ;ω f (14)

[0145] Among them, F fuse For the generated fused image, F ir Represents an infrared image, F vis This indicates that the light image ω can be represented f ω d Let Φ and ψ be the trainable parameters in the fusion network and the object detection network, respectively, and L be the parameters of the fusion network and the object detection network. d This is the target detection loss, and ω is optimized using the target detection loss. f ω d To obtain the optimal parameters;

[0146] Step 3.4, the total loss is expressed as:

[0147] L total =aL SIMM +bL v +cLd (15)

[0148] Where a, b, and c are equilibrium parameters, and L total L represents the total loss. SIMM Represents structural loss L v Indicates VGG16 loss, L d This indicates the target detection loss.

[0149] Step 4: Design an alternating cyclic learning strategy. After multiple alternating training sessions, a better YOLO v5 network is obtained. The object detection loss is used to guide the fusion network. Finally, the optimal fusion network is used to obtain the fused image.

[0150] Example 3

[0151] This invention presents a lightweight infrared and visible light image fusion method based on VGG16 and target detection, which is implemented according to the following steps:

[0152] Step 1: Design an infrared and visible light image fusion network based on DenseNet, where DenseNet uses grouped convolutions to reduce the number of parameters. DenseNet is used to obtain feature combinations at different levels of the image, and the final fused image is obtained through several convolutional layers.

[0153] Step 1 is implemented in the following steps:

[0154] Visible light and infrared image fusion networks are based on the DenseNet structure; the mainstream approach uses the Y channel of visible light images as input.

[0155] Step 1.1: Convert the RGB channels of the visible light image to CbCr channels, and concatenate the Y channel of the visible light image with the infrared image along the channel dimension to form a two-dimensional feature map. This process is represented as follows:

[0156] F c =concat(F vis ,F ir (1)

[0157] Among them, F c This represents the feature map after stitching visible light and infrared light images. `concat(,)` indicates stitching along the channel dimension. F vis The Y channel of a visible light image, F ir Represents an infrared image;

[0158] Step 1.2, F c First, a convolution operation and a batch normalization operation are performed, followed by a ReLU activation function to obtain shallow features. This process can be represented as follows:

[0159] F1 = ReLU(BN(Conv(F)) c (2)

[0160] F1 is the shallow feature obtained, ReLU is the non-linear activation function, BN is the batch normalization check, and Conv is a convolution operation with kernel_size (3,3) and stride (1,1).

[0161] Step 1.3: After the initial feature extraction in Step 1.2, the data is processed through a 3-layer DenseNet network. The results from each layer are spliced ​​to subsequent layers to avoid information loss during convolution and to ensure that earlier shallow features are incorporated into later deep features, resulting in richer features. The F1 score obtained earlier is then spliced ​​to the next three layers along the channel dimension. The convolution operation is performed using grouped convolutions to reduce the number of parameters. The process is represented as follows:

[0162] F2=ReLU(BN(Conv(F1))) (3)

[0163] F3=ReLU(BN(Conv(concat(F1,F2))) (4)

[0164] F4=ReLU(BN(Conv(concat(F1,F2,F3))) (5)

[0165] Where Conv represents a grouped convolution with kernel size (3,3), stride (1,1), and group size of 64, F2, F3, and F4 are the outputs of each layer. F2, F3, and F4 all contain information from the features of the previous layer. F2 is obtained by convolution, normalization, and activation of the previous F1 layer; F3 is obtained by convolution, normalization, and activation of the previous F1 and F2 layers; and F4 is obtained by convolution, normalization, and activation of the previous F1, F2, and F3 layers, so that the final output can contain both shallow detailed information and deep semantic information.

[0166] Step 1.4: After obtaining the F4 matrix with rich features, the features need to be fused to achieve the best results. After inputting the F4 matrix into the base convolution, the last layer uses the Tanh activation function. The process is represented as follows:

[0167] F5 = ReLU(Conv(F4)) (6)

[0168] F6 = ReLU(BN(Conv(F5))) (7)

[0169] F7 = ReLU(BN(Conv(F6))) (8)

[0170] Ffuse =Tanh(Conv(F7)) (9)

[0171] F5, F6, and F7 are the outputs of the intermediate layers. F5, F6, and F7 use grouped convolutions with a kernel size of (3,3), a stride of (1,1), and a group size of 64. fuse The final result of the fusion of visible and infrared images uses a normal convolution with kernel_size of (3,3), stride of (1,1), and group size of 64.

[0172] Step 2: Generate the final image fusion result using an infrared and visible light image fusion network;

[0173] Step 2 is implemented in the following steps:

[0174] First, prepare training data for infrared and visible light images. Infrared light has only one channel. Convert the RGB channel of the visible light image to a CbCr channel. Then, stitch the Y channel of the visible light image with the infrared image in the channel dimension. Input the stitching result into the fusion network constructed in step 1 to obtain the fused image. Construct the loss function in step 3 using the fused image.

[0175] Step 3: Design loss functions using VGG16 and YOLO v5 networks, and train an infrared and visible light image fusion algorithm based on visual enhancement.

[0176] Step 4: Design an alternating cyclic learning strategy. After multiple alternating training sessions, a better YOLO v5 network is obtained. The object detection loss is used to guide the fusion network, and finally the optimal fusion network is obtained.

[0177] Step 4 is implemented in the following steps:

[0178] Step 4.1: In the first round of training, the fusion network from Step 1 is initially trained to provide better data for the subsequent training of the object detection network. VGG16 is used for gradient truncation and is not trained; formulas (10) and (12) are used to ensure rapid convergence of the fusion network training.

[0179] Step 4.2: In the subsequent training, the fusion network and the object detection network are trained simultaneously using formula (15), which includes object detection loss, to train the fusion network of step 1. Formula (13) is used to train the object detection network introduced in step 3.2, so as to obtain a better object detection network to guide the training of the fusion network.

[0180] Step 4.3: Through multiple iterations of training, step 4.1 executes 50 epochs, and step 4.2 executes 50 epochs of alternating training. Through multiple alternating training, the fusion network and the object detection network are optimized. Finally, the optimal fusion network is used to obtain the fused image.

[0181] like Figure 4 The image shown is an example of the fused image effect in different scenarios on the LLVIP dataset. Figure 4 This image shows the fusion results of a lightweight infrared and visible light image fusion method based on VGG16 and object detection in different scenes and times on the LLVIP dataset (which is mostly nighttime data). (b) shows the fusion result during the day, while the others show the results at night. It can be observed that the target information from the infrared image is well fused at night, while the texture details from the visible light image are well fused during the day.

Claims

1. A lightweight infrared and visible light image fusion method based on VGG16 and target detection, characterized in that, The specific steps are as follows: Step 1: Design an infrared and visible light image fusion network based on DenseNet; Step 1 is implemented in the following steps: Visible light and infrared image fusion networks are based on the DenseNet structure; the mainstream approach uses the Y channel of visible light images as input. Step 1.1: Convert the RGB channels of the visible light image to CbCr channels, and concatenate the Y channel of the visible light image with the infrared image in the channel dimension to form a two-dimensional feature map. This process is represented as follows: (1) in, This represents the feature map after stitching together visible light and infrared light images. This indicates that splicing is performed at the channel level. The Y channel represents a visible light image. Represents an infrared image; Step 1.2 First, a convolution operation and a batch normalization operation are performed, followed by a ReLU activation function to obtain shallow features. This process can be represented as follows: (2) These are the shallow features obtained. ReLU is a non-linear activation function, BN is a check for batch normalization, and Conv is a convolution operation with kernel_size (3,3) and stride (1,1). Step 1.3: After the initial feature extraction in Step 1.2, the data is processed through a 3-layer DenseNet network. The results from each layer are then spliced ​​together with the results from the next layer. This will be concatenated into the subsequent three layers of the network along the channel dimension. The convolution operation is performed by grouped convolution, and the process is represented as follows: (3) (4) (5) in, This represents a grouped convolution with kernel_size (3,3), stride (1,1), and group size of 64. For the output of each layer, They all contain information from the features of the preceding layer. From the front layer Obtained through convolution, normalization, and activation; From the front layer Obtained through convolution, normalization, and activation; From the front layer Convolution, normalization, and activation are used to obtain the final output, which contains both shallow detailed information and deep semantic information. Step 1.4: Obtaining data with rich features Afterwards, the features need to be fused to achieve the best results. After inputting the base layer convolutions, the last layer uses the Tanh activation function, as shown in the following process: (6) (7) (8) (9) in, For the output of the intermediate layer, We used a grouped convolution with kernel_size (3,3), stride (1,1), and group size of 64. The final result of the fusion of visible and infrared light images uses a normal convolution with kernel_size (3,3), stride (1,1), and group size of 64. Step 2: Generate the final image fusion result using an infrared and visible light image fusion network; Step 3: Design loss functions using VGG16 and YOLO v5 networks, and train an infrared and visible light image fusion algorithm based on visual enhancement. Step 3 is implemented in the following steps: Step 3.1: To ensure the structural similarity between the fused image and the source image, a structural similarity loss function is introduced. , The loss function, based on the perceptual characteristics of the human visual system, evaluates image quality from three aspects: brightness, contrast, and structure. Specifically: (10) (11) in, These are two images to be compared. Representing an image The mean, Representing an image variance express and covariance, and It is a smoothing coefficient to prevent the denominator from being 0. and It's a hyperparameter; Step 3.2: The VGG16 algorithm has four downsampling operations. Through these four downsampling operations, the VGG16 is divided into four layers. The shallow layers have a good extraction effect on texture detail features, while the deep layers have a good extraction effect on semantic information of the target person, vehicle, and target. The initial input visible light image and the fused image obtained in step 2 are input into the pre-trained VGG16, and the first layer is used for L1 loss. The initial input infrared light image and the fused image obtained in step 2 are input into the pre-trained VGG16, and the third layer is used for L1 loss. Specifically: (12) in, This represents the output of the m-th layer of the VGG16 network, which is input as a fused image. This represents the output of the m-th layer of a VGG16 network when a visible light image is input. This represents the output of the nth layer of the VGG16 network, which is the input of the fused image. This represents the output of the nth layer of the VGG16 network, which is input as an infrared image. , It is a balance parameter; Step 3.3, Introduce object detection loss: A typical object detection network, YOLOv5, is represented as: (13) (14) in, For the generated fused image, Represents an infrared image. This indicates that light images can be displayed. , To fuse trainable parameters from the network and the object detection network, , To integrate networks and target detection networks, This refers to the object detection loss, which is then optimized. , To obtain the optimal parameters; Step 3.4, the total loss is expressed as: (15) in, , , It is a balance parameter. Indicates the total loss. Indicates structural loss. This indicates a loss in VGG16. Indicates the target detection loss; Step 4: Design an alternating cyclic learning strategy. After multiple alternating training sessions, use object detection loss to guide the fusion network and finally obtain the optimal fused image.

2. The lightweight infrared and visible light image fusion method based on VGG16 and target detection according to claim 1, characterized in that, Step 2 is implemented in the following steps: First, prepare training data for infrared and visible light images. Infrared light has only one channel. Convert the RGB channel of the visible light image to a CbCr channel. Then, stitch the Y channel of the visible light image with the infrared image in the channel dimension. Input the stitching result into the fusion network constructed in step 1 to obtain the fused image. Construct the loss function in step 3 using the fused image.

3. The lightweight infrared and visible light image fusion method based on VGG16 and target detection according to claim 2, characterized in that, Step 4 is implemented in the following steps: Step 4.1 In the first round of training, the fusion network from Step 1 is initially trained. VGG16 is not trained because gradient truncation is performed. Formulas (10) and (12) are used to ensure the rapid convergence of the fusion network training. Step 4.2: In the subsequent training, the fusion network and the object detection network are trained simultaneously using formula (15), which includes object detection loss, to train the fusion network of step 1. Formula (13) is used to train the object detection network introduced in step 3.2, so as to obtain a better object detection network to guide the training of the fusion network. Step 4.3: Through multiple iterations of training, step 4.1 executes 50 epochs, and step 4.2 executes 50 epochs of alternating training. Through multiple alternating training, the fusion network and the object detection network reach their optimal values. Finally, the optimal fusion network is used to obtain the fused image.

Images