Lightweight infrared and visible light image fusion method based on vg16 and target detection
Through the lightweight infrared and visible image fusion method based on vgg16 and object detection, the DenseNet network and alternating loop learning strategy, combined with vgg16 and yolo v5 networks, the fusion image prospects in the prior art are not outstanding and do not conform to human eye perception. The problem of this achieves high-quality image fusion and object detection effects.
Patent Information
- Application Number
- CN202411032024.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-30
- Publication Date
- 2024-11-15
- Estimated Expiration
- 2044-07-30
AI Technical Summary
The fusion image generated by the existing infrared and visible light image fusion algorithm makes the foreground target not prominent, and the image does not conform to the human eye's perception, and the loss function used cannot simulate the image quality perceived by the human eye.
Using a lightweight infrared and visible image fusion method based on vgg16 and object detection, DenseNet network is designed for image fusion, and the fusion network is optimized through alternating loop learning strategies and object detection loss function, combining vgg16 and yolo v5 networks, using structural similarity Loss function and object detection loss function ensure the quality of the fused image and the outstandingness of the prospective target.
The effect of fusion images in advanced visual tasks is significantly improved, making the generated fusion images closer to human eye perception, the prospective goals are more prominent, and semantic information is enriched.
Smart Images

Figure CN118967475A_ABST
Abstract
Description
Technical Field
[0001] The invention belongs to the technical field of computer digital image processing, and in particular relates to a lightweight infrared and visible light image fusion method based on vgg16 and target detection. Background Art
[0002] The core goal of infrared and visible light image fusion technology is to generate high-quality composite images for downstream tasks such as target detection and image segmentation. However, the current infrared and visible light image fusion algorithms using deep learning ignore the impact of fused images on downstream tasks, and use simple loss functions that cannot simulate images perceived by the human eye. Specifically, the fused images generated by such image fusion algorithms often make foreground targets less prominent, and the L2 loss used does not match the image quality perceived by the human eye. In order to solve this problem, it is urgent to develop an innovative infrared and visible light image fusion algorithm to improve the quality and applicability of the generated images. Summary of the invention
[0003] The purpose of the present invention is to provide a lightweight infrared and visible light image fusion method based on vgg16 and target detection, which solves the problems in the prior art that the foreground target of the fused image is not prominent and the image does not conform to the perception of the human eye, and significantly improves the overall effect in advanced visual tasks.
[0004] The technical solution adopted by the present invention is a lightweight infrared and visible light image fusion method based on vgg16 and target detection, which is specifically implemented according to the following steps:
[0005] Step 1: Design an infrared and visible light image fusion network based on DenseNet;
[0006] Step 2: Generate the final image fusion result through the infrared and visible light image fusion network;
[0007] Step 3: Use the vgg16 network and yolo v5 network to design the loss function and train the infrared and visible light image fusion algorithm based on visual enhancement;
[0008] Step 4: Design an alternating cyclic learning strategy. After multiple alternating trainings, use the target detection loss to guide the fusion network and finally obtain the optimal fused image.
[0009] The present invention is also characterized in that:
[0010] Step 1 is implemented according to the following steps:
[0011] The visible light and infrared image fusion network is based on the DenseNet structure; the mainstream uses the Y channel of the visible light image as input;
[0012] Step 1.1, convert the RGB channel of the visible light image into the CbCr channel, and concatenate the Y channel of the visible light image with the infrared image in the channel dimension to form a feature map with a dimension of two. The process is expressed as:
[0013] F c =concat(F vis ,F ir ) (1)
[0014] Among them, F c represents the feature map after the visible light and infrared light images are spliced together, concat(,) represents splicing in the channel dimension, F vis Represents the Y channel of the visible light image, F ir Indicates infrared light image;
[0015] Step 1.2, F c First, after a convolution operation and batch normalization operation, and then through the ReLU activation function, the shallow features are obtained. The process is expressed as:
[0016] F1 = ReLU(BN(Conv(F c ))) (2)
[0017] F1 is the shallow feature obtained, ReLU is the nonlinear activation function, BN is the batch normalization check, Conv is the convolution operation with kernel_size (3,3) and stride (1,1);
[0018] Step 1.3: After the initial feature extraction in step 1.2, the result of each layer of the DenseNet network is jumped and spliced to the next layer of the network. The F1 obtained in the previous layer will be spliced to the next three layers of the network in the channel dimension. The convolution operation is performed by group convolution. The process is expressed as follows:
[0019] F2=ReLU(BN(Conv(F1))) (3)
[0020] F3=ReLU(BN(Conv(concat(F1,F2))) (4)
[0021] F4=ReLU(BN(Conv(concat(F1,F2,F3))) (5)
[0022] Among them, Conv represents the group convolution with kernel_size (3,3), stride (1,1) and group size 64. F2, F3, F4 are the outputs of each layer. F2, F3, F4 all contain the information of the previous layer features. F2 is obtained by convolution, normalization and activation of the previous layer F1; F3 is obtained by convolution, normalization and activation of the previous layers F1 and F2; F4 is obtained by convolution, normalization and activation of the previous layers F1, F2 and F3, so that the final output can contain both shallow detail information and deep semantic information.
[0023] Step 1.4: After obtaining F4 with rich features, it is necessary to fuse the features to achieve the optimal effect. After F4 is input into the base convolution, the last layer uses the Tanh activation function. The process is expressed as:
[0024] F5=ReLU(Conv(F4)) (6)
[0025] F6=ReLU(BN(Conv(F5))) (7)
[0026] F7=ReLU(BN(Conv(F6))) (8)
[0027] F fuse =Tanh(Conv(F7)) (9)
[0028] Among them, F5, F6, and F7 are the outputs of the middle layer. F5, F6, and F7 use group convolution with kernel_size (3, 3), step size (1, 1), and group size 64. fuse This is the final result of the fusion of visible light and infrared light images, which uses normal convolution with kernel_size of (3,3), step size of (1,1), and group size of 64.
[0029] Step 2 is implemented as follows:
[0030] First, prepare the training data of infrared light images and visible light. Infrared light has only one channel. Convert the RGB channel of the visible light image to the CbCr channel. Concatenate the Y channel of the visible light image with the infrared image in the channel dimension. Input the concatenated result into the fusion network constructed in step 1 to obtain the fused image. Construct the loss function of step 3 through the fused image.
[0031] Step 3 is implemented as follows:
[0032] Step 3.1: In order to ensure the structural similarity between the fused image and the source image, the structural similarity loss function SIMM is introduced. The SIMM loss function is based on the perceptual characteristics of the human visual system and mainly evaluates the image quality from three aspects: brightness, contrast and structure. Specifically:
[0033] L SIMM =α1SIMM(F vis ,F fuse )+α2SIMM(F ir ,F fuse ) (10)
[0034]
[0035] Among them, x, y are the two images to be compared, μ x ,μ y represents the mean of image x,y, σ x ,σ y represents the variance of image x,y, σ xy represents the covariance of x and y, C1 and C2 are smoothing coefficients to prevent the denominator from being 0, and α1 and α2 are hyperparameters;
[0036] Step 3.2, vgg16 has 4 downsampling operations. Through the 4 downsampling operations, vgg16 is divided into 4 layers, among which the shallow layer has a good extraction effect on texture detail features, and the deep layer has a good extraction effect on the semantic information of target people, vehicles, and targets. The pre-trained vgg16 is input for the visible light image initially input and the fused image obtained in step 2, and the first layer is taken for L1 loss. The pre-trained vgg16 is input for the infrared image initially input and the fused image obtained in step 2, and the third layer is taken for L1 loss. Specifically:
[0037]
[0038] in, Represents the output of the mth layer of the fused image input vgg16 network, Represents the output of the mth layer of the VGG16 network with visible light image input, Represents the output of the nth layer of the fused image input vgg16 network, represents the output of the nth layer of the VGG16 network with infrared image input, β1 and β2 are balance parameters;
[0039] Step 3.3, introduce target detection loss: the typical target detection network yolo v5 is expressed as:
[0040]
[0041] f fuse= Φ(F vis ,F ir ;ω f ) (14)
[0042] Among them, F fuse is the generated fusion image, F ir represents infrared image, F vis Indicates that the light image ω f ,ω d are the trainable parameters in the fusion network and the target detection network, Φ and ψ are the trainable parameters in the fusion network and the target detection network, and L d That is the target detection loss, and ω is optimized by target detection loss f ,ω d To obtain the optimal parameters;
[0043] Step 3.4, the total loss is expressed as:
[0044] L total =aL SIMM +bL v +cL d (15)
[0045] Among them, a, b, c are equilibrium parameters, L total The total loss L SIMM Indicates the structural loss L v Indicates vgg16 loss, L d represents the object detection loss.
[0046] Step 4 is implemented according to the following steps:
[0047] Step 4.1: In the first round of training, the fusion network of step 1 is preliminarily trained, where vgg16 performs gradient truncation and no training is performed. Formulas (10) and (12) are used to ensure the rapid convergence of the fusion network training.
[0048] Step 4.2: In the subsequent training, the fusion network and the target detection network are trained simultaneously, using formula (15) in which the target detection loss is added to train the fusion network of step 1, and using formula (13) to train the target detection network introduced in step 3.2, so as to obtain a better target detection network to guide the training of the fusion network;
[0049] Step 4.3: Perform 50 epochs of step 4.1 and 50 epochs of step 4.2 through multiple cycles of training. The fusion network and the target detection network are optimized through multiple alternating trainings. Finally, the optimal fusion network is used to obtain the fused image.
[0050] The beneficial effect of the present invention is that the lightweight infrared and visible light image fusion method based on vgg16 and target detection adopts an alternating learning strategy. Through multiple alternating trainings, the fusion network and the target detection network are successfully optimized, which is convenient for the target detection network to guide the fusion network to pay more attention to the foreground target. In the specific implementation, the algorithm adds group convolution in the DenseNet network to reduce the amount of data. The target detection loss is used to guide the fusion network to pay attention to the foreground target, so that the fused image is more prominent for the foreground target. The vgg16 loss is used to make the generated fusion image closer to the human eye perception and enrich the semantic information. BRIEF DESCRIPTION OF THE DRAWINGS
[0051] Figure 1 This is the overall design idea of the lightweight infrared and visible light image fusion method based on vgg16 and target detection in the present invention;
[0052] Figure 2 It is a schematic diagram of the framework of the lightweight infrared and visible light image fusion method based on vgg16 and target detection of the present invention;
[0053] Figure 3 It is a schematic diagram of VGG16 loss in the lightweight infrared and visible light image fusion method based on VGG16 and target detection of the present invention;
[0054] Figure 4 This is a fusion image effect diagram of the lightweight infrared and visible light image fusion method based on vgg16 and target detection in different scenes in the LLVIP dataset. DETAILED DESCRIPTION
[0055] The present invention is described in detail below with reference to the accompanying drawings and specific embodiments.
[0056] like Figure 1 As shown in the figure, first, the Y channel of the visible light image is extracted and the infrared light is spliced in dimension, and then it is input into the constructed DenseNet network, which can extract details at different levels and avoid information loss. Group convolution is used to solve the problem of excessive computation caused by too many feature maps. Secondly, vgg16 loss and target detection loss are constructed. The fused image generated by vgg16 loss is closer to human eye perception and enriches semantic information; target detection loss can make the fusion network focus on the foreground target, making the fused image more prominent for the foreground target. Finally, an alternating learning strategy is constructed to achieve the mutual promotion of the two networks of image fusion and target detection.
[0057] The lightweight infrared and visible light image fusion method based on vgg16 and target detection of the present invention is implemented in the following steps:
[0058] Step 1: Design an infrared and visible light image fusion network based on DenseNet, where DenseNet uses group convolution to reduce the number of parameters. DenseNet is used to obtain the feature combination of different levels of the image, and the final fused image is obtained through several layers of convolution.
[0059] Step 1 is implemented according to the following steps:
[0060] The visible light and infrared image fusion network is based on the DenseNet structure; the mainstream uses the Y channel of the visible light image as input;
[0061] Step 1.1, convert the RGB channel of the visible light image into the CbCr channel, and concatenate the Y channel of the visible light image with the infrared image in the channel dimension to form a feature map with a dimension of two. The process is expressed as:
[0062] F c =concat(F vis ,F ir ) (1)
[0063] Among them, F c represents the feature map after the visible light and infrared light images are spliced together, concat(,) represents splicing in the channel dimension, F vis Represents the Y channel of the visible light image, F ir Indicates infrared light image;
[0064] Step 1.2, F c First, after a convolution operation and batch normalization operation, and then through the ReLU activation function, the shallow features are obtained. The process is expressed as:
[0065] F1 = ReLU(BN(Conv(F c ))) (2)
[0066] F1 is the shallow feature obtained, ReLU is the nonlinear activation function, BN is the batch normalization check, Conv is the convolution operation with kernel_size (3,3) and stride (1,1);
[0067] Step 1.3: After the initial feature extraction in step 1.2, the result of each layer of the DenseNet network will be spliced to the next layer of the network to avoid information loss during the convolution process, and to enable the previous shallow features to be spliced to the next deep features, so that it has richer features. The F1 obtained in the previous layer will be spliced to the next three layers of the network in the channel dimension, where the convolution operation is performed by group convolution to reduce the number of parameters in turn. The process is expressed as:
[0068] F2=ReLU(BN(Conv(F1))) (3)
[0069] F3=ReLU(BN(Conv(concat(F1,F2))) (4)
[0070] F4=ReLU(BN(Conv(concat(F1,F2,F3))) (5)
[0071] Among them, Conv represents the group convolution with kernel_size (3,3), stride (1,1) and group size 64. F2, F3, F4 are the outputs of each layer. F2, F3, F4 all contain the information of the previous layer features. F2 is obtained by convolution, normalization and activation of the previous layer F1; F3 is obtained by convolution, normalization and activation of the previous layers F1 and F2; F4 is obtained by convolution, normalization and activation of the previous layers F1, F2 and F3, so that the final output can contain both shallow detail information and deep semantic information.
[0072] Step 1.4: After obtaining F4 with rich features, it is necessary to fuse the features to achieve the optimal effect. After F4 is input into the base convolution, the last layer uses the Tanh activation function. The process is expressed as:
[0073] F5=ReLU(Conv(F4)) (6)
[0074] F6=ReLU(BN(Conv(F5))) (7)
[0075] F7=ReLU(BN(Conv(F6))) (8)
[0076] F fuse =Tanh(Conv(F7)) (9)
[0077] Among them, F5, F6, and F7 are the outputs of the middle layer. F5, F6, and F7 use group convolution with kernel_size (3, 3), step size (1, 1), and group size 64. fuse This is the final result of the fusion of visible light and infrared light images, which uses normal convolution with kernel_size of (3,3), step size of (1,1), and group size of 64.
[0078] Step 2: Generate the final image fusion result through the infrared and visible light image fusion network;
[0079] Step 2 is implemented as follows:
[0080] First, prepare the training data of infrared light images and visible light. Infrared light has only one channel. Convert the RGB channel of the visible light image to the CbCr channel. Concatenate the Y channel of the visible light image with the infrared image in the channel dimension. Input the concatenated result into the fusion network constructed in step 1 to obtain the fused image. Construct the loss function of step 3 through the fused image.
[0081] Step 3: Use the vgg16 network and yolo v5 network to design the loss function and train the infrared and visible light image fusion algorithm based on visual enhancement;
[0082] Step 3 is implemented as follows:
[0083] Step 3.1: In order to ensure the structural similarity between the fused image and the source image, the structural similarity loss function SIMM is introduced. The SIMM loss function is based on the perceptual characteristics of the human visual system and mainly evaluates the image quality from three aspects: brightness, contrast and structure. Specifically:
[0084] L SIMM =α1SIMM(F vis ,F fuse )+α2SIMM(F ir ,F fuse ) (10)
[0085]
[0086] Among them, x, y are the two images to be compared, μ x ,μ y represents the mean of image x,y, σ x ,σ y represents the variance of image x,y, σ xy represents the covariance of x and y, C1 and C2 are smoothing coefficients to prevent the denominator from being 0, and α1 and α2 are hyperparameters to balance the two source images;
[0087] Step 3.2: In order to solve the problem that L2 loss does not match the image quality perceived by the human eye, vgg16 loss is introduced, such as Figure 3 . vgg16 has 4 downsampling operations. Through the 4 downsampling operations, vgg16 is divided into 4 layers. The shallow layer has a good extraction effect on texture detail features, and the deep layer has a good extraction effect on the semantic information of target people, vehicles, and targets. For the initial input visible light image and the fused image obtained in step 2, the pre-trained vgg16 is input, and the first layer is taken for L1 loss to retain more detail information; for the initial input infrared light image and the fused image obtained in step 2, the pre-trained vgg16 is input, and the third layer is taken for L1 loss to retain more significant information. Specifically:
[0088]
[0089] in, Represents the output of the mth layer of the fused image input vgg16 network, Represents the output of the mth layer of the VGG16 network for visible light image input. The default value is 1. Represents the output of the nth layer of the fused image input vgg16 network, Indicates the output of the nth layer of the VGG16 network with infrared image input. The default value is 3. β1 and β2 are balance parameters;
[0090] Step 3.3, in order to solve the problem that the foreground of the fused image is not prominent and facilitate downstream high-level tasks, the target detection loss is introduced: the typical target detection network YOLO v5 is expressed as:
[0091]
[0092] f fuse = Φ(F vis ,F ir ;ω f ) (14)
[0093] Among them, F fuse is the generated fusion image, F ir represents infrared image, F vis Indicates that the light image ω f ,ω d are the trainable parameters in the fusion network and the target detection network, Φ and ψ are the trainable parameters in the fusion network and the target detection network, and L d That is the target detection loss, and ω is optimized by target detection loss f ,ω d To obtain the optimal parameters;
[0094] Step 3.4, the total loss is expressed as:
[0095] L total =aL SIMM +bL v +cL d (15)
[0096] Among them, a, b, c are equilibrium parameters, L total The total loss L SIMM Indicates the structural loss L v Indicates vgg16 loss, L d represents the object detection loss.
[0097] Step 4: Design an alternating cyclic learning strategy. After multiple alternating trainings, a better YOLO V5 network is obtained. The target detection loss is used to guide the fusion network, and finally the optimal fusion network is obtained.
[0098] Step 4 is implemented according to the following steps:
[0099] Step 4.1: In the first round of training, the fusion network of step 1 is preliminarily trained to provide better data for the subsequent target detection network training. VGG16 performs gradient truncation and does not perform training. Formulas (10) and (12) are used to ensure the rapid convergence of the fusion network training.
[0100] Step 4.2: In the subsequent training, the fusion network and the target detection network are trained simultaneously, using formula (15) in which the target detection loss is added to train the fusion network of step 1, and using formula (13) to train the target detection network introduced in step 3.2, so as to obtain a better target detection network to guide the training of the fusion network;
[0101] Step 4.3: Perform 50 epochs of step 4.1 and 50 epochs of step 4.2 through multiple cycles of training. The fusion network and the target detection network are optimized through multiple alternating trainings. Finally, the optimal fusion network is used to obtain the fused image.
[0102] Example 1
[0103] The lightweight infrared and visible light image fusion method based on vgg16 and target detection of the present invention is implemented in the following steps:
[0104] Step 1: Design an infrared and visible light image fusion network based on DenseNet, where DenseNet uses group convolution to reduce the number of parameters. DenseNet is used to obtain the feature combination of different levels of the image, and the final fused image is obtained through several layers of convolution.
[0105] Step 2: Generate the final image fusion result through the infrared and visible light image fusion network;
[0106] Step 3: Use the vgg16 network and yolo v5 network to design the loss function and train the infrared and visible light image fusion algorithm based on visual enhancement;
[0107] Step 4: Design an alternating cyclic learning strategy. After multiple alternating trainings, a better YOLO V5 network is obtained. The target detection loss is used to guide the fusion network, and finally the optimal fusion network is obtained.
[0108] Example 2
[0109] The lightweight infrared and visible light image fusion method based on vgg16 and target detection of the present invention is implemented in the following steps:
[0110] Step 1: Design an infrared and visible light image fusion network based on DenseNet, where DenseNet uses group convolution to reduce the number of parameters. DenseNet is used to obtain the feature combination of different levels of the image, and the final fused image is obtained through several layers of convolution.
[0111] Step 1 is implemented according to the following steps:
[0112] The visible light and infrared image fusion network is based on the DenseNet structure; the mainstream uses the Y channel of the visible light image as input;
[0113] Step 1.1, convert the RGB channel of the visible light image into the CbCr channel, and concatenate the Y channel of the visible light image with the infrared image in the channel dimension to form a feature map with a dimension of two. The process is expressed as:
[0114] F c =concat(F vis ,F ir ) (1)
[0115] Among them, F c represents the feature map after the visible light and infrared light images are spliced together, concat(,) represents splicing in the channel dimension, F vis Represents the Y channel of the visible light image, F ir Indicates infrared light image;
[0116] Step 1.2, F c First, after a convolution operation and batch normalization operation, and then through the ReLU activation function, the shallow features are obtained. The process is expressed as:
[0117] F1 = ReLU(BN(Conv(F c ))) (2)
[0118] F1 is the shallow feature obtained, ReLU is the nonlinear activation function, BN is the batch normalization check, Conv is the convolution operation with kernel_size (3,3) and stride (1,1);
[0119] Step 1.3: After the initial feature extraction in step 1.2, the result of each layer of the DenseNet network will be spliced to the next layer of the network to avoid information loss during the convolution process, and to enable the previous shallow features to be spliced to the next deep features, so that it has richer features. The F1 obtained in the previous layer will be spliced to the next three layers of the network in the channel dimension, where the convolution operation is performed by group convolution to reduce the number of parameters in turn. The process is expressed as:
[0120] F2=ReLU(BN(Conv(F1))) (3)
[0121] F3=ReLU(BN(Conv(concat(F1,F2))) (4)
[0122] F4=ReLU(BN(Conv(concat(F1,F2,F3))) (5)
[0123] Among them, Conv represents the group convolution with kernel_size (3,3), stride (1,1) and group size 64. F2, F3, F4 are the outputs of each layer. F2, F3, F4 all contain the information of the previous layer features. F2 is obtained by convolution, normalization and activation of the previous layer F1; F3 is obtained by convolution, normalization and activation of the previous layers F1 and F2; F4 is obtained by convolution, normalization and activation of the previous layers F1, F2 and F3, so that the final output can contain both shallow detail information and deep semantic information.
[0124] Step 1.4: After obtaining F4 with rich features, it is necessary to fuse the features to achieve the optimal effect. After F4 is input into the base convolution, the last layer uses the Tanh activation function. The process is expressed as:
[0125] F5=ReLU(Conv(F4)) (6)
[0126] F6=ReLU(BN(Conv(F5))) (7)
[0127] F7=ReLU(BN(Conv(F6))) (8)
[0128] F fuse =Tanh(Conv(F7)) (9)
[0129] Among them, F5, F6, and F7 are the outputs of the middle layer. F5, F6, and F7 use group convolution with kernel_size (3, 3), step size (1, 1), and group size 64. fuseThis is the final result of the fusion of visible light and infrared light images, which uses normal convolution with kernel_size of (3,3), step size of (1,1), and group size of 64.
[0130] Step 2: Generate the final image fusion result through the infrared and visible light image fusion network;
[0131] Step 2 is implemented as follows:
[0132] First, prepare the training data of infrared light images and visible light. Infrared light has only one channel. Convert the RGB channel of the visible light image to the CbCr channel. Concatenate the Y channel of the visible light image with the infrared image in the channel dimension. Input the concatenated result into the fusion network constructed in step 1 to obtain the fused image. Construct the loss function of step 3 through the fused image.
[0133] Step 3: Use the vgg16 network and yolo v5 network to design the loss function and train the infrared and visible light image fusion algorithm based on visual enhancement;
[0134] Step 3 is implemented as follows:
[0135] Step 3.1: In order to ensure the structural similarity between the fused image and the source image, the structural similarity loss function SIMM is introduced. The SIMM loss function is based on the perceptual characteristics of the human visual system and mainly evaluates the image quality from three aspects: brightness, contrast and structure. Specifically:
[0136] L SIMM =α1SIMM(F vis ,F fuse )+α2SIMM(F ir ,F fuse ) (10)
[0137]
[0138] Among them, x, y are the two images to be compared, μ x ,μ y represents the mean of image x,y, σ x ,σ y represents the variance of image x,y, σ xy represents the covariance of x and y, C1 and C2 are smoothing coefficients to prevent the denominator from being 0, and α1 and α2 are hyperparameters to balance the two source images;
[0139] Step 3.2: In order to solve the problem that L2 loss does not match the image quality perceived by the human eye, vgg16 loss is introduced, such as Figure 3. vgg16 has 4 downsampling operations. Through the 4 downsampling operations, vgg16 is divided into 4 layers. The shallow layer has a good extraction effect on texture detail features, and the deep layer has a good extraction effect on the semantic information of target people, vehicles, and targets. For the initial input visible light image and the fused image obtained in step 2, the pre-trained vgg16 is input, and the first layer is taken for L1 loss to retain more detail information; for the initial input infrared light image and the fused image obtained in step 2, the pre-trained vgg16 is input, and the third layer is taken for L1 loss to retain more significant information. Specifically:
[0140]
[0141] in, Represents the output of the mth layer of the fused image input vgg16 network, Represents the output of the mth layer of the VGG16 network for visible light image input. The default value is 1. Represents the output of the nth layer of the fused image input vgg16 network, Indicates the output of the nth layer of the VGG16 network with infrared image input. The default value is 3. β1 and β2 are balance parameters;
[0142] Step 3.3, in order to solve the problem that the foreground of the fused image is not prominent and facilitate downstream high-level tasks, the target detection loss is introduced: the typical target detection network YOLO v5 is expressed as:
[0143]
[0144] f fuse = Φ(F vis ,F ir ;ω f ) (14)
[0145] Among them, F fuse is the generated fusion image, F ir represents infrared image, F vis Indicates that the light image ω f ,ω d are the trainable parameters in the fusion network and the target detection network, Φ and ψ are the trainable parameters in the fusion network and the target detection network, and L d That is the target detection loss, and ω is optimized by target detection loss f ,ω d To obtain the optimal parameters;
[0146] Step 3.4, the total loss is expressed as:
[0147] L total =aL SIMM +bL v +cLd (15)
[0148] Among them, a, b, c are equilibrium parameters, L total The total loss L SIMM Indicates the structural loss L v Indicates vgg16 loss, L d represents the object detection loss.
[0149] Step 4: Design an alternating cyclic learning strategy. After multiple alternating trainings, a better YOLO V5 network is obtained. The target detection loss is used to guide the fusion network. Finally, the optimal fusion network is used to obtain the fused image.
[0150] Example 3
[0151] The lightweight infrared and visible light image fusion method based on vgg16 and target detection of the present invention is implemented in the following steps:
[0152] Step 1: Design an infrared and visible light image fusion network based on DenseNet, where DenseNet uses group convolution to reduce the number of parameters. DenseNet is used to obtain the feature combination of different levels of the image, and the final fused image is obtained through several layers of convolution.
[0153] Step 1 is implemented according to the following steps:
[0154] The visible light and infrared image fusion network is based on the DenseNet structure; the mainstream uses the Y channel of the visible light image as input;
[0155] Step 1.1, convert the RGB channel of the visible light image into the CbCr channel, and concatenate the Y channel of the visible light image with the infrared image in the channel dimension to form a feature map with a dimension of two. The process is expressed as:
[0156] F c =concat(F vis ,F ir ) (1)
[0157] Among them, F c represents the feature map after the visible light and infrared light images are spliced together, concat(,) represents splicing in the channel dimension, F vis Represents the Y channel of the visible light image, F ir Indicates infrared light image;
[0158] Step 1.2, F c First, after a convolution operation and batch normalization operation, and then through the ReLU activation function, the shallow features are obtained. The process is expressed as:
[0159] F1 = ReLU(BN(Conv(F c ))) (2)
[0160] F1 is the shallow feature obtained, ReLU is the nonlinear activation function, BN is the batch normalization check, Conv is the convolution operation with kernel_size (3,3) and stride (1,1);
[0161] Step 1.3: After the initial feature extraction in step 1.2, the result of each layer of the DenseNet network will be spliced to the next layer of the network to avoid information loss during the convolution process, and to enable the previous shallow features to be spliced to the next deep features, so that it has richer features. The F1 obtained in the previous layer will be spliced to the next three layers of the network in the channel dimension, where the convolution operation is performed by group convolution to reduce the number of parameters in turn. The process is expressed as:
[0162] F2=ReLU(BN(Conv(F1))) (3)
[0163] F3=ReLU(BN(Conv(concat(F1,F2))) (4)
[0164] F4=ReLU(BN(Conv(concat(F1,F2,F3))) (5)
[0165] Among them, Conv represents the group convolution with kernel_size (3,3), stride (1,1) and group size 64. F2, F3, F4 are the outputs of each layer. F2, F3, F4 all contain the information of the previous layer features. F2 is obtained by convolution, normalization and activation of the previous layer F1; F3 is obtained by convolution, normalization and activation of the previous layers F1 and F2; F4 is obtained by convolution, normalization and activation of the previous layers F1, F2 and F3, so that the final output can contain both shallow detail information and deep semantic information.
[0166] Step 1.4: After obtaining F4 with rich features, it is necessary to fuse the features to achieve the optimal effect. After F4 is input into the base convolution, the last layer uses the Tanh activation function. The process is expressed as:
[0167] F5=ReLU(Conv(F4)) (6)
[0168] F6=ReLU(BN(Conv(F5))) (7)
[0169] F7=ReLU(BN(Conv(F6))) (8)
[0170] Ffuse =Tanh(Conv(F7)) (9)
[0171] Among them, F5, F6, and F7 are the outputs of the middle layer. F5, F6, and F7 use group convolution with kernel_size (3, 3), step size (1, 1), and group size 64. fuse This is the final result of the fusion of visible light and infrared light images, which uses normal convolution with kernel_size of (3,3), step size of (1,1), and group size of 64.
[0172] Step 2: Generate the final image fusion result through the infrared and visible light image fusion network;
[0173] Step 2 is implemented as follows:
[0174] First, prepare the training data of infrared light images and visible light. Infrared light has only one channel. Convert the RGB channel of the visible light image to the CbCr channel. Concatenate the Y channel of the visible light image with the infrared image in the channel dimension. Input the concatenated result into the fusion network constructed in step 1 to obtain the fused image. Construct the loss function of step 3 through the fused image.
[0175] Step 3: Use the vgg16 network and yolo v5 network to design the loss function and train the infrared and visible light image fusion algorithm based on visual enhancement;
[0176] Step 4: Design an alternating cyclic learning strategy. After multiple alternating trainings, a better YOLO V5 network is obtained. The target detection loss is used to guide the fusion network, and finally the optimal fusion network is obtained.
[0177] Step 4 is implemented according to the following steps:
[0178] Step 4.1: In the first round of training, the fusion network of step 1 is preliminarily trained to provide better data for the subsequent target detection network training. VGG16 performs gradient truncation and does not perform training. Formulas (10) and (12) are used to ensure the rapid convergence of the fusion network training.
[0179] Step 4.2: In the subsequent training, the fusion network and the target detection network are trained simultaneously, using formula (15) in which the target detection loss is added to train the fusion network of step 1, and using formula (13) to train the target detection network introduced in step 3.2, so as to obtain a better target detection network to guide the training of the fusion network;
[0180] Step 4.3: Perform 50 epochs of step 4.1 and 50 epochs of step 4.2 through multiple cycles of training. The fusion network and the target detection network are optimized through multiple alternating trainings. Finally, the optimal fusion network is used to obtain the fused image.
[0181] like Figure 4 Shown are the fusion image effects of the present invention on the LLVIP dataset in different scenes in the LLVIP dataset. Figure 4 This is a fusion image effect diagram of the lightweight infrared and visible light image fusion method based on vgg16 and target detection in different scenes and time in the LLVIP dataset (LLVIP dataset is mostly night data). Among them, (b) is the fusion result in daytime, and the others are the fusion results at night. It can be found that the night scene has a good fusion of the target information of the infrared image, and the daytime scene has a good fusion of the texture details of the visible light image.
Claims
1. A lightweight infrared and visible light image fusion method based on vgg16 and target detection, characterized in that: Follow the steps below to implement it: Step 1: Design an infrared and visible light image fusion network based on DenseNet; Step 2: Generate the final image fusion result through the infrared and visible light image fusion network; Step 3: Use the vgg16 network and yolo v5 network to design the loss function and train the infrared and visible light image fusion algorithm based on visual enhancement; Step 4: Design an alternating cyclic learning strategy. After multiple alternating trainings, use the target detection loss to guide the fusion network and finally obtain the optimal fused image.
2. The lightweight infrared and visible light image fusion method based on vgg16 and target detection according to claim 1 is characterized in that: The step 1 is specifically implemented according to the following steps: The visible light and infrared image fusion network is based on the DenseNet structure; the mainstream uses the Y channel of the visible light image as input; Step 1.1, convert the RGB channel of the visible light image into the CbCr channel, and concatenate the Y channel of the visible light image with the infrared image in the channel dimension to form a feature map with a dimension of two. The process is expressed as: F c =concat(F vis ,F ir ) (1) Among them, F c represents the feature map after the visible light and infrared light images are spliced together, concat(,) represents splicing in the channel dimension, F vis Represents the Y channel of the visible light image, F ir Indicates infrared light image; Step 1.2, F c First, after a convolution operation and batch normalization operation, and then through the ReLU activation function, the shallow features are obtained. The process is expressed as: F1=ReLU(BN(Conv(F c ))) (2) F1 is the shallow feature obtained, ReLU is the nonlinear activation function, BN is the batch normalization check, Conv is the convolution operation with kernel_size (3,3) and stride (1,1); Step 1.3: After the initial feature extraction in step 1.2, the result of each layer of the DenseNet network is jumped and spliced to the next layer of the network. The F1 obtained in the previous layer will be spliced to the next three layers of the network in the channel dimension. The convolution operation is performed by group convolution. The process is expressed as follows: F2=ReLU(BN(Conv(F1))) (3) F3=ReLU(BN(Conv(concat(F1,F2))) (4) F4=ReLU(BN(Conv(concat(F1,F2,F3))) (5) Among them, Conv represents the group convolution with kernel_size (3,3), stride (1,1) and group size 64. F2, F3, F4 are the outputs of each layer. F2, F3, F4 all contain the information of the previous layer features. F2 is obtained by convolution, normalization and activation of the previous layer F1; F3 is obtained by convolution, normalization and activation of the previous layers F1 and F2; F4 is obtained by convolution, normalization and activation of the previous layers F1, F2 and F3, so that the final output can contain both shallow detail information and deep semantic information. Step 1.4: After obtaining F4 with rich features, it is necessary to fuse the features to achieve the optimal effect. After F4 is input into the base convolution, the last layer uses the Tanh activation function. The process is expressed as: F5=ReLU(Conv(F4)) (6) F6=ReLU(BN(Conv(F5))) (7) F7=ReLU(BN(Conv(F6))) (8) F fuse =Tanh(Conv(F7)) (9) Among them, F5, F6, and F7 are the outputs of the middle layer. F5, F6, and F7 use group convolution with kernel_size (3, 3), step size (1, 1), and group size 64. fuse This is the final result of the fusion of visible light and infrared light images, which uses normal convolution with kernel_size of (3,3), step size of (1,1), and group size of 64.
3. The lightweight infrared and visible light image fusion method based on vgg16 and target detection according to claim 2 is characterized in that: The step 2 is specifically implemented according to the following steps: First, prepare the training data of infrared light images and visible light. Infrared light has only one channel. Convert the RGB channel of the visible light image to the CbCr channel. Concatenate the Y channel of the visible light image with the infrared image in the channel dimension. Input the concatenated result into the fusion network constructed in step 1 to obtain the fused image. Construct the loss function of step 3 through the fused image.
4. The lightweight infrared and visible light image fusion method based on vgg16 and target detection according to claim 3 is characterized in that: The step 3 is specifically implemented according to the following steps: Step 3.1: Evaluate image quality in three aspects: brightness, contrast and structure. Specifically: L SIMM =α1SIMM(F vis ,F fuse )+α2SIMM(F ir ,F fuse ) (10) Among them, x, y are the two images to be compared, μ x ,μ y represents the mean of image x,y, σ x ,σ y represents the variance of image x,y, σ xy represents the covariance of x and y, C1 and C2 are smoothing coefficients to prevent the denominator from being 0, and α1 and α2 are hyperparameters; Step 3.2, vgg16 has 4 downsampling operations. Through the 4 downsampling operations, vgg16 is divided into 4 layers, among which the shallow layer has a good extraction effect on texture detail features, and the deep layer has a good extraction effect on the semantic information of target people, vehicles, and targets. The pre-trained vgg16 is input for the visible light image initially input and the fused image obtained in step 2, and the first layer is taken for L1 loss. The pre-trained vgg16 is input for the infrared image initially input and the fused image obtained in step 2, and the third layer is taken for L1 loss. Specifically: in, Represents the output of the mth layer of the fused image input vgg16 network, Represents the output of the mth layer of the VGG16 network with visible light image input, Represents the output of the nth layer of the fused image input vgg16 network, represents the output of the nth layer of the VGG16 network with infrared image input, β1 and β2 are balance parameters; Step 3.3, introduce target detection loss: the typical target detection network yolo v5 is expressed as: s.t F fuse = Φ(F vis ,F ir ;ω f ) (14) Among them, F fuse is the generated fusion image, F ir represents infrared image, F vis Indicates that the light image ω f ,ω d are the trainable parameters in the fusion network and the target detection network, Φ and ψ are the trainable parameters in the fusion network and the target detection network, and L d That is the target detection loss, and ω is optimized by target detection loss f ,ω d To obtain the optimal parameters; Step 3.4, the total loss is expressed as: <h2 style=";text-align:left;direction:ltr">L<h2 style=";text-align:left;direction:ltr"> total <h2 style=";text-align:left;direction:ltr"> =aL<h2 style=";text-align:left;direction:ltr"> SIMM <h2 style=";text-align:left;direction:ltr"> +bL<h2 style=";text-align:left;direction:ltr"> v <h2 style=";text-align:left;direction:ltr"> +cL<h2 style=";text-align:left;direction:ltr"> d <h2 style=";text-align:left;direction:ltr"> (15) Among them, a, b, c are equilibrium parameters, L total The total loss L SIMM Indicates the structural loss L v Indicates vgg16 loss, L d represents the object detection loss.
5. The lightweight infrared and visible light image fusion method based on vgg16 and target detection according to claim 4 is characterized in that: The step 4 is specifically implemented according to the following steps: Step 4.1: In the first round of training, the fusion network of step 1 is preliminarily trained, where vgg16 performs gradient truncation and no training is performed. Formulas (10) and (12) are used to ensure the rapid convergence of the fusion network training. Step 4.2: In the subsequent training, the fusion network and the target detection network are trained simultaneously, using formula (15) in which the target detection loss is added to train the fusion network of step 1, and using formula (13) to train the target detection network introduced in step 3.2, so as to obtain a better target detection network to guide the training of the fusion network; Step 4.3: Perform 50 epochs of step 4.1 and 50 epochs of step 4.2 through multiple cycles of training. The fusion network and the target detection network are optimized through multiple alternating trainings. Finally, the optimal fusion network is used to obtain the fused image.
Citation Information
Patent Citations
Infrared light and visible light image fusion method based on residual dense network and gradient loss
CN113592018A
Target detection method based on task-driven infrared and visible light image fusion
CN117893421A
Infrared and visible light image fusion method based on vision enhancement in low-light environment
CN117952846A
Underground coal mine pedestrian detection method based on image fusion and feature enhancement
WO2024037408A1
