Target detection method based on double adversarial learning visible light infrared fusion

Abstract

The invention provides a target detection method based on double adversarial learning visible light infrared fusion, and relates to the technical field of computer vision target detection. Comprising the following steps: acquiring and preprocessing an infrared-visible light image; an infrared-visible light fusion method based on double adversarial learning; the YOLOv5s is improved, and the model is optimized; evaluating an infrared-visible light fusion effect; compared with the prior art, the method realizes effective fusion and target detection of double optical modes, and has better target detection precision and robustness.

Description

Technical Field

[0001] The present invention relates to the technical field of computer vision target detection, and in particular to a target detection method based on dual adversarial learning visible light infrared fusion. Background Art

[0002] In the field of target detection, infrared images and visible light images are widely used in different application scenarios, such as military reconnaissance, security monitoring and unmanned driving. Infrared images have the advantage of good visibility in low-light and nighttime environments, while visible light images provide rich color and texture information. Infrared and visible light images have complementary characteristics, and fusing them can improve the accuracy and robustness of target detection.

[0003] Traditional infrared-visible light fusion methods are mainly based on image processing techniques, such as weighted fusion and transform domain fusion. However, these methods often rely on manually defined rules and feature extraction methods, making it difficult to fully utilize the information in the image. They are highly dependent on manual definition, the preliminary work is complicated, and they cannot effectively deal with unfamiliar images; they do not have autonomous learning functions and cannot fully utilize the complementary information of infrared and visible light images.

[0004] Therefore, there is an urgent need for a target detection method based on dual adversarial learning visible light infrared fusion to solve the above-mentioned problems. Summary of the invention

[0005] The purpose of the present invention is to provide a target detection method based on dual adversarial learning visible light infrared fusion, which realizes effective fusion and target detection of dual light modalities and has better target detection accuracy and robustness.

[0006] The embodiment of the present invention is achieved as follows:

[0007] The present application provides a target detection method based on dual adversarial learning visible light infrared fusion, which includes the following steps:

[0008] Step S1: construct visible light and infrared sequence image data sets respectively, and preprocess the images;

[0009] Step S2: Use TarDAL dual adversarial learning to generate a GAN adversarial network to fuse the visible light infrared image data set;

[0010] Step S3: construct an improved YOLOv5s visible light infrared image fusion target detection network model;

[0011] Step S4: training an improved YOLOv5s visible light infrared image fusion target detection network model;

[0012] Step S5: input the visible light and infrared sequence image data set constructed in step S1 into the visible light infrared image fusion target detection network of the improved YOLOv5s trained in step S3; test the detection performance of the visible light infrared image fusion target detection network based on the improved YOLOv5s;

[0013] Step S6: Evaluate the visible light infrared image fusion target detection network of the improved YOLOv5s and detect the detection effect of the ablation model.

[0014] In some embodiments of the present invention, the above step S1 specifically includes the following steps:

[0015] Step S1.1: Use an infrared-visible light camera with matching resolution and sensitivity wavelength range, and use a hardware trigger mechanism to trigger two cameras to collect images, and respectively construct a visible light and infrared sequence image dataset, and a visible light and infrared sequence image target detection and tracking dataset under ground or air background;

[0016] Step S1.2: The image data uses SIFT to detect key feature points in the image, and then performs feature point matching. After the matching is successful, the average H matrix calibration is used for preprocessing, and the convolutional neural network is used to perform image denoising;

[0017] Step S1.3: Construct the training set, validation set and test set of the visible light and infrared sequence image datasets, and randomly divide the selected images and labels in the ratio of 7:2:1. The obtained training set accounts for 70% of the total dataset, the test set accounts for 20% of the total dataset and the validation set accounts for 10% of the total dataset.

[0018] In some embodiments of the present invention, the above step S2 specifically includes the following steps:

[0019] Step S2.1: Extract the target perception features of infrared images and visible light images through the pre-trained target detection network, encourage the generator to deceive both discriminators and provide a realistic fused image; the generator network accepts the infrared image as input and generates a realistic visible light image;

[0020] Generator LSSIM loss function: L SSIM =(1-SSIM u，x ) / 2+(1-SSIM u，y ) / 2;

[0021] Target and detail discriminator LDT, LDD loss function:

[0022]

[0023] Step S2.2: The object discriminator evaluates the intensity consistency between the object extracted from the infrared image and the object in the fused image generated by the generator;

[0024] Step S2.3: The detail discriminator discriminates the gradient distribution between the visible light image and the fused image. The discriminator network receives the real visible light image and the generated visible light image as input and judges their differences;

[0025] Step S2.4: The generator network and the discriminator network are trained adversarially against each other.

[0026] In some embodiments of the present invention, the network structure of the improved YOLOv5s in the above step S3 consists of a backbone network and a feature pyramid network; the backbone network of the improved YOLOv5s usually adopts CSPDarknet53, which is used to extract image features, and the feature pyramid network consists of multiple feature layers of different scales, which are used to detect targets of different sizes; the improved YOLOv5s aims to improve the performance and accuracy of the network model, and the specific improvements include: using BotNet as the backbone network to replace the original backbone network, and adding a decoupled head structure: Decoupled.

[0027] In some embodiments of the present invention, in the above step 3, constraints are performed during the training process, and the pre-trained reference model obtains the relative confidence of the detection result; the relative confidence is determined as the ratio of the average confidence of a single poorly trained image in the parallel network to the average confidence of a single image in a pre-trained state.

[0028] In some embodiments of the present invention, the evaluation indicators of the model training in the above step S2 include mutual information, entropy, standard deviation, average precision mean and frame rate.

[0029] In some embodiments of the present invention, the evaluation indicators of the model training in the above step S4 include precision, recall, average precision and F1 score.

[0030] In some embodiments of the present invention, the above-mentioned decoupling head structure separates the prediction tasks of position and category and constructs two decoupling heads respectively.

[0031] Compared with the prior art, the embodiments of the present invention have at least the following advantages or beneficial effects:

[0032] The present invention provides a target detection method based on dual adversarial learning visible light infrared fusion. By using TarDAL to fuse infrared-visible light images and using improved YOLOv5s for target detection, the present invention realizes effective fusion and target detection of dual light modalities, and the effect of target detection after using TarDAL image fusion of the present invention under the practical MSRS data set is significantly improved. As shown in Table 1, it is observed that MI is improved to 3.21, EN is improved to 7.31, and SD is improved to 45.82, which proves the improvement of image quality and the effectiveness of target detection by dual light fusion of the present invention. At the same time, mAP@0.5 is improved to 76.25% and FPS is improved to 78, which further verifies the performance advantage of the method of the present invention. The accuracy of the fused image is significantly better than that of the single modality image, indicating that the target detection method based on dual adversarial learning visible light infrared fusion adopted by the present invention has better target detection accuracy and robustness. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for use in the embodiments are briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present invention and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without creative work.

[0034] Figure 1 is a flowchart of steps of an embodiment of the present invention;

[0035] Figure 2 The improved YOLOv5s network structure adopted in the embodiment of the present invention;

[0036] Figure 3 It is a Decoupled decoupling head according to an embodiment of the present invention;

[0037] Figure 4 It is the BotNet backbone network used in the target detection method of the embodiment of the present invention;

[0038] Figure 5 This is a comparison diagram of Map@0.5 changes before and after the infrared-visible light fusion effect of an embodiment of the present invention;

[0039] Figure 6 The figure is a comparison diagram of parameter changes of different models before and after improvement of the embodiment of the present invention. DETAILED DESCRIPTION

[0040] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Generally, the components of the embodiments of the present invention described and shown in the drawings here can be arranged and designed in various different configurations.

[0041] Therefore, the following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the invention claimed for protection, but merely represents selected embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0042] Example:

[0043] Please refer to Figure 1-6 This embodiment provides a target detection method based on dual adversarial learning visible light infrared fusion, which includes the following steps:

[0044] Step S1: Use an infrared-visible light camera with matching resolution and sensitivity wavelength range, and use a hardware trigger mechanism to trigger the two cameras to collect images, and respectively construct visible light and infrared sequence image data sets, and visible light and infrared sequence image target detection and tracking data sets under ground or air backgrounds; the image data uses SIFT to detect key feature points in the image, and then performs feature point matching. After the matching is successful, the average H matrix calibration is used for preprocessing. This calibration and alignment can ensure that the two images can be correctly registered and fused in the subsequent processing steps in infrared-visible light fusion, thereby improving the accuracy and stability of target detection and visual perception, and then a convolutional neural network (CNN) is used to perform image denoising to make the target clearer and more visible.

[0045] Step S2: using the TarDAL generative adversarial network (GAN) method to fuse the visible light infrared image dataset;

[0046] The application of TarDAL in infrared-visible light fusion is mainly used to improve the performance and robustness of target detection tasks. Through the TarDAL method, infrared images can be converted into realistic visible light images and fused with real visible light images to obtain more information-rich fused images. In order to more intuitively verify the advantages of image fusion images, target detection verification was performed on visible light, infrared, and fused images in two different scenes. Scene a is a nighttime urban street environment, and scene b is a nighttime urban road environment. Six groups of images of the two scenes were compared. The detection results show that the fused image has higher detection accuracy, enhanced feature extraction capabilities, and improved detection in different scenes.

[0047] Step S3: construct an improved YOLOv5s visible light infrared image fusion target detection network model;

[0048] YOLOv5s adopts a single-stage target detection method, which achieves fast and efficient target detection by directly predicting bounding boxes and categories on an image. The core principle of YOLOv5s is to divide the input image into different grids and predict a set of bounding boxes on each grid. Each bounding box contains the location and category information of the target. Compared with the traditional two-stage detection method, YOLOv5s can achieve real-time detection and has good performance in detection accuracy and speed. The network structure of YOLOv5s consists of a backbone network and a feature pyramid network. The backbone network usually uses CSPDarknet53 to extract image features. The feature pyramid network consists of multiple feature layers of different scales for detecting targets of different sizes. YOLOv5s also adopts some improvement strategies, such as data enhancement, loss function design and model lightweight, to improve the detection performance and efficiency of the model.

[0049] Step S4: training an improved YOLOv5s visible light infrared image fusion target detection network model;

[0050] The training and verification of the present invention uses a new multispectral data set of infrared and visible light image fusion provided by MSRS. The database uses People, Car, Bus, Motorcycle, Lamp, and Truck as labels for pedestrians, cars, buses, cyclists, bicycles, and trucks, respectively. There are 4200 infrared and visible light images, and the fused images are used for target detection. The target detection verification training set is 4200, the verification set is 500, and the test set is 280. Some important training parameters of the fusion network model are as follows: the training round is 200, the batch size is 4, the image size is 640*512, the feature basic size is 32, and the depth is 3. Some important training parameters of the target detection network model are set as follows: the training round is 200, the batch size is 16, the initial learning rate is 0.01, the periodic learning rate is 0.01, and the weight decay coefficient is 0.0005.

[0051] Step S5: input the visible light and infrared sequence image data set constructed in step 1 into the visible light infrared image fusion target detection network of the improved YOLOv5s trained in step 3, and test the detection performance of the visible light infrared image fusion target detection network based on the improved YOLOv5s;

[0052] The present invention uses a data set containing infrared images and visible light images for experimental evaluation, and inputs the fused image into the improved YOLOv5s target detector for target detection. The evaluation indicators include mutual information: MI: mutual information, entropy: EN, standard deviation: SD, mean average precision: mAP@0.5, frame rate: FPS. The experimental results show that the quality of the fused image is significantly improved, and the infrared-visible light fusion method based on dual adversarial learning has achieved significant performance improvement in the target detection task. Compared with the single-modal target detection method, the fusion method of the present invention can detect the target object more accurately and provide more stable detection results.

[0053] Step S6: Evaluate the detection effect of the improved YOLOv5s visible light infrared image fusion target detection network detection ablation model.

[0054] The present invention provides a target detection method based on dual adversarial learning visible light infrared fusion. By using TarDAL to fuse infrared-visible light images and using improved YOLOv5s for target detection, the present invention realizes effective fusion and target detection of dual light modalities, and the effect of target detection after using TarDAL image fusion of the present invention under the practical MSRS data set is significantly improved. As shown in Table 1, it is observed that MI is improved to 3.21, EN is improved to 7.31, and SD is improved to 45.82, which proves the effectiveness of the dual light fusion of the present invention in improving image quality and target detection. At the same time, mAP@0.5 is improved to 76.25% and FPS is improved to 78.

[0055] Table 1 Infrared-visible light fusion effect of the present invention

[0056] Dataset MI EN SD mAP@0.5 FPS IR 2.82 6.98 39.36 74.38 95 VI 2.81 7.09 41.26 75.73 71 IR-VI 3.21 7.31 45.82 76.25 78

[0057] Table 2 Comparison of infrared-visible light fusion effects

[0058] Dataset MI EN SD mAP@0.5 FPS IR 2.82 6.98 39.36 74.38 95 VI 2.81 7.09 41.26 75.73 71 IR-VI 3.21 7.31 45.82 76.25 78

[0059] Table 3 Comparison of target detection ablation experiment results

[0060] Model mAP@0.5 Precision Recall YOLOv5s 76.25 87.16 67.59 YOLOv5s-bot 84.59 91.12 76.70 YOLOv5s-BD 87.56 91.11 79.13

[0061] In some embodiments of the present invention, the above step S1 specifically includes the following steps:

[0062] Step S1.1: Use an infrared-visible light camera with matching resolution and sensitivity wavelength range, and use a hardware trigger mechanism to trigger two cameras to collect images, and respectively construct a visible light and infrared sequence image dataset, and a visible light and infrared sequence image target detection and tracking dataset under ground or air background;

[0063] Step S1.2: The image data uses SIFT to detect key feature points in the image, and then performs feature point matching. After the matching is successful, the average H matrix calibration is used for preprocessing, and the convolutional neural network is used to perform image denoising;

[0064] Step S1.3: Construct the training set, validation set and test set of the visible light and infrared sequence image datasets, and randomly divide the selected images and labels in the ratio of 7:2:1. The obtained training set accounts for 70% of the total dataset, the test set accounts for 20% of the total dataset and the validation set accounts for 10% of the total dataset.

[0065] In some embodiments of the present invention, the above step S2 specifically includes the following steps:

[0066] Step S2.1: Extract the target perception features of infrared images and visible light images through the pre-trained target detection network, encourage the generator to deceive both discriminators and provide a realistic fused image; the generator network accepts the infrared image as input and generates a realistic visible light image;

[0067] Generator LSSIM loss function: L SSIM =(1-SSIM u，x ) / 2+(1-SSIM u，y ) / 2;

[0068] Target and detail discriminator LDT, LDD loss function:

[0069]

[0070] Step S2.2: The object discriminator evaluates the intensity consistency between the object extracted from the infrared image and the object in the fused image generated by the generator;

[0071] Step S2.3: The detail discriminator discriminates the gradient distribution between the visible light image and the fused image. The discriminator network receives the real visible light image and the generated visible light image as input and judges their differences;

[0072] Step S2.4: The generator network and the discriminator network are trained adversarially against each other.

[0073] In the above technical solution, the TarDAL generative adversarial network (GAN) model is trained in step S2, and the evaluation indicators of model training are mainly analyzed from the following aspects: mutual information, entropy, standard deviation, mean average precision, and frame rate. MI: Mutual Information Mutual information is a concept used to measure the mutual dependence between two random variables. Entropy (EN): In information theory, entropy is a measure of the uncertainty of a random variable, which is a measure of information distribution. Standard deviation (SD): The standard deviation is a measure of the degree of discreteness of a data set. Mean average precision (mAP@0.5): Mean average precision is one of the indicators used to evaluate the performance of target detection algorithms. Frame rate (FPS): The frame rate is the number of frames displayed per second in a video, which is usually used to measure the speed of video playback or processing.

[0074] In some embodiments of the present invention, the network structure of the improved YOLOv5s in the above step S3 consists of a backbone network and a feature pyramid network; the backbone network of the improved YOLOv5s usually adopts CSPDarknet53, which is used to extract image features, and the feature pyramid network consists of multiple feature layers of different scales, which are used to detect targets of different sizes; the improved YOLOv5s aims to improve the performance and accuracy of the network model, and the specific improvements include: using BotNet as the backbone network to replace the original backbone network, and adding a decoupled head structure: Decoupled.

[0075] In the above technical solution, the evaluation indicators of the training of the visible light infrared image fusion target detection network model of the improved YOLOv5s in step 4 are mainly analyzed from the following aspects: Precision: refers to the ratio of the number of samples predicted as positive examples to the number of all predicted positive examples, Recall: refers to the ratio of the number of samples predicted as positive examples to the number of all truly positive examples, Average Precision: is an indicator obtained by averaging the AP of all categories, F1 score: is an indicator to measure the balance between the accuracy and recall of the model, the higher the F1 score, the better the overall performance of the model.

[0076] The present invention decouples the detection head of the YOLOv5s model. The decoupled head can accelerate the convergence speed of the network and improve the average precision mean of the detection. The Cls branch and Reg branch obtained after decoupling can be used for further feature information screening modules and inter-frame information link modules. The improved YOLOv5s decoupled head structure Decoupled in this project separates the prediction tasks of position and category, and constructs two decoupled heads respectively. This decoupled design enables the model to independently optimize the prediction tasks of position and category, thereby improving the accuracy and stability of target detection. The Decoupled head structure introduces more convolution and pooling layers to extract richer feature information, and adopts appropriate activation functions and loss functions for the prediction of position and category, respectively, so that the model can better understand the position and category information of the target.

[0077] Compared with the mainstream target detection network, the target detection method based on dual adversarial learning visible light infrared fusion provided by the present invention realizes effective fusion and target detection of dual light modalities by using TarDAL for infrared-visible light image fusion and improved YOLOv5s for target detection. It is observed that the effect of target detection using TarDAL image fusion under the MSRS dataset is significantly improved, and the warning rate is also reduced, reflecting good robustness and adaptability.

[0078] In some embodiments of the present invention, in the above step 3, constraints are performed during the training process, and the pre-trained reference model obtains the relative confidence of the detection result; the relative confidence is determined as the ratio of the average confidence of a single poorly trained image in the parallel network to the average confidence of a single image in a pre-trained state.

[0079] In some embodiments of the present invention, the evaluation indicators of the model training in step S2 include mutual information, entropy, standard deviation, average precision mean and frame rate. The evaluation indicators of the model training in step S4 include precision, recall, average precision and F1 score. The decoupling head structure separates the prediction tasks of position and category, and constructs two decoupling heads respectively.

[0080] In summary, the embodiments of the present invention provide a target detection method based on dual adversarial learning visible light infrared fusion. By using TarDAL to fuse infrared-visible light images and using improved YOLOv5s for target detection, the present invention achieves effective fusion and target detection of dual light modalities, and the effect of target detection after using TarDAL image fusion of the present invention under the practical MSRS data set is significantly improved. As shown in Table 1, it is observed that MI is improved to 3.21, EN is improved to 7.31, and SD is improved to 45.82, which proves the improvement of image quality and the effectiveness of target detection by dual light fusion of the present invention. At the same time, mAP@0.5 is improved to 76.25% and FPS is improved to 78, which further verifies the performance advantage of the method of the present invention. The accuracy of the fused image is significantly better than that of the single modality image, indicating that the target detection method based on dual adversarial learning visible light infrared fusion adopted by the present invention has better target detection accuracy and robustness.

[0081] The above are only preferred embodiments of the present invention and are not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A target detection method based on dual adversarial learning visible light infrared fusion, characterized in that: The following steps are involved: Step S1: construct visible light and infrared sequence image data sets respectively, and preprocess the images; Step S2: Use TarDAL dual adversarial learning to generate a GAN adversarial network to fuse the visible light infrared image data set; Step S3: construct an improved YOLOv5s visible light infrared image fusion target detection network model; Step S4: training an improved YOLOv5s visible light infrared image fusion target detection network model; Step S5: input the visible light and infrared sequence image datasets constructed in step S1 into the visible light infrared image fusion target detection network of the improved YOLOv5s trained in step S3; Test the detection performance of the visible light infrared image fusion target detection network based on the improved YOLOv5s; Step S6: Evaluate the visible light infrared image fusion target detection network of the improved YOLOv5s and detect the detection effect of the ablation model.

2. The target detection method based on dual adversarial learning visible light infrared fusion according to claim 1 is characterized in that: The step S1 specifically includes the following steps: Step S1.1: Use an infrared-visible light camera with matching resolution and sensitivity wavelength range, and use a hardware trigger mechanism to trigger two cameras to collect images, and respectively construct a visible light and infrared sequence image dataset, and a visible light and infrared sequence image target detection and tracking dataset under ground or air background; Step S1.2: The image data uses SIFT to detect key feature points in the image, and then performs feature point matching. After the matching is successful, the average H matrix calibration is used for preprocessing, and the convolutional neural network is used to perform image denoising; Step S1.3: Construct the training set, validation set and test set of the visible light and infrared sequence image datasets, and randomly divide the selected images and labels in the ratio of 7:2:

1. The obtained training set accounts for 70% of the total dataset, the test set accounts for 20% of the total dataset and the validation set accounts for 10% of the total dataset.

3. The target detection method based on dual adversarial learning visible light infrared fusion according to claim 2 is characterized in that: The step S2 specifically includes the following steps: Step S2.1: Extract the target perception features of infrared images and visible light images through the pre-trained target detection network, encourage the generator to deceive both discriminators and provide a realistic fused image; the generator network accepts the infrared image as input and generates a realistic visible light image; Generator LSSIM loss function: L SSIM =(1-SSIM u，x ) / 2+(1-SSIM u，y ) / 2; Target and detail discriminator LDT, LDD loss function: Step S2.2: The object discriminator evaluates the intensity consistency between the object extracted from the infrared image and the object in the fused image generated by the generator; Step S2.3: The detail discriminator discriminates the gradient distribution between the visible light image and the fused image. The discriminator network receives the real visible light image and the generated visible light image as input and judges their differences; Step S2.4: The generator network and the discriminator network are trained adversarially against each other.

4. The target detection method based on dual adversarial learning visible light infrared fusion according to claim 3 is characterized in that: The network structure of the improved YOLOv5s in step S3 consists of a backbone network and a feature pyramid network; the backbone network of the improved YOLOv5s usually adopts CSPDarknet53, which is used to extract image features, and the feature pyramid network consists of multiple feature layers of different scales, which are used to detect targets of different sizes; the improved YOLOv5s aims to improve the performance and accuracy of the network model, and the specific improvements include using BotNet as the backbone network to replace the original backbone network, and adding a decoupled head structure: Decoupled.

5. The target detection method based on dual adversarial learning visible light infrared fusion according to claim 4 is characterized in that: In step 3, constraints are imposed during the training process, and the pre-trained reference model obtains the relative confidence of the detection result; the relative confidence is determined as the ratio of the average confidence of a single poorly trained image in the parallel network to the average confidence of a single image in a pre-trained state.

6. The target detection method based on dual adversarial learning visible light infrared fusion according to claim 5 is characterized in that: The evaluation indicators of the model training in step S2 include mutual information, entropy, standard deviation, average precision mean and frame rate.

7. The target detection method based on dual adversarial learning visible light infrared fusion according to claim 6 is characterized in that: The evaluation indicators of the model training in step S4 include precision, recall, average precision and F1 score.

8. The target detection method based on dual adversarial learning visible light infrared fusion according to claim 7 is characterized in that: The decoupled head structure separates the prediction tasks of position and category and constructs two decoupled heads respectively.

Images