A multi-scale target detection method based on infrared and visible light

By combining multi-scale data enhancement and network structure design of infrared and visible images in the object detection algorithm, the problem of insufficient detection capabilities of small objects in complex scenarios is solved, and higher detection accuracy, adaptability and generalization capabilities are achieved. Improvement.

CN114332582BActive Publication Date: 2024-11-15中国人民解放军96901部队25分队 +1
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202111505346.3
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2021-12-10
Publication Date
2024-11-15
Estimated Expiration
2041-12-10

AI Technical Summary

Technical Problem

The existing object detection algorithm has weak detection capabilities for small objects in complex scenarios and is greatly affected by light changes and occlusion, making it difficult to effectively improve detection accuracy and accuracy, especially in applications combining infrared and visible light images.

Method used

Using multi-scale object detection method based on infrared and visible light, by combining similar and mixed data enhancement technologies, a multi-scale object detection network structure is designed, RFB structure and Gbneck module are added, upsampling operations are removed, and feature expression and detection head capabilities are enhanced. , the detection frame overlap redundancy is removed by using non-maximum suppression.

Benefits of technology

It significantly improves the detection accuracy and accuracy of object detection technology for complex scenes and small targets, improves the detection ability of infrared and visible images, and enhances the adaptability and generalization ability of the model.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN114332582B_ABST
    Figure CN114332582B_ABST
Patent Text Reader

Abstract

The present invention discloses a multi-scale target detection method based on infrared and visible light, collects infrared image and visible light image samples, performs calibration and data enhancement processing, forms training samples, constructs and uses the training samples to train a target detection network model, and then uses the trained target detection network model to detect the infrared image or visible light image to be detected, and outputs the detection result. The present invention utilizes the different characteristics of infrared and visible light images, and by designing a multi-scale target detection network structure and data augmentation and training strategy, it can greatly improve the detection precision and accuracy of target detection technology for complex scenes and small targets.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application belongs to the field of computer vision technology, and in particular, relates to a multi-scale target detection method based on infrared and visible light. Background Art

[0002] Object detection is an important research direction in the field of computer vision and is widely used in many fields, such as security monitoring, medical image analysis, and autonomous driving. With the rapid development of deep learning in recent years, object detection technology has also made major breakthroughs. For example, the emergence of algorithms such as Faster RCNN and the YOLO series has greatly improved the accuracy of object detection results. However, although these algorithm technologies have achieved good results in general object detection, in complex real-world application scenarios, the size of the target is often small, and will be affected by factors such as drastic changes in lighting, object occlusion, and changes in object scale. The detection rate of existing algorithm models will be greatly reduced, and the false alarm rate will increase.

[0003] At present, with the development of infrared technology, image acquisition equipment integrating infrared and visible light is becoming more and more popular. Target detection technology based on infrared and visible light can make use of the good adaptability of infrared images to weather and background and their night visibility. While compensating for the susceptibility of visible light images to interference from the external environment, it can also realize the detection of night targets.

[0004] Although some current technical solutions use the LSTM method to perform two rounds of up, down, left, and right four directions on the feature map output of the last layer of the deep residual convolutional neural network to obtain context information, and fuse it with the convolutional features to obtain multi-scale features for target detection, thus improving the accuracy of target detection, the existing technology does not use infrared and visible light image data, and there is still room for further improvement in network structure. Summary of the invention

[0005] The purpose of this application is to provide a multi-scale target detection method based on infrared and visible light, which uses infrared and visible light image data and combines the different characteristics of the two images to improve the adaptability of target detection technology to complex scenes. At the same time, the network structure is designed in a targeted manner to improve the algorithm model's ability to detect small targets.

[0006] In order to achieve the above purpose, the technical solution of this application is as follows:

[0007] A multi-scale target detection method based on infrared and visible light, comprising:

[0008] Collect infrared image and visible light image samples, perform calibration and data enhancement processing, and form training samples;

[0009] A target detection network model was constructed and trained using training samples. Based on YOLOV3, a group of RFB structures were added before the top convolution layer in the 8-fold downsampling and 16-fold downsampling layers in the YOLOV3 basic feature network Darknet-53. For the output branches y1, y2 and y3 of the three scales of YOLOV3, the upsampling operations used in the prediction layers of the y1 and y2 branches of the original YOLOV3 network were removed. In the feature map of the 4-fold downsampling, a Gb After downsampling by the neck module, it is concat-superimposed with the 8-fold downsampling, and then the number of channels is scaled by the Gbneck module with a step size of 1 as the final output of the y3 branch; at the same time, the Gbneck module with a step size of 2 is used to downsample the feature maps before the output of the y3 and y2 branches respectively, and then concat-connected with the original 16-fold downsampling and 32-fold downsampling in turn, and then the Gbneck module with a step size of 1 is used to scale the channels as the final output of the y2 and y1 branches respectively;

[0010] The infrared image or visible light image to be detected is detected using the trained target detection network model and the detection results are output.

[0011] Furthermore, the data enhancement processing includes:

[0012] The single-sample training data is processed using the same enhancement method;

[0013] The multi-sample training data are processed by mixed class enhancement, and new sample data are obtained by linear interpolation based on the neighborhood risk minimization principle.

[0014] Furthermore, the RFB structure performs the following operations:

[0015] The RFB structure first extracts features and compresses channels of the upper layer input through 1×1 convolution, 1×3 convolution and 3×3 convolution, and expands the feature map size through 3×3 dilated convolution. Then, the outputs of different branches are superimposed, added to the input feature map after passing through the 1×1 convolution layer, and finally passed through the Relu activation layer as the output of RFB.

[0016] Furthermore, after the detection is performed using the trained target detection network model and the detection result is output, the method further includes:

[0017] The non-maximum suppression method is used to remove the overlapping redundancy of detection boxes.

[0018] The present application proposes a multi-scale target detection method based on infrared and visible light, which utilizes the different characteristics of infrared and visible light images, and designs a multi-scale target detection network structure and data augmentation and training strategy, which can greatly improve the detection accuracy and precision of target detection technology for complex scenes and small targets. The present application collects a large number of infrared and visible light images, and uses the same and mixed data augmentation technology to augment the data set to improve the diversity of the data. A multi-scale target detection network structure model is designed, which adopts the RFB structure, multi-scale feature fusion design and lengthened feature backtracking detection head design, which can improve the detection effect of the algorithm model on small targets, and also has the target detection capability of infrared and visible light images. Improve the adaptability of target detection to complex scenes and the detection capability of small targets. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 This is a flow chart of the multi-scale target detection method based on infrared and visible light in this application;

[0020] Figure 2 This is a schematic diagram of the target detection network model structure of this application;

[0021] Figure 3 This is a schematic diagram of the RFB module structure of an embodiment of the present application. DETAILED DESCRIPTION

[0022] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0023] In view of the shortcomings of the existing technology in terms of weak detection capabilities in complex scenes and small targets, this application uses a camera device with infrared and visible light image acquisition functions to obtain infrared images that are more adaptable to weather and background and visible at night, and visible light images with richer scene information. The acquired infrared images and visible light images are augmented by the same and mixed enhancement methods, and a multi-scale network model is designed. The RFB (Receptive Field Block) structure is added to the basic network, the upsampling operation of the network is removed, and the detection head of the feature backtracking is added. Finally, the network model is trained by random multi-scale and dynamically adjusting the learning rate, thereby improving the detection accuracy and precision of the target detection algorithm in infrared and visible light scenes, and realizing all-weather target detection.

[0024] like Figure 1 As shown, the present application provides a multi-scale target detection method based on infrared and visible light, comprising:

[0025] Step S1: Collect infrared image and visible light image samples, perform calibration and data enhancement processing, and form training samples.

[0026] This application uses a camera device with infrared and visible light image acquisition functions to obtain infrared images that are more adaptable to weather and background and visible at night, and visible light images that have richer scene information.

[0027] Then, the collected infrared images and visible light images are calibrated and data enhanced to enrich the diversity of the training sample set. For example, the single-sample training data is processed using the same type of enhancement methods, including random geometric transformation, random color jitter and mosaic enhancement. Random geometric transformation methods include random cropping, random expansion, random horizontal flipping, and random scaling; random color jitter includes brightness, chroma, saturation, and contrast adjustment. Multi-sample training data is processed using mixed-class enhancement, and new sample data is obtained by linear interpolation based on the neighborhood risk minimization principle. A large number of practical verifications have shown that adding mixed-class enhancement can effectively improve the generalization ability and stability of the model.

[0028] Step S2, construct and use training samples to train the target detection network model. The target detection network model is based on YOLOV3. Before the top convolution layer in the 8-fold downsampling and 16-fold downsampling layers in the YOLOV3 basic feature network Darknet-53, a group of RFB structures are added respectively; for the output branches y1, y2 and y3 of the three scales of YOLOV3, the upsampling operation used in the prediction layer of the y1 and y2 branches of the original YOLOV3 network is removed; in the feature map of the 4-fold downsampling, a step size of 2 is added The Gbneck module with a step size of 2 is used to downsample the feature maps before the output of the y3 and y2 branches, and then they are concat-connected with the original 16-fold downsampling and 32-fold downsampling. Then, the Gbneck module with a step size of 1 is used to scale the channels as the final output of the y2 and y1 branches.

[0029] To ensure that the target detection algorithm can adapt to the complexity and diversity of application scenarios, this application specifically designs a deep learning network to enhance the ability to detect small targets, enhance the ability to detect multi-scale targets, and enhance the ability to express features. Specifically, this application first uses YOLOV3 as the prototype for network structure design, and makes improvements to enhance the ability to express features, small targets, and multi-scale detection. The overall network structure diagram is shown in the attached figure. Figure 2As shown in the figure, the design of three-scale feature output is retained. The output branches of the three scales are y1, y2 and y3, among which the y1 branch is mainly for the detection of large-scale targets, the y2 branch is mainly for the detection of medium-scale targets, and the y3 branch is mainly for the detection of small targets. The three-scale output structure is used to perform target detection on feature maps of different depths respectively, thereby enhancing the ability of multi-scale target detection.

[0030] The target detection network model of this application includes the following improvements:

[0031] 1) Add a group of RFB (Receptive Field Block) structures before the top convolutional layer in the 8x downsampling and 16x downsampling layers in the basic feature network Darknet-53 to expand the network structured receptive field and thus improve the network feature expression capability.

[0032] The RFB module structure diagram is as attached Figure 3 As shown in the figure, the RFB structure first extracts features and compresses channels of the upper layer input through 1×1 convolution, 1×3 convolution and 3×3 convolution, and expands the feature map size through 3×3 dilated convolution. Then the outputs of different branches are superimposed, added to the input feature map after passing through the 1×1 convolution layer, and finally used as the output of RFB after passing through the Relu activation layer. The features output by the RFB structure not only retain the characteristics of the previous layer features, but also can finely extract the features of the current layer, which helps to improve the detection rate of the target detection model.

[0033] 2) Remove the upsampling operation used in the y1 and y2 prediction layers of the original YOLOV3 network.

[0034] 3) In the feature map of the original YOLOV3 network downsampled by 4 times, a Gbneck module with a step size of 2 is added to downsample and parallelize with the original 8 times downsampling, and the two are concat-superimposed, and then the number of channels is scaled by the Gbneck module with a step size of 1 to improve the quality of high-dimensional features, and it is used as the final output of the y3 branch; at the same time, the Gbneck module with a step size of 2 is used to downsample the feature maps before the output of the y3 and y2 branches respectively, and then Concat-connected with the original 16 times downsampling and 32 times downsampling in turn, and then the Gbneck module with a step size of 1 is used to scale the channels, which are used as the final outputs of the y2 and y1 branches respectively.

[0035] The superposition of the Gbneck modules with different step sizes can effectively lengthen the feature backtracking of the detection head, improve the feature expression ability of the output feature maps of the y1, y2, and y3 prediction layers, thereby enhancing the accuracy of small target detection and improving the multi-scale detection capability of the network model.

[0036] The Gbneck module (Ghost bottleneck) is similar to the basic residual block in ResNet. The Gbneck module with a stride of 1 consists of two stacked Ghost modules. The first Ghost module is used as an expansion layer to increase the number of channels. The second Ghost module reduces the number of channels to match the shortcut path. Then, the shortcut is used to connect the input and output of the two Ghost modules. The second Ghost module in Ghost Bottleneck does not use ReLU, and other layers apply batch normalization (BN) and ReLu activation functions after each layer (mainly borrowing the idea of ​​MobileNetV2). In GhostBottleneck, for the case where the stride is equal to 2, the two Ghost modules are connected through a deep convolution with stride = 2. Regarding the Gbneck module, it is a relatively mature technology in this field and will not be repeated here.

[0037] The target detection network model constructed by training samples is trained, and the clustering calculation of Anchors is performed on the training data set by K-Means clustering method to obtain nine Anchors parameters of three different scales. In each training cycle, a scale is randomly selected for training, and the model is more robust to multiple scales by using different image size input training. At the same time, the training method of aliased images is added to speed up the convergence of the model. In addition, the training part adopts a general hyperparameter stage adjustment strategy. According to the different training stages corresponding to the number of training times, the size of parameters such as the learning rate is adjusted to make the network converge better.

[0038] It should be noted that the training of network models is already a relatively mature technology in this field and will not be elaborated here.

[0039] Step S3: Detect the infrared image or visible light image to be detected using the trained target detection network model and output the detection result.

[0040] For the infrared image or visible light image to be detected, first obtain the image to be detected or the real-time video stream, then load the target detection network model, crop the input image, adjust the image scale to 416*416 size and input it into the target detection network to obtain a large number of candidate boxes containing the target and their corresponding confidence levels.

[0041] Subsequently, the non-maximum suppression method can be used to remove the overlapping redundancy of the detection frames according to the size of the overlapping area between the target detection frames to obtain the final output result. The use of the non-maximum suppression method to obtain the final detection result is a relatively mature technical solution in this field and will not be described here.

[0042] The above-mentioned embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the patent of the present application shall be subject to the attached claims.

Claims

1. A multi-scale target detection method based on infrared and visible light, characterized in that: The multi-scale target detection method based on infrared and visible light includes: Collect infrared image and visible light image samples, perform calibration and data enhancement processing, and form training samples; A target detection network model was constructed and trained using training samples. Based on YOLOV3, a group of RFB structures were added before the top convolution layer in the 8-fold downsampling and 16-fold downsampling layers in the YOLOV3 basic feature network Darknet-53. For the output branches y1, y2 and y3 of the three scales of YOLOV3, the upsampling operations used in the prediction layers of the y1 and y2 branches of the original YOLOV3 network were removed. In the feature map of the 4-fold downsampling, a Gb After downsampling by the neck module, it is concat-superimposed with the 8-fold downsampling, and then the number of channels is scaled by the Gbneck module with a step size of 1 as the final output of the y3 branch; at the same time, the Gbneck module with a step size of 2 is used to downsample the feature maps before the output of the y3 and y2 branches respectively, and then concat-connected with the original 16-fold downsampling and 32-fold downsampling in turn, and then the Gbneck module with a step size of 1 is used to scale the channels as the final output of the y2 and y1 branches respectively; The infrared image or visible light image to be detected is detected using the trained target detection network model and the detection results are output.

2. The multi-scale target detection method based on infrared and visible light according to claim 1, characterized in that: The data enhancement processing includes: The single-sample training data is processed using the same enhancement method; The multi-sample training data are processed by mixed class enhancement, and new sample data are obtained by linear interpolation based on the neighborhood risk minimization principle.

3. The multi-scale target detection method based on infrared and visible light according to claim 1, characterized in that: The RFB structure performs the following operations: The RFB structure first extracts features and compresses channels of the upper layer input through 1×1 convolution, 1×3 convolution and 3×3 convolution, and expands the feature map size through 3×3 dilated convolution. Then, the outputs of different branches are superimposed, added to the input feature map after passing through the 1×1 convolution layer, and finally passed through the Relu activation layer as the output of RFB.

4. The multi-scale target detection method based on infrared and visible light according to claim 1, characterized in that: After the trained target detection network model is used to perform detection and output the detection result, the method further includes: The non-maximum suppression method is used to remove the overlapping redundancy of detection boxes.