A method for infrared image target detection based on bounding box regression

Through adaptive CLAHE image optimization and R-YOLOv4 network design, the problems of poor contrast in infrared image and inaccurate bounding box regression are solved, and the accuracy and flexibility of infrared image object detection are improved, especially in complex scenarios. The accuracy of bounding box regression.

CN115082672BActive Publication Date: 2024-11-15XIDIAN UNIV
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202210630721.5
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2022-06-06
Publication Date
2024-11-15
Estimated Expiration
2042-06-06

AI Technical Summary

Technical Problem

The existing infrared image object detection technology has problems such as poor infrared image contrast, low feature extraction capability of object detection networks, and inaccurate bounding box regression. It is especially difficult to achieve different levels of target bounding box accurate regression in complex road scenarios.

Method used

Adaptive CLAHE image optimization algorithm is used to improve infrared image contrast, and enhance receptive field by adding RFB modules on shallow networks, and adjust the regression accuracy based on α-IoU, and build an R-YOLOv4 object detection network to improve small Object detection accuracy and bounding box regression accuracy.

Benefits of technology

It effectively improves infrared image contrast and small object detection accuracy, achieves more flexible bounding box regression accuracy under different IoU thresholds, improves the accuracy and recall of object detection, and makes detection box positioning more accurate.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN115082672B_ABST
    Figure CN115082672B_ABST
Patent Text Reader

Abstract

The present invention discloses an infrared image target detection method based on bounding box regression, and the scheme is as follows: (1) constructing a new infrared image target detection network R‑YOLOv4; (2) constructing a FLIR infrared data set; (3) using a new adaptive CLAHE algorithm to enhance the contrast of infrared images; (4) using a new α‑IoU-based regression loss function to train the network, and obtaining a trained infrared image target detection network based on bounding box regression; (5) inputting the infrared image to be detected into the trained network for detection and annotation, and outputting the detection result of the infrared target. The target detection network of the present invention can adaptively enhance the contrast of infrared images, and the target detection network can effectively extract features, more flexibly realize different levels of bounding box regression accuracy, and when the IoU threshold is large, the accuracy and recall rate of the target can be improved, and the positioning of the detection box can be made more accurate.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention belongs to the field of computer vision technology, relates to target recognition, and in particular to an infrared image target detection method based on bounding box regression, which can be used in the fields of intelligent security, intelligent transportation, etc. Background Art

[0002] Computer vision is a science that uses computers to imitate the human visual system. Object detection is an important research direction of computer vision. Its main task is to locate the positions of all objects in an image or video frame and identify the category to which the object belongs. With the rapid development of computer technology and sensor technology, infrared images have been widely used in military and civilian fields due to their advantages such as wide night vision range, strong anti-interference ability, and all-weather operation. As the core technology of modern infrared systems, infrared image object detection has always been an important research content in the field of image processing. In recent years, with the outstanding research progress made by deep learning technology in the field of image processing, the development of infrared image object detection technology has been effectively promoted. Among many deep learning object detection algorithms, the regression target detection algorithm is one of the classic algorithms. It can directly convert the target bounding box positioning problem into a bounding box regression problem, and has the characteristics of high detection accuracy and fast speed. However, infrared images have poor contrast and lack of target texture features, so it is difficult for the target detection network to extract effective features of infrared images. Moreover, the IoU-type regression loss function used by the regression target detection algorithm makes it difficult to achieve accurate regression of target bounding boxes at different levels in complex road scenes.

[0003] Existing object detection methods based on bounding box regression can be divided into region proposal detection algorithms and regression detection algorithms. In 2014, Ross Girshick proposed a representative region proposal detection algorithm (Region-based CNN, R-CNN), which first uses the Selective Search method on the input image to screen out about 2,000 candidate boxes. Then, the features in each candidate box are obtained through CNN's convolution, pooling and other operations. It is worth noting that the input candidate boxes of CNN need to be fixed to the same size. Finally, the SVM classifier is used to determine whether the candidate box contains the target, and further predict the specific location and category of the target. During training, the above three steps are trained separately, and the training results of each step need to be transferred. The extraction of a large number of candidate box features and the fixed input feature map size of the CNN network lead to high model calculation complexity, which limits the training speed of the object detection network.

[0004] YOLOv1 (You Only Look Once) is one of the representative algorithms of regression-based object detection algorithms, proposed by JRedmon et al. in 2016. The main idea of ​​the YOLOv1 algorithm is to obtain the detection results of the target directly from the input image. It divides the input image into 7×7 grids. Assuming that the center point of a certain target falls in this grid, then this grid is responsible for predicting the target. Each grid generates 2 prediction boxes, and the parameters of each prediction box include its own position parameters and the confidence value of containing a certain type of target. Finally, a threshold is set for the output 7×7×2 prediction windows, and the non-maximum suppression method (NMS) is used to filter out windows with low probability. Although the regression-based detection algorithm can achieve real-time detection, the detection accuracy is lower than that of the region proposal network because it uses one network to predict the location and category of multiple targets. Summary of the invention

[0005] In order to overcome the shortcomings of the above-mentioned prior art, the purpose of the present invention is to provide an infrared image target detection method based on bounding box regression, in order to solve the problems of poor infrared image contrast, low target detection network feature extraction ability and inaccurate bounding box regression in the existing infrared image target detection technology. First, for the problem of poor infrared image contrast, a new adaptive CLAHE image optimization algorithm is used. According to the different contrasts of each image in the FLIR data set, the optimization parameters are adaptively selected to ensure that the contrast of all images in the data set is optimally improved. Then, the R-YOLOv4 target detection network is used. By adding the RFB module on the basis of the shallow network, the receptive field is greatly increased, the most significant contextual features can be separated, and the accuracy of small target detection is improved. Finally, a new regression loss function based on α-IoU is used. By adjusting the factor α of the regression loss function, the regression accuracy of bounding boxes at different levels is achieved, and the accuracy of target detection is improved.

[0006] In order to achieve the above object, the technical solution adopted by the present invention is:

[0007] An infrared image target detection method based on bounding box regression comprises the following steps:

[0008] Step (1) constructs the infrared image target detection network R-YOLOv4 based on deep learning. R-YOLOv4 consists of five modules, which are connected in the following order:

[0009] (1a) Backbone network CSPDarknet53 for preliminary feature extraction;

[0010] (1b) Feature pyramid network SPP for multi-scale feature fusion;

[0011] (1c) Receptive field module RFB for enhancing the receptive field of small objects;

[0012] (1d) Feature enhancement network PANet for enhanced feature extraction;

[0013] (1e) YOLO Head, a detection network used for object detection;

[0014] Step (2), constructing an infrared data set;

[0015] Step (3), preprocessing the infrared data set to enhance the contrast of the infrared image;

[0016] Step (4), based on the preprocessed infrared data set, train R-YOLOv4 using the regression loss function based on α-IoU to obtain the trained R-YOLOv4 based on bounding box regression;

[0017] Step (5), input the infrared image to be detected into the trained R-YOLOv4 for detection and annotation, and output the detection result of the infrared image target.

[0018] Compared with the prior art, the present invention has the following beneficial effects:

[0019] First, the present invention uses a new adaptive CLAHE image optimization algorithm to solve the problem that the fixed shear coefficient K cannot ensure that all images in the infrared data set have a good contrast enhancement effect. The method adaptively selects optimization parameters according to the different contrasts of each image in the FLIR data set to ensure that the contrast of all images in the data set is optimally improved.

[0020] Second, the present invention uses a new target detection network R-YOLOv4. By adding the RFB module on the basis of the shallow network, the hole convolution therein greatly increases the receptive field of small targets, can separate the most significant contextual features, and improve the detection accuracy of small targets. At the same time, the present invention replaces the 5×5 convolution in the original RFB module with two 3×3 convolutions, reducing the number of model parameters.

[0021] Third, the present invention uses a new α-IoU-based regression loss function. Compared with a certain IoU loss as the position regression loss, the loss function of the present invention can flexibly achieve different levels of bounding box regression accuracy by adjusting the parameter α.

[0022] Fourth, the present invention uses a frozen training method to enable the target detection network to quickly and accurately learn the characteristics of the data set, which is conducive to better convergence of the loss function, thereby ensuring the accuracy of target detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1 It is a flow chart for implementing the present invention.

[0024] Figure 2 Schematic diagram of the R-YOLOv4 network framework in the present invention.

[0025] Figure 3 It is a schematic diagram of the RFB structure in the present invention.

[0026] Figure 4 It is a schematic diagram of the fitting function of the adaptive CLAHE algorithm in the present invention.

[0027] Figure 5 This is a result diagram after contrast enhancement of the infrared image in the present invention.

[0028] Figure 6 It is the average accuracy of target detection on infrared images performed by the present invention.

[0029] Figure 7 This is a visualization result diagram of target detection on infrared images according to the present invention. DETAILED DESCRIPTION

[0030] The embodiments and effects of the present invention are further described in detail below in conjunction with the accompanying drawings.

[0031] In the existing infrared image target detection technology, it is difficult to achieve different levels of bounding box regression accuracy due to reasons such as the fixed IoU position regression loss function. Therefore, the present invention provides an infrared image target detection method based on bounding box regression to more flexibly achieve different levels of bounding box regression accuracy, especially when the IoU threshold is large, which can improve the accuracy and recall rate of the target and make the positioning of the detection box more accurate.

[0032] Reference Figure 1 The present invention is an infrared image target detection method based on bounding box regression, and the implementation steps are as follows:

[0033] Step 1: Build a new infrared image target detection network R-YOLOv4 based on deep learning.

[0034] The YOLOv4 algorithm is further improved on the basis of the YOLOv1 algorithm, and the detection accuracy is improved. In the present invention, R-YOLOv4 is further improved on the basis of the traditional YOLOv4 algorithm. Figure 2, R-YOLOv4 consists of five modules, which are: feature extraction network CSPDarknet53, multi-scale feature fusion network SPP, receptive field module RFB, feature enhancement network PANet and detection network YOLO Head in the order of network structure connection. Among them, CSPDarknet53 is the backbone network, which is used for preliminary feature extraction. SPP is a feature pyramid network, which is used for multi-scale feature fusion. RFB is used to enhance the receptive field of small targets. PANet is used to strengthen feature extraction. YOLO Head is used for target detection.

[0035] The key steps are as follows:

[0036] 1.1) Build feature extraction network CSPDarknet53

[0037] The structural relationship of the feature extraction network CSPDarknet53 is as follows: input layer → 1st convolutional layer → 2nd convolutional layer → 1st combination module CSP1 → 3rd convolutional layer → 4th convolutional layer → 2nd combination module CSP2 → 5th convolutional layer → 6th convolutional layer → 3rd combination module CSP8 → 7th convolutional layer → 8th convolutional layer → 4th combination module CSP8 → 9th convolutional layer → 10th convolutional layer → 5th combination module CSP4 → 11th convolutional layer.

[0038] The parameters of each module are set as follows:

[0039] The 1st to 11th convolutional layers are composed of CBM operations;

[0040] The number of channels of the first and second convolutional layers are 32 and 64 respectively, the convolution kernels are both 3×3, and the strides are 1 and 2 respectively;

[0041] The number of channels of the 3rd and 4th convolutional layers are 64 and 128 respectively, the convolution kernels are 1×1 and 3×3 respectively, and the strides are 1 and 2 respectively;

[0042] The number of channels of the 5th and 6th convolutional layers are 128 and 256 respectively, the convolution kernels are 1×1 and 3×3 respectively, and the strides are 1 and 2 respectively;

[0043] The number of channels of the 7th and 8th convolutional layers are 256 and 512 respectively, the convolution kernels are 1×1 and 3×3 respectively, and the strides are 1 and 2 respectively;

[0044] The number of channels of the 9th and 10th convolutional layers are 512 and 1024 respectively, the convolution kernels are 1×1 and 3×3 respectively, and the strides are 1 and 2 respectively;

[0045] The number of channels of the 11th convolutional layer is 512, the convolution kernel is 3×3, and the stride is 1.

[0046] The first combination module CSP1 is composed of three CBM convolutional layers (i.e., performing CBM operations) and one CSP residual module; the CBM convolutional layer is composed of a convolutional layer with 64 channels, a convolutional kernel size of 1×1, a stride of 1, a batch normalization layer, and a Mish activation function layer; the CSP residual module is composed of two CBM convolutional layers with 32 and 64 channels, 1×1 and 3×3 convolutional kernels, and a stride of 1.

[0047] The second combination module CSP2 is composed of three CBM convolutional layers (i.e., performing CBM operations) and two CSP residual modules; the CBM convolutional layer is composed of a convolutional layer with a convolution kernel channel number of 128, a convolution kernel size of 1×1, a stride of 1, a batch normalization layer, and a Mish activation function layer; the CSP residual module is composed of two CBM convolutional layers with 64 and 128 channels, 1×1 and 3×3 convolution kernels, and a stride of 1.

[0048] The third combination module CSP8 is composed of 3 CBM convolutional layers (i.e., performing CBM operations) and 8 CSP residual modules; the CBM convolutional layer is composed of a convolutional layer with a convolution kernel channel number of 256, a convolution kernel size of 1×1, a stride of 1, a batch normalization layer, and a Mish activation function layer; the CSP residual module is composed of 2 CBM convolutional layers with 128 and 256 channels, 1×1 and 3×3 convolution kernels, and a stride of 1.

[0049] The fourth combination module CSP8 is composed of 3 CBM convolutional layers (i.e., performing CBM operations) and 8 CSP residual modules; the CBM convolutional layer is composed of a convolutional layer with a convolution kernel channel number of 512, a convolution kernel size of 1×1, a stride of 1, a batch normalization layer, and a Mish activation function layer; the CSP residual module is composed of 2 CBM convolutional layers with 256 and 512 channels, 1×1 and 3×3 convolution kernels, and a stride of 1.

[0050] The fifth combination module CSP4 is composed of 3 CBM convolutional layers (i.e., performing CBM operations) and 4 CSP residual modules; the CBM convolutional layer is composed of a convolutional layer with a convolution kernel channel number of 1024, a convolution kernel size of 1×1, a stride of 1, a batch normalization layer, and a Mish activation function layer; the CSP residual module is composed of 2 CBM convolutional layers with 512 and 1024 channels, 1×1 and 3×3 convolution kernels, and a stride of 1.

[0051] The first combination module CSP1, the second combination module CSP2, the third combination module CSP8, the fourth combination module CSP8 and the fifth combination module CSP4 are all composed of two branches. The first branch is composed of the first CBM convolution layer, X CSP residual modules and the second CBM convolution layer. The input of the first CBM convolution layer of each combination module is the output of the previous convolution layer (such as the second convolution layer, the fourth convolution layer, the sixth convolution layer, the eighth convolution layer, and the tenth convolution layer), and its output is connected to the CSP residual module. The CSP residual modules are connected in sequence, and the output of the last CSP residual module is connected to the second CBM convolution layer; the second branch is the third CBM convolution layer, and its input is the output of the previous convolution layer; the output of the entire combination module is the feature map of the output of the second CBM convolution layer and the output of the third CBM convolution layer spliced ​​in the channel dimension.

[0052] In the present invention, the CBM operation consists of Conv convolution + Bn batch normalization layer + Mish activation function.

[0053] In the present invention, the CSP residual module is composed of two CBM convolutional layers connected in sequence, and the output of the second CBM convolutional layer is added to the input of the first CBM convolutional layer as the output of the CSP residual module.

[0054] In the present invention, the Mish activation function is expressed as: Mish = x × tanh (ln (1 + e x )), where x represents the output of the previous layer and tanh is the hyperbolic tangent function. The Mish activation function allows a smaller negative gradient to flow when the value is negative, ensuring the flow of information without the problem of gradient saturation.

[0055] 1.2) Build a feature pyramid network SPP

[0056] The structural relationship of SPP is: branch 1 → branch 2 → branch 3 → branch 4;

[0057] The parameters of each module are set as follows:

[0058] The first branch is a maximum pooling operation with a pooling kernel of 5;

[0059] The second branch is a maximum pooling operation with a pooling kernel of 9;

[0060] The third branch is a maximum pooling operation with a pooling kernel of 13;

[0061] The fourth branch is a maximum pooling operation with a pooling kernel of 1;

[0062] The input of SPP is the output of the last layer of CSPDarknet53 after CBL operation, that is, 13×13×512 feature map, and the output of SPP is the concatenation of the feature maps of the four branches in the channel dimension, that is, the output feature map is 13×13×2048. The feature pyramid network SPP uses the maximum pooling method of 1×1, 5×5, 9×9, and 13×13 to achieve multi-scale fusion.

[0063] In the present invention, the CBL operation consists of Conv convolution + Bn batch normalization + Leaky_relu activation function. The Leaky_relu activation function is expressed as: Where x represents the output of the previous layer.

[0064] 1.3) Build the receptive field module RFB

[0065] Reference Figure 3 , the structural relationship of RFB is: first branch → second branch → third branch → fourth branch → fifth branch.

[0066] The parameters of each module are set as follows:

[0067] The first branch consists of two 1×1 and 3×3 1_DCBL convolutional layers connected sequentially;

[0068] The second branch consists of two 1×1 and 3×3 1_DCBL convolutional layers and a 3×3 2_DCBL convolutional layer connected sequentially;

[0069] The third branch consists of a 1×1 1_DCBL convolutional layer, two 3×3 1_DCBL convolutional layers, and a 3×3 3_DCBL convolutional layer connected sequentially;

[0070] The fourth branch consists of a 1×1 1_DCBL convolutional layer;

[0071] The fifth branch consists of a 1×1 1_DCBL convolutional layer, and its input is the concatenation of the outputs of the first, second, and third branches;

[0072] In the present invention, the 1_DCBL convolution layer is composed of the Conv convolution layer + Bn batch normalization + Leaky_relu activation function; the 2_DCBL convolution layer is composed of the dilated convolution with a dilation rate of 2 + Bn batch normalization + Leaky_relu activation function; the 3_DCBL convolution layer is composed of the dilated convolution with a dilation rate of 3 + Bn batch normalization + Leaky_relu activation function; the dilation rate defines the interval distance between each value when the convolution kernel of the dilated convolution processes data.

[0073] In the present invention, there are two RFBs in the R-YOLOv4 network, which are respectively embedded after the 7th convolutional layer and the 9th convolutional layer of the backbone network CSPDarknet53. Compared with the maximum pooling operation, the hole convolution in the RFB can increase the receptive field with maximum efficiency without losing feature information.

[0074] 1.4) Build feature enhancement network PANet

[0075] The structural relationship of PANet is: a combination module → b combination module → c combination module → d combination module.

[0076] The parameters of each module are set as follows:

[0077] a Combination module, consisting of the first stacking module and a convolutional layer. The first stacking module is the feature layer after the output of the feature pyramid network SPP passes through the CBL structure and upsampling and the 26×26×256 effective feature layer output by the 9th convolutional layer of the backbone network CSPDarknet53 passes through the RFB structure.

[0078] b combination module, composed of the second stacking module and a convolutional layer. The second stacking module is the output of the a combination module after the CBL structure and upsampling of the feature layer and the 52×52×128 effective feature layer output by the 7th convolutional layer of the backbone network CSPDarknet53 after the RFB structure.

[0079] The c combination module is composed of the third stacking module and the convolution layer. The third stacking module is the stacking of the output feature layer of the a combination module and the feature layer after the output downsampling of the b combination module.

[0080] d Combination module, consisting of the 4th stacking module and the convolution layer. The 4th stacking module is a stack of the output downsampled feature layer of the c combination module and the output feature layer of the feature pyramid network SPP;

[0081] In the present invention, the convolutional layers of the combination modules a, b, c and d are all composed of CBL alternating convolutions of 1×1 pixels and 3×3 pixels.

[0082] 1.5) Build the detection network YOLO Head

[0083] In the present invention, there are three YOLO Head structures, each of which is composed of a 3×3 convolutional layer and a 1×1 convolutional layer. The input of the first YOLO Head is the output of the b combination module in the feature enhancement network PANet, the input of the second YOLO Head is the output of the c combination module in the feature enhancement network PANet, and the input of the third YOLO Head is the output of the d combination module in the feature enhancement network PANet.

[0084] The 3×3 convolution layer is a CBL convolution layer, and the 1×1 convolution layer is composed of only the Conv convolution layer. The 3×3 convolution layer integrates the obtained pixel features, and the 1×1 convolution layer uses the pixel features to obtain the final output result.

[0085] Step 2: Build infrared dataset

[0086] In the embodiment of the present invention, a FLIR infrared image dataset is used, which contains infrared image files and annotation files. The dataset construction process is as follows:

[0087] 2.1) Download the FLIR infrared image dataset, which contains an infrared image folder and an annotation folder. Since the dataset contains a small number of infrared images of dogs, it is not conducive to the learning of the target detection model and cannot objectively reflect the performance of the model in other categories. Therefore, infrared images containing dogs and infrared images without targets are removed. The final dataset contains a total of 9220 infrared images and 9220 annotation files. The images include three types of targets: Car, Person, and Bicycle. When the infrared image is input into the target detection network, the image resolution is adjusted to 416×416 pixels, and the annotation file format is adjusted to .xml format file;

[0088] 2.2) The number of images in the FLIR infrared image folder and the number of annotated files in the annotation folder are randomly divided according to the set ratio of training set: validation set: test set = 8:1:1. After the division, the training set contains 7376 images and 7376 annotated files, the validation set contains 922 images and 922 annotated files, and the test set contains 922 images and 922 annotated files.

[0089] Step 3: Preprocessing of infrared dataset

[0090] The infrared images in the FLIR dataset have problems such as low contrast and lack of target texture features, which are not conducive to feature extraction by the target detection network. Therefore, a new adaptive CLAHE image optimization algorithm is used to adaptively select optimization parameters according to the different contrasts of each image in the dataset to ensure that the contrast of all images in the dataset is optimally improved. The specific implementation process of the CLAHE algorithm is as follows:

[0091] (3a1) Divide the original image into M sub-regions of the same size.

[0092] (3b1) Calculate the grayscale histogram of each sub-region.

[0093] (3c1) Calculate the average number of pixels Avg distributed to each gray level from all pixels in each sub-region, limit the number of pixels contained in each gray level to not exceed K times Avg, and evenly distribute all pixels exceeding this limit to the gray level.

[0094] (3d1) After performing the above operations on each sub-region, grayscale histogram equalization is performed. The implementation process is as follows:

[0095] First, traverse all pixels in the image sub-region and calculate the number of pixels at each gray level; then find the ratio of the number of pixels at each gray level to the total number of pixels; then arrange the gray levels from small to large and calculate the cumulative probability of the number of pixels at each gray level; finally, establish a mapping relationship and map all pixels of the original image to new gray values ​​one by one. The mapping relationship is expressed as:

[0096]

[0097] Among them, q represents the output grayscale value, k represents the input pixel, H and W are the image width and height; Indicates the cumulative probability of the number of pixels in the gray value, which is between 0 and 1.

[0098] (3e1) The center point of the sub-region is taken as the sample point, and the linear interpolation method is used for each pixel of the image to obtain the pixel values ​​of all points.

[0099] The adaptive CLAHE image optimization algorithm of the present invention makes the following improvements based on the CLAHE image optimization algorithm:

[0100] 3.1) Calculate the contrast of all images in the FLIR infrared dataset. The contrast calculation formula is:

[0101]

[0102] Among them, δ(i,j) is the difference in grayscale values ​​between adjacent pixels, P δ (i, j) is the pixel distribution probability whose grayscale value difference between adjacent pixels is δ(i, j). After calculation, the contrast interval of the image in the FLIR dataset is (2×10 4 ~3×10 4 ).

[0103] 3.2) Select an image with the lowest contrast and an image with the highest contrast from the dataset, and select the best shear coefficient K and the number of sub-region divisions M for the two images. When selecting, the number of sub-regions and the shear coefficient K will affect the performance of the algorithm. If the number of sub-region divisions is small, it will not be conducive to enhancing the partial contrast and local details of the image. On the contrary, it will cause obvious boundaries between image blocks after linear interpolation, resulting in "jaggies". If the shear coefficient K is small, the effect of enhancing the contrast will not be achieved. On the contrary, the noise will be over-enhanced. After experiments, the contrast is determined to be 2×10 4 The optimal subregionalization score for the image is 8×8, and the optimal shear factor is 7. For a contrast of 3×10 4 For the image, the sub-region fraction is set to 8×8 and the shear coefficient is set to 1 to avoid increasing noise. Since the images in the FLIR infrared dataset have the same size, the number of sub-region divisions is fixed to 8×8, and only in the contrast interval of the entire dataset (2×10 4 ~3×10 4 )Fitting shear coefficient (1~7).

[0104] 3.3) Reference Figure 4 , the present invention proposes two fitting functions: linear and nonlinear. Among them, the linear fitting function is expressed as:

[0105] y=mx+n

[0106] Where x represents the infrared image contrast, y represents the shear coefficient, and m and n can be calculated based on the results in the above steps (2×10 4 , 7) and (3×10 4 , 1) Find the values ​​of m and n, respectively (-6)×10 -4 and 19.

[0107] The nonlinear fitting function is expressed as:

[0108]

[0109] Wherein, β represents a hyperparameter, which is set to 2 in the present invention, and b represents the middle value of the contrast, 2.5×10 4 , a and c can be calculated based on the results in the above steps (2×10 4 , 7) and (3×10 4 , 1) Find out, the values ​​of a and c are 6 and 1 respectively.

[0110] Step 4: Train the target detection network R-YOLOv4:

[0111] Specifically, the training process includes:

[0112] 4.1) Set up the environment required for R-YOLOv4. In this embodiment, the environment is set as: Windows 10 Professional Edition 64-bit operating system, GPU acceleration library cuda 10.1, cudnn4.1.5, graphics card name GeForce RTX 2060, software platform Anaconda3, Pycharm2020.1.2, programming language Python, deep learning framework Pytorch1.2.0, Torchvision=3.4.0;

[0113] 4.2) Use the VOC dataset for pre-training;

[0114] 4.3) Use K-means clustering algorithm to cluster the real frames of infrared dataset and get 9 representative prior frames (7,11), (11,13), (13,39), (21,21), (30,90), (34,39), (59,57), (97,101), (160,216), and evenly distribute them to the feature maps of 13×13, 26×26 and 52×52 of R-YOLOv4 network according to the size of the prior frame;

[0115] 4.4) Load the pre-trained weights and use the α-IoU-based regression loss function for training. The expression of the α-IoU-based regression loss function is:

[0116] L α-IoU =1-IoU α , α>0

[0117] Among them, IoU represents the ratio of the intersection and union of the real box and the predicted box, IoU α Represents the power IoU term. The α-IoU loss function has the following characteristics: ① It is a monotonically decreasing function. ② When α>1, the weighting factor w Lr It increases monotonically with the increase of IoU, which can help the model focus more on High IoU (large IoU value) targets, that is, give High IoU targets a greater loss, so that the network gradually corrects the high IoU prediction box and gradually reduces the focus on low IoU targets, thereby improving the accuracy of target position regression. ③ When α>1, the relative gradient ▽ r It increases monotonically with the increase of IoU. This relative gradient adjustment scheme allows the model to learn the target at an adaptive speed (different gradients) according to the target's IoU. The expressions of weighting factor and relative gradient are:

[0118]

[0119]

[0120] Since the YOLOv4 network uses the CIoU loss function as the regression loss function, the α-IoU loss is expanded to a more general form. CIoU The power penalty term is introduced into the formula as the final regression loss function of the YOLOv4 network, L CIoU The loss function expression is:

[0121]

[0122] Among them, (b,b gt ) represents the center point of the predicted box and the true box, ρ represents the distance between the center point of the predicted box and the true box, c represents the diagonal length of the minimum closure area of ​​the predicted box and the true box, βυ is the penalty term, β is used to measure the consistency of the aspect ratio, and υ is the trade-off parameter.

[0123] In L CIoU After the power penalty term is introduced into the formula, the expression is:

[0124]

[0125] By adjusting the factor α in this loss function, the object detection network can more flexibly achieve different levels of bounding box regression accuracy.

[0126] In terms of training methods, the target detection network adopts the frozen training method. The training set and validation set in the FLIR dataset are input into the R-YOLOv4 network. The model fits the data samples through the training set, and the hyperparameters of the model are adjusted through the validation set to make a preliminary evaluation of the model's capabilities. The specific process is: first freeze the pre-trained weights of the feature extraction network CSPDarknet53 for training, and put more computing resources on the parameters of other network layers. After training for a set time (for example, 25 generations), unfreeze the frozen CSPDarknet53 network, train all together, and continuously optimize the network training parameters until 50 generations of training. When the validation set loss function converges (for example, 50 generations of training), stop training. Finally, a trained infrared image target detection network is obtained.

[0127] Step 5: Input the image to be detected into the trained R-YOLOv4 network for detection and annotation, and output the infrared image with the target annotated.

[0128] Through the above steps, the target detection network of the present invention can adaptively enhance the contrast of infrared images. At the same time, the target detection network can effectively extract features and more flexibly achieve different levels of bounding box regression accuracy. When the IoU threshold is large, the accuracy and recall rate of the target can be improved, and the positioning of the detection frame can be made more accurate.

[0129] The effects of the present invention are further described below in conjunction with simulation experiments.

[0130] 1. Experimental conditions:

[0131] The hardware test platform of the simulation experiment of the present invention is: the CPU is Intel (R) Core (TM) i5-9400F, the main frequency is 2.90GHz, and the memory is 16GB.

[0132] The experimental evaluation indicators are: detection accuracy mAP@0.5, mAP@0.65, mAP@0.75, mAP@0.85 for the three categories of Car, Person, and Bicycle, and the average detection accuracy mAP under four IoU thresholds. Among them, mAP represents the average accuracy of all categories, and IoU represents the ratio of the intersection and union between the detection box and the true box. In general, there is an IoU threshold for the judgment of the detection box, that is, the detection box greater than the threshold indicates that the target is correctly predicted. mAP@0.5 represents the average accuracy of all categories when the IoU threshold is 0.5.

[0133] 2. Experimental content and results analysis:

[0134] The simulation experiment of the present invention adopts the method of the present invention to perform data set preprocessing on the constructed FLIR infrared image data set, trains the R-YOLOv4 network using the regression loss function based on α-IoU, and uses the trained target detection network to detect and annotate the input image to obtain the target detection result of the infrared image.

[0135] like Figure 5 As shown in the figure, (a) represents the original infrared image, and (b) represents the infrared image after contrast enhancement. It can be seen that the adaptive CLAHE image optimization method can effectively improve the contrast of infrared images, making the target details and textures clearer. The original infrared images in the first and second rows of the figure have low contrast. After selecting appropriate parameters for processing, the contours of various targets in the image are clearer. The original infrared image in the third row has high contrast, but the target is too bright. After selecting appropriate parameters for processing, the global contrast of the image is improved.

[0136] like Figure 6 As shown, the R-YOLOv4 network uses L CIoU L when the loss function and α are 2, 2.5 and 3 respectively α-CIoU The loss function is the average detection accuracy of the three types of targets under different IoU thresholds. It can be seen that when the threshold is 0.5, L CIoU The average detection accuracy of the three types of targets under the loss function is 78.3%. However, as the IoU threshold increases, when the threshold is 0.65, L 3-CIoUThe average detection accuracy of the three types of targets under the loss function is the highest, which is 54.2%. When the threshold is 0.75, L 3-CIoU The average detection accuracy of the three types of targets under the loss function is the highest, which is 28.5%. When the threshold is 0.85, L 3-CIoU The average detection accuracy of the three types of targets under the loss function is the highest, which is 5.8%. 3-CIoU Under the loss function, the average detection accuracy mAP under four thresholds is 41.7%, which is lower than that of L CIoU Loss function, the detection accuracy is improved by 1.1%.

[0137] The experimental results show that L α-CIoU The loss function provides the flexibility to achieve different levels of bounding box regression accuracy when training the object detection network. When the object detection network needs to locate the target more accurately, that is, when the IoU threshold is large, L α-CIoU The loss function shows better performance.

[0138] like Figure 7 As shown in Figure 2, the target detection results of the YOLOv4 network and the R-YOLOv4 network using different loss functions are shown, where (a) indicates that the YOLOv4 network uses L CIoU The detection results when the loss function is used, (b) indicates that the R-YOLOv4 network uses L 3-CIoU By comparing (a) and (b), we can see that in the first row of images, for objects such as cars, when using L CIoU When the vehicle is damaged, only part of it can be detected. 3-CIoU When using the loss function, the entire target can be completely located. For targets such as pedestrians, when using L CIoU When using the loss function, although pedestrians can be identified, the detection box is larger than the real box. 3-CIoU When the loss function is used, the detection box is very close to the real box, and the positioning is more accurate. In the second row of images, due to the RFB module in the R-YOLOv4 network, the receptive field of small targets is increased. Therefore, for targets such as pedestrians, more pedestrians can be identified in complex road scenes such as target occlusion and target edge blur. The detection results show that the R-YOLOv4 network based on the regression loss function based on α-IoU can effectively improve the detection accuracy of the target.

[0139] In summary, the infrared image target detection method based on bounding box regression proposed in the present invention can effectively extract infrared image target features and can more flexibly achieve different levels of bounding box regression accuracy. When the IoU threshold is large, the regression loss function based on α-IoU can improve the accuracy of the target and make the positioning of the detection box more accurate.

[0140] The above is only a specific example of the present invention to facilitate the understanding of the present invention by those skilled in the art, but the present invention is not limited to the scope of the specific example. For those of ordinary skill in the art, as long as various changes are within the spirit and scope of the present invention defined and determined by the attached claims, these changes are obvious, and all inventions and creations utilizing the concept of the present invention are protected by the present invention.

Claims

1. A method for infrared image target detection based on bounding box regression, characterized in that: The steps include: Step (1) constructs the infrared image target detection network R-YOLOv4 based on deep learning. R-YOLOv4 consists of five modules, which are connected in the following order: (1a) Backbone network CSPDarknet53 for preliminary feature extraction; (1b) Feature pyramid network SPP for multi-scale feature fusion; (1c) Receptive field module RFB for enhancing the receptive field of small objects; (1d) Feature enhancement network PANet for enhanced feature extraction; (1e) YOLO Head, a detection network used for object detection; Step (2), constructing an infrared data set; Step (3), preprocessing the infrared data set to enhance the contrast of the infrared image; Step (4), based on the preprocessed infrared data set, train R-YOLOv4 using the regression loss function based on α-IoU to obtain the trained R-YOLOv4 based on bounding box regression; Step (5), input the infrared image to be detected into the trained R-YOLOv4 for detection and annotation, and output the detection result of the infrared image target; In the step (3), the infrared data set is preprocessed using an adaptive CLAHE image optimization algorithm. According to the different contrasts of the images in the infrared data set, optimization parameters are adaptively selected to ensure that the contrasts of all images in the infrared data set are improved. The adaptive CLAHE image optimization algorithm includes: (3a1) Divide the original image into M sub-regions of the same size; (3b1) Calculate the grayscale histogram of each sub-region; (3c1) Calculate the number of pixels Avg that all pixels in each sub-region are evenly distributed to each gray level, limit the number of pixels contained in each gray level to not exceed K times Avg, and evenly distribute all pixels exceeding this limit to the gray level; (3d1) After performing the above operations on each sub-region, grayscale histogram equalization is performed; (3e1) Taking the center point of the sub-region as the sample point, use the linear interpolation method for each pixel of the image to obtain the pixel values ​​of all points; Among them, by calculating the contrast of all images in the infrared data set, an image with the lowest contrast and an image with the highest contrast are selected, and the optimal shearing coefficient K and the number of sub-region divisions M are selected for the two images. The number of sub-region divisions M remains fixed, and the shearing coefficient K is fitted using a fitting function in the contrast range of the entire infrared data set.

2. The infrared image target detection method based on bounding box regression according to claim 1, characterized in that: In step (1), the structural relationship of CSPDarknet53 is as follows: Input layer → 1st convolution layer → 2nd convolution layer → 1st combination module CSP1 → 3rd convolution layer → 4th convolution layer → 2nd combination module CSP2 → 5th convolution layer → 6th convolution layer → 3rd combination module CSP8 → 7th convolution layer → 8th convolution layer → 4th combination module CSP8 → 9th convolution layer → 10th convolution layer → 5th combination module CSP4 → 11th convolution layer, where: The 1st to 11th convolutional layers are composed of CBM operations; The first combination module CSP1 is composed of three CBM operations with 64 convolution kernel channels and one CSP residual module; The second combined module CSP2 is composed of three CBM operations with 128 convolution kernel channels and two CSP residual modules. The third combined module CSP8 is composed of three CBM operations with 256 convolution kernel channels and eight CSP residual modules. The fourth combined module CSP8 is composed of three CBM operations with 512 convolution kernel channels and eight CSP residual modules. The fifth combined module CSP4 is composed of three CBM operations with 1024 convolution kernel channels and four CSP residual modules. The first to fifth combination modules are composed of two branches. The first branch consists of the first CBM convolution layer, X CSP residual modules and the second CBM convolution layer. The input of the first CBM convolution layer of each combination module is the output of the second convolution layer, the fourth convolution layer, the sixth convolution layer, the eighth convolution layer and the tenth convolution layer, respectively. Its output is connected to the CSP residual module, and each CSP residual module is connected in sequence. The output of the last CSP residual module is connected to the second CBM convolution layer; the second branch is the third CBM convolution layer, and its input is the output of the previous convolution layer; the output of the entire combination module is the feature map of the output of the second CBM convolution layer and the output of the third CBM convolution layer spliced ​​in the channel dimension; Each of the CSP residual modules is composed of two CBM convolutional layers connected in sequence, and the output of the second CBM convolutional layer is added to the input of the first CBM convolutional layer as the output of the CSP residual module; The CBM operation consists of Conv convolution + Bn batch normalization layer + Mish activation function; the Mish activation function is expressed as: Mish = x × tanh (ln (1 + e x )), where x represents the output of the previous layer and tanh is the hyperbolic tangent function.

3. The infrared image target detection method based on bounding box regression according to claim 2 is characterized in that: In the step (1), the structural relationship of the SPP is: the first branch → the second branch → the third branch → the fourth branch, wherein: The first branch is a maximum pooling operation with a pooling kernel of 5; The second branch is a maximum pooling operation with a pooling kernel of 9; The third branch is a maximum pooling operation with a pooling kernel of 13; The fourth branch is a maximum pooling operation with a pooling kernel of 1; The input of SPP is the output of the last layer of CSPDarknet53 after CBL operation, and its output is the concatenation of the feature maps of the four branches in the channel dimension; The CBL operation consists of Conv convolution + Bn batch normalization + Leaky_relu activation function; The leaky_relu activation function is expressed as: Where x represents the output of the previous layer.

4. The infrared image target detection method based on bounding box regression according to claim 3 is characterized in that: In the step (1), the structural relationship of the RFB is: first branch→second branch→third branch→fourth branch→fifth branch, wherein: The first branch consists of two 1×1 and 3×3 1_DCBL convolutional layers connected sequentially; The second branch consists of two 1×1 and 3×3 1_DCBL convolutional layers and a 3×3 2_DCBL convolutional layer connected sequentially; The third branch consists of a 1×1 1_DCBL convolutional layer, two 3×3 1_DCBL convolutional layers, and a 3×3 3_DCBL convolutional layer connected sequentially; The fourth branch consists of a 1×1 1_DCBL convolutional layer; The fifth branch consists of a 1×1 1_DCBL convolutional layer, and its input is the concatenation of the outputs of the first, second, and third branches; The 1_DCBL convolution layer consists of Conv convolution layer + Bn batch normalization + Leaky_relu activation function; the 2_DCBL convolution layer consists of dilated convolution with a dilation rate of 2 + Bn batch normalization + Leaky_relu activation function; the 3_DCBL convolution layer consists of dilated convolution with a dilation rate of 3 + Bn batch normalization + Leaky_relu activation function; There are two RFBs in R-YOLOv4, which are embedded after the 7th and 9th convolutional layers of CSPDarknet53 respectively.

5. The infrared image target detection method based on bounding box regression according to claim 4 is characterized in that: In the step (1), the structural relationship of PANet is: a combination module → b combination module → c combination module → d combination module, wherein: a Combination module, consisting of the first stacking module and a convolutional layer. The first stacking module is a stack of the feature layer after the output of SPP passes through the CBL structure and upsampling and the 26×26×256 effective feature layer output by the 9th convolutional layer of the backbone network CSPDarknet53 passes through the RFB structure; b Combination module, consisting of the second stacking module and a convolutional layer. The second stacking module is a stack of the feature layer after the output of the combination module a passes through the CBL structure and upsampling, and the 52×52×128 effective feature layer output by the 7th convolutional layer of CSPDarknet53 passes through the RFB structure; c combination module, composed of the third stacking module and the convolution layer, the third stacking module is the stacking of the output feature layer of the a combination module and the feature layer after the output downsampling of the b combination module; d Combination module, consisting of the 4th stacking module and the convolution layer. The 4th stacking module is a stack of the output downsampled feature layer of the c combination module and the output feature layer of the feature pyramid network SPP; The convolutional layers of the combined modules a, b, c, and d consist of alternating CBL convolutions of 1×1 pixels and 3×3 pixels; There are three YOLO Head structures, each of which is composed of a 3×3 convolutional layer and a 1×1 convolutional layer; the 3×3 convolutional layer is a CBL convolutional layer, and the 1×1 convolutional layer is a Conv convolutional layer; among them, the input of the first YOLO Head is the output of the b combination module in PANet, the input of the second YOLO Head is the output of the c combination module in PANet, and the input of the third YOLOHead is the output of the d combination module in PANet.

6. The infrared image target detection method based on bounding box regression according to claim 1, characterized in that: In the step (2), the FLIR infrared data set is downloaded, infrared images without targets are removed, and the number of infrared image files and the number of annotated files are divided into a training set, a validation set, and a test set according to a set ratio.

7. The infrared image target detection method based on bounding box regression according to claim 6, characterized in that: The implementation process of the grayscale histogram equalization method is as follows: First, traverse all pixels in the image sub-region and calculate the number of pixels in each gray level; Then find the ratio of the number of pixels at each gray level to the total pixels; Then arrange the gray levels from small to large and calculate the cumulative probability of the number of pixels at each gray level; Finally, a mapping relationship is established to map all pixels of the original image to new grayscale values ​​one by one; The mapping relationship is expressed as: Among them, q represents the output grayscale value, k represents the input pixel, H and W are the image width and height; Indicates the cumulative probability of the number of pixels in the gray value, which is between 0 and 1.

8. The infrared image target detection method based on bounding box regression according to claim 1, characterized in that: In step (4), the training process includes: (4a) Set up the environment required for R-YOLOv4; (4b) Use the VOC dataset for pre-training; (4c) Use the K-means clustering algorithm to cluster the real frames of the infrared dataset, obtain 9 representative prior frames, and evenly distribute them to the feature maps of 13×13, 26×26, and 52×52 of R-YOLOv4; (4d) Load the pre-trained weights and train using the α-IoU based regression loss function.

9. The infrared image target detection method based on bounding box regression according to claim 1 or 8, characterized in that: The expression of the regression loss function based on α-IoU is: L α-IoU =1-IoU α ,α>0 Among them, IoU represents the ratio of the intersection and union of the real box and the predicted box, IoU α represents the power IoU term; In L CIoU The power penalty term is introduced into the formula, and the expression is: Among them, (b,b gt ) represents the center point of the predicted box and the true box, ρ represents the distance between the center point of the predicted box and the true box, c represents the diagonal length of the minimum closure area of ​​the predicted box and the true box, βυ is the penalty term, β is used to measure the consistency of the aspect ratio, and υ is a trade-off parameter. By adjusting the factor α in the loss function, different levels of bounding box regression accuracy can be achieved; In step (4d), the frozen training method is used to input the training set and validation set of the infrared dataset into R-YOLOv4. The training set is used to fit the model to the data samples, and the validation set is used to adjust the network hyperparameters and conduct a preliminary evaluation of the network detection capability.

10. The infrared image target detection method based on bounding box regression according to claim 9, characterized in that: The process of frozen training is as follows: first freeze the pre-trained weights of CSPDarknet53 for training, unfreeze the frozen CSPDarknet53 after the set training time, train all together, continuously optimize the network training parameters, stop training when the validation set loss function converges, and finally obtain the trained R-YOLOv4.