An infrared image pedestrian detection method
By combining YOLOv5, the attention module, and Unet to form the YOLO-Unet network, pseudo-color images are generated and features are extracted and classified, which solves the problem of insufficient performance of infrared pedestrian detection in low-light environments and achieves efficient pedestrian detection results.
Patent Information
- Application Number
- CN202310742124.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2023-06-21
- Publication Date
- 2025-12-09
- Estimated Expiration
- 2043-06-21
AI Technical Summary
In low-light environments, traditional infrared pedestrian detection methods suffer from insufficient detection performance, especially when infrared image features are obvious. Furthermore, the collection and annotation of large-scale datasets are difficult, making complex detection models impractical.
YOLOv5 is combined with the attention module and Unet to form the YOLO-Unet network. The Unet attention module generates pseudo-color images and transmits them to YOLOv5. The DIoU-NMS technology is used for post-processing and detection loss function to optimize feature extraction and pedestrian classification.
It improves the accuracy and performance of pedestrian detection in infrared images without requiring complex detection models or large datasets, especially achieving high detection accuracy in public infrared pedestrian datasets.
Smart Images
Figure CN116977917B_ABST
Abstract
Description
TECHNICAL FIELD
[0001] The present application relates to the technical field of pedestrian detection, and particularly relates to an infrared image pedestrian detection method. BACKGROUND
[0002] Pedestrian detection is a crucial task with wide practical applications, including enhancing pedestrian safety, enabling autonomous driving, and improving video surveillance. However, in low-light environments, due to factors such as reduced visibility, pedestrian detection remains a challenging task. Therefore, developing an effective infrared pedestrian detection system is a top priority. Infrared imaging has become a popular technology in surveillance systems due to its superior visual capabilities. However, due to the more pronounced features of infrared images compared to visible light images, traditional pedestrian detection methods that rely on manually designed features may not be sufficient for infrared pedestrian detection. Additionally, to train a pedestrian detection model with high accuracy, the collection and labeling of large-scale datasets is also a daunting task.
[0003] In recent years, machine learning models have been widely applied in infrared pedestrian detection and recognition tasks. Prior to the development of end-to-end detection methods, traditional computer vision techniques such as Histogram of Oriented Gradients (HOG), Local Binary Patterns (LBP), and Aggregated Channel Features (ACF) were commonly used in this field. However, these methods have limitations in accurately capturing all necessary information about pedestrians in thermal imaging. Although machine learning-based methods such as deep neural networks and the YOLO series have been proven to improve the performance of infrared pedestrian detection to some extent, they can result in complex detection models with high costs. Additionally, infrared pedestrian datasets are generally smaller than visible light datasets, so using complex detection models is not always feasible. SUMMARY
[0004] The present application aims to improve the performance of thermal image pedestrian detection simply. In view of the above-mentioned deficiencies, an infrared image pedestrian detection method is proposed.
[0005] The present application adopts the following technical solutions:
[0006] An infrared image pedestrian detection method, comprising the following steps:
[0007] S1: combining YOLOv5 with an attention module and Unet to form a YOLO-Unet network;
[0008] S2: the attention module and the Unet form an attention Unet, the attention Unet selects an infrared image or a pair of infrared and visible light pedestrian images, the attention Unet encodes and decodes the selected infrared image or the pair of infrared and visible light pedestrian images, generates a pseudo-color image and transmits the pseudo-color image to the YOLOv5;
[0009] S3: the YOLOv5 uses a DIoU-NMS technology for post-processing and detection loss function, and performs feature extraction and pedestrian classification;
[0010] The DIoU-NMS technology in step S3 involves the following formula:
[0011]
[0012]
[0013]
[0014] Wherein, S i is the score of DIoU-NMS, IoU is the intersection over union, R DIoU is a new penalty term of DIoU, ε is a predefined threshold, B is an estimated bounding box, B gt is a target reference true value, b is the center point of B, b gt is the center point of B gt , c is the diagonal length of the minimum bounding box C covering B and B gt , and ρ is the Euclidean distance between the two center points b and b gt .
[0015] Optionally, in step S3, the loss function of the YOLOv5 detection comprises three parts, namely, a bbox regression loss, a target confidence loss and a class loss.
[0016] Optionally, in step S3, the YOLOv5 detection loss function satisfies the following formula:
[0017]
[0018]
[0019]
[0020]
[0021]
[0022] Wherein, L(t p ,tgt ) is the total loss, t p is the prediction vector, t gt is the true value vector, K, S 2 and B are the output feature map, the predicted box and the number of anchors on each predicted box respectively, a box , a obj and a cls are the weights of the corresponding bbox regression loss, target confidence loss and class loss, is the kth output feature map, is the weight of the output feature map for balancing each scale, takes the value of or or corresponds to the output feature map of 80*80 when corresponds to the output feature map of 40*40 when corresponds to the output feature map of 20*20 when CIoU is the CIoU loss, CIoU is the IoU considering the overlapping area, the distance between the center points, and the aspect ratio, d v is the diagonal length of the minimum bounding box covering the true value bounding box and the predicted box, d e is the Euclidean distance between the center points of the true value bounding box and the predicted box, w gt is the width of the true value bounding box, h gt is the height of the true value bounding box, w is the width of the predicted box, and h is the height of the predicted box, L obj is the target confidence loss, p iou is the IoU value of the predicted box and the target box corresponding thereto, p o is the target confidence score in the predicted box, is the binary cross-entropy function with sigmoid BCEWithLogitsLoss, w obj is the weight of the adjusted sample, L cls is the classification loss, w cls is the weight of the adjusted sample, c p is the class score of the predicted box, c gt is the class score of the target box.
[0023] Optionally, in step S3, the YOLOv5 includes a backbone component, a neck component and a head component which are connected in communication with each other.
[0024] Optionally, in step S3, the backbone component uses the CSP-Darknet53 architecture, the neck component uses the SPPF and CSP-PAN architectures to generate a feature pyramid, and the head component uses the YOLO general prediction layer.
[0025] The beneficial effects obtained by the present application are:
[0026] 1. YOLOv5 is combined with an attention module and Unet to form a YOLO-Unet network, which focuses on common features, optimizes feature extraction, and classifies pedestrians;
[0027] 2. The YOLO-Unet network can share visible light information from multiple related data sets and achieve high pedestrian detection accuracy in a public infrared pedestrian data set;
[0028] 3. The YOLO-Unet network can improve the performance of thermal image pedestrian detection without the need for complex detection models or large data sets.
[0029] In order to further understand the features and technical contents of the present application, please refer to the following detailed description and drawings of the present application. However, the provided drawings are only used for reference and illustration, and are not used to limit the present application. BRIEF DESCRIPTION OF DRAWINGS
[0030] Figure 1 The flowchart of the present application;
[0031] Figure 2 The structural framework of the present application;
[0032] Figure 3 The structure and composition of the present application;
[0033] Figure 4 The corresponding FLIR data set in the present application;
[0034] Figure 5 The corresponding LLVIP data set in the present application. DETAILED DESCRIPTION
[0035] The following is to illustrate the embodiments of the present application by specific specific embodiments, and the person skilled in the art can understand the advantages and effects of the present application from the disclosed content of the present application. The present application can be implemented or applied by other different specific embodiments, and the details in the present application can be modified and changed based on different viewpoints and applications without departing from the spirit of the present application. In addition, the drawings of the present application are only simple schematic illustrations, not actual size drawings, and the prior declaration is made. The following embodiments will further illustrate the related technical contents of the present application, but the disclosed content is not used to limit the protection scope of the present application.
[0036] The present embodiment provides an infrared image pedestrian detection method, which combines Figures 1 to 5 as shown.
[0037] An infrared image pedestrian detection method, comprising the following steps:
[0038] S1: combine YOLOv5 with an attention module and Unet to form a YOLO-Unet network;
[0039] S2: the attention module and Unet form an attention Unet, the attention Unet selects an infrared image or a pair of infrared and visible light pedestrian images, the attention Unet encodes and decodes the selected infrared image or the pair of infrared and visible light pedestrian images, generates a false color image and transmits the false color image to the YOLOv5;
[0040] S3: the YOLOv5 uses a DIoU-NMS technique for post-processing and detection loss function, and performs feature extraction and pedestrian classification;
[0041] The DIoU-NMS technique in step S3 involves the following formula:
[0042]
[0043]
[0044]
[0045] wherein, S i is the score of DIoU-NMS, IoU is the intersection over union, R DIoU is a new penalty term of DIoU, ε is a predefined threshold value, B is an estimated bounding box, B gt is a target reference true value, b is the center point of B, b gt is the center point of B gt , c is the diagonal length of the minimum bounding box C that covers B and B gt at the same time, and ρ is the Euclidean distance between the two center points b and b gt . The formula involved in the DIoU-NMS technique considers the distance between the predicted bounding box and the actual true bounding box and the intersection measurement, and the purpose is to suppress the overlapping bounding boxes that are close to each other, while retaining those bounding boxes that are most relevant to the given target detection task.
[0046] Specifically, when the YOLOv5 performs post-processing, it mainly predicts the results and then filters the predicted frames of pedestrians. This process is achieved by using the distance intersection union non-maximum suppression technique, i.e., the DIoU-NMS technique.
[0047] Specifically, the combined YOLO-Unet network creates an improved end-to-end target detection method.
[0048] Specifically, the attention module is an attention mechanism that can learn to focus on structures of different sizes and shapes, making it suitable for infrared pedestrian detection. The attention module is incorporated into the Unet, which can automatically eliminate irrelevant areas in the input image and highlight significant features for subsequent processing. The combination of the attention module and the Unet forms an attention Unet, which removes the softmax layer and changes the output channel number of the last layer of convolution, allowing the attention Unet to output a three-channel pseudo-color image instead of a pixel-level segmentation image. The encoding part of the attention Unet gradually filters the input image and downsamples it by two at each scale. In the decoding process, convolution and upsampling are used to obtain deeper features, and AGs are used to selectively propagate features through skip connections. The attention coefficients calculated in the AGs are used to scale the input features, collecting gating signals from coarser scales to provide activation and contextual information for selecting spatial regions. Grid resampling of attention coefficients is achieved using tri-linear interpolation. The attention module is inserted into the C3 module and the decoder module of the Unet, enhancing network performance by improving infrared feature extraction accuracy and reducing unnecessary feature extraction.
[0049] Optionally, in step S3, the loss function of YOLOv5 detection is composed of three parts, namely bbox regression loss, target confidence loss and class loss.
[0050] Optionally, in step S3, the YOLOv5 detection loss function satisfies the following formula:
[0051]
[0052]
[0053]
[0054]
[0055]
[0056] wherein L(t p ,t gt ) is the total loss, t p is the predicted vector, t gt is the true value vector, K, S 2 and B are the output feature map, the predicted box and the number of anchors on each predicted box respectively, α box , α obj and α cls are the weights of the corresponding bbox regression loss, target confidence loss and class loss, is the kth output feature map, is the weight of the output feature map for balancing each scale, is the value of or or corresponds to the output feature map of 80*80, corresponds to the output feature map of 40*40, corresponds to the output feature map of 20*20, L CIoU is the CIoU loss, CIoU is the IoU considering the overlapping area, the distance between the center points, and the aspect ratio, d v is the diagonal length of the minimum bounding box covering the true value bounding box and the predicted box, d e is the Euclidean distance between the center points of the true value bounding box and the predicted box, w gt is the width of the true value bounding box, h gt is the height of the true value bounding box, w is the width of the predicted box, and h is the height of the predicted box, L obj is the target confidence loss, p iou is the IoU value of the predicted box and the target box corresponding thereto, p o is the target confidence score in the predicted box, is the binary cross-entropy function with sigmoid, BCEWithLogitsLoss, w obj is the weight of the sample, L cls is the classification loss, w cls is the weight of the sample, c p is the class score of the predicted box, c gt is the class score of the target box.
[0057] Specifically, k, i, and j are the loop quantities for summation. YOLOv5 divides the feature map into a plurality of cells, and each cell is a predicted box. Each cell outputs a vector of [t x , t y , t w , t h , p o , c1, c2, …], wherein t x , t y is used to calculate the offset of the center of the predicted box and the corresponding anchor box, t w , t h is used to calculate the width and height of the predicted box, p owhere c is the cell containing the object of interest, and c1, c2, … are the predicted values for the corresponding classes. The loss of each part is calculated by the matched positive sample pairs, and each output pseudo-color image is independent of each other, and the final loss value of each part is directly added. The bbox regression loss uses CioU LOSS, which uses b and b gt The object confidence loss is calculated by the sample pairs matched by the positive samples, and the final object confidence loss is calculated by the binary cross-entropy of the object confidence score in the prediction box and the IoU value of the prediction box and the target box corresponding to it; the class loss and the confidence loss are similar.
[0058] Optionally, in step S3, the YOLOv5 includes a backbone component, a neck component, and a head component which are communicatively connected to each other.
[0059] Optionally, in step S3, the backbone component uses the CSP-Darknet53 architecture, the neck component uses the SPPF and CSP-PAN architectures to generate a feature pyramid, and the head component uses the YOLO general prediction layer.
[0060] Specifically, the attention module is inserted after the backbone component of the YOLOv5, which enhances the network performance by improving the infrared feature extraction accuracy and reducing the extraction of unnecessary features.
[0061] The following settings correspond to the experiments and the results are analyzed.
[0062] In this study, two datasets were used to study pedestrian detection.
[0063] The first dataset is the FLIR dataset, which consists of annotated thermal infrared images and corresponding unannotated visible light images, with a total of 14452 infrared images. Among them, 10,228 images come from the collection of multiple short videos, and 4224 images come from a video lasting 144 seconds. All videos are shot on streets and highways, which makes them highly relevant to real-world scenarios for pedestrian detection.
[0064] The second dataset is the LLVIP dataset, which consists of strictly aligned thermal infrared images and visible light images. This dataset contains 30976 images or 15488 pairs of images, mainly taken in very dark environments.
[0065] Notably, all annotated objects in this dataset are pedestrians, which makes it highly relevant to the goal of this study.
[0066] In addition, this study evaluates the performance of the proposed method on the two datasets using three indicators.
[0067] First, the mean average precision (mAP@0.5) with an IoU threshold of 0.5 is used to measure the accuracy of the predicted bounding boxes with an IoU of 0.5 or higher compared to the ground truth boxes. Second, mAP@0.5-0.95 measures the accuracy of the predicted bounding boxes at different IoU thresholds, providing a more comprehensive evaluation. Finally, the frame-level F1 score, a commonly used metric in classification tasks, is used to measure the harmonic mean of precision and recall. The F1 score evaluates the performance of the classifier from both precision and recall. While mAP@0.5 and mAP@0.5-0.95 are specifically designed for object detection tasks, the F1 score is more generally used for classification tasks. These metrics provide a comprehensive evaluation of the performance of the proposed method on different datasets.
[0068] Furthermore, the model is implemented using PyTorch and experiments are conducted on an NVIDIA Tesla A100 GPU and an AMD Threadripper PRO 5995WX CPU. Table 1 (Comparison of Model Size and Inference Time) provides different network architectures of the model and their related key parameters. In each trained model, the number of epochs is set to 50. The loss function converges after 30 iterations, and the detection performance reaches a relatively stable state. The batch size is 64, which occupies nearly 80% of the available memory.
[0069]
[0070] Table 1
[0071] To verify the superiority of the proposed method, three different datasets, FLIR and LLVIP, are used for evaluation. Specifically, the FLIR dataset is divided into 8862 labeled thermal images for training and 1366 labeled thermal images for testing, maintaining the original division. Meanwhile, the LLVIP dataset is divided according to the original division, with 12025 image pairs for training and 3463 image pairs for testing. It is worth noting that all test results are determined using the fully converged network model weights.
[0072] Finally, the experimental results are analyzed. The performance of the two network architectures, YOLOv5 and Unet+YOLOv5, is compared on the FLIR dataset, which contains annotated heatmaps of pedestrians, bicycles, and cars. Due to the misalignment of the images in the dataset, only the heatmaps are used, and a selector is used to convert them into three-channel grayscale images. During training, the weights of Unet are randomly generated, while the YOLOv5 network is pre-trained on visible light images. The results shown in Table 2 (Comparison of precision for pedestrian detection) indicate that the performance of the Frozen-backbone training method, which only trains the backbone network of YOLOv5, is significantly lower than that of full training. The addition of Unet in the front end can better extract features, thereby improving the detection performance in terms of mAP@0.5 and mAP@0.5-0.95. These results demonstrate the superiority of the proposed Unet+YOLOv5 architecture over the YOLOv5 architecture alone in terms of heat image pedestrian detection.
[0073]
[0074] Table 2
[0075] The FLIR dataset is used for low-light pedestrian detection, consisting of aligned infrared and low-light visible light images. In this study, a 4-channel RGBT image is generated using a designed selector. The 4-channel input Unet+YOLO network architecture is compared with the original YOLO network. During training, the Unet weights are randomly generated, while YOLO loads the ready-made pre-trained YOLOv5x.pt weights. Figure 4 The results of pedestrian detection on the FLIR dataset are reported. The results of the study show that infrared image training is superior to visible light image training, as most images are captured under low-light conditions. The addition of the Unet structure enhances the detection performance, especially in terms of mAP@0.5-0.95, which is improved by 0.154 and 0.05 compared to visible light image training and infrared image training, respectively. Figure 4 In the results, (a) is the real pedestrian image, (b) is the visible light YOLOv5 prediction result obtained using pre-trained weights, i.e., YOLOv5x.pt, (c) is the Unet detection result obtained by training on the FLIR dataset, and (d) is the Unet+YOLOv5 detection result obtained by training on the FLIR dataset, where the images framed in red represent errors in the detection process.
[0076] Figure 5The proposed method is compared with the original YOLOv5 for different types of LLVIP dataset images. It is worth noting that the actual test shows that the model trained only on visible light data produces a high rate of false positives and false negatives. In contrast, the model trained with Unet+YOLO shows detection performance comparable to the pure infrared model, even in low light conditions. Post-processing of the Unet output image further enhances the pedestrian cues by generating detectable, clearer regions compared to the original visible light image. However, as mentioned above, some "staining" artifacts can still appear in certain regions of the post-processed Unet output image. Importantly, the overall brightness of the image is neutralized to match that of a normal visible light image under normal lighting settings. Figure 5 In the above, (a) is the pedestrian true value; (b) is the detection result of the original YOLOv5 trained only using visible light images; (c) is the detection result of the original YOLOv5 trained only using infrared images; (d) is the detection result based on paired infrared and visible light images, where the yellow arrow is a stained area, and the red circle is an error detection.
[0077] In summary, a method for infrared pedestrian detection uses an improved Unet and YOLO architecture to incorporate information from the visible light domain, i.e., using the structure of attention Unet and YOLOv5, the proposed method involves using an improved Unet to learn a mapping from pedestrian data in the infrared and visible light domains to a shared feature space. This mapping can generate pseudo-color infrared images for training and prediction of the detection network, particularly YOLO. By fusing these pseudo-color infrared images, a YOLOv5 model trained on visible light data can accurately detect pedestrian targets in the infrared domain. It is worth noting that the proposed method of utilizing information from the visible light domain still relies on available public datasets in this domain to achieve optimal performance. Nevertheless, the proposed method shows significant effectiveness in infrared pedestrian detection.
[0078] The above disclosed content is only a preferred feasible embodiment of the present application, and is not limited to the protection scope of the present application, so any equivalent technical changes made according to the content of the present application and the drawings are included in the protection scope of the present application, and in addition, the elements can be updated as technology develops.
Claims
1. An infrared image pedestrian detection method, characterized by, The method comprises the following steps: S1: combine YOLOv5 with an attention module and Unet to form a YOLO-Unet network; S2: the attention module and the Unet form an attention Unet, the attention Unet selects an infrared image or a pair of infrared and visible light pedestrian images, the attention Unet encodes and decodes the selected infrared image or the pair of infrared and visible light pedestrian images, generates a false color image and transmits the false color image to the YOLOv5; S3: the YOLOv5 uses a DIoU-NMS technology for post-processing and detection loss function, and performs feature extraction and pedestrian classification; The DIoU-NMS technology in step S3 involves the following formula: wherein S i is the score of DIoU-NMS, IoU is the intersection over union, R DIoU is the penalty term added by DIoU, ε is a predefined threshold, B is an estimated bounding box, B gt is a target ground truth, b is the center point of B, b gt is the center point of B gt , c is the diagonal length of the minimum bounding box C which covers B and B gt at the same time, and ρ is the Euclidean distance between b and b gt .
2. The method of claim 1, wherein, In step S3, the loss function of the YOLOv5 detection is composed of three parts, namely, the bbox regression loss, the target confidence loss and the class loss.
3. The method of claim 2, wherein the infrared image pedestrian detection method is characterized by, In step S3, the YOLOv5 detection loss function satisfies the following formula: Wherein, L(t) p , t gt ) represents the total loss, t p For the prediction vector, t gt Let K and S be truth vectors. 2 B and α represent the output feature map, predicted bounding box, and the number of anchors on each predicted bounding box, respectively. box α obj and α cls These are the weights for the corresponding bounding box regression loss, target confidence loss, and class loss. For the k-th output feature map, To balance the weights of the output feature maps at each scale, Values or or The corresponding output feature map is 80*80. The corresponding output feature map is 40*40. The corresponding output feature map is 20*20, L CIoU The loss is CioU, where CIoU is the IoU that takes into account the overlap area, center-to-center distance, and aspect ratio. v The diagonal length of the smallest bounding box that simultaneously covers both the ground truth bounding box and the predicted bounding box, d e w is the Euclidean distance between the center points of the ground truth bounding box and the predicted bounding box. gt h is the width of the truth bounding box. gt Let L be the height of the ground truth bounding box, w be the width of the predicted bounding box, h be the height of the predicted bounding box, and L be the height of the predicted bounding box. obj For the target confidence loss, p iou p represents the IoU value between the predicted bounding box and its corresponding target bounding box. o The target confidence score in the prediction box. For the binary cross-entropy function BCEWithLogitsLoss with sigmoid, w obj To adjust the weights of the samples, L cls For classification loss, w cls To adjust the weights of the samples, c p c represents the category score of the predicted bounding box. gt The category score for the target bounding box.
4. The method of claim 3, wherein the pedestrian is detected by using a pedestrian detection algorithm. In step S3, the YOLOv5 comprises a backbone component, a neck component and a head component which are connected in communication.
5. The method of claim 4, wherein the pedestrian is detected by using a pedestrian detection algorithm. In step S3, the backbone component uses a CSP-Darknet53 architecture, the neck component uses an SPPF and a CSP-PAN architecture to generate a feature pyramid, and the head component uses a YOLO general prediction layer.