A pedestrian detection method based on YOLOv7
Patent Information
- Application Number
- CN202311602425.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2023-11-28
- Publication Date
- 2026-10-27
- Estimated Expiration
- 2043-11-28
AI Technical Summary
[0004]本发明提出了一种基于YOLOv7的密集行人检测方法，该方法通过在网络的特定位置引入了特征过滤净化模块和采用基于深度可分离卷积的高效检测头等结构，解决了在密集行人检测任务中存在的复杂背景干扰和相互遮挡等问题
[0054](1)、本发明对YOLOv7进行了改进，本发明设计了一种聚合的特征过滤净化模块(FFPM)。首先对颈部预输入的多尺度特征进行新的级联融合，有效地过滤了跨层冲突；然后，受多尺度卷积注意力模块MSCA的启发，设计了多尺度特征增强模块(MSFE)，进一步优化级联融合后的特征，并为浅层特征提供丰富指导信息；接着，引入了CoordianateAttention(CA)模块与降维后的浅层特征相融合，它不仅保留了行人目标的空间位置信息，还进一步增强特征。
Smart Images
Figure CN117542082B_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of pedestrian detection technology, specifically relating to a dense pedestrian detection method based on an improved YOLOv7. Background Technology
[0002] Pedestrian detection plays a crucial role not only in intelligent transportation and assisted driving, but also provides the theoretical foundation and technical support for research on pedestrian fall detection and human behavior prediction. Computer vision-based pedestrian detection technology has played a decisive role in autonomous and assisted driving. This technology can acquire image data of the vehicle's surroundings through imaging devices during driving, accurately locating pedestrian targets in the image sequence, improving the safety of assisted driving, and protecting the lives of drivers and pedestrians. In practical applications, various pedestrian detection systems require high real-time performance and strong robustness, which are the challenges of pedestrian detection technology and make it a research hotspot in the field of object detection. However, pedestrian detection in real-world scenarios often suffers from overlapping targets, mutual occlusion, or inconsistent scales, leading to poor detection results. Therefore, there is an urgent need to design an efficient, automated, and dense pedestrian detection method.
[0003] To address the aforementioned issues, this invention proposes an improved method for dense pedestrian detection based on the YOLO (You Only Look Once) neural network architecture. YOLO is a widely used object detection algorithm, characterized by real-time object detection, but its performance in high-density pedestrian areas still has room for improvement. This invention introduces a series of innovative technologies aimed at enhancing the accuracy and efficiency of the YOLO network in dense pedestrian environments. Summary of the Invention
[0004] This invention proposes a dense pedestrian detection method based on YOLOv7. This method addresses issues such as complex background interference and mutual occlusion in dense pedestrian detection tasks by introducing feature filtering and purification modules at specific locations in the network and employing an efficient detection head structure based on depthwise separable convolution. Compared to existing advanced detection models, this invention offers significant advantages in both detection accuracy and efficiency for target detection in pedestrian images, effectively meeting the real-time requirements of practical scenarios.
[0005] The present invention adopts the following technical solution:
[0006] The pedestrian detection method based on YOLOv7 includes the following steps:
[0007] S1. Collect pedestrian image dataset. Preferably, each image is accompanied by a TXT format annotation file, which contains detailed annotation information on the category and location of each pedestrian target in the image, including the location coordinates and category of the pedestrian (c, x1, y1, x2, y2).
[0008] S2. To increase the diversity of the dataset and improve the robustness of the model, the collected pedestrian image dataset undergoes Mosaic data augmentation to generate enhanced versions of the pedestrian images. Preferably, Mosaic data augmentation includes operations such as rotation, scaling, mirroring, and brightness adjustment to increase the diversity of the training data.
[0009] S3. The enhanced dataset is divided into training, validation, and test sets according to a preset ratio. This division helps to evaluate and validate the model's performance and ensures the appropriate use of the dataset.
[0010] S4. In the training set, cluster the ground truth (GT) boxes of all images to obtain K prior boxes. These prior boxes will be used in the model training and detection process. This step preferably uses the K-means++ clustering algorithm to cluster the GT boxes of all images.
[0011] S5. Randomly select X images from the training set and input them into the YOLOv7 network to obtain N predicted feature maps for object detection.
[0012] S6. K prior boxes are evenly distributed across N prediction feature maps according to their scale. Then, the corresponding prior boxes are adjusted based on the anchor point information on the prediction feature maps to generate all the prediction boxes. These prediction boxes will be used for the pedestrian detection task.
[0013] S7. Using the predicted bounding boxes obtained in step S6 and the corresponding ground truth bounding boxes in the images, calculate the network loss value. Then, use gradient descent to update the parameters of the YOLOv7 network to minimize the loss function, thereby optimizing the model's performance and enabling it to accurately detect pedestrian targets.
[0014] S8. Repeat steps S5 to S7 until all images in the training set have been input into the network once; this constitutes one training epoch. After each training epoch, use the parameter-updated YOLOv7 network to predict each image in the validation set and calculate the average precision (AP) value for pedestrian categories in the validation set. The AP value is a metric for evaluating model performance and pedestrian detection accuracy. Continue until the calculated AP value stabilizes at a predetermined value or shows no significant improvement over several consecutive training epochs. At this point, training is considered complete, and a fully trained YOLOv7 network is obtained.
[0015] S9. Use the trained YOLOv7 network to predict all images in the test set. This will generate predicted bounding boxes on the feature maps. Then, based on the scaling relationship between the feature maps and the original images, these predicted boxes are mapped onto the corresponding original images to accurately locate pedestrian targets. This step is used for pedestrian detection and localization in practical applications.
[0016] As a preferred embodiment, in step S1, the obtained dataset specifically includes multiple 640x640 pixel images and corresponding label files. The label files are in txt format and record the target location information and category information of the images. The format of each label file is (c, x1, y1, x2, y2), where c represents the category of the object contained in the label box, x1 and y1 represent the x and y coordinates of the upper left corner vertex of the label box, respectively, and x2 and y2 represent the x and y coordinates of the lower right corner vertex of the label box.
[0017] As a preferred approach, in step S1, the category and location annotations are referred to as Ground Truth (GT), and these annotations will serve as the true object bounding boxes for model training. This helps the model understand the true location and category of the object for effective training. As a preferred approach, in step S2, Mosaic data augmentation is used to augment the pedestrian dataset: four images are randomly selected from the training set, randomly cropped, and then stitched together to create a new image, generating new training data. This method helps increase the diversity and complexity of the training data.
[0018] As a preferred option, in step S2, using the Mosaic data augmentation strategy helps generate more diverse training samples, thereby improving the robustness and generalization ability of the model.
[0019] As a preferred approach, in step S3, the ratio of the training set, validation set, and test set is 8:1:1. This ratio helps to fully train the model while also providing suitable validation and test data to evaluate model performance. To manage these datasets, train.txt, val.txt, and test.txt files are generated to store the corresponding image lists. These files help guide the training and evaluation of the model on different datasets.
[0020] As a preferred approach, in step S4, the width and height of the target are calculated based on the location information of the label files in the training set, thereby performing clustering. The format of the label files is (c, x1, y1, x2, y2), where c represents the category of the object within the bounding box, x1 and y1 represent the coordinates of the upper left vertex of the bounding box, and x2 and y2 represent the coordinates of the lower right vertex of the bounding box. The bounding boxes are called Ground Truth (GT) boxes. The K-means++ clustering algorithm is used to cluster the GT boxes based on their width and height to obtain K cluster center coordinates. The center coordinates are used as the width and height of the K prior boxes to accommodate targets of different sizes.
[0021] As a preferred approach, in step S4, the Kmeans++ algorithm is used to cluster all ground truth (GT) boxes in the training set. Each GT box contains (class, xmin, ymin, xmax, ymax) information, where class represents the pedestrian category, xmin and ymin represent the coordinates of the top-left vertex, and xmax and ymax represent the coordinates of the bottom-right vertex.
[0022] The clustering process in step S4 involves using the K-means++ clustering method to cluster the ground truth (GT) boxes based on their width and height, resulting in K cluster center coordinates. These center coordinates will serve as the width and height of the anchor boxes. Typically, K is chosen as 9 to accommodate targets of different sizes, providing the model with anchor boxes of multiple scales.
[0023] In step S4, the scales of the three training set network output feature maps obtained are 20×20, 40×40, and 80×80, respectively. For each training set network output feature map, three prior boxes of different sizes are assigned.
[0024] As a preferred approach, S5, X images are randomly selected from the training set and input into the YOLOv7 network. The backbone of the network is used to extract image features and generate effective feature maps at different scales. Subsequently, the feature aggregation module further fuses these effective feature maps to obtain fused feature maps at different scales. Finally, the channels of the fused feature maps are adjusted to obtain N predicted feature maps for object detection.
[0025] As a preferred embodiment, step S5 specifically includes the following steps:
[0026] S5.1. Randomly select X images from the training set and input them into the YOLOv7 backbone network CSPDarknet-53 for stepwise feature extraction. Extract the three deepest effective feature maps with different scales and channel numbers. The CSPDarknet-53 backbone network includes sequentially connected ELAN and MP modules, which simultaneously perform downsampling, outputting three feature layers of different scales (named M5, M4, and M3 in ascending order of scale). Input these layers into the feature filtering and purification module FFPM to filter cross-layer conflicts, outputting feature maps of three different scales: L5, L4, and L3. Insert the FFPM module into the backbone network after feature extraction to enhance feature fusion.
[0027] S5.2. The three effective feature maps obtained in step S5.1 (referred to as L5, L4, and L3 according to their scale from smallest to largest) are input into the FFPM module and the path aggregation network PANet for further fusion. This top-down and bottom-up fusion process fully mixes deep and shallow features. For each layer's output feature map P5, P4, and P3 of the feature fusion module, their scale is consistent with the input feature maps L5, L4, and L3. After FFPM fuses multi-scale features, an MSFE module is inserted to further optimize the cascaded fused features and provide rich guidance information for shallow features. The FFPM structure introduces a Coordianate Attention (CA) module in the shallow layer to fuse with the dimensionality-reduced shallow features. The three feature layers output by FFPM are then input into PANet for further fusion.
[0028] S5.3 Adjust the number of channels of the three fused feature maps output by PANet to num_anchor*(5+num_class) through the lightweight adaptive scale guidance head (ASGH) to obtain N output feature maps, where num_anchor represents the number of preset prior boxes on each anchor point of the predicted feature map, and num_class represents the number of predicted classes.
[0029] In ASGH, a more lightweight 5×5 depthwise separable convolution is used to replace the parameter-intensive 3×3 convolution kernel, which is improved according to the Selective-and-Kernel (SK) attention mechanism to select features that are more beneficial to the final prediction.
[0030] As a preferred embodiment, step S5.2, the feature fusion process specifically includes the following steps:
[0031] S5.2.1. The M5 feature map is processed through the SPPCSPC module to obtain the feature map K5. K5 is upsampled and fused with the M4 feature map. The fusion result is input into the ELAN-W module to obtain the feature map K4.
[0032] S5.2.2 Upsample K4 and fuse it with the M3 feature map. Input the fusion result into the ELAN-W module to obtain the shallowest output feature map P3.
[0033] S5.2.3 Input the P3 feature map into the MP module for downsampling and fuse it with the K4 feature map. Input the fusion result into the ELAN-W module to obtain the intermediate layer output feature map P4.
[0034] S5.2.4 Input the P4 feature map into the MP module for downsampling and fuse it with the K5 feature map. Input the fusion result into the ELAN-W module to obtain the deepest output feature map P5.
[0035] S5.2.5 For the obtained feature maps P3, P4, and P5, input the adaptive scaling head ASGH for prediction.
[0036] As a preferred embodiment, in step S6, the nine prior boxes are evenly distributed to three different scales of output feature maps according to their size. The network output feature map at each scale is divided into a different number of grid cells, with the center position of the grid as the reference. For each grid cell, a corresponding candidate box or prediction box is generated according to its center position and the size of the assigned prior box.
[0037] In step S6, firstly, the K prior boxes generated in step S4 are sorted according to their scale. Then, these prior boxes are evenly distributed across the N predicted feature maps generated in step S5. Each predicted feature map is divided into an H×W grid, where the center of each grid cell is called an anchor point. At each anchor point, K / N prior boxes belonging to the corresponding feature map are overlaid. This provides multiple prior boxes of different sizes and aspect ratios for each anchor point.
[0038] As a preferred approach, in step S6, the prior boxes corresponding to the anchor points on the predicted feature map are further adjusted to obtain the predicted boxes. Each anchor point on the predicted feature map corresponds to a vector of length num_anchor*(5+num_class). By splitting this vector dimensionally, a one-dimensional adjustment vector of length 5+num_class is obtained for each prior box. The first 5 values represent the adjustment information of the width and height of the corresponding prior box, the center point coordinates, and the bounding box confidence, respectively. Through these adjustment information, the position and size of the prior boxes are adjusted accordingly, forming the predicted boxes.
[0039] In step S6, the network outputs a feature map including position and scale adjustment information, classification confidence, and bounding box confidence. Based on the offset information and width and height adjustment information in the adjustment information, the corresponding prior boxes are adjusted to obtain the predicted boxes.
[0040] As a preferred approach, the network loss value is calculated based on the network output feature map, predicted boxes, and ground truth (GT) boxes. Specifically, the cross-union ratio (CUNR) loss is calculated based on the predicted boxes and GT boxes. The classification confidence loss and border confidence loss are calculated based on the classification confidence and border confidence of each predicted box contained in the network output feature map. The CUNR loss, classification confidence loss, and border confidence loss are then weighted and summed according to a preset ratio to obtain the overall network loss. Backpropagation is then performed to optimize the network parameters.
[0041] In step S7, firstly, the corresponding prior boxes are found based on the location information of the ground truth (GT) boxes. Then, the GT boxes and their class label information are converted into vectors of length 5 + num_class. Next, this vector is compared with the vector of each prior box on the predicted feature map to calculate the loss value. This typically includes the mean squared error loss for location information (such as the loss for bounding box coordinates), confidence loss (used to indicate the presence of a target), and class loss (used for class classification). The loss value is used to guide gradient descent to update the model parameters to optimize the performance of the pedestrian detection model.
[0042] As a preferred approach, in step S8, all images from the entire training set are input into the network for forward and backward propagation, constituting one epoch. After each epoch, the network with updated parameters is used to predict each image in the validation set. Based on the prediction and Ground Truth (GT) label for each image, the Average Precision (AP) metric for each category in the validation set is calculated. AP is a metric used to evaluate model performance, particularly suitable for object detection tasks.
[0043] As a preferred approach, in step S8, the convergence of the network is determined by observing whether the AP value remains constant or shows a decreasing trend over multiple consecutive rounds. If the AP value stabilizes or decreases, the model can be considered to have converged. This is an important indicator that the model has achieved good performance on the training data and no further training is needed.
[0044] In step S8, the YOLOv7 network model is trained, which specifically includes the following steps:
[0045] S8.1 uses Python 3.7 and the deep learning framework PyTorch 1.8, and leverages CUDA to accelerate computation;
[0046] S8.2 Set the initial learning rate to 0.001 and the learning rate adjustment strategy to cosine annealing decay;
[0047] S8.3 Set the number of images input to the network in each batch to 16;
[0048] S8.4 Initialize the network weights using the Kaiming method; after each training cycle, calculate the overall loss and continue iterative training until the mean accuracy (AP) of the validation set stabilizes at the set value and training stops.
[0049] As a preferred embodiment, step S9, obtaining the final prediction box from all adjusted test set prediction boxes, specifically includes the following steps:
[0050] S9.1 Sort all adjusted prediction boxes according to their confidence scores, which helps to find the most reliable prediction boxes;
[0051] S9.2. Use the Non-Maximum Suppression (NMS) method to remove redundant boxes to obtain the final predicted boxes. NMS helps to remove highly overlapping boxes, retaining only the most representative ones.
[0052] As a preferred embodiment, in step S9, for each image in the test set, the network outputs N corresponding predicted feature maps. Then, on each predicted feature map, the prior bounding boxes are adjusted according to the adjustment vector corresponding to each anchor point, thereby obtaining all predicted bounding boxes for each image. Next, non-maximum suppression (NMS) is used to remove redundant boxes to obtain the final predicted bounding boxes on the feature maps. Finally, the predicted bounding boxes on the feature maps are mapped to the scale of the original image according to the scaling relationship to locate the pedestrian target.
[0053] The beneficial effects of this invention are:
[0054] (1) This invention improves upon YOLOv7 by designing an aggregated feature filtering and purification module (FFPM). First, a novel cascaded fusion of multi-scale features pre-inputted to the neck effectively filters cross-layer conflicts. Then, inspired by the multi-scale convolutional attention module MSCA, a multi-scale feature enhancement module (MSFE) is designed to further optimize the cascaded fused features and provide rich guidance information for shallow features. Next, a CoordianateAttention (CA) module is introduced to fuse with the dimensionality-reduced shallow features, which not only preserves the spatial location information of the pedestrian target but also further enhances the features.
[0055] (2) In view of the problems of high complexity and fixed receptive field size of the original detection head, this invention proposes a lightweight adaptive scale guidance head (ASGH), which can dynamically adjust its receptive field size to adapt to pedestrian targets of different scales.
[0056] (3) For the prediction head of the network, this invention uses a lighter 5×5 depthwise separable convolution to replace the parameter-intensive 3×3 convolution kernel, and improves upon it based on the Selective-and-Kernel (SK) attention mechanism to select features that are more beneficial to the final prediction. Through this technique, both model accuracy and speed are improved. Attached Figure Description
[0057] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0058] Figure 1 This is a flowchart of a preferred embodiment of the present invention for a dense pedestrian detection method based on an improved YOLOv7;
[0059] Figure 2 This is a schematic diagram of the improved YOLOv7 structure according to a preferred embodiment of the present invention;
[0060] Figure 3 This is a schematic diagram of the FFPM (Filtration and Purification Module) feature of a preferred embodiment of the present invention;
[0061] Figure 4 This is a schematic diagram of the Multi-Scale Feature Enhancement Module (MSFE) according to a preferred embodiment of the present invention;
[0062] Figure 5 This is a schematic diagram of the ASGH structure according to a preferred embodiment of the present invention;
[0063] Figure 6 This is a schematic diagram of the AP value of the improved YOLOv7 in a pedestrian dataset according to a preferred embodiment of the present invention. Detailed Implementation
[0064] The following specific embodiments illustrate the implementation of the present invention. Those skilled in the art can easily understand other advantages and effects of the present invention from the content disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and various details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that, unless otherwise specified, the following embodiments and features described therein can be combined with each other.
[0065] like Figure 1 As shown, this embodiment provides a dense pedestrian detection method based on an improved YOLOv7. Figure 1This is a flowchart illustrating the method. (Refer to...) Figure 2 This is a schematic diagram of the improved YOLOv7 network structure. The image is processed according to the procedure of this embodiment. The following is a detailed explanation of the effect of the method described in this embodiment on improving detection accuracy.
[0066] This embodiment presents a dense pedestrian detection method based on an improved YOLOv7, which specifically includes the following steps:
[0067] S1. First, a pedestrian image dataset is collected, with each image accompanied by a TXT format annotation file. This annotation file contains detailed annotation information on the category and location of each pedestrian target in the image, including the pedestrian's location coordinates and category;
[0068] S2. To increase the diversity of the dataset and improve the robustness of the model, the collected pedestrian image dataset underwent data augmentation to generate enhanced versions of the pedestrian images. Data augmentation included operations such as rotation, scaling, mirroring, and brightness adjustment to increase the diversity of the training data;
[0069] S3. The enhanced dataset is divided into training, validation, and test sets according to a preset ratio. This division helps to evaluate and validate the model's performance and ensures the appropriate use of the dataset.
[0070] S4. In the training set, the K-means++ clustering algorithm is used to cluster the ground truth bounding boxes of all images to obtain K prior boxes. These prior boxes will be used in the model training and detection process;
[0071] S5. Randomly select X images from the training set and input them into the YOLOv7 network. The backbone of the network is used to extract image features and generate effective feature maps at different scales. Subsequently, the feature aggregation module further fuses these effective feature maps to obtain fused feature maps at different scales. Finally, the channels of the fused feature maps are adjusted to obtain N predicted feature maps for object detection.
[0072] S6. K prior boxes are evenly distributed across N prediction feature maps according to their scale. Then, the corresponding prior boxes are adjusted based on the anchor point information on the prediction feature maps to generate all the prediction boxes. These prediction boxes will be used for the pedestrian detection task.
[0073] S7. Using the predicted bounding boxes obtained in step S6 and the corresponding ground truth bounding boxes in the images, calculate the network loss value. Then, use gradient descent to update the parameters of the YOLOv7 network to minimize the loss function, thereby optimizing the model's performance and enabling it to accurately detect pedestrian targets.
[0074] S8. Repeat steps S5 to S7 until all images in the training set have been input into the network once; this constitutes one training epoch. After each training epoch, use the parameter-updated YOLOv7 network to predict each image in the validation set and calculate the average precision (AP) value for pedestrian targets in the validation set. AP is a metric for evaluating model performance and is used to assess the accuracy of pedestrian detection. Training continues until the calculated AP value stabilizes at a predetermined value or shows no significant improvement over several consecutive training epochs. At this point, training is considered complete, and a fully trained YOLOv7 network is obtained.
[0075] S9. Use the trained YOLOv7 network to predict all images in the test set. This will generate predicted bounding boxes on the feature maps. Then, based on the scaling relationship between the feature maps and the original images, these predicted boxes are mapped onto the corresponding original images to accurately locate pedestrian targets. This step is used for pedestrian detection and localization in practical applications.
[0076] The following is a more detailed description of each step.
[0077] In step S1, the obtained dataset specifically includes multiple 640x640 pixel images and corresponding label files. The label files are in txt format and record the target location and category information in the images. Each label file has the format (c, x1, y1, x2, y2), where c represents the pedestrian category contained within the bounding box, x1 and y1 represent the x and y coordinates of the top-left vertex of the bounding box, and x2 and y2 represent the x and y coordinates of the bottom-right vertex of the bounding box. These bounding boxes are called Ground Truth (GT).
[0078] In step S2, Mosaic data augmentation is used to augment the pedestrian dataset. Specifically, four images from the training set are randomly selected, randomly cropped, and then stitched together to create a new image, generating new training data. This method helps increase the diversity and complexity of the training data.
[0079] In step S3, the ratio of the training set, validation set, and test set is set to 8:1:1. To manage these datasets, train.txt, val.txt, and test.txt files are generated to store the corresponding image lists. These files help guide the training and evaluation of the model on different datasets.
[0080] In step S4, the width and height of the target are calculated based on the label file in the training set, specifically the positional information within the label file, to perform clustering. The label file format is (c, x1, y1, x2, y2), where c represents the pedestrian category within the labeled box, x1 and y1 represent the coordinates of the top-left vertex of the labeled box, and x2 and y2 represent the coordinates of the bottom-right vertex of the labeled box. These labeled boxes are called Ground Truth (GT). The K-means++ clustering algorithm is used to cluster these GT boxes based on their width and height, obtaining K cluster center coordinates. These center coordinates will be used as the width and height of the K prior boxes to accommodate targets of different sizes.
[0081] a) The K-means++ algorithm first randomly selects the width and height values of a real box as the first prior box when setting the initial prior box.
[0082] b) Calculate the IoU (Interchange of Value) between the width and height of all ground truth boxes and the existing prior boxes on the same plane, and then calculate the probability that each ground truth box will be selected as the next prior box. The smaller the IoU value between the ground truth box and the existing prior boxes, the greater the probability that it will be selected as the next prior box.
[0083] c) Repeat step b) until K prior boxes are obtained.
[0084] d) Calculate the IoU between all ground truth boxes and the K anchor boxes, and assign the ground truth box with the highest IoU value to its corresponding cluster. The most suitable prior box is obtained by calculating the average width and height of all ground truth boxes in each cluster.
[0085] e) Repeat step d) until the width and height of all prior boxes no longer change or change by a very small amount, thus generating the optimal result.
[0086] Step S5 specifically includes the following steps:
[0087] S5.1. Randomly select X images from the training set and input them into the YOLOv7 backbone network CSPDarknet-53 for stepwise feature extraction. Extract the three deepest effective feature maps with different scales and number of channels. The backbone network CSPDarknet-53 includes ELAN and MP modules connected in sequence and performs downsampling at the same time, outputting three feature layers of different scales (named M5, M4, and M3 according to the scale from smallest to largest). Input them into the feature filtering and purification module FFPM to filter cross-layer conflicts.
[0088] S5.2 The three effective feature maps at different scales extracted from the backbone network are further fused using an improved FFPM and PANet. The output feature maps correspond to the scales of the backbone network output feature maps (20×20, 40×40, and 80×80, respectively). The improved FFPM first performs multi-scale cascaded fusion to increase the interaction between inter-layer features and filter out noise and redundant background information. Then, a multi-scale feature enhancement module (MSFE) is designed to further optimize the cascaded fused features and provide richer semantic information to the shallow features to help distinguish pedestrian targets from complex backgrounds. In addition, a Coordinate Attention (CA) module is introduced, which further improves the network's spatial localization ability by embedding positional information into channel attention and fusing it with the dimensionality-reduced shallow features. PANet represents a bidirectional FPN structure, SPP is spatial pyramid pooling, and CSP is cross-stage local structure.
[0089] S5.3. Adjust the number of channels of the fused feature map to num_anchor*(5+num_class) using a lightweight ASGH, and output N predicted feature maps. The lightweight ASGH is a depthwise separable convolution that we improved from the original 3×3 convolution in the YOLO Head. num_anchor represents the number of prior boxes on each grid, and num_class represents the number of classes.
[0090] In step S5.2, the feature fusion process specifically includes the following steps:
[0091] S5.2.1. The M5 feature map is processed through the SPPCSPC module to obtain the feature map K5. K5 is upsampled and fused with the M4 feature map. The fusion result is input into the ELAN-W module to obtain the feature map K4.
[0092] S5.2.2 Upsample K4 and fuse it with the M3 feature map. Input the fusion result into the ELAN-W module to obtain the shallowest output feature map P3.
[0093] S5.2.3 Input the P3 feature map into the MP module for downsampling and fuse it with the K4 feature map. Input the fusion result into the ELAN-W module to obtain the intermediate layer output feature map P4.
[0094] S5.2.4 Input the P4 feature map into the MP module for downsampling and fuse it with the K5 feature map. Input the fusion result into the ELAN-W module to obtain the deepest output feature map P5.
[0095] S5.2.5 For the obtained feature maps P3, P4, and P5, input the adaptive scaling head ASGH for prediction.
[0096] Anchor box assignment and adjustment: In step S6, the nine prior boxes are evenly distributed across three different scales of output feature maps according to their size order. Each scale's network output feature map is divided into a different number of grid cells, typically based on the grid center position. For each grid cell, a corresponding candidate box or predicted box is generated based on its center position and the size of the assigned prior box.
[0097] In step S4, the resulting training set network output feature maps have scales of 20×20, 40×40, and 80×80, respectively. For each training set network output feature map, three prior boxes of different sizes are assigned. For example, for the 20×20 scale training set network output feature map, because its receptive field is the largest, the three largest prior boxes are assigned. Similarly, for the 40×40 scale output feature map, the three largest prior boxes are assigned. For the 80×80 scale training set network output feature map, the three smallest prior boxes are assigned.
[0098] In step S6, the network output feature map includes position and scale adjustment information, classification confidence, and bounding box confidence. Furthermore, offset information and width and height adjustment information are obtained from the model output and applied to the prior boxes. Typically, the offset information is added to the coordinates of the prior boxes, and the width and height adjustment information is applied to the width and height of the prior boxes. The adjusted boxes are then the predicted boxes.
[0099] In step S7, the overall network loss is calculated based on the network output feature map, predicted bounding box, and corresponding ground truth bounding box, specifically as follows:
[0100] L IoU =1-IoU
[0101] L local =(xx) * ) 2 +(yy * ) 2 +(hh * ) 2 +(ww * ) 2
[0102] y i =Softmax(x i )
[0103]
[0104] Loss = 0.1L IoU +0.05L local +0.125L class
[0105] Where x, y, h, and w represent the coordinates of the top-left corner of the border and its width and height, respectively. Represents the true value of the category (0 or 1), y i This represents the probability of a pedestrian's class. The Intersection over Union (IoU) loss is calculated based on the predicted bounding boxes and their corresponding ground truth (GT) boxes. Classification confidence loss and bounding box confidence loss are calculated based on the classification confidence and bounding box confidence of each predicted bounding box in the network's output feature map. The IoU loss, classification confidence loss, and bounding box confidence loss are then weighted and summed according to a preset ratio to obtain the overall network loss. Backpropagation is then performed to optimize the network parameters.
[0106] In step S8, the YOLO v7 network model is trained, which specifically includes the following steps:
[0107] A: In terms of network environment configuration, Python 3.7 and the deep learning framework PyTorch 1.8 were used, and CUDA was used to accelerate computation to improve training speed;
[0108] B: Set the initial learning rate to 0.001, and the learning rate adjustment strategy to cosine annealing decay;
[0109] C: Set the number of images input to the network in each batch to 16. This is the amount of data input each time the model is updated.
[0110] D: The network weights are initialized using the Kaiming initialization method, without using pre-trained weights. After each training epoch, the overall loss is calculated, and iterative training continues until the mean accuracy (AP) of the validation set stabilizes at a certain value, at which point training stops.
[0111] Step S9, obtaining the final prediction box from all adjusted test set prediction boxes, specifically includes the following steps:
[0112] S9.1 Sort all adjusted prediction boxes according to their confidence scores, which helps to find the most reliable prediction boxes;
[0113] S9.2. Use the Non-Maximum Suppression (NMS) method to remove redundant boxes to obtain the final predicted boxes. NMS helps to remove highly overlapping boxes, retaining only the most representative ones.
[0114] To verify the performance of the proposed method, an improved YOLOv7 network was used to predict images in the test set. Mean precision (AP), class accuracy, and recall were calculated. The proposed method achieved high accuracy in real-time detection of dense pedestrians.
[0115] The beneficial effects of this invention are:
[0116] (1) This invention improves upon YOLOv7 by designing an aggregated feature filtering and purification module (FFPM). First, a new cascade fusion is performed on the multi-scale features pre-inputted to the neck, effectively filtering cross-layer conflicts. Then, inspired by the multi-scale convolutional attention module MSCA, a multi-scale feature enhancement module (MSFE) is designed to further optimize the cascaded fused features and provide rich guidance information for shallow features. Next, a Coordianate Attention (CA) module is introduced and fused with the dimensionality-reduced shallow features, which not only preserves the spatial location information of the pedestrian target but also further enhances the features.
[0117] (2) In view of the problems of high complexity and fixed receptive field size of the original detection head, this invention proposes a lightweight adaptive scale guidance head (ASGH), which can dynamically adjust its receptive field size to adapt to pedestrian targets of different scales.
[0118] (3) For the prediction head of the network, this invention uses a lighter 5×5 depthwise separable convolution to replace the parameter-intensive 3×3 convolution kernel, and improves upon it according to the Selective-and-Kernel (SK) attention mechanism to select features that are more beneficial to the final prediction. In this way, both the model accuracy and speed are improved.
[0119] In summary, this invention discloses a dense pedestrian detection method based on an improved YOLO v7 algorithm. First, a pedestrian dataset is collected and augmented using Mosaic data, then divided into training, validation, and test sets in an 8:1:1 ratio. Next, K-means++ clustering is used to cluster the ground truth (GT) bounding boxes in the dataset's label files, and the coordinates of the K cluster centers are used as the width and height of the anchor boxes, with K typically set to 9. This invention also designs an aggregated feature filtering and purification module (FFPM). First, a novel cascaded fusion of multi-scale features pre-inputted to the neck area effectively filters cross-layer conflicts. Then, inspired by the multi-scale convolutional attention module MSCA, a multi-scale feature enhancement module (MSFE) is designed to further optimize the cascaded fused features and provide rich guidance information for shallow features. Finally, this invention introduces a Coordianate Attention (CA) module to fuse with the dimensionality-reduced shallow features, which not only preserves the spatial location information of the pedestrian target but also further enhances the features. To address the issues of high complexity and fixed receptive field scale in traditional detection heads, this invention proposes a lightweight Adaptive Scale Guided Head (ASGH), which dynamically adjusts its receptive field size to adapt to pedestrian targets of different scales. For the prediction head of the network, this invention replaces the parameter-intensive 3×3 convolutional kernels with lighter depthwise separable convolutions, and improves upon them using a Selective-and-Kernel (SK) attention mechanism to select features more beneficial for the final prediction. Through this technique, both model accuracy and speed are improved.
[0120] The embodiments described above are merely preferred embodiments of the present invention and are not intended to limit the scope of the present invention. Various modifications and improvements made by those skilled in the art to the technical solutions of the present invention without departing from the spirit of the present invention should fall within the protection scope of the present invention.
Claims
1. A pedestrian detection method based on YOLOv7, characterized in that, Includes the following steps: S1. Collect a pedestrian image dataset. Each image is accompanied by a TXT format annotation file, which contains the location coordinates of the pedestrian and its category. S2. The collected pedestrian image dataset undergoes data augmentation processing; S3. The enhanced dataset is divided into training set, validation set and test set according to a preset ratio; S4. In the training set, cluster the target ground boxes of all images to obtain K prior boxes; S5. Randomly select X images from the training set, input them into the YOLOv7 network, and obtain N predicted feature maps for object detection; Step S5 specifically includes the following steps: S5.1 Randomly select X images from the training set and input them into the YOLOv7 backbone network CSPDarknet-53 for stepwise feature extraction. Extract the three deepest effective feature maps with different scales and number of channels. The backbone network CSPDarknet-53 includes ELAN and MP modules connected in sequence. Downsampling is performed to output three feature layers with different scales, which are named M5, M4 and M3 in order of increasing scale. Input them into the feature filtering and purification module FFPM to filter cross-layer conflicts. S5.2 The three effective feature maps at different scales extracted from the backbone network are fused using FFPM and the path aggregation network PANet. The output feature map corresponds to the scale of the output feature map of the backbone network. FFPM first performs multi-scale cascaded fusion, increasing the interaction between inter-layer features and filtering out noise and redundant background information. Then, a multi-scale feature enhancement module MSFE is designed to optimize the cascaded fused features. Next, a CA module is introduced to improve the spatial localization capability of the network by embedding location information into channel attention and fusing it with the dimensionality-reduced shallow features. PANet represents a bidirectional FPN structure, SPP is spatial pyramid pooling, and CSP is cross-stage local structure. S5.
3. Adjust the number of channels of the fused feature map to num_anchor * (5 + num_class) using lightweight ASGH, and output N predicted feature maps; where lightweight ASGH is a depthwise separable convolution obtained by improving the 3×3 convolution in YOLO Head, num_anchor represents the number of prior boxes on each grid, and num_class represents the number of classes; S6. K prior boxes are evenly distributed across N prediction feature maps according to the scale. The corresponding prior boxes are adjusted according to the anchor point information on the prediction feature maps to generate all prediction boxes. S7. Using the predicted bounding boxes obtained in step S6 and the ground truth bounding boxes of the corresponding images, calculate the network loss value; then use gradient descent to update the parameters of the YOLOv7 network to minimize the loss function. S8. Repeat steps S5 to S7 until all images in the training set have been input into the network once, constituting one training cycle. After each training cycle, use the YOLOv7 network with updated parameters to predict each image in the validation set and calculate the average precision (AP) value of the pedestrian category in the validation set. The training is considered complete when the calculated AP value stabilizes at a predetermined value or shows no significant improvement over several consecutive training cycles, and a fully trained YOLOv7 network is obtained. S9. Use the trained YOLOv7 network to predict all images in the test set and generate prediction boxes on the feature maps. Based on the ratio between the feature maps and the original images, map the prediction boxes onto the corresponding original images to locate pedestrian targets.
2. The pedestrian detection method based on YOLOv7 as described in claim 1, characterized in that, In step S1, the obtained dataset specifically includes multiple 640x640 pixel images and corresponding label files. The label files are in txt format and record the target location information and category information of the images. The format of each label file is (c, x1, y1, x2, y2), where c represents the category of the object contained in the label box, x1 and y1 represent the x and y coordinates of the upper left corner vertex of the label box, respectively, and x2 and y2 represent the x and y coordinates of the lower right corner vertex of the label box.
3. The pedestrian detection method based on YOLOv7 as described in claim 1, characterized in that, In step S2, Mosaic data augmentation is used to expand the pedestrian dataset: four images from the training set are randomly selected, the four images are randomly cropped, and then stitched together to form a new image, generating new training data.
4. The pedestrian detection method based on YOLOv7 as described in claim 1, characterized in that, In step S3, the ratio of the training set, validation set, and test set is set to 8:1:1, and train.txt, val.txt, and test.txt files are generated to save the corresponding image lists.
5. The pedestrian detection method based on YOLOv7 as described in claim 1, characterized in that, In step S4, the width and height of the target are calculated based on the location information of the label files in the training set, thereby performing clustering. The format of the label file is (c, x1, y1, x2, y2), where c represents the category of the object within the bounding box, x1 and y1 represent the coordinates of the top-left vertex of the bounding box, and x2 and y2 represent the coordinates of the bottom-right vertex of the bounding box. The K-means++ clustering algorithm is used to cluster the ground truth boxes based on their width and height, obtaining K cluster center coordinates. These center coordinates are used as the width and height of the K prior boxes to accommodate targets of different sizes.
6. The pedestrian detection method based on YOLOv7 as described in claim 1, characterized in that, In step S5.2, the feature fusion process is as follows: S5.2.
1. The feature map M5 is processed through the SPPCSPC module to obtain the feature map K5. The feature map K5 is upsampled and fused with the feature map M4. The fusion result is input into the ELAN-W module to obtain the feature map K4. S5.2.2 Upsample feature map K4 and fuse it with feature map M3. Input the fusion result into the ELAN-W module to obtain the shallowest output feature map P3. S5.2.3 Input feature map P3 into the MP module for downsampling and fuse it with feature map K4. Input the fusion result into the ELAN-W module to obtain the intermediate layer output feature map P4. S5.2.4 Input feature map P4 into the MP module for downsampling and fuse it with feature map K5. Input the fusion result into the ELAN-W module to obtain the deepest output feature map P5. S5.2.5 For the obtained feature maps P3, P4, and P5, input the adaptive scaling head ASGH for prediction.
7. A pedestrian detection method based on YOLOv7 as described in any one of claims 1-5, characterized in that, In step S6, the nine prior boxes are evenly distributed to three different scales of output feature maps according to their size. Each scale of the network output feature map is divided into a different number of grid cells, with the center position of the grid as the reference. For each grid cell, a corresponding candidate box or prediction box is generated according to its center position and the size of the assigned prior box.
8. The pedestrian detection method based on YOLOv7 as described in claim 7, characterized in that, In step S7, the network loss value is calculated based on the network output feature map, predicted bounding box, and ground truth bounding box from step S6, specifically as follows: The cross-union ratio (CUI) loss is calculated based on the predicted bounding boxes and ground truth (GT) boxes. The classification confidence loss and bounding box confidence loss are calculated based on the classification confidence and bounding box confidence of each predicted bounding box contained in the network output feature map. The CUI loss, classification confidence loss, and bounding box confidence loss are then weighted and summed according to a preset ratio to obtain the overall network loss. Backpropagation is then performed to optimize the network parameters.
9. A pedestrian detection method based on YOLOv7 as described in any one of claims 1-5, characterized in that, Step S9, obtaining the final prediction box from all adjusted test set prediction boxes, specifically includes the following steps: S9.1 Sort all adjusted prediction boxes according to their confidence scores; S9.2 Use the nonmaximum suppression method to remove redundant boxes to obtain the final predicted boxes.