Lightweight pedestrian detection method based on BASFPN multi-scale feature fusion

Abstract

The invention provides a lightweight pedestrian detection method based on BASFPN multi-scale feature fusion, and belongs to the technical field of artificial intelligence and computer vision target detection. The technical problems of low precision and insufficient robustness in the aspect of detection of small target pedestrians and shielded pedestrians are solved. Comprising the following steps: S1, acquiring a pedestrian detection video, and preprocessing the video into an image; s2, performing data mixing enhancement processing on the acquired data; s3, constructing a lightweight network model MobileViT as a feature extraction network of the model; s4, constructing an FCOS network model; s5, a BASFPN feature pyramid layer is designed; and S7, inputting the data set into the improved model for training iteration. According to the invention, by improving the FPN structure and the fusion strategy, the utilization and expression ability of the model to different scale features is enhanced, so that the overall performance of pedestrian detection is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of artificial intelligence and computer vision target detection, and in particular to a lightweight pedestrian detection method based on BASFPN multi-scale feature fusion. Background Art

[0002] Pedestrian detection is an important research topic in the field of computer vision and plays a key role in practical applications such as video surveillance, autonomous driving, intelligent transportation and crowd counting. Traditional pedestrian detection methods mainly rely on manually designed feature extraction and classification algorithms, which show certain limitations when dealing with complex environments, especially in the case of lighting changes, occlusion and pedestrian posture diversity, the detection accuracy is often not ideal.

[0003] With the rise of deep learning technology, pedestrian detection methods based on convolutional neural networks (CNNs) have gradually become mainstream. Deep learning frameworks such as Faster R-CNN, YOLO, and SSD have achieved remarkable results in target detection tasks. However, these methods still face challenges in small target detection, processing of targets of different scales, and real-time performance. Especially in pedestrian detection tasks, pedestrians are usually small in scale and high in density, making it difficult for traditional deep learning detection models to accurately detect and classify them.

[0004] In recent years, feature pyramid networks (FPNs) have been widely used in the field of object detection. By fusing feature maps at different scales, the detection performance of detectors for multi-scale targets has been improved. For example, the FPN structure proposed by Tsung-Yi Lin et al. in the paper "Feature Pyramid Networks for Object Detection" (CVPR 2017) effectively improves the detection performance through bottom-up feature level fusion. However, this method still has some shortcomings in pedestrian detection tasks, especially when dealing with small-scale pedestrians and dense crowds, the detection accuracy and robustness are limited. The main reason is that the features of different scales in FPN are not fully fused, and the contextual information interaction between feature maps is less, resulting in insufficient feature expression capabilities.

[0005] In addition, S. Liu et al. pointed out in the paper "High-Level Semantic Feature Detection: A New Perspective for Pedestrian Detection" (CVPR 2019) that although deep learning models can extract high-level semantic features, the effectiveness of feature fusion in multi-scale pedestrian detection is still a challenge, especially for small targets and occluded pedestrians. This reflects the bottleneck of feature fusion in current pedestrian detection models, especially in complex scenes, how to effectively use multi-scale features becomes a key issue.

[0006] The present invention is proposed in this context, and aims to improve the overall performance of pedestrian detection by improving the FPN structure and fusion strategy, enhancing the model's ability to utilize and express features of different scales. Summary of the invention

[0007] The purpose of the present invention is to provide a lightweight pedestrian detection method based on BASFPN multi-scale feature fusion to overcome the deficiencies in the prior art, such as the difficulty in detecting small targets: the scale of pedestrians in the image is usually small, especially in long-distance cameras or complex scenes, pedestrians often only occupy a few pixels in the image. This makes the traditional detection method perform poorly when detecting small-scale targets. Challenges in occluded crowd detection: In crowded or high-density crowd scenes, due to the serious occlusion and overlap between pedestrians, traditional detection methods are difficult to effectively handle complex scenes of dense crowds; the effectiveness of feature fusion is insufficient: the existing feature fusion has less contextual information interaction between feature maps of different scales, resulting in insufficient model feature expression capabilities for small targets and occluded targets, affecting the overall detection performance. The present invention overcomes the deficiencies of the prior art, improves the accuracy and robustness of pedestrian detection, and achieves more comprehensive fusion and utilization of multi-scale features by improving the FPN structure of the existing algorithm, enhancing feature expression capabilities, thereby improving detection performance, and at the same time reducing the occupation of computing resources. Firstly, based on the anchor-free frame algorithm FCOS, an improvement is made by using hybrid data augmentation technology in the training process to improve the robustness of the model; secondly, for the feature extraction network, a lightweight convolutional network model is adopted. Compared with the common network, this model has fewer parameters and computational complexity, and has better detection effect while having a higher inference speed; finally, a dual-path attention separable feature pyramid BASFPN is proposed to perform multi-scale feature fusion to achieve effective detection of pedestrian targets, so that the model achieves a good balance in accuracy, speed and scale.

[0008] In order to achieve the above-mentioned invention object, the technical solution adopted by the present invention is specifically: a lightweight pedestrian detection method based on BASFPN multi-scale feature fusion, comprising the following steps:

[0009] S1: Obtain pedestrian detection video, pre-process it into images, standardize and normalize the images, and scale them to a fixed size to meet the model input requirements;

[0010] S2: Perform data enhancement processing on the acquired data and divide the data set into training set, validation set and test set;

[0011] S3: Build a lightweight network model MobileViT as the feature extraction network of the model;

[0012] S4: Construct the FCOS network model, which includes the input layer, backbone network layer, feature pyramid layer, head network layer and output layer;

[0013] S5: Design the BASFPN feature pyramid layer to replace the original FPN layer;

[0014] S6: Use Wise IOU loss function to replace Focal Loss;

[0015] S7: Input the data set into the improved model for training iterations until the loss value is stable. At the same time, input the validation set into the model to obtain the optimal model for training weights, and save the model parameter weights. Test the images in the test set and finally get the detection results.

[0016] Furthermore, the step S1: the public dataset Caltech is a vehicle camera video dataset, which converts the video file into an image file through preprocessing, wherein the image file is placed in a separate folder after processing, and the label file is placed in a single file after conversion. The label file contains the image name, image path, image label name, and target location coordinates. The label file and the corresponding image file are sampled repeatedly to delete the non-target file, reduce the sample training sample, and the image size is uniformly adjusted to 640×640 to complete the final data processing.

[0017] Furthermore, step S21: data enhancement is performed on the data set, wherein the most important data enhancement includes Random Erasing and Mosaic, wherein Random Erasing randomly selects the area to cover the pixel value to introduce a certain amount of noise, sets the pixel value to a random value or a fixed value to cover part of the image, and Mosaic randomly selects several pictures to be placed and spliced ​​in four directions, performs edge processing on the excess part, and keeps the label information consistent. In addition to these two enhancements, random scale scaling, length and width distortion, flipping, image color gamut transformation, etc. are performed during the training process to improve the robustness of the model, reduce the risk of model overfitting, and improve the training effect.

[0018] S22: The data set is divided into training set, validation set and test set according to the ratio of 7:1:2. The training set contains 10594 images, the validation set contains 1617 images and the test set contains 3329 images.

[0019] Furthermore, the step S3: constructing a lightweight network model MobileViT as a feature extraction network specifically comprises the following steps:

[0020] S31: Initialize the MobileViT network and use pre-trained weights to initialize the parameters of MobileViT to accelerate the convergence process of the model and improve the generalization ability of the model on the data set.

[0021] S32: Configure the feature extraction layer and adjust the size of the feature extraction layer of the network to meet the input requirements of the subsequent feature fusion pyramid. The effective feature layer height, width and channel number change matrix is:

[0022]

[0023] Furthermore, the step S5: designing a BASFPN feature pyramid for multi-scale feature fusion specifically comprises the following steps:

[0024] S51: The original FPN only uses one top-down path for feature fusion. By adding an additional bottom-up path for bidirectional fusion with the original path, the overall feature fusion capability of the model is improved.

[0025] S52: At the same time, the idea of ​​residual connection in residual network is used for skip connection to perform multi-scale feature fusion. In multi-scale feature fusion, simple addition or superposition processing is not used, but weighted feature fusion mechanism is used for differentiated fusion. Channel-by-channel convolution and point-by-point convolution are used to keep the model lightweight and maximize the retention of feature fusion and semantic richness between each level.

[0026] Furthermore, in step S6: the Focal Loss is replaced by the Wise IOU loss function. The Wise IOU loss function can more effectively handle the deviation of the model when predicting the boundary and improve the accuracy of target positioning. Wise IOU focuses on the overlapping area between the predicted bounding box and the true bounding box, provides accurate gradient updates in the bounding box regression process, and accurately fits the boundary of the target.

[0027] Furthermore, step S7: the data set is input for training iteration until the loss value is stable. The validation set obtains the optimal model of training weights and saves the model parameter weights. The test set is tested and the detection results are finally obtained. A total of 100 epochs are trained, using the Adam optimizer, the maximum learning rate of the model is set to 3e-4, the minimum learning rate is 3e-6, the internal momentum of the optimizer is set to 0.9, the model freeze training is 50, the batch is set to 8, the initial learning rate is 0.01, and after iterative training, the loss and accuracy of the model gradually stabilize, and the optimal model training parameters are saved.

[0028] Compared with the prior art, the present invention has the following beneficial effects:

[0029] 1. The present invention provides a lightweight pedestrian detection method based on BASFPN multi-scale feature fusion. The BASFPN feature fusion strategy is used to enhance the expression ability of multi-scale features. The model can more accurately identify pedestrian targets of different scales in complex scenes and improve the detection effect.

[0030] 2. The present invention adopts a lightweight MobileViT network as the feature extraction network, which greatly reduces the number of parameters and calculations of the model, so that the model can maintain good detection effects while having a high inference speed; the combination of hybrid data enhancement technology accelerates the convergence process of the model, improves the generalization ability on different data sets, and reduces the risk of overfitting.

[0031] 3. The present invention uses the Wise IOU loss function to replace the traditional Focal Loss, which improves the accuracy of the model's predicted bounding box and enhances the robustness of the model in occlusion and diverse scenarios, ultimately achieving a good balance between accuracy, speed, and scale. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] The accompanying drawings are used to provide further understanding of the present invention and constitute a part of the specification. They are used to explain the present invention together with the embodiments of the present invention and do not constitute a limitation of the present invention.

[0033] Figure 1 This is the overall flow chart for the FCOS-BASFPN model constructed in this paper.

[0034] Figure 2 This is the framework diagram of the BASFPN dual-path attention separable fusion strategy in the present invention.

[0035] Figure 3 This is the overall framework diagram of the final FCOS-BASFPN model in the present invention.

[0036] Figure 4This is a diagram showing the real-time detection effect of the model of the present invention on small-target pedestrians and occluded pedestrians in complex scenes. DETAILED DESCRIPTION

[0037] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. Of course, the specific embodiments described here are only used to explain the present invention and are not used to limit the present invention.

[0038] Example 1

[0039] As attached Figure 1 As shown, the technical solution provided by this embodiment is a lightweight pedestrian detection method based on BASFPN multi-scale feature fusion, comprising the following steps:

[0040] S1: Preprocess the input data, which includes the following steps:

[0041] S1.1: Obtain pedestrian detection videos from dashcams or public datasets such as Caltech, convert video files into image files, and perform standardization and normalization. Scale the images to a fixed size of 640×640 to meet the model input requirements.

[0042] S1.2: Read video data frame by frame, convert the format to RGB using the cvtColor() function, and finally convert it to Image for detection. The processed image files are placed in a separate folder, and the label file is converted into a single file containing the image name, image path, image label name, and target location coordinates. Delete files without targets in a loop to reduce the redundancy of the sample training set.

[0043] S2: Perform data enhancement processing on the acquired data and divide the data set into training set, validation set and test set. The specific method of this step is:

[0044] After the S2.1 image is converted to RGB, resize_image() is used to add gray bars to the image and scale the image to 640×640 without distortion. The dataset is subjected to various data enhancements including Random Erasing and Mosaic. Random erasing introduces noise or blocks part of the image by randomly selecting areas on the image and covering pixel values. Mosaic randomly selects several images for splicing, processes edges and keeps label information consistent. In addition, random scale scaling, length and width distortion, flipping, and image color gamut transformation are also included to further improve the robustness and generalization ability of the model.

[0045] S2.2: After data enhancement, the data set is divided into training set, validation set and test set in a ratio of 7:1:2; the training set is 10594 images, the validation set is 1617 images, and the test set is 3329 images. The data set is processed to generate train.txt and val.txt files containing the absolute path, number and specific coordinates of the detection target.

[0046] S3: Construction of lightweight network model MobileViT. This step specifically includes the following steps:

[0047] S3.1: Initialize the network and use pre-trained weights to initialize the MobileViT network to accelerate convergence and improve generalization ability.

[0048] S3.2: Configure the feature extraction layer and adjust the feature extraction layer of the MobileViT network to adapt to the input requirements of the subsequent feature fusion pyramid to ensure the change of the height and width channels of the effective feature layer. Specifically, the slicing operation is used to select the feature extraction layer with channels of 48, 64, and 320 as the input of the two-way attention separable feature pyramid.

[0049] S4: Construct the FCOS network model, including the input layer, backbone network layer, enhanced feature network layer, head network layer and output layer. Each layer is responsible for different functions, including feature extraction, multi-scale feature fusion and target classification regression positioning. The specific results of this step are shown in the attached figure. Figure 1 shown.

[0050] S5: Design the BASFPN feature pyramid, which includes the following steps:

[0051] S5.1: The original FPN structure uses feature layers of different scales for regression classification, namely P3, P4, P5, P6, and P7 layers. Each pixel in each layer will regress to a target within a fixed scale range. When the pedestrian target is far away and small, or the pedestrian target is occluded and displayed incompletely, the detection performance of the model will be reduced. In the detection algorithm, the pixel on the feature map can only be detected when the receptive field of the size of the mapping area in the original image is greater than or equal to the target size. The underlying feature receptive field is relatively small and contains rich geometric detail information, but due to frequent downsampling, the resolution is constantly reduced and detail information is lost. Therefore, on the basis of the top-down path, an additional bottom-up path is added to the original path for bidirectional fusion, and the idea of ​​residual connection in the residual network is used for jump connection to perform multi-scale feature fusion. In multi-scale feature fusion, simple addition or superposition processing is not used, but a weighted feature fusion mechanism is used for differentiated fusion. Channel-by-channel convolution and point-by-point convolution are used to keep the model lightweight and maximize the retention of feature fusion and semantic richness between levels. The framework of this step is shown in the attached figure. Figure 2 shown.

[0052] S5.2: Take out the three effective layers C3, C4, and C5 after feature extraction, and then obtain the input P3, P4, and P5 of multi-scale feature fusion through lightweight attention convolution. The original P6 and P7 outputs are simply the result of two downsampling from the P5 layer. This is to facilitate the output to the head network classification regression. There is no multi-scale feature fusion and does not contain rich feature semantic information. Therefore, in order to simplify the network, the original P6 and P7 layers are deleted. Because the low-level feature layer has more geometric detail information, the P6 and P7 layers are obtained by fusion of the downsampling attention mechanism after channel-by-channel convolution and point convolution downsampling on the basis of P5. Finally, the 5 effective feature layers are bidirectionally fused by upsampling and downsampling. Since P3 is located at the last layer of upsampling fusion and P7 is located at the last layer of downsampling fusion, there is no residual connection, and the semantic information fusion effect is not good enough. Therefore, the middle layer P5 is used as the residual path for multi-scale feature fusion.

[0053] S5.3: For upsampling fusion, take P7 as an example. After upsampling P7 and P6, the upsampled attention is fused with the original P6 for weighted feature fusion. P6, P5, P4, and P3 are upsampled and fused in sequence in a similar way. In order to distinguish the importance of different feature maps, simple addition or superposition processing is not used, but weighted fusion is adopted. The fusion method is: The range is [0,1]. The specific feature fusion calculation formula for each upsampling layer is as follows:

[0054]

[0055] Where w is a parameter of model learning, which is used to distinguish the importance of different features in the fusion process; P is the specific effective feature layer involved in feature fusion; USample operation is an upsampling operation; UE is an upsampling attention mechanism; Conv is a channel-by-channel convolution and a point-wise convolution.

[0056] S5.4: Downsampling fusion takes P3 as an example. After downsampling P3, the downsampling attention is fused with the upsampling fused P4 and the basic P4 connected horizontally by the residual jump to perform weighted feature fusion. The fusion method used is the same as above. The specific feature fusion calculation expression formula of each downsampling layer is as follows:

[0057]

[0058] In the formula, w is a parameter of model learning, which is used to distinguish the importance of different features in the fusion process; P is the specific effective feature layer involved in feature fusion; P′ is the feature layer of residual connection; DSample operation is a downsampling operation; DE is a downsampling attention mechanism; Conv is a channel-by-channel convolution and a point-by-point convolution. After the final fusion, 5 predicted feature maps are obtained and sent to the Head network to predict the regression box, category, and center degree of each point pixel by pixel. The overall framework of this step is shown in the attached figure. Figure 3 shown.

[0059] S6: Use Wise IOU loss function instead of Focal Loss. Wise IOU loss function can more effectively handle the deviation of the model when predicting the bounding box, and improve the accuracy of the model's positioning of the target. It pays more attention to the overlapping area between the predicted bounding box and the true bounding box, and can provide more accurate gradient updates in the bounding box regression process, helping the model to fit the boundary of the target more accurately. The definition of Wise IOU loss is:

[0060]

[0061] This loss gives a greater penalty when the IoU is low, improving the model's positioning ability. At the same time, according to the performance of the model at different training stages, the weight of the Wise IOU loss is adaptively adjusted to ensure that the model achieves the best balance between positioning and classification.

[0062] S7: Model training and testing. The dataset is input into the improved model for training iterations until the loss value stabilizes. The Adam optimizer is used in the training process, and the maximum learning rate is set to 3e-4, the minimum learning rate is set to 3e-6, the learning rate is adaptively adjusted, the momentum is set to 0.9, and the first 50 epochs of the frozen training use a smaller initial learning rate of 0.01, and the batch size is set to 8. During the training process, the validation set is input into the model to obtain the optimal model weights. Save the parameter weights of the optimal model. The test set is input into the model for the final detection result output. After training, the loss and accuracy of the model gradually stabilize, and finally a good balance is achieved in terms of accuracy, speed and scale. The model detection results are shown in the attached Figure 4 The above can improve the detection accuracy of small-target pedestrians and obscured pedestrians in complex scenes while ensuring the detection speed, and is suitable for fields such as intelligent monitoring, autonomous driving and intelligent transportation.

[0063] Example 2

[0064] In order to verify the effectiveness of the model, the detection effects of several benchmark models are compared, including Faster R-CNN, SSD, RetinaNet, YOLOV4, YOLOX-s, and FCOS. The detection effects of the models are shown in the following table.

[0065]

[0066] The above experimental results show that the model proposed in this paper has the best performance in terms of mAP index, reaching 96.31%, and FPS has also been greatly improved, reaching 52.35 frames / s. Compared with other models, the total number of parameters has reached the minimum, only 7.44M. Finally, a good balance is achieved between accuracy, speed and model scale.

[0067] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A lightweight pedestrian detection method based on BASFPN multi-scale feature fusion, characterized in that: The following steps are involved: S1: Obtain pedestrian detection video, pre-process it into images, standardize and normalize the images, and scale them to a fixed size to meet the model input requirements; S2: Perform data enhancement processing on the acquired data and divide the data set into training set, validation set and test set; S3: Build a lightweight network model MobileViT as the feature extraction network of the model; S4: Construct the FCOS network model, which includes the input layer, backbone network layer, feature pyramid layer, head network layer and output layer; S5: Design the BASFPN feature pyramid layer to replace the original FPN layer; S6: Use Wise IOU loss function to replace Focal Loss; S7: Input the data set into the improved model for training iterations until the loss value is stable. At the same time, input the validation set into the model to obtain the model with the optimal training weights. Save the model parameter weights, test the images in the test set, and finally obtain the detection results.

2. The lightweight pedestrian detection method based on BASFPN multi-scale feature fusion according to claim 1 is characterized in that: In step S1, the public dataset Caltech is a vehicle-mounted camera video dataset. The video files are converted into image files through preprocessing, wherein the image files are placed in a separate folder after processing, and the label files are placed in a single file after conversion. The label file contains the image name, image path, image label name and target location coordinates. The label files and corresponding image files are sampled repeatedly in a loop to delete files without targets, reduce sample training samples, and uniformly adjust the image size to 640×640 to complete the final data processing.

3. The lightweight pedestrian detection method based on BASFPN multi-scale feature fusion according to claim 1 is characterized in that: The step S2 specifically includes the following steps: S21: Data enhancement is performed on the dataset. Data enhancement includes Random Erasing and Mosaic. Random Erasing randomly selects area coverage pixel values ​​to introduce a certain amount of noise, sets the pixel values ​​to random values ​​or a fixed value to cover part of the image, and Mosaic randomly selects several images and places them in four directions for splicing. Edge processing is performed on the excess parts while keeping the label information consistent. In addition to these two enhancements, random scale scaling, length and width distortion, flipping, and image color gamut transformation are performed during the training process to improve the robustness of the model, reduce the risk of model overfitting, and improve the training effect. S22: The data set is divided into training set, validation set and test set according to the ratio of 7:1:2, with 10594 images in the training set, 1617 images in the validation set and 3329 images in the test set.

4. The lightweight pedestrian detection method based on BASFPN multi-scale feature fusion according to claim 1 is characterized in that: The step S3: constructing a lightweight network model MobileViT as a feature extraction network, specifically comprises the following steps: S31: Initialize the MobileViT network and use pre-trained weights to initialize the parameters of MobileViT to accelerate the convergence of the model and improve the generalization ability of the model on the data set; S32: Configure the feature extraction layer and adjust the size of the feature extraction layer of the network to meet the input requirements of the subsequent feature fusion pyramid. The effective feature layer height, width and channel number change matrix is:

5. The lightweight pedestrian detection method based on BASFPN multi-scale feature fusion according to claim 1 is characterized in that: The step S5: designing a BASFPN feature pyramid for multi-scale feature fusion specifically comprises the following steps: S51: The original FPN only uses one top-down path for feature fusion. By adding an additional bottom-up path for bidirectional fusion with the original path, the overall feature fusion capability of the model is improved; S52: At the same time, the idea of ​​residual connection in the residual network is adopted for skip connection to perform multi-scale feature fusion. In the multi-scale feature fusion, the weighted feature fusion mechanism is used for differentiated fusion. Channel-by-channel convolution and point-by-point convolution are used to keep the model lightweight and maximize the retention of feature fusion and semantic richness between each level.

6. The lightweight pedestrian detection method based on BASFPN multi-scale feature fusion according to claim 1 is characterized in that: In step S6, the Focal Loss is replaced by the Wise IOU loss function, which processes the deviation of the model when predicting the boundary and improves the accuracy of target positioning. The Wise IOU focuses on the overlapping area between the predicted bounding box and the true bounding box, provides precise gradient updates in the bounding box regression process, and accurately fits the boundary of the target.

7. The lightweight pedestrian detection method based on BASFPN multi-scale feature fusion according to claim 1 is characterized in that: In step S7, the data set input is trained iteratively until the loss value is stable, the validation set obtains the optimal model of training weights, the model parameter weights are saved, the test set is tested, and finally the detection results are obtained. A total of 100 epochs are trained, and the Adam optimizer is used. The maximum learning rate of the model is set to 3e-4, the minimum learning rate is 3e-6, the internal momentum of the optimizer is set to 0.9, the model freezing training is 50, the batch is set to 8, and the initial learning rate is 0.

01. After iterative training, the loss and accuracy of the model gradually stabilize, and the optimal model training parameters are saved.

Images