Pedestrian detection method and system for autonomous driving under complex background

Abstract

The present invention provides a method and system for pedestrian detection in autonomous driving under complex backgrounds. The method comprises: acquiring a dataset consisting of pedestrian images in different backgrounds and with varying degrees of occlusion; constructing an AD‑YOLO‑RPBNet model, the backbone of which comprises a serial structure of four convolutional blocks and diverse branch C3K2 modules; introducing one diverse branch C3K2 module in the first column of the model's neck, and alternating three diverse branch C3K2 modules in the second column; the output of each diverse branch C3K2 module in the second column is connected to a detection head through a DELA module; and training the model using the acquired dataset. The trained model is then used for pedestrian detection in autonomous driving under complex backgrounds. The model maintains high accuracy and robustness in various complex backgrounds while also enabling real-time processing, thereby improving the performance of vehicle-to-pedestrian detection in complex background environments.

Description

Technical Field

[0001] The present invention relates to the field of vehicle target detection and processing technology, and in particular to a pedestrian detection method and system for autonomous driving under the influence of complex backgrounds. Background Art

[0002] In the field of vehicle object detection and processing technology, accurately identifying and locating pedestrians is crucial for autonomous driving, intelligent transportation systems, and other fields. Pedestrian detection is a key technology in autonomous driving and is crucial for achieving highly automated driving. In autonomous driving systems, accurately detecting and identifying road conditions through precise object detection is a critical task for ensuring safe driving. In particular, after detecting pedestrians on the road, the system needs to make fast and accurate decisions based on real-time traffic conditions, enabling the autonomous vehicle to avoid obstacles by automatically controlling vehicle speed and braking.

[0003] Traditional pedestrian detection methods primarily rely on hand-crafted feature extractors and classifiers. While these methods can achieve some success under specific conditions, they often underperform in complex scenarios. For example, traditional methods based on features like edge detection tend to fail in dynamic scenes and complex backgrounds, resulting in inaccurate feature extraction and, consequently, poor detection performance.

[0004] With the development of deep learning technology, pedestrian detection methods based on convolutional neural networks (CNNs) and the You Only Look Once (YOLO) family of network models have attracted widespread attention due to their powerful feature learning capabilities. These methods can automatically learn features from images and are adaptable to dynamic and complex scenes. However, existing methods can easily interfere with pedestrian detection algorithms when the background contains a large number of cluttered objects or textures, or when pedestrians have similar colors or textures to the background, leading to false or missed detections. Moving objects in the background can also cause instability in the background model, and these objects may be mistakenly detected as pedestrians, affecting the accuracy and robustness of target detection.

[0005] Therefore, developing a pedestrian detection method and system that can effectively cope with the influence of complex backgrounds in autonomous driving, realizing high-precision pedestrian target detection under the influence of complex backgrounds during the driving process of autonomous driving vehicles, completing effective identification of pedestrians during autonomous driving, and thus accurately identifying pedestrian targets during driving for automatic vehicle obstacle avoidance, has important practical significance and application value. Summary of the Invention

[0006] In view of the shortcomings of the existing technology, the purpose of this invention is to provide a method and system for pedestrian detection in autonomous driving under complex backgrounds, addressing the aforementioned significant shortcomings of the existing technology. The method and system can maintain high accuracy and robustness in various complex backgrounds, while also possessing real-time processing capabilities to meet the needs of autonomous driving in identifying pedestrians and achieving automatic obstacle avoidance, thereby improving the performance of vehicle-to-vehicle pedestrian detection in complex background environments.

[0007] In order to solve the above technical problems, the technical solutions of the present invention are as follows:

[0008] In a first aspect, the present invention provides a method for detecting pedestrians in autonomous driving under complex background conditions, the method comprising the following steps:

[0009] Step 1: Get the dataset.

[0010] We used binocular vehicle-mounted cameras to capture images of actual pedestrian traffic at different times in shopping malls, bus stops, and on different streets. We also collected pedestrian images from the internet under different backgrounds and with varying degrees of occlusion in the city. These images, including simple and complex backgrounds with varying degrees of occlusion, form a dataset. We used the LabelImg tool to annotate the images in the dataset, identifying pedestrian targets in the images, including their location boxes and category information.

[0011] Step 2: Build the AD-YOLO-RPBNet model.

[0012] The backbone of the AD-YOLO-RPBNet model consists of four serial structures consisting of convolutional blocks and diverse branch C3K2 modules. In the neck of the AD-YOLO-RPBNet model, a diverse branch C3K2 module is introduced in the first column, and three diverse branch C3K2 modules are introduced alternately in the second column. The output of each diverse branch C3K2 module in the second column is connected to a detection head through a DELA module.

[0013] The diverse branch C3K2 module includes a convolution block, from which two branches are separated. One branch is processed by two diverse branch C3K modules and then concatenated with the other branch. After that, a 1*1 convolution operation is performed to obtain the output.

[0014] The diverse branch C3K module includes a sequentially connected convolution block and two bottleneck layers; the bottleneck layer includes two sequentially connected diverse branch convolutions;

[0015] Each diverse branch convolution consists of five branches. The first branch passes through a 1*1 convolution and a BN layer before output. The second branch passes through a 1*1 convolution, a BN layer, a 3*3 convolution and a BN layer before output. The third branch passes through a 1*1 convolution, a BN layer, a maximum pooling layer and a BN layer before output. The fourth branch passes through a 3*3 convolution, a BN layer, an average pooling layer and a BN layer before output. The last branch passes through a 3*3 convolution and a BN layer before output. Finally, the outputs of the five branches are spliced and processed by a nonlinear layer to obtain the output of the diverse branch convolution.

[0016] The DELA module has three branches, two of which use X maximum pooling and Y average pooling in the spatial dimension to obtain feature vectors in the horizontal and vertical directions, respectively, and then generate attention weights in the horizontal and vertical directions after processing through a 1*1 convolution, a GN layer, and a Sigmoid function. The attention weights in the horizontal and vertical directions are weighted with the input of the DELA module according to the corresponding directions to obtain the output of the DELA module;

[0017] Step 3: Use the data set obtained in step 1 to train the AD-YOLO-RPBNet model, and use the trained AD-YOLO-RPBNet model for pedestrian detection in autonomous driving under complex backgrounds.

[0018] Furthermore, the neck part of the AD-YOLO-RPBNet model is composed of three columns. The first column is composed of upsample, concat splicing operation, diverse branch C3K2 module, upsample, and concat splicing operation connected from bottom to top. The second column is composed of diverse branch C3K2 module, convolution block, concat splicing operation, diverse branch C3K2 module, convolution block, concat splicing operation, and diverse branch C3K2 module connected from top to bottom. The third column is composed of three parallel DELA modules.

[0019] The outputs of the C2PSA module in the trunk are connected to the upsample at the bottom of the first column and the Concat splicing operation at the bottom of the second column of the neck part respectively; the outputs of the third diverse branch C3K2 module and the second diverse branch C3K2 module in the trunk are connected to the two Concat splicing operations in the first column of the neck part respectively; the output of the diverse branch C3K2 module in the first column of the neck part is connected to the first Concat splicing operation in the second column; at the same time, the output of the Concat splicing operation at the top of the first column is input to the diverse branch C3K2 module at the top of the second column; the three diverse branch C3K2 modules in the second column are each connected to a DELA module;

[0020] In the output part of the AD-YOLO-RPBNet model, three detection heads are used to process feature maps of three different scales, which can identify pedestrian targets of different sizes in an image and display the category information and bounding box of the detected targets in the image.

[0021] Furthermore, the AD-YOLO-RPBNet model uses an occlusion-aware focus loss L that considers occlusion class weights during training. OWFL as a loss constraint.

[0022] Furthermore, when starting training, the model initialization parameters are set as follows: the epoch of the training model is set to 650, the optimizer uses the adaptive gradient descent Adagrad optimizer, and the initial learning rate of the Adagrad optimizer is set to 0.01, and the gradient weight decay parameter is set to 5e-2; training is stopped when the loss change error is within ±1e-4.

[0023] Furthermore, the image size in the dataset is 640×640.

[0024] In a second aspect, the present invention provides a pedestrian detection system for autonomous driving under the influence of complex backgrounds. The system performs the steps of the pedestrian detection method for autonomous driving under the influence of complex backgrounds, including:

[0025] Binocular vehicle-mounted camera module, used to obtain pedestrian images under different road conditions;

[0026] AD-YOLO-RPBNet model, used for real-time pedestrian detection during autonomous driving;

[0027] The vehicle-to-person distance calculation module is used to measure the distance, speed, and angle between the vehicle and the target using the millimeter-wave radar sensor based on the target detection results;

[0028] The feedback module is used to prompt the driver according to the calculation results of the vehicle-pedestrian distance calculation module and adjust the vehicle's driving status.

[0029] Furthermore, the target detection results, the distance between the vehicle and the person, and the position of the target relative to the vehicle are displayed on the vehicle screen. The target detection results include confidence level and occlusion level.

[0030] The feedback module is triggered based on the data from the target detection module and the vehicle-pedestrian distance calculation module. The feedback module determines the level of graded broadcast according to the calculation results of the vehicle-pedestrian distance calculation module, and broadcasts the instructions to pay attention to pedestrians, slow down or emergency braking in a graded manner.

[0031] Furthermore, the AD-YOLO-RPBNet model is used in autonomous driving pedestrian detection in complex backgrounds, achieving a mean pixel accuracy (mAP) greater than 70% and a frame rate (FPS) greater than 80, while maintaining a parameter count between 15 and 20 million. Preferably, the mean pixel accuracy (mAP) is greater than 75% and the frame rate (FPS) is greater than 90.

[0032] Compared with the prior art, the present invention has the following beneficial effects:

[0033] To address the influence of complex backgrounds, this paper proposes an Automatic Driving YOLO Respond for Pedestrian Background Influence Object Detection Network (AD-YOLO-RPBNet) model, which demonstrates significant advantages in multiple aspects:

[0034] The proposed model utilizes multiple C3K2 modules with diverse branches, while also introducing three DELA (Multi-branch Efficient Local Attention) modules before the three detection heads, significantly improving pedestrian detection performance and accuracy. The different pooling methods used in the X (horizontal) and Y (vertical) axes within the DELA modules enable the model to focus more closely on the pedestrian region within the image while effectively suppressing the interference of irrelevant background information, thereby reducing the impact of background noise on pedestrian detection. This enhances the model's adaptability to diverse scenes, lighting conditions, and shooting angles, improving detection robustness. Furthermore, the strategic placement of the DELA modules significantly improves detection accuracy and speed while also enhancing the model's generalization and environmental adaptability. This provides an efficient and powerful solution for pedestrian detection, helping to reduce false detections, particularly for small or low-contrast objects. Importantly, these improvements are achieved without significantly increasing the computational burden, ensuring that pedestrian detection speed is not compromised.

[0035] In terms of loss function, this paper proposes occlusion-weighted focal loss to solve the pedestrian occlusion problem by assigning different occlusion category weights to different occlusion levels (such as heavy occlusion ω c = 2.0), forcing the model to focus more on feature learning of occluded samples during training, so that the model balances the contribution of occluded samples and normal samples during training, specifically solving the problem of pedestrian occlusion in complex backgrounds and achieving a balance between accuracy, robustness, and efficiency.

[0036] The AD-YOLO-RPBNet model in this invention can cope with complex background influences in real time and automatically detect the position of pedestrians in front of the autonomous vehicle. After detecting pedestrians, the AD-YOLO-RPBNet model can provide the autonomous vehicle with accurate vehicle-pedestrian distance calculation, and issue early warnings and adjust driving decisions based on the relative position of the vehicle and the target and the distance between the vehicle and the pedestrian, ensuring driving safety and efficiency under various complex background influences, effectively avoiding obstacles and optimizing driving efficiency.

[0037] The present invention completes the task of pedestrian target detection in response to complex background influences. By combining the vehicle networking technology, the system enables autonomous driving vehicles to make optimal and safe autonomous driving decisions in complex urban environments.

[0038] In summary, the AD-YOLO-RPBNet model of the present invention enriches the scale of feature extraction and improves the ability to capture important features in pedestrian target detection under complex backgrounds. It also achieves deeper pyramid-shaped feature fusion and extraction while reducing the amount of computation. It can better cope with pedestrian target detection tasks under complex backgrounds. It not only improves the accuracy of target detection, but also achieves lightweight and enhances its applicability in autonomous driving vehicles. It takes into account accuracy, computational effect and parameter minimization, and can run efficiently on resource-constrained devices. It has demonstrated excellent performance in the field of autonomous driving target detection and has broad application prospects. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Figure 1 Schematic diagram of the structure of the AD-YOLO-RPBNet model in one embodiment of the present invention.

[0040] Figure 2 This is a schematic structural diagram of a diverse branch C3K2 module according to an embodiment of the present invention.

[0041] Figure 3 This is a structural diagram of a DELA module according to an embodiment of the present invention. DETAILED DESCRIPTION

[0042] In order to more clearly describe the technical problems, technical solutions and advantages of the present invention, the following will be described in detail with reference to the figures and embodiments. It should be noted that these embodiments are only used to illustrate the principles and application scope of the present invention and should not be regarded as limiting the present invention.

[0043] In the description of this specification, specific features, structures, or characteristics described in each embodiment can be combined in an appropriate manner in any one or more embodiments or examples.

[0044] Example 1:

[0045] This embodiment provides a method for detecting pedestrians in autonomous driving under complex backgrounds, including the following steps:

[0046] Step 1: Get the dataset

[0047] We used binocular vehicle-mounted cameras to capture images of actual pedestrian traffic at different times in shopping malls, bus stops, and on different streets. We also collected pedestrian images from the internet under different backgrounds and with varying degrees of occlusion in the city. These images, including simple and complex backgrounds with varying degrees of occlusion, form a dataset. We used the LabelImg tool to annotate the images in the dataset, identifying pedestrian targets in the images, including their location boxes and category information.

[0048] And the blocked pedestrians are divided into three categories:

[0049] Mild occlusion refers to a pedestrian occlusion ratio of less than 30% (including 30%), with key body parts visible, including the head, shoulders, and torso. The occlusion only affects local areas and does not affect overall shape recognition. For example, a pedestrian stands behind a road sign with a metal pole covering the waist; a pedestrian pushes a bicycle with the handlebar covering the right forearm; and branches in a park cover about 20% of the pedestrian's left shoulder.

[0050] Moderate occlusion refers to a pedestrian occlusion ratio of more than 30% and less than 60% (inclusive). The pedestrian's main body structure is missing, such as limbs and torso, but logical inference can be made based on the remaining limbs or context. For example, two people are walking side by side, and the chest and abdominal areas of the pedestrian behind are blocked by the pedestrian in front; the pedestrian is crossing the road, and the front of the car blocks the waist to thigh area.

[0051] Severe obstruction refers to a pedestrian being obscured by more than 60%, with only fragmented body features remaining, making it difficult for the naked eye to quickly identify the pedestrian. For example, the side of a truck completely obscures the pedestrian, with only the tip of the left shoe and the corner of the right elbow exposed; a pedestrian holding an umbrella in a rainstorm, with only the lower half of the calf and sneakers exposed.

[0052] Step 2: Build the AD-YOLO-RPBNet model.

[0053] The AD-YOLO-RPBNet model consists of three parts: backbone, neck and output.

[0054] The backbone consists of a head composed of convolution blocks with a kernel size of 2, an intermediate unit, and a tail composed of a series of SPPF modules and C2PSA modules. The intermediate unit includes four groups of serial structures composed of convolution blocks and diverse branch C3K2 modules.

[0055] The neck part consists of three columns. The first column is composed of upsampling Upsample, Concat splicing operation, diverse branch C3K2 module, upsampling Upsample, and Concat splicing operation connected from bottom to top. The second column is composed of diverse branch C3K2 module, convolution block, Concat splicing operation, diverse branch C3K2 module, convolution block, Concat splicing operation, and diverse branch C3K2 module connected from top to bottom. The third column consists of three parallel DELA modules.

[0056] The outputs of the C2PSA module in the trunk are connected to the upsample at the bottom of the first column and the Concat splicing operation at the bottom of the second column of the neck part respectively; the outputs of the third diverse branch C3K2 module and the second diverse branch C3K2 module in the trunk are connected to the two Concat splicing operations in the first column of the neck part respectively; the output of the diverse branch C3K2 module in the first column of the neck part is connected to the first Concat splicing operation in the second column; at the same time, the output of the Concat splicing operation at the top of the first column is input to the diverse branch C3K2 module at the top of the second column; the three diverse branch C3K2 modules in the second column are each connected to a DELA module;

[0057] In the output part, the outputs of the three DELA modules in the third column of the neck part are processed by three detection heads on feature maps of three different scales. This allows the identification of pedestrian targets of different sizes in an image and displays the category information and bounding box of the detected targets in the image.

[0058] The diverse branch C3K2 module includes a convolution block, from which two branches are separated. One branch is processed by two diverse branch C3K modules and then concatenated with the other branch. After that, a 1*1 convolution operation is performed to obtain the output.

[0059] The diverse branch C3K module includes a main path consisting of a convolutional block and two bottleneck layers connected in sequence. The outputs of the convolutional block and the two bottleneck layers are used as three branches. The three branches are concatenated with the output of the main path and then subjected to a 1*1 convolution process to obtain the output of the diverse branch C3K module.

[0060] The bottleneck layer includes two sequentially connected diversified branch convolutions, and the output of the bottleneck layer is obtained by splicing the output and input of the two diversified branch convolutions;

[0061] The diverse branch convolution consists of five branches. The first branch passes through a 1*1 convolution and a BN layer and is output. The second branch passes through a 1*1 convolution, a BN layer, a 3*3 convolution and a BN layer and is output. The third branch passes through a 1*1 convolution, a BN layer, a maximum pooling layer and a BN layer and is output. The fourth branch passes through a 3*3 convolution, a BN layer, an average pooling layer and a BN layer and is output. The last branch passes through a 3*3 convolution and a BN layer and is output. Finally, the outputs of the five branches are spliced and then processed by a nonlinear layer and output to obtain the output of the diverse branch convolution.

[0062] The DELA module (see Figure 3 ) Set up three branches, two of which use X maximum pooling and Y average pooling in the spatial dimension to obtain feature vectors in the horizontal and vertical directions, respectively. Then, they are processed by a 1*1 convolution, a GN layer (group normalization), and a Sigmoid function to generate attention weights in the horizontal and vertical directions. The attention weights in the horizontal and vertical directions are weighted with the input of the DELA module according to the corresponding directions to obtain the output of the DELA module;

[0063] This completes the construction of the AD-YOLO-RPBNet model;

[0064] Step 3: Use the data set obtained in step 1 to train the AD-YOLO-RPBNet model, and use the trained AD-YOLO-RPBNet model for pedestrian detection in autonomous driving under complex backgrounds.

[0065] In step 3, the occlusion-aware focus loss L is used during training. OWFL As a loss constraint, the occlusion-aware focal loss L OWFL Expressed as:

[0066] ;

[0067] Where N is the number of samples and C is the number of categories; is the true label of the i-th sample belonging to category c (one-hot encoding, only one element is 1 and the rest are 0); is the predicted probability that the i-th sample belongs to category c, is the occlusion category weight. When the category is no occlusion, background interference or pedestrians are too dense, it is considered to be light occlusion. =1.0; when the image is slightly occluded =1.0, when the image is moderately occluded =1.5, when the image is heavily occluded =2.0. α is a balancing factor used to adjust the weights between positive and negative samples, with a default value of α=2.0; γ is a focusing parameter used to control the degree of attention paid to difficult samples, with a default value of γ=4.0.

[0068] Example 2:

[0069] This embodiment provides a method for detecting pedestrians in autonomous driving under complex backgrounds. The method uses the Automatic Driving YOLO Respond for Pedestrian Background Influence Object Detection Network (AD-YOLO-RPBNet) model and includes the following steps:

[0070] Step 1: Get the dataset

[0071] A binocular vehicle-mounted camera was used to capture images in various urban settings, such as crowded streets, shopping malls, bus stops, and near schools. Actual street conditions were captured on several streets at different time periods, capturing images of both simple and complex backgrounds with varying degrees of occlusion. Images with simple and complex backgrounds were then categorized as lightly, moderately, or heavily occluded, based on the degree of pedestrian occlusion. Furthermore, complex background images included images with varying degrees of occlusion, as well as images with background interference and dense pedestrian traffic. These two conditions were considered unoccluded. The dataset for this example consists of 12,000 images, including 4,000 images of pedestrians with simple backgrounds, including 1,000 images with light occlusion, 1,500 images with moderate occlusion, and 1,500 images with heavy occlusion. The dataset also contains 8,000 images of pedestrians with complex backgrounds, including 2,000 images with light occlusion, 2,000 images with moderate occlusion, and 2,000 images with heavy occlusion. Furthermore, the dataset also contains 1,000 images with background noise and 1,000 images with dense pedestrians. The images in the dataset were annotated using the LabelImg tool to identify the pedestrian targets in the images, including their location, size, and category. The dataset was randomly divided into training and test sets in a 7:3 ratio. To ensure consistent image size when inputting into the network model, all images were set to 640×640.

[0072] Step 2: Build the AD-YOLO-RPBNet model, which includes the backbone, neck, and output.

[0073] The backbone consists of a head, an intermediate unit, and a tail consisting of a series of SPPF (Spatial Pyramid Pooling-Fast) modules and C2PSA modules. The intermediate unit consists of four serial structures consisting of convolutional blocks and diverse branch C3K2 modules. Each convolutional block consists of a 2D convolution, a 2D BN layer, and a SiLU activation function in series. The structure of the diverse branch C3K2 module can be found in [1]. Figure 2 .

[0074] The input image is first processed by the backbone part, and the convolution block with a convolution kernel size of 2 at the head of the backbone part performs initial feature extraction and expands its number of channels. The input image of size 640×640×3 (height, width, number of channels) is extracted into an initial feature map of size 320×320×16. Next, the initial feature map is subjected to deep feature extraction through the first set of serial structures consisting of convolutional blocks and diverse branch C3K2 modules through the intermediate units of the backbone, and the output feature map with a size of 160×160×32 is obtained. ; Then, the feature map is processed by the second set of serial structures Perform feature extraction again to obtain a feature map of size 80×80×64 ; The next step is to use the third group of serial structures to map the feature map Perform the same feature extraction operation to obtain a feature map of size 40×40×128 ; The last set of serial structure pair feature maps Feature extraction is performed to obtain a feature map of size 20×20×256 , and then the SPPF module performs feature mapping Perform spatial pyramid pooling, and the pooling result is processed by the C2PSA module to obtain a feature map of size 20×20×256 containing multi-scale information ,by Serves as part of the input to the next stage.

[0075] The neck part consists of two columns of structures with input and output in reverse order and one column of DELA modules. The first column is a feature map. As input, after an upsample, a feature map of size 40×40×256 is obtained. , after an upsample process and then combined with the feature map The splicing is performed to obtain a feature map of size 40×40×384 ; Then the feature map As input, a feature map of size 40×40×128 is obtained by feature extraction through a multi-branch C3K2 module. , after an upsample process and then combined with the feature map The splicing is performed to obtain a feature map of size 80×80×192 The second column is the feature map As input, a feature map of size 80×80×64 is obtained through feature extraction through a multi-branch C3K2 module. , feature map Then a convolution block is used to extract features to obtain a feature map of size 40×40×64. , feature map With feature map The splicing is performed to obtain a feature map of size 40×40×192 ; Next, the feature map Feature extraction is performed through a diverse branch C3K2 module to obtain a feature map of size 40×40×128 , feature map After a convolution block feature extraction, a feature map of size 20×20×128 is obtained. , the feature map With feature map The splicing is performed to obtain a feature map of size 20×20×384 ; Final feature map Feature extraction is performed through a diverse branch C3K2 module to obtain a feature map of size 20×20×256 The third column consists of feature maps After the DELA module, a feature map of size 80×80×64 is obtained. , by the feature map After the DELA module, a feature map of size 40×40×128 is obtained. , by the feature map After the DELA module, a feature map of size 20×20×256 is obtained. .

[0076] The output part includes three detection heads, which are respectively based on feature maps , feature map , feature map As input, by processing feature maps of three different scales, it is possible to identify pedestrian targets of different sizes in an image and display the location, category information and bounding box of the detected target in the image.

[0077] Step 3: Use the data set obtained in step 1 to train the AD-YOLO-RPBNet model, and use the trained AD-YOLO-RPBNet model for pedestrian detection in autonomous driving under complex backgrounds.

[0078] Example 3:

[0079] This embodiment addresses the pedestrian detection method for autonomous driving under complex backgrounds. The following steps are used to detect pedestrians under complex backgrounds and achieve automatic obstacle avoidance for the autonomous driving vehicle:

[0080] 1. Data collection and preparation stage

[0081] 1.1 Data Collection

[0082] Equipment: A binocular vehicle-mounted camera was used to capture real-world road conditions. Images with both simple and complex backgrounds were captured on various streets and at different time periods. Simple background images primarily featured clear pedestrian outlines, a small number of pedestrians (less than five), and varying degrees of occlusion. Complex background images primarily featured cluttered backgrounds, background textures similar to those of pedestrians, overly dense pedestrians, or obscured pedestrian outlines. The binocular vehicle-mounted camera was used to capture images in various urban settings, such as crowded streets, shopping malls, bus stops, and near schools.

[0083] The pedestrian's outline is obscured to different degrees of occlusion: mild occlusion, the pedestrian's occlusion ratio is less than 30%, and key body parts are visible; moderate occlusion, the pedestrian's occlusion ratio is above 30% and below 60%, and the pedestrian's main body structure is missing; severe occlusion, the pedestrian's occlusion ratio is above 60%, and only fragmented body features are retained.

[0084] In this embodiment, there are eight categories: simple background with light occlusion, simple background with moderate occlusion, simple background with heavy occlusion, background interference (including cluttered background, background texture similar to pedestrians), too many pedestrians, complex background with light occlusion, complex background with moderate occlusion, and complex background with heavy occlusion.

[0085] Operation: Capture pedestrian images for each street in the scene, ensuring coverage of images under varying background influences and multiple sets of pedestrian target images. For example, in this embodiment, the target is pedestrian images under varying background conditions on the road. The target is locked onto and captured from different angles using a binocular vehicle-mounted camera. Pedestrian images under varying backgrounds and with varying degrees of occlusion are then collected online and combined with the captured images to create the dataset used in this embodiment.

[0086] 1.2 Data Processing

[0087] Dataset Acquisition: The collected image data undergoes preliminary processing, including removing obvious noise and outliers, and cropping and resizing the images to ensure consistency and standardization of the image dataset. The collected images are divided into training and test sets in a 7:3 ratio.

[0088] Data augmentation: Perform data augmentation operations such as rotation, scaling, and flipping on the training set (excluding adjustments such as brightness) to improve the robustness and generalization ability of the model.

[0089] 2. Model training phase

[0090] 2.1 Model Construction

[0091] The AD-YOLO-RPBNet model is constructed, which includes three parts: backbone, neck, and output. The specific structure is the same as that of Example 1.

[0092] 2.2 AD-YOLO-RPBNet model training

[0093] When starting training, first complete the parameter settings for network model initialization: the epoch of the training network model is set to 650, the optimizer uses the adaptive gradient descent Adagrad optimizer, and the initial learning rate of the Adagrad optimizer is set to 0.01, and the gradient weight decay parameter is set to 5e-2. During the training process, the training set images are input to AD-YOLO-RPBNet and read according to the training set storage path; when the overall network target loss function L OWFL When there is no longer a significant decrease (error ±1e-4), the network model training is considered stable and the training process is completed.

[0094] Loss function: The occlusion-weighted focal loss function is used to improve the target detection performance by assigning occlusion category weights to different occlusion levels.

[0095] When testing the model, input the prepared test set images, import the network weights trained in the training set, obtain the test results, and complete the model target detection purpose.

[0096] 3. Processing and analysis stage

[0097] 3.1 Model Deployment

[0098] Deployment: In this embodiment, a binocular vehicle camera module is constructed using a left camera and a right camera with a resolution of 1920×1080, 30 FPS, and a field of view (FOV) of 72.6°×50°×82° (±3°). The two cameras must be mounted in parallel, with their optical axes aligned to ensure stereoscopic accuracy. Calibration is then performed on the left and right cameras to obtain their intrinsic and extrinsic parameters, distortion coefficients, and homography matrices. Based on the calibration results, the two cameras are calibrated so that the images captured by the two cameras are in the same plane and parallel to each other. Disparity map conversion is then performed, and the images captured by the two cameras are converted into depth maps using a reprojection matrix to obtain coordinates in three-dimensional space, completing the preparation of the binocular vehicle camera module.

[0099] The trained AD-YOLO-RPBNet model is deployed in autonomous vehicles for real-time pedestrian detection while driving. Driving on urban roads at speeds up to 40 km / h, the autonomous vehicle's object detection system detects pedestrians in real-time road conditions using images transmitted by the binocular onboard camera module. After identifying pedestrians, the recognition results are transmitted to the vehicle-pedestrian distance calculation module, which calculates the distance between the vehicle and the identified pedestrian. The feedback module then adjusts the vehicle's driving state based on the varying distances.

[0100] 3.2 Results Analysis

[0101] Model quality evaluation: The target detection performance is evaluated using mAP (average pixel accuracy) and FPS (frames per second), as well as the number of parameters.

[0102] Using a dataset of 12,000 pedestrian images captured by the same binocular vehicle camera module under different backgrounds, the AD-YOLO-RPBNet model of the present invention was compared with existing methods using the same loss function. The comparison results show that the object detection method of the present invention has lower computational complexity and better performance than YOLOv11, FasterRCNN, CenterNet, YOLOv8, or models with different DELA module positions. Table 1 shows the comparative results of different models trained for pedestrian detection.

[0103]

[0104] The results shown in Table 1 demonstrate that the AD-YOLO-RPBNet model achieves higher mAP than the existing, highly-performing YOLOv11, YOLOv8, CenterNet, and FasterRCNN models when detecting objects in a dataset of images of pedestrians on roads captured by a binocular vehicle camera module against various backgrounds. Furthermore, the model achieves a higher frame rate per second (FPS) while maintaining a lower parameter count. This significant difference demonstrates the superior performance of the proposed detection method compared to the aforementioned similar object detection methods.

[0105] The present invention introduces diverse branch convolution into the diverse branch C3K2 module, further enhancing the model's feature extraction capabilities, improving detection accuracy, maintaining a lightweight design, and enhancing adaptability. It also works in conjunction with the DELA module to improve the model's overall performance, making it easy to handle the aforementioned complex situations. In terms of the loss function, an occlusion-aware focus loss function is used. By assigning different occlusion category weights to different occlusion levels, the model is forced to focus more on feature learning of occluded samples during training, balancing the contributions of occluded samples with those of ordinary samples during training. This specifically addresses the problem of pedestrian occlusion in complex backgrounds, achieving a balance between accuracy, robustness, and efficiency. By combining diverse branch structures, the diverse branch convolution can capture richer feature representations and utilize complex branch structures to enhance feature learning, thereby improving the model's detection accuracy without significantly increasing computational complexity. The introduction of diverse branch convolution enables the model to accurately detect pedestrians under the influence of complex backgrounds, especially in urban environments where the background contains a large number of cluttered objects or textures, or where pedestrians and the background have similar colors or textures, or where moving objects in the background or changing lighting conditions can cause background model instability.

[0106] Example 4:

[0107] This embodiment addresses the pedestrian detection system for autonomous driving under complex backgrounds, including:

[0108] Binocular vehicle-mounted camera module, used to obtain pedestrian images under different road conditions;

[0109] AD-YOLO-RPBNet model for real-time pedestrian detection;

[0110] The vehicle-to-person distance calculation module uses a millimeter-wave radar sensor to measure the distance, speed, and angle between the vehicle and the target based on target detection results. By transmitting and receiving millimeter-wave signals, it measures the distance, angle, and other information between the vehicle and the target. It features high precision, high resolution, and strong anti-interference capabilities, making it suitable for various road and weather conditions.

[0111] The feedback module is used to prompt the driver according to the calculation results of the vehicle-pedestrian distance calculation module and adjust the vehicle's driving status.

[0112] The target detection results, vehicle-pedestrian distance, and position of the target relative to the vehicle are displayed on the vehicle screen. The target detection results include confidence level and occlusion level, for example, "a 60% occluded pedestrian is detected 10 meters in front of the right." The feedback module is triggered based on the target detection results and the data of the vehicle-pedestrian distance calculation module. The feedback module determines the level of graded broadcast according to the calculation results of the vehicle-pedestrian distance calculation module, and broadcasts the instructions of "watch out for pedestrians", "slow down" or "emergency brake" in a graded manner.

[0113] When the distance to a pedestrian is greater than 15 meters, it is considered a mild risk. The feedback module will display a warning on the vehicle screen to alert the driver, and at the same time, a voice broadcast will say "Watch out for pedestrians."

[0114] When the distance to a pedestrian is 5-15 meters, it is considered a moderate risk. The feedback module will display a warning on the vehicle screen and announce the "slow down" instruction to the driver. At the same time, it will automatically apply light brakes and tighten the seat belt pretensioners.

[0115] When the distance to a pedestrian is less than 5 meters, it is a severe risk. The feedback module will trigger automatic emergency braking with maximum braking force output, and then alert the driver by broadcasting the "emergency braking" command and displaying an early warning on the on-board screen.

[0116] Example 5:

[0117] The AD-YOLO-RPBNet model is applied to traffic flow monitoring, especially its ability to detect and count pedestrian targets under different background conditions.

[0118] Deployment: The trained AD-YOLO-RPBNet model is deployed into the urban traffic monitoring system for real-time pedestrian flow detection and counting on key roads.

[0119] This example verifies the AD-YOLO-RPBNet model's ability to detect and count pedestrian targets under different background conditions, providing a new solution for pedestrian flow monitoring on key urban roads.

[0120] The present invention aims to address the existing defects of current pedestrian target detection technology, such as insufficient target detection accuracy and slow processing speed caused by the influence of complex backgrounds. The AD-YOLO-RPBNet model in the present invention is used to realize pedestrian target detection under the influence of complex backgrounds, improve recognition efficiency, save computing resources, and thus realize timely distance control through driving feedback of autonomous vehicles, thereby improving driving efficiency.

[0121] The technical solution of the present invention improves the target detection efficiency and avoids the influence of complex background, which is of great significance to the efficiency and safety of pedestrian target detection in autonomous driving vehicles.

[0122] Any matters not described in the present invention are applicable to the prior art.

Claims

1. A pedestrian detection method for autonomous driving under complex background conditions, characterized in that: The detection method comprises the following steps: Step 1: Get the dataset. We used binocular vehicle-mounted cameras to capture images of actual pedestrian traffic at different times in shopping malls, bus stops, and on different streets. We also collected pedestrian images from the internet under different backgrounds and with varying degrees of occlusion in the city. These images, including simple and complex backgrounds with varying degrees of occlusion, form a dataset. We used the LabelImg tool to annotate the images in the dataset, identifying pedestrian targets in the images, including their location boxes and category information. Step 2: Build the AD-YOLO-RPBNet model. The backbone of the AD-YOLO-RPBNet model consists of four serial structures consisting of convolutional blocks and diverse branch C3K2 modules. In the neck of the AD-YOLO-RPBNet model, a diverse branch C3K2 module is introduced in the first column, and three diverse branch C3K2 modules are introduced alternately in the second column. The output of each diverse branch C3K2 module in the second column is connected to a detection head through a DELA module. The diverse branch C3K2 module includes a convolution block, from which two branches are separated. One branch is processed by two diverse branch C3K modules and then concatenated with the other branch. After that, a 1*1 convolution operation is performed to obtain the output. The diverse branch C3K module includes a sequentially connected convolution block and two bottleneck layers; the bottleneck layer includes two sequentially connected diverse branch convolutions; Each diverse branch convolution consists of five branches. The first branch passes through a 1*1 convolution and a BN layer before output. The second branch passes through a 1*1 convolution, a BN layer, a 3*3 convolution and a BN layer before output. The third branch passes through a 1*1 convolution, a BN layer, a maximum pooling layer and a BN layer before output. The fourth branch passes through a 3*3 convolution, a BN layer, an average pooling layer and a BN layer before output. The last branch passes through a 3*3 convolution and a BN layer before output. Finally, the outputs of the five branches are spliced and processed by a nonlinear layer to obtain the output of the diverse branch convolution. The DELA module has three branches, two of which use X maximum pooling and Y average pooling in the spatial dimension to obtain feature vectors in the horizontal and vertical directions, respectively, and then generate attention weights in the horizontal and vertical directions after processing through a 1*1 convolution, a GN layer, and a Sigmoid function. The attention weights in the horizontal and vertical directions are weighted with the input of the DELA module according to the corresponding directions to obtain the output of the DELA module; Step 3: Use the data set obtained in step 1 to train the AD-YOLO-RPBNet model, and use the trained AD-YOLO-RPBNet model for pedestrian detection in autonomous driving under complex backgrounds.

2. The pedestrian detection method for autonomous driving under complex background according to claim 1, characterized in that: The neck part of the AD-YOLO-RPBNet model is composed of three columns. The first column is composed of upsampling Upsample, Concat splicing operation, diverse branch C3K2 module, upsampling Upsample, and Concat splicing operation connected in sequence from bottom to top. The second column is composed of diverse branch C3K2 module, convolution block, Concat splicing operation, diverse branch C3K2 module, convolution block, Concat splicing operation, and diverse branch C3K2 module connected in sequence from top to bottom. The third column is composed of three parallel DELA modules. The outputs of the C2PSA module in the trunk are connected to the upsample at the bottom of the first column and the Concat splicing operation at the bottom of the second column of the neck part respectively; the outputs of the third diverse branch C3K2 module and the second diverse branch C3K2 module in the trunk are connected to the two Concat splicing operations in the first column of the neck part respectively; the output of the diverse branch C3K2 module in the first column of the neck part is connected to the first Concat splicing operation in the second column; at the same time, the output of the Concat splicing operation at the top of the first column is input to the diverse branch C3K2 module at the top of the second column; the three diverse branch C3K2 modules in the second column are each connected to a DELA module; In the output part of the AD-YOLO-RPBNet model, three detection heads are used to process feature maps of three different scales, which can identify pedestrian targets of different sizes in an image and display the category information and bounding box of the detected targets in the image.

3. The pedestrian detection method for autonomous driving under complex background conditions according to claim 1, characterized in that: The AD-YOLO-RPBNet model uses occlusion-aware focal loss that considers occlusion class weights as a loss constraint during training.

4. The pedestrian detection method for autonomous driving under complex background according to claim 1, characterized in that: When starting training, the model initialization parameters are set as follows: the epoch of the training model is set to 650, the optimizer uses the adaptive gradient descent Adagrad optimizer, and the initial learning rate of the Adagrad optimizer is set to 0.01, and the gradient weight decay parameter is set to 5e-2; training is stopped when the loss change error is within ±1e-4.

5. The pedestrian detection method for autonomous driving under complex background conditions according to claim 1, characterized in that: The image size in the dataset is 640×640.

6. A pedestrian detection system for autonomous driving under complex background conditions, characterized in that: The system executes the steps of the pedestrian detection method for autonomous driving under complex background influences as described in any one of claims 1 to 5, including: Binocular vehicle-mounted camera module, used to obtain pedestrian images under different road conditions; AD-YOLO-RPBNet model, used for real-time pedestrian detection during autonomous driving; The vehicle-to-person distance calculation module is used to measure the distance, speed, and angle between the vehicle and the target using the millimeter-wave radar sensor based on the target detection results; The feedback module is used to prompt the driver according to the calculation results of the vehicle-pedestrian distance calculation module and adjust the vehicle's driving status.

7. The pedestrian detection system for autonomous driving under complex background conditions according to claim 6, characterized in that: The target detection results, vehicle-to-person distance, and the position of the target relative to the vehicle are displayed on the vehicle screen. The target detection results include confidence level and occlusion level. The feedback module is triggered based on the results of target detection and the data of the vehicle-pedestrian distance calculation module. The feedback module determines the level of graded broadcast according to the calculation results of the vehicle-pedestrian distance calculation module, and broadcasts the instructions to pay attention to pedestrians, slow down or emergency braking instructions in a graded manner.

8. The pedestrian detection system for autonomous driving under complex background conditions according to claim 6, characterized in that: The AD-YOLO-RPBNet model is used in autonomous driving pedestrian target detection in complex backgrounds. The average pixel accuracy (mAP) is greater than 70%, the frame rate (FPS) is greater than 80, and the number of parameters is controlled between 15 and 20 million.

Images