Pedestrian crossing detection method, device and electronic equipment

Abstract

The application discloses a crosswalk detection method and device and electronic equipment. The application uses a target detection network to perform target detection on a picture frame to obtain a crosswalk in the picture frame. The target detection network adopts an improved YOLOv8 network, and the network input parameters are optimized as follows: for a picture of a flat and unobstructed road surface in front, the horizon of the same batch of pictures is obtained, and the area above the horizon is cropped for batch cropping and label processing of the same batch of original labeled samples; an ECA mechanism is introduced into the improved YOLOv8 network, and the improved method combining reasonable cropping of the input picture and introduction of the attention mechanism enables the model to pay more attention to the key target area without losing the detection speed, thereby improving the accuracy of the model and enhancing the robustness of the algorithm in a complex scene.

Description

Technical Field

[0001] This invention relates to the field of intelligent transportation technology, and in particular to a method, device and electronic equipment for detecting pedestrian crossings. Background Technology

[0002] Pedestrian crossing detection is an important technology in intelligent transportation systems, specifically in autonomous driving or driver assistance systems. In driving scenarios, it involves real-time monitoring of pedestrian crossings in the vehicle's direction of travel. When a pedestrian crossing is detected ahead and preset warning conditions are met, an appropriate warning is issued.

[0003] In autonomous driving systems, the primary target refers to any object that poses a collision risk to the vehicle, including vehicles, pedestrians, and obstacles. The pedestrian crossing closest to the vehicle on its forward trajectory is defined as the primary target pedestrian crossing and serves as the monitoring target for pedestrian crossing warnings. Existing technologies typically employ Q-DNN networks, improved RetinaNet networks, and YOLO networks for pedestrian crossing detection.

[0004] However, existing pedestrian crossing detection methods suffer from the following technical challenges and limitations: In real-world scenarios, lighting and road conditions are complex and variable. For example, there are various lighting conditions such as cloudy, sunny, rainy, foggy, and nighttime, as well as varying road conditions including normal, congested, obstructed, and damaged roads. This limits the edge computing capabilities of existing pedestrian crossing detection methods. Furthermore, the diversity of traffic signs can interfere with pedestrian crossing detection, leading to false detections and frequent abnormal warnings. Multiple pedestrian crossings at intersections can also interfere with the identification of the primary target pedestrian crossing ahead of the vehicle. Therefore, improving the accuracy of pedestrian crossing detection in complex scenarios is a problem that the industry needs to solve. Summary of the Invention

[0005] The purpose of this invention is to provide a pedestrian crossing detection method, device, and electronic device to solve the problem that the accuracy of existing pedestrian crossing detection methods in complex scenarios needs to be improved.

[0006] In a first aspect, the present invention provides a method for detecting pedestrian crossings, comprising:

[0007] The system acquires the vehicle's CAN information stream, the real-time video stream captured by the forward-view camera, and configuration parameters; the CAN information stream includes the vehicle's real-time speed; the real-time video stream is segmented at a preset frame rate to obtain image frames; the configuration parameters include the forward-view camera calibration parameters acquired through offline calibration, alarm distance thresholds, and vehicle speed alarm thresholds;

[0008] The image frames are used to detect objects to obtain the pedestrian crossings in the image frames. The object detection network adopts an improved YOLOv8 network, and the network input parameters are optimized as follows: for images with flat road surfaces and no obstructions in front, the skyline of the same batch of images is obtained, and the original labeled samples of the same batch are batch-cropped and labeled to remove the area above the skyline; an ECA mechanism is introduced into the improved YOLOv8 network.

[0009] The pedestrian crossings are screened to determine the primary target pedestrian crossing;

[0010] Based on the calibration parameters of the forward-looking camera obtained from the offline calibration, the pixel coordinate system to the vehicle coordinate system is transformed for the main target pedestrian crossing to obtain the distance from the main target pedestrian crossing to the vehicle.

[0011] Determine whether the distance is less than the alarm distance threshold;

[0012] If the distance is less than the alarm distance threshold, determine whether the current vehicle speed exceeds the vehicle speed alarm threshold.

[0013] If the current vehicle speed exceeds the speed warning threshold, an overspeed warning will be issued;

[0014] If the current vehicle speed does not exceed the vehicle speed warning threshold, determine whether there are pedestrians in the main target pedestrian crossing;

[0015] If there are pedestrians in the main target crosswalk, issue a courtesy reminder.

[0016] Furthermore, the ECA mechanism includes:

[0017] The image frame is used as the original input feature map and global average pooling is performed to obtain the feature map after global average pooling.

[0018] The feature map after global average pooling is subjected to a 1D convolution operation with a kernel size of k in the channel dimension, and the weights of each channel are obtained by passing it through the Sigmoid activation function.

[0019] The weights are multiplied by the corresponding elements of the original input feature map to obtain the final output feature map.

[0020] Furthermore, the feature map after global average pooling is subjected to a 1D convolution operation with a kernel size of k along the channel dimension, and then the weights of each channel are obtained by passing the Sigmoid activation function, including:

[0021] ω=σ(C1D k (y))

[0022] In the formula, y is the feature map after global average pooling; k represents the kernel size; C1D represents one-dimensional convolution, which involves only k parameters; C represents the channel dimension; and ω is the weight of each channel.

[0023] The mapping function φ(k) from k to C is defined as:

[0024] C=φ(k=2 (γ*k-b)

[0025] The mapping function ψ(C) from C to k is:

[0026]

[0027] In the formula, |t| odd represents the closest odd number to t; γ and b are mapping coefficients, with γ and b set to 2 and 1 respectively.

[0028] Furthermore, the pedestrian crossings are screened to determine the primary target pedestrian crossings, including:

[0029] Determine whether any one of the six equal divisions of the bottom edge of the target frame of the pedestrian crossing is within the vehicle warning zone; wherein, the configuration parameters include the vehicle warning zone;

[0030] If any one of the six equal divisions of the bottom edge of the target frame of the pedestrian crossing is within the vehicle warning zone, the pedestrian crossing will be regarded as the candidate primary target pedestrian crossing.

[0031] Based on the candidate primary target pedestrian crossings, the final primary target pedestrian crossing is determined.

[0032] Furthermore, the pedestrian crossings are screened to determine the primary target pedestrian crossings, including:

[0033] Based on the location frame of the pedestrian crossing and the coordinates of the center point of its lower edge, the pixel coordinates of the lower left and lower right corners of the location frame are obtained.

[0034] Based on the pixel coordinates of the bottom left and bottom right corners of the location box, the coordinates of the bottom left and bottom right vehicle body are obtained through coordinate transformation.

[0035] Calculate the minimum distance △X from each point on the bottom edge of the pedestrian crossing to the vehicle's centerline, based on the vehicle's coordinates at the bottom left and bottom right corners.

[0036] Targets with △X≤λ*(lane_width / 2) are selected as candidate primary pedestrian crossings; where λ is an empirical multiplier value and lane_width represents the lane width; the configuration parameters include the lane width.

[0037] Based on the candidate primary target pedestrian crossings, the final primary target pedestrian crossing is determined.

[0038] Further, based on the candidate primary target pedestrian crossings, the final primary target pedestrian crossing is determined, including:

[0039] By comparing and analyzing the candidate pedestrian crossings, the pedestrian crossing closest to the vehicle in the longitudinal direction can be identified.

[0040] Determine whether any one of the four equal divisions at the top of the pedestrian crossing target is within the vehicle's driving area;

[0041] If any one of the four equal division points at the top of the pedestrian crossing target is within the driving area of ​​this vehicle, the pedestrian crossing target will be determined as the final primary pedestrian crossing target.

[0042] Secondly, the present invention provides a pedestrian crossing detection device, comprising:

[0043] The acquisition unit is used to acquire the CAN information stream of the vehicle, the real-time video stream captured by the front-view camera, and configuration parameters; the CAN information stream includes the real-time vehicle speed; the real-time video stream is segmented into frames according to a preset frame rate to obtain image frames; the configuration parameters include the calibration parameters of the front-view camera obtained through offline calibration, the alarm distance threshold, and the vehicle speed alarm threshold;

[0044] The detection unit is used to perform target detection on the image frame using a target detection network to obtain the pedestrian crossing in the image frame. The target detection network adopts an improved YOLOv8 network, and the network input parameters are optimized as follows: for images with flat road surfaces and no obstructions in front, the skyline of the same batch of images is obtained, and the original labeled samples of the same batch are batch-cropped and labeled to crop out the area above the skyline; an ECA mechanism is introduced into the improved YOLOv8 network.

[0045] A filtering unit is used to filter the pedestrian crossings and determine the primary target pedestrian crossing;

[0046] The conversion unit is used to convert the main target pedestrian crossing from the pixel coordinate system to the vehicle coordinate system according to the calibration parameters of the forward-looking camera obtained by the offline calibration, so as to obtain the distance from the main target pedestrian crossing to the vehicle.

[0047] The first judgment unit is used to determine whether the distance is less than the alarm distance threshold.

[0048] The second judgment unit is used to determine whether the current vehicle speed exceeds the vehicle speed alarm threshold when the distance is less than the alarm distance threshold.

[0049] The alarm unit is used to issue an overspeed alarm when the current vehicle speed exceeds the vehicle speed alarm threshold;

[0050] The third judgment unit is used to determine whether there are pedestrians in the main target pedestrian crossing when the current vehicle speed does not exceed the vehicle speed warning threshold.

[0051] The reminder unit is used to issue a courtesy reminder when there are pedestrians in the main target pedestrian crossing.

[0052] Thirdly, the present invention provides an electronic device, comprising: a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor performs the steps of the method described above.

[0053] The beneficial effects of the present invention are as follows: The pedestrian crossing detection method, device and electronic device provided by the present invention utilize an improved YOLOv8 network. Through an improved method that combines reasonable cropping of input images and the introduction of an attention mechanism, the model can focus more on key target areas without sacrificing detection speed, thereby improving the accuracy of the model and enhancing the robustness of the algorithm in complex scenarios. Attached Figure Description

[0054] To further understand the features and technical content of this invention, please refer to the following detailed description and accompanying drawings. However, the drawings are provided for reference and illustration only and are not intended to limit the invention.

[0055] In the attached diagram,

[0056] Figure 1 This is a flowchart of the pedestrian crossing detection method of the present invention;

[0057] Figure 2 A flowchart of the ECA mechanism;

[0058] Figure 3 This is a flowchart of an optional implementation of S103;

[0059] Figure 4 This is a flowchart of another optional implementation of S103;

[0060] Figure 5 This is a schematic diagram of the vehicle body coordinate system;

[0061] Figure 6 These are schematic diagrams of image samples before and after cropping;

[0062] Figure 7 The diagram shows the Bottleneck module network structure before and after the improvement.

[0063] Figure 8 Diagram of the vehicle warning zone;

[0064] Figure 9 This is a block diagram of the pedestrian crossing detection device of the present invention;

[0065] Figure 10 This is a block diagram of the electronic device of the present invention. Detailed Implementation

[0066] To further illustrate the technical means and effects of the present invention, the following detailed description is provided in conjunction with the preferred embodiments of the present invention and their accompanying drawings.

[0067] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0068] Please see Figure 1 This invention provides a method for detecting pedestrian crossings, comprising:

[0069] S101, acquire the vehicle's CAN information stream, the real-time video stream captured by the forward-facing camera, and configuration parameters; the CAN information stream includes the vehicle's real-time speed; the real-time video stream is segmented at a preset frame rate to obtain image frames; the configuration parameters include the forward-facing camera calibration parameters acquired offline, alarm distance threshold, and vehicle speed alarm threshold.

[0070] Specifically, by reading the real-time CAN information stream from the vehicle control module, the vehicle's real-time speed and steering angle information are obtained. The input image frames are obtained by reading the real-time video stream from the forward-facing camera and segmenting it at a preset frame rate. Configuration parameters are read, including calibration parameters of the forward-facing camera obtained through offline calibration, such as intrinsic parameters, extrinsic parameters, and distortion coefficients; alarm distance thresholds, vehicle speed alarm thresholds, and vehicle warning zones or lane widths configured in the offline calibration.

[0071] S102, the image frame is subjected to target detection using an object detection network to obtain the pedestrian crossing in the image frame; the object detection network adopts an improved YOLOv8 network, and the network input parameters are optimized as follows: for images with flat road surfaces and no obstructions in front, the skyline of the same batch of images is obtained, and the original labeled samples of the same batch are batch-cropped and labeled to crop out the area above the skyline; an ECA mechanism is introduced into the improved YOLOv8 network.

[0072] YOLOv8 is a high-efficiency and advanced network model in the field of object detection, based on the YOLO series with numerous improvements and optimizations. This network not only inherits the high detection accuracy and fast inference speed of its predecessors but also introduces new design concepts and mechanisms, such as the C2F structure and decoupled head design, further enhancing model performance. Furthermore, YOLOv8 supports various visual AI tasks, including object detection and instance segmentation, and provides models at multiple scales to adapt to different application scenarios and computational resource requirements. In summary, YOLOv8 demonstrates strong competitiveness in the field of object detection due to its superior performance and flexibility. YOLOv8 supports a full range of visual AI tasks, including detection, segmentation, pose estimation, tracking, and classification, and its versatility allows users to leverage YOLOv8's capabilities in various applications and domains. Simultaneously, YOLOv8 provides models at multiple scales to adapt to different computational resource and accuracy requirements.

[0073] YOLOv8 can accurately identify and locate pedestrian crossings in images or videos, providing reliable information for vehicle driver assistance systems (ADAS). Optimized for real-time applications and limited computing resources, YOLOv8 achieves faster processing speeds while maintaining high accuracy, making it ideal for ADAS systems that require real-time response and rapid processing of image or video data. YOLOv8 employs a more advanced backbone network architecture, such as Darknet-53 or similar architectures, enabling better feature extraction and representation. This facilitates accurate pedestrian crossing identification in complex scenes, especially under challenging conditions such as changing lighting, occlusion, or blurriness. YOLOv8 supports a full range of visual AI tasks, including detection, segmentation, pose estimation, tracking, and classification, giving it greater flexibility in ADAS systems and allowing for expansion and customization of functionality as needed. Improved object classification capabilities enable more accurate identification and classification of detected targets. In pedestrian crossing detection, this means YOLOv8 can more accurately distinguish pedestrian crossings from other similar structures or objects, reducing the likelihood of false positives and false negatives. YOLOv8 offers pre-trained models of different sizes, which can be selected according to computing resources and accuracy requirements. This makes YOLOv8 more applicable to vehicle driver assistance systems and allows for flexible adjustments based on different application scenarios and hardware configurations.

[0074] Building an object detection network requires data preparation, network improvement, and training optimization. Data preparation combines publicly available datasets with proprietary urban vehicle-mounted datasets. Proprietary data preparation involves extracting frames from acquired offline video data to obtain labeled data, and then labeling them according to specific definitions and rules. For example, positive samples are pedestrian crossings, and negative samples are interference signs such as combined traffic dividers and continuous speed bumps. During algorithm execution, real-time or offline video streams are read, and frames are extracted at a specified frame rate to obtain input image frames. When processing publicly available datasets, since they only contain pedestrian crossing labels, interference sign labels need to be added; additionally, the pedestrian crossing labels may have inaccurate bounding boxes, requiring verification.

[0075] Please see Figure 6 Based on the characteristics of the images, such as observing the boundary between the sky and the ground on a flat, unobstructed road surface, this invention can obtain the skyline of the same batch of images, that is, the boundary line where the sky and the ground meet, thereby enabling batch cropping and labeling of the original labeled samples of the same batch. Figure 6 (a) is a schematic diagram of the image sample before cropping. Figure 6 (b) is a schematic diagram of the cropped image sample.

[0076] This method can improve the speed of model training and real-time detection to a certain extent, while eliminating interference from above the skyline and reducing false detections. By cropping the region above the skyline, the size of the input image can be reduced, thereby reducing the computational load during model processing. In object detection tasks, the region above the skyline often does not contain valuable detection targets, so cropping these regions can reduce unnecessary computation and improve the model's inference speed. The region above the skyline may contain some background information unrelated to the detection target, which may interfere with model training and increase the risk of overfitting. By cropping these regions, the interference of background information can be reduced, allowing the model to focus more on learning useful features, thereby reducing the risk of overfitting. In some cases, the region above the skyline may contain some noise or interference information that is harmful to the detection task, which may mislead the model to make incorrect predictions. By cropping these regions, the influence of this noise or interference information can be eliminated, thereby improving the model's detection performance. For images with a flat road surface and no obstructions in front, the input image after cropping the region above the skyline has higher similarity, which helps the model learn more general features. When the model encounters similar road surface scenarios in testing or real-world applications, these general features can help the model make more accurate predictions, thereby improving the model's generalization ability. In the data preprocessing stage, acquiring the skyline of the same batch of images and performing batch cropping and labeling can simplify the data preprocessing process and improve data processing efficiency. This not only reduces the time and manpower costs required for data preprocessing but also ensures the consistency and accuracy of the data preprocessing.

[0077] like Figure 7 As shown, this invention introduces the ECA (Efficient Channel Attention) mechanism into the Bottleneck module of the CSPrayer_2Conv module (C2F module) in the YOLOv8 network to improve the detection accuracy under complex lighting and working conditions. Its feature is that it uses 1D convolution to efficiently realize local cross-channel interaction and extract the dependencies between channels. Figure 7 (a) is a schematic diagram of the Bottleneck module network structure before the improvement. Figure 7 (b) is a schematic diagram of the improved Bottleneck module network structure.

[0078] The C2F module in the YOLOv8 network is a key component of the YOLOv8 network architecture, playing a crucial role in feature extraction and model performance. The YOLOv8 network uses the C2F module as its basic building block in the backbone, which internally employs the Bottleneck module to further enhance the model's expressive power and efficiency. The Bottleneck module has a smaller computational cost and fewer parameters, yet maintains good feature extraction capabilities, helping to reduce computational resource requirements while preserving model performance. The introduction of ECA (Elastic Computational Allocation) in the Bottleneck module allows the model to focus more on important channels during feature extraction. By globally adaptively weighting the feature maps of each channel, ECA improves the expressive power of features, thus helping the Bottleneck module better extract and convey useful feature information. The Bottleneck module with ECA is more accurate in feature extraction, capturing and strengthening important channel features. Due to the efficiency of ECA, this combination increases model performance without significantly increasing computational complexity or model size. In tasks such as object detection, this combination helps the model more accurately locate targets, improving detection accuracy and efficiency. In the Bottleneck module, the ECA module is added after the convolutional layers to process the feature maps output by the convolutional layers. The ECA module learns channel attention weights through 1D convolution and applies these weights to each channel of the feature map, achieving channel weighting. The weighted feature maps are then passed to subsequent parts of the Bottleneck module to continue participating in the model's training and inference processes.

[0079] Please see Figure 2 In this embodiment, the ECA mechanism specifically includes:

[0080] S1021, the image frame is used as the original input feature map and global average pooling is performed to obtain the feature map after global average pooling.

[0081] Specifically, the original input feature map has three dimensions: height (H), width (W), and number of channels (C). The height and width dimensions define the two-dimensional spatial size of the feature map. For the input image or feature map, the height and width are usually fixed, except in some networks with pooling or fully convolutional layers, which may change the spatial dimensions. In convolutional operations, the convolutional kernel, also called a filter, slides across the input feature map to produce an output feature map. The height and width of this output feature map are typically less than or equal to the height and width of the input feature map.

[0082] The channel dimension C typically corresponds to the number of color channels or feature maps in the input data. For example, for an RGB image, the number of input channels is usually 3, including red, green, and blue. In convolutional neural networks, each convolutional layer generates multiple feature maps that capture different aspects of the input data. Therefore, as the network depth increases, the number of channels in the feature maps may increase.

[0083] Global average pooling is a concise and effective technique that can be used in the last or penultimate layer of a convolutional neural network. It reduces the number of parameters and computational cost by averaging information from the entire feature map, generating one or more numerical values ​​as output. This improves computational efficiency and reduces the risk of overfitting. Furthermore, global average pooling helps the model generalize better to different input images.

[0084] S1022 performs a 1D convolution operation with a kernel size of k on the feature map after global average pooling, and then obtains the weights of each channel by passing it through the Sigmoid activation function.

[0085] Specifically, the weights of each channel are obtained according to the following formula:

[0086] ω=σ(C1D k (y))

[0087] In the formula, y is the feature map after global average pooling; k represents the kernel size, which represents the coverage of local cross-channel interactions, i.e., how many neighbors of each channel participate in the attention prediction of that channel; C1D represents a one-dimensional convolution, which involves only k parameters, where C represents the channel dimension; ω is the weight of each channel. The purpose of the ECA mechanism is to capture local cross-channel information interactions, therefore it is necessary to determine the approximate range of channel interaction information, i.e., the kernel size k of the 1D convolution.

[0088] The mapping function φ(k) from k to C is defined as:

[0089] C=φ(k=2 (γ*k-b)

[0090] The mapping function ψ(C) from C to k is:

[0091]

[0092] In the formula, |t| odd Let t represent the closest odd number to t; γ and b are mapping coefficients, set to 2 and 1 respectively. By using nonlinear mapping, high-dimensional channels have longer-distance interactions, while low-dimensional channels have shorter-distance interactions.

[0093] S1023, Multiply the weights by the corresponding elements of the original input feature map to obtain the final output feature map.

[0094] Specifically, the weights are multiplied element-wise with the corresponding channels of the original input feature map. If a channel has a higher weight, all elements in the feature map for that channel are correspondingly amplified; conversely, if the weight is lower, all elements in the feature map are correspondingly reduced. After this multiplication operation, a new feature map is obtained, which is the final output feature map after considering channel attention. Compared to the original input feature map, this output feature map places greater emphasis on the channels that the model deems important. This process allows the model to focus more on features useful for object detection, thereby improving model performance.

[0095] S103, the pedestrian crossings are screened to determine the primary target pedestrian crossing.

[0096] Specifically, the pedestrian crossings are screened to determine the main target pedestrian crossings, which may include two methods: area determination method and distance determination method.

[0097] Please see Figure 3 The regional determination method includes:

[0098] S1031, determine whether any one of the six equal divisions of the bottom edge of the target frame of the pedestrian crossing is within the vehicle warning zone; wherein, the configuration parameters include the vehicle warning zone.

[0099] Please see Figure 8 The process involves determining whether any one of the six equal divisions of the bottom edge of the pedestrian crossing's bounding box is within the vehicle warning zone. The purpose of this step is to identify pedestrian crossings that may intersect with vehicle paths, thereby improving the accuracy of identifying potential conflict areas.

[0100] S1032, if any one of the six equal divisions of the bottom edge of the pedestrian crossing's target frame is within the vehicle warning zone, then the pedestrian crossing is selected as a candidate primary target pedestrian crossing. Otherwise, proceed to S10321 to remove the pedestrian crossing.

[0101] S1033, Based on the candidate primary target pedestrian crossings, determine the final primary target pedestrian crossing.

[0102] By accurately identifying pedestrian crossings intersecting with vehicle warning zones, this method helps drivers recognize potential hazards earlier, giving them sufficient time to react and reducing the risk of traffic accidents. By using six equal divisions of the bottom edge of the pedestrian crossing target box as the basis for judgment, this method can more evenly cover the entire target box, improving the accuracy of determining the relationship between pedestrian crossings and vehicle warning zones while reducing unnecessary computation. This method allows for adjustment of the vehicle warning zone configuration parameters according to actual needs, adapting to different traffic environments and road conditions. For example, in areas with heavy traffic or high pedestrian volume, the vehicle warning zone can be expanded to improve the sensitivity of the warning.

[0103] Please see Figure 4 Distance determination methods include:

[0104] S1034. Based on the location frame of the pedestrian crossing and the coordinates of the center point of its lower edge, obtain the pixel coordinates of the lower left and lower right corners of the location frame.

[0105] The location bounding box coordinates (x, y) of the pedestrian crossing can be obtained through the object detection network. tl ,y tl ,x br ,y br ) and the coordinates of the center point of the lower edge (x bc ,y bc Then the lower left pixel coordinates of the pedestrian crossing target box are (x...). tl ,y br The bottom right pixel coordinates are (x br ,y br ).

[0106] S1035: Based on the pixel coordinates of the lower left and lower right corners of the position box, the coordinates of the lower left and lower right vehicle body are obtained through coordinate transformation.

[0107] Through coordinate transformation, the coordinates of the lower left corner of the vehicle can be obtained as (X...). bl ,Y bl The coordinates of the car body in the lower right corner are (X... br ,Y br ). bl represents the bottom left corner of the detection box, tl represents the top left corner, and br represents the bottom right corner.

[0108] S1036, Calculate the minimum distance △X from each point on the bottom edge of the pedestrian crossing to the vehicle's centerline, based on the vehicle's coordinates at the lower left and lower right corners.

[0109] In the vehicle coordinate system, let the minimum distance from each point on the bottom edge of the pedestrian crossing to the vehicle's centerline be ΔX. The formula for calculating the minimum distance is as follows:

[0110]

[0111] S1037, Select targets with △X≤λ*(lane_width / 2) as candidate primary targets for pedestrian crossings; where λ is an empirical multiplier value, and lane_width represents the lane width; wherein, the configuration parameters include the lane width.

[0112] After testing with multiple sets of offline videos, the default setting can be 1.25, as the width of each lane on an urban main road is generally 3.5 meters.

[0113] S1038, Based on the candidate primary target pedestrian crossings, determine the final primary target pedestrian crossing.

[0114] By converting pixel coordinates to vehicle coordinates and calculating the distances from each point on the bottom edge of the pedestrian crossing to the vehicle's centerline, this method can accurately determine the relative positional relationship between the pedestrian crossing and the vehicle. This helps to accurately identify pedestrian crossings that may conflict with the vehicle's travel path, improving the accuracy of traffic warnings. Since this method is based on coordinate calculation and distance comparison, its calculation process is relatively simple and efficient, allowing for real-time processing and analysis of large amounts of traffic data. This enables the system to quickly respond to changes in traffic conditions and provide timely warnings to drivers. By introducing an empirical multiplier value λ, the method can be adjusted according to different traffic environments and road conditions. For example, in areas with heavy traffic or high pedestrian flow, the value of λ can be appropriately increased to expand the range of candidate pedestrian crossings and improve the sensitivity of the warning. The steps of this method are clear and straightforward, easy to understand and implement, and do not require complex image processing algorithms or machine learning models; only basic coordinate transformation and distance calculation are needed. This makes the method highly feasible and reliable in practical applications. By considering the relationship between lane width and the location of pedestrian crossings, this method can identify pedestrian crossings that may conflict with the vehicle's travel path, helping drivers to better predict and avoid potential traffic hazards while adhering to traffic rules.

[0115] Specifically, a comparative analysis is performed on the candidate pedestrian crossings to identify the pedestrian crossing target that is closest to the vehicle in the longitudinal direction. The longitudinal distance comparison can be performed in either the vehicle coordinate system or the pixel coordinate system.

[0116] In a pixel coordinate system: the pedestrian crossing with the largest y-parameter among the pixel coordinates of the center point of the lower edge is the closest pedestrian crossing target to the vehicle. The coordinates of the center point of the m-th pedestrian crossing are represented as (x... bc_m ,y bc_m ), then y max =max(y bc_1 ,y bc_2 ,...,y bc_m ), y maxIf the serial number of the pedestrian crossing is i, the i-th pedestrian crossing in the candidate list is the pedestrian crossing target closest to the host vehicle.

[0117] In the vehicle body coordinate system: the pedestrian crossing with the smallest true longitudinal distance relative to the host vehicle is the main target pedestrian crossing. The true distance of the m-th candidate pedestrian crossing relative to the host vehicle is D min =min(Y bl_m ,Y br_m ), then D min =min(D1,D2,...,D m ), D min If the serial number of the pedestrian crossing where it is located is i, the i-th pedestrian crossing in the candidate list is the pedestrian crossing target closest to the host vehicle.

[0118] Determine whether any one of the four equal division points at the top of the pedestrian crossing target is within the driving area of the host vehicle; if any one of the four equal division points at the top of the pedestrian crossing target is within the driving area of the host vehicle, the pedestrian crossing target is determined as the final main target pedestrian crossing.

[0119] The left and right width parameters of the host vehicle are determined in the camera calibration process, fRight + fLeft = vehicle_width, fRight is the distance from the center of the ADAS front-view camera to the outer edge of the right tire of the host vehicle, fLeft is the distance from the center of the ADAS front-view camera to the outer edge of the left tire of the host vehicle, and vehicle_width represents the width of the vehicle, for example, the body width of a certain bus is 2.28 meters. If so, the pedestrian crossing is the main target pedestrian crossing. The judgment formula for the i-th point is: |X tl +i*detaX| < max(fLeft, fRight), 0 < i < 4, where detaX=(X tr -X tl ) / 4, representing the width of each segment after the detected width of the pedestrian crossing is equally divided into 4 parts.

[0120] S104: Convert the main target pedestrian crossing from the pixel coordinate system to the vehicle body coordinate system according to the front-view camera calibration parameters obtained by offline calibration, so as to obtain the distance from the main target pedestrian crossing to the host vehicle.

[0121] Through the internal and external parameters calibrated by the front-view camera, the conversion between the pixel coordinate system and the vehicle body coordinate system can be realized, and the pixel coordinates (x bc ,y bc ) of the center point of the lower edge of the pedestrian crossing can be converted into the vehicle body coordinates (X bc ,Y bc ), then the longitudinal coordinate Y in the vehicle body coordinates bc is the true distance D to the host vehicle real .

[0122] The vehicle body coordinate system of this invention can be defined as follows: X-axis: facing forward of the vehicle, in the direction indicated by the right hand; Y-axis: the direction of vehicle movement; Z-axis: perpendicular to the ground, pointing towards the roof; the origin is the intersection of the camera mounting position and the ground. Taking a sedan as an example, its vehicle body coordinate system is as follows: Figure 5 As shown.

[0123] S105, determine whether the distance is less than the alarm distance threshold.

[0124] Judge D real Is it less than the alarm distance threshold D? Threshold If the result is less than the given value, proceed to the next step; otherwise, return to the main process.

[0125] S106, if the distance is less than the alarm distance threshold, determine whether the current vehicle speed exceeds the vehicle speed alarm threshold.

[0126] S107 If the current vehicle speed exceeds the speed warning threshold, an overspeed warning will be issued.

[0127] S108, if the current vehicle speed does not exceed the vehicle speed warning threshold, determine whether there are pedestrians in the main target pedestrian crossing;

[0128] S109: If there are pedestrians in the main target crosswalk, issue a courtesy reminder.

[0129] If there are no pedestrians in the main target crosswalk, proceed to the next frame for detection.

[0130] As can be seen from the above embodiments, the pedestrian crossing detection method of the present invention detects pedestrian crossings around the vehicle during its movement using an improved YOLOv8 network model. The advantage of this method lies in its efficient implementation of local cross-lane interaction without changing the size of the input feature map by introducing an ECA mechanism. This strengthens lane features and improves the accuracy of pedestrian crossing detection in complex scenarios, making the system less susceptible to environmental interference and enhancing its robustness. The present invention integrates multi-angle information such as the detection area, lane lines, lane width, and distance to determine the primary target pedestrian crossing, further improving the system's robustness.

[0131] Please see Figure 9 This invention proposes a pedestrian crossing detection device, comprising:

[0132] The acquisition unit 91 is used to acquire the CAN information stream of the vehicle, the real-time video stream collected by the front-view camera, and configuration parameters; the CAN information stream includes the real-time vehicle speed; the real-time video stream is segmented into frames according to a preset frame rate to obtain image frames; the configuration parameters include the calibration parameters of the front-view camera obtained through offline calibration, the alarm distance threshold, and the vehicle speed alarm threshold.

[0133] The detection unit 92 is used to perform target detection on the image frame using a target detection network to obtain the pedestrian crossing in the image frame; the target detection network adopts an improved YOLOv8 network, and the network input parameters are optimized as follows: for images with flat road surfaces and no obstructions in front, the skyline of the same batch of images is obtained, and the original labeled samples of the same batch are batch-cropped and labeled to crop out the area above the skyline; an ECA mechanism is introduced into the improved YOLOv8 network;

[0134] The filtering unit 93 is used to filter the pedestrian crossings and determine the main target pedestrian crossing;

[0135] The conversion unit 94 is used to convert the main target pedestrian crossing from the pixel coordinate system to the vehicle coordinate system according to the calibration parameters of the forward-looking camera obtained by the offline calibration, so as to obtain the distance from the main target pedestrian crossing to the vehicle.

[0136] The first judgment unit 95 is used to determine whether the distance is less than the alarm distance threshold.

[0137] The second judgment unit 96 is used to determine whether the current vehicle speed exceeds the vehicle speed alarm threshold when the distance is less than the alarm distance threshold.

[0138] Alarm unit 97 is used to issue an overspeed alarm when the current vehicle speed exceeds the vehicle speed alarm threshold;

[0139] The third judgment unit 98 is used to determine whether there are pedestrians in the main target pedestrian crossing when the current vehicle speed does not exceed the vehicle speed alarm threshold.

[0140] The reminder unit 99 is used to issue a courtesy reminder when there are pedestrians in the main target pedestrian crossing.

[0141] As can be seen from the above embodiments, the pedestrian crossing detection device of the present invention detects pedestrian crossings around a vehicle during its journey using an improved YOLOv8 network model. The advantage of this device lies in its efficient implementation of local cross-lane interaction through the introduction of an ECA mechanism, without changing the size of the input feature map. This strengthens lane features and improves the accuracy of pedestrian crossing detection in complex scenarios, making the system less susceptible to environmental interference and enhancing its robustness. The present invention integrates multi-angle information such as the detection area, lane lines, lane width, and distance to determine the primary target pedestrian crossing, further improving the system's robustness.

[0142] Please see Figure 10 It includes: a memory 100 and a processor 200, wherein the memory 100 stores a computer program, and when the computer program is executed by the processor 200, the processor 200 performs the steps of the method described above.

[0143] This invention also provides a storage medium storing a computer program. When executed by a processor, the computer program implements some or all of the steps in the various embodiments of the pedestrian crossing detection method provided by this invention. The storage medium may be a magnetic disk, optical disk, read-only memory (ROM), or random access memory (RAM), etc.

[0144] Those skilled in the art will clearly understand that the techniques in the embodiments of the present invention can be implemented using software plus necessary general-purpose hardware platforms. Based on this understanding, the technical solutions in the embodiments of the present invention, or the parts that contribute to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in various embodiments or certain parts of the embodiments of the present invention.

[0145] The same or similar parts between the various embodiments in this specification can be referred to mutually. In particular, the pedestrian crossing detection device embodiment is basically similar to the method embodiment, so it is described more simply, and the relevant parts can be referred to the description in the method embodiment.

[0146] The embodiments of the present invention described above do not constitute a limitation on the scope of protection of the present invention.

Claims

1. A method for detecting pedestrian crossings, characterized in that, include: Obtain the vehicle's CAN information stream, real-time video stream captured by the forward-facing camera, and configuration parameters; The CAN information stream includes the vehicle's real-time speed; the real-time video stream is segmented at a preset frame rate to obtain image frames; The configuration parameters include the calibration parameters of the forward-looking camera obtained through offline calibration, the alarm distance threshold, and the vehicle speed alarm threshold. Object detection is performed on image frames using an object detection network to identify pedestrian crossings. The object detection network adopts an improved YOLOv8 network, and the network input parameters are optimized as follows: for images with flat road surfaces and no obstructions in front, the skyline of the same batch of images is obtained, and the original labeled samples of the same batch are batch-cropped and labeled to remove the area above the skyline; an ECA mechanism is introduced into the improved YOLOv8 network. The pedestrian crossings are filtered to determine the primary target pedestrian crossings. Specifically, based on the location frame of the pedestrian crossing and the coordinates of its bottom edge center point, the lower left and lower right pixel coordinates of the location frame are obtained. Based on these coordinates, the lower left and lower right vehicle vehicle coordinates are obtained through coordinate transformation. Based on these coordinates, the minimum distance ΔX from each point on the bottom edge of the pedestrian crossing to the vehicle's centerline is calculated. Pedestrian crossings with ΔX ≤ λ are selected. The target (lane_width / 2) is selected as the candidate primary target pedestrian crossing; where λ is an empirical multiplier value, and lane_width represents the lane width; the configuration parameters include the lane width; the final primary target pedestrian crossing is determined based on the candidate primary target pedestrian crossings; Based on the calibration parameters of the forward-looking camera obtained from the offline calibration, the pixel coordinate system of the main target pedestrian crossing is transformed to the vehicle coordinate system to obtain the distance from the main target pedestrian crossing to the vehicle. Determine whether the distance is less than the alarm distance threshold; If the distance is less than the alarm distance threshold, determine whether the current vehicle speed exceeds the vehicle speed alarm threshold. If the current vehicle speed exceeds the speed warning threshold, an overspeed warning will be issued; If the current vehicle speed does not exceed the vehicle speed warning threshold, determine whether there are pedestrians in the main target pedestrian crossing; If there are pedestrians in the main target crosswalk, issue a courtesy reminder.

2. The pedestrian crossing detection method as described in claim 1, characterized in that, The ECA mechanism includes: The image frame is used as the original input feature map and global average pooling is performed to obtain the feature map after global average pooling. The feature map after global average pooling is subjected to a 1D convolution operation with a kernel size of k in the channel dimension, and the weights of each channel are obtained by passing it through the Sigmoid activation function. The weights are multiplied by the corresponding elements of the original input feature map to obtain the final output feature map.

3. The pedestrian crossing detection method as described in claim 2, characterized in that, The feature map after global average pooling is subjected to a 1D convolution operation with a kernel size of k along the channel dimension, and then the weights of each channel are obtained by passing it through the Sigmoid activation function, including: ； In the formula, This is the feature map after global average pooling; k represents the kernel size; C1D represents one-dimensional convolution, which involves only k parameters, and C represents the channel dimension. The weights for each channel; Mapping function from k to C Defined as: C= ； Mapping function from C to k for: ； In the formula, Indicates the nearest odd number to t; and For mapping coefficients, and Set them to 2 and 1 respectively.

4. The pedestrian crossing detection method as described in claim 1, characterized in that, Based on the candidate primary target pedestrian crossings, the final primary target pedestrian crossing is determined, including: By comparing and analyzing the candidate pedestrian crossings, the pedestrian crossing closest to the vehicle in the longitudinal direction can be identified. Determine whether any one of the four equal divisions at the top of the pedestrian crossing target is within the vehicle's driving area; If any one of the four equal division points at the top of the pedestrian crossing target is within the driving area of ​​this vehicle, the pedestrian crossing target will be determined as the final primary pedestrian crossing target.

5. A pedestrian crossing detection device, characterized in that, include: The acquisition unit is used to acquire the vehicle's CAN information stream, the real-time video stream captured by the front-view camera, and configuration parameters; The CAN information stream includes the vehicle's real-time speed; the real-time video stream is segmented at a preset frame rate to obtain image frames; The configuration parameters include the calibration parameters of the forward-looking camera obtained through offline calibration, the alarm distance threshold, and the vehicle speed alarm threshold. The detection unit is used to perform target detection on the image frame using a target detection network to obtain the pedestrian crossing in the image frame. The target detection network adopts an improved YOLOv8 network, and the network input parameters are optimized as follows: for images with flat road surfaces and no obstructions in front, the skyline of the same batch of images is obtained, and the original labeled samples of the same batch are batch-cropped and labeled to crop out the area above the skyline; an ECA mechanism is introduced into the improved YOLOv8 network. A filtering unit is used to filter the pedestrian crossings and determine the main target pedestrian crossings. Specifically, based on the location frame of the pedestrian crossing and the coordinates of its bottom edge center point, the lower left and lower right pixel coordinates of the location frame are obtained. Based on the lower left and lower right pixel coordinates of the location frame, coordinate transformation is used to obtain the lower left and lower right vehicle vehicle coordinates. Based on the lower left and lower right vehicle vehicle coordinates, the minimum distance ΔX from each point on the bottom edge of the pedestrian crossing to the vehicle's centerline is calculated. Pedestrian crossings with ΔX ≤ λ are then filtered out. The target (lane_width / 2) is selected as the candidate primary target pedestrian crossing; where λ is an empirical multiplier value, and lane_width represents the lane width; the configuration parameters include the lane width; the final primary target pedestrian crossing is determined based on the candidate primary target pedestrian crossings; The conversion unit is used to convert the main target pedestrian crossing from the pixel coordinate system to the vehicle coordinate system according to the calibration parameters of the forward-looking camera obtained by the offline calibration, so as to obtain the distance from the main target pedestrian crossing to the vehicle. The first judgment unit is used to determine whether the distance is less than the alarm distance threshold. The second judgment unit is used to determine whether the current vehicle speed exceeds the vehicle speed alarm threshold when the distance is less than the alarm distance threshold. The alarm unit is used to issue an overspeed alarm when the current vehicle speed exceeds the vehicle speed alarm threshold; The third judgment unit is used to determine whether there are pedestrians in the main target pedestrian crossing when the current vehicle speed does not exceed the vehicle speed warning threshold. The reminder unit is used to issue a courtesy reminder when there are pedestrians in the main target pedestrian crossing.

6. An electronic device, characterized in that, include: A memory and a processor, the memory storing a computer program that, when executed by the processor, causes the processor to perform the steps of the method as described in any one of claims 1-4.

Images