Automatic driving pedestrian detection method and device, computer equipment and storage medium

Abstract

The invention relates to the technical field of automatic driving, and relates to an automatic driving pedestrian detection method and device, computer equipment, a computer readable storage medium and a computer program product. The method comprises the following steps: acquiring a road environment data set around a vehicle, wherein the road environment data set comprises a road environment image and a laser point cloud of a road environment; according to the laser point cloud of the road environment, acquiring a horizontal elevation angle feature map for representing a three-dimensional road environment; training by using the road environment image and the horizontal elevation angle feature map to obtain a pedestrian recognition model; road environment information around the vehicle is collected in real time; and identifying and obtaining pedestrian information in the road environment by using the pedestrian identification model according to the road environment information. By adopting the method, the accuracy of pedestrian detection can be improved by utilizing the three-dimensional space information acquired by the laser radar and the two-dimensional image information captured by the camera.

Description

Technical Field

[0001] The present application relates to the field of autonomous driving technology, and in particular to an autonomous driving pedestrian detection method, apparatus, computer equipment, computer-readable storage medium, and computer program product. Background Art

[0002] Traditional autonomous driving pedestrian detection systems often rely on a single sensor, such as using only a camera or lidar. However, a single sensor often has difficulty obtaining complete and accurate information in complex and changing road environments. For example, a camera may not be able to clearly capture pedestrians at night or in bad weather, and although lidar is sensitive to distance and depth information, it is not sensitive to visual features such as color and texture. Due to the complexity of the road environment and the diversity of pedestrians' postures and clothing, a single pedestrian detection algorithm often has difficulty achieving high accuracy and strong robustness. Pedestrians may fail to be detected or be misdetected due to factors such as occlusion, scale changes, and posture changes.

[0003] Therefore, there is an urgent need for an autonomous driving pedestrian detection method, device, computer equipment, computer readable storage medium and computer program product that can improve the accuracy of pedestrian detection. Summary of the invention

[0004] Based on this, it is necessary to provide an autonomous driving pedestrian detection method, device, computer equipment, computer-readable storage medium and computer program product that can improve the accuracy of pedestrian detection in response to the above-mentioned technical problems.

[0005] In a first aspect, the present application provides an autonomous driving pedestrian detection method, comprising:

[0006] Acquire a road environment dataset around the vehicle, wherein the road environment dataset includes a road environment image and a laser point cloud of the road environment;

[0007] Acquire a horizontal elevation angle characteristic map for representing a three-dimensional road environment according to the laser point cloud of the road environment;

[0008] Using the road environment image and the horizontal altitude angle feature map, a pedestrian recognition model is trained;

[0009] Collect road environment information around the vehicle in real time;

[0010] According to the road environment information, the pedestrian recognition model is used to identify and obtain pedestrian information in the road environment.

[0011] In one embodiment, obtaining a horizontal elevation angle feature map for representing a three-dimensional road environment according to the laser point cloud of the road environment includes:

[0012] Acquire a density depth map of the road environment according to the laser point cloud of the road environment;

[0013] Extracting horizontal disparity features, height features from the ground and angle features from the density depth map;

[0014] A horizontal altitude angle feature map for representing a three-dimensional road environment is generated according to the horizontal parallax feature, the height feature from the ground and the angle feature.

[0015] In one embodiment, the step of training a pedestrian recognition model using the road environment image and the horizontal altitude angle feature map includes:

[0016] Using the road environment image as input layer data of a convolutional neural network structure, respectively fusing and adding the horizontal altitude angle feature map into different types of structural layers of the convolutional neural network structure, and training to obtain at least one pedestrian recognition model;

[0017] A pedestrian recognition model with the highest recognition accuracy is determined from the at least one pedestrian recognition model.

[0018] In one embodiment, the step of identifying and obtaining pedestrian information in the road environment by using the pedestrian recognition model according to the road environment information includes:

[0019] Using the pedestrian recognition model, performing feature recognition processing on the road environment information to obtain a feature recognition result, wherein the feature recognition result includes scene features and pedestrian features;

[0020] According to the preset pedestrian attribute information and the preset image segmentation algorithm, the scene features and the pedestrian features are verified and identified again to obtain the pedestrian information in the road environment.

[0021] In one embodiment, after acquiring the road environment dataset around the vehicle, the method further includes:

[0022] Performing denoising processing on the road environment image;

[0023] The laser point cloud of the road environment is filtered and smoothed.

[0024] In one embodiment, the step of training a pedestrian recognition model using the road environment image and the horizontal altitude angle feature map includes:

[0025] Extracting a reflected wave signal for preliminary identification of pedestrian information from the laser point cloud of the road environment;

[0026] Referring to the reflected wave signal, a pedestrian recognition model is trained using the road environment image and the horizontal altitude angle feature map.

[0027] In a second aspect, the present application also provides an automatic driving pedestrian detection device, comprising:

[0028] An acquisition module is used to acquire a road environment dataset around the vehicle, wherein the road environment dataset includes a road environment image and a laser point cloud of the road environment;

[0029] The processing module is further used to obtain a horizontal elevation angle characteristic map for representing a three-dimensional road environment according to the laser point cloud of the road environment;

[0030] A model training module, used to train a pedestrian recognition model using the road environment image and the horizontal altitude angle feature map;

[0031] The acquisition module is also used to collect the road environment information around the vehicle in real time;

[0032] The detection module is used to identify and obtain pedestrian information in the road environment based on the road environment information and using the pedestrian recognition model.

[0033] In a third aspect, the present application further provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the following steps are implemented:

[0034] Acquire a road environment dataset around the vehicle, wherein the road environment dataset includes a road environment image and a laser point cloud of the road environment;

[0035] Acquire a horizontal elevation angle characteristic map for representing a three-dimensional road environment according to the laser point cloud of the road environment;

[0036] Using the road environment image and the horizontal altitude angle feature map, a pedestrian recognition model is trained;

[0037] Collect road environment information around the vehicle in real time;

[0038] According to the road environment information, the pedestrian recognition model is used to identify and obtain pedestrian information in the road environment.

[0039] In a fourth aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, the following steps are implemented:

[0040] Acquire a road environment dataset around the vehicle, wherein the road environment dataset includes a road environment image and a laser point cloud of the road environment;

[0041] Acquire a horizontal elevation angle characteristic map for representing a three-dimensional road environment according to the laser point cloud of the road environment;

[0042] Using the road environment image and the horizontal altitude angle feature map, a pedestrian recognition model is trained;

[0043] Collect road environment information around the vehicle in real time;

[0044] According to the road environment information, the pedestrian recognition model is used to identify and obtain pedestrian information in the road environment.

[0045] In a fifth aspect, the present application further provides a computer program product, including a computer program, which implements the following steps when executed by a processor:

[0046] Acquire a road environment dataset around the vehicle, wherein the road environment dataset includes a road environment image and a laser point cloud of the road environment;

[0047] Acquire a horizontal elevation angle characteristic map for representing a three-dimensional road environment according to the laser point cloud of the road environment;

[0048] Using the road environment image and the horizontal altitude angle feature map, a pedestrian recognition model is trained;

[0049] Collect road environment information around the vehicle in real time;

[0050] According to the road environment information, the pedestrian recognition model is used to identify and obtain pedestrian information in the road environment.

[0051] The above-mentioned autonomous driving pedestrian detection method, device, computer equipment, computer-readable storage medium and computer program product use the three-dimensional spatial information obtained by the laser radar and the two-dimensional image information captured by the camera, so that the system can understand the surrounding environment from multiple dimensions. The horizontal altitude angle feature map generated by the laser point cloud data provides the pedestrian recognition model with important clues about the possible location of pedestrians. Combined with the road environment image and the horizontal altitude angle feature map, the trained pedestrian recognition model can more accurately identify the location and posture of pedestrians. The road environment information around the vehicle collected in real time can be analyzed by the pedestrian recognition model to quickly identify pedestrians and provide instant feedback to the autonomous driving system. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] In order to more clearly illustrate the technical solutions in the embodiments of the present application or related technologies, the drawings required for use in the embodiments of the present application or related technical descriptions will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0053] FIG1 is a diagram of an application environment of an autonomous driving pedestrian detection method according to an embodiment;

[0054] FIG2 is a schematic diagram of a flow chart of a method for detecting pedestrians in an autonomous driving system in one embodiment;

[0055] FIG3 is a schematic diagram of a flow chart of a method for detecting pedestrians in an autonomous driving system in another embodiment;

[0056] FIG4 is a structural block diagram of an automatic driving pedestrian detection device in one embodiment;

[0057] FIG. 5 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0058] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0059] The autonomous driving pedestrian detection method provided in the embodiment of the present application can be applied in the application environment shown in FIG1 . The terminal 102 communicates with the server 104 via a network. The data storage system can store data that the server 104 needs to process. The data storage system can be integrated on the server 104, or placed on the cloud or other network servers.

[0060] Server 104 obtains a road environment data set around the vehicle, the road environment data set includes a road environment image and a laser point cloud of the road environment; obtains a horizontal altitude angle feature map for representing a three-dimensional road environment based on the laser point cloud of the road environment; uses the road environment image and the horizontal altitude angle feature map to train a pedestrian recognition model; collects road environment information around the vehicle in real time; and uses the pedestrian recognition model to identify and obtain pedestrian information in the road environment based on the road environment information.

[0061] The terminal 102 may be, but is not limited to, various personal computers, laptops, smart phones, tablet computers, IoT devices, and portable wearable devices. The IoT devices may be smart speakers, smart TVs, smart air conditioners, smart car devices, projection devices, etc. Portable wearable devices may be smart watches, smart bracelets, head-mounted devices, etc. Head-mounted devices may be virtual reality (VR) devices, augmented reality (AR) devices, smart glasses, etc. The server 104 may be an independent physical server, a server cluster or distributed system consisting of multiple physical servers, or a cloud server that provides cloud computing services.

[0062] In an exemplary embodiment, as shown in FIG2 , a method for detecting pedestrians in an autonomous driving system is provided, which is described by taking the method applied to the server in FIG1 as an example, and includes the following steps S202 to S210. Among them:

[0063] Step S202, obtaining a road environment dataset around the vehicle, where the road environment dataset includes a road environment image and a laser point cloud of the road environment.

[0064] Specifically, road environment images are pictures or video frames captured by the cameras on the vehicle, which contain visual information such as roads, traffic signs, pedestrians, and vehicles. These images are crucial for understanding the appearance characteristics of the vehicle's environment and can be used to train deep learning models for object detection, classification, and tracking. For example, the camera distribution range of a vehicle can be: front camera, rear camera, left rear camera, left front camera, right rear camera, and right front camera.

[0065] The laser point cloud of the road environment refers to the laser radar (LiDAR) device that captures the precise three-dimensional structure of the surrounding environment by emitting laser pulses and measuring the reflected light. These measurements generate dense point clouds that represent the spatial layout of the environment, including distance, height, and other geometric features. For example, the distribution range of the vehicle's LiDAR can be: front center LiDAR, front left LiDAR, front right LiDAR, rear left LiDAR, and rear right LiDAR.

[0066] Step S204, obtaining a horizontal altitude angle feature map for representing a three-dimensional road environment based on the laser point cloud of the road environment.

[0067] Specifically, since the laser point cloud data contains a large number of points, each point has a three-dimensional spatial coordinate (x, y, z). Before generating the horizontal elevation angle feature map, the laser point cloud data needs to be preprocessed, including denoising, filtering, ground segmentation and other steps to remove irrelevant information, such as the vehicle itself, small objects on the ground, etc. The generation of the horizontal elevation angle feature map involves extracting useful information from the laser point cloud data. This usually includes calculating the horizontal angle and height information of each point relative to the vehicle. The horizontal angle can refer to the angle of the point relative to the vehicle's forward direction, while the height information describes the position of the point in the vertical direction. Map these angle information onto a two-dimensional plane to create a feature map. In this feature map, the horizontal axis represents the horizontal angle, while the vertical axis represents the height. The intensity or color of each pixel can represent the point cloud density or other attributes at that angle and height. The generated feature map is further enhanced to highlight specific road elements, such as pedestrians, other vehicles, traffic signs, etc. This can be achieved through clustering algorithms, object detection algorithms, or specific image processing techniques. The horizontal elevation angle feature map can be fused with data from other sensors (such as camera images) to provide more comprehensive environmental information. This data fusion can improve the autonomous driving system's ability to understand the environment.

[0068] Step S206, using the road environment image and the horizontal altitude angle feature map, a pedestrian recognition model is trained.

[0069] Specifically, a deep learning model is trained to identify and locate pedestrians in the image by combining the road environment image and the horizontal altitude angle feature map. This model is able to learn the characteristics of pedestrians in the image and their spatial position relative to the vehicle. During the training process, the model not only learns to extract visual features from the image, but also learns to extract spatial features from the horizontal altitude angle feature map, and then effectively fuses these features to improve the accuracy and robustness of pedestrian detection. The trained pedestrian recognition model can be deployed in the autonomous driving system to process the road environment information around the vehicle in real time, quickly identify pedestrians, and provide support for decision making. In order to improve the generalization ability and adaptability of the model, the model can be optimized through data enhancement, hyperparameter adjustment and other technologies to ensure that it can maintain high accuracy in different environments and lighting conditions.

[0070] Step S208, collecting road environment information around the vehicle in real time.

[0071] Specifically, autonomous vehicles collect real-time information about the road environment around the vehicle through onboard sensors such as cameras, LiDAR, millimeter-wave radar, etc. This information includes but is not limited to image data, distance measurement, speed estimation, etc. The collected raw data needs to be preprocessed to improve the efficiency and accuracy of subsequent processing. The preprocessing steps include denoising, data synchronization, format conversion, etc. Useful features are extracted from the preprocessed data, such as color, texture, shape, etc. in the image, as well as spatial position information in the laser point cloud.

[0072] Step S210: According to the road environment information, a pedestrian recognition model is used to identify and obtain pedestrian information in the road environment.

[0073] Specifically, the features extracted in step S208 are input into the previously trained pedestrian recognition model. This model can identify pedestrians in the image and provide information such as the location and size of the pedestrians. The pedestrian recognition model processes the input feature data, identifies the presence of pedestrians, and outputs relevant information about the pedestrians, such as location coordinates, confidence, etc. If the system uses multimodal data (such as images and laser point clouds), the data from different sensors needs to be fused to obtain more accurate and robust pedestrian detection results. The identified pedestrian information will be used for further analysis, such as pedestrian tracking, behavior prediction, etc., and to provide support for vehicle decision making, such as path planning, obstacle avoidance, etc.

[0074] In the above-mentioned autonomous driving pedestrian detection method, the system can understand the surrounding environment from multiple dimensions by using the three-dimensional spatial information obtained by the laser radar and the two-dimensional image information captured by the camera. The horizontal altitude angle feature map generated by the laser point cloud data provides the pedestrian recognition model with important clues about the location of pedestrians. Combined with the road environment image and the horizontal altitude angle feature map, the trained pedestrian recognition model can more accurately identify the location and posture of pedestrians. The road environment information around the vehicle collected in real time can be analyzed by the pedestrian recognition model to quickly identify pedestrians and provide instant feedback to the autonomous driving system.

[0075] In an exemplary embodiment, as shown in FIG3 , obtaining a horizontal elevation angle feature map for representing a three-dimensional road environment according to a laser point cloud of a road environment includes:

[0076] Step S302, obtaining a density depth map of the road environment according to the laser point cloud of the road environment;

[0077] Step S304, extracting horizontal disparity features, height features from the ground and angle features from the density depth map;

[0078] Step S306, generating a horizontal altitude angle feature map for representing a three-dimensional road environment according to the horizontal parallax feature, the height feature from the ground and the angle feature.

[0079] Specifically, using the laser point cloud collected by LiDAR, a density depth map is first generated. This map usually reflects the distance of each point in the laser point cloud to the LiDAR sensor, as well as the density distribution of the points.

[0080] Then, from the density depth map, the following three key features are extracted:

[0081] Horizontal disparity feature: This usually refers to the horizontal position difference of each point in the point cloud relative to the vehicle from the perspective of the vehicle.

[0082] Height from ground feature: indicates the height of each point in the point cloud relative to the ground, which helps to distinguish different targets such as pedestrians and vehicles.

[0083] Angle feature: refers to the angle between each point in the point cloud and the vehicle's forward direction, which helps to determine the direction of the target.

[0084] Combining the above-extracted horizontal disparity features, ground height features, and angle features, a two-dimensional horizontal altitude angle feature map is generated. This feature map maps the three-dimensional spatial information onto a two-dimensional plane for subsequent processing and analysis. The horizontal altitude angle feature map provides a structured way to represent complex three-dimensional road environments, which is very important for the perception system of autonomous vehicles. It can help vehicles better understand the surrounding environment, such as distinguishing between pedestrians, vehicles, and road infrastructure. This feature map can be used as input data for training or running machine learning models such as deep neural networks in real time for tasks such as pedestrian detection and obstacle recognition.

[0085] In this embodiment, by converting the three-dimensional point cloud data into a two-dimensional feature map, the autonomous driving system can more accurately perceive the environment around the vehicle and improve the detection and classification capabilities of pedestrians, vehicles, etc. The information provided by the feature map can assist the autonomous driving system in decision-making, such as path planning, obstacle avoidance, and driving strategy selection, thereby improving driving safety.

[0086] In an exemplary embodiment, a pedestrian recognition model is trained using a road environment image and a horizontal altitude angle feature map, including:

[0087] The road environment image is used as the input layer data of the convolutional neural network structure, and the horizontal altitude angle feature map is fused and added to different types of structural layers of the convolutional neural network structure to train and obtain at least one pedestrian recognition model;

[0088] A pedestrian recognition model with the highest recognition accuracy is determined from at least one pedestrian recognition model.

[0089] Specifically, the road environment image is sent as input data to the input layer of the convolutional neural network (which may be TA-CNN in this application).

[0090] The horizontal elevation angle feature map contains spatial and angular information about pedestrians extracted from the laser point cloud data, which will be fused with the image data in different layers of TA-CNN. This fusion can be performed at multiple stages, such as before or after the convolutional layer, pooling layer, or fully connected layer, in order to combine visual and spatial information at different levels of abstraction of the network. In this way, multiple variants of the pedestrian recognition model are trained, each of which fuses the horizontal elevation angle feature map in a specific part of the network structure.

[0091] After training is complete, the recognition accuracy of each model needs to be evaluated. This is usually done by testing the model performance on a validation set, which contains known pedestrian images and corresponding labels. From all trained pedestrian recognition models, select the model with the highest recognition accuracy. This model is able to most accurately identify and classify pedestrians on a given dataset. Road environment images provide rich visual information, while horizontal elevation angle feature maps provide precise spatial information. The combination of the two can help the model better understand the posture, position, and size of pedestrians, even in occluded or complex environments. Convolutional neural networks are good at automatically learning features from images, and fusing additional spatial features can enhance the model's understanding of the input data and improve the accuracy and robustness of detection.

[0092] During the training process, strategies such as regularization techniques, hyperparameter adjustment, and data enhancement can be used to optimize model performance, prevent overfitting, and improve the model's ability to generalize to new data. The final pedestrian recognition model can be deployed on autonomous vehicles to process road environment images in real time, quickly and accurately identify pedestrians, and provide key information for the vehicle's decision-making system.

[0093] In this embodiment, through this process, multi-source data can be fully utilized to train a deep learning model that can accurately identify pedestrians.

[0094] In an exemplary embodiment, according to the road environment information, using the pedestrian recognition model to identify and obtain pedestrian information in the road environment includes:

[0095] Using the pedestrian recognition model, the road environment information is processed for feature recognition to obtain feature recognition results, which include scene features and pedestrian features;

[0096] According to the preset pedestrian attribute information and the preset image segmentation algorithm, the scene features and pedestrian features are verified and identified again to obtain the pedestrian information in the road environment.

[0097] Specifically, the pedestrian recognition model is used to process the road environment information collected in real time to identify the features in the image. These features include scene features and pedestrian features. Scene features may involve environmental elements such as roads, traffic signs, and vehicles, while pedestrian features focus on identifying pedestrians in the image. The feature recognition results obtained after model processing should be able to describe the contextual information of the scene and the specific features of the pedestrians, such as position, size, posture, etc.

[0098] According to the preset pedestrian attribute information (such as pedestrian clothing, height, average body proportions, etc.) and the preset image segmentation algorithm (such as the segmentation method based on deep learning), the identified scene features and pedestrian features are verified and recognized again. This step is to improve the accuracy of recognition, ensure that the recognition results do contain pedestrians, and eliminate the possibility of misidentification.

[0099] After re-verification and identification, the pedestrian information finally obtained in the road environment should include key information such as the pedestrian's specific location, possible movement trajectory, and relative position with other objects.

[0100] Since pedestrian recognition models combine multimodal data (such as visual images and lidar point clouds), this step also includes data fusion techniques to ensure that the information obtained from different sensors is consistent.

[0101] It should be noted that TA-CNN (Task-Assistant Convolutional Neural Network) not only focuses on pedestrian detection, but also considers pedestrian attributes (such as backpacks, hats) and scene attributes (such as vehicles, trees), which helps to distinguish objects with similar shapes and improve detection accuracy. In the field of autonomous driving, treating pedestrian detection as a binary classification problem (i.e. pedestrian or background) may not be sufficient to cope with complex scenes and is prone to false detection. By introducing semantic information, TA-CNN can identify richer category information and improve detection accuracy.

[0102] TA-CNN is able to coordinate training samples with different distributions and convergence speeds, which means it can effectively handle unbalanced data sets and improve the generalization ability of the model. Compared with other traditional methods such as CNN and HOG, TA-CNN has higher accuracy in target recognition and detection, further improving the accuracy and recognition efficiency of pedestrian detection. In autonomous driving, the accuracy of pedestrian detection is crucial. By improving detection accuracy and efficiency, deep learning models help generate navigation paths and driving strategies suitable for the current environment.

[0103] In this embodiment, through this continuous identification and verification process, it is possible to ensure that pedestrian information is accurately extracted from a complex road environment, providing key perception capabilities for autonomous driving and intelligent transportation systems.

[0104] In an exemplary embodiment, after acquiring the road environment dataset around the vehicle, the method further includes:

[0105] De-noising the road environment image;

[0106] Filter and smooth the laser point cloud of the road environment.

[0107] Specifically, the road environment image is denoised:

[0108] The purpose of denoising is to reduce or eliminate unnecessary noise in the image, which may come from the sensor itself, interference during transmission, or environmental factors. Various image processing techniques, such as median filtering, Gaussian filtering, bilateral filtering, etc., can be used to smooth the image and retain important edge information. The denoised image will have clearer details and reduce noise interference, which helps to improve the accuracy of image analysis and recognition algorithms.

[0109] Filter and smooth the laser point cloud of the road environment:

[0110] The purpose of filtering and smoothing is to improve the quality of point cloud data, remove outliers and noise, and make the point cloud smoother and more continuous. Common point cloud preprocessing techniques include statistical filtering (removing outliers), moving least squares smoothing, RANSAC plane fitting, etc. The processed point cloud data will more accurately reflect the actual physical environment and reduce measurement errors and data redundancy.

[0111] After denoising and filtering, data cleaning may also be required to ensure that the dataset does not contain erroneous or inconsistent data points.

[0112] In this embodiment, through these preprocessing steps, the quality of road environment data can be significantly improved, providing accurate and reliable information for the vehicle's perception system, decision-making and control system.

[0113] In an exemplary embodiment, a pedestrian recognition model is trained using a road environment image and a horizontal altitude angle feature map, including:

[0114] Extracting reflected wave signals for preliminary identification of pedestrian information from the laser point cloud of the road environment;

[0115] With reference to the reflected wave signal, the pedestrian recognition model is trained using the road environment image and the horizontal altitude angle feature map.

[0116] Specifically, LiDAR measures the distance and speed of surrounding objects by emitting laser beams and receiving reflected light waves. These reflected wave signals contain rich spatial information and can be used to preliminarily identify the location and possible movements of pedestrians. Combining the reflected wave signals extracted from the laser point cloud with the road environment image can provide more comprehensive pedestrian feature information. Image data can provide visual features of pedestrians, such as color, texture, and shape, while the reflected wave signals provide spatial position and dynamic information of pedestrians. The horizontal altitude angle feature map further enhances the model's understanding of the spatial distribution of pedestrians. This feature map may contain information about the height of pedestrians and horizontal angles relative to the vehicle, which helps to improve the accuracy of pedestrian detection. Using deep learning models such as convolutional neural networks, the fused data is used as input for training. The model learns to extract useful features from this data and recognize the presence of pedestrians.

[0117] In this embodiment, the training method based on multimodal data fusion fully utilizes the advantages of different data sources and improves the accuracy and robustness of the pedestrian recognition model.

[0118] It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0119] Based on the same inventive concept, the embodiment of the present application also provides an automatic driving pedestrian detection device for implementing the automatic driving pedestrian detection method involved above. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme recorded in the above method, so the specific limitations in one or more automatic driving pedestrian detection device embodiments provided below can refer to the limitations of the automatic driving pedestrian detection method above, and will not be repeated here.

[0120] In an exemplary embodiment, as shown in FIG4 , an autonomous driving pedestrian detection device is provided, comprising: an acquisition module 402 for acquiring a road environment dataset around a vehicle, the road environment dataset comprising a road environment image and a laser point cloud of the road environment;

[0121] The processing module 404 is further used to obtain a horizontal elevation angle feature map for representing a three-dimensional road environment according to the laser point cloud of the road environment;

[0122] The model training module 406 is used to train a pedestrian recognition model using the road environment image and the horizontal altitude angle feature map;

[0123] The acquisition module 402 is also used to collect the road environment information around the vehicle in real time;

[0124] The detection module 408 is used to identify and obtain pedestrian information in the road environment using a pedestrian recognition model based on the road environment information.

[0125] In an exemplary embodiment, the processing module 404 is also used to obtain a density depth map of the road environment based on the laser point cloud of the road environment; extract horizontal parallax features, height features from the ground and angle features from the density depth map; and generate a horizontal altitude angle feature map for representing the three-dimensional road environment based on the horizontal parallax features, height features from the ground and angle features.

[0126] In an exemplary embodiment, the model training module 406 is also used to use the road environment image as the input layer data of the convolutional neural network structure, and to fuse and add the horizontal altitude angle feature map in different types of structural layers of the convolutional neural network structure to train and obtain at least one pedestrian recognition model; and to determine the pedestrian recognition model with the highest recognition accuracy from the at least one pedestrian recognition model.

[0127] In an exemplary embodiment, the detection module 408 is also used to use a pedestrian recognition model to perform feature recognition processing on the road environment information to obtain feature recognition results, which include scene features and pedestrian features; based on preset pedestrian attribute information and a preset image segmentation algorithm, the scene features and pedestrian features are re-verified and recognized to obtain pedestrian information in the road environment.

[0128] In an exemplary embodiment, the processing module 404 is further used to perform denoising on the road environment image; and to perform filtering and smoothing on the laser point cloud of the road environment.

[0129] In an exemplary embodiment, the model training module 406 is also used to extract the reflected wave signal for preliminary identification of pedestrian information from the laser point cloud of the road environment; refer to the reflected wave signal, use the road environment image and the horizontal altitude angle feature map, and train to obtain the pedestrian recognition model.

[0130] Each module in the above-mentioned autonomous driving pedestrian detection device can be implemented in whole or in part by software, hardware, or a combination thereof. Each of the above-mentioned modules can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute the operations corresponding to each of the above modules.

[0131] In an exemplary embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be shown in FIG5. The computer device includes a processor, a memory, an input / output interface (Input / Output, referred to as I / O) and a communication interface. The processor, the memory and the input / output interface are connected via a system bus, and the communication interface is connected to the system bus via the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store a road environment data set around the vehicle. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a method for detecting pedestrians in an automatic driving is implemented.

[0132] Those skilled in the art will understand that the structure shown in FIG. 5 is merely a block diagram of a partial structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0133] In an exemplary embodiment, a computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and when the processor executes the computer program, the following steps are implemented:

[0134] Acquire a road environment dataset around the vehicle, where the road environment dataset includes a road environment image and a laser point cloud of the road environment;

[0135] According to the laser point cloud of the road environment, a horizontal elevation angle characteristic map for representing the three-dimensional road environment is obtained;

[0136] Using road environment images and horizontal height angle feature maps, a pedestrian recognition model is trained;

[0137] Collect road environment information around the vehicle in real time;

[0138] According to the road environment information, the pedestrian recognition model is used to identify and obtain the pedestrian information in the road environment.

[0139] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:

[0140] According to the laser point cloud of the road environment, a density depth map of the road environment is obtained;

[0141] Horizontal disparity features, ground height features and angle features are extracted from the density depth map;

[0142] A horizontal altitude angle feature map for representing a three-dimensional road environment is generated according to horizontal parallax features, ground height features and angle features.

[0143] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:

[0144] The road environment image is used as the input layer data of the convolutional neural network structure, and the horizontal altitude angle feature map is fused and added to different types of structural layers of the convolutional neural network structure to train and obtain at least one pedestrian recognition model;

[0145] A pedestrian recognition model with the highest recognition accuracy is determined from at least one pedestrian recognition model.

[0146] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:

[0147] Using the pedestrian recognition model, the road environment information is processed for feature recognition to obtain feature recognition results, which include scene features and pedestrian features;

[0148] According to the preset pedestrian attribute information and the preset image segmentation algorithm, the scene features and pedestrian features are verified and identified again to obtain the pedestrian information in the road environment.

[0149] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:

[0150] De-noising the road environment image;

[0151] Filter and smooth the laser point cloud of the road environment.

[0152] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:

[0153] Extracting reflected wave signals for preliminary identification of pedestrian information from the laser point cloud of the road environment;

[0154] With reference to the reflected wave signal, the pedestrian recognition model is trained using the road environment image and the horizontal altitude angle feature map.

[0155] In one embodiment, a computer readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the following steps are implemented:

[0156] Acquire a road environment dataset around the vehicle, where the road environment dataset includes a road environment image and a laser point cloud of the road environment;

[0157] According to the laser point cloud of the road environment, a horizontal elevation angle characteristic map for representing the three-dimensional road environment is obtained;

[0158] Using road environment images and horizontal height angle feature maps, a pedestrian recognition model is trained;

[0159] Collect road environment information around the vehicle in real time;

[0160] According to the road environment information, the pedestrian recognition model is used to identify and obtain the pedestrian information in the road environment.

[0161] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0162] According to the laser point cloud of the road environment, a density depth map of the road environment is obtained;

[0163] Horizontal disparity features, ground height features and angle features are extracted from the density depth map;

[0164] A horizontal altitude angle feature map for representing a three-dimensional road environment is generated according to horizontal parallax features, ground height features and angle features.

[0165] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0166] The road environment image is used as the input layer data of the convolutional neural network structure, and the horizontal altitude angle feature map is fused and added to different types of structural layers of the convolutional neural network structure to train and obtain at least one pedestrian recognition model;

[0167] A pedestrian recognition model with the highest recognition accuracy is determined from at least one pedestrian recognition model.

[0168] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0169] Using the pedestrian recognition model, the road environment information is processed for feature recognition to obtain feature recognition results, which include scene features and pedestrian features;

[0170] According to the preset pedestrian attribute information and the preset image segmentation algorithm, the scene features and pedestrian features are verified and identified again to obtain the pedestrian information in the road environment.

[0171] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0172] De-noising the road environment image;

[0173] Filter and smooth the laser point cloud of the road environment.

[0174] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0175] Extracting reflected wave signals for preliminary identification of pedestrian information from the laser point cloud of the road environment;

[0176] With reference to the reflected wave signal, the pedestrian recognition model is trained using the road environment image and the horizontal altitude angle feature map.

[0177] In one embodiment, a computer program product is provided, comprising a computer program, which, when executed by a processor, implements the following steps:

[0178] Acquire a road environment dataset around the vehicle, where the road environment dataset includes a road environment image and a laser point cloud of the road environment;

[0179] According to the laser point cloud of the road environment, a horizontal elevation angle characteristic map for representing the three-dimensional road environment is obtained;

[0180] Using road environment images and horizontal height angle feature maps, a pedestrian recognition model is trained;

[0181] Collect road environment information around the vehicle in real time;

[0182] According to the road environment information, the pedestrian recognition model is used to identify and obtain the pedestrian information in the road environment.

[0183] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0184] According to the laser point cloud of the road environment, a density depth map of the road environment is obtained;

[0185] Horizontal disparity features, ground height features and angle features are extracted from the density depth map;

[0186] A horizontal altitude angle feature map for representing a three-dimensional road environment is generated according to horizontal parallax features, ground height features and angle features.

[0187] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0188] The road environment image is used as the input layer data of the convolutional neural network structure, and the horizontal altitude angle feature map is fused and added to different types of structural layers of the convolutional neural network structure to train and obtain at least one pedestrian recognition model;

[0189] A pedestrian recognition model with the highest recognition accuracy is determined from at least one pedestrian recognition model.

[0190] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0191] Using the pedestrian recognition model, the road environment information is processed for feature recognition to obtain feature recognition results, which include scene features and pedestrian features;

[0192] According to the preset pedestrian attribute information and the preset image segmentation algorithm, the scene features and pedestrian features are verified and identified again to obtain the pedestrian information in the road environment.

[0193] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0194] De-noising the road environment image;

[0195] Filter and smooth the laser point cloud of the road environment.

[0196] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0197] Extracting reflected wave signals for preliminary identification of pedestrian information from the laser point cloud of the road environment;

[0198] With reference to the reflected wave signal, the pedestrian recognition model is trained using the road environment image and the horizontal altitude angle feature map.

[0199] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant regulations.

[0200] A person of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment method can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, an artificial intelligence (AI) processor, etc., but are not limited to this.

[0201] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0202] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A method for detecting pedestrians in an autonomous driving system, characterized in that: The method comprises: Acquire a road environment dataset around the vehicle, wherein the road environment dataset includes a road environment image and a laser point cloud of the road environment; Acquire a horizontal elevation angle characteristic map for representing a three-dimensional road environment according to the laser point cloud of the road environment; Using the road environment image and the horizontal altitude angle feature map, a pedestrian recognition model is trained; Collect road environment information around the vehicle in real time; According to the road environment information, the pedestrian recognition model is used to identify and obtain pedestrian information in the road environment.

2. The method according to claim 1, characterized in that The step of obtaining a horizontal elevation angle feature map for representing a three-dimensional road environment according to the laser point cloud of the road environment comprises: Acquire a density depth map of the road environment according to the laser point cloud of the road environment; Extracting horizontal disparity features, height features from the ground and angle features from the density depth map; A horizontal altitude angle feature map for representing a three-dimensional road environment is generated according to the horizontal parallax feature, the height feature from the ground and the angle feature.

3. The method according to claim 1, characterized in that The method of using the road environment image and the horizontal altitude angle feature map to train and obtain a pedestrian recognition model includes: Using the road environment image as input layer data of a convolutional neural network structure, respectively fusing and adding the horizontal altitude angle feature map into different types of structural layers of the convolutional neural network structure, and training to obtain at least one pedestrian recognition model; A pedestrian recognition model with the highest recognition accuracy is determined from the at least one pedestrian recognition model.

4. The method according to claim 1, characterized in that The method of identifying and obtaining pedestrian information in the road environment by using the pedestrian recognition model according to the road environment information includes: Using the pedestrian recognition model, performing feature recognition processing on the road environment information to obtain a feature recognition result, wherein the feature recognition result includes scene features and pedestrian features; According to the preset pedestrian attribute information and the preset image segmentation algorithm, the scene features and the pedestrian features are verified and identified again to obtain the pedestrian information in the road environment.

5. The method according to claim 1, characterized in that After obtaining the road environment dataset around the vehicle, the method further includes: Performing denoising processing on the road environment image; The laser point cloud of the road environment is filtered and smoothed.

6. The method according to claim 5, characterized in that The method of using the road environment image and the horizontal altitude angle feature map to train and obtain a pedestrian recognition model includes: Extracting a reflected wave signal for preliminary identification of pedestrian information from the laser point cloud of the road environment; Referring to the reflected wave signal, a pedestrian recognition model is trained using the road environment image and the horizontal altitude angle feature map.

7. An automatic driving pedestrian detection device, characterized in that: The device comprises: An acquisition module is used to acquire a road environment dataset around the vehicle, wherein the road environment dataset includes a road environment image and a laser point cloud of the road environment; The processing module is further used to obtain a horizontal elevation angle characteristic map for representing a three-dimensional road environment according to the laser point cloud of the road environment; A model training module, used to train a pedestrian recognition model using the road environment image and the horizontal altitude angle feature map; The acquisition module is also used to collect the road environment information around the vehicle in real time; The detection module is used to identify and obtain pedestrian information in the road environment based on the road environment information and using the pedestrian recognition model.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 6 are implemented.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images