A method for predicting the trajectory of an obstacle, a processing module, and an electronic device.

Abstract

This application proposes a method, processing module, and electronic device for predicting the trajectory of an obstacle. The method includes: acquiring a target image of a target area, millimeter-wave radar data of initial obstacles, and lidar data; processing the lidar data and target image using a first neural network model to extract the target obstacle from the initial obstacles and determine the position and velocity data of the target obstacle; labeling the target obstacle in the target image; generating multiple convex hulls based on the position data; calculating the number of first moving targets in each convex hull using the millimeter-wave radar data; inputting the labeled target image into a second neural network model to obtain the number of second moving targets in each convex hull; inputting the position and velocity data into a third neural network model to obtain the initial predicted trajectory of the target obstacle; and correcting the initial predicted trajectory based on the number of first and second moving targets to obtain the final predicted trajectory of the target obstacle.

Description

Technical Field

[0001] This application relates to the field of intelligent driving technology, and in particular to a method for predicting the trajectory of obstacles, a processing module, and an electronic device. Background Technology

[0002] With the development of intelligent driving technology, in order to improve traffic safety, vulnerable road users (VRU) perception systems are usually deployed at intersections or on vehicles where traffic accidents are frequent. The VRU perception system can calculate the trajectory information of the VRU object and determine whether the VRU object will collide with the vehicle based on the trajectory information of the VRU object. If a collision is likely to occur, a safety warning will be issued.

[0003] In related technologies, it is difficult to accurately predict the trajectory information of VRU objects. Summary of the Invention

[0004] This application proposes an obstacle trajectory prediction method, processing module, and electronic device to solve the technical problem of difficulty in accurately predicting the trajectory information of VRU objects.

[0005] To achieve the above objectives, according to a first aspect of this application, a method for predicting the trajectory of an obstacle is provided, comprising:

[0006] Acquire a target image of the target area, millimeter-wave radar data of the initial obstacle, and lidar data of the initial obstacle; wherein the initial obstacle is located in the target area;

[0007] The first neural network model is used to process the lidar data and the target image to extract the target obstacle from the initial obstacle and determine the position and velocity data of the target obstacle.

[0008] The target obstacles in the target image are labeled;

[0009] Based on the position data of the target obstacle, generate multiple convex hulls;

[0010] Using millimeter-wave radar data of the initial obstacles, the number of first moving targets in each convex hull is calculated;

[0011] The labeled target image is input into the second neural network model to obtain the number of second moving targets in each convex hull;

[0012] The position and velocity data of the target obstacle are input into the third neural network model to obtain the initial predicted trajectory of the target obstacle;

[0013] Based on the number of the first moving targets and the number of the second moving targets, the initial predicted trajectory is corrected to obtain the target predicted trajectory of the target obstacle.

[0014] In some embodiments, the millimeter-wave radar data characterizes a first state of the initial obstacle and the location information of the initial obstacle, wherein the first state includes movement and stillness.

[0015] In some embodiments, multiple convex hulls are generated based on the position data of the target obstacle, including:

[0016] Based on the location data of the target obstacles, the target obstacles are clustered to obtain multiple clustering results;

[0017] Calculate the minimum outer connection polygon for each clustering result to obtain the convex hull corresponding to each clustering result.

[0018] In some embodiments, using millimeter-wave radar data of the initial obstacle, the number of first moving targets in each convex hull is calculated, including:

[0019] Using millimeter-wave radar data of the initial obstacle, it is determined whether the initial obstacle is located inside the convex hull, thus obtaining a first detection result;

[0020] Based on the first detection result, the number of first moving targets in each convex hull is calculated; the first moving targets include: initial obstacles in a first state of motion.

[0021] In some embodiments, the labeled target image is input into a second neural network model to obtain the number of second moving targets in each convex hull, including:

[0022] The labeled target image is input into a second neural network model to obtain a second state of the target obstacle; wherein, the second state includes: motion and stillness;

[0023] Based on the second state of the target obstacle, the number of second moving targets in each convex hull is obtained; wherein, the second moving targets include: target obstacles in the second state of movement.

[0024] In some embodiments, the initial predicted trajectory is corrected based on the number of the first moving targets and the number of the second moving targets to obtain the target predicted trajectory of the target obstacle, including:

[0025] When the number of the first moving targets is not equal to 0 and the number of the second moving targets is not equal to 0, the first number of initial predicted trajectories are extracted from the initial predicted trajectories arranged in descending order of length to obtain the first trajectory; the first number is determined based on the number of the first moving targets and the number of the second moving targets.

[0026] The first trajectory and the third trajectory are combined to obtain the target predicted trajectory, wherein the third trajectory is obtained by trajectory compression of the second trajectory, and the second trajectory is the complement of the initial predicted trajectory and the first trajectory.

[0027] In some embodiments, the initial predicted trajectory is corrected based on the number of the first moving targets and the number of the second moving targets to obtain the target predicted trajectory of the target obstacle, including:

[0028] When the number of the first moving targets is not equal to 0 and the number of the second moving targets is equal to 0, the first second number of initial predicted trajectories are extracted from the initial predicted trajectories arranged in descending order of length to obtain the fourth trajectory; the second number is determined based on the number of the first moving targets.

[0029] The fourth and sixth trajectories are combined to obtain the target predicted trajectory, wherein the sixth trajectory is obtained by compressing the fifth trajectory, and the fifth trajectory is the complement of the initial predicted trajectory and the fourth trajectory.

[0030] In some embodiments, the initial predicted trajectory is corrected based on the number of the first moving targets and the number of the second moving targets to obtain the target predicted trajectory of the target obstacle, including:

[0031] When the number of the first moving target is equal to 0 and the number of the second moving target is equal to 0, the initial predicted trajectory is compressed to obtain the target predicted trajectory.

[0032] In some embodiments, the initial predicted trajectory is corrected based on the number of the first moving targets and the number of the second moving targets to obtain the target predicted trajectory of the target obstacle, including:

[0033] If the number of the first moving targets is determined to be 0 and the number of the second moving targets is not determined to be 0, the first third number of initial predicted trajectories are extracted from the initial predicted trajectories arranged in descending order of length to obtain the seventh trajectory; the third number is determined based on the number of the second moving targets.

[0034] The seventh and ninth trajectories are combined to obtain the target predicted trajectory, wherein the ninth trajectory is obtained by compressing the eighth trajectory, and the eighth trajectory is the complement of the initial predicted trajectory and the seventh trajectory.

[0035] According to a second aspect of this application, a processing module is provided, the processing module being configured to perform the following steps:

[0036] Acquire a target image of the target area, millimeter-wave radar data of the initial obstacle, and lidar data of the initial obstacle; wherein the initial obstacle is located in the target area;

[0037] The first neural network model is used to process the lidar data and the target image to extract the target obstacle from the initial obstacle and determine the position and velocity data of the target obstacle.

[0038] The target obstacles in the target image are labeled;

[0039] Based on the position data of the target obstacle, generate multiple convex hulls;

[0040] Using millimeter-wave radar data of the initial obstacles, the number of first moving targets in each convex hull is calculated;

[0041] The labeled target image is input into the second neural network model to obtain the number of second moving targets in each convex hull;

[0042] The position and velocity data of the target obstacle are input into the third neural network model to obtain the initial predicted trajectory of the target obstacle;

[0043] Based on the number of the first moving targets and the number of the second moving targets, the initial predicted trajectory is corrected to obtain the target predicted trajectory of the target obstacle.

[0044] According to a third aspect of this application, an electronic device is provided, the electronic device including a processor and a memory for storing processor-executable instructions, wherein the processor executes the instructions to implement the steps of any of the obstacle trajectory prediction methods described above.

[0045] According to a fourth aspect of this application, a storage medium is provided that stores computer instructions thereon, which, when executed by a processor, implement the steps described in any of the obstacle trajectory prediction methods above.

[0046] The technical solution of this application embodiment can achieve the following beneficial effects: Based on the obstacle trajectory prediction method provided in this application embodiment, a convex hull is generated using LiDAR data and image data. Using the convex hull as the processing unit, a first moving target with a motion tendency inside the convex hull is obtained using millimeter-wave radar data, and a second moving target with a motion tendency inside the convex hull is obtained using image data. Since multiple data sources are comprehensively utilized, the accuracy of motion tendency judgment is improved. Furthermore, the initial predicted trajectory is corrected using the number of the first moving target and the number of the second moving target, and the trajectory without a motion tendency is compressed to obtain a more accurate target prediction trajectory, which improves the accuracy of intelligent driving road condition judgment, can accurately help the vehicle avoid target obstacles, and improves intelligent driving safety.

[0047] Other features and advantages of this application will be described in detail in the following detailed description section. Attached Figure Description

[0048] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0049] To gain a more complete understanding of this application and its beneficial effects, the following description will be provided in conjunction with the accompanying drawings, wherein the same reference numerals in the following description denote the same parts.

[0050] Figure 1 A flowchart illustrating an obstacle trajectory prediction method provided in an embodiment of this application;

[0051] Figure 2 This is a flowchart illustrating an obstacle trajectory prediction method provided in an embodiment of this application. Detailed Implementation

[0052] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0053] In the description of this application, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of the stated features. In the description of this application, "multiple" means two or more, unless otherwise explicitly specified.

[0054] "A and / or B" includes the following three combinations: A only, B only, and a combination of A and B.

[0055] The use of "applies to" or "configured to" in this application implies open and inclusive language, which does not exclude the applicability to or configuration to devices performing additional tasks or steps. Additionally, the use of "based on" implies openness and inclusivity, because processes, steps, calculations, or other actions "based on" one or more of the stated conditions or values ​​may in practice be based on additional conditions or values ​​beyond those stated.

[0056] In this application, the term "exemplary" is used to mean "used as an example, illustration, or description." Any embodiment described as "exemplary" in this application is not necessarily to be construed as being more preferred or advantageous than other embodiments. The following description is provided to enable any person skilled in the art to make and use this application. Details are set forth in the following description for purposes of explanation. It should be understood that those skilled in the art will recognize that this application can be made without using these specific details. In other instances, well-known structures and processes are not described in detail to avoid obscuring the description of this application with unnecessary detail. Therefore, this application is not intended to be limited to the embodiments shown, but is consistent with the broadest scope of the principles and features disclosed in this application.

[0057] To improve traffic safety, vulnerable road user (VRU) perception systems are typically deployed at intersections or on vehicles where traffic accidents are frequent. The VRU perception system can calculate the trajectory information of the VRU object and determine whether a collision will occur between the VRU object and the vehicle. If a collision is likely, a safety warning will be issued.

[0058] In related technologies, in complex scenarios such as urban roads, VRU objects are numerous and relatively small in size, occupying limited pixel or signal features in image or radar data. This makes it easy for detection algorithms to miss or falsely detect objects and has low perception accuracy, making it difficult to accurately identify VRU objects.

[0059] Furthermore, due to factors such as sensor noise and the irregularity of moving targets within the VRU object, the position detection results of the VRU object fluctuate. This makes it easy to generate errors during target association, thereby affecting the accurate judgment of attributes such as the speed of moving targets and impacting trajectory prediction. In addition, mutual occlusion between groups of VRU objects further reduces the accuracy and reliability of detection.

[0060] To address the technical problem of accurately calculating the trajectory of VRU objects, this application proposes a trajectory prediction method, processing module, and electronic device for obstacles to overcome the aforementioned problem.

[0061] On one hand, embodiments of this application provide a method for predicting the trajectory of obstacles, which can be executed by a smart device installed on the roadside or a vehicle traveling on the road. For example... Figure 1 As shown, the method may include the following steps:

[0062] S101: Acquire target images of the target area, millimeter-wave radar data of the initial obstacle, and lidar data of the initial obstacle; wherein the initial obstacle is located in the target area.

[0063] S102: Use the first neural network model to process the lidar data and target image, extract the target obstacle from the initial obstacle, and determine the position and velocity data of the target obstacle.

[0064] S103: Label the target obstacles in the target image.

[0065] S104: Generate multiple convex hulls based on the position data of the target obstacle.

[0066] S105: Calculate the number of first moving targets in each convex hull using millimeter-wave radar data of the initial obstacles.

[0067] S106: Input the labeled target image into the second neural network model to obtain the number of second moving targets in each convex hull.

[0068] S107: Input the position and velocity data of the target obstacle into the third neural network model to obtain the initial predicted trajectory of the target obstacle.

[0069] S108: Based on the number of the first moving target and the number of the second moving target, correct the initial predicted trajectory to obtain the target predicted trajectory of the obstacle.

[0070] The following is a detailed description of an obstacle trajectory prediction method provided by an embodiment of this application.

[0071] S101: Acquire target images of the target area, millimeter-wave radar data of the initial obstacle, and lidar data of the initial obstacle; wherein the initial obstacle is located in the target area.

[0072] In some embodiments, the target area includes areas prone to traffic accidents, such as busy intersections, school drop-off and pick-up areas, construction areas, etc.

[0073] Initial obstacles include: VRU (vulnerable road users) objects, vehicles, traffic lights, road signs, streetlights, etc. VRU objects include: pedestrians, cyclists, and motorcyclists. Understandably, initial obstacles include both stationary obstacles (traffic lights, road signs, streetlights, stationary pedestrians, stationary cyclists, stationary vehicles, etc.) and moving obstacles (moving pedestrians, moving cyclists, moving vehicles, etc.).

[0074] LiDAR data is collected using LiDAR (Light Laser Detection and Ranging). LiDAR data is a type of point cloud data. The LiDAR emits a series of laser pulses (laser beams) towards a target object. These pulses are reflected back after encountering the object. The LiDAR receiver records information such as the time and intensity of each pulse's return. Based on the speed of light and the time difference, the distance to the target object is calculated. Combined with information such as the laser pulse's emission angle, the target object's position can be further determined.

[0075] Millimeter-wave radar data is acquired through millimeter-wave radar. Millimeter-wave radar is a sensor that uses millimeter waves with wavelengths of 1-10mm and frequencies of 30-300GHz for detection. Its advantage lies in accurate speed measurement.

[0076] In some embodiments, millimeter-wave radar data characterizes the first state of an initial obstacle and its position information. The first state includes moving and stationary. There are multiple initial obstacles, and different initial obstacles can have different first states. An initial obstacle may have a "moving" first state or a "stationary" first state.

[0077] S102: Use the first neural network model to process the lidar data and target image, extract the target obstacle from the initial obstacle, and determine the position and velocity data of the target obstacle.

[0078] In some embodiments, lidar data and target images are input into a first neural network model for processing. The first neural network model can extract the target obstacle from the initial obstacle and further determine the position and velocity data of the target obstacle.

[0079] The target obstacle includes VRU objects.

[0080] The first neural network model is a deep neural network, including the RT-DETR neural network model. The RT-DETR neural network model is an end-to-end object detection model based on Transformer, which can capture global information in the image, consider the whole image rather than just local regions, and help to understand the context of features, thereby improving the accuracy of detection.

[0081] S103: Label the target obstacles in the target image.

[0082] In some embodiments, the target image includes initial obstacles and a background. The initial obstacles include target obstacles. After the target obstacles are identified, the target obstacles in the target image are labeled to obtain the labeled target image.

[0083] S104: Generate multiple convex hulls based on the position data of the target obstacle.

[0084] In some embodiments, multiple convex hulls are generated based on the position data of the target obstacle, specifically including:

[0085] S1: Based on the location data of the target obstacles, cluster the target obstacles to obtain multiple clustering results;

[0086] S2: Calculate the minimum outer connection polygon for each clustering result to obtain the convex hull corresponding to each clustering result.

[0087] In some embodiments, the target obstacles are clustered based on their location data to obtain multiple clustering results. Each clustering result represents a group of nearby target obstacles. The Monoton Chain algorithm is used to solve for the minimum outer connection polygon, i.e., the convex hull, of each clustering result.

[0088] Specifically, the Monoton Chain algorithm is an efficient algorithm for finding the convex hull of a planar point set. After clustering the target obstacles, each cluster result can be regarded as a set of target points with similar spatial positions. The Monoton Chain algorithm can quickly and accurately find the smallest convex polygon boundary (smallest outer connected polygon) that can enclose this cluster result. This smallest convex polygon boundary (smallest outer connected polygon) is the convex hull.

[0089] S105: Calculate the number of first moving targets in each convex hull using millimeter-wave radar data of the initial obstacles.

[0090] In some embodiments, the number of first moving targets in each convex hull is calculated using millimeter-wave radar data of the initial obstacle, specifically including:

[0091] S1: Using millimeter-wave radar data of the initial obstacle, detect whether the initial obstacle is located inside the convex hull to obtain the first detection result;

[0092] S2: Based on the first detection result, calculate the number of first moving targets in each convex hull; the first moving targets include: the initial obstacles in the first state of motion.

[0093] In some embodiments, each initial obstacle has corresponding millimeter-wave radar data, which includes the location information of the initial obstacle it indicates. The location information of the millimeter-wave radar data is determined using a vector cross product method or a ray method to determine whether it is located inside the convex hull. If the location information of a certain millimeter-wave radar data is located inside the convex hull, then the initial obstacle indicated by that millimeter-wave radar data is considered to be located inside the convex hull, thus obtaining a first detection result. The first detection result corresponds one-to-one with the initial obstacle, indicating whether the initial obstacle is located inside the convex hull. The millimeter-wave radar data also includes the first state of the initial obstacle it indicates (the first state includes moving and stationary). Based on the first detection result, initial obstacles located inside the convex hull and whose first state is moving are selected as first moving targets. Then, the number of first moving targets in each convex hull is determined. The first moving target represents an obstacle with a tendency to move within the convex hull, determined based on the millimeter-wave radar data.

[0094] By using the minimum outer connection polygon, the potential erroneous association between millimeter-wave radar data and obstacles can be resolved. This embodiment only needs to focus on the number of first moving targets within the convex hull, without needing to further determine which initial obstacles within the convex hull are in a moving state. Subsequent processing is also unified using the convex hull as the processing unit, thus reducing data processing complexity. Furthermore, millimeter-wave radar's precise velocity measurement can better help distinguish whether there are first moving targets with a tendency to move within the convex hull.

[0095] S106: Input the labeled target image into the second neural network model to obtain the number of second moving targets in each convex hull.

[0096] In some embodiments, the labeled target image is input into a second neural network model to obtain the number of second moving targets in each convex hull, specifically including:

[0097] S1: Input the labeled target image into the second neural network model to obtain the second state of the target obstacle; wherein, the second state includes: moving and stationary;

[0098] S2: Based on the second state of the target obstacle, obtain the number of second moving targets in each convex hull; wherein, the second moving targets include: target obstacles in the second state of motion.

[0099] In some embodiments, the labeled target image and the convex hull obtained in step S104 are input into a second neural network model. The second neural network model is used to extract target obstacles from the labeled target image, obtain the pose of the target obstacles through a pose recognition algorithm, analyze the pose of the target obstacles to obtain the second state of the target obstacles, each target obstacle has its corresponding second state, and target obstacles located inside the convex hull and whose second state is motion are selected from the target obstacles as second moving targets. Then, the number of second moving targets in each convex hull is determined. The second moving target represents the obstacle with a tendency to move in the convex hull obtained based on the target image.

[0100] The second neural network model is a pose recognition (pose determination) model, which is also a type of deep neural network. This includes the RTMPOSE neural network model and the YOLOX-Pose neural network model. The YOLOX-Pose neural network model applies the YOLOX-Pose algorithm, a heatmap-free joint detection method based on the YOLOX object detection framework, used for 2D pose recognition of images. The RTMPOSE neural network model is a real-time multi-person pose estimation model based on MMPose (an open-source pose estimation toolkit based on PyTorch). It uses a top-down approach, using existing detectors to obtain human detection bounding boxes and then estimating the pose of each target obstacle separately.

[0101] Based on the above embodiments, millimeter-wave radar data is used to reflect the first state of the initial obstacle, and target image and attitude recognition algorithm are used to reflect the second state of the target obstacle. The two dimensions are used to determine whether the obstacle is moving or stationary, which improves the accuracy of the judgment and provides a data basis for subsequent trajectory correction.

[0102] In some embodiments, the labeled target image of the current frame and the labeled target images of historical frames can be input into the second neural network model to obtain the second state of the current frame. Using multiple frames to add a time dimension can better ensure perceptual stability.

[0103] S107: Input the position and velocity data of the target obstacle into the third neural network model to obtain the initial predicted trajectory of the target obstacle.

[0104] In some embodiments, the third neural network model is a trajectory prediction model and also a deep neural network, including the DTPP neural network model. The DTPP neural network model is mainly based on tree policy planning. Its core idea is to approximate the continuous space policy planning problem by sampling a set of discrete obstacle trajectories at multiple stages to form a trajectory tree and predicting the motion of other obstacles. A scene tree is formed with each obstacle trajectory segment as a condition, and then dynamic programming is used to derive the optimal trajectory, that is, to obtain the initial predicted trajectory of each target obstacle.

[0105] The initial predicted trajectory obtained through the third neural network model has low accuracy. It is necessary to combine the number of first moving targets and the number of second moving targets to correct the trajectory and obtain a more accurate target predicted trajectory.

[0106] In some embodiments, the position and velocity data of the target obstacle from multiple frames can be input into a third neural network model for trajectory prediction. Using multiple frames of data to add a time dimension can better ensure the stability of the prediction.

[0107] S108: Based on the number of the first moving target and the number of the second moving target, correct the initial predicted trajectory to obtain the target predicted trajectory of the obstacle.

[0108] Based on millimeter-wave radar data, the number of first moving targets in each convex hull can be obtained. The first state of the first moving targets is "moving". Based on the target image, the number of second moving targets in each convex hull can be obtained. The second state of the second moving targets is "moving". It is necessary to perform trajectory compression processing on the initial predicted trajectory corresponding to the "stationary" target obstacle, and not process the initial predicted trajectory corresponding to the "moving" target obstacle.

[0109] In some embodiments, the initial predicted trajectory is corrected based on the number of first moving targets and the number of second moving targets to obtain the target predicted trajectory of the obstacle, specifically including:

[0110] S1: When the number of the first moving target is not equal to 0 and the number of the second moving target is not equal to 0, extract the first number of initial predicted trajectories from the initial predicted trajectories arranged in descending order of length to obtain the first trajectory; the first number is determined based on the number of the first moving target and the number of the second moving target.

[0111] S2: Combine the first trajectory and the third trajectory to obtain the target predicted trajectory. The third trajectory is obtained by compressing the second trajectory. The second trajectory is the complement of the initial predicted trajectory and the first trajectory.

[0112] In some embodiments, when the number of first moving targets is not equal to 0 and the number of second moving targets is not equal to 0, for each target obstacle in the convex hull, the lengths of the initial predicted trajectories of the target obstacles are sorted in descending order. From the initial predicted trajectories sorted in descending order of length, the first number of initial predicted trajectories are extracted to obtain the first trajectory. The first number is determined based on the number of first moving targets and the number of second moving targets, and it can be assumed that there are a first number of "moving" target obstacles in the convex hull.

[0113] For example, the first quantity is equal to the maximum value between the quantity of the first moving target and the quantity of the second moving target. Or, the first quantity is equal to the minimum value between the quantity of the first moving target and the quantity of the second moving target.

[0114] Obtain the complement of the first trajectory from the initial predicted trajectory to obtain the second trajectory. The first trajectory is the one with the longer initial predicted trajectory, and the second trajectory is the one with the shorter initial predicted trajectory. Since there are a first number of "moving" target obstacles in the convex hull, the trajectories of these target obstacles (the first trajectory) are not compressed. The second trajectory can be considered as the trajectory of the "stationary" target obstacles. Trajectory compression is performed on the second trajectory by adjusting the coordinates of each data point in the second trajectory to 0.1 times its original value, thus compressing the second trajectory to 0.1 times its original value, resulting in the third trajectory. Combine the first and third trajectories to obtain the target predicted trajectory.

[0115] Based on the above embodiments, the trajectory is corrected by comprehensively utilizing the number of the first moving target and the number of the second moving target, which makes up for the defect that the number of the first moving target is inaccurate due to jitter in the single millimeter-wave radar data.

[0116] In some embodiments, the first quantity can be equal to the minimum of the number of first moving targets and the number of second moving targets. The fourth quantity is equal to the absolute value of the difference between the number of first moving targets and the number of second moving targets. Then, the top four quantities of trajectories from the second trajectories are selected as the tenth trajectory. Since the state of the tenth trajectory is between motion and stillness, a milder trajectory compression is performed, adjusting the coordinates of each data point in the tenth trajectory to 0.4-0.6 times its original value, thus compressing the tenth trajectory to 0.4-0.6 times its original value. Combining the first trajectory, the compressed tenth trajectory, and the uncompressed second trajectory yields the target prediction trajectory.

[0117] In some embodiments, the initial predicted trajectory is corrected based on the number of first moving targets and the number of second moving targets to obtain the target predicted trajectory of the obstacle, specifically including:

[0118] S1: When the number of the first moving target is not equal to 0 and the number of the second moving target is equal to 0, extract the first second number of initial predicted trajectories from the initial predicted trajectories arranged in descending order of length to obtain the fourth trajectory; the second number is determined based on the number of the first moving target.

[0119] S2: Combine the fourth and sixth trajectories to obtain the target predicted trajectory. The sixth trajectory is obtained by compressing the fifth trajectory, and the fifth trajectory is the complement of the initial predicted trajectory and the fourth trajectory.

[0120] In some embodiments, when the number of first moving targets is not equal to 0 and the number of second moving targets is equal to 0, for each target obstacle in the convex hull, the lengths of the initial predicted trajectories of the target obstacles are sorted in descending order. From the initial predicted trajectories sorted in descending order of length, the first second number of initial predicted trajectories are extracted to obtain the fourth trajectory. The second number is determined based on the number of first moving targets, and it can be assumed that there are a second number of "moving" target obstacles in the convex hull.

[0121] For example, the number of the first moving target can be multiplied by a first coefficient to obtain the second number. The first coefficient ranges from [0,1), and can be 0.8 or 0.9. To ensure safety, the first coefficient should be set larger; to consider driving comfort and avoid accidental braking, the first coefficient should be set smaller.

[0122] Obtain the complement of the fourth trajectory from the initial predicted trajectory to obtain the fifth trajectory. The fourth trajectory is the one with a longer initial predicted trajectory, and the fifth trajectory is the one with a shorter initial predicted trajectory. Since there are a second number of "moving" target obstacles in the convex hull, the trajectories of these target obstacles (the fourth trajectory) are not compressed. The fifth trajectory can be considered as the trajectory of the "stationary" target obstacles. Trajectory compression is performed on the fifth trajectory by adjusting the coordinates of each data point in the fifth trajectory to 0.1 times its original value, thus compressing the fifth trajectory to 0.1 times its original value, resulting in the sixth trajectory. Combining the fourth and sixth trajectories yields the target predicted trajectory.

[0123] In some embodiments, the initial predicted trajectory is corrected according to the number of the first moving target and the number of the second moving target to obtain the target predicted trajectory of the target obstacle. Specifically, this includes: when the number of the first moving target is equal to 0 and the number of the second moving target is equal to 0, the initial predicted trajectory is compressed to obtain the target predicted trajectory.

[0124] If the number of first moving targets is 0 and the number of second moving targets is 0, it means that all the target obstacles in the convex hull are "stationary". Then, all the initial predicted trajectories are compressed to 0.1 times their original size to obtain the target predicted trajectory.

[0125] In some embodiments, the initial predicted trajectory is corrected based on the number of first moving targets and the number of second moving targets to obtain the target predicted trajectory of the obstacle, specifically including:

[0126] S1: Given that the number of the first moving target is equal to 0 and the number of the second moving target is not equal to 0, extract the first third number of initial predicted trajectories from the initial predicted trajectories arranged in descending order of length to obtain the seventh trajectory; the third number is determined based on the number of the second moving target.

[0127] S2: Combine the seventh and ninth trajectories to obtain the target predicted trajectory. The ninth trajectory is obtained by compressing the eighth trajectory, which is the complement of the initial predicted trajectory and the seventh trajectory.

[0128] In some embodiments, when the number of first moving targets is equal to 0 and the number of second moving targets is not equal to 0, for each target obstacle in the convex hull, the lengths of the initial predicted trajectories of the target obstacles are sorted in descending order. From the initial predicted trajectories sorted in descending order of length, the first third number of initial predicted trajectories are extracted to obtain the seventh trajectory. The third number is determined based on the number of second moving targets, and it can be assumed that there are a third number of "moving" target obstacles in the convex hull.

[0129] For example, the number of the second moving target can be multiplied by the second coefficient to obtain the third number. The value range of the second coefficient is [0,1), for example, it can be 0.8 or 0.9. If you want to ensure safety, you should set the second coefficient to a larger value; if you want to consider the driving comfort of the vehicle and avoid accidental braking, you should set the second coefficient to a smaller value.

[0130] Obtain the complement of the seventh trajectory from the initial predicted trajectory to obtain the eighth trajectory. The seventh trajectory is the one with a longer initial predicted trajectory, and the eighth trajectory is the one with a shorter initial predicted trajectory. Since there are a third number of "moving" target obstacles in the convex hull, the trajectories of these target obstacles (the seventh trajectory) are not compressed. The eighth trajectory can be considered as the trajectory of the "stationary" target obstacles. Trajectory compression is performed on the eighth trajectory by adjusting the coordinates of each data point in the eighth trajectory to 0.1 times its original value, thus compressing the eighth trajectory to 0.1 times its original value, resulting in the ninth trajectory. Combining the seventh and ninth trajectories yields the target predicted trajectory.

[0131] In some embodiments, after obtaining the target predicted trajectory of the target obstacle, the target predicted trajectory can be sent to nearby vehicles to prompt them to plan intelligent driving routes reasonably and avoid the target obstacle.

[0132] This application embodiment generates a convex hull using LiDAR data and image data. Using the convex hull as the processing unit, millimeter-wave radar data is used to obtain a first moving target with a motion tendency inside the convex hull, and image data is used to obtain a second moving target with a motion tendency inside the convex hull. By comprehensively utilizing multiple data sources, the accuracy of motion tendency judgment is improved, which can enhance the perception effect of groups of target obstacles and overcome the defects of speed instability caused by relying solely on speed data and position jitter caused by relying solely on position data. Furthermore, by using the number of first moving targets and the number of second moving targets, the initial predicted trajectory is corrected, and the trajectory without motion tendency is compressed to obtain a more accurate target prediction trajectory, which improves the accuracy of intelligent driving road condition judgment and can accurately help vehicles avoid target obstacles, thereby improving the safety of intelligent driving.

[0133] In some embodiments, see Figure 2 As shown, the obstacle trajectory prediction method specifically includes the following steps:

[0134] Step 1: Acquire the target image of the target area, the millimeter-wave radar data of the initial obstacle, and the lidar data of the initial obstacle; wherein, the initial obstacle is located in the target area.

[0135] Step 2: Use the pre-fusion network (first neural network model) to process the LiDAR data and target image, extract the target obstacle from the initial obstacle, and determine the position and velocity data of the target obstacle.

[0136] Step 3: Based on the location data of the target obstacles, cluster the target obstacles to obtain multiple clustering results; calculate the minimum outer connection polygon of each clustering result to obtain the convex hull corresponding to each clustering result; the Monoton Chain algorithm can be used to solve for the convex hull.

[0137] Step 4: Using the vector cross product method or ray casting method, determine whether the location information of the millimeter-wave radar data is located inside the convex hull. If the location information of a certain millimeter-wave radar data is located inside the convex hull, it is considered that the initial obstacle indicated by the millimeter-wave radar data is located inside the convex hull, thus obtaining the first detection result. The first detection result corresponds one-to-one with the initial obstacle, indicating whether the initial obstacle is located inside the convex hull. The millimeter-wave radar data also includes the first state of the initial obstacle it indicates (the first state includes moving and stationary). Based on the first detection result, select the initial obstacles located inside the convex hull and whose first state is moving as the first moving targets, and then determine the number of first moving targets in each convex hull.

[0138] Step 5: Input the labeled target image into the pose recognition network (second neural network model) to obtain the second state of the target obstacle; wherein, the second state includes: moving and stationary; according to the second state of the target obstacle, obtain the number of second moving targets in each convex hull; wherein, the second moving targets include: target obstacles in the second state of moving.

[0139] Step 6: Input the position and velocity data of the target obstacle into the trajectory generation network (third neural network model) to obtain the initial predicted trajectory of the target obstacle.

[0140] Step 7: Based on the combined number of the first moving target and the number of the second moving target, the initial predicted trajectory is corrected and compressed to varying degrees.

[0141] This application embodiment also provides a processing module, which is used to perform the following steps:

[0142] Acquire target images of the target area, millimeter-wave radar data of the initial obstacles, and lidar data of the initial obstacles; wherein the initial obstacles are located in the target area;

[0143] The first neural network model is used to process lidar data and target images, extract the target obstacle from the initial obstacle, and determine the position and velocity data of the target obstacle;

[0144] Label the target obstacles in the target image;

[0145] Based on the location data of the target obstacle, generate multiple convex hulls;

[0146] Using millimeter-wave radar data of the initial obstacles, the number of first moving targets in each convex hull is calculated;

[0147] The labeled target image is input into the second neural network model to obtain the number of second moving targets in each convex hull;

[0148] The position and velocity data of the target obstacle are input into the third neural network model to obtain the initial predicted trajectory of the target obstacle;

[0149] Based on the number of the first moving target and the number of the second moving target, the initial predicted trajectory is corrected to obtain the target predicted trajectory of the obstacle.

[0150] This application provides an electronic device, which includes a processor and a memory for storing processor-executable instructions. When the processor executes the instructions, it implements the steps described in any of the obstacle trajectory prediction methods above.

[0151] This application provides a storage medium storing computer instructions, which, when executed by a processor, implement the steps described in any of the obstacle trajectory prediction methods above.

[0152] In the embodiments of this application, the storage medium may be a magnetic disk, an optical disk, a read-only memory (ROM), or a random access memory (RAM), etc.

[0153] In the above embodiments, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions in other embodiments.

[0154] The foregoing has provided a detailed description of an obstacle trajectory prediction method, processing module, and electronic device provided in the embodiments of this application. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the method and core ideas of this application. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this application. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A method for predicting the trajectory of an obstacle, characterized in that, include: Acquire a target image of the target area, millimeter-wave radar data of the initial obstacle, and lidar data of the initial obstacle; wherein the initial obstacle is located in the target area; The first neural network model is used to process the lidar data and the target image to extract the target obstacle from the initial obstacle and determine the position and velocity data of the target obstacle. The target obstacles in the target image are labeled; Based on the position data of the target obstacle, generate multiple convex hulls; Using millimeter-wave radar data of the initial obstacles, the number of first moving targets in each convex hull is calculated; The labeled target image is input into the second neural network model to obtain the number of second moving targets in each convex hull; The position and velocity data of the target obstacle are input into the third neural network model to obtain the initial predicted trajectory of the target obstacle; Based on the number of the first moving targets and the number of the second moving targets, the initial predicted trajectory is corrected to obtain the target predicted trajectory of the target obstacle; Based on the number of the first moving targets and the number of the second moving targets, the initial predicted trajectory is corrected to obtain the target predicted trajectory of the target obstacle, including: When the number of the first moving targets is not equal to 0 and the number of the second moving targets is not equal to 0, the first number of initial predicted trajectories are extracted from the initial predicted trajectories arranged in descending order of length to obtain the first trajectory; the first number is determined based on the number of the first moving targets and the number of the second moving targets. The first trajectory and the third trajectory are combined to obtain the target predicted trajectory, wherein the third trajectory is obtained by trajectory compression of the second trajectory, and the second trajectory is the complement of the initial predicted trajectory and the first trajectory.

2. The method according to claim 1, characterized in that, The millimeter-wave radar data characterizes the first state of the initial obstacle and the location information of the initial obstacle, the first state including motion and stationary.

3. The method according to claim 1, characterized in that, Based on the position data of the target obstacle, multiple convex hulls are generated, including: Based on the location data of the target obstacles, the target obstacles are clustered to obtain multiple clustering results; Calculate the minimum outer connection polygon for each clustering result to obtain the convex hull corresponding to each clustering result.

4. The method according to claim 1, characterized in that, Using millimeter-wave radar data of the initial obstacle, the number of first moving targets in each convex hull is calculated, including: Using millimeter-wave radar data of the initial obstacle, it is determined whether the initial obstacle is located inside the convex hull, thus obtaining a first detection result; Based on the first detection result, the number of first moving targets in each convex hull is calculated; the first moving targets include: initial obstacles in a first state of motion.

5. The method according to claim 1, characterized in that, The labeled target image is input into the second neural network model to obtain the number of second moving targets in each convex hull, including: The labeled target image is input into a second neural network model to obtain a second state of the target obstacle; wherein, the second state includes: motion and stillness; Based on the second state of the target obstacle, the number of second moving targets in each convex hull is obtained; wherein, the second moving targets include: target obstacles in the second state of movement.

6. The method according to claim 1, characterized in that, Based on the number of the first moving targets and the number of the second moving targets, the initial predicted trajectory is corrected to obtain the target predicted trajectory of the target obstacle, including: When the number of the first moving targets is not equal to 0 and the number of the second moving targets is equal to 0, the first second number of initial predicted trajectories are extracted from the initial predicted trajectories arranged in descending order of length to obtain the fourth trajectory; the second number is determined based on the number of the first moving targets. The fourth and sixth trajectories are combined to obtain the target predicted trajectory, wherein the sixth trajectory is obtained by compressing the fifth trajectory, and the fifth trajectory is the complement of the initial predicted trajectory and the fourth trajectory.

7. The method according to claim 1, characterized in that, Based on the number of the first moving targets and the number of the second moving targets, the initial predicted trajectory is corrected to obtain the target predicted trajectory of the target obstacle, including: When the number of the first moving target is equal to 0 and the number of the second moving target is equal to 0, the initial predicted trajectory is compressed to obtain the target predicted trajectory.

8. The method according to claim 1, characterized in that, Based on the number of the first moving targets and the number of the second moving targets, the initial predicted trajectory is corrected to obtain the target predicted trajectory of the target obstacle, including: If the number of the first moving targets is determined to be 0 and the number of the second moving targets is not determined to be 0, the first third number of initial predicted trajectories are extracted from the initial predicted trajectories arranged in descending order of length to obtain the seventh trajectory; the third number is determined based on the number of the second moving targets. The seventh and ninth trajectories are combined to obtain the target predicted trajectory, wherein the ninth trajectory is obtained by compressing the eighth trajectory, and the eighth trajectory is the complement of the initial predicted trajectory and the seventh trajectory.

9. A processing module, characterized in that, The processing module is used to perform the following steps: Acquire a target image of the target area, millimeter-wave radar data of the initial obstacle, and lidar data of the initial obstacle; wherein the initial obstacle is located in the target area; The first neural network model is used to process the lidar data and the target image to extract the target obstacle from the initial obstacle and determine the position and velocity data of the target obstacle. The target obstacles in the target image are labeled; Based on the position data of the target obstacle, generate multiple convex hulls; Using millimeter-wave radar data of the initial obstacles, the number of first moving targets in each convex hull is calculated; The labeled target image is input into the second neural network model to obtain the number of second moving targets in each convex hull; The position and velocity data of the target obstacle are input into the third neural network model to obtain the initial predicted trajectory of the target obstacle; Based on the number of the first moving targets and the number of the second moving targets, the initial predicted trajectory is corrected to obtain the target predicted trajectory of the target obstacle; Based on the number of the first moving targets and the number of the second moving targets, the initial predicted trajectory is corrected to obtain the target predicted trajectory of the target obstacle, including: When the number of the first moving targets is not equal to 0 and the number of the second moving targets is not equal to 0, the first number of initial predicted trajectories are extracted from the initial predicted trajectories arranged in descending order of length to obtain the first trajectory; the first number is determined based on the number of the first moving targets and the number of the second moving targets. The first trajectory and the third trajectory are combined to obtain the target predicted trajectory, wherein the third trajectory is obtained by trajectory compression of the second trajectory, and the second trajectory is the complement of the initial predicted trajectory and the first trajectory.

10. An electronic device, characterized in that, The electronic device includes a processor and a memory for storing processor-executable instructions, wherein the processor executes the instructions to implement the steps of the method according to any one of claims 1 to 8.

11. A storage medium, characterized in that, It stores computer instructions that, when executed by a processor, implement the steps of the method according to any one of claims 1 to 8.

Images