Automatic driving annotation data quality verification method and device, and storage medium

Abstract

The embodiment of the invention provides an automatic driving annotation data quality verification method and device and a storage medium. Comprising the steps that fragment data of vehicle driving and vehicle decision marking data of the fragment data are acquired, and the fragment data are at least used for describing driving conditions of a vehicle and a game obstacle interacting with the vehicle; verifying the self-vehicle decision marking data based on the fragment data, and outputting a verification result; wherein the verification comprises at least one of open-loop verification and closed-loop verification, the open-loop verification is to carry out frame-level verification on the fragment data, and the closed-loop verification is to carry out fragment-level verification on the fragment data. Therefore, the data labeling quality can be improved, and the regulation control data closed-loop efficiency can be improved.

Description

Technical Field

[0001] The present application relates to the field of vehicle technology, and in particular to a method, device, and storage medium for verifying the quality of autonomous driving annotation data. Background Art

[0002] In the field of autonomous driving, deep learning neural networks make autonomous driving systems more human-like by learning decision-making behaviors (such as cutting in / giving way, avoiding obstacles / following) from human driving data and deploying them to decision-making modules. The key to improving the effectiveness of neural network models is to obtain high-quality training data. In the data closed-loop process, data goes through: data collection, data mining, data labeling (including pre-labeling, manual labeling, pre-review, manual review, pre-acceptance, and data acceptance processes), and finally obtains human driving data fragments with decision-making behavior labels. However, there are obvious challenges in the data labeling link:

[0003] (1) It is difficult to label the decision-making behavior of human driving data in complex traffic scenarios. For example, in the scenario of going straight or turning left at an intersection, the human driver does not make the same decision of cutting in or giving way throughout the entire time segment. Instead, the decision-making behavior of the human driver may change during this process, and such changes are difficult to label.

[0004] (2) Relying on manual data quality verification wastes manpower costs and blocks the improvement of data closed-loop efficiency. For example, in a data closed-loop of 1 million frames, the test found that the time consumption of data labeling accounts for 30-50% of the total, which is a huge waste of time and manpower costs.

[0005] In summary, it is necessary to provide a new method for verifying the quality of labeled data. Summary of the invention

[0006] The present application discloses a method, device, and storage medium for verifying the quality of autonomous driving annotation data, which can improve the quality of data annotation and enhance the efficiency of the closed loop of regulatory control data.

[0007] In a first aspect, an embodiment of the present application provides a method for verifying the quality of autonomous driving annotation data, including:

[0008] Acquire segment data of the ego vehicle's travel and ego vehicle decision annotation data of the segment data, wherein the segment data is used to at least describe the travel conditions of the ego vehicle and the game obstacles interacting with the ego vehicle;

[0009] Verifying the self-vehicle decision-making annotation data based on the segment data, and outputting a verification result;

[0010] The verification includes at least one of an open-loop verification and a closed-loop verification, the open-loop verification is a frame-level verification of the fragment data, and the closed-loop verification is a fragment-level verification of the fragment data.

[0011] It can be understood that the frame-level verification is to check each frame of the sampled multi-frame data (i.e., the fragment data below) in turn, and confirm whether the frame data satisfies the kinematic constraints based on the ego-vehicle decision annotation data corresponding to each frame data, that is, whether the frame data can generate the corresponding trajectory. If yes, it means that the ego-vehicle decision annotation data corresponding to the frame data is accurate.

[0012] The segment-level verification is: for the sampled segment data, by comparing the difference between the simulation trajectory corresponding to the segment data and the posterior trajectory corresponding to the segment data, the accuracy of the vehicle decision labeling data is determined. The simulation trajectory corresponding to the segment data is obtained based on the simulation trajectory of each frame data. Among them, the simulation trajectory corresponding to the current frame data is obtained by controlling the vehicle based on the simulation trajectory of the previous frame data.

[0013] In an embodiment of the present application, segment data of the vehicle's travel and the vehicle's decision-making annotation data of the segment data are obtained, and then the vehicle's decision-making annotation data is verified based on the segment data, and a verification result is output. The verification can be an open-loop verification that performs frame-level verification on the segment data, or a closed-loop verification that performs segment-level verification on the segment data. The verification can be a combination of open-loop verification and closed-loop verification. By adopting this method, the annotation quality of training data decision labels in various traffic scenarios can be improved, and automated data quality inspection can effectively reduce the labor cost of annotation review and effectively improve the efficiency of the closed loop of regulatory control data.

[0014] This solution can be used to verify the results of manual and automatic annotation in the data closed-loop process. Based on the above verification, the manpower of manual review can be saved. For example, if the verification result shows that there is no problem with the result of automatic annotation, the subsequent manual scene annotation can be performed directly, skipping the review and acceptance. If the verification result shows that the annotation result is incorrect but there is no problem with the basic data (speed, acceleration, etc., etc., and other error messages other than the large mean square error obtained by the verification, such as Wrong IG and Wrong YD), the original process is followed for review and acceptance. Among them, the review and acceptance process can also continue to use the above verification to verify the modified results. For discardable data that has not passed the basic data cleaning, there is no need for subsequent manual annotation. Furthermore, by eliminating unreasonable data such as radical / conservative driving and drastic changes in intention, the efficiency of the data closed loop is improved. On the other hand, the quality of the returned data can also be improved, and the cost of data storage and transmission can be reduced.

[0015] In a possible implementation, the open-loop verification includes: determining whether N frames of data sampled from the segment data all satisfy kinematic constraint information based on the ego-vehicle decision annotation data, where N is a positive integer.

[0016] In this example, the open-loop check is to not control the vehicle, and solve the manual annotation results and kinematic constraints of the current data frame to determine whether the label of the current frame is correct. This can improve the annotation quality of the training data decision labels and reduce the labor and time costs of annotation review.

[0017] In a possible implementation, the closed-loop verification includes: detecting whether the difference between the simulated trajectory of the ego vehicle and the posterior trajectory of the ego vehicle is less than a preset threshold, the simulated trajectory of the ego vehicle is predicted by a multi-vehicle joint model based on the input ego vehicle decision annotation data and the fragment data, and the posterior trajectory of the ego vehicle is included in the fragment data.

[0018] In this example, the closed-loop verification is a segment-level simulation of the vehicle control. It mainly compares the difference between the simulation and the posterior trajectory to determine the overall accuracy of the annotations within the segment. This can improve the annotation quality of the training data decision labels and reduce the labor and time costs of annotation review.

[0019] In a possible implementation, the segment data includes driving parameters of the vehicle, driving parameters of the game obstacle, and road topology information.

[0020] In a possible implementation, determining whether N frames of data sampled from the segment data all satisfy kinematic constraint information based on the ego-vehicle decision annotation data includes:

[0021] Solving based on the kinematic constraint information, the ego-vehicle decision labeling data, the segment data, and the quadratic programming QP model to determine whether N predicted trajectories of the ego-vehicle can be generated, the N predicted trajectories corresponding to the N frames of data;

[0022] If N predicted trajectories can be generated, it is confirmed that the N frames of data all satisfy the kinematic constraint information.

[0023] This verification is based on the frame-level verification of the segment data based on the self-vehicle decision-making labeling data to confirm whether each frame of data satisfies the kinematic constraint information. This method can improve the labeling quality of training data decision labels in various traffic scenarios, and automated data quality inspection can effectively reduce the labor cost of labeling review and effectively improve the efficiency of the closed loop of regulatory control data.

[0024] In a possible implementation, the fragment data also includes the posterior trajectory of the game obstacle, and the solving based on the kinematic constraint information, the ego-vehicle decision labeling data, the fragment data and the quadratic programming QP model includes:

[0025] When it is determined based on the first distance-time ST graph of the game obstacle that there is a preset space between the ego vehicle and the game obstacle, solving is performed based on the kinematic constraint information, the ego vehicle decision annotation data, the fragment data, and the quadratic programming QP model;

[0026] The first ST diagram is established based on the posterior trajectory of the game obstacle and the self-vehicle decision labeling data.

[0027] In a possible implementation, the segment data further includes the posterior trajectory of the game obstacle, and the process of obtaining the simulated trajectory of the vehicle specifically includes:

[0028] Wherein, for the i-th frame data in N frames of data sampled from the fragment data, i is an integer greater than 1 and less than or equal to N:

[0029] The posterior trajectory of the game obstacle in the segment data, the driving parameters of the game obstacle, the road topology information, the posterior trajectory of the ego vehicle and the ego vehicle decision labeling data are input into the multi-vehicle joint model to obtain the simulation trajectory of each game obstacle in the i-th frame data;

[0030] Inputting the i-1th initial simulation trajectory of the self-vehicle, the simulation trajectory of each game obstacle in the i-th frame data, and the fragment data into the multi-vehicle joint model to obtain the trajectory result of the self-vehicle corresponding to the i-th frame data;

[0031] The i-1th initial simulation trajectory of the ego vehicle is obtained based on the i-2th initial simulation trajectory of the ego vehicle and the trajectory result of the ego vehicle corresponding to the i-1th frame data, wherein when i is equal to 2, the 0th initial simulation trajectory is the posterior trajectory of the ego vehicle;

[0032] The Nth initial simulation trajectory of the vehicle is obtained based on the N-1th initial simulation trajectory of the vehicle and the trajectory result of the vehicle corresponding to the Nth frame data;

[0033] The simulated trajectory of the vehicle includes the trajectory result of the vehicle corresponding to the Nth frame data and the trajectory result of the vehicle corresponding to the i-th frame data.

[0034] This verification is based on the self-vehicle decision-making labeling data to verify the segment data at the segment level, confirm the difference between the simulation trajectory and the posterior trajectory, and then confirm the accuracy of the labeling data of the entire segment data. This method can improve the labeling quality of training data decision labels in various traffic scenarios, and automated data quality inspection can effectively reduce the labor cost of labeling review and effectively improve the efficiency of the closed loop of regulatory control data.

[0035] In a possible implementation, the step of inputting the posterior trajectory of the game obstacle in the segment data, the driving parameters of the game obstacle, the road topology information, the posterior trajectory of the ego vehicle, and the ego vehicle decision labeling data into the multi-vehicle joint model to obtain the simulation trajectory of each game obstacle in the i-th frame data includes:

[0036] Based on the posterior trajectory of the game obstacle, the posterior trajectory of the ego vehicle and the ego vehicle decision labeling data, a second distance-time ST graph corresponding to the ego vehicle and the game obstacle in the i-th frame data is established;

[0037] Obtaining at least one collision point between the game obstacle and the vehicle in the i-th frame data based on the second ST graph;

[0038] Based on the time at which the self-vehicle and the game obstacle respectively arrive at the corresponding collision point in the i-th frame data, the game obstacles are sorted to obtain a sorting result of the game obstacles corresponding to the i-th frame data;

[0039] The sorting result of the game obstacles corresponding to the i-th frame data, the posterior trajectory of the game obstacles in the fragment data, the driving parameters of the game obstacles, the road topology information, the posterior trajectory of the ego vehicle and the ego vehicle decision annotation data are all input into the multi-vehicle joint model to obtain the simulation trajectory of each game obstacle in the i-th frame data, wherein the simulation trajectory of the k-th game obstacle in the i-th frame data is related to the simulation trajectory of the obstacle before the k-th game obstacle in the sorting result, and k is an integer greater than 1.

[0040] In a possible implementation, the sorting result of the game obstacles corresponding to the i-th frame data is displayed.

[0041] After obtaining the ranking results of the above-mentioned game obstacles, prompts such as display and playback can also be performed. For example, when applied to the human-machine interaction HMI visualization interface, it can effectively remind the driver of the obstacles that need the most attention / concern.

[0042] In a possible implementation, the segment data further includes a posterior trajectory of a non-gaming obstacle, and the method further includes:

[0043] Performing interpolation processing on the posterior trajectory of the non-game obstacle to obtain a simulated trajectory of the non-game obstacle;

[0044] The step of inputting the i-1th initial simulation trajectory of the self-vehicle, the simulation trajectory of each game obstacle in the i-th frame data, and the fragment data into the multi-vehicle joint model to obtain the trajectory result of the self-vehicle corresponding to the i-th frame data includes:

[0045] The simulation trajectory of the non-game obstacle, the i-1th initial simulation trajectory of the ego vehicle, the simulation trajectory of each game obstacle in the i-th frame data, and the fragment data are all input into the multi-vehicle joint model to obtain the trajectory result of the ego vehicle corresponding to the i-th frame data.

[0046] In a second aspect, an embodiment of the present application provides a method for displaying gaming obstacles, including:

[0047] Acquire segment data of the ego vehicle's travel and ego vehicle decision labeling data of the segment data, wherein the segment data is used to at least describe the travel conditions of the ego vehicle and a game obstacle interacting with the ego vehicle, and the segment data includes a posterior trajectory of the game obstacle and a posterior trajectory of the ego vehicle;

[0048] Based on the segment data and the ego vehicle decision labeling data, a ranking result of the game obstacles corresponding to the segment data is obtained; the ranking result is determined based on the time sequence of the ego vehicle and the game obstacles arriving at the corresponding collision points respectively;

[0049] Control and display the sorting results of the game obstacles corresponding to the fragment data.

[0050] In the embodiment of the present application, based on processing the segment data of the vehicle's travel and the vehicle's decision-making annotation data of the segment data, the ranking results of the game obstacles corresponding to the segment data are obtained and displayed, which can effectively remind the driver of the obstacles that need the most attention / concern.

[0051] In one possible implementation, the collision point is determined based on a distance-time ST diagram corresponding to the ego vehicle and the game obstacle, and the ST diagram is established based on the posterior trajectory of the game obstacle, the posterior trajectory of the ego vehicle, and the ego vehicle decision annotation data.

[0052] In a third aspect, an embodiment of the present application provides a device for verifying the quality of autonomous driving annotation data, including:

[0053] An acquisition module, used to acquire segment data of the ego vehicle and ego vehicle decision annotation data of the segment data, wherein the segment data is used to at least describe the driving conditions of the ego vehicle and the gaming obstacles interacting with the ego vehicle;

[0054] A verification module, used to verify the self-vehicle decision-making annotation data based on the segment data and output a verification result;

[0055] The verification includes at least one of an open-loop verification and a closed-loop verification, the open-loop verification is a frame-level verification of the fragment data, and the closed-loop verification is a fragment-level verification of the fragment data.

[0056] In a possible implementation, the open-loop verification includes: determining whether N frames of data sampled from the segment data all satisfy kinematic constraint information based on the ego-vehicle decision annotation data, where N is a positive integer.

[0057] In a possible implementation, the closed-loop verification includes: detecting whether the difference between the simulated trajectory of the ego vehicle and the posterior trajectory of the ego vehicle is less than a preset threshold, the simulated trajectory of the ego vehicle is predicted by a multi-vehicle joint model based on the input ego vehicle decision annotation data and the fragment data, and the posterior trajectory of the ego vehicle is included in the fragment data.

[0058] In a possible implementation, the segment data includes driving parameters of the vehicle, driving parameters of the game obstacle, and road topology information.

[0059] In a possible implementation, the verification module is used to:

[0060] Solving based on the kinematic constraint information, the ego-vehicle decision labeling data, the segment data, and the quadratic programming QP model to determine whether N predicted trajectories of the ego-vehicle can be generated, the N predicted trajectories corresponding to the N frames of data;

[0061] If N predicted trajectories can be generated, it is confirmed that the N frames of data all satisfy the kinematic constraint information.

[0062] In a possible implementation, the segment data further includes a posterior trajectory of the gaming obstacle, and the verification module is used to:

[0063] When it is determined based on the first distance-time ST graph of the game obstacle that there is a preset space between the ego vehicle and the game obstacle, solving is performed based on the kinematic constraint information, the ego vehicle decision annotation data, the fragment data, and the quadratic programming QP model;

[0064] The first ST diagram is established based on the posterior trajectory of the game obstacle and the self-vehicle decision labeling data.

[0065] In a possible implementation, the segment data further includes a posterior trajectory of the gaming obstacle, and the verification module is used to:

[0066] Wherein, for the i-th frame data in N frames of data sampled from the fragment data, i is an integer greater than 1 and less than or equal to N:

[0067] The posterior trajectory of the game obstacle in the segment data, the driving parameters of the game obstacle, the road topology information, the posterior trajectory of the ego vehicle and the ego vehicle decision labeling data are input into the multi-vehicle joint model to obtain the simulation trajectory of each game obstacle in the i-th frame data;

[0068] Inputting the i-1th initial simulation trajectory of the self-vehicle, the simulation trajectory of each game obstacle in the i-th frame data, and the fragment data into the multi-vehicle joint model to obtain the trajectory result of the self-vehicle corresponding to the i-th frame data;

[0069] The i-1th initial simulation trajectory of the ego vehicle is obtained based on the i-2th initial simulation trajectory of the ego vehicle and the trajectory result of the ego vehicle corresponding to the i-1th frame data, wherein when i is equal to 2, the 0th initial simulation trajectory is the posterior trajectory of the ego vehicle;

[0070] The Nth initial simulation trajectory of the vehicle is obtained based on the N-1th initial simulation trajectory of the vehicle and the trajectory result of the vehicle corresponding to the Nth frame data;

[0071] The simulated trajectory of the vehicle includes the trajectory result of the vehicle corresponding to the Nth frame data and the trajectory result of the vehicle corresponding to the i-th frame data.

[0072] In a possible implementation, the verification module is used to:

[0073] Based on the posterior trajectory of the game obstacle, the posterior trajectory of the ego vehicle and the ego vehicle decision labeling data, a second distance-time ST graph corresponding to the ego vehicle and the game obstacle in the i-th frame data is established;

[0074] Obtaining at least one collision point between the game obstacle and the vehicle in the i-th frame data based on the second ST graph;

[0075] Based on the time at which the self-vehicle and the game obstacle respectively arrive at the corresponding collision point in the i-th frame data, the game obstacles are sorted to obtain a sorting result of the game obstacles corresponding to the i-th frame data;

[0076] The sorting result of the game obstacles corresponding to the i-th frame data, the posterior trajectory of the game obstacles in the fragment data, the driving parameters of the game obstacles, the road topology information, the posterior trajectory of the ego vehicle and the ego vehicle decision annotation data are all input into the multi-vehicle joint model to obtain the simulation trajectory of each game obstacle in the i-th frame data, wherein the simulation trajectory of the k-th game obstacle in the i-th frame data is related to the simulation trajectory of the obstacle before the k-th game obstacle in the sorting result, and k is an integer greater than 1.

[0077] In a possible implementation, the device further includes a display module, which is used to:

[0078] Display the sorting results of the game obstacles corresponding to the i-th frame data.

[0079] In a possible implementation, the segment data further includes a posterior trajectory of a non-gaming obstacle, and the verification module is further configured to:

[0080] Performing interpolation processing on the posterior trajectory of the non-game obstacle to obtain a simulated trajectory of the non-game obstacle;

[0081] The simulation trajectory of the non-game obstacle, the i-1th initial simulation trajectory of the ego vehicle, the simulation trajectory of each game obstacle in the i-th frame data, and the fragment data are all input into the multi-vehicle joint model to obtain the trajectory result of the ego vehicle corresponding to the i-th frame data.

[0082] In a fourth aspect, an embodiment of the present application provides a display device for gaming obstacles, including:

[0083] an acquisition module, configured to acquire segment data of the ego vehicle's travel and ego vehicle decision labeling data of the segment data, wherein the segment data is used to at least describe the travel conditions of the ego vehicle and a game obstacle interacting with the ego vehicle, and the segment data includes a posterior trajectory of the game obstacle and a posterior trajectory of the ego vehicle;

[0084] A processing module, configured to obtain, based on the segment data and the ego vehicle decision labeling data, a ranking result of the game obstacles corresponding to the segment data; the ranking result is determined based on a time sequence in which the ego vehicle and the game obstacles respectively arrive at corresponding collision points;

[0085] A display module is used to control and display the sorting results of the game obstacles corresponding to the fragment data.

[0086] In one possible implementation, the collision point is determined based on a distance-time ST diagram corresponding to the ego vehicle and the game obstacle, and the ST diagram is established based on the posterior trajectory of the game obstacle, the posterior trajectory of the ego vehicle, and the ego vehicle decision annotation data.

[0087] In a fifth aspect, an embodiment of the present application provides an autonomous driving labeled data quality verification device, comprising a processor and a memory; wherein the memory is used to store program code, and the processor is used to call the program code to execute a method provided in any possible implementation manner of the first aspect.

[0088] In a sixth aspect, an embodiment of the present application provides a display device for gaming obstacles, comprising a processor and a memory; wherein the memory is used to store program code, and the processor is used to call the program code to execute a method provided in any possible implementation manner of the second aspect.

[0089] In the seventh aspect, an embodiment of the present application provides a computer-readable storage medium, which stores a computer program, and the computer program is executed by a processor to implement a method provided in any possible implementation of the first aspect and / or any possible implementation of the second aspect.

[0090] In an eighth aspect, an embodiment of the present application provides a computer program product. When the computer program product runs on a computer, the computer executes a method provided in any possible implementation of the first aspect and / or any possible implementation of the second aspect.

[0091] In a ninth aspect, an embodiment of the present application provides a vehicle, comprising a device provided by any possible implementation of the third aspect and / or any possible implementation of the fourth aspect.

[0092] It can be understood that the device described in the third aspect, the device described in the fourth aspect, the device described in the fifth aspect, the device described in the sixth aspect, the computer storage medium described in the seventh aspect or the computer program product described in the eighth aspect, and the vehicle described in the ninth aspect are all used to execute any method provided in the first aspect and any method provided in the second aspect. Therefore, the beneficial effects that can be achieved can refer to the beneficial effects in the corresponding methods, which will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0093] The following is an introduction to the drawings used in the embodiments of the present application.

[0094] Figure 1 It is a schematic diagram of the architecture of an autonomous driving annotation data quality verification system provided by an embodiment of the present application;

[0095] Figure 2a It is a flowchart of a method for verifying the quality of autonomous driving annotation data provided in an embodiment of the present application;

[0096] Figure 2b This is a schematic diagram of quality verification of autonomous driving annotation data provided by an embodiment of the present application;

[0097] Figure 3 is a schematic diagram of a method for verifying the quality of autonomous driving annotation data provided in an embodiment of the present application;

[0098] Figure 4 is a schematic diagram of another method for verifying the quality of autonomous driving annotation data provided in an embodiment of the present application;

[0099] Figure 5 It is a schematic diagram of data processing provided by an embodiment of the present application;

[0100] Figure 6 It is a flowchart of a method for displaying gaming obstacles provided in an embodiment of the present application;

[0101] Figure 7 It is a structural schematic diagram of an autonomous driving annotation data quality verification device provided in an embodiment of the present application;

[0102] Figure 8 It is a structural schematic diagram of a display device for gaming obstacles provided in an embodiment of the present application;

[0103] Fig. 9 It is a structural schematic diagram of another autonomous driving annotation data quality verification device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0104] The following describes the embodiments of the present application in conjunction with the drawings in the embodiments of the present application. The terms used in the implementation method part of the embodiments of the present application are only used to explain the specific embodiments of the present application, and are not intended to limit the present application.

[0105] To facilitate understanding, the following first introduces relevant terms that may be involved in the embodiments of the present application.

[0106] (1) Autonomous driving

[0107] Autonomous driving, also known as intelligent driving or assisted driving, is an important direction for the development of vehicle intelligence. With the development of perception technology and the improvement of chip capabilities, intelligent driving provides people with more and more rich driving functions and gradually realizes different levels of driving experience. The Society of Automotive Engineers (SAE) provides a driving automation classification standard, including driving levels L0 to L5, where L0 is no automation and the human driver has full control over the vehicle. During driving, the driver can get warnings or assistance from the driving system, such as autonomous emergency braking (AEB), blind spot monitoring (BSM) or lane departure warning (LDW). Level 1 is driving assistance, and the driving operation is completed by the human driver and the driving system. The driving system can provide driving assistance for the steering wheel or acceleration and deceleration operations through the driving environment, and other driving operations are performed by the human driver, such as adaptive cruise control (ACC) or lane keep assistance / support (LKA / LKS). Level 2 is partial automation, which provides driving assistance for the steering wheel and multiple acceleration and deceleration operations through the driving environment, and other driving actions are performed by the human driver, such as a combination of adaptive cruise control (ACC) and lane keep assistance (LKA / LKS). Level 3 is conditional automation, where the driving system can complete all driving operations, but the human driver needs to respond to the request of the driving system at the appropriate time, that is, the human driver needs to be prepared to take over the driving system; Level 4 is highly automated, where the driving system can complete all driving operations, and the human driver does not necessarily need to respond to the request of the driving system. For example, when the road and environmental conditions permit (such as closed parks, highways, urban roads or fixed driving routes, etc.), the human driver may not take over the driving; Level 5 is fully automated, where the driving system can autonomously complete driving operations under various road and environmental conditions that human drivers can cope with. It can be seen that at levels L0 to L2, the driving system mainly provides support for the driver, and the driver still needs to supervise the driving and steer, brake or accelerate as needed to ensure safety. At levels L3 to L5, the driving system can complete all driving operations on behalf of the driver. At level L3, the driver must be prepared to take over the driving. At levels L4 and L5, the driving system can achieve full driving under some conditions and all conditions, and the driver can choose whether to take over.

[0108] The above classification is an example. With the evolution of technology or different regulations in different countries or regions, the above classification may change. For example, the vehicle automation classification proposed by the Ministry of Industry and Information Technology of China includes 6 levels of vehicle driving automation, among which Levels 0-2 are driving assistance, the system assists humans in performing dynamic driving tasks, and the driving subject is still the driver; Levels 3-5 are automatic driving, the system replaces humans in performing dynamic driving tasks under the designed operating conditions, and when the function is activated, the driving subject is the system. The names and definitions of each level are as follows: Level 0 driving automation (emergency assistance) The system cannot continuously perform the lateral or longitudinal motion control of the vehicle in dynamic driving tasks, but has the ability to continuously perform partial target and event detection and response in dynamic driving tasks. Level 1 driving automation (partial driver assistance) The system continuously performs the lateral or longitudinal motion control of the vehicle in dynamic driving tasks under its designed operating conditions (or called the design operating range ODD), and has the ability to detect and respond to partial targets and events that are compatible with the lateral or longitudinal motion control of the vehicle being performed. Level 2 driving automation (combined driver assistance) systems continuously perform lateral and longitudinal motion control of the vehicle in dynamic driving tasks under their designed operating conditions, and have the ability to detect and respond to some targets and events that are compatible with the lateral and longitudinal motion control of the vehicle being performed. Level 3 driving automation (conditionally automated driving) systems continuously perform all dynamic driving tasks under their designed operating conditions. Level 4 driving automation (highly automated driving) systems continuously perform all dynamic driving tasks under their designed operating conditions and automatically implement minimum risk strategies. Level 5 driving automation (fully automated driving) systems continuously perform all dynamic driving tasks under any drivable conditions and automatically implement minimum risk strategies. Among them, lateral control is mainly used for vehicle steering control, for example, controlling the steering wheel torque or angle to control the direction of the vehicle; longitudinal control is mainly used for vehicle speed control, for example, controlling the brake pedal, accelerator pedal, or gear position to control the acceleration / deceleration, braking, etc. of the vehicle.

[0109] Regardless of the classification method adopted, the description of the embodiments of the present application can be applied to the above-mentioned automatic driving systems that require partial or full participation in vehicle driving.

[0110] (2) Obstacles

[0111] In the embodiments of the present application, an obstacle refers to an entity that the vehicle does not expect to collide with during driving, and the entity can be static or dynamic. Among them, static entities can be, for example, cartons on the road, road construction signs, road dividing railings, dirt piles, tires, overturned vehicles, lying people, animals, buildings beside the road, trees, parked vehicles, road signs, electric poles, roadside isolation strips and other stationary objects with volume and mass; dynamic entities can be, for example, pedestrians (such as walking pedestrians, pedestrians riding bicycles, etc.), animals, vehicles, vehicles carrying goods (such as loaded with cartons, branches or other goods) and other moving objects with volume and mass.

[0112] (3) Gaming Obstacles

[0113] It can be understood that the game obstacles are obstacles that the vehicle may overtake, yield, avoid, accelerate, etc.

[0114] (4) Non-gaming obstacles

[0115] It can be understood that non-game obstacles are obstacles that the vehicle has no intention of giving way to, etc.

[0116] (5) Frame-level verification

[0117] Each frame of the sampled multi-frame data (hereinafter referred to as the fragment data) is checked in turn, and the ego-vehicle decision annotation data corresponding to each frame of data is used to confirm whether the frame of data satisfies the kinematic constraints, that is, whether the frame of data can generate the corresponding trajectory. If yes, it means that the ego-vehicle decision annotation data corresponding to the frame of data is accurate.

[0118] (6) Fragment-level verification

[0119] For the sampled fragment data, the difference between the simulation trajectory corresponding to the fragment data and the posterior trajectory corresponding to the fragment data is compared to determine whether the self-vehicle decision labeling data is accurate. The simulation trajectory corresponding to the fragment data is obtained based on the simulation trajectory of each frame data. Among them, the simulation trajectory corresponding to the current frame data is obtained by controlling the vehicle based on the simulation trajectory of the previous frame data.

[0120] The above exemplary description of the concepts can be applied in the following embodiments.

[0121] The following will describe the system architecture of the embodiment of the present application in detail with reference to the accompanying drawings. Figure 1 , Figure 1 Schematic diagram of an autonomous driving annotation data quality verification system applicable to the embodiment of the present application, the system includes a vehicle and a network side device. Among them:

[0122] Here, the vehicle refers to a vehicle equipped with an automatic driving system. The automatic driving system is not limited to a fully automatic driving system, a highly automatic driving system, a conditional automatic driving system, or a partially automatic driving system. Those skilled in the art can understand that any non-fully manual driving system that provides intelligent driving can be covered under this concept.

[0123] Exemplarily, depending on the power source of the vehicle, the vehicle may be a new energy vehicle or a traditional vehicle, etc., wherein a traditional vehicle refers to a fuel vehicle, such as a gasoline vehicle, a diesel vehicle, etc., and a new energy vehicle may be, for example, an electric vehicle (EV), a hybrid electric vehicle (HEV), an extended-range electric vehicle (range extended EV), a plug-in hybrid vehicle (Plug-in HEV), a fuel cell vehicle or other new energy vehicles, without specific limitation herein.

[0124] Among them, cameras and radars are deployed on the vehicle, wherein the camera is used to collect image data of the vehicle's current surroundings, and the radar is used to collect point cloud data of the vehicle's current surroundings. The radar includes at least one of a laser radar Lidar, a millimeter-wave radar Radar, and the like. Based on the installation position of the camera on the vehicle, the camera can be divided into, for example, a front-view camera, a surround-view camera, a rear-view camera, and a side-view camera, and the like; based on the structural division of the camera, the camera can be divided into, for example, a monocular camera, a binocular camera, a wide-angle camera, and the like. Here, the embodiment of the present application does not limit the number of cameras configured for the vehicle. For safety reasons, the camera on the vehicle needs to be able to collect 360-degree image data around the vehicle body.

[0125] Here, the network side device is a device with computing capabilities. The network side device may be, for example, a server deployed on the network side (e.g., a server for intelligent driving processing), or a component or chip in the server. In some possible embodiments, the network side device may also be a system-level device or a computing device cluster composed of multiple servers. The network side device may be deployed in a cloud environment or an edge environment, and the embodiments of the present application are not specifically limited.

[0126] Above Figure 1 The figure shows an implementation method of an embodiment of the present application. In this implementation method, the autonomous driving labeling data quality verification method provided by the embodiment of the present application can be implemented on a network side device.

[0127] In another possible implementation, the embodiment of the present application can also be used in vehicles, for example, it can be an onboard device (onboard unit, OBU) in the vehicle with the function of verifying the quality of the autonomous driving annotation data, or it can be set in a sensor with the function of collecting and processing the quality verification of the autonomous driving annotation data. Among them, the onboard device can include but is not limited to a vehicle terminal, a vehicle controller, a vehicle module, a vehicle module, a vehicle component, a vehicle chip, a vehicle unit, a vehicle radar, an electronic control unit (electronic control unit, ECU), a domain controller (domain controller, DC) and other devices.

[0128] The embodiments of the present application can be applied to the field of autonomous driving vehicles, as well as to other fields such as smart homes, smart logistics, drones, etc., for example, involving judgment scenarios that consider the results of interactions between robots and targets.

[0129] The architecture of the embodiment of the present application is described above, and the method of the embodiment of the present application is described in detail below.

[0130] Reference Figure 2a As shown in FIG. 1 , it is a flow chart of a method for verifying the quality of autonomous driving annotation data provided by an embodiment of the present application. Optionally, the method can be applied to the aforementioned autonomous driving annotation data quality verification system. Figure 2a The autonomous driving annotation data quality verification method shown may include steps 201-202. It should be understood that for the convenience of description, this application is described in the order of 201-202, and is not intended to limit the execution to the above order. The embodiment of the present application does not limit the order of execution, execution time, number of executions, etc. of the above one or more steps. The following description takes the execution subject of steps 201-202 of the autonomous driving annotation data quality verification method as the vehicle-side chip as an example. This application is also applicable to other execution subjects such as servers. Steps 201-202 are as follows:

[0131] 201. Obtain segment data of the ego vehicle's travel and ego vehicle decision annotation data of the segment data, wherein the segment data is at least used to describe the travel conditions of the ego vehicle and a game obstacle interacting with the ego vehicle.

[0132] This fragment data, understandably, contains information about the game between the vehicle and the obstacle.

[0133] In a possible implementation, the segment data includes the driving parameters of the vehicle, the driving parameters of the game obstacle, and the road topology information. For example, the driving parameters include speed, acceleration, comfort, etc. The road topology information may include, for example, the road centerline, sideline, traffic light, speed limit sign, etc., and their corresponding attribute information. Among them, the attribute information of the centerline and sideline includes whether it is a dotted line, a solid line, and its length; the attribute information of the speed limit sign includes the speed limit value; the attribute information of the traffic light includes the color of the current traffic light and the waiting time, etc.

[0134] The above-mentioned ego vehicle decision labeling data indicates the predicted decision behavior of the ego vehicle towards the obstacle. The ego vehicle decision labeling data may be, for example, grabway (GW; the ego vehicle accelerates or does not change speed to pass the possible collision point with the obstacle first), yield (YD; the ego vehicle decelerates or does not change speed to let the obstacle pass the possible collision point first), ignore (IG; i.e., the longitudinal action is not changed due to the target vehicle), unknown (UN; i.e., the result of the grab and yield cannot be determined), etc.

[0135] The above-mentioned self-driving car decision-making labeling data can be manually labeled or automatically labeled by a machine, and this solution does not impose any restrictions on this.

[0136] Optionally, the above-mentioned segment data of the vehicle driving can be obtained after data cleaning processing, such as Figure 2b As shown. Among them, the verified data segment may not be human driving data in the actual process, or the entire segment trajectory may be short or even completely stagnant or congested, or there may be aggressive behaviors such as excessive acceleration and excessive speed during human driving, all of which will affect the results of data training or make it unusable. Therefore, data cleaning can be used for screening to improve the efficiency of screening. For example, basic data problems such as short segment length, frame loss, and database data loss can be initially screened.

[0137] 202. Verify the self-vehicle decision-making labeling data based on the segment data, and output a verification result.

[0138] Wherein, the verification includes at least one of an open-loop verification and a closed-loop verification. Figure 2b As shown, it is a schematic diagram of the quality verification of the autonomous driving annotation data provided by the embodiment of the present application. This example is illustrated by first performing an open-loop verification and then a closed-loop verification. Of course, it is also possible to perform a closed-loop verification first and then an open-loop verification. Alternatively, the open-loop verification and the closed-loop verification can also be performed in parallel, etc., and this solution does not limit this.

[0139] The open-loop calibration and closed-loop calibration are introduced below.

[0140] 1. The open-loop check is to perform frame-level check on the segment data.

[0141] In a possible implementation, the open-loop verification includes: determining, based on the ego-vehicle decision-labeled data, whether N frames of data sampled from the segment data all satisfy kinematic constraint information.

[0142] Exemplarily, the above segment data may be sampled to obtain N frames of data, for example, uniform sampling at a certain frequency (such as sampling once every 10 seconds), or random sampling.

[0143] Based on the ego-vehicle decision-making annotation data and N frames of data, calculate whether each frame of data satisfies the kinematic constraint information. If the N frames of data all satisfy the kinematic constraint information, it indicates that the verification result is successful, that is, the verification of the ego-vehicle decision-making annotation data is successful. If there is at least one frame of data that does not satisfy the kinematic constraint information, it indicates that the verification result is failed, that is, the verification of the ego-vehicle decision-making annotation data fails.

[0144] Exemplarily, the output verification result may include a total result and a result corresponding to each frame.

[0145] In a possible implementation, a solution is performed based on the kinematic constraint information, the ego vehicle decision labeling data, the segment data, and a quadratic programming (QP) model to determine whether N predicted trajectories of the ego vehicle can be generated, the N predicted trajectories corresponding to the N frames of data. If N predicted trajectories can be generated, it is determined that the N frames of data all satisfy the kinematic constraint information.

[0146] Among them, the quadratic programming QP model f1 can be expressed as:

[0147]

[0148] f1 satisfies the following conditions:

[0149] (1 (2) and, (3)g(x,u)≤0.

[0150] Where si represents the position of the vehicle in the i-th frame data; ai represents the acceleration of the vehicle in the i-th frame data; j i represents the comfort level of the ego vehicle in the i-th frame data; slack represents other constraint information of the ego vehicle and / or obstacles; s r is the preset distance; a r is the preset acceleration; j r is the preset comfort level; s 、wa 、w j and w k are the preset weights respectively.

[0151] The above condition (1) is the state equation, A is the state transfer matrix, B is the input matrix, x i represents the state of the system at time i, u i represents the input at time i. The above condition (2) is a hard constraint on the state and control quantities. represents the lower limit of the hard constraints on the state and control quantities, represents the upper limit of the hard constraints on the state and control quantities, represents the lower limit of the state at time i, represents the upper limit of the state at time i. The above condition (3) is an inequality constraint of the soft boundary and the slack variable.

[0152] The ego-vehicle decision annotation data, the segment data, and the kinematic constraint information are input into the QP model f1 for processing to confirm whether N solutions can be generated. If so, it indicates that the ego-vehicle decision annotation data verification is successful.

[0153] In a possible implementation, before solving the QP model, it also includes confirming a first distance-time (space-time, ST) diagram of the game obstacle to determine whether there is a preset space between the vehicle and the game obstacle.

[0154] Among them, the fragment data also includes the posterior trajectory of the game obstacle. Based on the posterior trajectory of the game obstacle and the decision-making annotation data of the ego vehicle, a first distance-time ST diagram of the game obstacle is established, and then based on the first ST diagram, it is confirmed whether there is a preset space between the ego vehicle and the game obstacle.

[0155] The ST diagram is a coordinate system with time as the horizontal axis and the distance of the planned path as the vertical axis; it can project the speed plan onto a two-dimensional plane and solve it in an optimized manner. Exemplarily, by projecting the posterior trajectory of the game obstacle onto the ST diagram, based on the labels such as grabbing and giving way corresponding to the decision-making annotation data of the ego vehicle, it can be determined to adopt the corresponding trajectory in the ST diagram, that is, to obtain the first ST diagram. Based on the first ST diagram, it can be seen that if the distance between the ego vehicle and the obstacle is small, it is confirmed that there is no room for passage; if the distance between the ego vehicle and the obstacle is large, it is confirmed that there is room for passage. Figure 3 As shown, based on the constraints generated by the ST diagram and the QP model solution, the verification result is obtained.

[0156] Based on this example, data frame-level decision label truth verification can be implemented.

[0157] 2. The closed-loop verification is to perform segment-level verification on the segment data.

[0158] In a possible implementation, the closed-loop verification includes: detecting whether a difference between a simulated trajectory of the ego vehicle and a posterior trajectory of the ego vehicle is less than a preset threshold.

[0159] The simulation trajectory of the ego vehicle is obtained by predicting the multi-vehicle joint model based on the input ego vehicle decision annotation data and the segment data.

[0160] For game obstacles, the state of the self-vehicle will affect the state of other vehicles. The joint optimization algorithm can simultaneously optimize the state of the self-vehicle and other vehicles and find a set of joint optimal solution actions, so the planning result will be more human-like. Among them, the multi-vehicle joint model minimizeL can be expressed as:

[0161] minimize L=L ego +L obj +L joint ;

[0162] Among them, L ego represents the loss associated with the ego vehicle and the non-joint optimization objective; L obj represents the loss of the joint optimization objective; L joint represents the loss reflecting the interaction between the ego vehicle and the joint optimization objective.

[0163] Optionally, the interactive game scenarios can be divided into two categories: grabbing and giving way, and then a set of joint optimization models can be maintained for different scenarios. Among them, the constraints and weights can be adjusted according to the posture relationship between the vehicle and the joint optimization target when grabbing and giving way.

[0164] Among them, the decision labeling data and fragment data of the ego vehicle are input into the above-mentioned multi-vehicle joint model minimize L to obtain the simulation trajectory of the ego vehicle.

[0165] Specifically, the fragment data also includes the posterior trajectory of the vehicle and the posterior trajectory of the game obstacle. The process of obtaining the simulated trajectory of the vehicle specifically includes:

[0166] For the i-th frame data among the N frames of data sampled from the fragment data, i is an integer greater than 1 and less than or equal to N:

[0167] S1, inputting the posterior trajectory of the game obstacle in the segment data, the driving parameters of the game obstacle, the road topology information, the posterior trajectory of the ego vehicle and the ego vehicle decision labeling data into the multi-vehicle joint model to obtain the simulation trajectory of each game obstacle in the i-th frame data;

[0168] S2, inputting the i-1th initial simulation trajectory of the self-vehicle, the simulation trajectory of each game obstacle in the i-th frame data, and the fragment data into the multi-vehicle joint model to obtain the trajectory result of the self-vehicle corresponding to the i-th frame data;

[0169] The i-1th initial simulation trajectory of the ego vehicle is obtained based on the i-2th initial simulation trajectory of the ego vehicle and the trajectory result of the ego vehicle corresponding to the i-1th frame data, wherein when i is equal to 2, the 0th initial simulation trajectory is the posterior trajectory of the ego vehicle;

[0170] Let i=i+1, and repeat the above steps until the trajectory result of the vehicle corresponding to the Nth frame data is obtained. The simulated trajectory of the vehicle includes the trajectory result of the vehicle corresponding to the Nth frame data and the trajectory result of the vehicle corresponding to the i-th frame data.

[0171] That is to say, the trajectory result of the current frame of the vehicle is obtained based on the obtained simulation trajectory of the game obstacle in each frame of data and the initial simulation trajectory of the vehicle in the previous frame of data, and then the initial simulation trajectory of the current frame of the vehicle is obtained based on the trajectory result of the current frame of the vehicle and the initial simulation trajectory of the vehicle in the previous frame of data. And so on, until the trajectory result of the last frame is obtained, wherein the simulation trajectory of the vehicle includes each obtained trajectory result.

[0172] The above implementation method is introduced in detail below.

[0173] In a possible implementation, for the i-th frame of data in N frames of data sampled from the fragment data, step S1 may include:

[0174] Based on the posterior trajectory of the game obstacle, the posterior trajectory of the ego vehicle and the ego vehicle decision labeling data, a second distance-time ST graph corresponding to the ego vehicle and the game obstacle in the i-th frame data is established. In other words, an ST graph containing relevant information of both the ego vehicle and the game obstacle is established. For the method of establishing the ST graph, please refer to the above-mentioned records, which will not be repeated here.

[0175] Then, at least one collision point between the game obstacle and the vehicle in the i-th frame data is obtained based on the second ST graph. It can be understood that the collision point is the intersection of the tracks corresponding to the game obstacle and the vehicle in the ST graph. For example, the collision point between the obstacle and the vehicle is found based on the ST graph and the grab-and-give relationship.

[0176] Based on the time when the self-vehicle and the game obstacle respectively arrive at the corresponding collision points in the i-th frame data, the game obstacles are sorted to obtain the sorting result of the game obstacles corresponding to the i-th frame data.

[0177] For example, the weight corresponding to the vehicle can be set to w e , the weight corresponding to the obstacle is w o , the time t for the ego vehicle and the obstacle to reach the collision point is calculated based on the following method eo :

[0178] t eo =w e t(ego,P c olli)+w o t(obj k , P c olli);

[0179] Among them, t(ego, P c olli) represents the time from the vehicle to the collision point, t(obj k , P c olli) represents the time from the kth obstacle to the collision point.

[0180] Based on this method, the time when the ego vehicle and the obstacle reach the collision point can be solved, and then the order from small to large can be sorted to obtain the sorting result of the game obstacles corresponding to the i-th frame data. In this way, the most critical obstacle can be obtained based on the sorting, that is, the obstacle with the greatest impact on the ego vehicle.

[0181] Based on the sorting results of the game obstacles obtained above, by inputting the sorting results of the game obstacles corresponding to the i-th frame data, the posterior trajectory of the game obstacles in the fragment data, the driving parameters of the game obstacles, the road topology information, the posterior trajectory of the ego vehicle and the ego vehicle decision annotation data into the multi-vehicle joint model, the simulation trajectory of each game obstacle in the i-th frame data can be obtained.

[0182] Among them, the simulation trajectory of the kth game obstacle in the i-th frame data is related to the simulation trajectory of the obstacle located before the kth game obstacle in the sorting result, and k is an integer greater than 1.

[0183] Exemplarily, based on the sorting result of the game obstacle corresponding to the i-th frame data, the simulation trajectory of the first game obstacle in the sorting result is first obtained, and then based on the simulation trajectory of the first game obstacle, the posterior trajectory of the game obstacle in the fragment data, the driving parameters of the game obstacle, the road topology information, the posterior trajectory of the ego vehicle, the ego vehicle decision-making annotation data, and the multi-vehicle joint model, the simulation trajectory of the second game obstacle is obtained; and then based on the simulation trajectory of the first game obstacle, the simulation trajectory of the second game obstacle, the posterior trajectory of the game obstacle in the fragment data, the driving parameters of the game obstacle, the road topology information, the posterior trajectory of the ego vehicle, the ego vehicle decision-making annotation data, and the multi-vehicle joint model, the simulation trajectory of the third game obstacle is obtained. By analogy, the simulation trajectory of each game obstacle can be obtained.

[0184] For step S2: by inputting the i-1th initial simulation trajectory of the self-vehicle, the simulation trajectory of each game obstacle in the i-th frame data, and the fragment data into the multi-vehicle joint model, the trajectory result of the self-vehicle corresponding to the i-th frame data is obtained.

[0185] The trajectory result of the vehicle corresponding to the i-th frame data may be, for example, a point.

[0186] The i-1th initial simulation trajectory of the ego vehicle is obtained based on the i-2th initial simulation trajectory of the ego vehicle and the trajectory result of the ego vehicle corresponding to the i-1th frame data. When i is equal to 2, the 0th initial simulation trajectory is the posterior trajectory of the ego vehicle.

[0187] That is to say, based on the previous initial simulation trajectory, the trajectory result of the vehicle corresponding to the current frame data can be obtained, and then based on the trajectory result of the vehicle corresponding to the current frame data and the previous initial simulation trajectory, the current initial simulation trajectory can be obtained.

[0188] For example, the 0th initial simulation trajectory of the vehicle is the posterior trajectory of the vehicle, and the trajectory result of the vehicle corresponding to the first frame data is obtained based on the 0th initial simulation trajectory. The posterior trajectory result of the vehicle corresponding to the first frame data in the 0th initial simulation trajectory (i.e., the posterior trajectory of the vehicle) is replaced with the trajectory result of the vehicle corresponding to the first frame data, and the simulation trajectory before the first frame data in the 0th initial simulation trajectory (here, it is none) can be obtained to obtain the first initial simulation trajectory. Then, based on the first initial simulation trajectory, the trajectory result of the vehicle corresponding to the second frame data is obtained. The posterior trajectory result of the vehicle corresponding to the second frame data in the first initial simulation trajectory is replaced with the trajectory result of the vehicle corresponding to the second frame data, and the simulation trajectory before the second frame data in the first initial simulation trajectory (here, it is the simulation trajectory corresponding to the first frame data) can be obtained to obtain the second initial simulation trajectory. By analogy, the Nth initial simulation trajectory of the vehicle and N trajectory results can be obtained. Wherein, the simulation trajectory of the vehicle includes the N trajectory results. That is to say, the trajectory result of the previous step is used as the starting point of the next step trajectory to perform the vehicle control operation, and the position information of the trajectory of the point is obtained based on the calculation, and the iteration is repeated until the segment (N frames of data) or the trajectory ends, and the trajectory result of the self-vehicle for joint optimization of the vehicle control is obtained. Exemplarily, the second initial simulation trajectory includes the trajectory result of the self-vehicle corresponding to the second frame of data, and also includes the posterior trajectory corresponding to the third frame of data to the N+1 frame of data. Correspondingly, the third initial simulation trajectory includes the trajectory result of the self-vehicle corresponding to the third frame of data, and also includes the posterior trajectory corresponding to the fourth frame of data to the N+2 frame of data... Generally, the above-mentioned segment data is sampled to obtain 2N, 3N and other frame data.

[0189] Among them, obstacles that the vehicle has no intention to give way to are non-game obstacles. In a possible implementation, the segment data also includes a posteriori trajectory of the non-game obstacle. The simulated trajectory of the non-game obstacle can be obtained by interpolating the posteriori trajectory of the non-game obstacle.

[0190] For example, for the entire trajectory of a non-game obstacle, a short-term a posteriori trajectory is first used, and then the acceleration and speed are fixed, and the uniform velocity trajectory and the uniform velocity trajectory are calculated by interpolation to replace the a posteriori trajectory. Of course, other methods can also be used, and this solution does not limit this.

[0191] Optionally, step S1 may include:

[0192] The simulation trajectory of the non-game obstacle, the posterior trajectory of the game obstacle in the fragment data, the driving parameters of the game obstacle, the road topology information, the posterior trajectory of the ego vehicle and the ego vehicle decision labeling data are all input into the multi-vehicle joint model to obtain the simulation trajectory of each game obstacle in the i-th frame data.

[0193] For example, based on the simulated trajectory of the non-game obstacle, the posterior trajectory of the game obstacle, the posterior trajectory of the ego vehicle and the ego vehicle decision annotation data, a second distance-time ST diagram corresponding to the ego vehicle and the game obstacle in the i-th frame data is established.

[0194] Correspondingly, by inputting the simulation trajectory of the non-game obstacle, the i-1th initial simulation trajectory of the ego vehicle, the simulation trajectory of each game obstacle in the i-th frame data, and the fragment data into the multi-vehicle joint model, the trajectory result of the ego vehicle corresponding to the i-th frame data is obtained.

[0195] Based on the simulated trajectory of the vehicle obtained above, for example, by setting a threshold for trajectory inconsistency, the overall accuracy of the label is evaluated according to the root mean square error value RMSE of the speed V as shown below.

[0196]

[0197] in, Indicates the actual speed, represents the simulation speed, t represents the time, and N represents the number of frames.

[0198] Of course, other evaluation methods can also be used, for example, based on the root mean square error RMSE of the distance s, to evaluate the overall accuracy of the label, where

[0199] The output result of the verification may be, for example, that the error is small and the label is correct; the error is slightly large and needs to be checked; the error is large and the label is wrong, etc.

[0200] like Figure 4 As shown in the figure, for the i-th frame data, i.e., the data at time t, the gaming obstacles and non-gaming obstacles are classified and sorted, and then the simulation trajectory of the non-gaming obstacles is generated based on the posterior trajectory of the non-gaming obstacles; then, based on the multi-vehicle joint model, the simulation trajectory of the ego vehicle is finally obtained through continuous iteration.

[0201] In this example, the ego vehicle trajectory is generated based on the labeled data and the vehicle is controlled, and the difference between the simulated and a posteriori trajectories is compared to determine the overall accuracy of the labeled data in the segment. That is, by adding the vehicle control, we can see whether a reasonable trajectory can be obtained after the entire segment is controlled with the current labeled data results.

[0202] It is understandable that when only one of the operations is used for verification, the final verification result is consistent with the verification result of the operation. When the above two operations are used for verification at the same time, if the verification result of at least one of the operations indicates failure, the final verification result is failure. If the verification results of both operations indicate success, the final verification result is success.

[0203] Among them, in the daily data collection and road testing on the vehicle side, a large amount of data will be collected for autonomous driving algorithm development and training evaluation. Based on the above data cleaning and / or verification of this solution, such as Figure 5 As shown, the massive amount of collected data and road test data can also be screened to distinguish, for example, radical or conservative data (such as exceeding the self-driving ability (speed, excessive acceleration, and data based on verification results), exceeding the speed limit, driving in congested traffic, etc., and far below the speed limit or traffic speed, giving way too early, etc.), so as to select and save high-value human driving data and improve data quality. At the same time, for some complex or difficult scenarios (such as continuous squeezing by other cars, incorrect behaviors of giving way, etc., and large differences between the planned and actual trajectories), the data can be screened to obtain relevant difficult examples, which is convenient for training for complex or difficult scenarios.

[0204] In an embodiment of the present application, segment data of the vehicle's travel and the vehicle's decision-making annotation data of the segment data are obtained, and then the vehicle's decision-making annotation data is verified based on the segment data, and a verification result is output. The verification can be an open-loop verification that performs frame-level verification on the segment data, or a closed-loop verification that performs segment-level verification on the segment data. The verification can be a combination of open-loop verification and closed-loop verification. By adopting this method, the annotation quality of training data decision labels in various traffic scenarios can be improved, and automated data quality inspection can effectively reduce the labor cost of annotation review and effectively improve the efficiency of the closed loop of regulatory control data.

[0205] This solution can be used to verify the results of manual and automatic annotation in the data closed-loop process. Based on the above verification, the manpower of manual review can be saved. For example, if the verification result shows that there is no problem with the result of automatic annotation, the subsequent manual scene annotation can be performed directly, skipping the review and acceptance. If the verification result shows that the annotation result is incorrect but there is no problem with the basic data (speed, acceleration, etc., etc., and other error messages other than the large mean square error obtained by the verification, such as Wrong IG and Wrong YD), the original process is followed for review and acceptance. Among them, the review and acceptance process can also continue to use the above verification to verify the modified results. For discardable data that has not passed the basic data cleaning, there is no need for subsequent manual annotation. Furthermore, by eliminating unreasonable data such as radical / conservative driving and drastic changes in intention, the efficiency of the data closed loop is improved. On the other hand, the quality of the returned data can also be improved, and the cost of data storage and transmission can be reduced.

[0206] Reference Figure 6As shown, it is a flow chart of a method for displaying gaming obstacles provided by an embodiment of the present application. The method may include steps 601-603. It should be understood that, for the convenience of description, the present application is described in the order of 601-603, and is not intended to be limited to execution in the above order. The present application embodiment does not limit the order of execution, execution time, number of executions, etc. of the above one or more steps. The following description takes the execution subject of steps 601-603 of the method for displaying gaming obstacles as a vehicle as an example, and the present application is also applicable to other execution subjects. Steps 601-603 are as follows:

[0207] 601. Obtain segment data of the ego vehicle's travel and ego vehicle decision labeling data of the segment data, wherein the segment data is used at least to describe the travel conditions of the ego vehicle and a game obstacle interacting with the ego vehicle, and the segment data includes a posterior trajectory of the game obstacle and a posterior trajectory of the ego vehicle.

[0208] For an introduction to this section, see Figure 2a The description of step 201 in the illustrated embodiment will not be repeated here.

[0209] 602. Based on the fragment data and the ego-vehicle decision labeling data, obtain a ranking result of the game obstacles corresponding to the fragment data; the ranking result is determined based on the time sequence of the ego-vehicle and the game obstacles arriving at corresponding collision points respectively.

[0210] In a possible implementation, the collision point is determined based on the distance-time ST graph corresponding to the ego vehicle and the game obstacle, and the ST graph is established based on the posterior trajectory of the game obstacle, the posterior trajectory of the ego vehicle, and the ego vehicle decision-making annotation data. Figure 2a The description of step 202 in the illustrated embodiment will not be repeated here.

[0211] 603. Control and display the sorting results of the game obstacles corresponding to the fragment data.

[0212] After obtaining the ranking results of the above-mentioned game obstacles, prompts such as display and playback can also be performed. For example, when applied to a human-machine interaction (HMI) visualization interface, it can effectively remind the driver of the obstacles that need the most attention / concern.

[0213] In a possible implementation, gaming obstacles and non-gaming obstacles may be distinguished and displayed.

[0214] In the embodiment of the present application, based on processing the segment data of the vehicle's travel and the vehicle's decision-making annotation data of the segment data, the ranking results of the game obstacles corresponding to the segment data are obtained and displayed, which can effectively remind the driver of the obstacles that need the most attention / concern.

[0215] It should be noted that in the various embodiments of the present application, unless otherwise specified or there is a logical conflict, the terms and / or descriptions between the various embodiments are consistent and can be referenced mutually, and the technical features in different embodiments can be combined to form new embodiments according to their internal logical relationships.

[0216] The above detailed description of the method of the embodiment of the present application, the following provides a device of the embodiment of the present application. It can be understood that in the various device embodiments of the present application, the division of multiple units or modules is only a logical division according to function, and is not used as a limitation on the specific structure of the device. In a specific implementation, some functional modules may be subdivided into more small functional modules, and some functional modules may also be combined into one functional module, but no matter whether these functional modules are subdivided or combined, the general process performed by the device is the same. For example, some devices contain a receiving unit and a sending unit. In some designs, the sending unit and the receiving unit can also be integrated into a communication unit, which can implement the functions implemented by the receiving unit and the sending unit. Usually, each unit corresponds to its own program code (or program instruction), and when the program code corresponding to each of these units is run on the processor, the unit is controlled by the processing unit to execute the corresponding process to implement the corresponding function.

[0217] The embodiments of the present application also provide a device for implementing any of the above methods, for example, providing an autonomous driving annotation data quality verification device including a module (or means) for implementing each step performed by the vehicle-side chip in any of the above methods. For another example, another display device for gaming obstacles is also provided, including a module (or means) for implementing each step performed by the vehicle in any of the above methods.

[0218] For example, refer to Figure 7 , which is a schematic diagram of the structure of an automatic driving annotation data quality verification device provided by an embodiment of the present application. The automatic driving annotation data quality verification device is used to implement the aforementioned automatic driving annotation data quality verification method, for example Figure 2a The autonomous driving labeling data quality verification method shown.

[0219] like Figure 7 As shown, the device may include an acquisition module 701 and a verification module 702, which are specifically as follows:

[0220] An acquisition module 701 is used to acquire segment data of the ego vehicle and ego vehicle decision annotation data of the segment data, wherein the segment data is used to at least describe the driving conditions of the ego vehicle and the game obstacles interacting with the ego vehicle;

[0221] A verification module 702, configured to verify the self-vehicle decision-making annotation data based on the segment data and output a verification result;

[0222] The verification includes at least one of an open-loop verification and a closed-loop verification, the open-loop verification is a frame-level verification of the fragment data, and the closed-loop verification is a fragment-level verification of the fragment data.

[0223] In a possible implementation, the open-loop verification includes: determining whether N frames of data sampled from the segment data all satisfy kinematic constraint information based on the ego-vehicle decision annotation data, where N is a positive integer.

[0224] In a possible implementation, the closed-loop verification includes: detecting whether the difference between the simulated trajectory of the ego vehicle and the posterior trajectory of the ego vehicle is less than a preset threshold, the simulated trajectory of the ego vehicle is predicted by a multi-vehicle joint model based on the input ego vehicle decision annotation data and the fragment data, and the posterior trajectory of the ego vehicle is included in the fragment data.

[0225] In a possible implementation, the segment data includes driving parameters of the vehicle, driving parameters of the game obstacle, and road topology information.

[0226] In a possible implementation, the verification module 702 is used to:

[0227] Solving based on the kinematic constraint information, the ego-vehicle decision labeling data, the segment data, and the quadratic programming QP model to determine whether N predicted trajectories of the ego-vehicle can be generated, the N predicted trajectories corresponding to the N frames of data;

[0228] If N predicted trajectories can be generated, it is confirmed that the N frames of data all satisfy the kinematic constraint information.

[0229] In a possible implementation, the segment data further includes a posterior trajectory of the gaming obstacle, and the verification module 702 is used to:

[0230] When it is determined that there is a preset space between the ego vehicle and the game obstacle based on the first distance-time ST diagram of the game obstacle, a solution is performed based on the kinematic constraint information, the ego vehicle decision annotation data, the fragment data and the quadratic programming QP model; wherein the first ST diagram is established based on the posterior trajectory of the game obstacle and the ego vehicle decision annotation data.

[0231] In a possible implementation, the segment data further includes a posterior trajectory of the gaming obstacle, and the verification module 702 is used to:

[0232] Wherein, for the i-th frame data in N frames of data sampled from the fragment data, i is an integer greater than 1 and less than or equal to N:

[0233] The posterior trajectory of the game obstacle in the segment data, the driving parameters of the game obstacle, the road topology information, the posterior trajectory of the ego vehicle and the ego vehicle decision labeling data are input into the multi-vehicle joint model to obtain the simulation trajectory of each game obstacle in the i-th frame data;

[0234] Inputting the i-1th initial simulation trajectory of the self-vehicle, the simulation trajectory of each game obstacle in the i-th frame data, and the fragment data into the multi-vehicle joint model to obtain the trajectory result of the self-vehicle corresponding to the i-th frame data;

[0235] The i-1th initial simulation trajectory of the ego vehicle is obtained based on the i-2th initial simulation trajectory of the ego vehicle and the trajectory result of the ego vehicle corresponding to the i-1th frame data, wherein when i is equal to 2, the 0th initial simulation trajectory is the posterior trajectory of the ego vehicle;

[0236] The Nth initial simulation trajectory of the vehicle is obtained based on the N-1th initial simulation trajectory of the vehicle and the trajectory result of the vehicle corresponding to the Nth frame data;

[0237] The simulated trajectory of the vehicle includes the trajectory result of the vehicle corresponding to the Nth frame data and the trajectory result of the vehicle corresponding to the i-th frame data.

[0238] In a possible implementation, the verification module 702 is used to:

[0239] Based on the posterior trajectory of the game obstacle, the posterior trajectory of the ego vehicle and the ego vehicle decision labeling data, a second distance-time ST graph corresponding to the ego vehicle and the game obstacle in the i-th frame data is established;

[0240] Obtaining at least one collision point between the game obstacle and the vehicle in the i-th frame data based on the second ST graph;

[0241] Based on the time at which the self-vehicle and the game obstacle respectively arrive at the corresponding collision point in the i-th frame data, the game obstacles are sorted to obtain a sorting result of the game obstacles corresponding to the i-th frame data;

[0242] The sorting result of the game obstacles corresponding to the i-th frame data, the posterior trajectory of the game obstacles in the fragment data, the driving parameters of the game obstacles, the road topology information, the posterior trajectory of the ego vehicle and the ego vehicle decision annotation data are all input into the multi-vehicle joint model to obtain the simulation trajectory of each game obstacle in the i-th frame data, wherein the simulation trajectory of the k-th game obstacle in the i-th frame data is related to the simulation trajectory of the obstacle before the k-th game obstacle in the sorting result, and k is an integer greater than 1.

[0243] In a possible implementation, the device further includes a display module, configured to display a sorting result of the game obstacles corresponding to the i-th frame data.

[0244] In a possible implementation, the segment data further includes a posterior trajectory of a non-gaming obstacle, and the verification module 702 is further configured to:

[0245] Performing interpolation processing on the posterior trajectory of the non-game obstacle to obtain a simulated trajectory of the non-game obstacle;

[0246] The simulation trajectory of the non-game obstacle, the i-1th initial simulation trajectory of the ego vehicle, the simulation trajectory of each game obstacle in the i-th frame data, and the fragment data are all input into the multi-vehicle joint model to obtain the trajectory result of the ego vehicle corresponding to the i-th frame data.

[0247] For example, refer to Figure 8 FIG. 1 is a schematic diagram of the structure of a display device for gaming obstacles provided in an embodiment of the present application. The display device for gaming obstacles is used to implement the aforementioned display method for gaming obstacles, for example Figure 6 The method of displaying the gaming obstacles shown.

[0248] like Figure 8 As shown, the device may include an acquisition module 801, a processing module 802 and a display module 803, which are specifically as follows:

[0249] An acquisition module 801 is used to acquire segment data of the ego vehicle and ego vehicle decision labeling data of the segment data, wherein the segment data is used to at least describe the driving conditions of the ego vehicle and the game obstacle interacting with the ego vehicle, and the segment data includes the posterior trajectory of the game obstacle and the posterior trajectory of the ego vehicle;

[0250] The processing module 802 is used to obtain a ranking result of the game obstacles corresponding to the fragment data based on the fragment data and the ego vehicle decision labeling data; the ranking result is determined based on the time sequence of the ego vehicle and the game obstacles arriving at the corresponding collision points respectively;

[0251] The display module 803 is used to control the display of the sorting results of the game obstacles corresponding to the fragment data.

[0252] In one possible implementation, the collision point is determined based on a distance-time ST diagram corresponding to the ego vehicle and the game obstacle, and the ST diagram is established based on the posterior trajectory of the game obstacle, the posterior trajectory of the ego vehicle, and the ego vehicle decision annotation data.

[0253] It should be understood that the division of each module in the above devices is only a division of logical functions. In actual implementation, they can be fully or partially integrated into one physical entity, or they can be physically separated. In addition, the modules in the automatic driving labeling data quality verification device and the display device of the gaming obstacle can be implemented in the form of a processor calling software; for example, the automatic driving labeling data quality verification device and the display device of the gaming obstacle include a processor, the processor is connected to a memory, and instructions are stored in the memory. The processor calls the instructions stored in the memory to implement any of the above methods or implement the functions of each module of the device, wherein the processor is, for example, a general-purpose processor, such as a central processing unit (CPU) or a microprocessor, and the memory is a memory inside the device or a memory outside the device. Alternatively, the modules in the device can be implemented in the form of hardware circuits, and the functions of some or all units can be realized by designing the hardware circuits, and the hardware circuits can be understood as one or more processors; for example, in one implementation, the hardware circuit is an application-specific integrated circuit (ASIC), and the functions of some or all of the above units are realized by designing the logical relationship of the components in the circuit; for another example, in another implementation, the hardware circuit can be realized by a programmable logic device (PLD), taking a field programmable gate array (FPGA) as an example, which can include a large number of logic gate circuits, and the connection relationship between the logic gate circuits is configured by configuring the configuration file, so as to realize the functions of some or all of the above units. All modules of the above devices can be realized in the form of a processor calling software, or in the form of hardware circuits, or in part by a processor calling software, and the rest by hardware circuits.

[0254] Reference Fig. 9 As shown in FIG. 1 , it is a hardware structure diagram of another automatic driving labeling data quality verification device provided in an embodiment of the present application. Fig. 9 The autonomous driving labeling data quality verification device 900 (the device 900 may be a computer device) includes a memory 901, a processor 902, a communication interface 903, and a bus 904. The memory 901, the processor 902, and the communication interface 903 are connected to each other through the bus 904.

[0255] The memory 901 may be a read only memory (ROM), a static storage device, a dynamic storage device or a random access memory (RAM).

[0256] The memory 901 can store programs. When the program stored in the memory 901 is executed by the processor 902, the processor 902 and the communication interface 903 are used to execute the various steps of the autonomous driving annotation data quality verification method of the embodiment of the present application.

[0257] The processor 902 is a circuit with signal processing capability. In one implementation, the processor 902 may be a circuit with instruction reading and running capability, such as a central processing unit CPU, a microprocessor, a graphics processing unit (GPU) (which may be understood as a microprocessor), or a digital signal processor (DSP); in another implementation, the processor 902 may implement certain functions through the logical relationship of a hardware circuit, and the logical relationship of the hardware circuit may be fixed or reconfigurable, such as the processor 902 being a hardware circuit implemented by an ASIC or a programmable logic device PLD, such as an FPGA. In a reconfigurable hardware circuit, the process of the processor loading a configuration document to implement the hardware circuit configuration may be understood as the process of the processor loading instructions to implement the functions of some or all of the above modules. In addition, it may also be a hardware circuit designed for artificial intelligence, which may be understood as an ASIC, such as a neural network processing unit (NPU), a tensor processing unit (TPU), a deep learning processing unit (DPU), etc. Processor 902 is used to execute relevant programs to implement the functions required to be performed by the units in the autonomous driving labeling data quality verification device of the embodiment of the present application, or to execute the autonomous driving labeling data quality verification method of the method embodiment of the present application.

[0258] It can be seen that each module in the above device can be one or more processors (or processing circuits) configured to implement the above method, such as: CPU, GPU, NPU, TPU, DPU, microprocessor, DSP, ASIC, FPGA, or a combination of at least two of these processor forms.

[0259] In addition, the modules in the above device can be fully or partially integrated together, or can be implemented independently. In one implementation, these modules are integrated together and implemented in the form of a system-on-a-chip (SOC). The SOC may include at least one processor for implementing any of the above methods or implementing the functions of the modules of the device. The type of the at least one processor may be different, for example, including a CPU and an FPGA, a CPU and an artificial intelligence processor, a CPU and a GPU, etc.

[0260] The communication interface 903 uses a transceiver device such as, but not limited to, a transceiver to implement communication between the device 900 and other devices or a communication network. For example, data can be obtained through the communication interface 903 .

[0261] The bus 904 may include a path for transmitting information between various components of the device 900 (eg, the memory 901 , the processor 902 , and the communication interface 903 ).

[0262] It should be noted that although Fig. 9 The device 900 shown only shows a memory, a processor, and a communication interface, but in the specific implementation process, those skilled in the art should understand that the device 900 also includes other devices necessary for normal operation. At the same time, according to specific needs, those skilled in the art should understand that the device 900 may also include hardware devices for implementing other additional functions. In addition, those skilled in the art should understand that the device 900 may also only include the devices necessary to implement the embodiments of the present application, and does not necessarily include Fig. 9 All devices shown in .

[0263] The above only introduces the autonomous driving labeling data quality verification device as an example. The display device of the game obstacle can be found in Fig. 9 The description of the illustrated embodiment will not be repeated here.

[0264] An embodiment of the present application also provides a computer-readable storage medium, which stores instructions. When the computer-readable storage medium is executed on a computer or a processor, the computer or the processor executes one or more steps in any of the above methods.

[0265] The embodiment of the present application further provides a computer program product including instructions. When the computer program product is executed on a computer or a processor, the computer or the processor executes one or more steps in any of the above methods.

[0266] It should be understood that in the description of the present application, unless otherwise specified, " / " indicates that the objects associated before and after are in an "or" relationship, for example, A / B can represent A or B; wherein A and B can be singular or plural. Also, in the description of the present application, unless otherwise specified, "multiple" refers to two or more than two. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single items or plural items. For example, at least one of a, b, or c can represent: a, b, c, ab, ac, bc, or abc, wherein a, b, c can be single or multiple. In addition, in order to facilitate the clear description of the technical solutions of the embodiments of the present application, in the embodiments of the present application, the words "first", "second", etc. are used to distinguish the same items or similar items with substantially the same functions and effects. Those skilled in the art can understand that the words "first", "second", etc. do not limit the quantity and execution order, and the words "first", "second", etc. do not limit them to be necessarily different. Meanwhile, in the embodiments of the present application, words such as "exemplary" or "for example" are used to indicate examples, illustrations or descriptions. Any embodiment or design described as "exemplary" or "for example" in the embodiments of the present application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "exemplary" or "for example" is intended to present related concepts in a concrete manner for ease of understanding.

[0267] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the division of the unit is only a logical function division, and there may be other division methods in actual implementation, for example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. The mutual coupling, direct coupling, or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0268] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0269] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function according to the embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions can be stored in a computer-readable storage medium or transmitted through the computer-readable storage medium. The computer instructions can be transmitted from a website site, computer, server or data center to another website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (digital subscriber line, DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that contains one or more available media integrations. The available medium may be a read-only memory (ROM), or a random access memory (RAM), or a magnetic medium, such as a floppy disk, a hard disk, a tape, a disk, or an optical medium, such as a digital versatile disc (DVD), or a semiconductor medium, such as a solid state disk (SSD), etc.

[0270] The above is only a specific implementation of the embodiment of the present application, but the protection scope of the embodiment of the present application is not limited thereto, and any changes or replacements within the technical scope disclosed in the embodiment of the present application should be included in the protection scope of the embodiment of the present application. Therefore, the protection scope of the embodiment of the present application should be based on the protection scope of the claims.

Claims

1. A method for verifying the quality of autonomous driving annotation data, characterized in that: include: Acquire segment data of the ego vehicle's travel and ego vehicle decision annotation data of the segment data, wherein the segment data is used to at least describe the travel conditions of the ego vehicle and the game obstacles interacting with the ego vehicle; Verifying the self-vehicle decision-making annotation data based on the segment data, and outputting a verification result; The verification includes at least one of an open-loop verification and a closed-loop verification, the open-loop verification is a frame-level verification of the fragment data, and the closed-loop verification is a fragment-level verification of the fragment data.

2. The method according to claim 1, characterized in that The open-loop verification includes: determining whether N frames of data sampled from the segment data all satisfy kinematic constraint information based on the ego-vehicle decision-labeled data, where N is a positive integer.

3. The method according to claim 1 or 2, characterized in that: The closed-loop verification includes: detecting whether the difference between the simulated trajectory of the ego vehicle and the posterior trajectory of the ego vehicle is less than a preset threshold, the simulated trajectory of the ego vehicle is obtained by predicting the multi-vehicle joint model based on the input ego vehicle decision annotation data and the fragment data, and the posterior trajectory of the ego vehicle is included in the fragment data.

4. The method according to any one of claims 1 to 3, characterized in that: The segment data includes the driving parameters of the vehicle, the driving parameters of the game obstacle and the road topology information.

5. The method according to claim 2, characterized in that: The determining, based on the ego-vehicle decision-marking data, whether all N frames of data sampled from the segment data satisfy kinematic constraint information includes: Solving based on the kinematic constraint information, the ego-vehicle decision labeling data, the segment data, and the quadratic programming QP model to determine whether N predicted trajectories of the ego-vehicle can be generated, the N predicted trajectories corresponding to the N frames of data; If N predicted trajectories can be generated, it is confirmed that the N frames of data all satisfy the kinematic constraint information.

6. The method according to claim 5, characterized in that The fragment data also includes the posterior trajectory of the game obstacle, and the solution based on the kinematic constraint information, the self-vehicle decision labeling data, the fragment data and the quadratic programming QP model includes: When it is determined based on the first distance-time ST graph of the game obstacle that there is a preset space between the ego vehicle and the game obstacle, solving is performed based on the kinematic constraint information, the ego vehicle decision annotation data, the fragment data, and the quadratic programming QP model; The first ST diagram is established based on the posterior trajectory of the game obstacle and the self-vehicle decision labeling data.

7. The method according to claim 3, characterized in that The fragment data also includes the posterior trajectory of the game obstacle, and the process of obtaining the simulated trajectory of the vehicle is specifically include: Wherein, for the i-th frame data in N frames of data sampled from the fragment data, i is an integer greater than 1 and less than or equal to N: The posterior trajectory of the game obstacle in the segment data, the driving parameters of the game obstacle, the road topology information, the posterior trajectory of the ego vehicle and the ego vehicle decision labeling data are input into the multi-vehicle joint model to obtain the simulation trajectory of each game obstacle in the i-th frame data; Inputting the i-1th initial simulation trajectory of the self-vehicle, the simulation trajectory of each game obstacle in the i-th frame data, and the fragment data into the multi-vehicle joint model to obtain the trajectory result of the self-vehicle corresponding to the i-th frame data; The i-1th initial simulation trajectory of the ego vehicle is obtained based on the i-2th initial simulation trajectory of the ego vehicle and the trajectory result of the ego vehicle corresponding to the i-1th frame data, wherein when i is equal to 2, the 0th initial simulation trajectory is the posterior trajectory of the ego vehicle; The Nth initial simulation trajectory of the vehicle is obtained based on the N-1th initial simulation trajectory of the vehicle and the trajectory result of the vehicle corresponding to the Nth frame data; The simulated trajectory of the vehicle includes the trajectory result of the vehicle corresponding to the Nth frame data and the trajectory result of the vehicle corresponding to the i-th frame data.

8. The method according to claim 7, characterized in that The step of inputting the posterior trajectory of the game obstacle in the segment data, the driving parameters of the game obstacle, the road topology information, the posterior trajectory of the ego vehicle and the ego vehicle decision labeling data into the multi-vehicle joint model to obtain the simulation trajectory of each game obstacle in the i-th frame data includes: Based on the posterior trajectory of the game obstacle, the posterior trajectory of the ego vehicle and the ego vehicle decision labeling data, a second distance-time ST graph corresponding to the ego vehicle and the game obstacle in the i-th frame data is established; Obtaining at least one collision point between the game obstacle and the vehicle in the i-th frame data based on the second ST graph; Based on the time at which the self-vehicle and the game obstacle respectively arrive at the corresponding collision point in the i-th frame data, the game obstacles are sorted to obtain a sorting result of the game obstacles corresponding to the i-th frame data; The sorting result of the game obstacles corresponding to the i-th frame data, the posterior trajectory of the game obstacles in the fragment data, the driving parameters of the game obstacles, the road topology information, the posterior trajectory of the ego vehicle and the ego vehicle decision annotation data are all input into the multi-vehicle joint model to obtain the simulation trajectory of each game obstacle in the i-th frame data, wherein the simulation trajectory of the k-th game obstacle in the i-th frame data is related to the simulation trajectory of the obstacle before the k-th game obstacle in the sorting result, and k is an integer greater than 1.

9. The method according to claim 8, characterized in that The method further comprises: Display the sorting results of the game obstacles corresponding to the i-th frame data.

10. The method according to any one of claims 7 to 9, characterized in that: The segment data also includes a posteriori trajectory of a non-gaming obstacle, and the method further includes: Performing interpolation processing on the posterior trajectory of the non-game obstacle to obtain a simulated trajectory of the non-game obstacle; The step of inputting the i-1th initial simulation trajectory of the self-vehicle, the simulation trajectory of each game obstacle in the i-th frame data, and the fragment data into the multi-vehicle joint model to obtain the trajectory result of the self-vehicle corresponding to the i-th frame data includes: The simulation trajectory of the non-game obstacle, the i-1th initial simulation trajectory of the ego vehicle, the simulation trajectory of each game obstacle in the i-th frame data, and the fragment data are all input into the multi-vehicle joint model to obtain the trajectory result of the ego vehicle corresponding to the i-th frame data.

11. A method for displaying gaming obstacles, characterized in that: include: Acquire segment data of the ego vehicle's travel and ego vehicle decision labeling data of the segment data, wherein the segment data is used to at least describe the travel conditions of the ego vehicle and a game obstacle interacting with the ego vehicle, and the segment data includes a posterior trajectory of the game obstacle and a posterior trajectory of the ego vehicle; Based on the segment data and the ego vehicle decision labeling data, a ranking result of the game obstacles corresponding to the segment data is obtained; the ranking result is determined based on the time sequence of the ego vehicle and the game obstacles arriving at the corresponding collision points respectively; Control and display the sorting results of the game obstacles corresponding to the fragment data.

12. The method according to claim 11, characterized in that The collision point is determined based on a distance-time ST diagram corresponding to the ego vehicle and the game obstacle, and the ST diagram is established based on the posterior trajectory of the game obstacle, the posterior trajectory of the ego vehicle and the ego vehicle decision annotation data.

13. An automatic driving annotation data quality verification device, characterized in that: include: An acquisition module, used to acquire segment data of the ego vehicle and ego vehicle decision annotation data of the segment data, wherein the segment data is used to at least describe the driving conditions of the ego vehicle and the gaming obstacles interacting with the ego vehicle; A verification module, used to verify the self-vehicle decision-making annotation data based on the segment data and output a verification result; The verification includes at least one of an open-loop verification and a closed-loop verification, the open-loop verification is a frame-level verification of the fragment data, and the closed-loop verification is a fragment-level verification of the fragment data.

14. The device according to claim 13, characterized in that The open-loop verification includes: determining whether N frames of data sampled from the segment data all satisfy kinematic constraint information based on the ego-vehicle decision-labeled data, where N is a positive integer.

15. The device according to claim 13 or 14, characterized in that The closed-loop verification includes: detecting whether the difference between the simulated trajectory of the ego vehicle and the posterior trajectory of the ego vehicle is less than a preset threshold, the simulated trajectory of the ego vehicle is obtained by predicting the multi-vehicle joint model based on the input ego vehicle decision annotation data and the fragment data, and the posterior trajectory of the ego vehicle is included in the fragment data.

16. The device according to any one of claims 13 to 15, characterized in that The segment data includes the driving parameters of the vehicle, the driving parameters of the game obstacle and the road topology information.

17. The device according to claim 14, characterized in that The verification module is used for: Solving based on the kinematic constraint information, the ego-vehicle decision labeling data, the segment data, and the quadratic programming QP model to determine whether N predicted trajectories of the ego-vehicle can be generated, the N predicted trajectories corresponding to the N frames of data; If N predicted trajectories can be generated, it is confirmed that the N frames of data all satisfy the kinematic constraint information.

18. The device according to claim 17, characterized in that The segment data also includes the posterior trajectory of the game obstacle, and the verification module is used to: When it is determined based on the first distance-time ST graph of the game obstacle that there is a preset space between the ego vehicle and the game obstacle, solving is performed based on the kinematic constraint information, the ego vehicle decision annotation data, the fragment data, and the quadratic programming QP model; The first ST diagram is established based on the posterior trajectory of the game obstacle and the self-vehicle decision labeling data.

19. The device according to claim 15, characterized in that The segment data also includes the posterior trajectory of the game obstacle, and the verification module is used to: Wherein, for the i-th frame data in N frames of data sampled from the fragment data, i is an integer greater than 1 and less than or equal to N: The posterior trajectory of the game obstacle in the segment data, the driving parameters of the game obstacle, the road topology information, the posterior trajectory of the ego vehicle and the ego vehicle decision labeling data are input into the multi-vehicle joint model to obtain the simulation trajectory of each game obstacle in the i-th frame data; Inputting the i-1th initial simulation trajectory of the self-vehicle, the simulation trajectory of each game obstacle in the i-th frame data, and the fragment data into the multi-vehicle joint model to obtain the trajectory result of the self-vehicle corresponding to the i-th frame data; The i-1th initial simulation trajectory of the ego vehicle is obtained based on the i-2th initial simulation trajectory of the ego vehicle and the trajectory result of the ego vehicle corresponding to the i-1th frame data, wherein when i is equal to 2, the 0th initial simulation trajectory is the posterior trajectory of the ego vehicle; The Nth initial simulation trajectory of the vehicle is obtained based on the N-1th initial simulation trajectory of the vehicle and the trajectory result of the vehicle corresponding to the Nth frame data; The simulated trajectory of the vehicle includes the trajectory result of the vehicle corresponding to the Nth frame data and the trajectory result of the vehicle corresponding to the i-th frame data.

20. The device according to claim 19, characterized in that The verification module is used for: Based on the posterior trajectory of the game obstacle, the posterior trajectory of the ego vehicle and the ego vehicle decision labeling data, a second distance-time ST graph corresponding to the ego vehicle and the game obstacle in the i-th frame data is established; Obtaining at least one collision point between the game obstacle and the vehicle in the i-th frame data based on the second ST graph; Based on the time at which the self-vehicle and the game obstacle respectively arrive at the corresponding collision point in the i-th frame data, the game obstacles are sorted to obtain a sorting result of the game obstacles corresponding to the i-th frame data; The sorting result of the game obstacles corresponding to the i-th frame data, the posterior trajectory of the game obstacles in the fragment data, the driving parameters of the game obstacles, the road topology information, the posterior trajectory of the ego vehicle and the ego vehicle decision annotation data are all input into the multi-vehicle joint model to obtain the simulation trajectory of each game obstacle in the i-th frame data, wherein the simulation trajectory of the k-th game obstacle in the i-th frame data is related to the simulation trajectory of the obstacle before the k-th game obstacle in the sorting result, and k is an integer greater than 1.

21. The device according to claim 20, characterized in that The device also includes a display module, which is used for: Display the sorting results of the game obstacles corresponding to the i-th frame data.

22. The device according to any one of claims 19 to 21, characterized in that The segment data also includes a posteriori trajectory of non-game obstacles, and the verification module is further used to: Performing interpolation processing on the posterior trajectory of the non-game obstacle to obtain a simulated trajectory of the non-game obstacle; The simulation trajectory of the non-game obstacle, the i-1th initial simulation trajectory of the ego vehicle, the simulation trajectory of each game obstacle in the i-th frame data, and the fragment data are all input into the multi-vehicle joint model to obtain the trajectory result of the ego vehicle corresponding to the i-th frame data.

23. A display device for gaming obstacles, characterized in that: include: an acquisition module, configured to acquire segment data of the ego vehicle's travel and ego vehicle decision labeling data of the segment data, wherein the segment data is used to at least describe the travel conditions of the ego vehicle and a game obstacle interacting with the ego vehicle, and the segment data includes a posterior trajectory of the game obstacle and a posterior trajectory of the ego vehicle; A processing module, configured to obtain a ranking result of the game obstacles corresponding to the fragment data based on the fragment data and the self-vehicle decision labeling data; The sorting result is determined based on the time sequence in which the ego vehicle and the game obstacle arrive at the corresponding collision points respectively; A display module is used to control and display the sorting results of the game obstacles corresponding to the fragment data.

24. The device according to claim 23, characterized in that The collision point is determined based on a distance-time ST diagram corresponding to the ego vehicle and the game obstacle, and the ST diagram is established based on the posterior trajectory of the game obstacle, the posterior trajectory of the ego vehicle and the ego vehicle decision annotation data.

25. An automatic driving annotation data quality verification device, characterized in that: It comprises a processor and a memory; wherein the memory is used to store program code, and the processor is used to call the program code to execute the method according to any one of claims 1 to 10.

26. A display device for gaming obstacles, characterized in that: It comprises a processor and a memory; wherein the memory is used to store program code, and the processor is used to call the program code to execute the method as claimed in claim 11 or 12.

27. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and the computer program is executed by a processor to implement the method according to any one of claims 1 to 10, or the method according to claim 11 or 12.

28. A computer program product, characterized in that When the computer program product runs on a computer, the computer is caused to execute the method according to any one of claims 1 to 10, or the method according to claim 11 or 12.

29. A vehicle, characterized in that: The vehicle comprises the device as described in any one of claims 13 to 22 and / or the device as described in claim 23 or 24.

Images