Model evaluation method and device, computer equipment and readable storage medium

Abstract

The embodiment of the invention discloses a model evaluation method and device, computer equipment and a readable storage medium. The method comprises the following steps: acquiring original data; the original data at least comprise video data collected by a camera in the actual driving process of the automatic driving vehicle on the test road section, a lidar sensing result in the actual driving process of the automatic driving vehicle on the test road section, and actual driving track data of the automatic driving vehicle on the test road section; based on the video data, a to-be-evaluated perception model is adopted to obtain a simulation perception result; the simulation perception result at least comprises position information of each target at each moment; on the basis of the simulation perception result, obtaining own vehicle planning path data by adopting a preset regulation and control model; and comparing the lidar sensing result, the simulation sensing result, the actual driving track data and the own vehicle planning path data, and determining an evaluation result of the to-be-evaluated sensing model. By applying the scheme provided by the embodiment of the invention, the accuracy of model evaluation can be improved.

Description

Technical Field

[0001] The present invention relates to the field of autonomous driving technology, and in particular to a model evaluation method, device, computer equipment and readable storage medium. Background Art

[0002] In the field of autonomous driving, autonomous vehicles need to be equipped with various cameras, sensors, lidar and other equipment to obtain information about the vehicle's surrounding environment, so that the vehicle can decide its driving path based on the acquired environmental information. After the camera collects video data, the perception model can perform target recognition based on the video data. For example, it can recognize pedestrians, other vehicles and other targets around the vehicle, and then plan the vehicle's driving path based on information such as the location of each target.

[0003] It is understandable that the recognition accuracy of the perception model is an important factor in determining the driving safety of the vehicle. Therefore, the perception model needs to be evaluated before it is put into operation. In other words, the accuracy of the perception model in identifying the target needs to be evaluated. Only the perception model that meets the accuracy requirements can be officially put into use. Therefore, how to accurately evaluate the perception model has become a technical problem that needs to be solved urgently. Summary of the invention

[0004] The present application provides a model evaluation method, apparatus, computer equipment and readable storage medium to accurately evaluate the perception model. The specific technical solution is as follows.

[0005] In a first aspect, an embodiment of the present application provides a model evaluation method, the method comprising:

[0006] Acquire raw data; the raw data includes at least: video data collected by a camera during the actual driving of the autonomous driving vehicle on the test section, lidar perception results of the autonomous driving vehicle during the actual driving of the test section, and actual driving trajectory data of the autonomous driving vehicle on the test section;

[0007] Based on the video data, a simulated perception result is obtained by using the perception model to be evaluated; the simulated perception result at least includes: position information of each target at each moment;

[0008] Based on the simulation perception results, a preset regulation and control model is used to obtain the planned path data of the vehicle;

[0009] The lidar perception result, the simulated perception result, the actual driving trajectory data, and the self-vehicle planned path data are compared to determine the evaluation result of the perception model to be evaluated.

[0010] Optionally, the step of comparing the lidar perception result, the simulation perception result, the actual driving trajectory data, and the self-vehicle planned path data to determine the evaluation result of the perception model to be evaluated includes:

[0011] The trajectory composed of the positions of each target in the lidar perception result within the preset time period is taken as a perfect prediction; the trajectory composed of the positions of each target in the simulation perception result within the preset time period is taken as an imperfect prediction;

[0012] The actual driving trajectory is regarded as a perfect plan, and the planned path of the vehicle is regarded as an imperfect plan;

[0013] Determining whether the perception model to be evaluated has false detection based on the perfect plan and the imperfect prediction; and / or

[0014] It is determined whether the perception model to be evaluated has any missed detections based on the imperfect planning, the perfect prediction, and the imperfect prediction.

[0015] Optionally, the step of determining whether the perception model to be evaluated has false detection according to the perfect planning and the imperfect prediction includes:

[0016] When it is determined that the perfect plan and the imperfect prediction have overlapping points, it is determined that the perception model to be evaluated has false detection;

[0017] The step of determining whether the perception model to be evaluated has missed detection according to the imperfect planning, the perfect prediction and the imperfect prediction comprises:

[0018] When it is determined that there are overlapping points between the imperfect plan and the perfect prediction, and when it is determined that there are no overlapping points between the imperfect plan and the imperfect prediction, it is determined that there is a missed detection in the perception model to be evaluated.

[0019] Optionally, before comparing the lidar perception result with the simulated perception result, and the actual driving trajectory data with the self-vehicle planned path data, and determining the evaluation result of the perception model to be evaluated, the method further includes:

[0020] Identifying first abnormal data in the lidar sensing result and correcting the first abnormal data; and / or

[0021] Identify second abnormal data in the actual driving trajectory data, and correct the second abnormal data.

[0022] Optionally, the step of identifying first abnormal data in the lidar sensing result and correcting the first abnormal data includes:

[0023] In the lidar sensing result, identifying discontinuous position information as first abnormal data, and displaying the first abnormal data;

[0024] The input manual annotation result is received, and the manual annotation result is added to the lidar perception result.

[0025] Optionally, the step of identifying first abnormal data in the lidar sensing result and correcting the first abnormal data includes:

[0026] In the lidar sensing result, identifying first abnormal data corresponding to a vehicle speed greater than or equal to a preset speed threshold;

[0027] In the first abnormal data, the data corresponding to the lidar sensing distance being less than or equal to the preset distance threshold is deleted.

[0028] Optionally, the step of identifying first abnormal data in the lidar sensing result and correcting the first abnormal data includes:

[0029] In the lidar sensing result, identifying first abnormal data corresponding to the abnormal behavior;

[0030] Acquire historical data within a preset time period before the first abnormal data, and generate a prediction result based on a preset prediction model according to the historical data;

[0031] The first abnormal data is replaced by the prediction result.

[0032] Optionally, before using the prediction result to replace the first abnormal data, the method further includes:

[0033] displaying the prediction results;

[0034] A revision operation on the prediction result is received, and the prediction result is updated according to the revision operation; the revision operation includes adding and / or deleting.

[0035] Optionally, the step of identifying second abnormal data in the actual driving trajectory data and correcting the second abnormal data includes:

[0036] In the actual driving trajectory data, second abnormal data corresponding to a preset type of data packet is identified;

[0037] Obtaining the target lidar perception result in the preset type data packet, and obtaining the vehicle trajectory data based on the preset regulation and control model according to the target lidar perception result;

[0038] When there is no overlap between the trajectory of each target included in the target lidar perception result and the own vehicle trajectory, the second abnormal data is replaced by the own vehicle trajectory data.

[0039] Optionally, the lidar perception result and the simulation perception result are both generated based on the rear axle system of the autonomous driving vehicle, and the perfect prediction, the imperfect prediction, the perfect planning, and the imperfect planning are all generated based on the power-on system of the autonomous driving vehicle;

[0040] Before taking the trajectory composed of the positions of each target in the lidar perception result within the preset time period as a perfect prediction and taking the trajectory composed of the positions of each target in the simulation perception result within the preset time period as an imperfect prediction, the method further includes:

[0041] Get the transformation matrix;

[0042] According to the transformation matrix, the lidar sensing result and the simulation sensing result are converted into the power-on system.

[0043] In a second aspect, an embodiment of the present application provides a model evaluation device, the device comprising:

[0044] A data acquisition module is used to acquire raw data; the raw data includes at least: video data collected by a camera during the actual driving of the autonomous driving vehicle on the test section, lidar perception results of the autonomous driving vehicle during the actual driving of the test section, and actual driving trajectory data of the autonomous driving vehicle on the test section;

[0045] A data perception module, used to obtain a simulated perception result based on the video data using a perception model to be evaluated; the simulated perception result at least includes: position information of each target at each time;

[0046] A path planning module, used to obtain the self-vehicle planned path data based on the simulated perception results using a preset regulation and control model;

[0047] The model evaluation module is used to compare the lidar perception result, the simulated perception result, the actual driving trajectory data, and the self-vehicle planned path data to determine the evaluation result of the perception model to be evaluated.

[0048] Optionally, the model evaluation module includes:

[0049] The first determination submodule is used to take the trajectory composed of the positions of each target in the lidar perception result within the preset time period as a perfect prediction; and take the trajectory composed of the positions of each target in the simulation perception result within the preset time period as an imperfect prediction;

[0050] A second determination submodule is used to take the actual driving trajectory as a perfect plan and the self-vehicle planned path as an imperfect plan;

[0051] The model evaluation submodule is used to determine whether the perception model to be evaluated has any false detection based on the perfect plan and the imperfect prediction; and / or to determine whether the perception model to be evaluated has any missed detection based on the imperfect plan, the perfect prediction and the imperfect prediction.

[0052] Optionally, the model evaluation submodule is specifically used to:

[0053] When it is determined that the perfect plan and the imperfect prediction have overlapping points, it is determined that the perception model to be evaluated has false detection;

[0054] When it is determined that there are overlapping points between the imperfect plan and the perfect prediction, and when it is determined that there are no overlapping points between the imperfect plan and the imperfect prediction, it is determined that there is a missed detection in the perception model to be evaluated.

[0055] Optionally, the device also includes: a data processing module, used to identify first abnormal data in the lidar perception result and correct the first abnormal data; and / or, identify second abnormal data in the actual driving trajectory data and correct the second abnormal data.

[0056] Optionally, the data processing module is specifically used to:

[0057] In the lidar sensing result, identifying discontinuous position information as first abnormal data, and displaying the first abnormal data;

[0058] The input manual annotation result is received, and the manual annotation result is added to the lidar perception result.

[0059] Optionally, the data processing module is specifically used to:

[0060] In the lidar sensing result, identifying first abnormal data corresponding to a vehicle speed greater than or equal to a preset speed threshold;

[0061] In the first abnormal data, the data corresponding to the lidar sensing distance being less than or equal to the preset distance threshold is deleted.

[0062] Optionally, the data processing module is specifically used to:

[0063] In the lidar sensing result, identifying first abnormal data corresponding to the abnormal behavior;

[0064] Acquire historical data within a preset time period before the first abnormal data, and generate a prediction result based on a preset prediction model according to the historical data;

[0065] The first abnormal data is replaced by the prediction result.

[0066] Optionally, the data processing module is further used to:

[0067] displaying the prediction results;

[0068] Receiving a revision operation on the prediction result, and updating the prediction result according to the revision operation; the revision operation includes adding and / or deleting.

[0069] Optionally, the data processing module is specifically used to:

[0070] In the actual driving trajectory data, second abnormal data corresponding to a preset type of data packet is identified;

[0071] Obtaining the target lidar perception result in the preset type data packet, and obtaining the vehicle trajectory data based on the preset regulation and control model according to the target lidar perception result;

[0072] When there is no overlap between the trajectory of each target included in the target lidar perception result and the own vehicle trajectory, the second abnormal data is replaced by the own vehicle trajectory data.

[0073] Optionally, the lidar perception result and the simulation perception result are both generated based on the rear axle system of the autonomous driving vehicle, and the perfect prediction, the imperfect prediction, the perfect planning, and the imperfect planning are all generated based on the power-on system of the autonomous driving vehicle; the device further includes:

[0074] A matrix acquisition module is used to obtain a transformation matrix;

[0075] A coordinate conversion module is used to convert the lidar perception result and the simulation perception result into the startup system according to the transformation matrix.

[0076] In a third aspect, an embodiment of the present application provides a computer device, including: a memory and a processor, wherein the memory and the processor are coupled;

[0077] The memory is used to store one or more computer instructions;

[0078] The processor is used to execute the one or more computer instructions to implement the model evaluation method as described in the first aspect.

[0079] In a fourth aspect, an embodiment of the present invention provides a computer-readable storage medium having one or more computer instructions stored thereon, which are executed by a processor to implement the model evaluation method as described in the first aspect above.

[0080] In a fifth aspect, the present application provides a computer program product, which includes a computer program, and when the computer program is executed by a processor, it implements the model evaluation method described in the first aspect.

[0081] From the above content, it can be seen that the model evaluation method, device, computer equipment and readable storage medium provided in the embodiments of the present application can, after acquiring the original data, use the perception model to be evaluated to perceive based on the video data in the original data to obtain a simulated perception result, that is, the position information of each target at each time can be identified, and then the self-vehicle planned path can be obtained by using the regulation and control model based on the simulated perception result, and then the lidar perception result, the simulated perception result, the actual driving trajectory data, and the self-vehicle planned path data are compared to determine the evaluation result of the perception model to be evaluated; during the actual driving process of the autonomous driving vehicle, the lidar perception result is accurate, and the actual driving trajectory data of the vehicle is the expert trajectory, which is also accurate. Therefore, by comparing the simulated perception result of the perception model with the accurate data, it is possible to accurately determine whether there is an error in the perception result of the perception model, thereby ensuring the accuracy of the perception model evaluation. BRIEF DESCRIPTION OF THE DRAWINGS

[0082] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the prior art descriptions. Obviously, the drawings described below are only some embodiments of the present invention. For those of ordinary skill in the art, other drawings can also be obtained based on these drawings without creative work.

[0083] Figure 1 A schematic diagram of a flow chart of a model evaluation method provided by an embodiment of the present invention;

[0084] Figure 2 Another schematic diagram of a flow chart of a model evaluation method provided by an embodiment of the present invention;

[0085] Figure 3 Another schematic diagram of a flow chart of a model evaluation method provided by an embodiment of the present invention;

[0086] Figure 4 A schematic diagram of a vehicle coordinate system provided by an embodiment of the present invention;

[0087] Figure 5 A schematic diagram of the structure of a model evaluation device provided by an embodiment of the present invention;

[0088] Figure 6 A schematic diagram of the structure of a computer device provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0089] The following will be combined with the accompanying drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0090] It should be noted that the terms "including" and "having" and any variations thereof in the embodiments of the present invention and the accompanying drawings are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device comprising a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to these processes, methods, products or devices.

[0091] The embodiment of the present application discloses a model evaluation method, device, computer equipment and readable storage medium, which can accurately evaluate the perception model. The embodiment of the present application is described in detail below.

[0092] Figure 1 Schematic diagram of the model evaluation process of the embodiment of the present application. Figure 1 As shown, the original data packet stores the data collected by all sensors during the driving process of the autonomous vehicle, as well as the data produced by the perception module, regulation and control module, etc., and also includes data such as the actual driving trajectory of the autonomous vehicle. Based on the data collected by the sensors in the original data packet, full module playback can generate a new data packet. The new data packet can be unpacked and the data converted to obtain various types of data to be evaluated. In other words, the perception model to be evaluated can be used to identify the target of the original data, and the position information of each target at each time and the trajectory of each target can be obtained, which are respectively used as imperfect perception and imperfect prediction. Based on the imperfect perception and imperfect prediction results, the vehicle trajectory predicted by the regulation and control module is used as an imperfect plan.

[0093] Directly unpacking the data packet can obtain various types of true value data. Specifically, the perception result of the lidar perception model is used as the perfect perception, the corresponding trajectory of each target is used as the perfect prediction, and the actual driving trajectory of the vehicle is used as the perfect planning.

[0094] After obtaining the data to be evaluated and the true value data, the corresponding evaluation results can be obtained by comparing the data of the same type. The corresponding indicators may include: MPI (Mileage Per Issue) or MPD (Mileage Per Disengagement). MPI units can include miles or kilometers, and problems can include abnormal performance such as misbraking and abnormal lane changes that do not affect the experience. MPD units can also include miles or kilometers to measure how many kilometers are taken over once.

[0095] Figure 2 A flow chart of a model evaluation method according to an embodiment of the present application is shown in FIG. The method is applied to electronic equipment and specifically includes the following steps.

[0096] S210: Obtaining raw data; the raw data includes at least: video data collected by a camera during the actual driving of the autonomous driving vehicle on the test section, lidar perception results during the actual driving of the autonomous driving vehicle on the test section, and actual driving trajectory data of the autonomous driving vehicle on the test section.

[0097] For example, various types of data generated during the driving of an autonomous vehicle can be stored in a fixed storage location. When model evaluation is required, the electronic device can obtain the required original data from the fixed storage location.

[0098] S220: Based on the video data, a simulated perception result is obtained using the perception model to be evaluated; the simulated perception result at least includes: position information of each target at each moment.

[0099] Specifically, the video data can be input into the perception model to be evaluated, and the position information of each target at each moment can be obtained as the simulated perception result.

[0100] S230: Based on the simulation perception result, a preset regulation and control model is used to obtain the vehicle planning path data.

[0101] It is understandable that the trajectories of surrounding targets (other vehicles, pedestrians, etc.) will affect the path planning results of the vehicle. In the embodiment of the present application, after the simulated perception results are obtained based on the video data using the perception model to be evaluated, the simulated perception results can be input into the regulation model, that is, the trajectories of the surrounding targets are input into the regulation model, and the regulation model can output the planned path data of the vehicle under the influence of the targets.

[0102] S240: Compare the lidar perception result, the simulation perception result, the actual driving trajectory data, and the self-vehicle planned path data to determine the evaluation result of the perception model to be evaluated.

[0103] The lidar perception result is the accurate target perception result, and the simulated perception result is the perception result output by the model to be evaluated. By comparing the two, we can determine whether the perception results of each target are consistent, and then determine whether the perception results of the perception model to be evaluated are accurate. For example, we can compare the position of each target at each time. When the position of each target at each time is consistent within a certain error range, we can determine that the accuracy of the perception result of the perception model to be evaluated meets the requirements.

[0104] The actual driving trajectory data is the expert route of the autonomous driving vehicle, that is, the route driven by the manual driver. The actual driving route can be considered to be the optimal route. The self-driving vehicle planned path is the self-driving vehicle path output by the control model under the influence of the simulated perception results output by the perception model to be evaluated. By comparing the two, the gap between the self-driving vehicle planned path and the actual driving trajectory can be determined, and then the accuracy of the self-driving vehicle planned path can be determined, which can also reflect whether the perception results of the perception model to be evaluated are accurate. For example, each position point in the actual driving trajectory and the closest point in the self-driving vehicle planned path can be used as a position point pair. When the distance between the two positions included in each position point pair is less than the preset value, it can be determined that the actual driving trajectory is consistent with the self-driving vehicle planned path, that is, the accuracy of the perception results of the perception model to be evaluated meets the requirements.

[0105] When determining the evaluation results of the perception model to be evaluated, it is possible to determine that the evaluation results of the perception model to be evaluated meet the requirements only when it is determined based on the lidar perception results and the simulation perception results that the accuracy of the perception model to be evaluated meets the requirements, or only when it is determined based on the actual driving trajectory data and the self-vehicle planned path data that the accuracy of the perception model to be evaluated meets the requirements; or, it is possible to determine that the evaluation results of the perception model to be evaluated meet the requirements only when it is determined based on the lidar perception results and the simulation perception results that the accuracy of the perception model to be evaluated meets the requirements, and when it is determined based on the actual driving trajectory data and the self-vehicle planned path data that the accuracy of the perception model to be evaluated meets the requirements. Both are possible, and the embodiments of the present application are not limited to this.

[0106] The model evaluation method provided in the embodiment of the present application can, after acquiring the original data, use the perception model to be evaluated to perform perception based on the video data in the original data to obtain a simulated perception result, that is, the position information of each target at each moment can be identified, and then the self-vehicle planned path can be obtained by using the regulation and control model based on the simulated perception result, and then the lidar perception result, the simulated perception result, the actual driving trajectory data, and the self-vehicle planned path data are compared to determine the evaluation result of the perception model to be evaluated; during the actual driving process of the autonomous driving vehicle, the lidar perception result is accurate, and the actual driving trajectory data of the vehicle is the expert trajectory, which is also accurate. Therefore, by comparing the simulated perception result of the perception model with the accurate data, it is possible to accurately determine whether there is an error in the perception result of the perception model, thereby ensuring the accuracy of the perception model evaluation.

[0107] In one implementation, Figure 3 As shown, in the above step S240, the lidar perception result, the simulation perception result, the actual driving trajectory data, and the vehicle planning path data are compared to determine the evaluation result of the perception model to be evaluated, which may specifically include:

[0108] S310: taking the trajectory composed of the positions of each target in the lidar perception result within the preset time period as a perfect prediction; taking the trajectory composed of the positions of each target in the simulation perception result within the preset time period as an imperfect prediction.

[0109] In other words, the lidar perception results are used as absolutely accurate true value data, and the simulated perception results are the data that need to be evaluated.

[0110] S320: The actual driving trajectory is regarded as a perfect plan, and the planned path of the vehicle is regarded as an imperfect plan.

[0111] S330: Determine whether the perception model to be evaluated has any false detection based on the perfect plan and the imperfect prediction; and / or determine whether the perception model to be evaluated has any missed detection based on the imperfect plan, the perfect prediction and the imperfect prediction.

[0112] Specifically, when it is determined that the perfect plan and the imperfect prediction have overlapping points, it can be determined that the perception model to be evaluated has false positives. The perfect plan is the actual driving trajectory, which is unlikely to collide with any target. Therefore, when the perfect plan and the imperfect prediction have overlapping points, it can be shown that there is an error in the imperfect prediction, specifically a false positive, that is, the target position is incorrectly positioned, and the position output by the imperfect prediction is not the actual position of the target.

[0113] When it is determined that there are overlapping points between the imperfect plan and the perfect prediction, and when there are no overlapping points between the imperfect plan and the imperfect prediction, it can be determined that there are missed detections in the perception model to be evaluated. There are overlapping points between the imperfect plan and the perfect prediction, and when there are no overlapping points between the imperfect plan and the imperfect prediction, that is, the planned route of the ego vehicle collides with the target in the lidar perception result, but the planned route of the ego vehicle does not collide with the target in the simulated perception result. Therefore, it can be determined that the target is lost in the simulated perception result, and the target that originally existed is not detected, which means that there are missed detections in the perception model to be evaluated.

[0114] When there is a false detection, the corresponding indicator may include MPI, which means there is a problem mileage; when there is a missed detection, the corresponding indicator may include MPD, which means that takeover is required. In addition, after processing a large number of data packets, the comprehensive situation of the data packets can be measured by calculating MPI / MPD.

[0115] Assuming that N_g of the Good Events failed and N_b of the Bad Events failed, the overall problem will be converted into MPI / MPD = (M_b*M_g) / (M_b*N_g+M_g*N_b). Among them, N_g is the number of Good Events num_good, N_b is the number of Bad Events num_bad, M_b is the total mileage of bad events Mile_bad, and M_g is the total mileage of good events Mile_good. The inverse of this formula is the number of good events with problems per kilometer + the number of bad events with problems per kilometer.

[0116] In this embodiment, it is possible to determine whether the perception model to be evaluated has false detections based on perfect planning and imperfect predictions, and to determine whether the perception model to be evaluated has missed detections based on imperfect planning, perfect predictions, and imperfect predictions. This allows the problem of the perception model to be evaluated to be accurately located, thereby improving the accuracy of model evaluation.

[0117] It is understandable that due to sensor failure or algorithm abnormality, the raw data obtained by the electronic device may contain abnormal data, such as missing data, or data with other range abnormalities, etc. Data abnormalities will affect the model evaluation results, especially when there are abnormalities in the lidar perception results and the actual driving trajectory data, which will directly lead to errors in the model evaluation results.

[0118] As an implementation method of an embodiment of the present application, in order to ensure the accuracy of model evaluation, the electronic device compares the lidar perception results and the simulated perception results, as well as the actual driving trajectory data and the vehicle planned path data. Before determining the evaluation results of the perception model to be evaluated, the first abnormal data in the lidar perception results can be first identified and the first abnormal data can be corrected; and / or, the second abnormal data in the actual driving trajectory data can be identified and the second abnormal data can be corrected.

[0119] Before model evaluation, abnormal data should be identified and corrected to ensure that accurate data is used to evaluate the model, thereby improving the accuracy of model evaluation.

[0120] In one implementation, there may be insufficient perception results in the lidar perception results, that is, some perception results are lost. In this case, non-continuous location information can be identified as the first abnormal data in the lidar perception results, and the first abnormal data can be displayed; the input manual annotation results are received, and the manual annotation results are added to the lidar perception results. In other words, the missing data can be supplemented by manual annotation. Specifically, the evaluator can mark the location of the target on the picture, and the electronic device recognizes the evaluator's annotation results and adds the annotation results to the lidar perception results.

[0121] It can be understood that when the vehicle speed is fast, such as greater than 70km / h or 80km / h, it is necessary to sense targets at a longer distance to meet planning requirements. In other words, data with a relatively small lidar sensing distance will not meet the model evaluation requirements. Therefore, in an embodiment of the present application, the first abnormal data corresponding to a vehicle speed greater than or equal to a preset speed threshold can be identified in the lidar sensing result; and in the first abnormal data, the data corresponding to a lidar sensing distance less than or equal to the preset distance threshold can be deleted.

[0122] During the driving process of an autonomous driving vehicle, some targets may have abnormal behaviors due to interaction with the vehicle, which will also cause abnormal lidar perception results. Therefore, in an embodiment of the present application, the first abnormal data corresponding to the abnormal behavior can be identified in the lidar perception results; and the historical data within a preset time period before the first abnormal data is obtained, and a prediction result is generated based on the preset prediction model according to the historical data; and the first abnormal data is replaced with the prediction result. In addition, before the prediction result is used to replace the first abnormal data, the prediction result can also be displayed; a revision operation on the prediction result is received, and the prediction result is updated according to the revision operation; wherein the above-mentioned revision operation includes addition and / or deletion. In other words, in the case of abnormal data generated by abnormal behavior, a historical perfect perception input prediction model can be used to generate a prediction, and further manual annotation and screening can be used to generate a "perfect prediction" to replace the abnormal lidar perception results.

[0123] In the original data obtained by the electronic device, there may be a situation where a Bad Event is recorded in the state of automatic driving. The Bad Event is based on the imperfect perception and imperfect prediction of the vehicle at that time. The regulation and control model will take actions such as braking avoidance. In this case, the actual driving trajectory, that is, the perfect planning, does not meet the evaluation requirements. Therefore, in the embodiment of the present application, when the data of the actual driving trajectory is corrected, the second abnormal data corresponding to the preset type data packet can be identified in the actual driving trajectory data; the target lidar perception result in the preset type data packet is obtained, and the vehicle trajectory data is obtained based on the preset regulation and control model according to the target lidar perception result; when the trajectory of each target included in the target lidar perception result and the vehicle trajectory do not have overlapping points, the vehicle trajectory data is used to replace the second abnormal data. In other words, for BadEvent, expert trajectory generation, that is, perfect planning, requires the use of perfect perception and perfect prediction re-run regulation and control modules. When the given planning trajectory does not collide with perfect perception and perfect prediction, a "perfect plan" is forged.

[0124] In the embodiment of the present application, the lidar perception results and the simulation perception results are generated based on the rear axle system of the autonomous driving vehicle, and the perfect prediction, imperfect prediction, perfect planning, and imperfect planning are all generated based on the power-on system of the autonomous driving vehicle, and data from different coordinate systems cannot be directly compared.

[0125] As an implementation method of an embodiment of the present application, the electronic device takes the trajectory composed of the various positions of each target within a preset time period in the lidar perception result as a perfect prediction; before taking the trajectory composed of the various positions of each target within a preset time period in the simulation perception result as an imperfect prediction, it can also obtain a transformation matrix, and convert the lidar perception result and the simulation perception result into the power-on system according to the transformation matrix.

[0126] like Figure 4 As shown, the rear axle system of the vehicle, that is, the vehicle coordinate system, has its origin at the center of the rear axle of the vehicle, the x-axis points to the right side of the vehicle, the y-axis points to the front of the vehicle, and the z-axis is perpendicular to the xoy plane and upward. The startup system is an imaginary coordinate system with the vehicle system at the beginning of recording the first frame of the data packet as the origin.

[0127] When recording data packets, the pose of the ego-vehicle at each timestamp can be obtained, namely egomotion. The pose consists of a quaternion representing rotation and a position representing position (based on the power-on system). The length, width and height of the ego-vehicle can also be obtained. In order to overcome the inconsistency between the egomotion timestamp and the sensor timestamp, the nearest neighbor sampling method can be used for fitting so that the egomotion and perception result timestamps can match. In other words, egomotion can be used as a transformation matrix (the transformation matrix of the ego-vehicle relative to the power-on system) to convert the description of the perception result in the ego-vehicle system to the description in the power-on system.

[0128] In this embodiment, by converting the data coordinate system, that is, converting the lidar perception results and the simulation perception results into the power-on system, it can be ensured that the data to be compared are in the same coordinate system, thereby ensuring the accuracy of the model evaluation.

[0129] like Figure 5 As shown, it shows a schematic diagram of the structure of a model evaluation device provided in an embodiment of the present application, the device comprising:

[0130] The data acquisition module 510 is used to acquire raw data; the raw data at least includes: video data collected by a camera during the actual driving of the autonomous driving vehicle on the test section, lidar perception results of the autonomous driving vehicle during the actual driving of the test section, and actual driving trajectory data of the autonomous driving vehicle on the test section;

[0131] The data perception module 520 is used to obtain a simulated perception result based on the video data using the perception model to be evaluated; the simulated perception result at least includes: the position information of each target at each time;

[0132] A path planning module 530 is used to obtain the vehicle's planned path data based on the simulated perception result using a preset regulation model;

[0133] The model evaluation module 540 is used to compare the lidar perception result, the simulation perception result, the actual driving trajectory data, and the self-vehicle planned path data to determine the evaluation result of the perception model to be evaluated.

[0134] Optionally, the model evaluation module 540 includes:

[0135] The first determination submodule is used to take the trajectory composed of the positions of each target in the lidar perception result within the preset time period as a perfect prediction; and take the trajectory composed of the positions of each target in the simulation perception result within the preset time period as an imperfect prediction;

[0136] A second determination submodule is used to take the actual driving trajectory as a perfect plan and the self-vehicle planned path as an imperfect plan;

[0137] The model evaluation submodule is used to determine whether the perception model to be evaluated has any false detection based on the perfect plan and the imperfect prediction; and / or to determine whether the perception model to be evaluated has any missed detection based on the imperfect plan, the perfect prediction and the imperfect prediction.

[0138] Optionally, the model evaluation submodule is specifically used to:

[0139] When it is determined that the perfect plan and the imperfect prediction have overlapping points, it is determined that the perception model to be evaluated has false detection;

[0140] When it is determined that there are overlapping points between the imperfect plan and the perfect prediction, and when it is determined that there are no overlapping points between the imperfect plan and the imperfect prediction, it is determined that there is a missed detection in the perception model to be evaluated.

[0141] Optionally, the device also includes: a data processing module, used to identify first abnormal data in the lidar perception result and correct the first abnormal data; and / or, identify second abnormal data in the actual driving trajectory data and correct the second abnormal data.

[0142] Optionally, the data processing module is specifically used to:

[0143] In the lidar sensing result, identifying discontinuous position information as first abnormal data, and displaying the first abnormal data;

[0144] The input manual annotation result is received, and the manual annotation result is added to the lidar perception result.

[0145] Optionally, the data processing module is specifically used to:

[0146] In the lidar sensing result, identifying first abnormal data corresponding to a vehicle speed greater than or equal to a preset speed threshold;

[0147] In the first abnormal data, the data corresponding to the lidar sensing distance being less than or equal to the preset distance threshold is deleted.

[0148] Optionally, the data processing module is specifically used to:

[0149] In the lidar sensing result, identifying first abnormal data corresponding to the abnormal behavior;

[0150] Acquire historical data within a preset time period before the first abnormal data, and generate a prediction result based on a preset prediction model according to the historical data;

[0151] The first abnormal data is replaced by the prediction result.

[0152] Optionally, the data processing module is further used to:

[0153] displaying the prediction results;

[0154] Receiving a revision operation on the prediction result, and updating the prediction result according to the revision operation; the revision operation includes adding and / or deleting.

[0155] Optionally, the data processing module is specifically used to:

[0156] In the actual driving trajectory data, second abnormal data corresponding to a preset type of data packet is identified;

[0157] Obtaining the target lidar perception result in the preset type data packet, and obtaining the vehicle trajectory data based on the preset regulation and control model according to the target lidar perception result;

[0158] When there is no overlap between the trajectory of each target included in the target lidar perception result and the own vehicle trajectory, the second abnormal data is replaced by the own vehicle trajectory data.

[0159] Optionally, the lidar perception result and the simulation perception result are both generated based on the rear axle system of the autonomous driving vehicle, and the perfect prediction, the imperfect prediction, the perfect planning, and the imperfect planning are all generated based on the power-on system of the autonomous driving vehicle; the device further includes:

[0160] A matrix acquisition module is used to obtain a transformation matrix;

[0161] A coordinate conversion module is used to convert the lidar perception result and the simulation perception result into the startup system according to the transformation matrix.

[0162] From the above content, it can be seen that the model evaluation device provided in the embodiment of the present application can, after acquiring the original data, use the perception model to be evaluated to perform perception based on the video data in the original data to obtain a simulated perception result, that is, it can identify the position information of each target at each moment, and then use the regulation and control model based on the simulated perception result to obtain the planned path of the self-vehicle, and then compare the lidar perception result, the simulated perception result, the actual driving trajectory data, and the planned path data of the self-vehicle to determine the evaluation result of the perception model to be evaluated; in the actual driving process of the autonomous driving vehicle, the lidar perception result is accurate, and the actual driving trajectory data of the vehicle is the expert trajectory, which is also accurate. Therefore, by comparing the simulated perception result of the perception model with the accurate data, it is possible to accurately determine whether there is any error in the perception result of the perception model, thereby ensuring the accuracy of the perception model evaluation.

[0163] The above device embodiment corresponds to the method embodiment and has the same technical effect as the method embodiment. For specific description, please refer to the method embodiment. The device embodiment is obtained based on the method embodiment. For specific description, please refer to the method embodiment part, which will not be repeated here.

[0164] Next, a computer device provided by an embodiment of the present application is introduced. Figure 6 , Figure 6 A schematic diagram of a computer device provided in an embodiment of the present application, the computer device comprising:

[0165] one or more processors 40;

[0166] The processor 40 is coupled to a storage device 41, and the storage device 41 is used to store one or more programs.

[0167] When the one or more programs are executed by the one or more processors 40, the electronic device implements the following Figures 1 to 3 A technical solution for the model evaluation method.

[0168] The present application also provides a computer-readable storage medium on which a computer program is stored. When the program is executed by a processor, the following is achieved: Figures 1 to 3 The model evaluation method described.

[0169] The present application provides a computer program product, the computer program product comprising a computer program, the computer program when executed by a processor to achieve the following Figures 1 to 3 The model evaluation method described.

[0170] Those skilled in the art can understand that the accompanying drawings are only schematic diagrams of an embodiment, and the modules or processes in the accompanying drawings are not necessarily required to implement the present invention.

[0171] Those skilled in the art can understand that the modules in the device in the embodiment can be distributed in the device in the embodiment according to the description of the embodiment, or can be changed accordingly and located in one or more devices different from the embodiment. The modules in the above embodiment can be combined into one module, or can be further divided into multiple sub-modules.

[0172] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A model evaluation method, characterized in that: The method comprises: Acquire raw data; the raw data includes at least: video data collected by a camera during the actual driving of the autonomous driving vehicle on the test section, lidar perception results of the autonomous driving vehicle during the actual driving of the test section, and actual driving trajectory data of the autonomous driving vehicle on the test section; Based on the video data, a simulated perception result is obtained by using the perception model to be evaluated; the simulated perception result at least includes: position information of each target at each moment; Based on the simulation perception results, a preset regulation and control model is used to obtain the planned path data of the vehicle; The lidar perception result, the simulated perception result, the actual driving trajectory data, and the self-vehicle planned path data are compared to determine the evaluation result of the perception model to be evaluated.

2. The method according to claim 1, characterized in that The step of comparing the lidar perception result, the simulation perception result, the actual driving trajectory data, and the self-vehicle planned path data to determine the evaluation result of the perception model to be evaluated includes: The trajectory composed of the positions of each target in the lidar perception result within the preset time period is taken as a perfect prediction; the trajectory composed of the positions of each target in the simulation perception result within the preset time period is taken as an imperfect prediction; The actual driving trajectory is regarded as a perfect plan, and the planned path of the vehicle is regarded as an imperfect plan; Determining whether the perception model to be evaluated has false detection based on the perfect plan and the imperfect prediction; and / or It is determined whether the perception model to be evaluated has any missed detections based on the imperfect planning, the perfect prediction, and the imperfect prediction.

3. The method according to claim 2, characterized in that The step of determining whether the perception model to be evaluated has a false detection according to the perfect planning and the imperfect prediction includes: When it is determined that the perfect plan and the imperfect prediction have overlapping points, it is determined that the perception model to be evaluated has false detection; The step of determining whether the perception model to be evaluated has missed detection according to the imperfect planning, the perfect prediction and the imperfect prediction comprises: When it is determined that there are overlapping points between the imperfect plan and the perfect prediction, and when it is determined that there are no overlapping points between the imperfect plan and the imperfect prediction, it is determined that there is a missed detection in the perception model to be evaluated.

4. The method according to claim 2, characterized in that: Before comparing the lidar perception result and the simulated perception result, as well as the actual driving trajectory data and the self-vehicle planned path data, and determining the evaluation result of the perception model to be evaluated, the method further includes: Identifying first abnormal data in the lidar sensing result and correcting the first abnormal data; and / or Identify second abnormal data in the actual driving trajectory data, and correct the second abnormal data.

5. The method according to claim 4, characterized in that The step of identifying the first abnormal data in the lidar sensing result and correcting the first abnormal data includes: In the lidar sensing result, identifying discontinuous position information as first abnormal data, and displaying the first abnormal data; The input manual annotation result is received, and the manual annotation result is added to the lidar perception result.

6. The method according to claim 4, characterized in that The step of identifying the first abnormal data in the lidar sensing result and correcting the first abnormal data includes: In the lidar sensing result, identifying first abnormal data corresponding to a vehicle speed greater than or equal to a preset speed threshold; In the first abnormal data, the data corresponding to the lidar sensing distance being less than or equal to the preset distance threshold is deleted.

7. The method according to claim 4, characterized in that The step of identifying the first abnormal data in the lidar sensing result and correcting the first abnormal data includes: In the lidar sensing result, identifying first abnormal data corresponding to the abnormal behavior; Acquire historical data within a preset time period before the first abnormal data, and generate a prediction result based on a preset prediction model according to the historical data; The first abnormal data is replaced by the prediction result.

8. The method according to claim 7, characterized in that Before using the prediction result to replace the first abnormal data, the method further includes: displaying the prediction results; Receiving a revision operation on the prediction result, and updating the prediction result according to the revision operation; the revision operation includes adding and / or deleting.

9. The method according to claim 4, characterized in that The step of identifying the second abnormal data in the actual driving trajectory data and correcting the second abnormal data comprises: In the actual driving trajectory data, second abnormal data corresponding to a preset type of data packet is identified; Obtaining the target lidar perception result in the preset type data packet, and obtaining the vehicle trajectory data based on the preset regulation and control model according to the target lidar perception result; When there is no overlap between the trajectory of each target included in the target lidar perception result and the own vehicle trajectory, the second abnormal data is replaced by the own vehicle trajectory data.

10. The method according to any one of claims 2 to 9, characterized in that: The lidar perception result and the simulation perception result are both generated based on the rear axle system of the autonomous driving vehicle, and the perfect prediction, the imperfect prediction, the perfect planning, and the imperfect planning are all generated based on the power-on system of the autonomous driving vehicle; The trajectory composed of the positions of each target in the lidar perception result within a preset time period is used as a perfect prediction; Before taking the trajectory composed of the positions of each target in the simulated perception result within the preset time period as an imperfect prediction, the method further includes: Get the transformation matrix; According to the transformation matrix, the lidar sensing result and the simulation sensing result are converted into the power-on system.

11. A model evaluation device, characterized in that: The device comprises: A data acquisition module is used to acquire raw data; the raw data includes at least: video data collected by a camera during the actual driving of the autonomous driving vehicle on the test section, lidar perception results of the autonomous driving vehicle during the actual driving of the test section, and actual driving trajectory data of the autonomous driving vehicle on the test section; A data perception module, used to obtain a simulated perception result based on the video data using a perception model to be evaluated; the simulated perception result at least includes: position information of each target at each time; A path planning module, used to obtain the self-vehicle planned path data based on the simulated perception results using a preset regulation and control model; The model evaluation module is used to compare the lidar perception result, the simulated perception result, the actual driving trajectory data, and the self-vehicle planned path data to determine the evaluation result of the perception model to be evaluated.

12. The device according to claim 11, characterized in that The model evaluation module includes: The first determination submodule is used to take the trajectory composed of the positions of each target in the lidar perception result within the preset time period as a perfect prediction; and take the trajectory composed of the positions of each target in the simulation perception result within the preset time period as an imperfect prediction; A second determination submodule is used to take the actual driving trajectory as a perfect plan and the self-vehicle planned path as an imperfect plan; The model evaluation submodule is used to determine whether the perception model to be evaluated has any false detection based on the perfect plan and the imperfect prediction; and / or to determine whether the perception model to be evaluated has any missed detection based on the imperfect plan, the perfect prediction and the imperfect prediction.

13. The device according to claim 12, characterized in that The model evaluation submodule is specifically used for: When it is determined that the perfect plan and the imperfect prediction have overlapping points, it is determined that the perception model to be evaluated has false detection; When it is determined that there are overlapping points between the imperfect plan and the perfect prediction, and when it is determined that there are no overlapping points between the imperfect plan and the imperfect prediction, it is determined that there is a missed detection in the perception model to be evaluated.

14. The device according to claim 12, characterized in that The device also includes: a data processing module, which is used to identify first abnormal data in the lidar perception result and correct the first abnormal data; and / or, identify second abnormal data in the actual driving trajectory data and correct the second abnormal data.

15. The device according to claim 14, characterized in that The data processing module is specifically used for: In the lidar sensing result, identifying discontinuous position information as first abnormal data, and displaying the first abnormal data; The input manual annotation result is received, and the manual annotation result is added to the lidar perception result.

16. The device according to claim 14, characterized in that The data processing module is specifically used for: In the lidar sensing result, identifying first abnormal data corresponding to a vehicle speed greater than or equal to a preset speed threshold; In the first abnormal data, the data corresponding to the lidar sensing distance being less than or equal to the preset distance threshold is deleted.

17. The device according to claim 14, characterized in that The data processing module is specifically used for: In the lidar sensing result, identifying first abnormal data corresponding to the abnormal behavior; Acquire historical data within a preset time period before the first abnormal data, and generate a prediction result based on a preset prediction model according to the historical data; The first abnormal data is replaced by the prediction result.

18. The device according to claim 17, characterized in that The data processing module is also used for: displaying the prediction results; Receiving a revision operation on the prediction result, and updating the prediction result according to the revision operation; the revision operation includes adding and / or deleting.

19. The device according to claim 14, characterized in that The data processing module is specifically used for: In the actual driving trajectory data, second abnormal data corresponding to a preset type of data packet is identified; Obtaining the target lidar perception result in the preset type data packet, and obtaining the vehicle trajectory data based on the preset regulation and control model according to the target lidar perception result; When there is no overlap between the trajectory of each target included in the target lidar perception result and the own vehicle trajectory, the second abnormal data is replaced by the own vehicle trajectory data.

20. The device according to any one of claims 12 to 19, characterized in that The lidar perception result and the simulation perception result are both generated based on the rear axle system of the autonomous driving vehicle, and the perfect prediction, the imperfect prediction, the perfect planning, and the imperfect planning are all generated based on the power-on system of the autonomous driving vehicle; The device also includes: A matrix acquisition module is used to obtain a transformation matrix; A coordinate conversion module is used to convert the lidar perception result and the simulation perception result into the startup system according to the transformation matrix.

21. A computer device, characterized in that: include: A memory and a processor, wherein the memory and the processor are coupled; The memory is used to store one or more computer instructions; The processor is used to execute the one or more computer instructions to implement the model evaluation method as described in any one of claims 1 to 10.

22. A readable storage medium having one or more computer instructions stored thereon, characterized in that: The instruction is executed by the processor to implement the model evaluation method as described in any one of claims 1 to 10.

Images