Automatic driving NOA third-party test and evaluation system

Abstract

The invention relates to the technical field of automatic driving evaluation systems, and discloses an automatic driving NOA third-party test and evaluation system, which comprises the following steps: SS01, a scene construction module for constructing an automatic driving scene in which a specified weather scene, a road scene and a driving scene are preset, SS02, a feature recognition and labeling module, and SS04, a third-party test and evaluation module for performing third-party test and evaluation after the step SS01, a worker carries out automatic marking of feature points and key points on a specified weather scene, a road scene and a driving scene, and after marking is completed, a feature data set in an automatic driving scene is generated. According to the invention, when the third party of the automatic driving NOA is tested and evaluated; according to four dimensions of driving result information of automatic driving, driving acquisition and driving perception information in an automatic driving process, humanized evaluation information of a safety officer and driving simulation stability information of automatic driving, the safety degree, the flow degree, the convenience and the precision of automatic driving can be comprehensively evaluated.

Description

Technical Field

[0001] The present invention relates to the technical field of autonomous driving evaluation systems, and more specifically, to an autonomous driving NOA third-party testing and evaluation system. Background Art

[0002] Driving behavior is an important factor that must be considered for both human-driven vehicles and self-driving vehicles. Most traffic accidents are caused by improper driving operations, and bad driving habits are the main cause of traffic accidents. Self-driving vehicles help to improve the intelligence of the "human-vehicle-environment" system, thereby improving traffic efficiency and traffic safety. With the development of self-driving technology, self-driving vehicles are gradually being promoted in closed scenarios, freeing human drivers from driving operations. However, there is still no reliable and effective evaluation method for how to objectively and accurately evaluate the driving behavior of self-driving vehicles so that self-driving vehicles can absorb the correct driving methods of human drivers, abandon the bad driving habits of human drivers, and make the driving behavior of self-driving vehicles as safe and effective as excellent drivers.

[0003] In the prior art, a patent document with publication number CN114692713A discloses a driving behavior evaluation method and device for an autonomous driving vehicle, the method comprising the following steps: obtaining driving behavior information of the autonomous driving vehicle on the route to be evaluated, the driving behavior information comprising: instantaneous vehicle speed, acceleration / deceleration, steering wheel angle, total vehicle operation time, total vehicle operation mileage, vehicle cumulative energy consumption, vehicle surrounding obstacle information, vehicle surrounding road data information; the vehicle surrounding road data information comprises ramps, curves, tunnels, accident-prone areas, road section speed limits, and traffic light information. The above-mentioned driving behavior evaluation method can realize both quantitative evaluation of the driving behavior of the autonomous driving vehicle and qualitative evaluation of the driving behavior of the autonomous driving vehicle. The above-mentioned evaluation method mainly uses the autonomous driving result as the evaluation value, and does not realize multi-dimensional testing and evaluation of the autonomous driving NOA during the evaluation. Based on this, the present invention provides an autonomous driving NOA third-party testing and evaluation system to solve the technical problems raised in the above-mentioned background technology. Summary of the invention

[0004] In order to overcome the shortcomings of the prior art, the present invention provides an autonomous driving NOA third-party testing and evaluation system. When conducting the autonomous driving NOA third-party testing and evaluation, the present invention can conduct a comprehensive evaluation of the safety, fluidity, convenience and accuracy of autonomous driving based on four dimensions, namely, the driving result information of autonomous driving, the driving collection and driving perception information during the autonomous driving process, the humanized evaluation information of the safety officer, and the driving simulation stability information of the autonomous driving, thereby effectively improving the evaluation accuracy of the autonomous driving system and the credibility of the evaluation results.

[0005] To achieve the above object, the present invention provides the following technical solutions: An automatic driving NOA third-party testing and evaluation system, including the following steps:

[0006] SS01. A scenario construction module that constructs an automatic driving scenario. The automatic driving scenario is preset with a specified weather scenario, a road scenario, and a driving scenario;

[0007] SS02. A feature recognition and annotation module. After the SS01 step, the staff automatically annotates the feature points and key points of the specified weather scenario, road scenario, and driving scenario. After the annotation is completed, a feature data set within the automatic driving scenario is generated;

[0008] SS03. A data acquisition module that constructs a data acquisition module in the automatic driving vehicle. The data acquisition module acquires real-time driving data during the operation of the automatic driving vehicle and generates a driving data set;

[0009] SS04. A sample reference construction module that presets multiple drivers. The drivers should drive in this automatic driving scenario for the first time. After the preset, a driving plan is established. Multiple drivers, according to the driving plan, successively conduct driving simulations in the automatic driving scenario. After each driver conducts a driving simulation, a driving sample data set is generated;

[0010] SS05. An average data construction module that constructs an average data set based on the multiple driving sample data sets in the SS04 step. The average data set includes the average vehicle speed, average steering wheel angle, and central trajectory parameters of the vehicle operation under the specified driving time;

[0011] SS06. An evaluation sample construction module. The automatic driving vehicle conducts an automatic driving simulation according to the driving plan in the SS03 step. After the automatic driving simulation is completed, the data acquisition module obtains a data set to be evaluated. When the automatic driving behavior is in progress, the automatic driving vehicle is equipped with a safety officer;

[0012] SS07. A driving evaluation module that calculates the driving weight ∑1 based on the data set to be evaluated obtained in the SS04 step. After the automatic driving behavior is completed, the safety officer obtains a humanized evaluation weight value ∑2 according to the evaluation of the safety officer;

[0013] SS08. A perception evaluation module. When the automatic driving vehicle conducts an automatic driving behavior, the perception evaluation module analyzes the data to judge the driving feature information collected by the data acquisition module during the driving process. The perception evaluation module calculates the perception information evaluation weight value ∑3 and the travel feature collection accuracy weight ∑4 based on the obtained driving feature information;

[0014] SS09. Driving Simulation Weight Evaluation Module. After the autonomous vehicle performs an autonomous driving behavior, the data acquisition module places the collected driving data set into the computer driving simulation system for simulation reproduction. After each driving simulation by the computer driving simulation system, a simulated driving data set is obtained. After each simulated driving data set is obtained, the matching degree K between the simulated driving data set and the driving sample data set is automatically calculated. After multiple K values are calculated, the average matching degree KD, the extreme difference value KR, and the weight ∑5 of the autonomous driving system stability value are calculated;

[0015] SS10. Comprehensive Evaluation Module. The comprehensive evaluation module calculates the evaluation value ∑ of the autonomous driving NOA based on ∑1, ∑2, ∑3, ∑4, and ∑5;

[0016] ∑ = ∑1 × ∑2 × ∑3 × ∑4 × ∑5

[0017] After ∑ is calculated, it is automatically recorded and backed up.

[0018] As a preferred technical solution of the present invention, in the SS01 step, the road scene includes the specified road type, specified road markings, specified traffic lights, signboards, street lights, intersection type, number of lanes, lane lines, lane type, speed limit type, road type, road angle, the weather scene includes weather, light, road visibility, and environmental temperature and humidity, and the driving scene includes the traffic flow scene and the vehicle interference simulation scene.

[0019] As a preferred technical solution of the present invention, in the SS02 step, the feature data set includes the total number of feature points and key points, and the types of feature points and key points.

[0020] As a preferred technical solution of the present invention, in the SS03 step, the driving data set includes driving time, vehicle instantaneous speed, steering wheel angle, vehicle GPS position information, vehicle edge driving trajectory, vehicle radar information, autonomous driving background operation code, and driving image information.

[0021] As a preferred technical solution of the present invention, in the SS07 step, when calculating the driving weight ∑1, the data of each item in the data set to be evaluated in the SS06 step is used as the calculation data, and the average data set in the SS05 step is used as the parameter data to calculate the data difference X under each type of driving item. After all types of data differences X are calculated, the average difference MD and the extreme difference MR under the automatic driving item are calculated;

[0022]

[0023] When calculating the extreme difference value MR, the maximum data difference value Xmax and the minimum data difference value Xmin after comparing the data differences of each item are obtained through a numerical comparison algorithm;

[0024] MR = X max -X min

[0025]

[0026] After the calculation of ∑1, it is automatically recorded and backed up.

[0027] As a preferred technical solution of the present invention, in the SS08:

[0028]

[0029] When calculating ∑4, first calculate the matching degree between the driving characteristics collected by the data acquisition module and the characteristic data set. Each characteristic corresponds to a characteristic matching degree F. After the characteristic matching degrees FN of each characteristic are calculated, calculate the travel characteristic acquisition accuracy weight ∑4;

[0030]

[0031] After the calculation of ∑4, it is automatically recorded and backed up.

[0032] As a preferred technical solution of the present invention, in the SS09 step:

[0033]

[0034] When calculating the matching extreme difference KR, obtain the maximum matching data difference Kmax and the minimum matching data difference Kmin after comparing the data differences of each item through a numerical comparison algorithm;

[0035] KR = K max -K min

[0036]

[0037] After the calculation of ∑5, it is automatically recorded and backed up.

[0038] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0039] When the present invention conducts third-party testing and evaluation on autonomous driving NOA, it can comprehensively evaluate the safety, fluency, convenience, and accuracy of autonomous driving from four dimensions, namely, the driving result information of autonomous driving, the driving collection and driving perception information during the autonomous driving process, the humanized evaluation information of the safety driver, and the driving simulation stability information of autonomous driving, thereby effectively improving the evaluation accuracy of the autonomous driving system and the credibility of the evaluation results. Brief Description of the Drawings

[0040] Figure 1Schematic diagram of the process structure of the third-party testing and evaluation system for the NOA of autonomous driving of the present invention; Specific embodiments

[0041] Next, the technical solutions in the embodiments of the present invention will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present invention.

[0042] As Figure 1 shown, the present invention provides a third-party testing and evaluation system for the NOA of autonomous driving, including the following steps:

[0043] SS01. Scenario construction module, which constructs an autonomous driving scenario. A specified weather scenario, road scenario, and driving scenario are preset in the autonomous driving scenario;

[0044] In step SS01, the road scenario includes specified road types, road markings, traffic lights, signs, street lights, intersection types, number of lanes, lane lines, lane types, speed limit types, road types, and road angles. The weather scenario includes weather, lighting, road visibility, and environmental temperature and humidity. The driving scenario includes a traffic flow scenario and a vehicle interference simulation scenario;

[0045] SS02. Feature recognition and annotation module. After step SS01, the staff automatically annotates the feature points and key points of the specified weather scenario, road scenario, and driving scenario. After the annotation is completed, a feature data set within the autonomous driving scenario is generated;

[0046] In step SS02, the feature data set includes the total number of feature points and key points, and the types of feature points and key points;

[0047] The feature data set includes weather scenario features, road scenario features, and driving scenario features;

[0048] SS03. Data acquisition module. A data acquisition module is constructed in the autonomous driving vehicle. The data acquisition module acquires real-time driving data during the operation of the autonomous driving vehicle and generates a driving data set;

[0049] In step SS03, the driving data set includes driving time, vehicle instantaneous speed, steering wheel angle, vehicle GPS position information, vehicle edge driving trajectory, vehicle radar information, autonomous driving background operation code, and driving image information;

[0050] SS04, Sample Reference Construction Module, presets multiple drivers who should be driving in this autonomous driving scenario for the first time. After presetting, a driving plan is established. Multiple drivers drive in the autonomous driving scenario in sequence according to the driving plan. After each driver's driving simulation, a set of driving sample data is generated;

[0051] The driving sample data set includes driving time, vehicle instantaneous speed, steering wheel angle, vehicle GPS position information, vehicle edge driving trajectory, vehicle radar information, autonomous driving background operation code, and driving image information;

[0052] SS05, Average Data Construction Module, the average data construction module establishes an average data set based on the multiple driving sample data sets in step SS04. The average data set includes the average vehicle speed, average steering wheel angle, and central trajectory parameters of vehicle operation at a specified driving time;

[0053] SS06, Evaluation Sample Construction Module, the autonomous vehicle conducts an autonomous driving simulation according to the driving plan in step SS03. After the autonomous driving simulation is completed, the data acquisition module obtains a data set to be evaluated. When the autonomous driving behavior is in progress, the autonomous vehicle is equipped with a safety officer;

[0054] SS07, Driving Evaluation Module, calculates the driving weight ∑1 based on the data set to be evaluated obtained in step SS04. After the autonomous driving behavior is completed, the safety officer obtains a personalized evaluation weight value ∑2 according to the safety officer's evaluation;

[0055] In step SS07, when calculating the driving weight ∑1, the data of the data set to be evaluated in step SS06 is used as the calculation data, and the average data set in step SS05 is used as the parameter data to calculate the data difference X under various driving items. After calculating the data differences X of all types, calculate the average difference MD and the extreme difference MR under the automatic driving items;

[0056]

[0057] Under a certain specified condition, if X1 is 2, X2 is 4, X3 is 6, X4 is 8, and X5 is 10, then:

[0058]

[0059] When calculating the extreme difference MR, obtain the maximum data difference Xmax and the minimum data difference Xmin after comparing the sizes of the data differences of each item through a numerical comparison algorithm;

[0060] MR = X max -X min

[0061] When X1 = 2, X2 = 4, X3 = 6, X4 = 8, and X5 = 10:

[0062] MR = 10 - 2 = 8

[0063] When X1 = 2, X2 = 4, X3 = 6, X4 = 8, and X5 = 10:

[0064]

[0065] ∑1 = 0.75

[0066] After ∑1 is calculated, it is automatically recorded and backed up;

[0067] SS08, Perception Evaluation Module. When the autonomous vehicle performs autonomous driving behavior, the Perception Evaluation Module analyzes data to judge the driving feature information collected by the Data Acquisition Module during the driving process. The Perception Evaluation Module calculates the perception information evaluation weight value ∑3 and the travel feature acquisition accuracy weight ∑4 based on the obtained driving feature information;

[0068] In SS08:

[0069]

[0070] When calculating ∑4, first calculate the matching degree between the driving features collected by the Data Acquisition Module and the feature data set. Each feature corresponds to a feature matching degree F. After calculating the feature matching degrees FN of each feature, calculate the travel feature acquisition accuracy weight ∑4;

[0071]

[0072] After ∑4 is calculated, it is automatically recorded and backed up.

[0073] SS09, Driving Simulation Weight Evaluation Module. After the autonomous vehicle performs autonomous driving behavior, the Data Acquisition Module places the collected driving data set into the computer driving simulation system for simulation reproduction. After each driving simulation by the computer driving simulation system, a simulated driving data set is obtained. After each simulated driving data set is obtained, the matching degree K between the simulated driving data set and the driving sample data set is automatically calculated. After calculating multiple K values, calculate the matching degree mean KD, the matching extreme difference KR, and the autonomous driving system stability value weight ∑5;

[0074] In the steps of SS09:

[0075]

[0076] When calculating the matching extreme difference KR, obtain the maximum matching data difference Kmax and the minimum matching data difference Kmin after comparing the data differences of each item through the numerical comparison algorithm;

[0077] KR = K max -K min

[0078]

[0079] After the ∑5 calculation, automatic recording and backup are performed;

[0080] SS10, Comprehensive Evaluation Module. The Comprehensive Evaluation Module calculates the evaluation value ∑ of the autonomous driving NOA based on ∑1, ∑2, ∑3, ∑4, and ∑5.

[0081] ∑ = ∑1 × ∑2 × ∑3 × ∑4 × ∑5

[0082] After the ∑ calculation is completed, automatic recording and backup are performed.

[0083] It should be noted that in this article, relational terms such as first and second are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the term "comprising", "including" or any other variation thereof is intended to cover non-exclusive inclusion, so that a process, method, article or device comprising a series of elements not only includes those elements, but also includes other elements not expressly listed, or also includes elements inherent to such process, method, article or device.

[0084] Although the embodiments of the present invention have been shown and described, it will be understood by those of ordinary skill in the art that various changes, modifications, substitutions and variations can be made in these embodiments without departing from the principles and spirit of the present invention, and the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. An autonomous driving NOA third-party testing and evaluation system, characterized in that It includes the following steps: SS01. The scenario construction module constructs an autonomous driving scenario. A specified weather scenario, road scenario, and driving scenario are preset in the autonomous driving scenario; SS02. The feature recognition and annotation module. After step SS01, the staff automatically annotates the feature points and key points of the specified weather scenario, road scenario, and driving scenario. After the annotation is completed, a feature data set within the autonomous driving scenario is generated; SS03. The data collection module. A data collection module is constructed in the autonomous driving vehicle. The data collection module collects real-time driving data during the operation of the autonomous driving vehicle and generates a driving data set; SS04. The sample reference construction module. Preset multiple drivers. The drivers should be driving in this autonomous driving scenario for the first time. After the preset, a driving plan is established. Multiple drivers drive in the autonomous driving scenario in sequence according to the driving plan. After each driver's driving simulation, a driving sample data set is generated; SS05. The average data construction module. The average data construction module establishes an average data set based on the multiple driving sample data sets in step SS04. The average data set includes the average vehicle speed, average steering wheel angle, and central trajectory parameters of the vehicle operation at a specified driving time; SS06. The evaluation sample construction module. The autonomous driving vehicle conducts an autonomous driving simulation according to the driving plan in step SS03. After the autonomous driving simulation is completed, the data collection module obtains an evaluation data set to be evaluated. When the autonomous driving behavior is in progress, a safety officer is configured in the autonomous driving vehicle; SS07. The driving evaluation module. According to the evaluation data set to be evaluated obtained in step SS04, the driving weight ∑1 is calculated. After the autonomous driving behavior is completed, the safety officer obtains a humanized evaluation weight value ∑2 according to the evaluation of the safety officer; SS08. The perception evaluation module. When the autonomous driving vehicle conducts the autonomous driving behavior, the perception evaluation module analyzes the data to judge the driving feature information collected by the data collection module during the driving process. The perception evaluation module calculates the perception information evaluation weight value ∑3 and the travel feature collection accuracy weight ∑4 based on the obtained driving feature information; SS09. The driving simulation weight evaluation module. After the autonomous driving vehicle conducts the autonomous driving behavior, the data collection module places the collected driving data set into the computer driving simulation system for simulation reproduction. After each driving simulation by the computer driving simulation system, a simulated driving data set is obtained. After each simulated driving data set is obtained, the matching degree K between the simulated driving data set and the driving sample data set is automatically calculated. After multiple K values are calculated, the average matching degree KD, the matching extreme difference KR, and the autonomous driving system stability value weight ∑5 are calculated; SS10. The comprehensive evaluation module. The comprehensive evaluation module calculates the evaluation value ∑ of the autonomous driving NOA based on ∑1, ∑2, ∑3, ∑4, and ∑5; ∑＝∑1×∑2×∑3×∑4×∑5 After ∑ is calculated, it is automatically recorded and backed up.

2. The third-party testing and evaluation system for autonomous driving NOA according to claim 1, wherein: In the SS01 step, the road scenario includes specified road types, specified road markings, specified traffic lights, signposts, street lights, intersection types, number of lanes, lane lines, lane types, speed limit types, road types, and road angles. The weather scenario includes weather, lighting, road visibility, and ambient temperature and humidity. The driving scenario includes traffic flow scenarios and vehicle interference simulation scenarios.

3. The third-party testing and evaluation system for autonomous driving NOA according to claim 1, characterized in that: In the SS02 step, the feature dataset includes the total number of feature points and key points, and the types of feature points and key points.

4. The third-party testing and evaluation system for autonomous driving NOA according to claim 1, wherein: In the SS03 step, the driving dataset includes driving time, instantaneous vehicle speed, steering wheel angle, vehicle GPS location information, vehicle edge driving trajectory, vehicle radar information, autonomous driving background operation code, and driving image information.

5. The third-party testing and evaluation system for autonomous driving NOA according to claim 1, wherein: In the SS07 step, when calculating the driving weight ∑1, the data of the dataset to be evaluated in the SS06 step is used as the calculation data, and the average dataset in the SS05 step is used as the parameter data to calculate the data difference X under each type of driving project. After calculating the data differences X for all types, calculate the average difference MD and the extreme difference MR under the automatic driving project; When calculating the extreme difference MR, obtain the maximum data difference Xmax and the minimum data difference Xmin after comparing the sizes of the data differences for each project through a numerical comparison algorithm; MR = X max -X min After calculating ∑1, record and back up automatically.

6. The third-party testing and evaluation system for autonomous driving NOA according to claim 1, wherein: In the SS08: When calculating ∑4, first calculate the matching degree between the driving features collected by the data acquisition module and the feature dataset. Each feature corresponds to a feature matching degree F. After calculating the feature matching degrees FN for all features, calculate the travel feature acquisition accuracy weight ∑4; After calculating ∑4, record and back up automatically.

7. The third-party testing and evaluation system for autonomous driving NOA according to claim 1, characterized in that: In the SS09 step: When calculating the matching extreme difference KR, obtain the maximum matching data difference Kmax and the minimum matching data difference Kmin after comparing the sizes of the data differences for each project through a numerical comparison algorithm; KR = K max -K min After calculating ∑5, record and back up automatically.

Images