Real vehicle testing method for vehicle automatic driving system based on artificial intelligence model
By collecting user portraits and setting target testers, combining testing of closed sites and open roads, the problem of inability to effectively test high-level autonomous driving vehicles in the existing technology is solved, and dynamic evaluation and safety of AI model performance is achieved Guarantee, providing reliable evaluation results.
Patent Information
- Application Number
- CN202410998865.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-24
- Publication Date
- 2024-11-08
AI Technical Summary
It is difficult for the existing technology to effectively test and evaluate the performance of high-level autonomous driving vehicles based on artificial intelligence models. Traditional testing methods assume that decision-making strategies are fixed and cannot adapt to the dynamic changes of the AI model, resulting in the lack of testing methods and the difficulty in ensuring safety and reliability.
By collecting user portraits, selecting target testers, setting vehicle training mileage and conducting road training, combining automatic driving tests on closed sites and open roads, and combining subjective and objective evaluation, a dynamic testing framework is built to simulate a real usage environment and evaluate the performance of the AI model. change.
It realizes objective and subjective combination testing of high-level autonomous driving systems, simulates the real usage environment, and the evaluation results are more in line with actual performance, providing reliable and quantifiable evaluation references, ensuring the safety and reliability of autonomous driving vehicles.
Smart Images

Figure CN118913711A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of autonomous driving technology, and in particular to a real vehicle testing method for a vehicle autonomous driving system based on an artificial intelligence model. Background Art
[0002] The driving automation level of intelligent connected vehicles is carefully divided into six levels, from level 0 to level 5, in accordance with my country's national standard "Automobile Driving Automation Classification", to evaluate the driving automation level of vehicles through different levels. Among them, levels 0 to 2 cover driving assistance systems, which mainly focus on providing driver assistance, including but not limited to automatic emergency braking, lane keeping assistance and other functions, while levels 3 to 5 indicate that the vehicle has different degrees of autonomous driving capabilities, among which level 3 is conditional autonomous driving, level 4 is highly autonomous driving, and level 5 is fully autonomous driving, where the vehicle can independently complete all driving tasks without human intervention.
[0003] The mainstream products on the market are currently concentrated in the driving assistance categories of Level 0, Level 1 and Level 2. These systems rely on preset rules and fixed algorithms to assist drivers. However, with the evolution of technology, higher-level autonomous driving products are being developed and tested. They widely use advanced artificial intelligence technologies such as neural networks, deep learning and reinforcement learning (such as AI algorithm models), which give vehicles unprecedented learning and adaptability.
[0004] The core advantage of the AI algorithm model lies in its self-learning and adaptive features, which means that autonomous vehicles can autonomously adjust the parameters in their decision-making models based on dynamic factors such as the actual driving environment, driver habits, and vehicle status. In the early days of vehicle delivery, due to the uniformity of AI model parameters, vehicles of the same model will have similar performance under the same conditions. However, as mileage accumulates and the scenarios that vehicles are exposed to become increasingly diverse, the AI model will continue to optimize its own parameters through actual operating data, resulting in even vehicles of the same model exhibiting different behavior patterns in the same scenario.
[0005] This change poses a new challenge to the testing methods of intelligent driving systems. Traditional testing techniques usually assume that the decision-making strategies of intelligent driving products are solidified when they leave the factory and will not change over time and environment. However, for AI-based autonomous driving products, this assumption is no longer valid. The decision-making strategies of autonomous driving vehicles are dynamic and will be adjusted according to the actual operation of the vehicle. This requires that the testing methods also need to evolve to adapt to the continuous evolution of AI models and ensure the safety and reliability of autonomous driving vehicles. In addition, the current autonomous driving products based on advanced technologies such as artificial intelligence large models and cloud computing are still in the research process, and the products have not yet reached the mass production stage. Therefore, vehicle manufacturers, equipment suppliers, testing institutions, etc., when faced with autonomous driving vehicles based on AI models, how to conduct vehicle testing, there are few related solutions, and there is a research gap in the evaluation scheme for high-level autonomous driving vehicles. Summary of the invention
[0006] The present invention aims to provide a real vehicle testing method for a vehicle automatic driving system based on an artificial intelligence model, which can be adapted to high-level automatic driving system testing and objectively evaluate the real vehicle performance of the automatic driving system equipped with an AI model.
[0007] The basic solution provided by the present invention is: a real vehicle testing method of a vehicle automatic driving system based on an artificial intelligence model, comprising the following steps:
[0008] Step 1: Collect the user profile of the vehicle under test, and select the target tester for the vehicle under test based on the user profile;
[0009] Step 2, setting the vehicle training mileage L1 and training the vehicle under test; the target tester drives the vehicle under test to complete the first section of road training, and the autonomous driving system drives the vehicle under test to complete the second section of road training; the mileage of the first section of road training and the second section of road training are both greater than or equal to L1 / 2;
[0010] Step 3: Use the trained vehicle to conduct autonomous driving tests in closed areas and open roads;
[0011] Step 4: After all target testers have completed the test, the test phase ends;
[0012] Step 5, collect the subjective scores of the target testers on the tested vehicle; and calculate the objective score of the tested vehicle based on the test data of the tested vehicle; perform weighted summation of the subjective score and the objective score and convert them proportionally into the final evaluation score to complete the evaluation of the tested vehicle.
[0013] The working principle and advantages of the present invention are:
[0014] The present invention provides a real vehicle testing method for an automatic driving system of a vehicle based on an artificial intelligence model, which can be adapted to high-level automatic driving system testing and objectively evaluate the actual vehicle performance of the automatic driving system with an AI model. This solution can effectively solve the problem of the lack of a real vehicle testing method for an automatic driving system based on an AI model in the prior art.
[0015] In particular, this solution can conduct an objective and subjective test on the autonomous driving system from the perspective of normal vehicle use by users. Compared with the existing single objective or subjective test methods, and the test methods involving only test engineers, this solution can construct driver inputs with different driving styles for the tested vehicle through the setting of target testers and the setting of training steps (corresponding to step 2), so that the tested vehicle can be equivalent to a real application environment, that is, it is equivalent to importing different driving habits, vehicle usage scenarios, and expectations of vehicle behavior performance of different drivers and passengers into the AI model of the vehicle's autonomous driving system, so that the AI model is in a real self-learning environment. The overall test framework built can simulate the dynamic changes of AI model parameters, and on the basis of the changes, it combines different evaluation environments (closed venues, open roads), combines subjective and objective evaluations, and conducts a real and effective evaluation of autonomous driving performance. The test results are more in line with the actual performance of the vehicle, and the evaluation results are more valuable for product design reference. BRIEF DESCRIPTION OF THE DRAWINGS
[0016] Figure 1 It is a structural schematic diagram of an embodiment of a real vehicle testing method for a vehicle automatic driving system based on an artificial intelligence model of the present invention;
[0017] Figure 2 It is a structural schematic diagram of an embodiment of a real vehicle testing method for a vehicle automatic driving system based on an artificial intelligence model of the present invention. DETAILED DESCRIPTION
[0018] The following is a further detailed description through specific implementation methods:
[0019] The embodiment is basically as shown in the attached Figure 1 As shown: A real vehicle testing method for a vehicle automatic driving system based on an artificial intelligence model, comprising the following steps:
[0020] Step 1: Collect the user profile of the vehicle under test, and select the target tester for the vehicle under test based on the user profile.
[0021] In this step, the target testers are set to K, and the value of K is positively correlated with the user portrait type. The user portrait type classification indicators include gender, occupation, age stage, and driving style. This setting can ensure that the vehicle state after subsequent training is close to the actual application state of this type of vehicle.
[0022] Specifically, the user portrait refers to the target user (customer) group portrait of the same model of the tested vehicle. In specific applications, this part of the data can be extracted from the historical sales data or historical user database of the model of the vehicle or the information of the intended sales object. In this embodiment, a plurality of drivers and passengers of different genders, different occupations, different age groups, and different driving styles (aggressive, conservative) are selected as target testers to participate in the test.
[0023] Step 2, set the vehicle training mileage L1 and train the vehicle under test; the target tester drives the vehicle under test to complete the first section of road training, and the autonomous driving system drives the vehicle under test to complete the second section of road training; the mileage of the first section of road training and the second section of road training are both greater than or equal to L1 / 2.
[0024] In this embodiment, the vehicle under test is a high-level intelligent driving vehicle with an automated driving level of 3-5 and an AI model in the autonomous driving system.
[0025] In addition, in the first road training, the vehicle under test was in manual driving mode and completed the training within the ODD range of the autonomous driving system; in the second road training, the vehicle under test was in autonomous driving mode and the autonomous driving system completed the training. ODD (Operational Design Domain) is the abbreviation of the operational design domain of the autonomous driving system, which defines the prerequisites and scope of application of the autonomous driving system.
[0026] Step 3: Use the trained vehicle to conduct K rounds of autonomous driving tests in closed areas and open roads.
[0027] When conducting autonomous driving tests in a closed field, K rounds of tests are conducted, and the autonomous driving performance of the tested vehicle is also verified; and the autonomous driving performance verification results are collected as the closed field verification score A; among them, if the autonomous driving performance verification result of the i-th round is passed, one point is scored, that is, a i-1 =1; if the result of the i-th round of autonomous driving performance verification is not passed, zero score will be given, i.e. a i-1 = 0. If all K rounds of autonomous driving performance verification in the closed field are passed, the road verification phase will get full marks, recorded as A = K; if all K rounds fail, A = 0.
[0028] After completing vehicle training, target testers need to evaluate the safety of the AI model in the vehicle's autonomous driving system. This part of the autonomous driving test needs to cover three dimensions: longitudinal and lateral safety test driving scenarios and automatic parking scenarios for autonomous driving vehicles, that is, to verify the lateral autonomous driving performance, longitudinal autonomous driving performance and automatic parking performance of the vehicle's autonomous driving system AI model. In specific applications, the design of this part of the test scenario can refer to national standards, industry standards or enterprise-designed standards. Through autonomous driving tests in closed venues, it can be effectively verified whether the vehicle's active safety functions and performance are attenuated after the AI model training is completed, thereby causing unsafe problems.
[0029] When conducting autonomous driving tests on open roads, the open training mileage is set to L2, and K rounds of testing are performed; the calculation of L2 is performed under the functional conditions of the autonomous driving system of the tested vehicle and is within the ODD range of the autonomous driving system. The value of L2 is 200 to 500km. In specific applications, L2 can be flexibly set within the range.
[0030] At this stage, road test equipment is used for testing. In this embodiment, conventional intelligent driving system real vehicle testing equipment (e.g., i-Tester) is selected for testing. After completing the test on the open road, the test data such as the vehicle data, traffic participant motion state data, test road environment data, etc. in the test are copied to the data analysis computer, and the test data is analyzed using the road test data analysis software. During the test data analysis process, the focus is on analyzing the number of road adaptability problems such as the automatic driving system actively reminding manual takeover, the automatic driving system false alarm or missed alarm, and malfunction during driving. The statistical indicators of each type of road adaptability problem (manual takeover, false alarm, etc.) are calculated based on a total score of 100 points, and 1 point is deducted for each occurrence (e.g., 1 malfunction).
[0031] Step 4: After all target testers have completed the test (i.e., after all K target testers have participated in and completed the test), the test phase ends.
[0032] Step 5, collect the subjective scores of the target testers on the tested vehicle; and calculate the objective score of the tested vehicle based on the test data of the tested vehicle; perform weighted summation of the subjective score and the objective score and convert them proportionally into the final evaluation score to complete the evaluation of the tested vehicle.
[0033] For automated driving tests conducted on open roads, the objective score of the vehicle under test is calculated using the following steps:
[0034] Based on the test data of the vehicle under test, the number of times the autonomous driving system exhibits behaviors corresponding to objective evaluation indicators during the vehicle's driving process is extracted; wherein the objective evaluation indicators refer to road adaptability issues such as the autonomous driving system actively reminding manual takeover, the autonomous driving system false alarms or missed alarms, and malfunctions.
[0035] The following formula is used to calculate the objective score of the i-th target tester in the i-th round of testing;
[0036]
[0037] Among them, f(n) is the scoring formula of the nth evaluation index, x is the number of objective evaluation indicators in the road test, and y n represents the number of occurrences of the nth objective evaluation index, c i It indicates the objective score of the i-th target tester during the i-th round of testing.
[0038] For the autonomous driving test conducted on the open road, in the active scoring, the target testers participating in the test will subjectively score the tested vehicle from the dimensions of comfort, safety, convenience, intelligence, and sense of technology during the test (i.e. the first level); and a scoring evaluation table is designed for each subjective evaluation indicator; the full score of each subjective evaluation indicator or dimension of the first level is set to 100 points. The subjective score is calculated using the following formula:
[0039]
[0040] Where g(n) is the subjective evaluation index (first level) score of the nth road test (the full score is 100), y is the number of subjective evaluation indexes (first level) in the road test, and b i Describes the subjective score of the i-th driver or passenger during the road test.
[0041] The final evaluation score is calculated using the following formula:
[0042]
[0043] S0=m×A+n×B; d i =b i +c i ;
[0044] Among them, S is the final evaluation score, which is a percentage score; S0 is the total evaluation score; A is the closed site verification score, and B is the open road adaptability score; d i Subjective rating b i and objective score c iThe sum; m and n are weight coefficients, and m + n = 2; K is the number of target testers and is equal to the total number of rounds of autonomous driving tests.
[0045] If A < K or S < 60, the evaluation result of the autonomous driving system of the vehicle under test is unqualified; if 60 ≤ S < 70, the evaluation result of the autonomous driving system of the vehicle under test is qualified; if 70 ≤ S < 80, the evaluation result of the autonomous driving system of the vehicle under test is medium; if 80 ≤ S < 90, the evaluation result of the autonomous driving system of the vehicle under test is good; if S ≥ 90, the evaluation result of the autonomous driving system of the vehicle under test is excellent, as Figure 2 shown.
[0046] A real vehicle test method for a vehicle autonomous driving system based on an artificial intelligence model provided by this embodiment can be adapted to the test of high-level autonomous driving systems, objectively evaluate the real vehicle performance of the autonomous driving system equipped with an AI model; and provides a six-grade evaluation scheme for the evaluation results, which can provide a reliable and quantifiable evaluation reference for the autonomous driving system test.
[0047] The above are only embodiments of the present invention. Specific structures and common knowledge such as characteristics well known in the art are not described in detail here. Those of ordinary skill in the art know all the common technical knowledge in the technical field to which the invention belongs before the application date or the priority date, can know all the existing technologies in this field, and have the ability to apply the conventional experimental means before this date. Those of ordinary skill in the art can, under the inspiration given in this application, complete and implement this solution in combination with their own abilities. Some typical well-known structures or well-known methods should not become an obstacle for those of ordinary skill in the art to implement this application. It should be noted that for those skilled in the art, without departing from the structure of the present invention, several deformations and improvements can still be made, and these should also be regarded as the protection scope of the present invention, and these will not affect the implementation effect of the present invention and the practicality of the patent.
Claims
1. A real vehicle testing method for a vehicle automatic driving system based on an artificial intelligence model, characterized in that: It includes the following steps: Step 1: Collect the user profile of the vehicle to be tested, and based on the user profile, select the target test personnel for the vehicle to be tested; Step 2: Set the vehicle training mileage L1 and train the vehicle to be tested; The target test personnel drive the vehicle to be tested to complete the first-stage road training, and the autonomous driving system drives the vehicle to be tested to complete the second-stage road training; the driving mileage of both the first-stage road training and the second-stage road training is greater than or equal to L1 / 2; Step 3: Use the trained vehicle to conduct autonomous driving tests in a closed site and an open road respectively; Step 4: After all target test personnel have completed the tests, the test phase ends; Step 5: Collect the subjective scores of the target test personnel for the vehicle to be tested; and based on the test data of the vehicle to be tested, calculate the objective score of the vehicle to be tested; perform weighted summation on the subjective score and the objective score and convert it into the final evaluation score according to a ratio to complete the evaluation of the vehicle to be tested.
2. The method for testing a vehicle automatic driving system based on an artificial intelligence model according to claim 1, characterized in that: In Step 1, there are K target test personnel, and the value of K is positively correlated with the type of user profile.
3. The method for testing a vehicle automatic driving system based on an artificial intelligence model according to claim 2, characterized in that: The classification indicators of the type of user profile include gender, occupation, age group, and driving style.
4. The method for testing a vehicle automatic driving system based on an artificial intelligence model according to claim 1, characterized in that: When conducting autonomous driving tests on an open road, set the open training mileage as L2; the calculation of L2 is carried out under the functional conditions of the autonomous driving system of the vehicle to be tested and within the ODD range of the autonomous driving system.
5. The method for testing a vehicle automatic driving system based on an artificial intelligence model according to claim 4, characterized in that: The value of L2 is 200 - 500 km.
6. The method for testing a vehicle automatic driving system based on an artificial intelligence model according to claim 1, characterized in that: For the autonomous driving test conducted on an open road, when calculating the objective score of the vehicle to be tested, it includes the following steps: Based on the test data of the vehicle to be tested, extract the number of times the autonomous driving system exhibits the corresponding behaviors of the objective evaluation indicators during vehicle driving; Use the following formula to calculate the objective score when the i-th target test personnel conducts the i-th round of test; Among them, f(n) is the scoring formula of the nth evaluation index, x is the number of objective evaluation indicators in the road test, and y n represents the number of occurrences of the nth objective evaluation index, c i It indicates the objective score of the i-th target tester during the i-th round of testing.
7. The method for testing a vehicle automatic driving system based on an artificial intelligence model according to claim 1, characterized in that: For the autonomous driving test conducted on an open road, the subjective score is calculated using the following formula: Among them, g(n) is the subjective evaluation index score of the nth road test, y is the number of subjective evaluation indicators in the road test, and b i Describes the subjective score of the i-th driver or passenger during the road test.
8. The method for testing a vehicle automatic driving system based on an artificial intelligence model according to claim 2, characterized in that: When conducting autonomous driving tests in a closed field, the autonomous driving performance of the tested vehicle is also verified; and the autonomous driving performance verification results are collected as the closed field verification score A; among them, if the autonomous driving performance verification result of the i-th round is passed, one point is scored, that is, a i-1 =1; if the result of the i-th round of autonomous driving performance verification is not passed, zero score will be given, i.e. a i-1 =0.
9. The method for testing a vehicle automatic driving system based on an artificial intelligence model according to claim 8, characterized in that: The final evaluation score is calculated using the following formula: Among them, S is the final evaluation score, which is a percentage score; S0 is the total evaluation score; A is the closed site verification score, and B is the open road adaptability score; d i Subjective rating b i and objective score c i The sum of m and n; m and n are weight coefficients, and m+n=2; K is the number of target testers and is equal to the total number of rounds of autonomous driving tests.
10. A real vehicle testing method for a vehicle automatic driving system based on an artificial intelligence model according to claim 9, characterized in that: If A < K or S < 60, the evaluation result of the autonomous driving system of the vehicle to be tested is unqualified; if 60 ≤ S < 70, the evaluation result of the autonomous driving system of the vehicle to be tested is qualified; if 70 ≤ S < 80, the evaluation result of the autonomous driving system of the vehicle to be tested is medium; if 80 ≤ S < 90, the evaluation result of the autonomous driving system of the vehicle to be tested is good; if S ≥ 90, the evaluation result of the autonomous driving system of the vehicle to be tested is excellent.
Citation Information
Cited By
Calculation method for evaluating end-to-end intelligent driving safety generalization capability
CN120121316A
Automatic parameter adjusting method and system for automatic driving system
CN121069994A
