Test evaluation method for ADAS navigation intelligent driving system

By constructing a dynamically updated test scenario library and comprehensive evaluation method, the problems of scene authenticity and integrity in the test of navigation intelligent driving system are solved, and efficient and accurate system evaluation and comparative evaluation results are achieved in the closed test site.

CN119065970APending Publication Date: 2024-12-03INTELLIGENT CONNECTED TECH OF CAERI CO LTD +1
View PDF 0 Cites 4 Cited by

Patent Information

Application Number
CN202411101346.0
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-12
Publication Date
2024-12-03

AI Technical Summary

Technical Problem

The existing navigation intelligent driving system test cannot take into account the integrity of the test and the authenticity of the scene, resulting in the incompleteness of the test results being incomplete and accurate enough, and the different test scenarios of different manufacturers are different, resulting in the lack of comparison reference value for the evaluation results.

Method used

Build a test scenario library for ADAS navigation intelligent driving system, combine the optimization of the intelligent connected vehicle test site and the release of new cars, dynamically update the test scenarios to ensure the comprehensiveness and authenticity of the scene, and subjective evaluation and performance through human-computer interaction Express comprehensive evaluation of subjective and objective methods and form standardized testing methods.

Benefits of technology

It has realized the construction of comprehensive testing scenarios in a closed test site, reduced testing costs, improved the feasibility and reference value of test data, ensured the integrity and contrast of evaluation results, and adapted to the continuous update of system technology.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119065970A_ABST
    Figure CN119065970A_ABST
Patent Text Reader

Abstract

The invention relates to the technical field of vehicle-mounted system testing, and discloses a test evaluation method for an ADAS navigation intelligent driving system, and the method comprises the steps: constructing a test scene library for evaluating the intelligent driving capability of the navigation intelligent driving system; continuously monitoring preset conditions including intelligent network connection automobile test field optimization and new automobile release, and performing dynamic change of the test scene library in a preset mode to form a current test scene library; determining a test vehicle carrying a to-be-tested navigation intelligent driving system; according to the business scene of the test vehicle, determining the priority of a target test scene in a current test scene library according to a preset scene arrangement strategy; the test vehicle runs in the target test scene according to requirements, and evaluation is carried out based on the test data through a preset evaluation strategy including man-machine interaction subjective evaluation and performance expression subjective and objective comprehensive evaluation; according to the scheme, the test field test method which is dynamic and comprehensive in scene, variable in test priority and selectable in actual test is provided for the navigation intelligent driving system.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field of vehicle-mounted system testing, and in particular to a testing and evaluation method for an ADAS navigation intelligent driving system. Background Art

[0002] Most companies currently test the intelligent driving assistance system (ADAS) as a whole, and rarely test its subsystems individually. This is mainly because the ultimate goal of the ADAS system is to provide a safe and reliable driving assistance function. It is highly integrated and there are complex interdependencies between the various subsystems. Overall testing can more comprehensively evaluate the performance and safety of the ADAS system and ensure that the system can operate reliably in the actual driving environment. Therefore, it is generally more critical to evaluate the performance of the entire system rather than just focusing on the performance of a single subsystem.

[0003] However, in the actual development process, key subsystems, such as the rapidly developing navigation assistance system, are usually tested separately to ensure that each part meets the requirements, and then the overall test is carried out to verify the overall performance of the system.

[0004] For navigation assistance systems, existing separate tests such as Chinese patent CN115985096B provide a method for testing navigation driving systems in limited conventional scenarios on public roads to ensure the consistency and repeatability of test evaluations on public roads. However, the evaluation scenarios of this method are only some conventional scenarios provided on the basis of limited driving modes in limited intersection types. Although it can achieve accurate evaluation in a representative scenario on public roads to a certain extent, for navigation assistance systems with high requirements for safe operation, the test is still not comprehensive and complete, the scenario construction is not ideal, and it is impossible to take into account both the integrity of the test and the authenticity of the scenario. Although Chinese patent CN118277257A discloses a method for constructing a complete scene by developing a scene and comprehensively adopting a scene combing method and a scene expansion method to address the problem that there are too many types of traffic participants at urban intersections in the test of urban navigation assistance systems, resulting in complex scenes and unsatisfactory construction effects, this method takes into account a variety of scene types and constructs a relatively complete test scene, but the following problems may exist in the actual adoption process: with the technological update of the system, different manufacturers select different test scenes, resulting in too many different factors in the entire test process. From the consumer's perspective, the evaluation results of systems from different manufacturers have no comparative reference value; in addition, although many test scenes are constructed, the limitations of existing testable conditions will result in limited selectable test scenes. Perfect construction does not mean that it can be fully and comprehensively applied in the actual test process, and it is still impossible to take into account both the integrity of the test and the authenticity of the scene. Summary of the invention

[0005] The present invention aims to provide a test and evaluation method for an ADAS navigation intelligent driving system, so as to solve the technical problem that the existing navigation intelligent system test cannot take into account both test integrity and scene authenticity.

[0006] The basic solution provided by the present invention is: a test and evaluation method for an ADAS navigation intelligent driving system, based on the characteristics of the navigation intelligent driving system, to construct a test scenario library for evaluating the intelligent driving capability of the navigation intelligent driving system;

[0007] Continuously monitor preset conditions, dynamically change the test scenario library in a preset manner, and form the current test scenario library; preset conditions include intelligent connected vehicle test site optimization and new vehicle release;

[0008] Determine the test vehicle equipped with the navigation intelligent driving system to be tested; determine the target test scenario in the current test scenario library according to the business scenario of the test vehicle and the preset scenario arrangement strategy;

[0009] The test vehicle drives as required in the target test scenario to obtain test data;

[0010] Evaluation is performed based on the test data using a preset evaluation strategy to obtain an evaluation result; the preset evaluation strategy includes a subjective evaluation of human-computer interaction and a subjective and objective comprehensive evaluation of performance.

[0011] The working principle and advantages of the present invention are as follows: In the existing testing methods for automatic driving systems including navigation intelligent driving systems, there is a general technical bias that the error of the test results in the test field is greater than the error of the test results on the public road. Therefore, the test is generally carried out on the public road, and the public road background is used as the optimization focus of the test method. Although the public road test has a certain authenticity in the environment, the test cost is high, the test data volume is large, the processing process is complicated, the test scenario is limited, and it is difficult to form a comprehensive and standard test method.

[0012] In the face of existing challenges in testing autonomous driving systems, this solution overcomes the technical bias that public road test results are more accurate. It is believed that based on the current reality of the continuous improvement of the construction of intelligent connected vehicle test sites, it is actually possible to ensure that the test scenarios are constructed realistically enough, the test scenarios are selected comprehensively enough, and the test errors of the test sites can be controlled within a reasonable range, which can make the test site test better than the public road test. In addition, the comprehensiveness of the test scenario should not be a certain limited comprehensive state, because the system technology is constantly developing, and the test conditions and test technologies in various aspects are also constantly updated. It should be regarded as a dynamic and comprehensive state of change. The test method under the dynamic and comprehensive test scenario is more applicable to adapt to the constantly updated and developing technology.

[0013] This solution first builds a relatively comprehensive test scenario library for the navigation intelligent driving system under the testable conditions of the existing closed test site, which can fully simulate the normal traffic environment, traffic participants, and traffic flow, and can be normally built in the closed site to ensure the feasibility of the test site test; on this basis, it monitors the optimization of the intelligent connected vehicle test site and the emergence of new vehicles in real time, judges the construction status and adds new preset business scenarios, and dynamically increases the test scenarios accordingly, so that the test scenarios used for the system are in a dynamic and comprehensive state, and can maintain the authenticity of the test scenarios in the test site; during specific testing, different business scenarios are tested, and the actual testable items are selected according to different business scenarios, and the test priority is changed. On the basis of the test site being sufficiently real, the test error can be controlled within an acceptable range; from the subjective evaluation of human-computer interaction and the subjective and objective comprehensive evaluation of performance, both driving experience and objectivity are taken into account, making the system's intelligent driving ability evaluation more complete and the results more referenceable.

[0014] It can be said that this scheme proposes a test site test method for navigation intelligent driving systems, which has dynamic and comprehensive scenarios, variable test priorities, and selectable actual tests. Within the limited service life and usage scenarios of the vehicle, and based on monitoring the existing available intelligent connected vehicle test sites, the constructed test scenarios are comprehensive and realistic enough, with low testing costs and small amounts of data. It is possible to form a feasible and standardized test method in the test site with strong operability, and can reproduce certain typical scenarios multiple times to find problems, thereby improving the integrity of the test scenarios and the ability to reproduce scenarios. It can not only accurately evaluate the navigation intelligent driving system, but also make the evaluation results have comparative reference value. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 A flow chart of a test and evaluation method for an ADAS navigation intelligent driving system provided by an embodiment of the present invention;

[0016] Figure 2 An evaluation flow chart of a test and evaluation method for an ADAS navigation intelligent driving system provided by an embodiment of the present invention;

[0017] Figure 3 A schematic diagram of acceleration threshold requirements in objective performance evaluation provided by an embodiment of the present invention;

[0018] Figure 4 A schematic diagram of the deceleration threshold requirement in the objective performance evaluation provided by an embodiment of the present invention;

[0019] Figure 5 A schematic diagram of the threshold requirement for the average rate of change of deceleration in the objective evaluation of performance provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0020] The following is a further detailed description through specific implementation methods:

[0021] The embodiment is basically as shown in the attached Figure 1 As shown: A test and evaluation method for an ADAS navigation intelligent driving system comprises the following steps:

[0022] S100, based on the characteristics of the navigation intelligent driving system, builds a test scenario library for evaluating the intelligent driving capabilities of the navigation intelligent driving system.

[0023] Specifically, based on the current construction status of closed test sites in China, and taking into account the road types of existing application scenarios of navigation intelligent driving systems in highways, cities, commuting, etc., as well as the performance application requirements of the navigation intelligent driving system itself, a series of test scenarios are obtained as shown in Table 1.

[0024] The test scenario library is a collection of multi-level test scenarios, including several main scenarios and at least one layer of sub-scenario under each main scenario; several main scenarios include intelligent vehicle speed control longitudinal, intelligent vehicle following test longitudinal, intelligent lane keeping lateral, lane and lane line type change passability test, autonomous lane change lateral, entry and exit ramps, U-turn traffic, intersection traffic, driving in impassable areas, traffic accident areas, areas where communication signals are weakened or blocked, traffic efficiency, traffic sign recognition and response, motor vehicle signal light recognition and response, and direction indicator signal light recognition and response.

[0025] Table 1 Test scenarios in the test scenario library

[0026]

[0027]

[0028]

[0029]

[0030]

[0031] Under the testable conditions of the existing closed test site, the series of test scenarios in Table 1 can fully simulate the normal traffic environment, traffic participants, and traffic flow, and can be normally built in the closed site to ensure the feasibility of the test site test; plus the test conditions directly hit the pain points of the navigation intelligent driving system, subsequent tests will be carried out around important functions such as vehicle lateral and longitudinal adaptive control, autonomous on and off ramps, function inhibition, degradation, human-computer interaction, etc., which can specifically discover problems in the system and give a comprehensive evaluation of the entire system, helping the test subject unit and the company to find loopholes in their own products and give suggestions for improvement, thus providing effective support for assisted driving and even autonomous driving.

[0032] S200 continuously monitors preset situations and dynamically changes the test scenario library in a preset manner to form the current test scenario library; the preset situations include the optimization of the intelligent connected vehicle test site and the release of new vehicles.

[0033] Specifically, the preset situation is the optimization of the intelligent connected vehicle test site, and the dynamic change of the test scenario library in a preset manner is to capture the construction status of all intelligent connected vehicle test sites, match the test scenarios in the test scenario library, and change the test scenarios according to the preset change strategy based on the matching status of the test scenarios.

[0034] The default situation is the release of a new car. The dynamic changes of the test scenario library in a preset manner are to capture the characteristics of the new car and its navigation intelligent driving system and the estimated business scenarios, match the test scenarios in the current test scenario library, and change the test scenarios according to the preset change strategy based on the matching of the test scenarios. Since the main function of the constructed scenario is to test the passability of the vehicle, the vehicle system characteristics involved, such as the vehicle's own lateral and longitudinal control capabilities, recognition and response capabilities, and vehicle performance stability in complex environments, the estimated business scenarios mainly involve the vehicle's operating environment.

[0035] Among them, crawler tools can be used to capture data. Python can be used to write crawler scripts, and the Scrapy framework, BeautifulSoup library, etc. can be used to crawl web content. For example, data related to the construction of intelligent connected vehicle test sites and new car releases can be captured from the homepages of various car companies, major media for mainstream test information releases in the automotive industry, and messages on automotive forums. The crawling frequency can be set, such as in units of days, weeks, or months, for 3 days, 2 weeks, or 1 month to ensure appropriate data flow and system computing power. When capturing a new car release preview, it can also be captured on the same day based on the specific release time to ensure timeliness of information acquisition.

[0036] Among them, the test scene is matched in the test scene library, and the test scene is changed according to the preset change strategy based on the matching situation of the test scene. In this embodiment, the test scene includes a main scene and a sub-scene. When matching, the main scene and the sub-scene can be matched in sequence. The preset change strategy is that if the test scene is matched, the test scene will not be changed; if it is not matched, the necessity of the test will continue to be judged, and according to the result of the test necessity judgment, the test scene will be changed, and the changes include addition, deletion or replacement; the captured content is marked with the matching situation and the change of the test scene.

[0037] Constructing a matching model to reflect the matching relationship between the data content captured under preset conditions (such as the construction status of the intelligent networked vehicle test site and the release of new cars) and the test scenes in the test scene library can be done by using training samples, i.e., the construction characteristics, vehicle characteristics, navigation intelligent driving system characteristics, and test requirement characteristics of the existing intelligent networked vehicle test site, and training an artificial intelligence model based on a deep learning algorithm to obtain a model with the ability to learn and match test scenes. Of course, technicians in the relevant field can also use other methods to achieve the conversion between the two, which does not affect the implementation of the present invention. The construction status of the intelligent networked vehicle test site or the release of new cars and the test scenes in the test scene library are input into the matching model, and the matching model reflects the matching relationship between the two with similarity. If the similarity is high, it is considered to be matched, and if the similarity is low, it is not matched.

[0038] A test necessity analysis model is constructed to reflect the necessary relationship between the data content captured under preset circumstances (such as the construction status of the intelligent connected vehicle test site and the release of new cars) and the characteristics of the navigation intelligent driving system. The model is trained in the same way as above to obtain a model with the necessary learning test capabilities. The construction status of the intelligent connected vehicle test site or the release of new cars, and the characteristics of the navigation intelligent driving system are input into the test necessity analysis model. The test necessity analysis model reflects the necessary relationship between the two in terms of importance. If the importance is high, it is considered that there is a need to change the test scenario and the test scenario is changed; if the importance is low, it is considered that there is no need to change the test scenario and the test scenario is not changed.

[0039] The optimized captured data content of the intelligent connected vehicle test field includes lane changes, regional scene changes and equipment changes. For example, a newly added special road is captured, with a wet circle radius of 60m, inner and outer safety zones of 3m each, and four lanes, namely basalt, polished concrete, low-adhesion asphalt and high-adhesion asphalt roads. Its main functions are used for various data on vehicle circular driving, stability testing and vehicle driving conditions under extreme conditions; special test environments are captured, and tunnel scenes can be used to test the perception, calculation, decision-making, execution and other functions of intelligent connected vehicles when there is no GPS positioning signal and the network signal is shielded, and meteorological scenes can simulate extreme meteorological conditions such as "wind, rain, thunder and lightning" to allow vehicles to be tested under extreme weather conditions; 32,000 square meters of laboratories and supporting test equipment are captured to achieve more accurate data collection and conduct corresponding tests with high-precision data.

[0040] For example, we can capture newly added special roads and their achievable tests, traverse the main scenes in the test scene library through keyword extraction and other technologies, calculate the similarity with all main scenes through the matching model to be 0, and determine that there is no main scene on this special road. Then, we use the test necessity analysis model to calculate the importance of the characteristics of the navigation intelligent driving system and the special road. If the importance is above 60%-70%, we will add this special road to the main scene, and generate corresponding sub-scenes based on the characteristics of the navigation intelligent driving system and the test requirements. The specific scene generation technology can be implemented using existing technologies; if the importance is below 70%-60%, we will retain this special road as a candidate for the main scene, and provide possible test scene expansion for subsequent system technology updates to adapt to the development of the system.

[0041] The captured data content of new car releases includes new car configurations and preset business scenarios. New car configurations include car models and navigation intelligent driving systems in new and old arrangements, such as new car models + new navigation intelligent driving systems, new car models + existing navigation intelligent driving systems, and existing car models + new navigation intelligent driving systems. Basically, all car models and systems are considered to be variable combinations. Compared with simply focusing on the update of the navigation intelligent driving system, because the final performance of the system is still based on the performance of the vehicle, the test scenario changes carried out in this combination can achieve a more comprehensive test effect; preset business scenarios include regional business scenarios, road business scenarios, and special business scenarios. For example, a new car model + a new system is about to be applied to the racing track and its specific characteristics, and the specific matching method is the same as above.

[0042] Through the above technical updates based on real-time monitoring of actual roads and vehicles, dynamic changes in test scenarios can be made, which can keep the test scenarios dynamic and comprehensive, and can maintain the uninterrupted applicability of system testing; at the same time, the test scenario library of this solution can be used as a unified and standardized test scenario library for the industry. Different car manufacturers can conduct scenario construction tests in any existing intelligent connected vehicle test site, and the test results have reference and comparison value.

[0043] S300, determining a test vehicle equipped with the navigation intelligent driving system to be tested; and determining the priority of a target test scenario in a current test scenario library according to a preset scenario arrangement strategy based on the business scenario of the test vehicle.

[0044] Specifically, the preset scenario arrangement strategy is to use the current test scenario library as a unified standard test scenario selection library. In the current test scenario library, the test scenarios that cannot be measured in the intelligent connected vehicle test site at the test location are the untestable scenarios at the location. For example, there is no "car accident area" at the test site at the test location, which reflects that the test vehicle and its system cannot be tested in the "car accident area" scenario at the test site; at the same time, the "car accident area" that cannot be tested is marked, so that when comparing the test results of different manufacturers, it can be clear based on the unified standard test scenario library which test scenarios of the system have been tested and which test scenarios have not been tested, thereby evaluating the comparative value of the test results.

[0045] Among the remaining test scenarios, the first target test scenario is selected according to the estimated business scenario of the test vehicle, and the second target test scenario different from the first target test scenario is selected according to the performance of the navigation intelligent driving system to be tested. The unselected test scenario is the third target test scenario; the first target test scenario, the second target test scenario and / or the third target test scenario are tested and sorted according to the preset sorting method. Among them, the preset sorting method includes calculating the switching paths of all test scenarios in the first target test scenario, the second target test scenario and the third target test scenario respectively according to the layout of the intelligent connected vehicle test site at the test location. The smaller the path, the priority is given to the test. It also includes selecting whether to add the test of the third target test scenario according to the importance of the test. If the test of the third target test scenario is not added, the switching path calculation is not performed and marking is performed at the same time.

[0046] S400, the test vehicle drives as required in the target test scene to obtain test data; based on the test data, the vehicle is evaluated using a preset evaluation strategy to obtain an evaluation result; the preset evaluation strategy includes subjective evaluation of human-computer interaction and subjective and objective comprehensive evaluation of performance, such as Figure 2 shown.

[0047] Specifically, for a vehicle model equipped with navigation intelligent driving, the i-tester (AVE2100) testing equipment is used for testing. The i-tester (AVE2100) field testing equipment can record the vehicle's front and rear perspectives and the driver's perspective video of the in-vehicle instruments during the entire test process, and collect the main vehicle's speed, acceleration, acceleration change rate, and relative position relationship with the target vehicle during the navigation intelligent driving test, the target vehicle's speed, acceleration, acceleration change rate and other commonly used GPS positioning information, as well as some bus signals required for test evaluation.

[0048] When the navigation intelligent driving system is turned on, the speed will be adaptively controlled according to the current road environment (speed limit on high-precision maps or speed limit signs). In this embodiment, the speed can also be set to: 80km / h on urban roads and 120km / h on expressways to ensure a balance between driving safety and test requirements.

[0049] The subjective evaluation of human-computer interaction is to build a subjective evaluation index system for human-computer interaction. When the test vehicle is static, multiple test engineers make subjective evaluations on the test vehicle's navigation intelligent driving system based on the subjective evaluation indicators of human-computer interaction in turn. Multiple subjective evaluations are combined to form a subjective evaluation result of human-computer interaction for the navigation intelligent driving system.

[0050] The subjective evaluation index system of human-computer interaction includes test indicators for the aesthetics of the interactive interface, test indicators for the convenience and comfort of the interactive interface, test indicators for visual, auditory, and tactile warning prompts, test indicators for voice interaction, test indicators for anti-interference, and test indicators for ease of operation, as shown in Table 2:

[0051] Table 2 Subjective evaluation indicators of human-computer interaction

[0052]

[0053]

[0054] For the navigation intelligent driving system to be tested (hereinafter referred to as the system), the test engineer uses the system to conduct corresponding evaluation. The specific evaluation process is as follows:

[0055] 1. Beautiful interactive interface

[0056] 1) Open each interface of the system, subjectively compare the style of the operation interaction interface with the style of the system interface, and evaluate whether the interaction interface style is simple, stylish, and luxurious.

[0057] 2. Convenience and comfort of human-computer interaction interface

[0058] 1) Subjectively evaluate whether the picture quality of the interactive display screen makes people feel comfortable and whether the size ratio is appropriate;

[0059] 2) Adjust the brightness of the display screen to the minimum and maximum to observe the color changes, and subjectively evaluate the rationality of the design of each icon pattern in each interface according to your preferences;

[0060] 3) Whether you can directly identify the general meaning of each system icon when operating the system interface for the first time.

[0061] 3. Visual, auditory and tactile warning prompts

[0062] 1) Trigger the visual alarm of the system in a relatively safe manner according to the functional definition of the system, observe the color of the warning signal on the display screen at this moment, and subjectively evaluate whether the warning degree is reasonable and whether there is any obvious delay, etc.;

[0063] 2) Trigger the auditory alarm of the system in a relatively safe manner according to the functional definition of the system, and determine whether the alarm sound is highly recognizable, whether the timing of the alarm sound is accurate, and whether there is any obvious delay;

[0064] 3) Trigger the tactile alarm of this system in a relatively safe manner according to the functional definition of the system, and subjectively experience the feedback method and intensity, rationality, and warning degree of the tactile alarm.

[0065] 4. Voice interaction

[0066] 1) Send relevant and recognizable voice commands to the system, and subjectively evaluate whether the sound rate is reasonable, whether there is obvious delay in the interaction process, whether the voice interaction is smooth, whether the fluency of the interaction is affected in a noisy environment, and whether the system can correctly understand the driver's commands and respond accordingly.

[0067] 5. Anti-interference

[0068] 1) Drive the vehicle to a road section with strong sunlight, causing the display screen to be exposed to strong sunlight for a long time. In this environment, frequently click various function switches of this system to observe the anti-interference ability of the system in this state.

[0069] 6. Ease of operation

[0070] 1) Open various prompt interfaces of the system and subjectively feel whether the prompt interface is easy to view;

[0071] 2) Use the system according to the content of the help interface, and subjectively judge whether the guidance interface can help you understand it, and whether the user does not feel confused or blocked during use due to excessive professional information;

[0072] 3) Use this system for a long time and subjectively feel whether the operating logic of the system is clear;

[0073] 4) When driving a vehicle for a long time and operating the interface when needed, will the driver feel that there are cumbersome continuous operation steps that distract the driver from driving for a long time?

[0074] 5) After using this system for a long time, you will subjectively feel whether the system's prompts and help terms maintain a certain consistency.

[0075] Multiple test engineers make subjective evaluations of the navigation intelligent driving system of the test vehicle according to the subjective evaluation indicators of human-computer interaction, and use the subjective evaluation of the test engineers (judging the quality of this function) as the evaluation criteria, as shown in Table 3:

[0076] Table 3 Subjective evaluation scoring table

[0077] Test items Subjective evaluation score weight (total score 100 points) Interactive interface aesthetics 15 points Convenient and comfortable interactive interface 15 points Visual, auditory, and tactile warning prompts 20 points Voice interaction 20 points Anti-interference 15 points Ease of operation 15 points

[0078] Combining multiple subjective evaluations, the final score can be calculated by weighted average to form the subjective evaluation result of the human-computer interaction of the navigation intelligent driving system. At the same time, the basis records of each score are retained, which can be used in the later stage of model training to improve the accuracy of subjective evaluation.

[0079] Test scenarios can be built in closed venues according to scenario requirements, and it has been verified through actual practice that closed test sites can meet the construction requirements.

[0080] The subjective and objective comprehensive evaluation of performance is to drive the test vehicle through the target test scenes one by one, turn on the positioning and data acquisition equipment to collect the test vehicle's operating data, and record the test vehicle's passing conditions in the target test scenes and the subjective evaluation content of the performance corresponding to the target test scenes as the basis for the subjective and objective comprehensive evaluation.

[0081] The subjective evaluation of performance focuses on longitudinal control experience, lateral control experience, traffic efficiency and passability. The specific evaluation indicators are shown in Table 4 below:

[0082] Table 4 Subjective evaluation index of performance

[0083]

[0084]

[0085] Subjective data analysis and processing methods for subjective evaluation of performance:

[0086] According to the scoring criteria in the above table, the subjective evaluation content is scored by multiple (usually 5-8) experienced test engineers on the performance of the system in the corresponding test scenarios, and the final score results are summed and averaged to obtain a more professional subjective test evaluation result.

[0087] Objective performance evaluation focuses on longitudinal performance, lateral performance, and motor vehicle signal light recognition performance. The specific evaluation indicators are shown in Table 5 below:

[0088] Table 5 Objective evaluation indicators of performance

[0089]

[0090] The objective evaluation of performance includes: taking the passing situation of the test vehicle in the target test scenario as the objective evaluation basis, and adding operation data in some scenarios for comprehensive objective evaluation. Specifically, the objective data processing method for objective evaluation of performance is as follows:

[0091] For intelligent vehicle speed control (longitudinal), intelligent vehicle following test (longitudinal), intelligent lane keeping (lateral), autonomous lane change (lateral), traffic efficiency, motor vehicle signal light recognition and response, if the objective data in the longitudinal and horizontal directions do not meet the passing requirements in Table 6 (including passing conditions and operation data), no points will be awarded. If they meet the requirements, full marks will be awarded. For other scenarios, such as lane and lane line type change passability test, entry and exit ramps, U-turn road passage (interchange), intersection passage (traffic lights), impassable areas (construction areas), accident areas, communication signal weakening or shielding areas, traffic sign recognition and response, direction indicator light recognition and response, etc., it is only necessary to pay attention to whether they can pass smoothly, that is, only make a passing situation judgment, and no requirements are made for performance indicator data.

[0092] Table 6 Objective data processing basis

[0093]

[0094] Among them, the acceleration requirements are as follows: Figure 3 As shown in the figure, when the vehicle speed exceeds 20m / s (72km / h), the average automatic acceleration of the adaptive cruise system should not exceed 2m / s 2 (The average interval is 2s); when the vehicle speed is lower than 5m / s (18km / h), the system average automatic acceleration should not exceed 4m / s 2 (The average interval is 2s).

[0095] Deceleration requirements, such as Figure 4 As shown in the figure, when the vehicle speed is higher than 20m / s (72km / h), the average automatic deceleration of the adaptive cruise system should not exceed 3.5m / s 2 (The average interval is 2s); when the vehicle speed is lower than 5m / s (18km / h), the system's average automatic deceleration should not exceed 5m / s 2 (The average interval is 2s).

[0096] Deceleration rate requirement, such as Figure 5 As shown in the figure, when the vehicle speed is higher than 20m / s (72km / h), the average change rate of the automatic deceleration of the adaptive cruise system should not exceed 2.5m / s 2 (The average interval is 1s); when the vehicle speed is lower than 5m / s (18km / h), the average change rate should not exceed 5m / s 2 (The average interval is 2s).

[0097] In this embodiment, the score weights of each test scenario in the subjective and objective comprehensive evaluation of performance are shown in Table 7, and the final subjective and objective comprehensive evaluation score of performance can be obtained by weighted average method.

[0098] Table 7 Score weights

[0099]

[0100]

[0101] The total score of the subjective evaluation part of human-computer interaction is 100 points, and the subjective evaluation and objective evaluation of performance are 100 points in total. In this embodiment, weights can be set, with human-computer interaction accounting for 30% and performance accounting for 70%. The final test score of the system to be tested is obtained by weighted sum of the scores. Through a complete evaluation system, the system functions are subjectively and objectively scored and evaluated, which can directly and effectively evaluate the intelligent driving capability of the navigation intelligent driving system equipped in the vehicle.

[0102] The present embodiment provides a test and evaluation method for an ADAS navigation intelligent driving system. First, a relatively comprehensive test scenario library is constructed for the navigation intelligent driving system. On this basis, the optimization of the intelligent connected vehicle test site and the emergence of new vehicles are monitored in real time, the construction status and the addition of preset business scenarios are judged, and the test scenarios are dynamically increased accordingly, so that the test scenarios used for the system are in a dynamic and comprehensive state, and the authenticity of the test scenarios in the test site can be maintained. During the specific test, different business scenarios are tested, and the actually testable items are selected according to different business scenarios, and the test priority is changed. On the basis that the test site is sufficiently real, the test error can be within an acceptable range. From the subjective evaluation of human-computer interaction and the subjective and objective comprehensive evaluation of performance, both the driving experience and the objective performance are taken into account, so that the intelligent driving ability evaluation of the system is more complete and the results are more referenceable. It can be said that this scheme proposes a test site test method for navigation intelligent driving systems with dynamic and comprehensive scenarios, variable test priorities, and selectable actual tests. Within the limited service life and usage scenarios of the vehicle, and based on monitoring the existing available intelligent connected vehicle test sites, the constructed test scenarios are comprehensive and realistic enough, with low testing costs and small amounts of data. A feasible and standardized test method can be formed in the test site, which can not only accurately evaluate the navigation intelligent driving system, but also make the evaluation results have comparative reference value.

[0103] The above is only an embodiment of the present invention. The common sense such as the known specific structure and characteristics in the scheme is not described in detail here. The ordinary technicians in the relevant field are aware of all the common technical knowledge in the technical field of the invention before the application date or priority date, can obtain all the existing technologies in the field, and have the ability to apply the conventional experimental means before that date. The ordinary technicians in the relevant field can improve and implement the scheme in combination with their own abilities under the enlightenment given by this application. Some typical known structures or known methods should not become obstacles for the ordinary technicians in the relevant field to implement this application. It should be pointed out that for the technicians in this field, without departing from the structure of the present invention, several deformations and improvements can be made, which should also be regarded as the protection scope of the present invention, and these will not affect the effect of the implementation of the present invention and the practicality of the patent.

Claims

1. A test and evaluation method for an ADAS navigation intelligent driving system, characterized in that: S100, based on the characteristics of the navigation intelligent driving system, builds a test scenario library for evaluating the intelligent driving capabilities of the navigation intelligent driving system; S200, continuously monitors preset conditions, dynamically changes the test scenario library in a preset manner, and forms the current test scenario library; preset conditions include intelligent connected vehicle test site optimization and new vehicle release; S300, determining a test vehicle equipped with the navigation intelligent driving system to be tested; According to the business scenarios of the test vehicle, the priority of the target test scenarios is determined according to the preset scenario arrangement strategy in the current test scenario library; S400, the test vehicle drives as required in the target test scene to obtain test data; Evaluation is performed based on the test data using a preset evaluation strategy to obtain an evaluation result; the preset evaluation strategy includes a subjective evaluation of human-computer interaction and a subjective and objective comprehensive evaluation of performance.

2. A test and evaluation method for an ADAS navigation intelligent driving system according to claim 1, characterized in that: The test scenario library is a multi-level scenario collection, including several main scenarios and at least one layer of sub-scenario under each main scenario; several main scenarios include intelligent vehicle speed control longitudinal, intelligent vehicle following test longitudinal, intelligent lane keeping lateral, lane and lane line type change passability test, autonomous lane change lateral, entry and exit ramps, U-turn traffic, intersection traffic, driving in impassable areas, traffic accident areas, areas where communication signals are weakened or blocked, traffic efficiency, traffic sign recognition and response, motor vehicle signal light recognition and response, and direction indicator signal light recognition and response.

3. A test and evaluation method for an ADAS navigation intelligent driving system according to claim 1, characterized in that: The preset situation is the optimization of the intelligent connected vehicle test site. The dynamic change of the test scenario library is performed in a preset manner, which is to capture the construction status of all intelligent connected vehicle test sites, match the test scenarios in the test scenario library, and change the test scenarios according to the preset change strategy based on the matching status of the test scenarios; the construction status includes lane changes, equipment changes and regional scene changes.

4. A test and evaluation method for an ADAS navigation intelligent driving system according to claim 1, characterized in that: New cars include vehicles whose models and navigation intelligent driving systems are arranged in new and old combinations; the preset situation is the release of a new car, and the dynamic change of the test scenario library is performed in a preset manner, which is to capture the characteristics and estimated business scenarios of the new car and its navigation intelligent driving system, match the test scenarios in the test scenario library, and change the test scenarios according to the preset change strategy based on the matching situation of the test scenarios.

5. A test and evaluation method for an ADAS navigation intelligent driving system according to claim 3 or 4, characterized in that: The preset change strategy is that if the test scenario is matched, no change will be made to the test scenario; if not matched, the necessity of the test will be judged, and according to the result of the necessity judgment, the test scenario will be changed, including addition, deletion or replacement; the captured content will be marked with the matching status and the test scenario change status.

6. A test and evaluation method for an ADAS navigation intelligent driving system according to claim 1, characterized in that: The preset scenario arrangement strategy is to mark the test scenarios that cannot be actually measured at the intelligent connected vehicle test site at the test site as untestable scenarios at the test site in the current test scenario library; Among the remaining test scenarios, a first target test scenario is selected according to the estimated business scenario of the test vehicle, a second target test scenario different from the first target test scenario is selected according to the performance of the navigation intelligent driving system to be tested, and the remaining unselected test scenarios are third target test scenarios; the first target test scenario, the second target test scenario and / or the third target test scenario are tested and sorted according to a preset sorting method.

7. A test and evaluation method for an ADAS navigation intelligent driving system according to claim 1, characterized in that: The subjective evaluation of human-computer interaction is to build a subjective evaluation index system for human-computer interaction. When the test vehicle is static, multiple test engineers make subjective evaluations on the test vehicle's navigation intelligent driving system based on the subjective evaluation indicators of human-computer interaction in turn. Multiple subjective evaluations are combined to form a subjective evaluation result of human-computer interaction for the navigation intelligent driving system.

8. A test and evaluation method for an ADAS navigation intelligent driving system according to claim 1, characterized in that: The subjective evaluation index system for human-computer interaction includes test indicators for the aesthetics of the interactive interface, test indicators for the convenience and comfort of the interactive interface, test indicators for audio-visual and tactile warning prompts, test indicators for voice interaction, test indicators for anti-interference, and test indicators for ease of operation.

9. A test and evaluation method for an ADAS navigation intelligent driving system according to claim 1, characterized in that: The subjective and objective comprehensive evaluation of performance is to drive the test vehicle through the target test scenes one by one, turn on the positioning and data acquisition equipment to collect the test vehicle's operating data, and record the test vehicle's passing conditions in the target test scenes and the subjective evaluation content of the performance corresponding to the target test scenes as the basis for the subjective and objective comprehensive evaluation.

10. A test and evaluation method for an ADAS navigation intelligent driving system according to claim 9, characterized in that: The objective evaluation of performance includes: taking the test vehicle's passing performance in the target test scenario as the objective evaluation basis, and adding operating data in some scenarios for comprehensive objective evaluation.

Citation Information

Cited By

  • Intelligent cockpit evaluation method and system

    CN119647791A

  • Calculation method for evaluating end-to-end intelligent driving safety generalization capability

    CN120121316A

  • A computational method for evaluating the safety generalization ability of end-to-end intelligent driving

    CN120121316B

  • Intelligent network-in-loop test method and system based on multi-source data acquisition

    CN121174210A