An automated evaluation method and device for intelligent driving

Abstract

The present invention provides an automated evaluation method and device for intelligent driving. Among them, the method includes: obtaining a test case, inputting the test case into an intelligent driving controller, determining the control driving parameters of the test case in a simulation scenario, inputting the control driving parameters into a vehicle model to obtain vehicle state parameters; determining whether the simulation system is normal according to the test case, the control driving parameters and the vehicle state parameters; if it is normal, waiting for the test case to end the test to obtain a test result, and evaluating the test result according to a preset evaluation criterion. It realizes the automated evaluation and testing of intelligent driving. It realizes the independent operation and effective cooperation of the simulation test system and the evaluation work.

Description

Technical Field

[0001] The present invention relates to the field of intelligent driving tests, and particularly to an automated evaluation method and device for intelligent driving. Background Art

[0002] Intelligent driving vehicles send relevant data such as their high-precision longitude, latitude, altitude, and destination to the control center through a communication system. The control center processes all vehicle status information, destination information, etc., plans the optimal route for all vehicles, and conveys the vehicle control information to all vehicles in real time, thereby realizing intelligent driving within the entire system. However, compared with traditional vehicles, the driving tests of intelligent driving vehicles have an exponentially increasing verification volume, posing great challenges in terms of time, manpower, and test costs.

[0003] In the existing technologies of intelligent driving tests, the quantitative evaluation methods in different dimensions mainly rely on the experience of test engineers, and almost all the test evaluations of a huge simulation scenario library are completed by test engineers. Therefore, for the preliminary positioning and specific analysis of test problems found in the existing technologies, they all rely on test engineers to check one by one.

[0004] In addition, the output of the existing technologies for intelligent driving simulation tests and evaluation results is basically carried out in two steps. The test result evaluation mainly relies on the engineer to output the evaluation result after the test ends, lacking linkage and real-time performance. Moreover, the simulation test system and the evaluation work operate independently, resulting in poor overall linkage and real-time performance and being unable to effectively cooperate in parallel. Summary of the Invention

[0005] The present invention provides an automated evaluation method and device for intelligent driving.

[0006] In a first aspect, the present invention provides an automated evaluation method for intelligent driving, including: obtaining a test case, inputting the test case into an intelligent driving controller, determining the control driving parameters of the test case in a simulation scenario, inputting the control driving parameters into a vehicle model to obtain vehicle state parameters; determining whether the simulation system is normal according to the test case, the control driving parameters, and the vehicle state parameters; if normal, waiting for the test case to end the test to obtain a test result, and evaluating the test result according to a preset evaluation criterion.

[0007] Further, the obtaining a test case, inputting the test case into an intelligent driving controller, and determining the control driving parameters of the test case in a simulation scenario includes: obtaining a test case, determining the parameters of the simulation scenario according to the test case, inputting the parameters of the simulation scenario and the test case into the intelligent driving controller, and determining the control driving parameters of the test case in the simulation scenario.

[0008] Further, the steps of obtaining a test case, inputting the test case into an intelligent driving controller, determining the control driving parameters of the test case in a simulation scenario, and inputting the control driving parameters into a vehicle model to obtain vehicle state parameters include: obtaining a test case, inputting the test case into an intelligent driving controller, determining the control driving parameters of the test case at the current moment in the simulation scenario, inputting the control driving parameters at the current moment into the vehicle model to obtain the vehicle state parameters at the current moment; inputting the vehicle state parameters at the current moment into the intelligent driving controller, determining the control driving parameters at the next moment in the simulation scenario corresponding to the vehicle state parameters at the current moment, inputting the control driving parameters at the next moment into the vehicle model to obtain the vehicle state parameters at the next moment; and so on in a loop until the test case ends the test.

[0009] Further, determining whether the simulation system is normal according to the test case, the control driving parameters, and the vehicle state parameters includes: determining whether the simulation system is normal according to the test case, the control driving parameters at the current moment, and the vehicle state parameters at the current moment; and / or determining whether the simulation system is normal according to the test case, the control driving parameters at the next moment, and the vehicle state parameters at the next moment.

[0010] Further, determining whether the simulation system is normal according to the test case, the control driving parameters, and the vehicle state parameters includes: inputting the test case, the control driving parameters, and the vehicle state parameters into an observer to determine whether the simulation system is normal.

[0011] Further, the observer includes a system state observer, a scenario observer, and a communication diagnostic; and inputting the test case, the control driving parameters, and the vehicle state parameters into the observer to determine whether the simulation system is normal includes: inputting the control driving parameters and the vehicle state parameters into the system state observer to determine whether the state of the simulation system is normal; inputting the test case and the parameters of the simulation scenario into the scenario observer to determine whether the simulation scenario is normal; inputting a preset communication protocol and the signal of the vehicle state parameters into the communication diagnostic to determine whether the communication is normal.

[0012] Further, inputting the control driving parameters and the vehicle state parameters into the system state observer to determine whether the state of the simulation system is normal includes: determining the predicted vehicle state parameters at the next moment according to the vehicle state parameters at the current moment and the control driving parameters at the current moment; obtaining the vehicle state parameters at the next moment, determining the deviation value between the vehicle state parameters at the next moment and the predicted vehicle state parameters at the next moment, and if the deviation value does not meet the preset range, the state of the simulation system is abnormal.

[0013] Further, the test case includes a corresponding design operation domain, and the simulation scenario also includes a corresponding design operation domain, where the design operation domain is used to represent the key parameters of the driving scenario simulated by the test case; and, inputting the test case and the simulation scenario into the scenario observer to determine whether the simulation scenario is normal includes: inputting the design operation domain corresponding to the test case and the design operation domain corresponding to the simulation scenario into the scenario observer to determine whether the simulation scenario is normal.

[0014] Further, inputting the preset communication protocol and the signal of the vehicle state parameters into the communication diagnostic to determine whether the communication is normal includes: the communication diagnostic determines the sending period, signal name, and check information of the signal according to the received signal of the vehicle state parameters, and checks whether the sending period, signal name, and check information of the signal are normal through the preset communication protocol.

[0015] Further, the evaluation criteria include at least one of the following: comfort, reliability, fuel economy, and safety.

[0016] In a second aspect, the present invention also provides an automated evaluation device for intelligent driving, including: a simulation module, configured to obtain a test case, input the test case into an intelligent driving controller, determine the control driving parameters of the test case in a simulation scenario, and input the control driving parameters into a vehicle model to obtain vehicle state parameters; a simulation system evaluation module, configured to determine whether the simulation system is normal according to the test case, the control driving parameters, and the vehicle state parameters; an intelligent driving controller evaluation module, configured to, if normal, wait for the test case to end the test to obtain a test result, and evaluate the test result according to the preset evaluation criteria.

[0017] In a third aspect, the present invention also provides an electronic device, including a memory, a processor, and a computer program stored on the memory and executable on the processor, where the processor implements the steps of the automated evaluation method for intelligent driving as described in any one of the above when executing the program.

[0018] In a fourth aspect, the present invention also provides a non-transitory computer-readable storage medium, on which a computer program is stored, and the computer program implements the steps of the automated evaluation method for intelligent driving as described in any one of the above when executed by a processor.

[0019] In a fifth aspect, the present invention also provides a computer program product, including a computer program, and the computer program implements the steps of the automated evaluation method for intelligent driving as described in any one of the above when executed by a processor.

[0020] An automated evaluation method and device for intelligent driving provided by the present invention determine control driving parameters of a test case in a simulation scenario by inputting the test case into an intelligent driving controller, and input the control driving parameters into a vehicle model to obtain vehicle state parameters, shortening the test cycle of the simulation test work of the huge test scenario library of intelligent driving and improving the test efficiency. Determine whether the simulation system is normal according to the test case, the control driving parameters and the vehicle state parameters. If it is normal, wait for the test case to end the test to obtain a test result, and evaluate the test result according to a preset evaluation criterion, so that the simulation test and evaluation can be effectively coordinated and parallel, improving the overall linkage and real-time performance of the simulation test and evaluation. Moreover, the simulation system and the test result are respectively evaluated, realizing the preliminary classification and positioning of possible problems in the driving test process. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following will briefly introduce the drawings required for the description of the embodiments or the prior art. Obviously, the drawings in the following description are some embodiments of the present invention. For those of ordinary skill in the art, without creative efforts, other drawings can be obtained according to these drawings.

[0022] Figure 1 is a schematic flowchart of some embodiments of the automated evaluation method for intelligent driving provided by the present invention;

[0023] Figure 2 is a schematic diagram of a framework of the automated evaluation method for intelligent driving;

[0024] Figure 3 is a schematic structural diagram of some embodiments of the automated evaluation device for intelligent driving provided by the present invention;

[0025] Figure 4 is a schematic structural diagram of an electronic device provided by the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0026] To make the objectives, technical solutions and advantages of the present invention clearer, the following will clearly and completely describe the technical solutions in the present invention with reference to the accompanying drawings in the present invention. Obviously, the described embodiments are some, but not all, of the embodiments of the present invention. Based on the embodiments in the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts fall within the scope of protection of the present invention.

[0027] In addition, it should be noted that for the sake of convenience of description, only the parts related to the relevant invention are shown in the drawings. Without conflict, the embodiments in the present invention and the features in the embodiments can be combined with each other.

[0028] It should be noted that the concepts such as "first" and "second" mentioned in the present invention are only used to distinguish different devices, modules or units, and are not used to limit the order or mutual dependence relationship of the functions performed by these devices, modules or units.

[0029] It should be noted that the modifications of "one" and "multiple" mentioned in the present invention are illustrative rather than restrictive. Those skilled in the art should understand that unless clearly specified otherwise in the context, it should be understood as "one or more".

[0030] The names of the messages or information exchanged between multiple devices in the embodiments of the present invention are only for illustrative purposes, and are not used to limit the scope of these messages or information.

[0031] The present invention will be described in detail below with reference to the drawings and in conjunction with embodiments.

[0032] Please refer to Figure 1 , Figure 1 which is a schematic flowchart of some embodiments of the automated evaluation method for intelligent driving provided by the present invention. As Figure 1 shown, the method includes the following steps:

[0033] Step 101, obtain a test case, input the test case into the intelligent driving controller, determine the control driving parameters of the test case in the simulation scenario, and input the control driving parameters into the vehicle model to obtain vehicle state parameters.

[0034] The present invention mainly relates to intelligent driving function performance test cases, simulation test platforms and evaluation systems. The intelligent driving controller and the simulation test platform can be automatically evaluated from the perspectives of function specifications, performance indicators, design operating ranges, etc. The overall framework process is as Figure 2 shown.

[0035] As Figure 2As shown in the figure, the automated evaluation method for intelligent driving may include a simulation system and an evaluation system. Among them, the simulation system includes a simulation scenario, an intelligent driving controller, and a vehicle model (including vehicle dynamics models, vehicle kinematics models, etc.). The system requirements include multiple test cases, and the test cases may include case numbers, functional and performance requirements for intelligent driving, output requirements for test results, design operating domains (i.e., key parameters of the simulation scenario), and requirements for the simulation system, etc. The functional requirements for intelligent driving may be lane-changing requirements, deceleration requirements, etc. The performance requirements for intelligent driving may be curves that need to be met for lane-changing, whether the deceleration speed meets the requirements, etc. The output requirements for test results and the requirements for the simulation system may be the expected values of vehicle state parameters.

[0036] The simulation scenario in the simulation system can be determined by the test cases.

[0037] Input the test cases into the intelligent driving controller, and the intelligent driving controller can determine the control driving parameters of the test cases in the simulation scenario according to the functional and performance requirements for intelligent driving, the output requirements for test results, the design operating domain, etc. of the input test cases.

[0038] The control driving parameters are equivalent to controlling the actions of the vehicle. The actions of the vehicle can be turning the steering wheel to the left, stepping on the accelerator, etc. The corresponding control driving parameters can be the angle of turning left, acceleration.

[0039] The vehicle model is equivalent to the intelligent driving vehicle to be tested. Inputting the control driving parameters into the vehicle model is equivalent to conducting a driving test on the intelligent driving vehicle to be tested. The vehicle state parameters output by the vehicle model are the states of the vehicle after completing the driving. As an example, the vehicle state parameters can be the current speed of the vehicle, the position in the simulation scenario where it is located, etc.

[0040] Step 102, determine whether the simulation system is normal according to the test cases, control driving parameters, and vehicle state parameters.

[0041] As Figure 2 shown, as an example, after the control driving parameters are input into the vehicle model, the vehicle state parameters are obtained by the output of the vehicle model. The test cases may include the expected vehicle state parameters. Therefore, it is possible to match the expected vehicle state parameters with the vehicle state parameters obtained by the output of the vehicle model. If the match is unsuccessful, it is determined that the simulation system is abnormal. The test cases may also include the expected control driving parameters. If the match between the expected control driving parameters and the control driving parameters input into the vehicle model is unsuccessful, then it is determined that the simulation system is abnormal.

[0042] Step 103, if it is normal, wait for the test cases to end the test, obtain the test results, and evaluate the test results according to the preset evaluation criteria.

[0043] If the simulation system is normal, it is necessary to wait for the test cases to be tested. As an example, the test time for each test case can be preset to ensure the integrity and coherence of the test data, which is beneficial to improving the accuracy of the evaluation of the test results. The preset evaluation criteria can be defined in the test cases. The simulation system and the evaluation system run simultaneously, realizing the linkage and real-time performance of the simulation test and the simulation evaluation.

[0044] As Figure 2 shown, the evaluation system includes a signal processor, an observer, an evaluator, etc. Among them, the evaluator evaluates the cases other than those test cases that fail the test due to the simulation system and outputs the test results.

[0045] The automated evaluation method for intelligent driving disclosed in some embodiments of the present invention determines the control driving parameters of the test case in the simulation scenario by inputting the test case into the intelligent driving controller, and inputs the control driving parameters into the vehicle model to obtain the vehicle state parameters, shortening the test cycle of the simulation test work in the huge test scenario library of intelligent driving and improving the test efficiency. Determine whether the simulation system is normal according to the test case, the control driving parameters and the vehicle state parameters. If it is normal, wait for the test case to end the test to obtain the test result, and evaluate the test result according to the preset evaluation criteria, so that the simulation test and the evaluation can be effectively coordinated and run in parallel, improving the overall linkage and real-time performance of the simulation test and the evaluation. Moreover, the simulation system and the test result are respectively evaluated, realizing the preliminary classification and positioning of the possible problems in the driving test process.

[0046] In some alternative implementation manners, obtaining the test case and inputting the test case into the intelligent driving controller to determine the control driving parameters of the test case in the simulation scenario includes: obtaining the test case, determining the parameters of the simulation scenario according to the test case, and inputting the parameters of the simulation scenario and the test case into the intelligent driving controller to determine the control driving parameters of the test case in the simulation scenario.

[0047] As an example, after obtaining the test case, the pre-stored simulation scenario can be obtained by matching according to the number of the test case, the parameters of the simulation scenario (or the designed operation domain of the simulation scenario) are obtained, and the parameters of the simulation scenario and the test case are input into the intelligent driving controller.

[0048] In some alternative implementations, a test case is obtained and input into an intelligent driving controller. The control driving parameters of the test case in a simulation scenario are determined, and the control driving parameters are input into a vehicle model to obtain vehicle state parameters, including: obtaining a test case, inputting the test case into an intelligent driving controller, determining the control driving parameters of the test case at the current moment in the simulation scenario, inputting the control driving parameters at the current moment into the vehicle model to obtain the vehicle state parameters at the current moment; inputting the vehicle state parameters at the current moment into the intelligent driving controller, determining the control driving parameters at the next moment in the simulation scenario corresponding to the vehicle state parameters at the current moment, and inputting the control driving parameters at the next moment into the vehicle model to obtain the vehicle state parameters at the next moment; and so on in a loop until the test case ends the test.

[0049] It may take some time to complete the test of a test case in the corresponding simulation scenario. For example, a simulation scenario is a left-turn intersection, that is, the vehicle needs to go straight for a certain distance and then turn left. When the parameters of a test case and the simulation scenario are input into the intelligent driving controller, the intelligent driving controller needs to determine the operation actions (i.e., control driving parameters, such as going straight and decelerating) according to the parameters of the initial vehicle defined in the test case, such as the position and speed, and the current simulation scenario faced by the vehicle (e.g., going straight). Inputting the control driving parameters into the vehicle model is equivalent to driving the vehicle to decelerate and go straight according to the operation actions. At the same time, the vehicle state will change (the changed vehicle state is the obtained current vehicle state parameters, and the changed vehicle state can be going straight and with a reduced speed), and the changed vehicle state is the vehicle state parameters at the current moment. During the driving process of the vehicle in the simulation scenario, the specific position of the vehicle in the simulation scenario also changes (for example, after the vehicle goes straight, it is currently in a state of waiting to turn left). Therefore, the intelligent driving controller needs to determine the control driving parameters at the next moment (e.g., turning left) according to the vehicle state parameters at the current moment and the current simulation scenario faced by the vehicle (e.g., waiting to turn left), and so on in a loop until the simulated driving in the simulation scenario is completed.

[0050] As an example, the time for simulated driving can be preset in each test case, and it is stipulated to complete the simulated driving within this time.

[0051] In some alternative implementations, it is determined whether the simulation system is normal according to the test case, the control driving parameters, and the vehicle state parameters, including: determining whether the simulation system is normal according to the test case, the control driving parameters at the current moment, and the vehicle state parameters at the current moment; and / or determining whether the simulation system is normal according to the test case, the control driving parameters at the next moment, and the vehicle state parameters at the next moment.

[0052] For the control driving parameters and vehicle state parameters obtained in each loop, it is necessary to determine whether the simulation system is normal based on them.

[0053] In some alternative implementation manners, determining whether the simulation system is normal based on the test case, control driving parameters, and vehicle state parameters includes: inputting the test case, control driving parameters, and vehicle state parameters into an observer to determine whether the simulation system is normal.

[0054] The observer is used to determine whether the simulation system is normal. As Figure 2 shown, the observer can determine whether the simulation system is normal based on the control driving parameters and vehicle state parameters output by the intelligent driving controller and the vehicle model, as well as the test case. As Figure 2 shown, the signal of the control driving parameters sent by the intelligent driving controller and the signal of the vehicle state parameters sent by the vehicle model will be input into the signal processor. The signal processor parses and sends the signals to the observer according to the preset rules. At the same time, the observer receives the test case and determines whether the simulation system is normal through the received test case, control driving parameters, and vehicle state parameters. As an example, the control driving parameters are going straight and decelerating, the vehicle state parameters are going straight and decelerating, and the expected vehicle state included in the test case is going straight and decelerating. Therefore, the observer can determine that the vehicle state parameters have executed the control driving parameters and the execution is effective, indicating that the vehicle model is normal; the expected vehicle state included in the test case is consistent with the vehicle state parameters, indicating that the instruction sent by the intelligent driving controller is correct and the intelligent driving controller is normal. Thus, the simulation system is normal.

[0055] In some alternative implementation manners, the observer includes a system state observer, a scenario observer, and a communication diagnoser; and, inputting the test case, control driving parameters, and vehicle state parameters into the observer to determine whether the simulation system is normal includes: inputting the control driving parameters and vehicle state parameters into the system state observer to determine whether the simulation system state is normal; inputting the test case and the parameters of the simulation scenario into the scenario observer to determine whether the simulation scenario is normal; inputting the preset communication protocol and the signal of the vehicle state parameters into the communication diagnoser to determine whether the communication is normal.

[0056] In some embodiments, intelligent driving function performance test cases are written based on the system requirements document (the system requirements document contains multiple test cases), and each test case includes a case number, a description (including key evaluation index information), an expected result (including key evaluation index information), etc. The simulation system determines the simulation scenario according to the specific case, and the parameters of the simulation scenario (the parameters of the simulation scenario will change with the change of the vehicle state. For example, after the vehicle travels forward for a certain distance, the vehicle may be about to enter the intersection of the simulation scenario. Therefore, the parameters of the simulation scenario will add traffic light information) will be injected into the algorithm in the intelligent driving controller, and then act on the vehicle model. Both the vehicle model and the intelligent driving controller output real-time signals to the signal processor of the evaluation system. The signal value processed by the signal processor is output to the observer, and the observer can determine whether the current test case fails due to problems in the simulation system. The observer can start from three aspects: system state, scenario, and communication diagnosis.

[0057] As Figure 2 shown, the evaluation system includes a signal processor, an observer, an evaluator, etc. The signal processor can parse and process the output signals of the vehicle and the signals output by the intelligent driving controller. The observer (or the simulation system observer) can determine whether the simulation system is normal, including the simulation system state observer, the scenario observer, and the communication diagnoser, and observe the simulation system from these three aspects to exclude the problem of failed tests due to the simulation system.

[0058] As an example, the control driving parameters are straight ahead and deceleration, the vehicle state parameters are straight ahead and deceleration, and the expected vehicle state included in the test case is straight ahead and deceleration. The consistency between the control driving parameters and the vehicle state parameters indicates that the vehicle model correctly executes the control driving parameters, and the control driving parameters are determined according to the simulation scenario and the test case. When the control driving parameters are correct, it indicates that the simulation scenario and the intelligent driving controller are normal. Therefore, the simulation system state is normal.

[0059] As an example, the test case can include a simple description of the simulation scenario, which is used to determine whether the obtained simulation scenario is correct. Therefore, the simple description of the simulation scenario in the test case and the obtained simulation scenario can be input into the scenario observer to determine whether the scenario observer is normal. In addition, the acquisition of the simulation scenario can also be obtained by matching the simple description of the simulation scenario in the test case with the database.

[0060] As an example, the preset communication protocol can be a preset range that stipulates the signal sending period, sending frequency, etc. of the vehicle state parameters, and the communication diagnoser is used to determine whether the signal of the vehicle state parameters conforms to the preset communication protocol.

[0061] In some alternative implementation manners, input the control driving parameters and vehicle state parameters into the system state observer to determine whether the simulation system state is normal, including: determining the predicted vehicle state parameters at the next moment according to the vehicle state parameters at the current moment and the control driving parameters at the current moment; obtaining the vehicle state parameters at the next moment, and determining the deviation value between the vehicle state parameters at the next moment and the predicted vehicle state parameters at the next moment. If the deviation value does not meet the preset range, the simulation system state is abnormal.

[0062] Since the vehicle state parameters and control driving parameters are obtained cyclically in the simulation system, it is necessary to judge the vehicle state parameters and control driving parameters at adjacent moments. Because the vehicle state parameters and control driving parameters at adjacent moments generally do not differ greatly, if the vehicle state parameters at adjacent moments differ greatly, it indicates that the simulation system state is abnormal.

[0063] In some embodiments, the observer can obtain vehicle state information from the vehicle model, and estimate the vehicle state value at the current moment (k + 1 moment) based on the vehicle state at the previous moment (k moment). The formula is as follows. Then compare the vehicle state value at the current moment with the actual vehicle measurement value. If the deviation exceeds the reasonable range, it is considered that there is a problem with the simulation system, and the test case result is a failure caused by the simulation system. Otherwise, the simulation system is okay, wait for the current test case to end, and the evaluator will conduct an evaluation.

[0064]

[0065] Among them,

[0066] v: (velocity) vehicle speed

[0067] a: (acceleration) acceleration

[0068] j: (jerk) acceleration change rate

[0069] Φ: (yaw rate) yaw angular velocity

[0070] F k : state transition matrix

[0071] u k : external input matrix (intelligent driving controller command input matrix, for example: acceleration, steering wheel angle)

[0072] As an example, if the intelligent driving controller outputs control signals such as turn signals, brake lights, and windshield wipers to the observer, and at the same time, the vehicle model signals received by the observer do not have corresponding responses, it is considered that there is a problem with the simulation system, and the test case result is a failure caused by the simulation system. Otherwise, the simulation system is okay, wait for the current test case to end, and the evaluator will conduct an evaluation.

[0073] In some alternative implementation manners, the test case includes a corresponding design operation domain, and the simulation scenario also includes a corresponding design operation domain, where the design operation domain is used to represent the key parameters of the driving scenario simulated by the test case; and, inputting the test case and the simulation scenario into a scenario observer to determine whether the simulation scenario is normal, including: inputting the design operation domain corresponding to the test case and the design operation domain corresponding to the simulation scenario into the scenario observer to determine whether the simulation scenario is normal.

[0074] The design operation domain corresponding to the test case represents the key parameters of the simulation scenario corresponding to the test case, such as the road information of the simulation scenario. The simulation scenario also includes a corresponding design operation domain, and the design operation domain corresponding to the simulation scenario represents the key parameters in this simulation scenario, such as the road information of the simulation scenario. This simulation scenario may also include other parameters, such as the information of the trees and flowers beside the road.

[0075] Each test case contains a corresponding design operation domain. According to the NHTSA construction framework, the design operation domain includes infrastructure (such as road type, etc.), driving operation restrictions (such as speed limit, etc.), surrounding objects (such as road signs, etc.), interconnection (such as vehicles, etc.), environmental conditions (such as weather, lighting, etc.), regions (such as geofences, traffic control regions, etc.). The observer obtains the key parameter information in the specified test case scenario description (i.e., the design operation domain corresponding to the test case) and the corresponding simulation scenario information in the simulation system (i.e., the design operation domain corresponding to the simulation scenario), and compares the two design operation domains to exclude test failures caused by scenario problems. If the simulation scenario does not match the scenario description in the design operation domain of the test case, there is a problem with the simulation system, and the test case result is a test failure caused by the simulation system. Otherwise, the simulation system is okay, waiting for the current test case to end, and then the evaluator will conduct an evaluation.

[0076] In some alternative implementation manners, inputting a preset communication protocol and the signal of the vehicle state parameters into a communication diagnostic to determine whether the communication is normal, including: the communication diagnostic determines the sending period, signal name, and check information of the signal according to the received signal of the vehicle state parameters, and checks whether the sending period, signal name, and check information of the signal are normal through the preset communication protocol.

[0077] The communication diagnostic in the observer can check information such as the signal sending period, signal name, and check information from the vehicle model according to the communication protocol. If the information does not match, there is a communication problem with the simulation system, and the test case result is a test failure caused by the simulation system. Otherwise, the simulation system is okay, waiting for the current test case to end, and then the evaluator will conduct an evaluation.

[0078] In some alternative implementation manners, the evaluation criteria include at least one of the following: comfort, reliability, fuel economy, and safety.

[0079] The evaluator evaluates the current test case according to the evaluation criteria. If the evaluation criteria meet the system requirements, the test result is normal; otherwise, the test result is failed, and the intelligent driving controller of the object under test fails this test. For the huge intelligent driving scenario library, the present invention can realize the automated evaluation of the execution results of test cases and the linkage mechanism between the simulation system and the automated evaluation system, which can release some human resources and test resources, improve the test efficiency, and shorten the test cycle.

[0080] As Figure 2 shown, when the observer completes the judgment on whether the simulation system is normal, a preliminary classification of the failure reasons for the unpassed test cases is realized: one category is no simulation system problem, that is, the test failure caused by the problem of the simulation system; the other category is with simulation system problem, that is, the test failure caused by the intelligent driving controller, enabling the test engineer to output efficiently according to the problem location. After excluding whether it is a problem of the simulation system, the test execution situation of the current case (that is, all the control driving parameters and vehicle state parameters obtained after the test case is tested) and the evaluation criteria of the test case from the system requirements (the evaluation criteria can be recorded in the test case) are injected into the evaluator. The evaluation criteria of the test case from the system requirements include performance indicators such as comfort (for example, if the acceleration change exceeds the threshold, it indicates poor comfort), reliability (for example, the number of unsuccessful lane changes), fuel economy (the vehicle model itself can define an initial fuel quantity, and the fuel situation is calculated according to the parameters output by the vehicle model to judge the fuel economy), and safety (for example, the number of collisions). On the one hand, the evaluation criteria of the object under test are quantified from the requirement definition level (that is, the evaluation criteria for quantifying whether the algorithm of the intelligent driving control is normal), realizing the evaluation of the performance test results of the intelligent driving controller from different requirement perspectives such as comfort, reliability, fuel economy, and safety; on the other hand, the confidence of the simulation system is quantitatively evaluated from the perspective of the simulation system.

[0081] Please refer to Figure 3 , Figure 3 which is a schematic structural diagram of some embodiments of an automated evaluation device for intelligent driving according to the present invention. As an implementation of the methods shown in the above figures, the present invention also provides some embodiments of an automated evaluation device for intelligent driving. These device embodiments correspond to Figure 1 some method embodiments shown, and the device can be applied to various electronic devices.

[0082] As Figure 3As shown, the automated evaluation device for intelligent driving in some embodiments includes a simulation module 301, a simulation system evaluation module 302, and an intelligent driving controller evaluation module 303: The simulation module 301 is configured to obtain a test case, input the test case into the intelligent driving controller, determine the control driving parameters of the test case in the simulation scenario, and input the control driving parameters into the vehicle model to obtain vehicle state parameters; The simulation system evaluation module 302 is configured to determine whether the simulation system is normal according to the test case, the control driving parameters, and the vehicle state parameters; The intelligent driving controller evaluation module 303 is configured to, if it is normal, wait for the test case to end the test, obtain the test result, and evaluate the test result according to the preset evaluation criteria.

[0083] In an alternative implementation of some embodiments, the simulation module 301 is further configured to: obtain a test case, determine the parameters of the simulation scenario according to the test case, and input the parameters of the simulation scenario and the test case into the intelligent driving controller to determine the control driving parameters of the test case in the simulation scenario.

[0084] In an alternative implementation of some embodiments, the simulation module 301 is further configured to: obtain a test case, input the test case into the intelligent driving controller, determine the control driving parameters at the current moment of the test case in the simulation scenario, input the control driving parameters at the current moment into the vehicle model to obtain the vehicle state parameters at the current moment; Input the vehicle state parameters at the current moment into the intelligent driving controller, determine the control driving parameters at the next moment of the vehicle state parameters corresponding to the current moment in the simulation scenario, and input the control driving parameters at the next moment into the vehicle model to obtain the vehicle state parameters at the next moment; Repeat this cycle until the test case ends the test.

[0085] In an alternative implementation of some embodiments, the simulation system evaluation module 302 is further configured to: determine whether the simulation system is normal according to the test case, the control driving parameters at the current moment, and the vehicle state parameters at the current moment; and / or determine whether the simulation system is normal according to the test case, the control driving parameters at the next moment, and the vehicle state parameters at the next moment.

[0086] In an alternative implementation of some embodiments, the simulation system evaluation module 302 is further configured to: input the test case, the control driving parameters, and the vehicle state parameters into an observer to determine whether the simulation system is normal.

[0087] In an alternative implementation of some embodiments, the observer includes a system state observer, a scenario observer, and a communication diagnostic; and the simulation system evaluation module 302 is further configured to: input the control driving parameters and vehicle state parameters into the system state observer to determine whether the simulation system state is normal; input the test case and the parameters of the simulation scenario into the scenario observer to determine whether the simulation scenario is normal; input the preset communication protocol and the signal of the vehicle state parameters into the communication diagnostic to determine whether the communication is normal.

[0088] In an alternative implementation of some embodiments, the simulation system evaluation module 302 is further configured to: determine the predicted vehicle state parameters at the next moment according to the vehicle state parameters at the current moment and the control driving parameters at the current moment; obtain the vehicle state parameters at the next moment, and determine the deviation value between the vehicle state parameters at the next moment and the predicted vehicle state parameters at the next moment. If the deviation value does not meet the preset range, the simulation system state is abnormal.

[0089] In an alternative implementation of some embodiments, the test case includes a corresponding designed operating domain, and the simulation scenario also includes a corresponding designed operating domain. The designed operating domain is used to represent the key parameters of the driving scenario simulated by the test case; and the simulation system evaluation module 302 is further configured to: input the designed operating domain corresponding to the test case and the designed operating domain corresponding to the simulation scenario into the scenario observer to determine whether the simulation scenario is normal.

[0090] In an alternative implementation of some embodiments, the simulation system evaluation module 302 is further configured to: the communication diagnostic determines the transmission period, signal name, and check information of the signal according to the received signal of the vehicle state parameters, and checks whether the transmission period, signal name, and check information of the signal are normal through the preset communication protocol.

[0091] In an alternative implementation of some embodiments, the evaluation criteria include at least one of the following: comfort, reliability, fuel economy, and safety.

[0092] It can be understood that each module described in this device corresponds to each step in the method described with reference to Figure 1 Therefore, the operations, features, and beneficial effects described above for the method also apply to the device and the modules and units included therein, and will not be repeated here.

[0093] Figure 4 Illustrates a schematic diagram of the physical structure of an electronic device, such as Figure 4As shown in the figure, the electronic device may include: a processor 410, a communications interface 420, a memory 430, and a communication bus 440. Among them, the processor 410, the communications interface 420, and the memory 430 communicate with each other through the communication bus 440. The processor 410 can call the logical instructions in the memory 430 to execute an automated evaluation method for intelligent driving. The method includes: obtaining a test case, inputting the test case into an intelligent driving controller, determining the control driving parameters of the test case in a simulation scenario, inputting the control driving parameters into a vehicle model to obtain vehicle state parameters; determining whether the simulation system is normal according to the test case, the control driving parameters, and the vehicle state parameters; if it is normal, waiting for the test case to end the test to obtain a test result, and evaluating the test result according to a preset evaluation criterion.

[0094] In addition, when the logical instructions in the above-mentioned memory 430 are implemented in the form of software functional units and sold or used as an independent product, they can be stored in a computer-readable storage medium. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the above methods in various embodiments of the present invention. The foregoing storage medium includes: various media such as a USB flash drive, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a magnetic disk, or an optical disc that can store program codes.

[0095] On the other hand, the present invention also provides a computer program product. The above computer program product includes a computer program stored on a non-transitory computer-readable storage medium. The above computer program includes program instructions. When the above program instructions are executed by a computer, the computer can execute the automated evaluation method for intelligent driving provided by the above various methods. The method includes: obtaining a test case, inputting the test case into an intelligent driving controller, determining the control driving parameters of the test case in a simulation scenario, inputting the control driving parameters into a vehicle model to obtain vehicle state parameters; determining whether the simulation system is normal according to the test case, the control driving parameters, and the vehicle state parameters; if it is normal, waiting for the test case to end the test to obtain a test result, and evaluating the test result according to a preset evaluation criterion.

[0096] In another aspect, the present invention also provides a non-transitory computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, it is used to execute the above-mentioned automated evaluation method for intelligent driving, and the method includes: obtaining a test case, inputting the test case into an intelligent driving controller, determining the control driving parameters of the test case in a simulation scenario, inputting the control driving parameters into a vehicle model to obtain vehicle state parameters; determining whether the simulation system is normal according to the test case, the control driving parameters and the vehicle state parameters; if it is normal, waiting for the test case to end the test to obtain a test result, and evaluating the test result according to a preset evaluation criterion.

[0097] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed to multiple network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the solution of this embodiment. Those of ordinary skill in the art can understand and implement it without creative work.

[0098] Through the description of the above embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on this understanding, the essence of the above technical solution, or the part that contributes to the prior art, can be embodied in the form of a software product. The computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to execute each embodiment or some parts of the above-mentioned method.

[0099] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that they can still modify the technical solutions described in the foregoing embodiments, or perform equivalent replacements for some of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. An automated evaluation method for intelligent driving, characterized in that, Including: Obtain a test case, input the test case into the intelligent driving controller, determine the control driving parameters of the test case in the simulation scenario, and input the control driving parameters into the vehicle model to obtain vehicle state parameters; Determine whether the simulation system is normal according to the test case, the control driving parameters, and the vehicle state parameters; If it is normal, wait for the test case to end the test to obtain a test result, and evaluate the test result according to a preset evaluation criterion; Determine whether the simulation system is normal according to the test case, the control driving parameters, and the vehicle state parameters, including: Input the control driving parameters and the vehicle state parameters into the system state observer to determine whether the simulation system state is normal; Input the test case and the parameters of the simulation scenario into the scenario observer to determine whether the simulation scenario is normal; Input a preset communication protocol and the signal of the vehicle state parameters into the communication diagnostic to determine whether the communication is normal.

2. The automated evaluation method for intelligent driving according to claim 1, wherein The obtaining a test case, inputting the test case into the intelligent driving controller, and determining the control driving parameters of the test case in the simulation scenario includes: Obtain a test case, determine the parameters of the simulation scenario according to the test case, input the parameters of the simulation scenario and the test case into the intelligent driving controller, and determine the control driving parameters of the test case in the simulation scenario.

3. The automated evaluation method for intelligent driving according to claim 2, wherein The obtaining a test case, inputting the test case into the intelligent driving controller, determining the control driving parameters of the test case in the simulation scenario, and inputting the control driving parameters into the vehicle model to obtain vehicle state parameters includes: Obtain a test case, input the test case into the intelligent driving controller, determine the control driving parameters of the test case at the current moment in the simulation scenario, input the control driving parameters at the current moment into the vehicle model to obtain the vehicle state parameters at the current moment; Input the vehicle state parameters at the current moment into the intelligent driving controller, determine the control driving parameters at the next moment in the simulation scenario corresponding to the vehicle state parameters at the current moment, and input the control driving parameters at the next moment into the vehicle model to obtain the vehicle state parameters at the next moment; Repeat in this way until the test case ends the test.

4. The automated evaluation method for intelligent driving according to claim 3, wherein The determining whether the simulation system is normal according to the test case, the control driving parameters, and the vehicle state parameters includes: Determine whether the simulation system is normal according to the test case, the control driving parameters at the current moment, and the vehicle state parameters at the current moment; and / or Determine whether the simulation system is normal according to the test case, the control driving parameters at the next moment, and the vehicle state parameters at the next moment.

5. The automated evaluation method for intelligent driving according to claim 4, wherein The inputting the control driving parameters and the vehicle state parameters into the system state observer to determine whether the simulation system state is normal includes: Determine the predicted vehicle state parameters at the next moment according to the vehicle state parameters at the current moment and the control driving parameters at the current moment; Obtain the vehicle state parameters at the next moment, determine the deviation value between the vehicle state parameters at the next moment and the predicted vehicle state parameters at the next moment. If the deviation value does not meet the preset range, the simulation system state is abnormal.

6. The automated evaluation method for intelligent driving according to claim 1, characterized in that, The test case includes a corresponding design operation domain, and the simulation scenario also includes a corresponding design operation domain. The design operation domain is used to represent the key parameters of the driving scenario simulated by the test case; And, The step of inputting the test case and the simulation scenario into the scenario observer to determine whether the simulation scenario is normal includes: Input the design operation domain corresponding to the test case and the design operation domain corresponding to the simulation scenario into the scenario observer to determine whether the simulation scenario is normal.

7. The automated evaluation method for intelligent driving according to claim 1, wherein The step of inputting the preset communication protocol and the signal of the vehicle state parameters into the communication diagnostic to determine whether the communication is normal includes: The communication diagnostic determines the transmission period, signal name, and check information of the signal according to the received signal of the vehicle state parameters, and checks whether the transmission period, signal name, and check information of the signal are normal through the preset communication protocol.

8. The automated evaluation method for intelligent driving according to claim 1, wherein, The evaluation criteria include at least one of the following: comfort, reliability, fuel economy, and safety.

9. An automated evaluation device for intelligent driving, characterized in that, Include: A simulation module, configured to obtain a test case, input the test case into an intelligent driving controller, determine the control driving parameters of the test case in the simulation scenario, and input the control driving parameters into a vehicle model to obtain vehicle state parameters; A simulation system evaluation module, configured to determine whether the simulation system is normal according to the test case, the control driving parameters, and the vehicle state parameters; Determining whether the simulation system is normal according to the test case, the control driving parameters, and the vehicle state parameters includes: inputting the control driving parameters and the vehicle state parameters into a system state observer to determine whether the simulation system state is normal; inputting the parameters of the test case and the simulation scenario into a scenario observer to determine whether the simulation scenario is normal; inputting the preset communication protocol and the signal of the vehicle state parameters into a communication diagnostic to determine whether the communication is normal; An intelligent driving controller evaluation module, configured to, if it is normal, wait for the test case to end the test to obtain a test result, and evaluate the test result according to the preset evaluation criteria.

10. An electronic device, comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the steps of the automated evaluation method for intelligent driving according to any one of claims 1 to 8.

11. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the steps of the automated evaluation method for intelligent driving according to any one of claims 1 to 8.

12. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the steps of the automated evaluation method for intelligent driving according to any one of claims 1 to 8.

Images