Simulation evaluation method and device, electronic equipment and storage medium

Abstract

The application relates to the technical field of automation simulation, in particular to a simulation evaluation method and device, a vehicle and a storage medium, wherein the method comprises the following steps: obtaining a target test function of a software to be tested; calling a corresponding test case according to the target test function, loading the test case to a simulation model, and performing simulation test on the target test function by using the simulation model; and generating a test result of the target test function according to test data in the simulation test process. Therefore, the problems of low work efficiency, time-consuming and labor-consuming caused by manual test in the related art are solved.

Description

Technical Field

[0001] This application relates to the field of automated simulation technology, and in particular to a simulation evaluation method, device, electronic device and storage medium. Background Technology

[0002] Testing, verification, and rapid, effective iteration of intelligent driving technologies are crucial for advancing these technologies. Common testing and verification methods include on-road testing with real vehicles and simulation testing. However, on-road testing is time-consuming, labor-intensive, and lacks specificity, while manual testing is inefficient. Summary of the Invention

[0003] This application provides a simulation testing method, apparatus, electronic device, and storage medium to solve the problems of low work efficiency, time-consuming and labor-intensive work caused by manual testing in related technologies.

[0004] The first aspect of this application provides a simulation testing method, comprising the following steps: obtaining the target test function of the software under test; invoking corresponding test cases according to the target test function, loading the test cases into a simulation model, and using the simulation model to perform simulation testing on the target test function; generating test results for the target test function based on the test data from the simulation testing process.

[0005] Optionally, the target testing function includes one or more of the following: perception function, prediction function, planning function, and control function.

[0006] Optionally, the simulation model includes one or more of a vehicle dynamics model, a sensor model, a driver model, and a scene rendering model.

[0007] Optionally, generating the test results of the target test function based on the test data from the simulation test process includes: acquiring the simulation test data of the simulation model in the simulation test process; and performing online evaluation on the simulation test data to obtain the test results of the target test function.

[0008] Optionally, generating the test results of the target test function based on the test data from the simulation test process further includes: acquiring the simulation test data of the simulation model in the simulation test process; and performing offline evaluation on the simulation test data to obtain the test results of the target test function.

[0009] Optionally, during the simulation test of the target test function using the simulation model, the process further includes: setting the scene information and vehicle pose information of the simulation model; and controlling the vehicle to perform preset actions based on the target signal emitted by the software under test.

[0010] Optionally, after generating the test results of the target test function based on the test data from the simulation test process, the method includes: sending the test results to a preset terminal.

[0011] A second aspect of this application provides a simulation testing device, comprising: an acquisition module for acquiring a target test function of the software under test; an invocation module for invoking corresponding test cases according to the target test function, loading the test cases into a simulation model, and using the simulation model to perform simulation testing on the target test function; and a generation module for generating test results of the target test function based on test data from the simulation testing process.

[0012] Optionally, the target testing function includes one or more of the following: perception function, prediction function, planning function, and control function.

[0013] Optionally, the simulation model includes one or more of a vehicle dynamics model, a sensor model, a driver model, and a scene rendering model.

[0014] Optionally, the generation module is further configured to: acquire simulation test data of the simulation model during the simulation test process; and perform online evaluation on the simulation test data to obtain the test results of the target test function.

[0015] Optionally, the generation module is further configured to: acquire simulation test data of the simulation model during the simulation test process; and perform offline evaluation on the simulation test data to obtain test results of the target test function.

[0016] Optionally, the calling module is further used to: set the scene information and vehicle pose information of the simulation model; and control the vehicle to perform preset actions based on the target signal emitted by the software under test.

[0017] Optionally, it further includes: a sending module, used to send the test results to a preset terminal after generating the test results of the target test function based on the test data of the simulation test process.

[0018] A third aspect of this application provides an electronic device, including: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the simulation evaluation method as described in the above embodiments.

[0019] A fourth aspect of this application provides a computer-readable storage medium having a computer program stored thereon, which is executed by a processor to implement the simulation evaluation method as described in the above embodiments.

[0020] Therefore, this application has at least the following beneficial effects:

[0021] The embodiments of this application can call the corresponding test cases according to the target test functions of the software under test, load them into the simulation model, use the simulation model to perform simulation testing on the target test functions of the software under test, and generate test results, thereby realizing automated testing without the need for manual or real vehicle testing, effectively improving testing efficiency.

[0022] Additional aspects and advantages of this application will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of this application. Attached Figure Description

[0023] The above and / or additional aspects and advantages of this application will become apparent and readily understood from the following description of the embodiments taken in conjunction with the accompanying drawings, wherein:

[0024] Figure 1 This is a flowchart of a simulation evaluation method provided according to an embodiment of this application;

[0025] Figure 2 This is a schematic diagram of the overall composition structure of the simulation evaluation provided according to the embodiments of this application;

[0026] Figure 3 This is a schematic diagram of the overall data flow of the in-loop automation simulation evaluation method provided according to the embodiments of this application;

[0027] Figure 4 This is a schematic diagram of the overall process of the in-loop automation simulation evaluation method provided in the embodiments of this application;

[0028] Figure 5 This is a schematic diagram of an online evaluation provided according to an embodiment of this application;

[0029] Figure 6 This is a schematic diagram of offline evaluation provided according to an embodiment of this application;

[0030] Figure 7 This is an example diagram of a simulation testing device according to an embodiment of this application;

[0031] Figure 8 This is a schematic diagram of the structure of an electronic device according to an embodiment of this application. Detailed Implementation

[0032] The embodiments of this application are described in detail below. Examples of these embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and intended to explain this application, and should not be construed as limiting this application.

[0033] The simulation evaluation method, apparatus, electronic device, and storage medium of this application are described below with reference to the accompanying drawings. Addressing the problems of low efficiency and time-consuming nature caused by manual testing mentioned in the background art, this application provides a simulation evaluation method. In this method, corresponding test cases are invoked according to the target test function of the software under test, and these cases are loaded into a simulation model. The simulation model is then used to perform simulation testing on the target test function of the software under test, and test results are generated. This enables automated testing and effectively improves testing efficiency. Therefore, it solves the problems of low efficiency and time-consuming nature caused by manual testing in related technologies.

[0034] Specifically, Figure 1 This is a flowchart illustrating a simulation evaluation method provided in an embodiment of this application.

[0035] like Figure 1 As shown, this simulation evaluation method includes the following steps:

[0036] In step S101, the target test function of the software under test is obtained.

[0037] The target testing functions include one or more of the following: perception, prediction, planning, and control functions.

[0038] It is understood that the embodiments of this application can obtain the target test functions of the software under test, so as to facilitate subsequent simulation testing of the target test functions.

[0039] In step S102, the corresponding test cases are called according to the target test function, and the test cases are loaded into the simulation model. The simulation model is then used to perform simulation testing on the target test function.

[0040] The simulation model includes one or more of the following: vehicle dynamics model, sensor model, driver model, and scene rendering model.

[0041] It is understood that, according to the embodiments of this application, the corresponding test cases can be called according to the target test function and loaded into the simulation model. The simulation model is used to perform simulation testing on the target test function so as to generate the test results of the target test function based on the test data of the simulation test process.

[0042] It should be noted that the test case library is designed based on the driver assistance function. The functions include both horizontal and vertical functions, covering warning and action functions. The test case library for smoke test and full test is constructed by analyzing the basic elements of the scenario.

[0043] In this embodiment of the application, the process of using a simulation model to perform simulation testing on the target test function further includes: setting the scene information and vehicle pose information of the simulation model; and controlling the vehicle to perform preset actions based on the target signal emitted by the software under test.

[0044] The preset actions can be actions such as turning, moving forward, or reversing, and can be adjusted according to the actual situation without specific limitations.

[0045] It is understood that the embodiments of this application can set the scene information and vehicle pose information of the simulation model, thereby rendering the scene and vehicle data of the entire simulation model, and controlling the vehicle to perform preset actions based on the target signal emitted by the software under test, so as to achieve automated testing.

[0046] Specifically, communication is established between the desktop runtime environment of the software under test and the VTD, transmitting the scene information and vehicle pose information of the VTD to the software under test, and simultaneously providing the vehicle control signals calculated by the software under test to the vehicle dynamics model of the VTD.

[0047] In step S103, test results for the target test function are generated based on the test data from the simulation test process.

[0048] It is understood that the embodiments of this application can generate test results of the target test function based on the test data of the simulation test process, thereby realizing automated testing without the need for manual or actual vehicle testing, and effectively improving testing efficiency.

[0049] In this embodiment of the application, generating test results for the target test function based on test data from the simulation test process includes: acquiring simulation test data of the simulation model in the simulation test process; and performing online evaluation on the simulation test data to obtain test results for the target test function.

[0050] It is understood that the embodiments of this application can perform online evaluation of simulation test data to obtain test results for the target test function.

[0051] It should be noted that online evaluation rules are established to determine whether the current test has ended, mainly considering dangerous situations such as collisions or veering off the road.

[0052] Specifically, the test cases to be tested are selected based on the software functions, and the test cases in the test case library are loaded one by one; the software and VTD (Virtual Test Drive) in the desktop environment are started to confirm that the functional software is running normally, the VTD is executing the preset test scenarios normally, and real-time communication is normal; during the operation of the software under test and the VTD, the running status is monitored in real time. If dangerous situations such as collisions or deviations from the road occur, or if the test case termination conditions are met, such as meeting time, mileage, speed, etc., the current test is terminated; the data of the current test is stored locally for subsequent offline evaluation; after the current test case ends, it is determined whether there are any test cases that need to be tested. If so, the test case is loaded, tested, and the results are saved until all tests are completed.

[0053] In this embodiment of the application, generating test results for the target test function based on test data from the simulation test process further includes: obtaining simulation test data of the simulation model in the simulation test process; and performing offline evaluation on the simulation test data to obtain test results for the target test function.

[0054] It is understood that the embodiments of this application obtain the test results of the target test function by performing offline evaluation on simulation test data.

[0055] It should be noted that offline evaluation rules are established for different functions and scenarios. Based on the different functions, safety evaluation indicators, comfort evaluation indicators, and compliance evaluation indicators are set, including sub-indicators such as whether a collision, sudden braking, swerving, or shaking occurs. After the online test is completed, the offline evaluation module is called to read the test result data in sequence and determine whether the test result is passed according to the evaluation rules. Then, the results are filled into the test report template.

[0056] Specifically, the test result data saved in the online test is loaded one by one; the scenario pass criteria are loaded according to the test item category, and relevant items are extracted from the data according to the indicators designed in the pass criteria; it is determined whether the data meets the scenario pass criteria and the results are stored in the test report; it is determined whether there are any data items that still need to be evaluated, and if so, the evaluation process is carried out until all data evaluation is completed; the overall pass status is statistically analyzed, the results are filled into the test report, and stored on the cloud platform.

[0057] In this embodiment of the application, after generating the test results of the target test function based on the test data of the simulation test process, the method includes: sending the test results to a preset terminal.

[0058] The preset terminal can be a computer terminal or a mobile phone terminal, without specific restrictions.

[0059] It is understood that, in this embodiment of the application, after generating test results, the results are sent to a preset terminal so that the software and test case library can be updated based on these results, and iterative updates can be completed.

[0060] According to the simulation evaluation method proposed in the embodiments of this application, the corresponding test cases are called according to the target test function of the software under test, and loaded into the simulation model. The simulation model is used to perform simulation testing on the target test function of the software under test and generate test results, which can realize automated testing and effectively improve testing efficiency.

[0061] The following will combine Figures 2 to 6 The simulation evaluation method of this application is described in detail, and the specific steps are as follows:

[0062] Step 1: Design a test case library based on the driver assistance functions. The functions include horizontal and vertical functions, covering warning and action functions. By analyzing the basic elements of the scenario composition, construct a test case library for smoke testing and full-scale testing.

[0063] Step 2: Establish offline evaluation rules for different functions and scenarios. Based on the different functions, set safety evaluation indicators, comfort evaluation indicators, and compliance evaluation indicators, including sub-indicators such as whether a collision occurred, sudden braking, erratic movement, or vibration. Simultaneously, establish online evaluation rules to determine whether the current test has ended, primarily considering dangerous situations such as collisions or veering off the road.

[0064] Step 3: Establish communication between the desktop runtime environment of the software under test and the VTD, transmit the scene information and vehicle pose information of the VTD to the software under test, and at the same time, transmit the vehicle control signals calculated by the software under test to the vehicle dynamics model of the VTD.

[0065] Deploy the Genkins service so that when developers update the software code, the updated code is automatically downloaded to the test server and integrated into the test environment.

[0066] Step 4: The automated testing module integrates the latest version of the software under test into the testing environment. At the same time, it calls different test case sets according to the functions of the software under test and loads them into the VTD in sequence. It synchronously calls the runtime environment of the software under test to execute online model testing and stores the test data in local files.

[0067] If a serious hazard such as a collision or deviation from the road occurs during the test, or if the test's termination conditions are met, such as satisfying time, distance, or speed requirements, then the test will terminate. After the test concludes, check if there are any remaining test cases to execute. If so, perform the tests; otherwise, conduct offline evaluation.

[0068] Select the test cases to be tested based on the software functions, and load the test cases from the test case library one by one; start the software and VTD in the desktop environment, and confirm that the functional software is running normally, the VTD is executing the preset test scenarios normally, and real-time communication is normal; during the operation of the software under test and the VTD, monitor the running status in real time. If dangerous situations such as collisions or deviations from the road occur, or if the test case termination conditions are met, such as time, mileage, speed, etc., the current test ends; store the current test data locally for subsequent offline evaluation; after the current test case ends, determine whether there are any test cases that need to be tested. If so, load the test case, perform the test, and save the results until all tests are completed.

[0069] Step 5: After the online test is completed, call the offline evaluation module to read the test result data in sequence and determine whether the test result is passed according to the evaluation rules. Then fill the results into the test report template.

[0070] Load the test result data saved in the online test one by one; load the scenario pass criteria according to the test item category, and extract the relevant items from the data according to the indicators designed in the pass criteria; determine whether the data meets the scenario pass criteria, and store the results in the test report; determine whether there are any data items that still need to be evaluated, and if so, load the evaluation process until all data evaluation is completed; summarize the overall pass status, fill the results in the test report, and store them on the cloud platform.

[0071] Step 6: After the test report is generated, upload the report to the cloud platform and notify the testers and developers via email. This method can quickly help developers check for software logic errors during the software development process, effectively reduce the workload of testers, and accelerate software iteration efficiency.

[0072] In summary, this application addresses the issue of inaccurate vehicle models by introducing professional vehicle dynamics software, resulting in simulation results that more closely resemble actual performance. The automated testing and evaluation modules effectively improve testing efficiency and consistency, avoiding the time developers would otherwise spend on testing and enhancing software iteration efficiency. Compared to on-road testing by personnel driving test vehicles, this method is not limited by personnel, vehicles, or time constraints. After deploying the automated testing framework, software versions can be monitored in real time, and test tasks are automatically triggered and test reports generated when software updates occur, saving manpower and resources. Each test is executed automatically and results are stored, ensuring good test consistency and facilitating problem reproduction and analysis. Furthermore, it allows for comparison of results with historical software versions, enabling the statistical analysis of the number of issues and additions / closures during software iteration.

[0073] Next, the simulation evaluation device proposed according to the embodiments of this application is described with reference to the accompanying drawings.

[0074] Figure 7 This is a block diagram of the simulation evaluation device according to an embodiment of this application.

[0075] like Figure 7 As shown, the simulation evaluation device 10 includes: an acquisition module 100, a calling module 200, and a generation module 300.

[0076] The acquisition module 100 is used to acquire the target test function of the software under test; the invocation module 200 is used to invoke the corresponding test cases according to the target test function, load the test cases into the simulation model, and use the simulation model to perform simulation testing on the target test function; the generation module 300 is used to generate the test results of the target test function based on the test data of the simulation test process.

[0077] In this application embodiment, the target testing function includes one or more of the following: perception function, prediction function, planning function, and control function.

[0078] In this embodiment of the application, the simulation model includes one or more of the following: a vehicle dynamics model, a sensor model, a driver model, and a scene rendering model.

[0079] In this embodiment of the application, the generation module 300 is further used to: acquire simulation test data of the simulation model in the simulation test process; and perform online evaluation of the simulation test data to obtain the test results of the target test function.

[0080] In this embodiment of the application, the generation module 300 is further configured to: acquire simulation test data of the simulation model in the simulation test process; and perform offline evaluation on the simulation test data to obtain test results of the target test function.

[0081] In this embodiment, the calling module 200 is further used to: set the scene information and vehicle pose information of the simulation model; and control the vehicle to perform preset actions based on the target signal emitted by the software under test.

[0082] In this embodiment of the application, it further includes: a sending module, used to send the test results to a preset terminal after generating the test results of the target test function based on the test data of the simulation test process.

[0083] It should be noted that the foregoing explanation of the simulation evaluation method embodiment also applies to the simulation evaluation device of this embodiment, and will not be repeated here.

[0084] According to the simulation testing device proposed in the embodiments of this application, the corresponding test cases are called according to the target test function of the software under test, and loaded into the simulation model. The simulation model is used to perform simulation testing on the target test function of the software under test and generate test results, which can realize automated testing and effectively improve testing efficiency.

[0085] Figure 8 A schematic diagram of the structure of an electronic device provided in an embodiment of this application. The electronic device may include:

[0086] The memory 801, the processor 802, and the computer program stored on the memory 801 and capable of running on the processor 802.

[0087] When the processor 802 executes the program, it implements the simulation evaluation method provided in the above embodiments.

[0088] Furthermore, electronic devices also include:

[0089] Communication interface 803 is used for communication between memory 801 and processor 802.

[0090] The memory 801 is used to store computer programs that can run on the processor 802.

[0091] The memory 801 may include high-speed RAM (Random Access Memory) memory, and may also include non-volatile memory, such as at least one disk storage.

[0092] If the memory 801, processor 802, and communication interface 803 are implemented independently, then the communication interface 803, memory 801, and processor 802 can be interconnected via a bus to complete communication between them. The bus can be an ISA (Industry Standard Architecture) bus, a PCI (Peripheral Component Interconnect) bus, or an EISA (Extended Industry Standard Architecture) bus, etc. The bus can be divided into address bus, data bus, control bus, etc. For ease of representation, Figure 8 The bus is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.

[0093] Optionally, in a specific implementation, if the memory 801, processor 802, and communication interface 803 are integrated on a single chip, then the memory 801, processor 802, and communication interface 803 can communicate with each other through an internal interface.

[0094] The processor 802 may be a CPU (Central Processing Unit), an ASIC (Application Specific Integrated Circuit), or one or more integrated circuits configured to implement the embodiments of this application.

[0095] This application also provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the simulation evaluation method described above.

[0096] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of this application. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0097] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this application, "N" means at least two, such as two, three, etc., unless otherwise explicitly specified.

[0098] Any process or method described in the flowchart or otherwise herein can be understood as representing a module, segment, or portion of code comprising one or N executable instructions for implementing custom logic functions or processes, and the scope of the preferred embodiments of this application includes additional implementations in which functions may be performed not in the order shown or discussed, including substantially simultaneously or in reverse order depending on the functions involved, as should be understood by those skilled in the art to which embodiments of this application pertain.

[0099] It should be understood that the various parts of this application can be implemented using hardware, software, firmware, or a combination thereof. In the above embodiments, the N steps or methods can be implemented using software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, it can be implemented using any one or more of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (FPGAs), field-programmable gate arrays (FPGAs), etc.

[0100] Those skilled in the art will understand that all or part of the steps of the methods in the above embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, the program includes one or a combination of the steps of the method embodiments.

[0101] Although embodiments of this application have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting this application. Those skilled in the art can make changes, modifications, substitutions and variations to the above embodiments within the scope of this application.

Claims

1. A simulation evaluation method, characterized in that, Includes the following steps: Obtain the target test functions of the software under test, wherein the target test functions include one or more of the following: perception function, prediction function, planning function, and control function; The corresponding test cases are invoked according to the target test function, and the test cases are loaded into the simulation model. The simulation model is used to perform simulation testing on the target test function. The simulation model includes one or more of the following: vehicle dynamics model, sensor model, driver model, and scene rendering model. Generating test results for the target test function based on test data from the simulation testing process, wherein generating test results for the target test function based on test data from the simulation testing process includes: acquiring simulation test data of the simulation model in the simulation testing process; performing online evaluation on the simulation test data to obtain test results for the target test function, and / or, acquiring simulation test data of the simulation model in the simulation testing process; performing offline evaluation on the simulation test data to obtain test results for the target test function.

2. The simulation evaluation method according to claim 1, characterized in that, The process of simulating and testing the target test function using the simulation model also includes: Set the scene information and vehicle pose information of the simulation model; The software under test controls the vehicle to perform preset actions based on the target signal emitted by the software.

3. The simulation evaluation method according to claim 1, characterized in that, After generating the test results for the target test function based on the test data from the simulation test process, the process includes: The test results are sent to a preset terminal.

4. A simulation testing device, characterized in that, include: The acquisition module is used to acquire the target test functions of the software under test, wherein the target test functions include one or more of the following: perception function, prediction function, planning function, and control function. The calling module is used to call the corresponding test cases according to the target test function, load the test cases into the simulation model, and use the simulation model to perform simulation testing on the target test function. The simulation model includes one or more of the following: vehicle dynamics model, sensor model, driver model, and scene rendering model. A generation module is used to generate test results for the target test function based on test data from the simulation test process. The generation of test results for the target test function based on test data from the simulation test process includes: acquiring simulation test data of the simulation model in the simulation test process; performing online evaluation on the simulation test data to obtain test results for the target test function; and / or, acquiring simulation test data of the simulation model in the simulation test process; and performing offline evaluation on the simulation test data to obtain test results for the target test function.

5. An electronic device, characterized in that, include: A memory, a processor, and a computer program stored in the memory and executable on the processor, the processor executing the program to implement the simulation evaluation method as described in any one of claims 1-3.

6. A computer-readable storage medium having a computer program stored thereon, characterized in that, The program is executed by the processor to implement the simulation evaluation method as described in any one of claims 1-3.

Images