Driving software testing method and device, storage medium and electronic device

Abstract

The invention provides a driving software testing method and device, a storage medium and an electronic device.The method comprises the steps that vehicle driving data of driving domain controllers of multiple vehicle ends are obtained, and the driving domain controllers are used for operating target driving software to be tested; test cases of the multiple driving scenes are generated by adopting the vehicle driving data, evaluation cases are generated according to the test cases, and the evaluation cases are used for evaluating test results of the test cases; creating a test task of the target driving software based on the test case and the evaluation case; the test task is issued to the rack cluster, an evaluation report fed back after the rack cluster executes the test task is received, and the rack cluster is used for conducting data recharging based on the test case and evaluating the recharged data based on the evaluation case. Through the embodiment of the invention, the technical problem of low driving software testing efficiency in related technologies is solved, the testing efficiency is greatly improved, the testing period is remarkably shortened, and the testing cost is effectively reduced.

Description

Technical Field

[0001] The present invention relates to the field of autonomous driving technology, and in particular to a testing method and device for driving software, a storage medium, and an electronic device. Background Art

[0002] In related technologies, with the rapid development of autonomous driving technology, intelligent driving systems are gradually being applied on actual roads. In order to ensure the safety and reliability of intelligent driving systems, a large amount of testing and evaluation is required. However, traditional testing methods often rely on manual testing and verification of a single scenario, which are inefficient, have a small coverage, low reliability, long testing cycles, and high costs, making it difficult to meet the increasingly complex testing needs of autonomous driving applications. At present, the testing systems in the industry are usually decentralized and lack an integrated solution. Vehicle-side data collection, cloud-based data processing and analysis, test task management, and execution are often independent systems, resulting in information islands and inefficiency.

[0003] With respect to the above-mentioned problems existing in the related technologies, no efficient and accurate solutions have been found yet. Summary of the invention

[0004] The present invention provides a driving software testing method and device, a storage medium, and an electronic device to solve technical problems in related technologies.

[0005] According to one embodiment of the present invention, a method for testing driving software is provided, comprising: obtaining vehicle driving data from multiple vehicle-side driving domain controllers, wherein the driving domain controller is used to run a target driving software to be tested; using the vehicle driving data to generate test cases for multiple driving scenarios, and generating evaluation cases based on the test cases, wherein the evaluation cases are used to evaluate the test results of the test cases; creating a test task for the target driving software based on the test cases and the evaluation cases; issuing the test task to a test bench cluster, and receiving an evaluation report fed back by the test bench cluster after executing the test task, wherein the test bench cluster is used to perform data re-injection based on the test case and to evaluate the re-injected data based on the evaluation case.

[0006] Optionally, obtaining vehicle driving data from multiple vehicle-side driving domain controllers includes: obtaining the following vehicle driving data transmitted by multiple vehicle-side data acquisition modules: vehicle control instructions, vehicle status data, and sensor data, wherein the data acquisition module is communicatively connected with the vehicle-side driving domain controller.

[0007] Optionally, using the vehicle driving data to generate test cases for multiple driving scenarios includes: performing data cleaning and scene classification on the vehicle driving data to obtain multiple sets of driving scene data; and responding to a first creation request from a user, using the multiple sets of driving scene data to generate test cases for multiple driving scenarios.

[0008] Optionally, the vehicle driving data is cleaned and scene classified to obtain multiple sets of driving scene data, including: cleaning the vehicle driving data to obtain intermediate data; clustering the intermediate data to obtain multiple data clusters, wherein each data cluster corresponds to a driving scene category; and classifying each data cluster in the multiple data clusters to obtain multiple sets of driving scene data.

[0009] Optionally, classifying each of the multiple data clusters includes: for each data cluster among the multiple data clusters, extracting key features in the data cluster, wherein the key features include at least one of the following: road type, traffic signs, environmental objects, weather features; and classifying the data cluster into multiple driving scenario items based on the key features.

[0010] Optionally, using the multiple driving scenario data to generate test cases corresponding to multiple driving scenarios includes: determining a target test scenario according to the first creation request; searching for target test data for the target test scenario from the multiple driving scenario data; configuring a software package of the target driving software and a bench configuration file of the target test scenario, wherein the bench configuration file is used to define reinjection configuration items; taking the target driving software as a test object, and generating a target test case for the target test scenario using the target test data and the bench configuration file.

[0011] Optionally, using the target test data and the bench configuration file to generate a target test case for the target test scenario includes: obtaining a test case template for the target test scenario, wherein the test case template includes typical input data and expected output results under the target test scenario; randomly adjusting the target test data within a data range of the typical input data to obtain multiple groups of use case data under the target test scenario; and using the multiple groups of use case data and the bench configuration file to generate a plurality of target test cases for the target test scenario.

[0012] Optionally, generating an evaluation case based on the test case includes: responding to the user's second creation request, configuring evaluation indicators and evaluation rules of the test results according to the test case, and configuring operation dependency parameters; and generating an evaluation case using the evaluation indicators, evaluation rules and the operation dependency parameters.

[0013] Optionally, configuring evaluation indicators and evaluation rules for test results according to the test case includes: determining key scenario data of the driving scenario corresponding to the test case; and searching for evaluation indicators and evaluation rules that match the key scenario data in a preset evaluation database.

[0014] Optionally, using the vehicle driving data to generate test cases for multiple driving scenarios includes: obtaining historical evaluation reports of historical test tasks of the target driving software; locating abnormal parameters to be optimized in the historical evaluation reports; iteratively updating template parameters in the test case template based on the abnormal parameters; and using the vehicle driving data and the updated test case template to generate test cases for multiple driving scenarios of the current test task.

[0015] Optionally, sending the test task to the rack cluster includes: obtaining operating status information of each rack in the rack cluster; using the operating status information to calculate the task waiting time and task execution time of each rack; using the task waiting time and the task execution time to calculate the response ratio of each rack; configuring the priority of each rack in the rack cluster based on the response ratio, and sending the test task to the racks in the rack cluster according to the priority.

[0016] According to another embodiment of the present invention, a testing device for driving software is provided, including: an acquisition module, used to acquire vehicle driving data of multiple vehicle-side driving domain controllers, wherein the driving domain controller is used to run the target driving software to be tested; a generation module, used to generate test cases for multiple driving scenarios using the vehicle driving data, and generate evaluation cases based on the test cases, wherein the evaluation cases are used to evaluate the test results of the test cases; a creation module, used to create a test task for the target driving software based on the test cases and the evaluation cases; a testing module, used to issue the test task to a test bench cluster, and receive an evaluation report fed back by the test bench cluster after executing the test task, wherein the test bench cluster is used to perform data re-injection based on the test case and to evaluate the re-injected data based on the evaluation case.

[0017] Optionally, the acquisition module includes: an acquisition unit, used to acquire the following vehicle driving data transmitted by multiple vehicle-side data acquisition modules: vehicle control instructions, vehicle status data, and sensor data, wherein the data acquisition module is communicatively connected to the vehicle-side driving domain controller.

[0018] Optionally, the generation module includes: a classification unit, used to perform data cleaning and scene classification on the vehicle driving data to obtain multiple driving scene data; a first generation unit, used to respond to a user's first creation request and use the multiple driving scene data to generate test cases corresponding to multiple driving scenarios.

[0019] Optionally, the classification unit includes: a cleaning subunit, used to clean the vehicle driving data to obtain intermediate data; a clustering subunit, used to cluster the intermediate data to obtain multiple data clusters, wherein each data cluster corresponds to a driving scene category; and a classification subunit, used to classify each data cluster in the multiple data clusters to obtain multiple driving scene data.

[0020] Optionally, the classification subunit is also used to: extract key features in each data cluster among the multiple data clusters, wherein the key features include at least one of the following: road type, traffic signs, environmental objects, weather characteristics; and classify the data cluster into multiple driving scene items based on the key features.

[0021] Optionally, the first generation unit includes: a determination subunit, used to determine the target test scenario according to the first creation request; a search subunit, used to search for target test data of the target test scenario from the multiple driving scenario data; a configuration subunit, used to configure the software package of the target driving software and the bench configuration file of the target test scenario, wherein the bench configuration file is used to define the reinjection configuration items; a generation subunit, used to take the target driving software as the test object, and use the target test data and the bench configuration file to generate a target test case for the target test scenario.

[0022] Optionally, the generation subunit is also used to: obtain a test case template for the target test scenario, wherein the test case template includes typical input data and expected output results under the target test scenario; randomly adjust the target test data within the data range of the typical input data to obtain multiple sets of use case data under the target test scenario; and use the multiple sets of use case data and the bench configuration file to generate a plurality of target test cases for the target test scenario.

[0023] Optionally, the generation module includes: a configuration unit, used to respond to the user's second creation request, configure the evaluation indicators and evaluation rules of the test results according to the test case, and configure the operation dependency parameters; a second generation unit, used to generate an evaluation case using the evaluation indicators, evaluation rules and the operation dependency parameters.

[0024] Optionally, the configuration unit includes: a determination subunit, used to determine key scenario data of the driving scenario corresponding to the test case; and a search subunit, used to search for evaluation indicators and evaluation rules that match the key scenario data in a preset evaluation database.

[0025] Optionally, the generation module includes: an acquisition unit, used to acquire historical evaluation reports of historical test tasks of the target driving software; a positioning unit, used to locate abnormal parameters to be optimized in the historical evaluation report; an updating unit, used to iteratively update template parameters in the test case template based on the abnormal parameters; and a third generation unit, used to use the vehicle driving data and the updated test case template to generate test cases for multiple driving scenarios of the current test task.

[0026] Optionally, the test module includes: an acquisition unit, used to acquire the operating status information of each rack in the rack cluster; a first calculation unit, used to use the operating status information to calculate the task waiting time and task execution time of each rack; a second calculation unit, used to use the task waiting time and the task execution time to calculate the response ratio of each rack; and a sending unit, used to configure the priority of each rack in the rack cluster based on the response ratio, and send the test task to the racks in the rack cluster according to the priority.

[0027] According to another aspect of an embodiment of the present application, a storage medium is further provided, which includes a stored program, and the above steps are executed when the program is run.

[0028] According to another aspect of an embodiment of the present application, there is also provided an electronic device, including a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other via the communication bus; wherein: the memory is used to store computer programs; and the processor is used to execute the steps in the above method by running the program stored in the memory.

[0029] The embodiment of the present application also provides a computer program product including instructions, which, when executed on a computer, enables the computer to execute the steps in the above method.

[0030] Beneficial effects of the present invention:

[0031] 1. Through deep integration of multiple links of the test process, the test efficiency is greatly improved, the test cycle is significantly shortened, and the test cost is effectively reduced;

[0032] 2. Efficient data collection and secure transmission mechanism on the vehicle side ensures the integrity, accuracy and availability of data, providing a solid data foundation for subsequent testing;

[0033] 3. The cloud's powerful data processing capabilities and accurate scene classification ensure data quality and create favorable conditions for the automatic generation and optimization of test cases and evaluation cases;

[0034] 4. Automatically generate and optimize test cases and evaluation cases to significantly reduce manual intervention, improve the quality, pertinence and effectiveness of test cases and evaluation cases, and enhance the comprehensive testing and evaluation capabilities of intelligent driving systems;

[0035] 5. The task scheduling algorithm allocates resources reasonably, further improving the overall test efficiency and ensuring the balanced execution of test tasks;

[0036] 6. Comprehensive and iteratively optimized testing process covers a variety of testing businesses, greatly improving the coverage of test scenarios and the accuracy of results. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] The drawings described herein are used to provide a further understanding of the present invention and constitute a part of this application. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the drawings:

[0038] Figure 1 is a hardware structure block diagram of a server according to an embodiment of the present invention;

[0039] Figure 2 is a flow chart of a driving software testing method according to an embodiment of the present invention;

[0040] Figure 3 is a schematic diagram of the architecture of an embodiment of the present invention;

[0041] Figure 4 It is a schematic diagram of the operation flow of the intelligent driving test system in an embodiment of the present invention;

[0042] Figure 5 It is a structural block diagram of a driving software testing device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0043] In order to enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only embodiments of a part of the present application, not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the field without creative work should fall within the scope of protection of the present application. It should be noted that the embodiments in the present application and the features in the embodiments can be combined with each other without conflict.

[0044] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0045] Example 1

[0046] The method embodiment provided in the first embodiment of the present application can be executed in a server, a car, a processor, or a similar processing device. Taking running on a server as an example, Figure 1 FIG. 1 is a hardware structure diagram of a server according to an embodiment of the present invention. Figure 1 As shown, the server may include one or more ( Figure 1 Only one is shown in the figure) a processor 102 (the processor 102 may include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA) and a memory 104 for storing data. Optionally, the server may also include a transmission device 106 and an input / output device 108 for communication functions. It can be understood by those skilled in the art that Figure 1 The structure shown is only for illustration and does not limit the structure of the above server. Figure 1 More or fewer components as shown, or with Figure 1 Different configurations shown.

[0047] The memory 104 can be used to store server programs, for example, software programs and modules of application software, such as a server program corresponding to a testing method of driving software of a server in an embodiment of the present invention. The processor 102 executes various functional applications and data processing by running the server program stored in the memory 104, that is, to implement the above method. The memory 104 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 104 may further include a memory remotely arranged relative to the processor 102, and these remote memories may be connected to the server via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0048] The transmission device 106 is used to receive or send data via a network. The specific example of the above network may include a wireless network provided by a communication provider of the server. In one example, the transmission device 106 includes a network adapter (Network Interface Controller, referred to as NIC), which can be connected to other network devices through a base station so as to communicate with the Internet. In one example, the transmission device 106 can be a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.

[0049] In this embodiment, a driving software testing method is provided. Figure 2 is a flow chart of a method for testing driving software according to an embodiment of the present invention, such as Figure 2 As shown, the process includes the following steps:

[0050] Step S202, obtaining vehicle driving data of multiple vehicle-side driving domain controllers, wherein the driving domain controller is used to run the target driving software to be tested;

[0051] The target driving software of this embodiment may be autonomous driving software, intelligent driving software, smart driving software, etc.

[0052] Step S204, using the vehicle driving data to generate test cases for multiple driving scenarios, and generating evaluation cases based on the test cases, wherein the evaluation cases are used to evaluate the test results of the test cases;

[0053] Step S206, creating a test task for the target driving software based on the test case and the evaluation case;

[0054] Step S208, sending the test task to the test bench cluster, and receiving the evaluation report fed back by the test bench cluster after executing the test task, wherein the test bench cluster is used to re-inject data based on the test case and evaluate the re-injected data based on the evaluation case.

[0055] Through the above steps, vehicle driving data of driving domain controllers on multiple vehicle sides are obtained, wherein the driving domain controller is used to run the target driving software to be tested; test cases for multiple driving scenarios are generated using the vehicle driving data, and evaluation cases are generated according to the test cases, wherein the evaluation cases are used to evaluate the test results of the test cases; a test task for the target driving software is created based on the test cases and the evaluation cases; the test task is issued to a test bench cluster, and an evaluation report fed back by the test bench cluster after executing the test task is received, wherein the test bench cluster is used to perform data re-injection based on the test case and evaluate the re-injected data based on the evaluation case, thereby realizing an automatic testing solution, solving the technical problem of low efficiency in testing driving software in related technologies, greatly improving testing efficiency, significantly shortening testing cycle, and effectively reducing testing costs.

[0056] Figure 3 It is an architectural diagram of an embodiment of the present invention, including a cloud server, a storage unit, a WEB terminal (user terminal), a vehicle-side acquisition module, and a test bench cluster. The test bench cluster includes a task calling service, a switch, and multiple test benches. Each test bench includes an industrial computer and an intelligent driving domain controller. The intelligent driving domain controller is used to deploy and run the test object, that is, the target driving software.

[0057] In one implementation of this embodiment, obtaining vehicle driving data of multiple vehicle-side driving domain controllers includes: obtaining the following vehicle driving data transmitted by multiple vehicle-side data acquisition modules: vehicle control instructions, vehicle status data, and sensor data, wherein the data acquisition module is communicatively connected with the vehicle-side driving domain controller.

[0058] The data acquisition module is integrated inside the vehicle, and communicates with the vehicle's intelligent driving domain controller through interfaces such as CAN bus and Ethernet, to collect the vehicle's control instructions and status information in real time, as well as environmental perception data from sensors, including multi-channel cameras, millimeter-wave radars, and lidars. The acquisition module has a built-in cellular mobile network module to provide data cloud network support, a high-performance processor to ensure efficient data processing capabilities, and a large-capacity storage device to provide temporary data storage capabilities. The collected data is encrypted and compressed before transmission to ensure data security and reduce transmission bandwidth requirements.

[0059] When the cloud server collects various types of data from the actual vehicle, the timestamps of the data sent by different devices and sensors are synchronized to the local time and written into the file. After completing the recording of a file, the file is compressed into a zip package and waits for uploading to the cloud; the data naming rules are: car series (uppercase letters)_the last six digits of vin_date (year-month-day)_time (hour-minute-second).

[0060] After receiving the compressed package, the file upload service initiates an upload request to the cloud. After the cloud responds, it starts sending the compressed package. After all the compressed packages to be uploaded are sent, an upload completion message is sent to the cloud to end the file upload. If a file fails to upload, the failed file will be stored in the breakpoint resume queue. After other files are uploaded successfully, the breakpoint resume function will be enabled and uploaded to the cloud again.

[0061] In one implementation of this embodiment, using the vehicle driving data to generate test cases for multiple driving scenarios includes: performing data cleaning and scene classification on the vehicle driving data to obtain multiple sets of driving scene data; and responding to a first creation request from a user, using the multiple sets of driving scene data to generate test cases for multiple driving scenarios.

[0062] In one example, the vehicle driving data is cleaned and scene classified to obtain multiple sets of driving scene data, including: cleaning the vehicle driving data to obtain intermediate data; clustering the intermediate data to obtain multiple data clusters, wherein each data cluster corresponds to a driving scene category; and classifying each data cluster in the multiple data clusters to obtain multiple sets of driving scene data.

[0063] After receiving the uploaded data, the cloud server will decompress it to the storage path set according to the rules of vehicle model / frame number / collection date for subsequent management and processing. The user creates a data cleaning task, and the cloud calls the data cleaning operator that supports custom rules. The operator can run automatically according to the set cycle to ensure that the data quality meets the test requirements and reduce the deviation of test results caused by data problems. The cleaning rules cover a variety of functions such as data naming specification checking, data integrity verification of specified types, frame loss detection, and clearing specified data to meet different test requirements.

[0064] Optionally, classifying each of the multiple data clusters includes: for each data cluster among the multiple data clusters, extracting key features in the data cluster, wherein the key features include at least one of the following: road type, traffic signs, environmental objects, weather features; and classifying the data cluster into multiple driving scenario items based on the key features.

[0065] This embodiment uses a combination of clustering algorithms and classification algorithms in machine learning algorithms to classify the scenes of the collected data. First, the clustering algorithm is used to divide the data into different clusters, each cluster represents a potential scene category, such as weather category, road category, and time category. Then, the classification algorithm is used to further classify these clusters and clearly divide them into different driving scenes, such as road categories including urban roads, highways, and mountain roads, time categories including night driving and daytime driving, and weather categories including driving scene items such as clear weather, foggy weather, rainy and snowy weather. Key features are extracted from the collected data for scene segmentation. These features include road types, traffic signs and markings, types and quantities of objects around the vehicle, weather conditions, etc. The accurately classified data is labeled with corresponding labels and stored in the warehouse, and test cases are subsequently established based on the scene labels.

[0066] Data is the data foundation of the development and testing process of intelligent driving. However, because the hardware and software status of the actual vehicle may have unpredictable problems, data cleaning must be performed to ensure that the data quality meets the requirements. Data cleaning can greatly reduce the unreliability of test results caused by data problems in later test businesses. Accurate scene classification ensures that testers can quickly obtain data that meets the test conditions for the test business.

[0067] In one example, using the multiple driving scenario data to generate test cases corresponding to multiple driving scenarios includes: determining a target test scenario according to the first creation request; searching for target test data for the target test scenario from the multiple driving scenario data; configuring a software package of the target driving software and a bench configuration file of the target test scenario, wherein the bench configuration file is used to define a reinjection configuration item; taking the target driving software as a test object, and using the target test data and the bench configuration file to generate a target test case for the target test scenario.

[0068] In this example, the user creates a test case on the web page, defines the test scenario and test parameters, where the test parameters include test data, the intelligent driving software package to be tested, the test bench configuration file, etc. It supports modification and deletion of existing test cases. Test cases can be manually created and automatically generated by users. Users manually create test cases based on the necessary test scenarios, and the system can automatically create a batch of similar test cases based on the basic test cases as templates.

[0069] The intelligent driving software package to be tested is the test object of the use case. Different types of software can be selected according to different test scenarios. Generally, they include APP package, MCU package and OS package. APP package is a required option and other types of software packages are optional. Software packages are generally automatically assembled by CI pipeline at scheduled times or manually assembled by users and then uploaded to the cloud.

[0070] The test bench configuration file is the dependency of test bench deployment and recharge, which defines various recharge configuration items. For example, "replay_mode" defines the recharge mode, which supports single-point recharge, continuous recharge, and cyclic recharge; "replay_type" defines the recharge data communication type, which supports CAN, TCP, UDP, ZMQ, DDS and other communication methods. The test bench supports recharge with multiple communication types at a time; "topic_name" defines the topic name of the data to be recharged.

[0071] Optionally, using the target test data and the bench configuration file to generate a target test case for the target test scenario includes: obtaining a test case template for the target test scenario, wherein the test case template includes typical input data and expected output results under the target test scenario; randomly adjusting the target test data within a data range of the typical input data to obtain multiple groups of use case data under the target test scenario; and using the multiple groups of use case data and the bench configuration file to generate a plurality of target test cases for the target test scenario.

[0072] According to the results of scenario classification, test cases can be automatically generated by combining random generation and template generation. For each driving scenario, some templates are pre-defined, which contain typical input data and expected output results for the scenario. Then, multiple different test cases are generated by making random changes based on the template. For example, in an urban road scenario, the template can include the parking behavior of a vehicle when it encounters a red light at an intersection, and random changes can include different vehicle speeds, different vehicle spacing, etc. The automatic test case generation function reduces the time and cost of manually designing test cases, can quickly build a test case library, can cover more driving scenarios, and improve the reliability and safety of the intelligent driving system.

[0073] In an implementation scenario of this embodiment, generating an evaluation case based on the test case includes: responding to the user's second creation request, configuring the evaluation indicators and evaluation rules of the test results according to the test case, and configuring the operation dependency parameters; and generating the evaluation case using the evaluation indicators, evaluation rules and the operation dependency parameters.

[0074] Optionally, configuring evaluation indicators and evaluation rules for test results according to the test case includes: determining key scenario data of the driving scenario corresponding to the test case; and searching for evaluation indicators and evaluation rules that match the key scenario data in a preset evaluation database.

[0075] On the one hand, users select test cases, priorities, and evaluation cases to create test tasks. The test bench cluster executes the re-injection and evaluation tasks according to the information of the test tasks, supports the start, stop, and restart of tasks, supports real-time viewing of task progress and re-injection results, and supports online preview and download of re-injection results and evaluation reports after the task is completed. It supports dynamic adjustment of task priorities to meet the needs of queue jumping during emergency testing. Test tasks can be created manually by users or by calling the API interface on a scheduled basis through the pipeline. The test business can flexibly configure the creation of test tasks according to needs, which can effectively improve task execution efficiency and test bench utilization.

[0076] On the other hand, based on the scenarios and data involved in the test cases, as well as the key performance indicators of the intelligent driving system, the evaluation indicators, evaluation parameters, evaluation algorithms and operation dependencies are automatically determined to generate evaluation cases. For example, for the above-mentioned highway overtaking scenario test case, the evaluation indicators may include overtaking safety, overtaking efficiency, etc.; the evaluation parameters may involve the speed and distance when overtaking; the evaluation algorithm can adopt a risk assessment algorithm based on machine learning; and the operation dependency specifies the required computing resources and environment configuration.

[0077] In an implementation scenario of the present embodiment, using the vehicle driving data to generate test cases for multiple driving scenarios includes: obtaining historical evaluation reports of historical test tasks of the target driving software; locating abnormal parameters to be optimized in the historical evaluation reports; iteratively updating template parameters in the test case template based on the abnormal parameters; and using the vehicle driving data and the updated test case template to generate test cases for multiple driving scenarios of the current test task.

[0078] In the implementation scenario of test case optimization, after each test task is completed, the parameters of the test case are iteratively optimized using machine learning algorithms (such as genetic algorithms, particle swarm algorithms, etc.) based on the results in the evaluation report (such as case pass rate, performance indicators, etc.). For example, if a test case is found to have poor acceleration performance of the intelligent driving system in a specific scenario during the evaluation, the system will adjust the relevant parameters of the test case in that scenario (such as initial vehicle speed, accelerator pedal travel, etc.) through the optimization algorithm and regenerate the test case for the next round of testing.

[0079] In an example of this embodiment, sending the test task to the rack cluster includes: obtaining the operating status information of each rack in the rack cluster; using the operating status information to calculate the task waiting time and the task execution time of each rack; using the task waiting time and the task execution time to calculate the response ratio of each rack; configuring the priority of each rack in the rack cluster based on the response ratio, and sending the test task to the racks in the rack cluster according to the priority.

[0080] This embodiment monitors the racks in the rack cluster, and each rack uploads the rack's operating status information to the cloud every 10 seconds, including network connection status, CAN bus status, rack CPU, disk space, memory usage, task status, task progress, etc. Users can view the latest status of the rack and task operation status on the web page, which is convenient for users to arrange task distribution reasonably.

[0081] A server is deployed in the test bench cluster to interact with the cloud through the task scheduling service. It is used to receive the test task information sent by the cloud, and use the high response ratio priority scheduling algorithm HRRN (Highest Response Ratio Next) to dynamically allocate test tasks to each test bench according to the data volume and priority of the test tasks, to ensure the balance of work between the various benches and improve the overall test efficiency. HRRN is a compromise algorithm between FCFS (First Come First Served Algorithm) and SJF (Short Job First Algorithm). It takes into account both the execution time and the waiting time of the job, and combines the characteristics of the first-come first-served and shortest job first algorithms. The response ratio in this algorithm refers to the ratio of the job waiting time to the running time. The response ratio formula is defined as follows:

[0082] Response ratio = (task waiting time + task execution time) / task execution time, that is, RR = (w + s) / s = 1 + w / s, so the response ratio must be greater than or equal to 1. The task waiting time is the time the test bench needs to wait to start executing the test task, and the task execution time is the time the test bench needs to execute the test task from the beginning to the end.

[0083] The task scheduling service arranges the test tasks and sends them to the test bench. The re-injection test tool software is deployed on the test bench industrial computer. Its main functions are: software deployment, data re-injection, data recording, evaluation, etc.

[0084] After the test bench on the test bench cluster receives the test task, it downloads the intelligent driving software package to be tested according to the task information. The intelligent driving software package generally includes MCU, OS, and APP, of which MCU and OS are firmware upgrade packages and are optional; APP is a full upgrade package for the intelligent driving application software and is a required item. Each task must have this software package to start the evaluation. After the software package is downloaded, the MD5 value of the verification file is added to ensure that the file is not damaged, and then each software package is deployed to the intelligent driving domain controller. The deployed APP defaults to the configuration items of the actual vehicle, but there are differences between the test bench environment and the actual vehicle. It is necessary to modify some configurations in the APP to the test bench mode and turn off some verification switches to enable the normal operation of the test bench intelligent driving software.

[0085] The test bench configuration file sent in the test task is the input for data re-injection, which defines the re-injection mode, re-injection channel and re-injection data type.

[0086] The refill mode is divided into single data refill and continuous data set refill. In the single data refill mode, the smart driving application is started before refill, and the smart driving application is closed after refilling one data until all data are refilled. In the continuous data set refill mode, the smart driving application is started before refill, and the smart driving application is closed after the data of the entire data set are refilled until all data sets are refilled.

[0087] The feedback channels include CAN (Controller Area Network) bus, UDP (Open Systems Interconnection), ZMQ (Zero Message Queue), DDS (Data Distribution Service) and other channels, which are combined in various ways according to the communication methods of intelligent driving applications. The feedback data type specifies the type of data that needs to be fed back, such as laser, vision, millimeter-wave radar, and complete vehicle.

[0088] Before re-injection, open the data recording module to receive the processed data from the intelligent driving domain controller. After the re-injection is completed, stop recording and write the data into a file for subsequent evaluation.

[0089] Download the evaluation case sent from the cloud to the test bench, deploy the evaluation environment in the test bench to the local virtual environment, run the evaluation.py evaluation algorithm in the virtual environment to evaluate and analyze the injected data, and generate a detailed evaluation report, including case pass rate, performance indicators, problem location, and improvement suggestions. Package the evaluation report into report.zip and upload it to the cloud, and the test task is completed.

[0090] Figure 4 It is a schematic diagram of the operation process of the intelligent driving test system in the embodiment of the present invention. It proposes an efficient and accurate intelligent driving test system. The data acquisition module is deployed on the vehicle side to collect the real road data of the actual vehicle test and upload it to the cloud server; the cloud server pre-processes the data, including data cleaning and quality inspection, scene mining, classification storage, test case self-generation, etc.; the user creates a test task according to the test requirements through the WEB page and sends it to the test bench cluster. The test task includes a re-injection task and an evaluation task. The re-injection task includes the data to be re-injected and the intelligent driving software package to be tested; the task scheduling service is deployed on the test bench cluster. The task scheduling service intelligently schedules the task to each test bench to perform the re-injection test task according to the test bench operation status and task resource requirements. The main hardware of each test bench includes a high-performance industrial computer and an intelligent driving domain controller. The re-injection tool and the evaluation tool are deployed on the high-performance industrial computer to execute the re-injection task and the evaluation task after re-injection, and output the evaluation report. Finally, all process data are transmitted back to the cloud server, and the user views the task results through the WEB end.

[0091] Based on the solution of this embodiment, a high degree of coverage of services such as intelligent driving software module testing, system testing, and integration testing is achieved, which greatly improves the test scenario coverage, test efficiency, and accuracy of test results of intelligent driving software. The present invention can provide strong support for the development of intelligent connected vehicle technology, can greatly improve the test efficiency and coverage of intelligent driving systems, and has high commercial value and application prospects.

[0092] Through the description of the above implementation methods, those skilled in the art can clearly understand that the method according to the above embodiment can be implemented by means of software plus a necessary general hardware platform, and of course can also be implemented by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, a magnetic disk, or an optical disk), and includes a number of instructions for enabling a terminal device (which can be a mobile phone, a computer, a server, or a network device, etc.) to execute the methods described in each embodiment of the present invention.

[0093] Example 2

[0094] In this embodiment, a driving software testing device is also provided, which is used to implement the above-mentioned embodiments and preferred implementation modes, and the descriptions that have been made will not be repeated. As used below, the term "module" can be a combination of software and / or hardware that implements a predetermined function. Although the devices described in the following embodiments are preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and conceivable.

[0095] Figure 5 is a structural block diagram of a driving software testing device according to an embodiment of the present invention, such as Figure 5 As shown, the device comprises:

[0096] An acquisition module 50 is used to acquire vehicle driving data of a plurality of vehicle-side driving domain controllers, wherein the driving domain controller is used to run the target driving software to be tested;

[0097] A generating module 52, configured to generate test cases of multiple driving scenarios using the vehicle driving data, and to generate evaluation cases based on the test cases, wherein the evaluation cases are used to evaluate the test results of the test cases;

[0098] A creation module 54, configured to create a test task for the target driving software based on the test case and the evaluation case;

[0099] The test module 56 is used to send the test task to the test bench cluster and receive the evaluation report fed back by the test bench cluster after executing the test task, wherein the test bench cluster is used to re-inject data based on the test case and evaluate the re-injected data based on the evaluation case.

[0100] Optionally, the acquisition module includes: an acquisition unit, used to acquire the following vehicle driving data transmitted by multiple vehicle-side data acquisition modules: vehicle control instructions, vehicle status data, and sensor data, wherein the data acquisition module is communicatively connected to the vehicle-side driving domain controller.

[0101] Optionally, the generation module includes: a classification unit, used to perform data cleaning and scene classification on the vehicle driving data to obtain multiple driving scene data; a first generation unit, used to respond to a user's first creation request and use the multiple driving scene data to generate test cases corresponding to multiple driving scenarios.

[0102] Optionally, the classification unit includes: a cleaning subunit, used to clean the vehicle driving data to obtain intermediate data; a clustering subunit, used to cluster the intermediate data to obtain multiple data clusters, wherein each data cluster corresponds to a driving scene category; and a classification subunit, used to classify each data cluster in the multiple data clusters to obtain multiple driving scene data.

[0103] Optionally, the classification subunit is also used to: extract key features in each data cluster among the multiple data clusters, wherein the key features include at least one of the following: road type, traffic signs, environmental objects, weather characteristics; and classify the data cluster into multiple driving scene items based on the key features.

[0104] Optionally, the first generation unit includes: a determination subunit, used to determine the target test scenario according to the first creation request; a search subunit, used to search for target test data of the target test scenario from the multiple driving scenario data; a configuration subunit, used to configure the software package of the target driving software and the bench configuration file of the target test scenario, wherein the bench configuration file is used to define the reinjection configuration items; a generation subunit, used to take the target driving software as the test object, and use the target test data and the bench configuration file to generate a target test case for the target test scenario.

[0105] Optionally, the generation subunit is also used to: obtain a test case template for the target test scenario, wherein the test case template includes typical input data and expected output results under the target test scenario; randomly adjust the target test data within the data range of the typical input data to obtain multiple sets of use case data under the target test scenario; and use the multiple sets of use case data and the bench configuration file to generate a plurality of target test cases for the target test scenario.

[0106] Optionally, the generation module includes: a configuration unit, used to respond to the user's second creation request, configure the evaluation indicators and evaluation rules of the test results according to the test case, and configure the operation dependency parameters; a second generation unit, used to generate an evaluation case using the evaluation indicators, evaluation rules and the operation dependency parameters.

[0107] Optionally, the configuration unit includes: a determination subunit, used to determine key scenario data of the driving scenario corresponding to the test case; and a search subunit, used to search for evaluation indicators and evaluation rules that match the key scenario data in a preset evaluation database.

[0108] Optionally, the generation module includes: an acquisition unit, used to acquire historical evaluation reports of historical test tasks of the target driving software; a positioning unit, used to locate abnormal parameters to be optimized in the historical evaluation report; an updating unit, used to iteratively update template parameters in the test case template based on the abnormal parameters; and a third generation unit, used to use the vehicle driving data and the updated test case template to generate test cases for multiple driving scenarios of the current test task.

[0109] Optionally, the test module includes: an acquisition unit, used to acquire the operating status information of each rack in the rack cluster; a first calculation unit, used to use the operating status information to calculate the task waiting time and task execution time of each rack; a second calculation unit, used to use the task waiting time and the task execution time to calculate the response ratio of each rack; and a sending unit, used to configure the priority of each rack in the rack cluster based on the response ratio, and send the test task to the racks in the rack cluster according to the priority.

[0110] It should be noted that the above modules can be implemented by software or hardware. For the latter, it can be implemented in the following ways, but not limited to: the above modules are all located in the same processor; or the above modules are located in different processors in any combination.

[0111] Example 3

[0112] An embodiment of the present invention further provides a storage medium, in which a computer program is stored, wherein the computer program is configured to execute the steps of any of the above method embodiments when running.

[0113] Optionally, in this embodiment, the storage medium may be configured to store a computer program for performing the following steps:

[0114] S1, obtaining vehicle driving data of multiple vehicle-side driving domain controllers, wherein the driving domain controller is used to run the target driving software to be tested;

[0115] S2, using the vehicle driving data to generate test cases for multiple driving scenarios, and generating evaluation cases based on the test cases, wherein the evaluation cases are used to evaluate the test results of the test cases;

[0116] S3, creating a test task for the target driving software based on the test case and the evaluation case;

[0117] S4, sending the test task to the test bench cluster, and receiving the evaluation report fed back by the test bench cluster after executing the test task, wherein the test bench cluster is used to re-inject data based on the test case and evaluate the re-injected data based on the evaluation case.

[0118] Optionally, in this embodiment, the above-mentioned storage medium may include but is not limited to: a USB flash drive, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk or an optical disk, and other media that can store computer programs.

[0119] An embodiment of the present invention further provides an electronic device, including a memory and a processor, wherein a computer program is stored in the memory, and the processor is configured to run the computer program to execute the steps in any one of the above method embodiments.

[0120] Optionally, the electronic device may further include a transmission device and an input / output device, wherein the transmission device is connected to the processor, and the input / output device is connected to the processor.

[0121] Optionally, in this embodiment, the processor may be configured to perform the following steps through a computer program:

[0122] S1, obtaining vehicle driving data of multiple vehicle-side driving domain controllers, wherein the driving domain controller is used to run the target driving software to be tested;

[0123] S2, using the vehicle driving data to generate test cases for multiple driving scenarios, and generating evaluation cases based on the test cases, wherein the evaluation cases are used to evaluate the test results of the test cases;

[0124] S3, creating a test task for the target driving software based on the test case and the evaluation case;

[0125] S4, sending the test task to the test bench cluster, and receiving the evaluation report fed back by the test bench cluster after executing the test task, wherein the test bench cluster is used to re-inject data based on the test case and evaluate the re-injected data based on the evaluation case.

[0126] Optionally, the specific examples in this embodiment may refer to the examples described in the above embodiments and optional implementation modes, and this embodiment will not be described in detail here.

[0127] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0128] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a general hardware platform, and of course, by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the relevant technology can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0129] It should be understood that the terms used herein are only for the purpose of describing specific example embodiments and are not intended to be limiting. Unless the context clearly indicates otherwise, the singular forms "one", "an" and "said" as used herein may also be meant to include plural forms. The terms "include", "comprise", "contain", and "have" are inclusive, and therefore specify the existence of stated features, steps, operations, elements and / or parts, but do not exclude the existence or addition of one or more other features, steps, operations, elements, parts, and / or combinations thereof. The method steps, processes, and operations described herein are not interpreted as necessarily requiring them to be performed in the specific order described or illustrated, unless the execution order is clearly indicated. It should also be understood that additional or alternative steps may be used.

[0130] The foregoing is merely a specific embodiment of the present invention, which enables those skilled in the art to understand or implement the present invention. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features claimed herein.

Claims

1. A driving software testing method, characterized in that: include: Acquire vehicle driving data of multiple vehicle-side driving domain controllers, wherein the driving domain controller is used to run the target driving software to be tested; Generating test cases of multiple driving scenarios using the vehicle driving data, and generating evaluation cases based on the test cases, wherein the evaluation cases are used to evaluate the test results of the test cases; Creating a test task for the target driving software based on the test case and the evaluation case; The test task is issued to the test bench cluster, and an evaluation report fed back by the test bench cluster after executing the test task is received, wherein the test bench cluster is used to re-inject data based on the test case and evaluate the re-injected data based on the evaluation case.

2. The method according to claim 1, characterized in that Obtaining vehicle driving data from multiple vehicle-side driving domain controllers includes: Acquire the following vehicle driving data transmitted by multiple vehicle-side data acquisition modules: vehicle control instructions, vehicle status data, and sensor data, wherein the data acquisition module is communicatively connected with the vehicle-side driving domain controller.

3. The method according to claim 1, characterized in that The test cases for generating multiple driving scenarios using the vehicle driving data include: Performing data cleaning and scene classification on the vehicle driving data to obtain multiple sets of driving scene data; In response to a first creation request from a user, test cases for a plurality of driving scenarios are generated using the plurality of driving scenario data.

4. The method according to claim 3, characterized in that The vehicle driving data is cleaned and scene classified to obtain multiple driving scene data, including: Performing data cleaning on the vehicle driving data to obtain intermediate data; Clustering the intermediate data to obtain multiple data clusters, wherein each data cluster corresponds to a driving scene category; Each data cluster in the multiple data clusters is classified to obtain multiple pieces of driving scene data.

5. The method according to claim 4, characterized in that Classifying each of the plurality of data clusters includes: For each data cluster in the plurality of data clusters, extract key features in the data cluster, wherein the key features include at least one of the following: road type, traffic signs, environmental objects, and weather features; The data cluster is classified into a plurality of driving scenario items based on the key features.

6. The method according to claim 3, characterized in that The test cases for generating multiple driving scenarios using the multiple driving scenario data include: Determine a target test scenario according to the first creation request; Searching for target test data of the target test scene from the plurality of driving scene data; Configuring the software package of the target driving software and the bench configuration file of the target test scenario, wherein the bench configuration file is used to define the recharge configuration item; The target driving software is used as a test object, and the target test data and the bench configuration file are used to generate a target test case of the target test scenario.

7. The method according to claim 6, characterized in that Generating a target test case of the target test scenario using the target test data and the bench configuration file includes: Obtaining a test case template for the target test scenario, wherein the test case template includes typical input data and expected output results under the target test scenario; Randomly adjusting the target test data within the data range of the typical input data to obtain multiple groups of use case data under the target test scenario; The multiple groups of use case data and the bench configuration files are used to generate a plurality of target test cases for the target test scenarios.

8. The method according to claim 1, characterized in that Generating evaluation cases according to the test cases includes: In response to the user's second creation request, configure the evaluation indicators and evaluation rules of the test results according to the test case, and configure the operation dependency parameters; The evaluation indicators, evaluation rules and the operation dependent parameters are used to generate evaluation cases.

9. The method according to claim 8, characterized in that The evaluation indicators and evaluation rules for configuring the test results according to the test case include: Determine key scenario data of the driving scenario corresponding to the test case; Search a preset evaluation database for evaluation indicators and evaluation rules that match the key scenario data.

10. The method according to claim 1, characterized in that The test cases for generating multiple driving scenarios using the vehicle driving data include: Obtaining historical evaluation reports of historical test tasks of the target driving software; Locating abnormal parameters to be optimized in the historical evaluation report; Iteratively update the template parameters in the test case template based on the abnormal parameters; The vehicle driving data and the updated test case template are used to generate test cases for multiple driving scenarios of the current test task.

11. The method according to claim 1, characterized in that: Sending the test task to the test bench cluster includes: Obtaining the operating status information of each rack in the rack cluster; The operation status information is used to calculate the task waiting time and task execution time of each rack; Calculating the response ratio of each rack using the task waiting time and the task execution time; The priority of each rack in the rack cluster is configured based on the response ratio, and the test task is issued to the racks in the rack cluster according to the priority.

12. A driving software testing device, characterized in that: include: An acquisition module, used to acquire vehicle driving data of multiple vehicle-side driving domain controllers, wherein the driving domain controller is used to run the target driving software to be tested; A generating module, configured to generate test cases of multiple driving scenarios using the vehicle driving data, and to generate evaluation cases based on the test cases, wherein the evaluation cases are used to evaluate the test results of the test cases; A creation module, used for creating a test task of the target driving software based on the test case and the evaluation case; The test module is used to send the test task to the test bench cluster and receive the evaluation report fed back by the test bench cluster after executing the test task, wherein the test bench cluster is used to re-inject data based on the test case and evaluate the re-injected data based on the evaluation case.

13. A storage medium, characterized in that: The storage medium stores a computer program, wherein the computer program is configured to execute the method according to any one of claims 1 to 11 when executed.

14. An electronic device comprising a memory and a processor, characterized in that: A computer program is stored in the memory, and the processor is configured to run the computer program to perform the method according to any one of claims 1 to 11.

Images