Test data processing method and device of automatic driving system and storage medium

Abstract

The embodiment of the invention provides a test data processing method and device of an automatic driving system and a storage medium, and the method comprises the steps: obtaining simulation test data of a specified vehicle; the simulation test data has a first category label; the first category label is used for identifying a target scene category of the preset simulation scene; according to the first category label, analyzing the simulation test data according to a group of evaluation indexes matched with the target scene category identified by the first category label; the group of evaluation indexes comprises a group of general indexes and a group of self-defined indexes; each user-defined index in the group of user-defined indexes is an index which is preset according to the target scene category and is different from the general index; and generating an evaluation report of the specified vehicle in the preset simulation scene according to the analyzed index quantized value of each evaluation index in the group of evaluation indexes. According to the invention, the technical problem that the evaluation result of the automatic driving system is not accurate enough due to the single evaluation index is solved.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of autonomous driving simulation testing. Specifically, the embodiments relate to a method, an apparatus, and a storage medium for processing test data of an autonomous driving system. Background Art

[0002] With the continuous development of autonomous driving technology and the increasing popularity of vehicles with autonomous driving functions, it has become increasingly important to improve autonomous driving technology. The autonomous driving system is software used in autonomous vehicles to implement driving control functions. To ensure the accuracy and safety of the driving of autonomous vehicles, it is necessary to test and evaluate the autonomous driving system on them before the autonomous vehicles are put into road operation. Especially for high-level autonomous vehicles, during operation, the driver no longer always has the motion control right of the vehicle, and the autonomous driving system becomes the main body for monitoring the driving environment and controlling the vehicle operation.

[0003] Currently, the commonly used autonomous driving systems mainly adopt simulation testing methods. Starting from standard regulations, scene segments are pre-designed, and then the safety of the autonomous driving system is evaluated based on passability.

[0004] However, the existing evaluation methods for autonomous driving systems ignore the randomness of the traffic environment and natural environmental factors, resulting in a large difference between the real environment and the simulation environment when the vehicle is driving, and relying on a single evaluation index (such as passability) for evaluation, resulting in inaccurate evaluation results of the autonomous driving system. Summary of the Invention

[0005] The embodiments of the present application provide a method, an apparatus, and a storage medium for processing test data of an autonomous driving system, so as to at least solve the technical problem that the evaluation results of the autonomous driving system are inaccurate due to a single evaluation index in the related art.

[0006] According to one aspect of the embodiments of the present application, a method for processing test data of an autonomous driving system is provided, including: obtaining simulation test data of a specified vehicle; the simulation test data refers to the simulation data obtained after the specified vehicle undergoes simulation testing in a preset simulation scenario; the simulation test data has a first category label; the first category label is used to identify the target scenario category of the preset simulation scenario; according to the first category label, parsing the simulation test data according to a set of evaluation indexes matching the target scenario category identified by the first category label; the set of evaluation indexes includes a set of general indexes and a set of custom indexes; each custom index in the set of custom indexes is an index different from the general indexes preset according to the target scenario category; generating an evaluation report of the specified vehicle in the preset simulation scenario according to the index quantization value of each evaluation index in the parsed set of evaluation indexes.

[0007] According to another aspect of the embodiments of the present application, there is also provided a test data processing device for an autonomous driving system, including: a simulation module, configured to obtain simulation test data of a specified vehicle; the simulation test data refers to the simulation data obtained after the specified vehicle undergoes simulation tests in a preset simulation scenario; the simulation test data has a first category label; the first category label is used to identify the target scenario category of the preset simulation scenario; an evaluation module, configured to parse the simulation test data according to the first category label according to a set of evaluation indicators matching the target scenario category identified by the first category label; the set of evaluation indicators includes a set of general indicators and a set of custom indicators; each custom indicator in the set of custom indicators is an indicator preset to be different from the general indicators according to the target scenario category; a report generation module, configured to generate an evaluation report of the specified vehicle in the preset simulation scenario according to the index quantization value of each evaluation indicator in the parsed set of evaluation indicators.

[0008] According to yet another aspect of the embodiments of the present application, there is also provided a computer-readable storage medium, in which a computer program is stored, wherein the computer program is configured to execute the steps in any one of the above method embodiments when running.

[0009] According to yet another aspect of the embodiments of the present application, there is provided a computer program product or a computer program, the computer program product or the computer program includes computer instructions, and the computer instructions are stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes the steps in any one of the above method embodiments.

[0010] According to yet another aspect of the embodiments of the present application, there is also provided an electronic device, including a memory and a processor, a computer program is stored in the memory, and the processor is configured to execute the steps in any one of the above method embodiments through the computer program.

[0011] Through this application, simulation test data obtained after a specified vehicle undergoes simulation tests in a preset simulation scenario is acquired. The simulation test data has a first category label for identifying the target scenario category of the preset simulation scenario. According to the first category label, the simulation test data is analyzed according to a set of evaluation metrics that match the target scenario category identified by the first category label. Among them, a set of evaluation metrics that match the target scenario category identified by the first category label includes a set of general metrics and a set of custom metrics. Different from traditional methods, first, in the embodiments of this application, the evaluation metrics used to evaluate the simulation test data are associated with the preset simulation scenario. As the preset simulation scenario changes, a set of evaluation metrics that match the target scenario category identified by the first category label also changes dynamically. That is, the evaluation metrics in the embodiments of this application change dynamically following the preset simulation scenario. Second, the evaluation metrics in the embodiments of this application have the characteristic of diversification. A set of evaluation metrics that match the target scenario category identified by the first category label includes a set of general metrics and a set of custom metrics. The custom metrics are preset according to the target scenario category. Different from the general metrics, the custom metrics can evaluate specific challenges in a specific scenario, avoiding the limitations of a single evaluation metric (such as passability), and being able to more comprehensively evaluate the performance of the autonomous driving system. Therefore, according to the metric quantization values of each evaluation metric in the parsed set of evaluation metrics, the generated evaluation report of the specified vehicle in the preset simulation scenario is more accurate, improving the accuracy and comprehensiveness of the evaluation results and solving the technical problem in the related art that the evaluation results of the autonomous driving system are not accurate enough due to a single evaluation metric. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] Figure 1 is a schematic diagram of an application scenario of a method for processing test data of an autonomous driving system according to an embodiment of the present application;

[0013] Figure 2 is a schematic flowchart of an optional method for processing test data of an autonomous driving system according to an embodiment of the present application;

[0014] Figure 3 is a structural diagram of an optional evaluation report generation system according to an embodiment of the present application;

[0015] Figure 4 is an application schematic diagram of an optional evaluation model according to an embodiment of the present application;

[0016] Figure 5 is a training schematic diagram of an optional evaluation model according to an embodiment of the present application;

[0017] Figure 6 is a structural block diagram of an optional device for processing test data of an autonomous driving system according to an embodiment of the present application;

[0018] Figure 7 It is a block diagram of the computer system structure of an optional electronic device according to an embodiment of the present application. Detailed implementation manners

[0019] In order to enable those skilled in the art of the present technology to better understand the solution of the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present application.

[0020] It should be noted that the terms "first", "second", etc. in the description and claims of the present application and the above accompanying drawings are used to distinguish similar objects, and do not necessarily need to be used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged under appropriate circumstances so that the embodiments of the present application described here can be implemented in an order different from those illustrated or described here. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion. For example, a process, method, system, product or device including a series of steps or units does not necessarily need to be limited to those steps or units clearly listed, but may include other steps or units not clearly listed or inherent to these processes, methods, products or devices.

[0021] According to one aspect of the embodiments of the present application, a method for processing test data of an autonomous driving system is provided. Optionally, in this embodiment, the method for processing test data of the autonomous driving system described above may be but is not limited to being applied to a hardware environment such as Figure 1 shown in the figure, including a terminal device 102 and a server 104. The server 104 can be connected to the terminal device 102 through a network, and can be used to provide services (such as application services, etc.) for the terminal device 102 or a client installed on the terminal device 102. A database can be set on the server 104 or independently of the server 104 to provide data storage services for the server 104.

[0022] The above network may include, but is not limited to, at least one of the following: a wired network, a wireless network. The above wired network may include, but is not limited to, at least one of the following: a wide area network, a metropolitan area network, a local area network. The above wireless network may include, but is not limited to, at least one of the following: WIFI (Wireless Fidelity), Bluetooth. The terminal device 102 may be, but is not limited to, a PC (Personal Computer), a mobile phone, a tablet computer, etc. The server 104 may be, but is not limited to, a cloud server, a server cluster, or other server types.

[0023] The test data processing method of the autonomous driving system according to the embodiments of the present application may be executed by the server 104, or may be executed by the terminal device 102, or may also be jointly executed by the server 104 and the terminal device 102. Among them, when the terminal device 102 executes the test data processing method of the autonomous driving system according to the embodiments of the present application, it may also be executed by the client installed thereon.

[0024] Taking the execution of the test data processing method of the autonomous driving system in this embodiment by the terminal device 102 as an example, Figure 2 is a schematic flowchart of an optional test data processing method of an autonomous driving system according to the embodiments of the present application, as Figure 2 shown, the process of this method may include the following steps:

[0025] Step S202, obtain the simulation test data of a specified vehicle; the simulation test data refers to the simulation data obtained after the specified vehicle undergoes a simulation test in a preset simulation scenario; the simulation test data has a first category label; the first category label is used to identify the target scenario category of the preset simulation scenario.

[0026] Among them, the test data processing method of the autonomous driving system in this embodiment can be applied to the field of autonomous driving simulation testing, and is applied to driving scenarios such as urban road environments, highway environments, tunnel environments, road bridges with crosswinds, and rainy and snowy weather.

[0027] The specified vehicle refers to a vehicle that needs to be tested for an autonomous driving system. The specified vehicle is equipped with an autonomous driving system and can perform autonomous driving tests in a simulation environment.

[0028] The preset simulation scenario refers to a virtual environment designed and set in advance for testing the autonomous driving system. The preset simulation scenario simulates factors such as real road conditions, traffic conditions, static environments, meteorological environments, environmental temperature and humidity, and vehicle conditions, and is used to evaluate the performance and safety of the autonomous driving system. The preset simulation scenario can be various scenarios such as urban roads, highways, rural roads, tunnels, bridges, rainy and snowy weather, and night driving.

[0029] Simulation test data refers to the data obtained after a designated vehicle undergoes simulation tests in a preset simulation scenario. The simulation test data records information such as the driving state, environmental perception, and decision-making process of the vehicle in the simulation environment.

[0030] The target scenario category refers to one or more specific driving environment categories in the preset simulation scenario. For example, one or more driving environment categories such as urban roads, highways, tunnels, road bridges with crosswinds, or rainy and snowy weather.

[0031] The first category label refers to the label used to identify the target scenario category of the preset simulation scenario. The first category label identifies the type of preset simulation scenario from which the simulation test data is derived. For example, urban roads, highways, tunnels, road bridges with crosswinds, or rainy and snowy weather. The first category label can identify a single scenario or a combined simulation scenario.

[0032] It should be noted that: simulation software usually has a scenario recognition function, which can analyze environmental information, vehicle status, traffic participant behavior, etc. in the simulation test data to determine the scenario category to which the current simulation scenario belongs. Once the scenario category is determined, the simulation software can automatically generate a first category label corresponding to the scenario category according to preset rules or algorithms. It can be understood that: the first category label is generated by the simulation software according to the recognized scenario category of the preset simulation scenario.

[0033] Optionally, the terminal device constructs a vehicle dynamics model of the designated vehicle, places the vehicle dynamics model in the simulation environment, and starts the autonomous driving system for testing. It collects the simulation test data obtained after the designated vehicle undergoes simulation tests in the preset simulation scenario from the simulation environment, and performs preprocessing operations such as cleaning, annotation, and feature extraction on the collected simulation test data. Among them, outliers and noise data are removed through the cleaning operation, the collected simulation test data is associated, annotated, and classified with the corresponding first category label through the annotation operation, and key features in the data are extracted through the feature extraction operation to obtain the preprocessed simulation test data.

[0034] In some embodiments, Figure 3 is a structural diagram of an evaluation report generation system in an embodiment, as Figure 3 shown, the autonomous driving simulation test evaluation report generation system includes a sensor module, a vehicle dynamics module, a simulation test module, a scenario library module, a data recombination module, a data processing module, and a meteorological information module.

[0035] Among them, the sensor module, including the sensor module, uses devices such as lidar, cameras, ultrasonic sensors, GPS, and radars to provide information about the vehicle's surrounding environment, helping the autonomous driving system to perceive and understand the surrounding roads and traffic conditions, and improving the perception ability and safety of the autonomous driving system.

[0036] The vehicle dynamics module can establish an accurate vehicle size model and a vehicle power system model through the vehicle dynamics module, which are used to simulate and control the vehicle's motion behavior. Specifically, through the vehicle size model, the vehicle's mass, inertia, size, and suspension system are determined, and the vehicle's acceleration, braking, and steering are determined. Through the vehicle's power system model, the engine, transmission system, tires, and tire pressure are used to calculate the vehicle's acceleration, speed, and steering force, and the vehicle's power performance and energy consumption characteristics are calculated. In this way, when the vehicle is driving, its engine, transmission system, tires, and tire pressure are used to calculate the vehicle's acceleration, speed, and steering force, and the vehicle's power performance and energy consumption characteristics are calculated through the vehicle mechanics module, enabling the vehicle to simulate driving in different environments and road conditions and obtain simulation test data.

[0037] The simulation test module verifies the accuracy and reliability of the vehicle's power and braking by performing simulations in a simulation environment and simulating different driving scenarios and road conditions.

[0038] The scenario library module is used to test the adaptability, safety, and reliability of the autonomous driving system in response to different scenarios by establishing rich and accurate virtual sites. The scenario library module selects appropriate perception, decision-making, and control strategies according to the current driving scenario. By arranging urban road environments, highway environments, tunnel environments, road bridges with crosswinds, and rain and snow weather, it helps the system understand and adapt to different driving scenarios and provides appropriate decision-making and control strategies.

[0039] The data reorganization module integrates data from different sensors and modules to generate a unified data stream. After data comparison and calibration, it accurately reflects the real environmental conditions, and performs noise reduction processing on the data to remove noise and interference, which can improve the system's perception and decision-making abilities.

[0040] The data processing module processes and analyzes the simulation test data, extracts useful information and features, and helps the system understand and predict changes in the driving environment through statistics, machine learning, and deep learning under different working conditions. Finally, the data is visualized in the form of charts, images, and maps to provide accurate and reliable data information for the system's perception, decision-making, and control. By processing the data, the system's understanding and cognitive abilities of the driving environment are improved, and more precise, safe, and intelligent autonomous driving is achieved.

[0041] The meteorological information module sets information such as the atmosphere and environment in the simulation environment.

[0042] Step S204: According to the first category label, parse the simulation test data according to a set of evaluation metrics that match the target scenario category identified by the first category label; the set of evaluation metrics includes a set of general metrics and a set of custom metrics; each custom metric in the set of custom metrics is a metric that is preset to be different from the general metrics according to the target scenario category.

[0043] Among them, a set of evaluation metrics that match the target scenario category identified by the first category label refers to a set of specific criteria or parameters used to evaluate the performance and safety of the autopilot system of a specified vehicle in a preset simulation scenario. The specific metric types of a set of evaluation metrics that match the target scenario category identified by the first category label are associated with the preset simulation scenario. Specifically, a set of evaluation metrics includes a set of general metrics and a set of custom metrics. Among them, a set of general metrics is generally applicable to various driving scenarios and is an evaluation of the basic performance of the autopilot system. For example, vehicle driving stability, compliance with traffic rules, obstacle detection accuracy, vehicle power, and braking, etc. A set of custom metrics is different from a set of general metrics. The custom metrics are preset for a specific preset simulation scenario and are designed to more deeply evaluate the specific performance of the autopilot system in this scenario. For example, in rainy and snowy weather, the custom metrics may include visibility adaptability, braking distance under slippery road conditions, etc.; on a crosswind road bridge, vehicle stability and crosswind resistance may be concerned.

[0044] Optionally, the terminal device extracts the first category label from the simulation test data, according to the first category label, extracts a set of evaluation metrics that match the target scenario category, and parses the simulation test data according to each evaluation metric in the extracted set of evaluation metrics to obtain the metric quantization value corresponding to each evaluation metric in the set of evaluation metrics.

[0045] In some embodiments, the target scenario category identified by the first category label may be a single scenario. For example, the target scenario category identified by the first category label may be any one of an urban road scenario, a highway scenario, a sunny day scenario, and a rainy day scenario.

[0046] When the preset simulation scenario is an urban road scenario, the target scenario category identified by the first category label is an urban road, and a set of custom metrics that match the target scenario category identified by the first category label includes road marking recognition ability, traffic participant behavior prediction, and intersection passing ability, etc.

[0047] When the preset simulation scenario is a highway scenario, the target scenario category identified by the first category label is the highway scenario, and a set of custom metrics matching the target scenario category identified by the first category label include the ability to maintain the highest speed, safety during overtaking, the ability to maintain a safe distance from the vehicle in front, the braking reaction time in emergency situations, etc.

[0048] When the preset simulation scenario is a sunny day scenario, the target scenario category identified by the first category label is the sunny day scenario, and a set of custom metrics matching the target scenario category identified by the first category label include the clarity of sensors (such as cameras, lidar), the color recognition ability under lighting conditions, the recognition accuracy of road markings, etc.

[0049] When the preset simulation scenario is a rainy day scenario, the target scenario category identified by the first category label is the rainy day scenario, and a set of custom metrics matching the target scenario category identified by the first category label include the penetration ability of sensors in rain and fog, the braking distance under wet road conditions, the impact of windshield wipers on the sensor's field of view, and the impact of unique light refraction in rainy days on the perception system, etc.

[0050] It can be understood that when the preset simulation scenario changes, a set of evaluation metrics matching the target scenario category identified by the first category label will also change. For example, if the current preset simulation scenario is an urban road scenario, the target scenario category identified by the first category label is the urban road, and a set of custom metrics matching the target scenario category identified by the first category label include the road marking recognition ability, traffic participant behavior prediction, and intersection passing ability, etc. When the preset simulation scenario switches from the urban road scenario to the highway scenario, the current preset simulation scenario changes from the urban road scenario to the highway scenario, and the target scenario category identified by the first category label will also switch from the urban road to the highway scenario. At this time, a set of custom metrics matching the target scenario category identified by the first category label will also switch to the ability to maintain the highest speed, safety during overtaking, the ability to maintain a safe distance from the vehicle in front, the braking reaction time in emergency situations, etc. Therefore, when the preset simulation scenario changes from an urban road to a highway, a set of custom metrics will correspondingly change from the specific requirements of urban roads to the specific requirements of highways.

[0051] In some embodiments, the target scenario category identified by the first category label may be a combined simulation scenario. For example, the first category label may represent a combined simulation scenario such as "urban road scenario in snowy weather". In this case, the sum of a set of custom metrics matching each simulation scenario in the combined scenario can be used as a set of custom metrics matching the target scenario category identified by the first category label. Alternatively, the weight of each custom evaluation metric in a set of custom metrics matching each simulation scenario in the combined simulation scenario can be preset, and according to the preset weights, a preset number of custom metrics can be selected from a set of custom metrics corresponding to each simulation scenario in the combined simulation scenario as a set of custom metrics matching the target scenario category identified by the first category label.

[0052] For example, the target scenario category identified by the first category label is a combined simulation scenario of "urban road scenario in snowy weather". Among them, a set of custom metrics preset for the snowy weather simulation scenario includes: visibility recognition ability (weight 0.3), snowflake interference adaptation ability (weight 0.2), and wind speed impact assessment (weight 0.1); a set of custom metrics preset for the urban road scenario includes: road marking recognition ability (weight 0.2), traffic participant behavior prediction (weight 0.3), and intersection passing ability (weight 0.2); according to the requirement of a preset number (such as selecting 5 evaluation metrics), the following 5 are selected from a set of custom metrics corresponding to the above two simulation scenarios as a set of custom metrics matching the first category label "urban road scenario in snowy weather": visibility recognition ability (weight 0.3), snowflake interference adaptation ability (weight 0.2), road marking recognition ability (weight 0.2), traffic participant behavior prediction (weight 0.3), intersection passing ability (weight 0.2).

[0053] Step S206: Generate an evaluation report of the specified vehicle in the preset simulation scenario according to the metric quantization value of each evaluation metric in the parsed set of evaluation metrics.

[0054] Optionally, the terminal device sorts each evaluation metric and its corresponding metric quantization value to form a correspondence table of evaluation metrics and quantization values, and fills the data in the sorted correspondence table of evaluation metrics and quantization values into the corresponding positions in the preset evaluation report framework to obtain an evaluation report of the specified vehicle in the preset simulation scenario. Among them, the evaluation report of the specified vehicle in the preset simulation scenario includes the summary, visual display, and comparative analysis content of the evaluation metrics, providing a comprehensive evaluation and analysis of the performance of the autonomous driving system.

[0055] Existing evaluation methods for autonomous driving systems ignore the randomness of the traffic environment and natural environmental factors, resulting in a large difference between the real environment and the simulation environment during vehicle driving, and relying on a single evaluation index (such as passability) for evaluation, resulting in inaccurate evaluation results of autonomous driving systems.

[0056] Through this embodiment, simulation test data obtained after a specified vehicle undergoes simulation testing in a preset simulation scenario is acquired. The simulation test data has a first category label for identifying the target scenario category of the preset simulation scenario. According to the first category label, the simulation test data is parsed according to a set of evaluation indexes matching the target scenario category identified by the first category label. Among them, a set of evaluation indexes matching the target scenario category identified by the first category label includes a set of general indexes and a set of custom indexes. Different from the traditional method, first, the evaluation indexes used to evaluate the simulation test data in the embodiments of the present application are associated with the preset simulation scenario. As the preset simulation scenario changes, a set of evaluation indexes matching the target scenario category identified by the first category label also changes dynamically. That is, the evaluation indexes in the embodiments of the present application change dynamically following the preset simulation scenario. Second, the evaluation indexes in the embodiments of the present application have the characteristic of diversification. A set of evaluation indexes matching the target scenario category identified by the first category label includes a set of general indexes and a set of custom indexes. The custom indexes are preset according to the target scenario category. Different from the general indexes, the custom indexes can evaluate specific challenges in a specific scenario, avoiding the limitations of a single evaluation index (such as passability), and being able to more comprehensively evaluate the performance of the autonomous driving system. Therefore, according to the index quantization values of each evaluation index in the parsed set of evaluation indexes, the accuracy of the evaluation report of the specified vehicle in the preset simulation scenario is higher, improving the accuracy and comprehensiveness of the evaluation results, and solving the technical problem in the related art that the single evaluation index leads to inaccurate evaluation results of the autonomous driving system.

[0057] In an exemplary embodiment, parsing the simulation test data according to a set of evaluation indexes matching the target scenario category identified by the first category label includes:

[0058] According to the first category label, extract the evaluation indexes matching the target scenario category from the preset evaluation index set to obtain a set of evaluation indexes; the evaluation index set includes multiple general indexes and multiple custom indexes; each general index in the multiple general indexes corresponds to at least one scenario category; each custom index in the multiple custom indexes corresponds to at least one scenario category; parse the simulation test data according to each evaluation index to obtain the index quantization value of each evaluation index.

[0059] Among them, the evaluation index set is a comprehensive set that includes multiple general indicators and multiple custom indicators. General indicators are basic evaluation indicators applicable to multiple scenario categories and have wide applicability. For example, driving distance, driving time, average speed, etc. Each general indicator corresponds to at least one scenario category, but may also be applicable to multiple scenario categories. Custom indicators are evaluation indicators designed for specific scenario categories and can evaluate specific challenges in specific scenarios. For example, obstacle avoidance ability in complex traffic environments, stability in adverse weather conditions, etc. Each custom indicator also corresponds to at least one scenario category, but may also be extended to other relevant scenario categories according to requirements.

[0060] Optionally, the terminal device extracts the first category label from the simulation test data. When the target scenario category identified by the first category label is a single scenario, for each general indicator in the preset evaluation index set, it detects whether each general indicator corresponds to the target scenario category identified by the first category label. If it corresponds, it selects that general indicator; for each custom indicator in the preset evaluation index set, it detects whether each custom indicator corresponds to the target scenario category identified by the first category label. If it corresponds, it selects that custom indicator, and obtains a set of evaluation indicators that match the target scenario category. This set of evaluation indicators includes both a set of general indicators and a set of custom indicators. When the target scenario category identified by the first category label is a combined simulation scenario, a set of general indicators that match the target scenario category is extracted from the preset evaluation index set. The sum of a set of custom indicators that match each simulation scenario in the combined scenario can be used as a set of custom indicators that match the target scenario category identified by the first category label. Alternatively, the weight of each custom evaluation indicator in a set of custom indicators that match each simulation scenario in the combined simulation scenario can be preset in advance. According to the preset weight, a preset number of custom indicators are selected from a set of custom indicators corresponding to each simulation scenario in the combined simulation scenario as a set of custom indicators that match the target scenario category identified by the first category label. The terminal device parses the simulation test data according to each general indicator, based on the definition and calculation method of each general indicator, to obtain the index quantization value corresponding to each general indicator. The terminal device parses the simulation test data according to each custom indicator, based on the definition and calculation method of each custom indicator, to obtain the index quantization value corresponding to each custom indicator.

[0061] Through this embodiment, an evaluation index set including a plurality of general indexes and a plurality of custom indexes is provided. Among them, each general index in the evaluation index set corresponds to at least one scenario category, ensuring the generality and applicability of the evaluation indexes. At the same time, each custom index in the evaluation index set also corresponds to at least one scenario category, further enhancing the pertinence and flexibility of the evaluation indexes. The custom indexes can evaluate specific challenges in specific scenarios, so as to more accurately reflect the performance of the autonomous driving system in complex environments; According to the first category label, evaluation indexes matching the target scenario category are extracted from the preset evaluation index set to obtain a set of evaluation indexes including a plurality of general indexes and a plurality of custom indexes. This approach breaks the limitation of a single evaluation index, enabling the evaluation process to cover more dimensions and aspects, thus improving the accuracy and comprehensiveness of the evaluation.

[0062] In an exemplary embodiment, each evaluation index corresponds to an evaluation function, and each evaluation function corresponding to an evaluation index has at least one input parameter.

[0063] Among them, the evaluation function refers to the mathematical model or algorithm corresponding to each evaluation index, and is used to calculate and obtain the quantization value of the evaluation index according to the input parameters. The evaluation function is the specific implementation method of the evaluation index, and it can convert the simulation test data into numerical results that can be compared and analyzed.

[0064] The input parameter is the input data or variable required by the evaluation function. The input parameters usually come from simulation test data, such as vehicle state information (speed, acceleration, position, etc.), environmental information (road type, traffic conditions, weather conditions, etc.), and system control parameters, etc.

[0065] In some embodiments, the simulation test data is parsed according to each evaluation index to obtain the index quantization value of each evaluation index, including:

[0066] According to the input parameters of the evaluation function corresponding to each evaluation index, the parameter values of the input parameters of the evaluation function corresponding to each evaluation index are extracted from the simulation test data; According to the evaluation function corresponding to each evaluation index and the parameter values of the input parameters of the evaluation function corresponding to each evaluation index, the index quantization value of each evaluation index is determined.

[0067] Optionally, the terminal device pre-sets an evaluation function corresponding to each evaluation metric and stores it in the terminal device. After determining a set of evaluation metrics that match the target scenario category identified by the first category label, for each evaluation metric in the set of evaluation metrics, the terminal device looks up the corresponding evaluation function, identifies the input parameters required by the evaluation function, extracts the parameter values of the input parameters corresponding to the evaluation function from the simulation test data, and substitutes the parameter values of the input parameters into the corresponding evaluation function for calculation to obtain the corresponding metric quantization value.

[0068] Through this embodiment, when determining the metric quantization value of each evaluation metric, by calculating according to the evaluation function and the parameter values of its input parameters, it can be ensured that each metric has a specific and quantized numerical result, which can more accurately reflect the performance differences of the autonomous driving system in various aspects and improve the evaluation accuracy.

[0069] In an exemplary embodiment, parsing the simulation test data according to the first category label according to a set of evaluation metrics that match the target scenario category identified by the first category label includes:

[0070] Inputting the first category label and the simulation test data into a pre-trained evaluation model to obtain the metric quantization value of each evaluation metric output by the evaluation model; the evaluation model is used to extract, according to the input category label, the evaluation metrics that match the scenario category identified by the input category label from the evaluation metric set, and parse out the metric quantization values of the extracted evaluation metrics from the input simulation data.

[0071] Among them, the evaluation model is a pre-trained model that receives two inputs: the first category label and the simulation test data, and outputs the metric quantization value of each evaluation metric. The evaluation model not only integrates the evaluation metric set, but also integrates the evaluation function corresponding to each evaluation metric in the evaluation metric set. The main function of the evaluation model is to extract, according to the input category label, the evaluation metrics that match the scenario category identified by the input category label from the predefined evaluation metric set. Then, the evaluation model will parse out the metric quantization values corresponding to these extracted evaluation metrics from the input simulation test data. Finally, the evaluation model outputs the metric quantization value of each evaluation metric.

[0072] Optionally, Figure 4 is a schematic diagram of the application of the evaluation model in an embodiment, as Figure 4As shown, the terminal device inputs the simulation test data and the first category label of the simulation test data into a pre-trained evaluation model. According to the input first category label, the evaluation model extracts a set of evaluation metrics that match the current scenario category from a predefined set of evaluation metrics, parses out the metric quantization values corresponding to the extracted evaluation metrics based on the input simulation test data, and outputs the calculated metric quantization values for each evaluation metric.

[0073] Through this embodiment, using the evaluation model to output the metric quantization values of each evaluation metric according to the input simulation test data and the first category label of the simulation test data. Throughout the process, the evaluation model is not only responsible for extracting the evaluation metrics but also for parsing out the quantization values corresponding to these evaluation metrics from the input simulation test data, which can ensure the quantization accuracy of the evaluation metrics.

[0074] In an exemplary embodiment, each evaluation metric corresponds to an evaluation function, and each evaluation function corresponding to an evaluation metric has at least one input parameter.

[0075] In some embodiments, inputting the first category label and the simulation test data into a pre-trained evaluation model to obtain the metric quantization values of each evaluation metric output by the evaluation model includes:

[0076] Inputting the first category label and the simulation test data into a pre-trained evaluation model so that the evaluation model obtains the metric quantization values of each evaluation metric by performing the following parsing operations:

[0077] According to the first category label, extract the evaluation metrics that match the target scenario category from the set of evaluation metrics to obtain a set of evaluation metrics; extract the parameter values of the input parameters of the evaluation function corresponding to each evaluation metric from the simulation test data according to the input parameters of the evaluation function corresponding to each evaluation metric; determine the metric quantization values of each evaluation metric according to the parameter values of the input parameters of the evaluation function corresponding to each evaluation metric.

[0078] Optionally, as Figure 4As shown in the figure, the terminal device inputs the simulation test data and the first category label of the simulation test data into a pre-trained evaluation model. When the target scenario category identified by the first category label is a single scenario, the evaluation model extracts a set of evaluation indicators (including a set of general indicators and a set of custom indicators) that match the target scenario category from a preset evaluation indicator set according to the input first category label; when the target scenario category identified by the first category label is a combined simulation scenario, a set of general indicators that match the target scenario category is extracted from the preset evaluation indicator set. The sum of a set of custom indicators that match each simulation scenario in the combined scenario can be used as a set of custom indicators that match the target scenario category identified by the first category label. Alternatively, weights for each custom evaluation indicator in a set of custom indicators that match each simulation scenario in the combined simulation scenario can be preset. According to the preset weights, a preset number of custom indicators are selected from a set of custom indicators corresponding to each simulation scenario in the combined simulation scenario as a set of custom indicators that match the target scenario category identified by the first category label. For each extracted evaluation indicator, the evaluation model extracts the corresponding parameter values from the simulation test data according to the input parameter requirements of its corresponding evaluation function. The evaluation model determines the index quantization value of each evaluation indicator through corresponding calculation processes according to the evaluation function corresponding to each evaluation indicator and the parameter values of its input parameters, and outputs the calculated index quantization value of each evaluation indicator.

[0079] Through this embodiment, the evaluation model extracts a set of evaluation indicators that match the target scenario category from the evaluation indicator set according to the first category label, achieving precise screening of the evaluation indicators, ensuring that the selected evaluation indicators are highly relevant to the scenario characteristics of the current simulation test data, thereby improving the pertinence and accuracy of the evaluation. Secondly, before the index calculation, the evaluation functions of each evaluation indicator in the evaluation indicator set are integrated in the evaluation model. During the process of calculating the index quantization value corresponding to the evaluation indicator, the evaluation model extracts the corresponding parameter values from the simulation test data according to the input parameters of the evaluation function corresponding to each evaluation indicator, not only considering the corresponding relationship between the evaluation indicator and the simulation test data, but also ensuring the accuracy of the evaluation indicator quantization value through precise calculation methods, which helps to reduce errors and improve the reliability and credibility of the evaluation.

[0080] In an exemplary embodiment, the training method of the evaluation model includes the following steps:

[0081] 1. Obtain multiple training samples; each training sample among the multiple training samples includes historical simulation test data of a historical vehicle; the historical simulation test data in each training sample refers to the simulation data obtained after the historical vehicle undergoes a simulation test in a preset simulation scenario; each training sample has a second category label and a corresponding historical index quantization value; the second category label of each training sample is used to identify the historical scenario category corresponding to each training sample; the historical index quantization value corresponding to each training sample is the historical index quantization value of the evaluation index matched with the historical scenario category corresponding to each training sample.

[0082] Among them, a training sample refers to the sample data used to train an evaluation model. The training sample contains the historical simulation test data of a historical vehicle and its corresponding information.

[0083] A historical vehicle refers to a vehicle that underwent a simulation test in the past, and its test data is used to train the model.

[0084] Historical simulation test data refers to the simulation data obtained after the historical vehicle undergoes a simulation test in a preset simulation scenario, including vehicle driving status, environmental perception information, system control decisions, etc.

[0085] The second category label is a label used to identify the historical scenario category corresponding to each training sample, such as urban roads, highways, complex intersections, etc.

[0086] The historical index quantization value refers to the historical quantization result of the evaluation index matched with the historical scenario category corresponding to each training sample, and is used to measure the performance of the vehicle in the historical scenario.

[0087] Optionally, Figure 5 is a training schematic diagram of the evaluation model in an embodiment, such as Figure 5 shown, the terminal device collects the simulation data obtained after the simulation test of multiple historical vehicles in different preset simulation scenarios, assigns a second category label to each training sample to identify the historical scenario category corresponding to it; according to the historical scenario category of each training sample, determines a set of historical evaluation indicators matched with the second category label of each training sample, and parses the historical simulation test data according to each historical evaluation indicator in the set of historical evaluation indicators to obtain the historical index quantization value corresponding to each training sample.

[0088] 2. Use multiple training samples to train the evaluation model to be trained to obtain a trained evaluation model. Among them, during the process of training the evaluation model, the model parameters of the evaluation model are adjusted according to the difference between the predicted index quantization value corresponding to each training sample output by the evaluation model and the historical index quantization value corresponding to each training sample.

[0089] Optionally, such asFigure 5 As shown in Figure 5 , the terminal device inputs each training sample into the evaluation model to obtain the predicted metric quantization value corresponding to each training sample. According to the difference between the predicted metric quantization value of each training sample and the corresponding historical metric quantization value, the value of the loss function is calculated. Using the backpropagation algorithm, the gradient of the loss function is passed to the model parameters for parameter update, completing one iteration training process. Repeat the above iteration training process until the value of the loss function reaches the preset convergence condition or the number of training rounds reaches the preset upper limit, then stop training to obtain the trained evaluation model.

[0090] Through this embodiment, each training sample has a second category label and the corresponding historical metric quantization value. These labels and quantization values are used to identify and measure the performance in different scenarios. Through the training of multiple samples, the evaluation model can learn the performance characteristics in different scenarios, thereby improving the accuracy of evaluation; during the model training process, the model parameters of the evaluation model are adjusted according to the difference between the predicted metric quantization value output by the model and the historical metric quantization value. This adjustment method enables the model to gradually reduce the prediction error and optimize the model parameters, thereby improving the accuracy of evaluation; using multiple training samples for model training is actually training the model with multiple evaluation metrics and data in multiple scenarios, fundamentally solving the problem of inaccurate evaluation results caused by a single evaluation metric in the background art.

[0091] In an exemplary embodiment, obtaining the simulation test data of a specified vehicle includes:

[0092] Constructing a preset simulation scenario; the preset simulation scenario is selected from a pre-constructed simulation scenario library; the simulation scenario library includes at least one of the following simulation scenarios: urban road environment, highway environment, tunnel environment, rain and snow weather, road bridge with crosswind; testing the specified vehicle in the preset simulation scenario to obtain the simulation test data.

[0093] Among them, the existing simulation test methods only use pre-designed scenario segments for testing, which is difficult to meet the requirements of test scenario coverage. And the existing methods cannot test and evaluate the power and braking performance of the autonomous driving system for meteorological factors and mountain road driving vehicles, resulting in inaccurate evaluation results of the autonomous driving system. Therefore, to solve the above problems, when designing the preset simulation scenario in this embodiment, instead of using fixed scenario segments, a simulation scenario library is constructed based on the historical driving data of the vehicle driving in different scenarios. Each simulation scenario in the simulation scenario library includes parameters such as meteorological factors, power and braking performance of the vehicle in different sections. The formed simulation scenario not only covers more test scenarios, but also can use the simulation scenario of this application to test and evaluate the power and braking performance of the autonomous driving system for meteorological factors and mountain road driving vehicles, thereby improving the accuracy of the evaluation results.

[0094] The simulation scenario library is a database containing various simulation scenarios. The simulation scenarios in the simulation scenario library are designed according to real road environments and traffic conditions and are used for simulating and testing autonomous vehicles. The scenarios in the simulation scenario library can include urban road environments, highway environments, tunnel environments, rainy and snowy weather, road and bridge sections with crosswinds, etc.

[0095] Optionally, the terminal device pre - designs and constructs various simulation scenarios and converts the constructed various simulation scenarios into a simulation scenario library recognizable by a computer. The terminal device selects a suitable preset simulation scenario from the simulation scenario library according to the test requirements, and configures a simulation test environment according to the selected preset simulation scenario, including setting vehicle parameters, road environment parameters, traffic flow parameters, etc. The terminal device constructs a dynamic model of the specified vehicle and conducts simulation tests on the dynamic model of the specified vehicle according to the preset simulation scenario, and records simulation test data such as the driving state of the vehicle, environmental perception information, and system control decisions.

[0096] Through this embodiment, a preset simulation scenario library containing various simulation scenarios such as urban road environments, highway environments, tunnel environments, rainy and snowy weather, road and bridge sections with crosswinds, etc. is constructed, providing a more comprehensive and diverse test environment for the performance evaluation of autonomous vehicles, meeting the requirements for test scenario coverage. The simulation scenario test of this application can be used to evaluate the autonomous driving system for meteorological factors and the power and braking performance of vehicles driving on mountain roads, thereby improving the accuracy and reliability of the evaluation.

[0097] It should be noted that for the foregoing method embodiments, for the sake of simple description, they are all expressed as a series of action combinations. However, those skilled in the art should know that this application is not limited by the described action sequence, because according to this application, certain steps can be performed in other sequences or simultaneously. Secondly, those skilled in the art should also know that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily essential for this application.

[0098] Through the description of the above embodiments, those skilled in the art can clearly understand that the method according to the above embodiments can be implemented by means of software plus a necessary general hardware platform. Of course, it can also be implemented by hardware, but in many cases the former is a better implementation. Based on such an understanding, the technical solution of the present application, in essence, or the part that contributes to the prior art can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM (Read-Only Memory), RAM (Random Access Memory), magnetic disk, optical disk), and includes several instructions for causing a terminal device (which can be a mobile phone, computer, server, or network device, etc.) to execute the methods described in various embodiments of the present application.

[0099] According to another aspect of the embodiments of the present application, there is also provided a test data processing device for an autonomous driving system. This test data processing device for an autonomous driving system can be used to implement the test data processing method for the autonomous driving system provided in the above embodiments, and those that have been described will not be repeated. As used hereinafter, the term "module" can be a combination of software and / or hardware that can achieve a predetermined function. Although the devices described in the following embodiments are preferably implemented in software, implementation in hardware, or a combination of software and hardware is also possible and contemplated.

[0100] Figure 6 is a structural block diagram of an optional test data processing device for an autonomous driving system according to the embodiments of the present application. As Figure 6 shown in, this test data processing device for an autonomous driving system includes:

[0101] A simulation module 602, configured to obtain simulation test data of a specified vehicle; the simulation test data refers to the simulation data obtained after the specified vehicle undergoes simulation tests in a preset simulation scenario; the simulation test data has a first category label; the first category label is used to identify the target scenario category of the preset simulation scenario;

[0102] An evaluation module 604, configured to parse the simulation test data according to the first category label according to a set of evaluation indicators matching the target scenario category identified by the first category label; the set of evaluation indicators includes a set of general indicators and a set of custom indicators; each custom indicator in the set of custom indicators is an indicator preset according to the target scenario category and different from the general indicators;

[0103] A report generation module 606, configured to generate an evaluation report of the specified vehicle in the preset simulation scenario according to the index quantization value of each evaluation indicator in the parsed set of evaluation indicators.

[0104] It should be noted that the simulation module 602 in this embodiment can be used to execute the above step S202, the evaluation module 604 in this embodiment can be used to execute the above step S204, and the report generation module 606 in this embodiment can be used to execute the above step S206.

[0105] Through the embodiment provided by the present application, simulation test data obtained after a specified vehicle undergoes a simulation test in a preset simulation scenario is acquired. The simulation test data has a first category label for identifying the target scenario category of the preset simulation scenario. According to the first category label, the simulation test data is parsed according to a set of evaluation indicators that match the target scenario category identified by the first category label. Among them, a set of evaluation indicators that match the target scenario category identified by the first category label includes a set of general indicators and a set of custom indicators. Different from the traditional method, first, the evaluation indicators used to evaluate the simulation test data in the embodiment of the present application are associated with the preset simulation scenario. As the preset simulation scenario changes, a set of evaluation indicators that match the target scenario category identified by the first category label also changes dynamically, that is, the evaluation indicators in the embodiment of the present application change dynamically following the preset simulation scenario. Second, the evaluation indicators in the embodiment of the present application have the characteristic of diversification. A set of evaluation indicators that match the target scenario category identified by the first category label includes a set of general indicators and a set of custom indicators. The custom indicators are preset according to the target scenario category. Different from the general indicators, the custom indicators can evaluate specific challenges in a specific scenario, avoiding the limitations of a single evaluation indicator (such as passability), and can more comprehensively evaluate the performance of the autonomous driving system. Therefore, according to the index quantization value of each evaluation indicator in the parsed set of evaluation indicators, the accuracy of the evaluation report of the specified vehicle in the preset simulation scenario is higher, improving the accuracy and comprehensiveness of the evaluation result, and solving the technical problem that the evaluation result of the autonomous driving system is not accurate enough due to a single evaluation indicator in the related art.

[0106] In an exemplary embodiment, the evaluation module 604 is further configured to extract, according to the first category label, evaluation indicators that match the target scenario category from a preset set of evaluation indicators to obtain a set of evaluation indicators; the set of evaluation indicators includes multiple general indicators and multiple custom indicators; each general indicator among the multiple general indicators corresponds to at least one scenario category; each custom indicator among the multiple custom indicators corresponds to at least one scenario category; the simulation test data is parsed according to each evaluation indicator respectively to obtain the index quantization value of each evaluation indicator.

[0107] In an exemplary embodiment, each evaluation metric corresponds to an evaluation function, and the evaluation function corresponding to each evaluation metric has at least one input parameter; the evaluation module 604 is further configured to extract, from the simulation test data, the parameter values of the input parameters of the evaluation function corresponding to each evaluation metric according to the input parameters of the evaluation function corresponding to each evaluation metric; and determine the metric quantization values of each evaluation metric according to the evaluation function corresponding to each evaluation metric and the parameter values of the input parameters of the evaluation function corresponding to each evaluation metric.

[0108] In an exemplary embodiment, the evaluation module 604 is further configured to input the first category label and the simulation test data into a pre-trained evaluation model, and obtain the metric quantization values of each evaluation metric output by the evaluation model; the evaluation model is configured to extract, from the evaluation metric set, the evaluation metrics that match the scenario category identified by the input category label according to the input category label, and parse out the metric quantization values of the extracted evaluation metrics from the input simulation data.

[0109] In an exemplary embodiment, each evaluation metric corresponds to an evaluation function, and the evaluation function corresponding to each evaluation metric has at least one input parameter; the evaluation module 604 is further configured to input the first category label and the simulation test data into a pre-trained evaluation model, so that the evaluation model can perform the following parsing operations to obtain the metric quantization values of each evaluation metric: extract, from the evaluation metric set, the evaluation metrics that match the target scenario category according to the first category label, and obtain a set of evaluation metrics; extract, from the simulation test data, the parameter values of the input parameters of the evaluation function corresponding to each evaluation metric according to the input parameters of the evaluation function corresponding to each evaluation metric; and determine the metric quantization values of each evaluation metric according to the parameter values of the input parameters of the evaluation function corresponding to each evaluation metric.

[0110] In an exemplary embodiment, the evaluation module 604 is further configured to obtain a plurality of training samples; each training sample in the plurality of training samples includes the historical simulation test data of a historical vehicle; the historical simulation test data in each training sample refers to the simulation data obtained after the historical vehicle has undergone a simulation test in a preset simulation scenario; each training sample has a second category label and a corresponding historical metric quantization value; the second category label of each training sample is used to identify the historical scenario category corresponding to each training sample; the historical metric quantization value corresponding to each training sample is the historical metric quantization value of the evaluation metric that matches the historical scenario category corresponding to each training sample; use the plurality of training samples to perform model training on the evaluation model to be trained, and obtain a trained evaluation model, wherein, in the process of performing model training on the evaluation model, the model parameters of the evaluation model are adjusted according to the difference between the predicted metric quantization value corresponding to each training sample output by the evaluation model and the historical metric quantization value corresponding to each training sample.

[0111] In an exemplary embodiment, the simulation module 602 is further configured to construct a preset simulation scenario; the preset simulation scenario is selected from a pre-constructed simulation scenario library; the simulation scenario library includes at least one of the following simulation scenarios: urban road environment, highway environment, tunnel environment, rain and snow weather, and road and bridge with crosswind; test a specified vehicle in the preset simulation scenario to obtain simulation test data.

[0112] It should be noted that the above-mentioned various modules can be implemented by software or hardware. For the latter, it can be implemented in the following ways, but not limited to: the above-mentioned modules are all located in the same processor; or, the above-mentioned various modules are respectively located in different processors in any combination form.

[0113] According to another aspect of the embodiments of the present application, a computer-readable storage medium is provided. The computer-readable storage medium includes a stored program, wherein when the program runs, it executes the steps in any one of the above method embodiments.

[0114] In an exemplary embodiment, the above computer-readable storage medium may include, but is not limited to: USB flash drive, ROM, RAM, mobile hard disk, magnetic disk or optical disc and other various media that can store computer programs.

[0115] According to another aspect of the embodiments of the present application, an electronic device is provided, including a memory, a processor, and a computer program stored on the memory and executable on the processor. The processor is configured to execute the steps in any one of the above method embodiments through the computer program. In an exemplary embodiment, the above electronic device may further include a transmission device and an input / output device, wherein the transmission device is connected to the above processor, and the input / output device is connected to the above processor.

[0116] The specific examples in this embodiment may refer to the examples described in the above embodiments and exemplary embodiments, and will not be repeated here.

[0117] According to another aspect of the embodiments of the present application, a computer program product is further provided. The computer program product includes computer programs / instructions, and the computer programs / instructions include program codes for executing the methods shown in the flowcharts. In such an embodiment, the computer program can be downloaded and installed from the network through the communication part 709, and / or installed from the removable medium 711. When the computer program is executed by the central processing unit 701, it executes various functions provided by the embodiments of the present application. The serial numbers of the above embodiments of the present application are only for description and do not represent the advantages and disadvantages of the embodiments.

[0118] Figure 7A computer system block diagram for implementing the electronic device according to the embodiments of the present application is schematically shown. As Figure 7 shown, the computer system 700 includes a CPU (Central Processing Unit) 701, which can perform various appropriate actions and processes according to the program stored in the ROM 702 or the program loaded from the storage section 708 into the RAM 703. In the random access memory 703, various programs and data required for system operation are also stored. The central processing unit 701, the read-only memory 702, and the random access memory 703 are connected to each other via a bus 704. An I / O (Input / Output) interface 705 is also connected to the bus 704.

[0119] The following components are connected to the I / O interface 705: an input section 706 including a keyboard, a mouse, etc.; an output section 707 including, for example, a CRT (Cathode Ray Tube), an LCD (Liquid Crystal Display), etc. and a speaker, etc.; a storage section 708 including a hard disk, etc.; and a communication section 709 including a network interface card such as a LAN card, a modem, etc. The communication section 709 performs communication processing via a network such as the Internet. A drive 710 is also connected to the input / output interface 705 as needed. A removable medium 711, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 710 as needed, so that the computer program read from it can be installed into the storage section 708 as needed.

[0120] In particular, according to the embodiments of the present application, the processes described in each method flowchart can be implemented as computer software programs. For example, the embodiments of the present application include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program contains program codes for executing the methods shown in the flowcharts. In such an embodiment, the computer program can be downloaded and installed from the network through the communication section 709, and / or installed from the removable medium 711. When the computer program is executed by the central processing unit 701, various functions defined in the system of the present application are executed.

[0121] It should be noted that Figure 7 the computer system 700 of the shown electronic device is only an example, and should not impose any limitation on the functions and usage scope of the embodiments of the present application.

[0122] Obviously, those skilled in the art should understand that the various modules or steps of the present application described above can be implemented by a general-purpose computing device. They can be concentrated on a single computing device or distributed on a network composed of multiple computing devices. They can be implemented by program codes executable by the computing device. Thus, they can be stored in a storage device and executed by the computing device. And in some cases, the steps shown or described can be executed in a different order from here, or they can be separately fabricated into individual integrated circuit modules, or multiple modules or steps among them can be fabricated into a single integrated circuit module for implementation. In this way, the present application is not limited to any specific combination of hardware and software.

[0123] The above are only the preferred embodiments of the present application and are not used to limit the present application. For those skilled in the art, the present application can have various changes and modifications. Any modification, equivalent replacement, improvement, etc. made within the principle of the present application shall be included within the protection scope of the present application.

Claims

1. A test data processing method for an automatic driving system, characterized in that: include: Acquire simulation test data of a specified vehicle; the simulation test data refers to simulation data obtained after the specified vehicle is simulated and tested under a preset simulation scenario; The simulation test data has a first category label; The first category label is used to identify the target scene category of the preset simulation scene; According to the first category label, the simulation test data is parsed according to a set of evaluation indicators matching the target scene category identified by the first category label; the set of evaluation indicators includes a set of general indicators and a set of custom indicators; each custom indicator in the set of custom indicators is a pre-set indicator different from the general indicator according to the target scene category; An evaluation report of the designated vehicle in the preset simulation scenario is generated according to the index quantization value of each evaluation index in the group of evaluation indexes obtained by parsing.

2. The method according to claim 1, characterized in that The step of parsing the simulation test data according to the first category label and a set of evaluation indicators matching the target scene category identified by the first category label includes: According to the first category label, extracting evaluation indicators matching the target scene category from a preset evaluation indicator set to obtain the set of evaluation indicators; the evaluation indicator set includes multiple general indicators and multiple custom indicators; each of the multiple general indicators corresponds to at least one scene category; each of the multiple custom indicators corresponds to at least one scene category; The simulation test data is analyzed according to each evaluation index to obtain the index quantization value of each evaluation index.

3. The method according to claim 2, characterized in that Each evaluation index corresponds to an evaluation function, and the evaluation function corresponding to each evaluation index has at least one input parameter; The step of respectively parsing the simulation test data according to each evaluation index to obtain the index quantization value of each evaluation index includes: According to the input parameters of the evaluation function corresponding to each evaluation indicator, extracting the parameter values ​​of the input parameters of the evaluation function corresponding to each evaluation indicator from the simulation test data; The indicator quantization value of each evaluation indicator is determined according to the evaluation function corresponding to each evaluation indicator and the parameter value of the input parameter of the evaluation function corresponding to each evaluation indicator.

4. The method according to claim 1, characterized in that: The step of parsing the simulation test data according to the first category label and a set of evaluation indicators matching the target scene category identified by the first category label includes: The first category label and the simulation test data are input into a pre-trained evaluation model to obtain the indicator quantification value of each evaluation indicator output by the evaluation model; the evaluation model is used to extract the evaluation indicator that matches the scene category identified by the input category label from the evaluation indicator set according to the input category label, and parse the indicator quantification value of the extracted evaluation indicator from the input simulation data.

5. The method according to claim 4, characterized in that Each evaluation index corresponds to an evaluation function, and the evaluation function corresponding to each evaluation index has at least one input parameter; The step of inputting the first category label and the simulation test data into a pre-trained evaluation model to obtain the index quantization value of each evaluation index output by the evaluation model includes: The first category label and the simulation test data are input into the pre-trained evaluation model, so that the evaluation model obtains the indicator quantization value of each evaluation indicator by performing the following parsing operation: Extracting, according to the first category label, evaluation indicators matching the target scene category from the evaluation indicator set to obtain the set of evaluation indicators; According to the input parameters of the evaluation function corresponding to each evaluation indicator, extracting the parameter values ​​of the input parameters of the evaluation function corresponding to each evaluation indicator from the simulation test data; The indicator quantization value of each evaluation indicator is determined according to the parameter value of the input parameter of the evaluation function corresponding to each evaluation indicator.

6. The method according to claim 4, characterized in that The method further comprises: Acquire multiple training samples; each of the multiple training samples includes historical simulation test data of a historical vehicle; the historical simulation test data in each training sample refers to simulation data obtained after the historical vehicle has been simulated and tested under a preset simulation scenario; each training sample has a second category label and a corresponding historical indicator quantization value; the second category label of each training sample is used to identify the historical scene category corresponding to each training sample; the historical indicator quantization value corresponding to each training sample is the historical indicator quantization value of the evaluation indicator matching the historical scene category corresponding to each training sample; The evaluation model to be trained is trained using the multiple training samples to obtain the trained evaluation model, wherein, in the process of training the evaluation model, the model parameters of the evaluation model are adjusted according to the difference between the quantitative value of the prediction index corresponding to each training sample output by the evaluation model and the quantitative value of the historical index corresponding to each training sample.

7. The method according to any one of claims 1 to 6, characterized in that The step of obtaining the simulation test data of the specified vehicle includes: Constructing the preset simulation scene; the preset simulation scene is selected from a pre-constructed simulation scene library; the simulation scene library includes at least one of the following simulation scenes: urban road environment, high-speed environment, tunnel environment, rainy and snowy weather, and road and bridge with crosswind; The designated vehicle is tested in the preset simulation scenario to obtain the simulation test data.

8. A test data processing device for an automatic driving system, characterized in that: include: A simulation module, used to obtain simulation test data of a specified vehicle; the simulation test data refers to simulation data obtained after the specified vehicle is simulated and tested under a preset simulation scenario; The simulation test data has a first category label; The first category label is used to identify the target scene category of the preset simulation scene; An evaluation module, configured to parse the simulation test data according to a set of evaluation indicators matching the target scene category identified by the first category label according to the first category label; the set of evaluation indicators includes a set of general indicators and a set of custom indicators; each custom indicator in the set of custom indicators is a pre-set indicator different from the general indicator according to the target scene category; The report generation module is used to generate an evaluation report of the specified vehicle in the preset simulation scenario according to the index quantization value of each evaluation index in the group of evaluation indexes analyzed.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, wherein the computer program implements the steps of any one of the methods of claims 1 to 7 when executed by a processor.

10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.

Images