Test method and device of automatic driving decision-making system and electronic equipment

Abstract

The invention discloses a test method and device for an automatic driving decision-making system and electronic equipment. The method comprises the following steps: acquiring initial capture information collected by a target vehicle in a driving process; based on the initial capture information, generating a target driving track matched with the preset driving scene through an automatic driving decision-making system carried on the simulation equipment; determining a plurality of track data matched with the preset driving scene in the target driving track; and testing the automatic driving decision-making system based on the multiple pieces of trajectory data to obtain a test result of the automatic driving decision-making system. According to the invention, the technical problem of low efficiency of testing the automatic driving decision-making system in the prior art is solved.

Description

Technical Field

[0001] The present invention relates to the field of autonomous driving, and in particular to a testing method, device and electronic equipment for an autonomous driving decision system. Background Art

[0002] In autonomous driving technology, the autonomous driving decision system is responsible for analyzing road conditions and making driving decisions based on the information provided by the perceived vehicle and preset goals to achieve safe driving of the vehicle. The decision-making ability of the autonomous driving decision system is related to the safety and reliability of the vehicle. Therefore, it is necessary to verify and evaluate the decision-making ability of the autonomous driving decision system, that is, to test the autonomous driving decision system, which can include testing the performance and stability of the system under various conditions to ensure that the system can drive safely and efficiently, to discover possible problems and defects in the system, and to improve and optimize the system.

[0003] At present, the relevant technologies mostly use real vehicle road testing methods to test the autonomous driving decision-making system. The vehicles and driving conditions that need to be tested are diverse and complex, resulting in low efficiency in testing the autonomous driving decision-making system.

[0004] To address the above-mentioned problems, no effective solution has been proposed yet. Summary of the invention

[0005] Embodiments of the present invention provide a method, device, and electronic device for testing an autonomous driving decision system to at least solve the technical problem of low efficiency in testing the autonomous driving decision system in related technologies.

[0006] According to one aspect of an embodiment of the present invention, there is provided a method for testing an autonomous driving decision system, comprising: obtaining initial capture information collected by a target vehicle during driving; based on the initial capture information, generating a target driving trajectory that matches a preset driving scenario by simulating an autonomous driving decision system mounted on a device; determining a plurality of trajectory data in the target driving trajectory that matches the preset driving scenario; and testing the autonomous driving decision system based on the plurality of trajectory data to obtain a test result of the autonomous driving decision system.

[0007] Optionally, based on the initial capture information, an autonomous driving decision system mounted on a simulation device is used to generate a target driving trajectory that matches a preset driving scenario, including: filtering out target capture information that matches the preset driving scenario from the initial capture information; and generating a trajectory based on the target capture information through the autonomous driving decision system to obtain a target driving trajectory.

[0008] Optionally, based on the initial captured information, a target driving trajectory matching a preset driving scenario is generated by simulating an autonomous driving decision system mounted on a device, including: generating a trajectory based on the initial captured information by the autonomous driving decision system to obtain an initial driving trajectory; and screening out a target driving trajectory matching the preset driving scenario from the initial driving trajectory.

[0009] Optionally, based on multiple trajectory data, the autonomous driving decision system is tested to obtain test results of the autonomous driving decision system, including: obtaining preset trajectory data corresponding to the multiple trajectory data; comparing each trajectory data with the corresponding preset trajectory data to obtain a verification result of the trajectory data, wherein the verification result is used to characterize the degree of difference between the trajectory data and the preset trajectory data; and determining the test result based on the verification results of the multiple trajectory data.

[0010] Optionally, each trajectory data is compared with the corresponding preset trajectory data to obtain a verification result of the trajectory data, including: obtaining vehicle condition information of the target vehicle, wherein the vehicle condition information is used to characterize the vehicle type and / or current state of the target vehicle; determining a trajectory test limit corresponding to the preset trajectory data based on the vehicle condition information; and comparing each trajectory data with the trajectory test limit to obtain a verification result.

[0011] Optionally, based on the verification results of multiple trajectory data, the test results are determined, including: determining target weights corresponding to the multiple verification results based on a preset driving scenario, wherein the target weights are used to characterize the degree of influence of the corresponding verification results on the decision-making ability of the autonomous driving decision-making system; and weighting the multiple verification results based on the target weights to obtain the test results.

[0012] Optionally, the autonomous driving decision system includes multiple versions of decision systems; after testing the autonomous driving decision system based on multiple trajectory data to obtain test results of the autonomous driving decision system, the method also includes one of the following: determining a target version of the decision system from the multiple versions of the decision systems based on the test results of the multiple versions of the decision systems, and updating the target vehicle based on the target version of the decision system; sorting the test results of the multiple versions of the decision systems according to the iteration time of the multiple versions of the decision systems to obtain a test result sequence, and determining the quality change of the target driving trajectory based on the test result sequence.

[0013] According to another aspect of an embodiment of the present invention, a testing device for an autonomous driving decision system is also provided, including: an acquisition module, used to acquire initial capture information collected by a target vehicle during driving; a generation module, used to generate a target driving trajectory matching a preset driving scenario by simulating the autonomous driving decision system mounted on a device based on the initial capture information; a determination module, used to determine multiple trajectory data in the target driving trajectory matching the preset driving scenario; and a test module, used to test the autonomous driving decision system based on the multiple trajectory data to obtain a test result of the autonomous driving decision system.

[0014] According to another aspect of an embodiment of the present invention, there is further provided an electronic device, comprising: a memory storing an executable program; and a processor for running the program, wherein the method in each embodiment of the present invention is executed when the program is running.

[0015] According to another aspect of an embodiment of the present invention, a computer-readable storage medium is provided. The computer-readable storage medium includes a stored executable program, wherein when the executable program is running, the device where the computer-readable storage medium is located is controlled to execute the methods in various embodiments of the present invention.

[0016] According to another aspect of an embodiment of the present invention, a computer program product is provided, including a computer program. When the computer program is executed by a processor, the method in each embodiment of the present invention is implemented.

[0017] According to another aspect of an embodiment of the present invention, a computer program product is provided, including a non-volatile computer-readable storage medium, wherein the non-volatile computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method in each embodiment of the present invention is implemented.

[0018] According to another aspect of the embodiments of the present invention, a computer program is further provided. When the computer program is executed by a processor, the methods in the embodiments of the present invention are implemented.

[0019] In an embodiment of the present invention, initial capture information collected by a target vehicle during driving is obtained; based on the initial capture information, a target driving trajectory matching a preset driving scenario is generated by simulating an automatic driving decision system mounted on a device; multiple trajectory data matching the preset driving scenario in the target driving trajectory are determined; based on the multiple trajectory data, the automatic driving decision system is tested to obtain a test result of the automatic driving decision system. It is easy to notice that, in the process of testing the autonomous driving decision system, the present application generates a target driving trajectory by carrying a simulation device of the autonomous driving decision system, simulating the actual driving scene of the target vehicle, and can simulate the operation of the autonomous driving decision system in different driving scenes. By setting preset driving scenes, testers can simulate various situations that may be encountered in the real world, verify the performance and stability of the autonomous driving system in different situations, and verify whether the behavior and decision of the system in various scenarios meet the design expectations. It can effectively simulate the behavior and decision of the autonomous driving system in various driving scenarios, help verify the reliability and performance of the system, and help discover potential problems of the system and make improvements. By screening the multiple trajectory data obtained, parameters related to the specific preset driving scene can be extracted, so as to evaluate the performance of the system in the scene, which can help understand the behavior and performance of the system in different situations. By comparing different trajectory data, the performance of the system in different driving scenarios can be evaluated, and the problems and improvement space of the autonomous driving decision system can be found, so as to achieve a more comprehensive test of the autonomous driving decision system, thereby solving the technical problem of low efficiency in testing the autonomous driving decision system in related technologies. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] The drawings described herein are used to provide a further understanding of the present invention and constitute a part of this application. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the drawings:

[0021] Figure 1 is a flow chart of a method for testing an automatic driving decision system according to an embodiment of the present invention;

[0022] Figure 2 is a schematic diagram of an optional automatic driving decision system testing process according to an embodiment of the present invention;

[0023] Figure 3 is a schematic diagram of a testing device for an automatic driving decision system according to an embodiment of the present invention. DETAILED DESCRIPTION

[0024] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0025] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0026] According to one aspect of an embodiment of the present invention, a test of an autonomous driving decision-making system is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in an order different from that shown here.

[0027] Figure 1 is a flow chart of a method for testing an automatic driving decision system according to an embodiment of the present invention. Figure 1 As shown, the method comprises the following steps:

[0028] Step S102, obtaining initial capture information collected during the driving process of the target vehicle.

[0029] The above-mentioned autonomous driving decision system may refer to a system in an autonomous driving vehicle that is responsible for analyzing and processing vehicle surrounding environment information, formulating driving routes and controlling vehicle driving. The autonomous driving decision system may have multiple versions as it is upgraded and iterated. The target vehicle may be equipped with a certain version of the autonomous driving decision system and may be upgraded and iterated. The autonomous driving decision system may obtain environmental information through perception sensors, such as radars and cameras, and then use algorithms and models to process and analyze the environmental information and finally make a decision, that is, it may generate a running trajectory of the target vehicle, or multiple trajectory points, which may include vehicle speed, steering, acceleration and deceleration, etc., to ensure that the vehicle drives safely and efficiently on the road.

[0030] The above-mentioned target vehicle may refer to a vehicle that can be equipped with an autonomous driving decision-making system and can perform autonomous driving. The target vehicle may include but is not limited to sedans, sport utility vehicles (SUV), multi-purpose vehicles (MPV), buses, trucks, and may include engineering vehicles such as excavators, loaders, and mine cars. It may also include three-wheeled vehicles, two-wheeled vehicles, etc. that can be equipped with an autonomous driving decision-making system. The target vehicle can be determined according to actual needs and is not limited here.

[0031] The above-mentioned initial capture information may refer to information about the surrounding environment, road conditions, other vehicles and pedestrians perceived and located by the target vehicle through various sensors, cameras and other equipment during driving. The initial capture information may include but is not limited to the position, speed, acceleration of the target vehicle, the position and speed of surrounding vehicles, road signs, traffic light status, etc. The initial capture information can be used as input information for the autonomous driving system to make decisions and plans. The specific content of the initial capture information can be determined according to actual needs and is not limited here.

[0032] In an optional embodiment, during the test of the autonomous driving decision system, the initial captured information collected by the target vehicle during driving can be obtained, which may include collecting various types of data and information of the target vehicle during driving, and the data of the target vehicle during driving collected by various sensor devices on the target vehicle, such as cameras, lidars, millimeter-wave radars, etc., may include the vehicle's position, speed, acceleration, surrounding environment, etc.; the collected data may be stored in a database or file for subsequent processing and analysis; the collected data may be processed, including data cleaning, screening, noise removal and other operations to ensure the accuracy and reliability of the data; the processed data may be analyzed to extract the driving characteristics and behavior patterns of the target vehicle in order to evaluate its performance and performance under different circumstances; the analysis results may also be displayed and visualized. Through the above steps, the initial captured information collected by the target vehicle during driving can be obtained, providing data support for subsequent testing and evaluation, which can help evaluate the performance of the autonomous driving system, discover potential problems and room for improvement, and optimize the system's decision-making and planning algorithms to improve the safety and reliability of the system.

[0033] Step S104, based on the initial captured information, a target driving trajectory matching a preset driving scenario is generated by simulating the autonomous driving decision system mounted on the device.

[0034] The above-mentioned simulation equipment may refer to equipment used to simulate the driving conditions of a target vehicle equipped with an autonomous driving decision-making system on the road. The simulation equipment may include a computer system, equipment equipped with virtual simulation software, and may also include various types of simulated vehicles corresponding to the target vehicle.

[0035] The above-mentioned preset driving scenarios may refer to pre-set specific driving environments and scenarios. The preset driving scenarios may include different road conditions, traffic conditions, weather conditions, obstacles and other factors, as well as specific situations in which the autonomous driving system needs to make decisions. For example, the preset driving scenario may be set to be driving on an urban road, and encountering a vehicle in front that suddenly brakes, requiring the autonomous driving decision system to respond quickly and avoid a collision; another preset driving scenario may be driving on a highway, when the road is slippery due to heavy rain, the response capability and driving stability of the autonomous driving decision system can be tested; the preset driving scenario can also be set to a scenario in which a mine car is driving on the same section of road with different load states, so as to test the decision-making processing capability of the autonomous driving decision system for different vehicle states. By setting various preset driving scenarios, the performance of the autonomous driving system in various different situations can be comprehensively evaluated, providing an important reference for the optimization and improvement of the system. The preset driving scenarios can be determined according to actual needs and are not limited here.

[0036] The above-mentioned target driving trajectory may refer to a vehicle driving trajectory generated by an autonomous driving decision system mounted on a simulation device based on a preset driving scenario and initial captured information. The target driving trajectory may be composed of trajectory points generated by the autonomous driving decision system.

[0037] In an optional embodiment, during the testing of the autonomous driving decision-making system, a target driving trajectory can be generated by a simulation device equipped with the autonomous driving decision-making system to simulate the actual driving scenario of the target vehicle to verify whether the system's behavior and decision-making in various scenarios meet the design expectations. Specifically, a preset driving scenario can be set according to the actual driving scenario and test requirements, and the autonomous driving decision-making system carried on the simulation device can generate a target driving trajectory that matches the preset driving scenario based on the initial captured information and the preset driving scenario, which can include the autonomous driving decision-making system's perception of the surrounding environment in the initial captured information, path planning and decision-making based on the preset driving scenario.

[0038] Through the above steps, the behavior and decision-making of the autonomous driving system in various driving scenarios can be effectively simulated to help verify the reliability and performance of the system. Through the simulation equipment, the autonomous driving decision-making system can be simulated to run and make decisions in different driving scenarios, such as urban roads, highways, complex intersections, etc., to generate corresponding driving trajectories and evaluate the performance and decision-making ability of the autonomous driving system in these scenarios, which can help developers and researchers evaluate the performance of the system, improve algorithms, and verify the stability and safety of the system in various scenarios; by setting preset driving scenarios, testers can simulate various situations that may be encountered in the real world, verify the performance and stability of the autonomous driving system in different situations, help discover potential problems in the system and make improvements, and improve the reliability and safety of the autonomous driving system on actual roads.

[0039] Step S106, determining a plurality of trajectory data in the target driving trajectory that match the preset driving scene.

[0040] The above-mentioned multiple trajectory data may refer to a series of driving trajectory data generated according to different driving scenarios and different vehicle states. The multiple trajectory data may include but are not limited to data such as the vehicle's speed, direction, acceleration, etc., which can be determined according to actual needs and are not limited here.

[0041] In an optional embodiment, the obtained target driving trajectory can be screened based on a preset driving scenario to obtain multiple trajectory data. For example, when the preset driving scenario is that the vehicle is driving on a curve, the trajectory data related to the vehicle turning in the target driving trajectory can be determined. At this time, the multiple trajectory data can include parameters such as the vehicle's turning radius and roll angular velocity. For another example, when the preset driving scenario is that the vehicle is in an overtaking scenario, the trajectory data related to the vehicle overtaking scenario in the target driving trajectory can be determined. At this time, the multiple trajectory data can include parameters such as the vehicle's speed and acceleration.

[0042] This application can obtain multiple trajectory data by screening the target driving trajectory based on preset driving scenarios, which is helpful for evaluating and testing the performance of the system in different driving scenarios. By setting different driving scenarios, such as curve driving and overtaking scenarios, different driving situations can be simulated to determine the parameters related to the scenario in the target driving trajectory. By screening the multiple trajectory data obtained, parameters related to the specific driving scenario can be extracted to evaluate the performance of the system in the scenario, which can help understand the behavior and performance of the system in different situations, and then make adjustments and optimizations.

[0043] Step S108: testing the autonomous driving decision system based on the multiple trajectory data to obtain a test result of the autonomous driving decision system.

[0044] In an optional embodiment, the performance and stability of the autonomous driving decision system can be analyzed based on multiple trajectory data, and the performance of the system in different driving scenarios can be evaluated by comparing different trajectory data to obtain the test results of the autonomous driving decision system. The performance of the autonomous driving decision system can also be evaluated based on the test results, which can include but are not limited to indicators such as accuracy, response time, and decision accuracy, to find problems and room for improvement in the autonomous driving decision system. Improvement plans can also be proposed based on the test results to optimize the performance of the autonomous driving decision system and improve the stability and safety of the system. Through the above steps, the autonomous driving decision system can be tested more comprehensively to ensure that the system can make correct decisions in various situations and improve the reliability and safety of the autonomous driving system.

[0045] In an embodiment of the present invention, initial capture information collected by a target vehicle during driving is obtained; based on the initial capture information, a target driving trajectory matching a preset driving scenario is generated by simulating an automatic driving decision system mounted on a device; multiple trajectory data matching the preset driving scenario in the target driving trajectory are determined; based on the multiple trajectory data, the automatic driving decision system is tested to obtain a test result of the automatic driving decision system. It is easy to notice that, in the process of testing the autonomous driving decision system, the present application generates a target driving trajectory by carrying a simulation device of the autonomous driving decision system, simulating the actual driving scene of the target vehicle, and can simulate the operation of the autonomous driving decision system in different driving scenes. By setting preset driving scenes, testers can simulate various situations that may be encountered in the real world, verify the performance and stability of the autonomous driving system in different situations, and verify whether the behavior and decision of the system in various scenarios meet the design expectations. It can effectively simulate the behavior and decision of the autonomous driving system in various driving scenarios, help verify the reliability and performance of the system, and help discover potential problems of the system and make improvements. By screening the multiple trajectory data obtained, parameters related to the specific preset driving scene can be extracted, so as to evaluate the performance of the system in the scene, which can help understand the behavior and performance of the system in different situations. By comparing different trajectory data, the performance of the system in different driving scenarios can be evaluated, and the problems and improvement space of the autonomous driving decision system can be found, so as to achieve a more comprehensive test of the autonomous driving decision system, thereby solving the technical problem of low efficiency in testing the autonomous driving decision system in related technologies.

[0046] Optionally, based on the initial capture information, an autonomous driving decision system mounted on a simulation device is used to generate a target driving trajectory that matches a preset driving scenario, including: filtering out target capture information that matches the preset driving scenario from the initial capture information; and generating a trajectory based on the target capture information through the autonomous driving decision system to obtain a target driving trajectory.

[0047] In an optional embodiment, initial capture information of the entire process can be obtained from the target vehicle, which may include information such as the vehicle's position, speed, acceleration, and the positions of surrounding vehicles and obstacles. It can also be obtained through various on-board sensors, positioning systems and other equipment. The collected data can also be processed and screened to remove irrelevant information and extract target capture information related to preset driving scenarios. Machine learning algorithms can also be used to classify and cluster the data to identify target driving scenarios. The target capture information extraction process may include identifying target capture information that matches a preset driving scenario in the processed data. Based on the extracted target capture information, the autonomous driving decision system may use a path planning algorithm to generate a target driving trajectory. The generated target driving trajectory may be verified and tested in a simulation environment to evaluate the performance of the autonomous driving system in the preset scenario. Through the above steps, it is possible to filter out target capture information that matches a preset driving scenario from the initial capture information, and generate a target driving trajectory through the autonomous driving decision system. In the above process, the target capture information that matches a preset driving scenario is filtered out through the preset driving scenario, which can reduce the amount of data that the autonomous driving decision system needs to process for decision-making, shorten the test time of the autonomous driving decision system, and improve the test efficiency. Only data that matches the preset driving scenario will be used by the user to test the autonomous driving decision system, thereby avoiding the impact of invalid test data on system performance evaluation. By accurately selecting target capture information, it is possible to ensure that the test data is representative and accurate, thereby improving the accuracy of the test of the autonomous driving decision system.

[0048] Optionally, based on the initial captured information, a target driving trajectory matching a preset driving scenario is generated by simulating an autonomous driving decision system mounted on a device, including: generating a trajectory based on the initial captured information by the autonomous driving decision system to obtain an initial driving trajectory; and screening out a target driving trajectory matching the preset driving scenario from the initial driving trajectory.

[0049] In an optional embodiment, based on the initial capture information, an automatic driving decision system can be used to generate an initial driving trajectory. The initial driving trajectory can be obtained after comprehensive consideration of factors such as the vehicle's current position, speed, acceleration, and surrounding environment. After the initial driving trajectory is generated, a matching target driving trajectory can be screened out according to a preset driving scenario. Specifically, the initial driving trajectory can be evaluated, including indicators such as matching degree with the preset scenario, safety, and efficiency. Screening conditions can be set according to the preset driving scenario and requirements, such as avoiding collisions, maintaining vehicle distance, and complying with traffic rules. Through the above steps, the automatic driving decision system can generate a target driving trajectory. By generating the initial driving trajectory based on the initial capture information, the system can flexibly adjust the driving path according to the acquired environmental information and target position, thereby improving the adaptability and intelligence of the system, and screening out a target driving trajectory that matches the preset driving scenario from the initial driving trajectory, so as to achieve targeted testing of the automatic driving decision system based on the preset driving scenario.

[0050] Optionally, based on multiple trajectory data, the autonomous driving decision system is tested to obtain test results of the autonomous driving decision system, including: obtaining preset trajectory data corresponding to the multiple trajectory data; comparing each trajectory data with the corresponding preset trajectory data to obtain a verification result of the trajectory data, wherein the verification result is used to characterize the degree of difference between the trajectory data and the preset trajectory data; and determining the test result based on the verification results of the multiple trajectory data.

[0051] The above-mentioned preset trajectory data may refer to the pre-set trajectory data conditions, thresholds or ranges that need to be met for the vehicle to drive normally and safely under the current preset driving scenario. The preset trajectory data may be from simulation, real road driving records or professional road databases, etc., and may include vehicle position, speed, acceleration, direction and other information. Different preset driving scenarios and different types of trajectory data may correspond to different preset trajectory data. The preset trajectory data may be set according to actual needs and is not limited here.

[0052] In an optional embodiment, the process of comparing each trajectory data with the corresponding preset trajectory data, specifically, the generated trajectory data and the preset trajectory data can be formatted to ensure that the multiple trajectory data and the preset trajectory data have the same data structure and unit, each data point can be compared and calculated, and the comparison methods such as Euclidean distance and dynamic time warping can be used to calculate the degree of difference between the multiple trajectory data and the preset trajectory data, and the verification result of each trajectory data can be generated according to the result of the comparison calculation, which can be a numerical value or a sign; the test result can be determined based on the verification results of the multiple trajectory data, specifically, the verification results of each trajectory data can be comprehensively evaluated, and the weighted average, maximum and minimum values ​​and other methods can be used to determine the overall test result, and the comprehensive evaluation results can be compared with the standards according to the preset test standards or rules to determine whether the autonomous driving decision system has passed the test, and a test report can be generated to evaluate the performance and stability of the system. Through the above steps, the autonomous driving decision system can be tested and evaluated more comprehensively to ensure its stability and safety in various complex scenarios.

[0053] Optionally, each trajectory data is compared with the corresponding preset trajectory data to obtain a verification result of the trajectory data, including: obtaining vehicle condition information of the target vehicle, wherein the vehicle condition information is used to characterize the vehicle type and / or current state of the target vehicle; determining a trajectory test limit corresponding to the preset trajectory data based on the vehicle condition information; and comparing each trajectory data with the trajectory test limit to obtain a verification result.

[0054] The above-mentioned vehicle type may refer to a specific vehicle model or vehicle category to which the target vehicle belongs, such as a sedan, bus, truck, or engineering vehicle, etc. It may also be further divided. For example, a sedan may be further divided into a small sedan (Type A), a compact sedan (Type B), a mid-size sedan (Type C), and a large sedan (Type D). Different types of vehicles may have different performance characteristics and behavior patterns when driving.

[0055] The above-mentioned current state of the vehicle may refer to various actual states of the target vehicle, including but not limited to the load state of the target vehicle, the vehicle's mileage, the lubricating oil and coolant temperature in the vehicle, the degree of tire wear of the vehicle, etc. The current state of the vehicle can be determined according to actual needs and is not limited here.

[0056] The above-mentioned trajectory test limit can be calculated based on vehicle condition information and preset trajectory data, or it can be a standard value set in advance. Through the trajectory test limit, the driving condition of the target vehicle can be tested more accurately. The trajectory test limit can be determined according to actual needs and is not limited here.

[0057] In an optional embodiment, during the testing of the autonomous driving decision system, vehicle condition information of the target vehicle may be obtained to more accurately simulate the behavior and status of the target vehicle. The vehicle condition information may include the vehicle type and / or the current status of the vehicle. Based on the obtained vehicle condition information, a corresponding trajectory test limit may be determined according to preset trajectory data. The trajectory test limit may include the vehicle's maximum speed, maximum acceleration, maximum steering angle, etc. For example, if the target vehicle is a truck, then when simulating the target vehicle's behavior, it is necessary to consider the characteristics of the truck's slower acceleration performance and larger turning radius; if the target vehicle is a car, it is possible to consider the characteristics of the target vehicle's faster acceleration performance and more flexible steering. Based on the vehicle type and current state information, determining the trajectory test limit corresponding to the preset trajectory data can set the test conditions more accurately, which can ensure the comprehensiveness and effectiveness of the test. By comparing each trajectory data with the trajectory test limit, a verification result can be obtained to evaluate the performance of the autonomous driving decision system under different vehicle types and states. Each trajectory data can be compared with the corresponding trajectory test limit to obtain a verification result. Each trajectory data can be compared with the trajectory test limit to obtain a verification result to evaluate whether the actual driving conditions of the target vehicle meet expectations. During the comparison process, the difference between the actual state of the target vehicle and the preset trajectory data and test limit can be checked to determine whether the driving of the target vehicle is safe and stable. The verification result can help optimize the algorithm of the decision system and improve the performance and reliability of the autonomous driving system.

[0058] Optionally, based on the verification results of multiple trajectory data, the test results are determined, including: determining target weights corresponding to the multiple verification results based on a preset driving scenario, wherein the target weights are used to characterize the degree of influence of the corresponding verification results on the decision-making ability of the autonomous driving decision-making system; and weighting the multiple verification results based on the target weights to obtain the test results.

[0059] The above-mentioned target weight may refer to the weight value set for different verification results during the testing process of the autonomous driving decision-making system. By setting different target weights, the present application can prioritize different verification results in order to more accurately evaluate the performance and stability of the system.

[0060] In an optional embodiment, a preset driving scenario in a test case can be determined. For each preset driving scenario, the test case can be executed and the verification result of the system can be recorded. The target weight corresponding to each verification result can be determined according to the test requirements and actual conditions. The target weight can be determined according to expert experience, historical data or model analysis, etc., and multiple verification results can be weighted based on the target weight. Specifically, for each preset driving scenario, the various verification results under the scenario can be weighted and summed according to their corresponding target weights to obtain a comprehensive verification result under the scenario. Each preset driving scenario can be tested and the comprehensive verification result can be calculated. Finally, the performance of the system under different circumstances can be evaluated according to the comprehensive verification results under each scenario, and corresponding adjustments and improvements can be made. Through the above steps, the decision-making ability of the automatic driving decision system in various driving scenarios can be effectively evaluated, providing a basis for further optimization of the system; in the process of determining the target weights corresponding to multiple verification results based on the preset driving scenario, the importance of each verification result can be determined according to the actual road conditions and system design requirements, and the corresponding weight value can be assigned. For example, when the target vehicle performs a passenger transport task, a higher target weight can be assigned to the verification result of avoiding collision, which is related to the safety of passengers. When the present application obtains the test results after weighted processing of multiple verification results, the use of target weights can enable the impact of different verification results to be reasonably reflected in the test results. By reasonably setting the target weights, the overall performance of the system can be more accurately evaluated, and a useful reference can be provided for further optimization and improvement of the autonomous driving decision-making system.

[0061] Optionally, the autonomous driving decision system includes multiple versions of decision systems; after testing the autonomous driving decision system based on multiple trajectory data to obtain test results of the autonomous driving decision system, the method also includes one of the following: determining a target version of the decision system from the multiple versions of the decision systems based on the test results of the multiple versions of the decision systems, and updating the target vehicle based on the target version of the decision system; sorting the test results of the multiple versions of the decision systems according to the iteration time of the multiple versions of the decision systems to obtain a test result sequence, and determining the quality change of the target driving trajectory based on the test result sequence.

[0062] In an optional embodiment, multiple versions of decision systems can be used to compare and verify the performance of the autonomous driving decision system. These versions may include different algorithms, parameter settings or models. Multiple versions of decision systems can be prepared based on simulation equipment. These systems may be from different development teams or obtained through training with different training data sets. Test indicators and test cases can be determined. The indicators and test cases can cover multiple aspects of the system, including performance, accuracy, robustness, etc.; multiple versions of decision systems can be tested using the prepared test indicators and test cases, and the test results and performance data can be recorded. The test results can be compared and analyzed to compare the performance of multiple versions of decision systems in various indicators. Based on the results of the comparative analysis, the target version of the decision system can be determined, that is, the version with better performance and reliability. Finally, the target vehicle can be updated based on the target version of the decision system, which can include software updates or hardware upgrades to ensure that the vehicle can obtain a good decision system. Through the above steps, a better decision system version can be effectively selected and applied to the target vehicle, thereby improving the performance and safety of the autonomous driving system.

[0063] In another optional embodiment, during the testing of the autonomous driving decision system, the performance and quality of different versions of the decision system can be evaluated, and the specific requirements and conditions of the test scenario can be determined, including factors such as road environment, traffic conditions, and weather, to ensure the comprehensiveness and accuracy of the test. According to the requirements of the test scenario, a series of specific test cases can be designed, including various road conditions, traffic rules, obstacles, etc., to cover the behavior of the decision system in different situations. The designed test cases can be executed one by one in a simulator or an actual road environment to record the behavior and results of each version of the decision system. The behavior data of each version of the decision system under different test cases can be collected, including indicators such as vehicle speed, steering, and obstacle avoidance capabilities. Based on the collected data, the performance of each version of the decision system can be evaluated and the performance of different versions can be compared. According to the test results, multiple versions of the decision system can be sorted to determine the better version. According to the test result sequence, the quality changes of the target driving trajectory can be analyzed, and the adaptability and stability of different versions of the decision system in actual scenarios can be compared. The test results can be statistically analyzed and visualized to better understand and compare the performance of each version of the decision system.

[0064] In the current autonomous driving decision system, the decision planning module sends the trajectory of the expected vehicle movement to the control module. This trajectory may contain several trajectory points with information such as vehicle position, vehicle orientation, turning curvature, vehicle speed limit, speed, acceleration, etc. The quality of the trajectory points can affect the rationality, smoothness and safety of the autonomous driving vehicle movement. For example, if the turning speed of the trajectory point is too large at a certain moment, it may cause the vehicle to roll over. In the related technology, when paying attention to the quality of the trajectory points, it is usually checked by checking the operation log of the autonomous driving vehicle. Although this method can determine that the quality of the trajectory points in a certain driving process meets the requirements, the amount of data for manual inspection is small, and it is easy to miss problems. In particular, the vehicle operation log in the above method is fixed, and the R&D personnel can only obtain the trajectory points output by the current version of the program in the current environment. It is difficult to compare the quality of trajectory points between versions, so as to ensure that the quality of trajectory points has not deteriorated during the version iteration process. In addition, most autonomous driving companies only verify the correctness of the decision and the rationality of the trajectory during the testing process, and there is no systematic verification of the quality of the trajectory points.

[0065] This application can simulate the actual vehicle operation scenario through data playback, and can obtain the trajectory points generated by different versions of the decision planning program. After that, the trajectory point quality can be fully verified to ensure that the trajectory points meet the requirements of the dynamic model, vehicle model restrictions, etc. Finally, the verification results can be statistically analyzed to obtain a comparison of the trajectory point quality of different versions of the decision planning program, thereby helping to make targeted improvements to the autonomous driving decision system.

[0066] The present application can obtain data for data playback, can record the entire process data of the operation of an autonomous driving vehicle, or can only record scene data that is prone to problems, which can include scenes such as cornering, sudden braking, and short-distance starting. Data that meets the requirements can be filtered out from the entire process data based on pre-set features. For example, data in specific scenarios, such as sharp turns and sudden braking, can be filtered using an industrial computer. Raw data can be directly obtained from the autonomous driving control system (ADC) on the actual vehicle. Data related to trajectory generation, such as perception information, prediction, positioning, application, vehicle-to-everything (V2X), etc., can be obtained from the actual vehicle or constructed manually.

[0067] The autonomous driving control system (ADC) can also generate complete trajectory points based on the full process data. The industrial computer can screen them during verification, and then the recorded data can be placed in the cloud for data set management. A pipeline for data playback testing can be built on a pipeline website (for example, Jenkins). The data playback process can include that after the user selects the test version and data set, the test program will be deployed to the ADC, and the test set data can be deployed to an industrial computer connected to the ADC. The data set data can be played back, which can be various data packets encapsulated by the communication protocol and sent to the ADC. The decision program running on the ADC can process the data set data and generate a trajectory in the same way as on the autonomous driving vehicle. The generated trajectory can be recorded by the industrial computer and used for subsequent comprehensive quality verification.

[0068] Figure 2 is a schematic diagram of an optional automatic driving decision system testing process according to an embodiment of the present invention, such as Figure 2 As shown, an engineering machine and an autonomous driving control system can be used in the testing process of the autonomous driving decision system. The data management module in the engineering machine can replay the initial data packet required by the decision module to the decision module in the autonomous driving control system. The decision module can obtain the initial data packet frame by frame and make algorithmic decisions. The verification module in the engineering machine can record the data packet generated by the decision module.

[0069] The above-mentioned initial data packet may be initial capture information or target capture information, and the data packet generated by the above-mentioned decision module may be the target driving trajectory; the verification module may determine multiple trajectory data in the target driving trajectory that match the preset driving scene; and based on the multiple trajectory data, the autonomous driving decision system is tested to obtain the test results of the autonomous driving decision system.

[0070] This application performs trajectory point quality verification to check whether the trajectory points meet various kinematic model indicators and vehicle model restrictions. For example, the maximum acceleration of a certain vehicle model under no load is 1.3m / s 2 , the maximum acceleration under heavy load is 0.8m / s 2 , directly check whether the trajectory point meets the conditions; for example, the curvature change rate speed limit check can be performed, the curvature change rate of the trajectory point can be calculated, and the maximum speed limit value under the curvature change rate can be obtained according to the linear difference method. Finally, it can be checked whether the planned speed of the trajectory point meets the speed limit value.

[0071] The present application can obtain the pass rates of various checks of a certain version of a decision-making program under a playback test of a certain data set. Multiple versions of the test program can be used to perform playback tests on the same data set to check whether the quality of the trajectory points is optimized or degraded in a certain indicator. For different scenarios, different dimensions and different test contents can be used to test data of different scenarios. The test contents can be different, or the test contents can be the same but the constraint values ​​are different. For different scenarios, a single decision-making program, multiple decision-making programs, a single data set, multiple data sets, etc. can be tested. Different weights, that is, the importance of indicators, can be given for test contents of different dimensions, and the pass rate can be determined in combination with the weights to select the test version.

[0072] Based on the above-mentioned method for testing the autonomous driving decision-making system proposed in the present application, the trajectory point quality verification capability can be improved, and various customized and flexible verifications can be performed on the trajectory points output by the unmanned vehicle, and this can be completed automatically; the trajectory point quality verification scope can be improved, and testers can record data sets under various scenarios, build a wide range of scenario sets, and ensure the integrity of the verification; the trajectory point quality version monitoring capability can be obtained, and by using multiple versions of test programs to replay the same data set for testing, it can be intuitively viewed that with the iteration of versions, the optimization and degradation of the trajectory point quality can be visualized, and the decision-making program can make targeted improvements on the degraded verification indicators based on the monitoring results.

[0073] According to another aspect of an embodiment of the present invention, a testing device for an autonomous driving decision-making system is also provided. The device can execute the testing method for the autonomous driving decision-making system of the above-mentioned embodiment. The specific implementation method and preferred application scenario are the same as those of the above-mentioned embodiment and will not be elaborated here.

[0074] Figure 3 is a schematic diagram of a test device for an autonomous driving decision system according to an embodiment of the present application, such as Figure 3 As shown, the device includes the following: an acquisition module 302 , a generation module 304 , a determination module 306 and a testing module 308 .

[0075] Among them, the acquisition module 302 is used to obtain the initial capture information collected by the target vehicle during the driving process; the generation module 304 is used to generate a target driving trajectory matching the preset driving scene based on the initial capture information by simulating the automatic driving decision system installed on the device; the determination module 306 is used to determine multiple trajectory data in the target driving trajectory that match the preset driving scene; the test module 308 is used to test the automatic driving decision system based on multiple trajectory data to obtain the test results of the automatic driving decision system.

[0076] Among them, the generation module is also used to filter out target capture information that matches the preset driving scenario from the initial capture information; and generate a trajectory based on the target capture information through the automatic driving decision system to obtain the target driving trajectory.

[0077] Among them, the generation module is also used to generate a trajectory based on the initial captured information through the automatic driving decision system to obtain an initial driving trajectory; and to screen out a target driving trajectory that matches a preset driving scenario from the initial driving trajectory.

[0078] Among them, the test module is also used to obtain preset trajectory data corresponding to multiple trajectory data; compare each trajectory data with the corresponding preset trajectory data to obtain a verification result of the trajectory data, wherein the verification result is used to characterize the degree of difference between the trajectory data and the preset trajectory data; and determine the test result based on the verification results of the multiple trajectory data.

[0079] Among them, the test module is also used to obtain vehicle condition information of the target vehicle, wherein the vehicle condition information is used to characterize the vehicle type and / or current state of the target vehicle; determine the trajectory test limit corresponding to the preset trajectory data based on the vehicle condition information; compare each trajectory data with the trajectory test limit to obtain a verification result.

[0080] Among them, the test module is also used to determine the target weights corresponding to multiple verification results based on the preset driving scenario, wherein the target weights are used to characterize the degree of influence of the corresponding verification results on the decision-making ability of the autonomous driving decision-making system; based on the target weights, multiple verification results are weighted to obtain test results.

[0081] Among them, the autonomous driving decision system includes multiple versions of decision systems; the testing module is also used to determine the target version of the decision system from the multiple versions of the decision systems based on the test results of the multiple versions of the decision systems, and update the target vehicle based on the target version of the decision system; the test results of the multiple versions of the decision systems are sorted according to the iteration time of the multiple versions of the decision systems to obtain a test result sequence, and based on the test result sequence, the quality change of the target driving trajectory is determined.

[0082] An embodiment of the present application further provides an electronic device, comprising: a memory storing an executable program; and a processor for running the program, wherein the program executes the methods in various embodiments of the present invention when running.

[0083] The above-mentioned memory may refer to a device inside a computer for storing data and programs, and may include memory, hard disk, etc., wherein the memory may be used to temporarily store running programs and data, the hard disk may be used to store programs and data for a long time, and the memory may be used to enable the computer to read and write data, and execute programs; the above-mentioned processor may be responsible for executing instructions in computer programs and performing data processing, and may be responsible for controlling and executing various operations, including arithmetic operations, logical operations, data transmission, etc.

[0084] An embodiment of the present application further provides a computer-readable storage medium, which includes a stored executable program, wherein when the executable program is running, the device where the computer-readable storage medium is located is controlled to execute the methods in various embodiments of the present invention.

[0085] The above-mentioned computer storage medium may refer to a medium in a computer memory used to store certain discontinuous physical quantities. Computer storage media mainly include semiconductors, magnetic cores, magnetic drums, magnetic tapes, laser disks, etc.; the stored program included in the computer-readable storage medium may be a set of instructions that can be recognized and executed by a computer, running on an electronic computer, and is an information tool that meets certain needs of people.

[0086] An embodiment of the present application further provides a computer program product, including a computer program, which implements the methods in various embodiments of the present invention when executed by a processor.

[0087] The above-mentioned computer program product may refer to a software program that has been written, tested and released and can be run on a computer or other device. The computer program product may include an application, an operating system, tool software, etc., which is used to implement specific functions or solve specific problems.

[0088] An embodiment of the present application further provides a computer program product, including a non-volatile computer-readable storage medium, wherein the non-volatile computer-readable storage medium is used to store a computer program, and when the computer program is executed by a processor, the method in each embodiment of the present invention is implemented.

[0089] The above-mentioned non-volatile computer-readable storage medium may refer to a medium for storing data. The non-volatile computer-readable storage medium can keep the data from being lost when the power is off, and can be used to store long-term data, such as operating systems, applications, and user files. The non-volatile storage medium may include hard disk drives, solid-state drives, optical disks, and flash memory storage devices, etc.

[0090] The embodiments of the present application further provide a computer program, which implements the methods in the above-mentioned embodiments of the present invention when executed by a processor.

[0091] The above-mentioned computer program may refer to a collection of instructions used to tell a computer to perform a specific task or operation. A computer program may be written by a programmer using a specific programming language and may include algorithms, data structures, logic, and control flows. A computer program may be used for a variety of purposes, including application software, operating systems, and the like.

[0092] In the above embodiments of the present invention, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0093] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device embodiments described above are only schematic. For example, the division of the units can be a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.

[0094] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple units. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0095] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0096] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), mobile hard disk, magnetic disk or optical disk and other media that can store program codes.

[0097] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principle of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A testing method for an autonomous driving decision system, characterized in that: include: Obtaining initial capture information collected by the target vehicle during driving; Based on the initial captured information, a target driving trajectory matching a preset driving scenario is generated by simulating an autonomous driving decision system mounted on the device; Determining a plurality of trajectory data in the target driving trajectory that matches the preset driving scene; Based on the multiple trajectory data, the autonomous driving decision system is tested to obtain a test result of the autonomous driving decision system.

2. The method for testing the automatic driving decision system according to claim 1, characterized in that: Based on the initial captured information, a target driving trajectory matching a preset driving scenario is generated by simulating an autonomous driving decision system mounted on the device, including: Filtering target capture information matching the preset driving scene from the initial capture information; The target driving trajectory is obtained by performing trajectory generation based on the target capture information through the automatic driving decision system.

3. The method for testing the automatic driving decision system according to claim 1, characterized in that: Based on the initial captured information, a target driving trajectory matching a preset driving scenario is generated by simulating an autonomous driving decision system mounted on the device, including: Generate a trajectory based on the initial captured information by the automatic driving decision system to obtain an initial driving trajectory; The target driving trajectory matching the preset driving scene is selected from the initial driving trajectory.

4. The method for testing an automatic driving decision system according to claim 1, characterized in that: Based on the plurality of trajectory data, the autonomous driving decision system is tested to obtain a test result of the autonomous driving decision system, including: Acquire preset trajectory data corresponding to the plurality of trajectory data; Comparing each trajectory data with the corresponding preset trajectory data to obtain a verification result of the trajectory data, wherein the verification result is used to characterize the degree of difference between the trajectory data and the preset trajectory data; The test result is determined based on the verification results of the plurality of trajectory data.

5. The method for testing the automatic driving decision system according to claim 4, characterized in that: Each trajectory data is compared with the corresponding preset trajectory data to obtain the verification result of the trajectory data, including: Acquiring vehicle status information of the target vehicle, wherein the vehicle status information is used to characterize the vehicle type and / or current state of the target vehicle; determining a trajectory test limit corresponding to the preset trajectory data based on the vehicle condition information; Each of the trajectory data is compared with the trajectory test limit to obtain the verification result.

6. The method for testing the automatic driving decision system according to claim 4, characterized in that: Determining the test result based on the verification results of the plurality of trajectory data includes: Determining target weights corresponding to the plurality of verification results based on the preset driving scenario, wherein the target weights are used to characterize the degree of influence of the corresponding verification results on the decision-making ability of the autonomous driving decision-making system; The multiple verification results are weighted based on the target weight to obtain the test result.

7. The method for testing the automatic driving decision system according to claim 1, characterized in that: The autonomous driving decision system includes multiple versions of decision systems; after testing the autonomous driving decision system based on the multiple trajectory data and obtaining the test result of the autonomous driving decision system, the method further includes one of the following: Based on the test results of the multiple versions of the decision systems, determining a target version of the decision system from the multiple versions of the decision systems, and updating the target vehicle based on the target version of the decision system; The test results of the multiple versions of the decision systems are sorted according to the iteration times of the multiple versions of the decision systems to obtain a test result sequence, and based on the test result sequence, the quality change of the target driving trajectory is determined.

8. A test device for an automatic driving decision system, characterized in that: include: An acquisition module is used to acquire initial capture information collected by the target vehicle during driving; A generation module, configured to generate a target driving trajectory matching a preset driving scenario by simulating an automatic driving decision system mounted on a device based on the initial captured information; A determination module, used to determine a plurality of trajectory data in the target driving trajectory that matches the preset driving scene; A testing module is used to test the autonomous driving decision system based on the multiple trajectory data to obtain a test result of the autonomous driving decision system.

9. An electronic device, characterized in that: include: A memory storing an executable program; A processor is used to run the program, wherein the program, when running, executes the testing method of the autonomous driving decision system described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium includes a stored executable program, wherein when the executable program is running, the device where the storage medium is located is controlled to execute the testing method of the autonomous driving decision-making system according to any one of claims 1 to 7.

Images