Field evaluation method and system for safety of self-driving automobile sensing system

Abstract

The invention discloses an on-site evaluation method and system for safety of a sensing system of an automatic driving automobile. The method comprises the following steps: acquiring an automatic driving scene for testing; determining a perception problem corresponding to the accident based on the automatic driving scene; determining testable parameters based on the perceptual problem; determining an accident occurrence degree grade and an accident severity grade based on the perception problem; determining the scene risk number of all perception problems based on the accident occurrence degree grade and the accident severity grade; establishing a sensing system safety evaluation table based on the scene risk number and the accident severity level; and performing safety evaluation on the to-be-tested sensing system based on the testable parameters and the sensing system safety evaluation table. The system overcomes the limitation of virtual simulation test, whole vehicle test and single sensor test of an automatic driving vehicle, and achieves the convenient, comprehensive and accurate evaluation of the safety of a sensing system.

Description

Technical Field

[0001] The present invention relates to the technical field of autonomous driving performance testing, and more specifically to an on-site evaluation method and system for the safety of an autonomous driving vehicle perception system. Background Art

[0002] The progress of artificial intelligence technology has promoted the vigorous development of autonomous driving technology. Compared with traditional driving methods, autonomous driving technology can reduce traffic accidents caused by human errors and improve the safety of vehicle operation. The survey found that the research focus of autonomous driving vehicles has gradually shifted from breakthroughs in autonomous driving technology to functional testing and evaluation technology. As a key link in the perception, decision-making, and execution functions of autonomous driving, the perception system collects and processes environmental and vehicle posture information in real time by integrating data from sensors such as cameras, lidar, and millimeter-wave radars, which is a prerequisite for the safe driving of autonomous vehicles. Once the autonomous driving system lacks perception functions, the decision-making system in the next link will not be able to respond correctly in time, which will eventually lead to accidents. Therefore, it is particularly important to carry out functional safety testing of the perception system to ensure the safety of the entire vehicle.

[0003] At present, the test methods commonly used in the industry are mainly concentrated in the following aspects: First, the use of simulation testing methods to test perception performance in a virtual environment, but a single performance cannot measure the safety of the perception system, and the set virtual environment cannot guarantee the validity of the test results; second, qualitative testing of the safety of the entire vehicle, but the "black box effect" in the test will lead to the unexplainability of the problem, and it is difficult to accurately identify the cause of the perception risk. Third, testing a single sensor, but the autonomous driving perception system is a complex system containing multi-modal sensors and fusion algorithms. Single sensor problems can be compensated at the system level through multi-source data fusion. Conversely, the sensor is normal but the fusion algorithm has problems, which can also lead to system-level perception problems; therefore, the measurement of a single sensor cannot reflect the safety of the perception system.

[0004] Therefore, how to overcome the limitations of using virtual simulation testing, whole vehicle testing and single sensor testing for autonomous vehicles, and thereby achieve a convenient, comprehensive and accurate evaluation of the safety of the perception system is an issue that technical personnel in this field urgently need to solve. Summary of the invention

[0005] In view of this, the present invention provides an on-site evaluation method and system for the safety of an autonomous driving vehicle's perception system, which overcomes the limitations of using autonomous driving vehicle virtual simulation testing, whole vehicle testing, and single sensor testing, and achieves a convenient, comprehensive, and accurate evaluation of the safety of the perception system.

[0006] In order to achieve the above object, the present invention adopts the following technical solution:

[0007] A field evaluation method for the safety of a perception system of an autonomous vehicle, comprising:

[0008] Obtain autonomous driving scenarios for testing;

[0009] Determining a perception problem corresponding to the accident based on the autonomous driving scenario;

[0010] determining a testable parameter based on the perception problem;

[0011] determining an accident occurrence level and an accident severity level based on the perception problem;

[0012] Determining a scenario risk number for all of the perceived problems based on the accident occurrence level and the accident severity level;

[0013] Establishing a perception system safety evaluation table based on the scenario risk number and the accident severity level;

[0014] A safety evaluation is performed on the perception system to be tested based on the testable parameters and the perception system safety evaluation table.

[0015] Preferably, the autonomous driving scenario includes:

[0016] Scenario 1: Following a car on a long straight road;

[0017] Scene 2: Pedestrians crossing the road on a long straight road;

[0018] Scene 3: Pedestrians are crossing the road at a traffic light intersection;

[0019] Scenario 4: There is a stationary obstacle on a long straight road with multiple lanes.

[0020] Preferably, the perception problems include: perception problems in scene one, perception problems in scene two, perception problems in scene three, and perception problems in scene four;

[0021] Among them, the perception problems in scenario one include: failure to perceive the leading vehicle, failure to perceive the position information of the leading vehicle, perception that the leading vehicle is too far away, failure to perceive the speed of the leading vehicle, perception that the speed of the leading vehicle is too high, slow response to changes in the position and speed of the leading vehicle, inability to perceive its own position information, perception of its own position information with too large an error, failure to perceive lane line information, too large a lane line positioning error, and misperception of obstacles.

[0022] Preferably, the perception problems in scenario two include: not perceiving pedestrians or electric bicycles, not perceiving the distance of pedestrians or electric bicycles, perceiving that the distance of pedestrians or electric bicycles is too large, perceiving that the lateral position of pedestrians or electric bicycles deviates too much, not perceiving the speed of pedestrians or electric bicycles, perceiving that the speed error of pedestrians or electric bicycles is too large, perceiving the changes in the lateral position and speed of pedestrians or electric bicycles too slowly, being unable to perceive its own position information, perceiving that the error in its own position information is too large, not perceiving lane line information and having too large a lane line positioning error.

[0023] Preferably, the perception problems in scenario three include: normal pedestrian detection when the traffic light is not detected, normal pedestrian detection when the traffic light color is not detected, normal pedestrian detection when the traffic light color is detected incorrectly, normal pedestrian detection when the stop line is not detected, normal pedestrian detection when the stop line distance is perceived to be too large, traffic light or stop line detection failure and no pedestrian is perceived, traffic light or stop line detection failure and no pedestrian distance is perceived, traffic light or stop line detection failure and the perceived pedestrian distance is too large, traffic light or stop line detection failure and the perceived pedestrian lateral position deviation is too large, traffic light or stop line detection failure and no pedestrian speed is perceived, traffic light or stop line detection failure and the perceived pedestrian speed error is too large, traffic light or stop line detection failure and the perception of changes in the pedestrian lateral position and speed is too slow, failure to perceive the lane line crossing the solid line into other lanes at the intersection, too large lane line positioning error at the intersection, crossing the solid line into other lanes at the intersection, inability to perceive one's own position information and crossing the solid line into other lanes at the intersection, and perceiving one's own position information with too large an error and crossing the solid line into other lanes at the intersection.

[0024] Preferably, the perception problems in scenario four include: not perceiving an obstacle, not perceiving the obstacle's location information, perceiving that the obstacle's distance is too large, being unable to perceive its own location information, perceiving that the location information error is too large, not perceiving lane line information, and perceiving that the lane line positioning error is too large.

[0025] Preferably, the accident occurrence level includes:

[0026] Accident occurrence level 5: Accident occurrence ≥ 80 / 100 times;

[0027] Accident occurrence level 4: 80 / 100 times > accident occurrence ≥ 45 / 100 times;

[0028] Accident occurrence level 3: 45 / 100 times > accident occurrence ≥ 17.5 / 100 times;

[0029] Accident occurrence level 2: 17.5 / 100 times > accident occurrence ≥ 0.1 / 100 times;

[0030] Accident occurrence level 1: Accident occurrence ≤ 0.1 / 100 times;

[0031] The higher the accident occurrence level, the higher the possibility that the perception problem will lead to an accident.

[0032] Preferably, the accident severity levels include:

[0033] Accident severity level 5: death of personnel and vehicle damage or death of personnel;

[0034] Accident severity level 4: personal injury and vehicle damage or personal injury;

[0035] Accident severity level 3: Vehicle damaged, no one injured;

[0036] Accident severity level 2: Damage to other facilities, violation of traffic regulations, shock to people or loss of vehicle;

[0037] Accident severity level 1: disrupting normal driving;

[0038] The higher the accident severity level, the higher the severity of the accident.

[0039] Preferably, a perception system security evaluation table is established, specifically including:

[0040] A total risk number is obtained by summing up the risk numbers of all the scenarios;

[0041] Establishing the perception system safety evaluation table based on the total risk number and the accident severity level;

[0042] The perception system safety evaluation table includes:

[0043] Low safety level: total risk number ≥ 43 or accident severity level is 5;

[0044] Safety level: 10≤total risk number<43;

[0045] High security level: total risk number <10.

[0046] An on-site evaluation system for the safety of a perception system of an autonomous driving vehicle comprises: a scene acquisition module, a problem perception module, a test parameter determination module, a scene risk number determination module, an evaluation table determination module and a result output module;

[0047] The scene acquisition module is used to acquire an autonomous driving scene for testing;

[0048] The problem perception module is used to determine the perception problem corresponding to the accident based on the autonomous driving scenario;

[0049] The test parameter determination module is used to determine the testable parameters based on the perception problem;

[0050] The scenario risk number determination module is used to determine the accident occurrence level and the accident severity level based on the perceived problem; determine the scenario risk number of all perceived problems based on the accident occurrence level and the accident severity level;

[0051] The evaluation table determination module is used to establish a perception system safety evaluation table based on the scenario risk number and the accident severity level;

[0052] The result output module is used to perform safety evaluation on the perception system to be tested based on the testable parameters and the perception system safety evaluation table.

[0053] It can be seen from the above technical solutions that, compared with the prior art, the present invention discloses a field evaluation method and system for the safety of the perception system of an autonomous driving vehicle. Based on the systematic analysis of perception problems in driving scenarios, quantitative evaluation indicators for perception risks in different scenarios are proposed. The safety of the perception system is comprehensively evaluated based on the results of multi-scenario risk analysis, which has the following beneficial effects:

[0054] 1. Through on-site testing in different scenarios, the limitations of using whole vehicle testing and single sensor testing for autonomous vehicles have been overcome, and potential problems in the perception system have been clearly identified, so that the perception system under test can be optimized and improved in a targeted manner, providing convenience for perception system R&D companies.

[0055] 2. The safety assessment method proposed in the present invention directly assesses the perception system of the autonomous driving vehicle. The test method is convenient and overcomes the shortcomings of high cost of autonomous driving vehicle testing, poor interpretability of test results, and the inability of a single sensor test to reflect overall safety.

[0056] 3. This invention proposes a method for quantitatively analyzing the perception risk of autonomous driving, uses the risk number safety level, and provides the identification method of safety evaluation indicators in different scenarios and the perception safety evaluation criteria, guiding perception system manufacturers to effectively evaluate the safety of their products.

[0057] 4. The method proposed in the present invention is based on field testing, which overcomes the shortcomings of questionable test results and lack of real feedback in a virtual environment, and ensures the validity of the test results.

[0058] 5. The method proposed in the present invention is a perception system safety assessment method based on quantitative risk indicators. The results obtained can provide necessary input for the upgrade and optimization of the perception system of intelligent driving vehicles and the research and development of decision-making systems. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.

[0060] Figure 1 A flow chart of an on-site evaluation method for the safety of an autonomous driving vehicle perception system provided by the present invention.

[0061] Figure 2 This is a schematic diagram of scene 1 in the autonomous driving scenario provided by the present invention.

[0062] Figure 3 This is a schematic diagram of scene 2 in the autonomous driving scenario provided by the present invention.

[0063] Figure 4 This is a schematic diagram of scene 3 in the autonomous driving scenario provided by the present invention.

[0064] Figure 5 This is a schematic diagram of scene 4 in the autonomous driving scenario provided by the present invention.

[0065] Figure 6 This is a fault tree diagram of the long straight road following scenario provided by Scenario 1 of the present invention.

[0066] Figure 7 A schematic diagram of a fault tree for scenario 2 of a long straight road with pedestrians crossing the road provided by the present invention.

[0067] Figure 8 A fault tree diagram of scenario 3 provided by the present invention, in which an electric bicycle crosses the road at an intersection.

[0068] Fig. 9 A fault tree diagram of scenario 4 provided by the present invention, which is a scenario with a stationary obstacle on a long straight road with multiple lanes.

[0069] Fig.10 A schematic diagram of the structure of an on-site evaluation system for the safety of an autonomous driving vehicle perception system provided by the present invention. DETAILED DESCRIPTION

[0070] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0071] Example 1

[0072] like Figure 1 As shown, an embodiment of the present invention discloses a method for on-site evaluation of the safety of a perception system of an autonomous driving vehicle, comprising:

[0073] Obtain autonomous driving scenarios for testing;

[0074] Determine the perception issues that lead to accidents based on autonomous driving scenarios;

[0075] Determine testable parameters based on perceived issues;

[0076] Determine accident occurrence level and accident severity level based on perceived problems;

[0077] Determine the scenario risk number for all perceived problems based on the accident occurrence level and accident severity level;

[0078] Establish a safety evaluation table for the perception system based on the scenario risk number and accident severity level;

[0079] The safety of the perception system to be tested is evaluated based on the testable parameters and the perception system safety evaluation table.

[0080] Example 2

[0081] The embodiment of the present invention discloses a method for on-site evaluation of the safety of a perception system of an autonomous driving vehicle, comprising:

[0082] Get the autonomous driving scenario for testing:

[0083] Preferably, the risks caused by perception problems are closely related to the driving scenarios, and the same perception problems will present different risks in different scenarios. Therefore, when conducting a perception safety evaluation, it is necessary to first construct a test scenario. By analyzing typical driving scenarios, the risks caused by all perception problems in the scenarios can be systematically evaluated, and effective evaluation from local to overall can be achieved. With reference to relevant standards, the present invention selects four typical test scenarios, covering elements such as roads, traffic participants, traffic signs, and traffic lights, to analyze the risk performance of the perception system in each scenario.

[0084] Preferably, based on the requirements for test scene elements such as roads, traffic signs and markings, and targets in the "Field Test Methods and Requirements for Autonomous Driving Functions of Intelligent Connected Vehicles", the present invention constructs four typical closed driving scenarios to quantitatively evaluate the risks of the perception system.

[0085] Preferably, Figure 2-Figure 5 As shown, the autonomous driving scenarios include:

[0086] Scenario 1: Following a car on a long straight road;

[0087] Scene 2: Pedestrians crossing the road on a long straight road;

[0088] Scene 3: Pedestrians are crossing the road at a traffic light intersection;

[0089] Scenario 4: There is a stationary obstacle on a long straight road with multiple lanes.

[0090] Determine the perception issues that lead to accidents based on autonomous driving scenarios:

[0091] Preferably, by systematically analyzing the perception problems that may lead to accidents and thus generate risks, the present invention identifies all the perception problems in the scenario that may lead to accidents through a fault tree method.

[0092] Preferably, for the above four scenarios, a fault tree analysis method is used to determine the possible accidents and their risk causes in each scenario, and analyze the perception problems in the risk causes.

[0093] Preferably, the perception problems include: scene one perception problem, scene two perception problem, scene three perception problem and scene four perception problem;

[0094] Among them, Figure 6 As shown, there are three potential accidents in scenario 1, namely, hitting the vehicle in front, running off the road, and sudden braking. 11 perception problems were identified through the fault tree;

[0095] The perception problems in scenario one include: failure to perceive the vehicle in front, failure to perceive the position information of the vehicle in front, perception that the vehicle in front is too far away, failure to perceive the speed of the vehicle in front, perception of the speed of the vehicle in front with too large an error, slow response to changes in the position and speed of the vehicle in front, inability to perceive its own position information, perception of its own position information with too large an error, failure to perceive lane line information, lane line positioning error with too large an error, and mistaken perception of obstacles.

[0096] Preferably, Figure 7 As shown, there are two potential accidents in scenario 2, namely, hitting pedestrians and running off the road. 11 perception problems were identified through the fault tree;

[0097] The perception problems in scenario two include: not perceiving pedestrians, not perceiving the distance of pedestrians, perceiving the distance of pedestrians with too large an error, perceiving the lateral position of pedestrians with too large a deviation, not perceiving the speed of pedestrians, perceiving the speed of pedestrians with too large an error, perceiving the changes in the lateral position and speed of pedestrians too slowly, being unable to perceive its own position information, perceiving its own position information with too large an error, not perceiving lane line information, and having too large an error in lane line positioning.

[0098] Preferably, the above-mentioned too large or too slow refers to exceeding a corresponding preset value, and the preset value is set according to relevant standards or actual needs.

[0099] Preferably, Figure 8As shown, there are three potential accidents in scenario three, namely, running a red light without hitting a pedestrian, running a red light and hitting a pedestrian, and vehicles entering other lanes at the intersection. 16 perception problems were identified through the fault tree; among them, M1 running a red light without hitting a pedestrian and M2 running a red light and hitting a pedestrian have the same perception problems X1-X5, and the same events in the fault tree have the same number.

[0100] The perception problems in scenario three include: normal pedestrian detection when the traffic light is not detected, normal pedestrian detection when the traffic light color is not detected, normal pedestrian detection when the traffic light color is detected incorrectly, normal pedestrian detection when the stop line is not detected, normal pedestrian detection when the stop line distance error is too large, traffic light or stop line detection failure and no pedestrian is perceived, traffic light or stop line detection failure and no pedestrian distance is perceived, traffic light or stop line detection failure and the perceived pedestrian distance is too large, traffic light or stop line detection failure and the perceived pedestrian lateral position deviation is too large, traffic light or stop line detection failure and no pedestrian speed is perceived, traffic light or stop line detection failure and the perceived pedestrian speed error is too large, traffic light or stop line detection failure and the perception of changes in the pedestrian lateral position and speed is too slow, lane line is not perceived to cross the solid line into other lanes at the intersection, lane line positioning error is too large to cross the solid line into other lanes at the intersection, unable to perceive own position information to cross the solid line into other lanes at the intersection, and perceive own position information with too large error to cross the solid line into other lanes at the intersection.

[0101] Preferably, the above-mentioned too large or too slow refers to exceeding a corresponding preset value, and the preset value is set according to relevant standards or actual needs.

[0102] Preferably, Fig. 9 As shown, there are two potential accidents in scenario 4, namely, collision with obstacles and driving off the road. Seven perception problems are identified through the fault tree;

[0103] The perception problems in scenario four include: failure to perceive obstacles, failure to perceive obstacle location information, perception that the obstacle is too far away, inability to perceive its own location information, perception that the location information error is too large, failure to perceive lane line information, and perception that the lane line positioning error is too large.

[0104] Determine testable parameters based on perceived issues:

[0105] Preferably, testable parameters are determined based on perception problems, and the testable parameters are used as indicators for perceptual safety evaluation; the testable parameters are used for actual evaluation, and the existing perceptual problems are judged by testing the relevant testable parameters.

[0106] Preferably, the testable parameters include: scenario one testable parameters, scenario two testable parameters, scenario three testable parameters and scenario four testable parameters, which are respectively obtained based on the perception problem of scenario one, the perception problem of scenario two, the perception problem of scenario three and the perception problem of scenario four.

[0107] Preferably, the testable parameter corresponds to at least one perceptual question.

[0108] Preferably, the testable parameters of scenario one include: whether the preceding vehicle can be sensed, the relative distance error between the target vehicle and the preceding vehicle, the relative speed error between the target vehicle and the preceding vehicle, the time it takes for the target vehicle to sense the preceding vehicle information, the absolute position error of the target vehicle, the relative distance error between the target vehicle and the lane line, and whether a non-existent obstacle is sensed.

[0109] Preferably, different perception problems may be determined by the same testable parameter. For example, the two perception problems of not perceiving the speed of the preceding vehicle and perceiving that the preceding vehicle has too high a speed may be measured according to the testable parameter, the relative speed error between the target vehicle and the preceding vehicle.

[0110] Preferably, the testable parameters of scenario two include: whether pedestrians can be sensed, the relative distance error between the target vehicle and pedestrians, the relative position error between the target vehicle and pedestrians, the speed error of pedestrians, the absolute position error of the target vehicle, and the relative distance error between the target vehicle and the lane line.

[0111] Preferably, the testable parameters of scene three include: whether the traffic light and its color can be perceived, whether the stop line can be perceived, the relative distance error between the target vehicle and the stop line, whether the pedestrian can be perceived, the relative distance error between the target vehicle and the pedestrian, the relative position information error between the target vehicle and the pedestrian, the pedestrian speed error, the absolute position information error of the target vehicle, and the relative distance error between the target vehicle and the lane line.

[0112] Preferably, the testable parameters of scenario four include: whether a stationary obstacle can be sensed, the relative position error between the target vehicle and the obstacle, the absolute position error of the target vehicle, whether the lane line can be sensed, and the relative distance error between the target vehicle and the lane line.

[0113] Determine the accident occurrence level and accident severity level based on the perceived problem:

[0114] Preferably, GB / T 34402-2017, Guidelines for Safety Risk Assessment and Risk Control of Automobile Products, is used as the basis, combined with relevant literature: Cha Zhiyong. Research on reverse construction and driving risk assessment of ICV accident scenarios based on in-depth investigation [D]. Xihua University, 2023. Determine the grading standards for accident occurrence level and accident severity level.

[0115] Preferably, the accident occurrence level includes:

[0116] Accident occurrence level 5: Accident occurrence ≥ 80 / 100 times;

[0117] Accident occurrence level 4: 80 / 100 times > accident occurrence ≥ 45 / 100 times;

[0118] Accident occurrence level 3: 45 / 100 times > accident occurrence ≥ 17.5 / 100 times;

[0119] Accident occurrence level 2: 17.5 / 100 times > accident occurrence ≥ 0.1 / 100 times;

[0120] Accident occurrence level 1: Accident occurrence ≤ 0.1 / 100 times;

[0121] The accident occurrence degree indicates the possibility that a perception problem will lead to an accident; the higher the accident occurrence degree level, the higher the possibility that a perception problem will lead to an accident.

[0122] Preferably, the accident severity levels include:

[0123] Accident severity level 5: death of personnel and vehicle damage or death of personnel;

[0124] Accident severity level 4: personal injury and vehicle damage or personal injury;

[0125] Accident severity level 3: Vehicle damaged, no one injured;

[0126] Accident severity level 2: Damage to other facilities, violation of traffic regulations, shock to people or loss of vehicle;

[0127] Accident severity level 1: disrupting normal driving;

[0128] The severity of an accident is divided into five levels, 1-5, based on the casualties, vehicle damage and traffic violations caused. The higher the accident severity level, the more serious the accident.

[0129] Determine the scenario risk number for all perceived issues based on the incident occurrence level and incident severity level:

[0130] Preferably, the scenario risk number is based on the product of the accident occurrence level and the accident severity level, that is, the risk number of a single perceived problem, and the scenario risk number is used to measure the risk size of the perceived problem.

[0131] Preferably, taking scenario 1 as an example, "failure to perceive the vehicle in front" leads to a collision with the vehicle in front, the accident occurrence level is 4, the accident severity level is 4, and the corresponding scenario risk number is 16. Similarly, the scenario risk numbers corresponding to perception problems in other scenarios can be calculated. Scenario 1: The scenario risk numbers of perception problems in the long straight lane following scenario are shown in Table 1:

[0132] Table 1. Number of risk scenarios for perception problems in long straight lane following scenarios

[0133]

[0134] Scenario 2: Pedestrians crossing the road on a long straight road. The number of perceived problem scenarios is shown in Table 2:

[0135] Table 2. Number of risk scenarios for perception problems of electric vehicles crossing the road on a long straight road

[0136]

[0137]

[0138] Scenario 3: Electric vehicles crossing the road at an intersection. The number of perceived problem scenarios is shown in Table 3:

[0139] Table 3 Number of risk scenarios for the scenario of electric vehicles crossing the road at the intersection

[0140]

[0141]

[0142] Scenario 4: There is a stationary obstacle on a long straight road with multiple lanes. The number of scenario risks for perception problems is shown in Table 4:

[0143] Table 4 Risk number of scene perception problems with stationary obstacles on a long straight road with multiple lanes

[0144]

[0145] A safety evaluation table for the perception system is established based on the scenario risk number and accident severity level:

[0146] Preferably, the scenario risk numbers of individual perception problems in the scenario are added together to obtain the risk number of the perception system in the scenario, and the risk numbers of the four scenarios are further added together to obtain the total risk number of the perception system.

[0147] Preferably, a perception system security evaluation table is established, specifically including:

[0148] The total risk number is obtained by summing up the risk numbers of all scenarios;

[0149] Establish a safety evaluation table for the perception system based on the total risk number and accident severity level;

[0150] The perception system safety evaluation form includes:

[0151] Low safety level: total risk number ≥ 43 or accident severity level is 5;

[0152] Safety level: 10≤total risk number<43;

[0153] High security level: total risk number <10.

[0154] Preferably, the perception system safety evaluation table is determined by the expert group based on the scenario risk numbers in different scenarios in Tables 1-4, combined with the accident severity level. The perception system safety is divided into three levels: high, medium, and low. It is worth noting that if a perception problem with a severity of 5 occurs during the test, it means that the problem can cause death. No matter how large the risk number is, the safety of the perception system will be rated as low.

[0155] The safety of the perception system to be tested is evaluated based on the testable parameters and the perception system safety evaluation table.

[0156] Preferably, the safety evaluation specifically includes:

[0157] Based on the perception system to be tested, the system is tested in the autonomous driving scenario according to the testable parameters to obtain the test results;

[0158] Determine the perception issues corresponding to different autonomous driving scenarios based on the test results;

[0159] Based on the perceived problems, the scenario risk number is obtained;

[0160] The total risk number of the system to be tested is obtained by summing up the risk numbers of all scenarios;

[0161] Based on the total system risk number to be tested, the perception system security evaluation table is queried to obtain the security evaluation result.

[0162] Example 3

[0163] Practical application of the method of the present invention:

[0164] The autonomous driving perception system to be tested produced by a certain manufacturer includes cameras, lidar, millimeter-wave radar, ultrasonic radar, IMU, RTK and fusion algorithms.

[0165] The autonomous driving perception system to be tested was installed in an ordinary commercial vehicle and tested in the above four autonomous driving scenarios. In the testable parameter test of the system, the relative speed of the vehicle in front was not sensed in scenario 1, the speed error of pedestrians or electric bicycles was sensed to be 5km / h in scenario 2, the stop line was not sensed in scenario 3, and the speed of pedestrians was sensed to be 0.2m / s, and no perception failure occurred in scenario 4.

[0166] The analysis shows that the perception problems of the autonomous driving perception system under test are: failure to perceive the relative speed of the vehicle ahead, large errors in the perceived speed of pedestrians or electric vehicles, and traffic light or stop line detection failures, and large errors in the perceived speed of pedestrians. Query Tables 1 to 4, and the test results are shown in Table 5:

[0167] Table 5 Test results

[0168]

[0169]

[0170] It can be seen from Table 5 that the total risk number of the autonomous driving perception system to be tested is 14. According to the perception system safety evaluation table, the safety of the perception system is medium.

[0171] Through the analysis of the measurement results of the testable parameters, the perception system still needs to be improved or optimized in terms of speed perception. In actual testing, the higher the risk score of a certain perception problem detected, the greater its impact on the safety of the perception system under test. Further testing can be carried out on this problem to determine the specific cause, improve the corresponding software and hardware performance, and thus improve the safety level of the perception system. In this case, for the speed and error problems of the front vehicle, pedestrians, and electric bicycles in scenes 1 and 2, the performance of the speed radar should be improved; while the detection failure of traffic lights or stop lines in scene 3 should focus on testing the performance of the camera, and the perception of excessive pedestrian speed errors further illustrates the necessity of testing the performance of the radar; at the same time, for multi-modal intelligent perception sensors, the accuracy of their fusion algorithms should be improved to provide more accurate inputs for the decision-making system, and a more prudent decision-making plan should be adopted for the speed perception results when developing the decision-making system.

[0172] Example 4

[0173] like Fig.10 As shown, an embodiment of the present invention discloses an on-site evaluation system for the safety of a perception system of an autonomous driving vehicle, including: a scene acquisition module, a problem perception module, a test parameter determination module, a scene risk number determination module, an evaluation table determination module, and a result output module;

[0174] A scene acquisition module, used to acquire the autonomous driving scene for testing;

[0175] The problem perception module is used to determine the corresponding perception problems that lead to accidents based on the autonomous driving scenario;

[0176] A test parameter determination module, for determining testable parameters based on the perception problem;

[0177] A scenario risk number determination module is used to determine the accident occurrence level and the accident severity level based on the perceived problem; and to determine the scenario risk number of all perceived problems based on the accident occurrence level and the accident severity level;

[0178] An evaluation table determination module is used to establish a perception system safety evaluation table based on the scenario risk number and accident severity level;

[0179] The result output module is used to conduct safety evaluation on the perception system to be tested based on the testable parameters and the perception system safety evaluation table.

[0180] Preferably, the method implemented by the system module of the present invention corresponds one-to-one to the content of the corresponding method described above, and will not be described in detail here.

[0181] Example 5

[0182] Based on the same inventive concept, the present invention also provides a computer device, including a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other via the communication bus;

[0183] Memory, used to store computer programs;

[0184] The processor, when used to execute the program stored in the memory, can implement an on-site evaluation method for the safety of an autonomous driving vehicle perception system as in Example 1 or 2.

[0185] The electronic device may include: a processor, a communications interface, a memory, and a communications bus, wherein the processor, the communications interface, and the memory communicate with each other via the communications bus. The processor may call logic instructions in the memory to execute a field evaluation method for the safety of an autonomous driving vehicle perception system in Embodiment 1 or 2.

[0186] In addition, the logic instructions in the above-mentioned memory can be implemented in the form of software functional units and can be stored in a computer-readable storage medium when sold or used as an independent product. Based on such an understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art or the part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods of each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program codes.

[0187] Through the above technical solutions, it can be seen that the present invention discloses a field evaluation method and system for the safety of the perception system of an autonomous driving vehicle. Based on the systematic analysis of the perception problems in the driving scene, quantitative evaluation indicators of perception risks in different scenes are proposed. The safety of the perception system is comprehensively evaluated based on the multi-scene risk analysis results, which has the following beneficial effects:

[0188] 1. Through on-site testing in different scenarios, the limitations of using whole vehicle testing and single sensor testing for autonomous vehicles have been overcome, and potential problems in the perception system have been clearly identified, so that the perception system under test can be optimized and improved in a targeted manner, providing convenience for perception system R&D companies.

[0189] 2. The safety assessment method proposed in the present invention directly assesses the perception system of the autonomous driving vehicle. The test method is convenient and overcomes the shortcomings of high cost of autonomous driving vehicle testing, poor interpretability of test results, and the inability of a single sensor test to reflect overall safety.

[0190] 3. This invention proposes a method for quantitatively analyzing the perception risk of autonomous driving, uses the risk number safety level, and provides the identification method of safety evaluation indicators in different scenarios and the perception safety evaluation criteria, guiding perception system manufacturers to effectively evaluate the safety of their products.

[0191] 4. The method proposed in the present invention is based on field testing, which overcomes the shortcomings of questionable test results and lack of real feedback in a virtual environment, and ensures the validity of the test results.

[0192] 5. The method proposed in the present invention is a perception system safety assessment method based on quantitative risk indicators. The results obtained can provide necessary input for the upgrade and optimization of the perception system of intelligent driving vehicles and the research and development of decision-making systems.

[0193] In this specification, each embodiment is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part.

[0194] The above description of the disclosed embodiments enables one skilled in the art to implement or use the present invention. Various modifications to these embodiments will be apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for on-site evaluation of the safety of a perception system of an autonomous driving vehicle, characterized in that: include: Obtain autonomous driving scenarios for testing; Determining a perception problem corresponding to the accident based on the autonomous driving scenario; determining a testable parameter based on the perception problem; determining an accident occurrence level and an accident severity level based on the perception problem; Determining a scenario risk number for all of the perceived problems based on the accident occurrence level and the accident severity level; Establishing a perception system safety evaluation table based on the scenario risk number and the accident severity level; A safety evaluation is performed on the perception system to be tested based on the testable parameters and the perception system safety evaluation table.

2. The on-site evaluation method for the safety of the perception system of an autonomous driving vehicle according to claim 1 is characterized in that: The autonomous driving scenarios include: Scenario 1: Following a car on a long straight road; Scene 2: Pedestrians crossing the road on a long straight road; Scene 3: Pedestrians are crossing the road at a traffic light intersection; Scenario 4: There is a stationary obstacle on a long straight road with multiple lanes.

3. The on-site evaluation method for the safety of the perception system of an autonomous driving vehicle according to claim 2 is characterized in that: The perception problems include: perception problem of scene 1, perception problem of scene 2, perception problem of scene 3 and perception problem of scene 4; Among them, the perception problems in scenario one include: failure to perceive the leading vehicle, failure to perceive the position information of the leading vehicle, perception that the leading vehicle is too far away, failure to perceive the speed of the leading vehicle, perception that the speed of the leading vehicle is too high, slow response to changes in the position and speed of the leading vehicle, inability to perceive its own position information, perception of its own position information with too large an error, failure to perceive lane line information, too large a lane line positioning error, and misperception of obstacles.

4. The on-site evaluation method for the safety of the perception system of an autonomous driving vehicle according to claim 3 is characterized in that: The perception problems in scenario two include: not perceiving pedestrians or electric bicycles, not perceiving the distance of pedestrians or electric bicycles, perceiving the distance of pedestrians or electric bicycles to be too large, perceiving the lateral position of pedestrians or electric bicycles to be too deviated, not perceiving the speed of pedestrians or electric bicycles, perceiving the speed of pedestrians or electric bicycles to be too large, perceiving the changes in the lateral position and speed of pedestrians or electric bicycles to be too slow, being unable to perceive its own position information, perceiving its own position information to be too large an error, not perceiving lane line information and having a lane line positioning error to be too large.

5. The on-site evaluation method for the safety of the perception system of an autonomous driving vehicle according to claim 3 is characterized in that: The perception problems in scenario three include: normal pedestrian detection without detecting the traffic light, normal pedestrian detection without detecting the traffic light color, normal pedestrian detection with incorrect traffic light color detection, normal pedestrian detection without detecting the stop line, normal pedestrian detection with perceived stop line distance being too large, traffic light or stop line detection failure and no pedestrian perception, traffic light or stop line detection failure and no pedestrian perception, traffic light or stop line detection failure and perceived pedestrian distance being too large, traffic light or stop line detection failure and perceived pedestrian lateral position deviation being too large, traffic light or stop line detection failure and no pedestrian speed being perceived, traffic light or stop line detection failure and perceived pedestrian speed error being too large, traffic light or stop line detection failure and perception of changes in pedestrian lateral position and speed being too slow, failure to perceive lane line crossing the solid line into other lanes at an intersection, too large lane line positioning error when crossing the solid line into other lanes at an intersection, inability to perceive own position information when crossing the solid line into other lanes at an intersection, and too large error in perceiving own position information when crossing the solid line into other lanes at an intersection.

6. The on-site evaluation method for the safety of the perception system of an autonomous driving vehicle according to claim 3 is characterized in that: The perception problems in scenario four include: not perceiving an obstacle, not perceiving the obstacle's location information, perceiving that the obstacle is too far away, being unable to perceive its own location information, perceiving that the location information error is too large, not perceiving lane line information, and perceiving that the lane line positioning error is too large.

7. The on-site evaluation method for the safety of the perception system of an autonomous driving vehicle according to claim 1, characterized in that: The accident occurrence levels include: Accident occurrence level 5: Accident occurrence ≥ 80 / 100 times; Accident occurrence level 4: 80 / 100 times > accident occurrence ≥ 45 / 100 times; Accident occurrence level 3: 45 / 100 times > accident occurrence ≥ 17.5 / 100 times; Accident occurrence level 2: 17.5 / 100 times > accident occurrence ≥ 0.1 / 100 times; Accident occurrence level 1: Accident occurrence ≤ 0.1 / 100 times; The higher the accident occurrence level, the higher the possibility that the perception problem will lead to an accident.

8. The on-site evaluation method for the safety of the perception system of an autonomous driving vehicle according to claim 1, characterized in that: The accident severity levels include: Accident severity level 5: death of personnel and vehicle damage or death of personnel; Accident severity level 4: personal injury and vehicle damage or personal injury; Accident severity level 3: Vehicle damaged, no one injured; Accident severity level 2: Damage to other facilities, violation of traffic regulations, shock to people or loss of vehicle; Accident severity level 1: disrupting normal driving; The higher the accident severity level, the higher the severity of the accident.

9. The on-site evaluation method for the safety of the perception system of an autonomous driving vehicle according to claim 1, characterized in that: Establish a perception system safety evaluation table, including: A total risk number is obtained by summing up the risk numbers of all the scenarios; Establishing the perception system safety evaluation table based on the total risk number and the accident severity level; The perception system safety evaluation table includes: Low safety level: total risk number ≥ 43 or accident severity level is 5; Safety level: 10≤total risk number<43; High security level: total risk number <10.

10. An on-site safety evaluation system for a perception system of an autonomous driving vehicle, applied to an on-site safety evaluation method for a perception system of an autonomous driving vehicle as claimed in any one of claims 1 to 9, characterized in that: include: Scenario acquisition module, problem perception module, test parameter determination module, scenario risk number determination module, evaluation table determination module and result output module; The scene acquisition module is used to acquire an autonomous driving scene for testing; The problem perception module is used to determine the perception problem corresponding to the accident based on the autonomous driving scenario; The test parameter determination module is used to determine the testable parameters based on the perception problem; The scenario risk number determination module is used to determine the accident occurrence level and the accident severity level based on the perception problem; Determine the scenario risk number for all perceived problems based on the accident occurrence level and accident severity level; The evaluation table determination module is used to establish a perception system safety evaluation table based on the scenario risk number and the accident severity level; The result output module is used to perform safety evaluation on the perception system to be tested based on the testable parameters and the perception system safety evaluation table.

Images