A method and system for field evaluation of safety of an autonomous vehicle perception system

Abstract

The application discloses a kind of automatic driving car perception system safety's field evaluation method and system, wherein method includes: obtaining automatic driving scene for test;Determine the corresponding perception problem that causes accident based on automatic driving scene;Determine testable parameter based on perception problem;Determine accident occurrence degree grade and accident severity grade based on perception problem;Determine the scene risk number of all perception problems based on accident occurrence degree grade and accident severity grade;Establish perception system safety evaluation table based on scene risk number and accident severity grade;Safety evaluation is carried out to the perception system to be tested based on testable parameter and perception system safety evaluation table.It overcomes the limitation of using automatic driving car virtual simulation test, whole vehicle test and single sensor test, and realizes convenient, comprehensive and accurate evaluation of perception system safety.

Description

Technical Field

[0001] This invention relates to the field of autonomous driving performance testing technology, and more specifically to a method and system for on-site evaluation of the safety of an autonomous vehicle's perception system. Background Technology

[0002] Advances in artificial intelligence have driven the rapid development of autonomous driving technology. Compared to traditional driving methods, autonomous driving technology can reduce traffic accidents caused by human error and improve vehicle safety. Research has found that the research focus for autonomous vehicles has gradually shifted from breakthroughs in autonomous driving technology to functional testing and evaluation techniques. The perception system, as a crucial component of autonomous driving's perception, decision-making, and execution functions, integrates data from sensors such as cameras, LiDAR, and millimeter-wave radar to collect and process environmental and vehicle position information in real time. This is a prerequisite for the safe operation of autonomous vehicles. If the autonomous driving system lacks perception capabilities, the subsequent decision-making system will be unable to react correctly in a timely manner, ultimately leading to an accident. Therefore, conducting functional safety testing of the perception system is particularly important for ensuring overall vehicle safety.

[0003] Currently, the industry's commonly used testing methods mainly focus on the following aspects: First, simulation testing is used to test perception performance in a virtual environment. However, single performance metrics cannot measure the safety of the perception system, and the virtual environment cannot guarantee the validity of the test results. Second, qualitative testing of the overall vehicle safety is conducted, but the "black box effect" in the testing leads to the uninterpretability of problems and makes it difficult to accurately identify the causes of perception risks. Third, testing of single sensors is also used. However, autonomous driving perception systems are complex systems containing multimodal sensors and fusion algorithms. Problems with a single sensor can be compensated for at the system level through multi-source data fusion. Conversely, even if the sensor is normal, problems with the fusion algorithm can lead to system-level perception problems. Therefore, measurements of a single sensor cannot reflect the safety of the perception system.

[0004] Therefore, how to overcome the limitations of using virtual simulation testing, whole vehicle testing, and single sensor testing for autonomous vehicles, and thus achieve a convenient, comprehensive, and accurate evaluation of the safety of perception systems, is a problem that urgently needs to be solved by those skilled in the art. Summary of the Invention

[0005] In view of this, the present invention provides a method and system for on-site evaluation of the safety of the perception system of autonomous vehicles, which overcomes the limitations of using virtual simulation testing, whole vehicle testing and single sensor testing of autonomous vehicles, and realizes a convenient, comprehensive and accurate evaluation of the safety of the perception system.

[0006] To achieve the above objectives, the present invention adopts the following technical solution:

[0007] A method for on-site evaluation of the safety of an autonomous vehicle's perception system, comprising:

[0008] Obtain autonomous driving scenarios for testing;

[0009] Based on the aforementioned autonomous driving scenario, identify the perception issues that led to the accident;

[0010] Based on the aforementioned perception problem, testable parameters are determined;

[0011] Based on the aforementioned perception problem, the accident occurrence level and accident severity level are determined;

[0012] The scenario risk number for all the aforementioned perception problems is determined based on the accident occurrence level and the accident severity level.

[0013] A safety evaluation table for the perception system is established based on the scenario risk number and the accident severity level;

[0014] The security of the sensing system under test is evaluated based on the testable parameters and the sensing system security evaluation table.

[0015] Preferably, the autonomous driving scenario includes:

[0016] Scenario 1: Following another car on a long, straight road;

[0017] Scene 2: Pedestrians are crossing the road on a long, straight road;

[0018] Scene 3: Pedestrians are crossing the road at a traffic light intersection;

[0019] Scenario 4: There is a stationary obstacle on a long, straight road with multiple lanes.

[0020] Preferably, the perception problem includes: a scenario-one perception problem, a scenario-two perception problem, a scenario-three perception problem, and a scenario-four perception problem;

[0021] The perception problems in scenario one include: not sensing the vehicle in front, not sensing the position information of the vehicle in front, sensing that the position of the vehicle in front is too far away, not sensing the speed of the vehicle in front, sensing that the speed of the vehicle in front is too high, reacting slowly to changes in the position and speed of the vehicle in front, being unable to sense one's own position information, sensing that the error of one's own position information is too large, not sensing lane line information, lane line positioning error is too large, and misperceiving obstacles.

[0022] Preferably, the perception problems in scenario two include: not detecting pedestrians or electric bicycles, not detecting the distance to pedestrians or electric bicycles, detecting that the distance to pedestrians or electric bicycles is too large, detecting that the lateral position of pedestrians or electric bicycles is too large, not detecting the speed of pedestrians or electric bicycles, detecting that the speed of pedestrians or electric bicycles has too large an error, perceiving changes in the lateral position and speed of pedestrians or electric bicycles too slowly, being unable to perceive one's own position information, perceiving that the error in one's own position information is too large, not perceiving lane line information, and having too large an error in lane line positioning.

[0023] Preferably, the three perception problems in scenario three include: pedestrian detection is normal even when no traffic light is detected; pedestrian detection is normal even when no traffic light color is detected; pedestrian detection is normal even when the traffic light color is detected incorrectly; pedestrian detection is normal even when no stop line is detected; pedestrian detection is normal even when the perceived stop line distance is too large; pedestrian detection is normal even when the traffic light or stop line detection is faulty and no pedestrian is detected; pedestrian detection is faulty and the perceived pedestrian distance is too large even when the traffic light or stop line detection is faulty; pedestrian lateral position deviation is too large even when the traffic light or stop line detection is faulty; pedestrian speed is not detected even when the traffic light or stop line detection is faulty; pedestrian speed error is too large even when the traffic light or stop line detection is faulty; pedestrian lateral position and speed changes are perceived too slowly even when the traffic light or stop line detection is faulty; lane lines are not perceived when crossing solid lines at intersections to enter other lanes; lane line positioning error is too large when crossing solid lines at intersections to enter other lanes; inability to perceive one's own position information when crossing solid lines at intersections to enter other lanes; and perception of one's own position information error is too large when crossing solid lines at intersections to enter other lanes.

[0024] Preferably, the four perception problems in scenario four include: not perceiving obstacles, not perceiving obstacle location information, perceiving obstacles at too large distances, being unable to perceive one's own location information, perceiving location information with excessive errors, not perceiving lane line information, and perceiving lane line positioning errors with excessive errors.

[0025] Preferably, the accident occurrence level includes:

[0026] Accident Occurrence Level 5: Accident occurrence ≥ 80 / 100 times;

[0027] Accident Occurrence Level 4: 80 / 100 times > Accident Occurrence Level ≥ 45 / 100 times;

[0028] Accident Occurrence Level 3: 45 / 100 times > Accident Occurrence Rate ≥ 17.5 / 100 times;

[0029] Accident Occurrence Level 2: 17.5 / 100 times > Accident Occurrence Level ≥ 0.1 / 100 times;

[0030] Accident Occurrence Level 1: Accident occurrence ≤ 0.1 / 100 times;

[0031] The higher the accident occurrence level, the higher the likelihood that the perception problem will lead to an accident.

[0032] Preferably, the severity level of the accident includes:

[0033] Accident severity level 5: Fatalities and vehicle damage or fatalities;

[0034] Accident severity level 4: Personal injury and vehicle damage or personal injury;

[0035] Accident severity level 3: Vehicle damaged, no injuries;

[0036] Accident Severity Level 2: Damage to other facilities, violation of traffic rules, fright of persons, or damage to the vehicle;

[0037] Accident severity level 1: Disruption of normal driving;

[0038] The higher the severity level of the accident, the more serious the accident.

[0039] Preferably, a security evaluation table for the sensing system is established, which specifically includes:

[0040] The total number of risks is obtained by summing all the risk numbers for the described scenarios.

[0041] A safety evaluation table for the perception system is established based on the total number of risks and the severity level of the accidents.

[0042] The security evaluation table for the sensing system includes:

[0043] Low safety level: Total risk number ≥ 43 or accident severity level is 5;

[0044] In the security level: 10 ≤ Total risk number < 43;

[0045] High security level: Total risk number <10.

[0046] An on-site safety evaluation system for an autonomous vehicle perception system includes: a scene acquisition module, a problem perception module, a test parameter determination module, a scene risk number determination module, an evaluation table determination module, and a result output module.

[0047] The scene acquisition module is used to acquire autonomous driving scenarios for testing;

[0048] The problem perception module is used to determine the perceived problem that leads to the accident based on the autonomous driving scenario;

[0049] The test parameter determination module is used to determine testable parameters based on the perception problem;

[0050] The scenario risk number determination module is used to determine the accident occurrence level and accident severity level based on the perception problem; and to determine the scenario risk number for all perception problems based on the accident occurrence level and accident severity level.

[0051] The evaluation table determination module is used to establish a safety evaluation table for the perception system based on the number of scenario risks and the severity level of accidents.

[0052] The result output module is used to evaluate the security of the sensing system under test based on testable parameters and a sensing system security evaluation table.

[0053] As can be seen from the above technical solution, compared with the prior art, the present invention discloses a method and system for on-site evaluation of the safety of the perception system of autonomous vehicles. Based on a systematic analysis of perception problems in driving scenarios, it proposes quantitative evaluation indicators for perception risks in different scenarios, and comprehensively evaluates the safety of the perception system based on the results of multi-scenario risk analysis, which has the following beneficial effects:

[0054] 1. By conducting field tests in different scenarios, the limitations of using whole-vehicle testing and single-sensor testing for autonomous vehicles have been overcome. Potential problems in the perception system have been clearly identified, enabling targeted optimization and improvement of the perception system under test, thus providing convenience for perception system R&D companies.

[0055] 2. The safety evaluation method proposed in this invention directly evaluates the perception system of autonomous vehicles. The testing method is convenient and overcomes the shortcomings of high testing costs for autonomous vehicles, poor interpretability of test results, and the inability of single sensor testing to reflect overall safety.

[0056] 3. This invention proposes a method for quantitatively analyzing the perception risks of autonomous driving, using risk numbers and safety levels, and provides identification methods for safety evaluation indicators in different scenarios and perception safety evaluation criteria. This guides perception system manufacturers in effectively evaluating the safety of their products.

[0057] 4. The method proposed in this invention is based on on-site testing, which overcomes the shortcomings of questionable testing results and lack of real feedback in virtual environments, thus ensuring the validity of the test results.

[0058] 5. The method proposed in this invention is a safety evaluation method for perception systems based on quantitative risk indicators. The results obtained can provide necessary input for the upgrading and optimization of intelligent driving vehicle perception systems and the research and development of decision-making systems. Attached Figure Description

[0059] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0060] Figure 1 A flowchart of an on-site safety evaluation method for an autonomous vehicle perception system provided by the present invention.

[0061] Figure 2 This is a schematic diagram of scenario 1 in the autonomous driving scenario provided by the present invention.

[0062] Figure 3 This is a schematic diagram of scenario 2 in the autonomous driving scenario provided by the present invention.

[0063] Figure 4 This is a schematic diagram of scenario 3 in the autonomous driving scenario provided by the present invention.

[0064] Figure 5 This is a schematic diagram of scenario 4 in the autonomous driving scenario provided by the present invention.

[0065] Figure 6 This is a schematic diagram of the fault tree for a long straight road following vehicle scenario (Scenario 1) provided by the present invention.

[0066] Figure 7 This is a fault tree diagram illustrating a scenario 2 (a long, straight road with pedestrians crossing the road) provided by the present invention.

[0067] Figure 8 A fault tree diagram illustrating scenario 3, where electric bicycles are crossing the road at an intersection, provided by this invention.

[0068] Figure 9 This is a fault tree diagram illustrating a scenario 4 involving a stationary obstacle on a long, straight multi-lane road, provided by the present invention.

[0069] Figure 10 This invention provides a schematic diagram of the structure of an on-site safety evaluation system for an autonomous vehicle perception system. Detailed Implementation

[0070] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0071] Example 1

[0072] like Figure 1 As shown in the figure, this invention discloses a field evaluation method for the safety of an autonomous vehicle perception system, including:

[0073] Obtain autonomous driving scenarios for testing;

[0074] Identify the perception issues that led to the accident based on the autonomous driving scenario;

[0075] Determine testable parameters based on perception problems;

[0076] Determine the occurrence level and severity level of an accident based on perception issues;

[0077] The scenario risk number for all perceived problems is determined based on the accident occurrence level and accident severity level;

[0078] Establish a safety evaluation table for the perception system based on the number of scenario risks and the severity level of accidents;

[0079] The security of the sensing system under test is evaluated based on the testable parameters and the security evaluation table of the sensing system.

[0080] Example 2

[0081] This invention discloses a method for on-site evaluation of the safety of an autonomous vehicle's perception system, comprising:

[0082] Obtain autonomous driving scenarios for testing:

[0083] Preferably, the risks caused by perception problems are closely related to driving scenarios; the same perception problem will exhibit different risks in different scenarios. Therefore, when conducting perception safety evaluation, it is first necessary to construct test scenarios. By analyzing typical driving scenarios, the risks caused by all perception problems in the scenario can be systematically assessed, achieving an effective evaluation from the local to the overall level. This invention, referencing relevant standards, selected four typical test scenarios covering elements such as roads, traffic participants, traffic signs, and traffic lights to analyze the risk performance of the perception system in each scenario.

[0084] Preferably, based on the requirements for test scenario elements such as roads, traffic signs and markings, and targets in the "Test Methods and Requirements for Automated Driving Functions of Intelligent Connected Vehicles", this invention constructs four typical closed driving scenarios to quantitatively assess the risks of the perception system.

[0085] Preferred, such as Figures 2-5 As shown, autonomous driving scenarios include:

[0086] Scenario 1: Following another car on a long, straight road;

[0087] Scene 2: Pedestrians are crossing the road on a long, straight road;

[0088] Scene 3: Pedestrians are crossing the road at a traffic light intersection;

[0089] Scenario 4: There is a stationary obstacle on a long, straight road with multiple lanes.

[0090] Identify the perception issues that led to the accident based on autonomous driving scenarios:

[0091] Preferably, through systematic analysis of perception problems that can lead to accidents and thus generate risks, this invention identifies all perception problems in a scenario that may lead to accidents using a fault tree method.

[0092] Preferably, for the above four scenarios, fault tree analysis is used to determine the possible accidents and their risk causes in each scenario, and to analyze the perception problems in the risk causes.

[0093] Preferably, the perception problems include: scenario 1 perception problem, scenario 2 perception problem, scenario 3 perception problem, and scenario 4 perception problem;

[0094] Among them, such as Figure 6 As shown, there are three potential accidents in Scenario 1: hitting the car in front, driving off the road, and sudden braking. Eleven perception problems were identified through the fault tree.

[0095] The perception problems in scenario one include: not sensing the vehicle in front, not sensing the position information of the vehicle in front, sensing that the vehicle in front is too far away, not sensing the speed of the vehicle in front, sensing that the speed of the vehicle in front has too large an error, reacting slowly to changes in the position and speed of the vehicle in front, not being able to sense one's own position information, sensing that one's own position information has too large an error, not sensing lane line information, lane line positioning error has too large an error, and misperceiving obstacles.

[0096] Preferred, such as Figure 7 As shown, there are two potential accidents in Scenario 2: hitting pedestrians and driving off the road. Eleven perception problems were identified through the fault tree.

[0097] The perception problems in scenario two include: not detecting pedestrians, not detecting pedestrian distance, excessive error in detecting pedestrian distance, excessive deviation in the perceived lateral position of pedestrians, not detecting pedestrian speed, excessive error in perceiving pedestrian speed, slow perception of changes in pedestrian lateral position and speed, inability to perceive one's own position information, excessive error in perceiving one's own position information, inability to perceive lane line information, and excessive error in lane line positioning.

[0098] Preferably, the terms "too large" or "too slow" refer to exceeding the corresponding preset value, which is set according to relevant standards or actual needs.

[0099] Preferred, such as Figure 8As shown, there are three potential accidents in scenario three: running a red light without hitting a pedestrian, running a red light and hitting a pedestrian, and vehicles entering other lanes at an intersection. Sixteen perception problems were identified through the fault tree. Among them, M1 running a red light without hitting a pedestrian and M2 running a red light and hitting a pedestrian have the same perception problems X1-X5, and the same events are numbered the same in the fault tree.

[0100] Scenario 3 perception problems include: pedestrian detection is normal even if traffic light is not detected; pedestrian detection is normal even if traffic light color is not detected; pedestrian detection is normal even if traffic light color is detected incorrectly; pedestrian detection is normal even if stop line is not detected; pedestrian detection is normal even if stop line distance is perceived too much; pedestrian detection is normal even if traffic light or stop line detection is faulty; pedestrian detection is faulty even if pedestrian distance is not perceived; pedestrian detection is faulty even if pedestrian distance is perceived too much even if traffic light or stop line detection is faulty; pedestrian lateral position deviation is perceived too much even if traffic light or stop line detection is faulty; pedestrian speed is not perceived even if traffic light or stop line detection is faulty; pedestrian speed perception error is too large even if traffic light or stop line detection is faulty; pedestrian lateral position and speed changes are perceived too slowly even if traffic light or stop line detection is faulty; lane lines are not perceived when crossing solid lines at intersections to enter other lanes; lane line positioning error is too large when crossing solid lines at intersections to enter other lanes; inability to perceive one's own position information when crossing solid lines at intersections to enter other lanes; and perception error of one's own position information is too large when crossing solid lines at intersections to enter other lanes.

[0101] Preferably, the terms "too large" or "too slow" refer to exceeding the corresponding preset value, which is set according to relevant standards or actual needs.

[0102] Preferred, such as Figure 9 As shown, there are two potential accidents in scenario four: collision with an obstacle and driving off the road. Seven perception problems were identified through the fault tree.

[0103] The perception problems in scenario four include: not detecting obstacles, not detecting obstacle location information, detecting obstacles at too great a distance, being unable to perceive one's own location information, perceiving location information with excessive error, not perceiving lane line information, and perceiving lane line positioning error with excessive error.

[0104] Determining testable parameters based on perception problems:

[0105] Preferably, testable parameters are determined based on the perception problem, and the testable parameters serve as indicators for perception security evaluation; the testable parameters are used for actual evaluation, and the existing perception problems are identified by testing the relevant testable parameters.

[0106] Preferably, the testable parameters include: testable parameters for scenario one, testable parameters for scenario two, testable parameters for scenario three, and testable parameters for scenario four, which are obtained based on the perception problems of scenario one, scenario two, scenario three, and scenario four, respectively.

[0107] Preferably, the testable parameters correspond to at least one perception problem.

[0108] Preferably, the testable parameters for Scenario 1 include: whether the target vehicle can sense the vehicle in front, the relative distance error between the target vehicle and the vehicle in front, the relative speed error between the target vehicle and the vehicle in front, the time it takes for the target vehicle to sense the information of the vehicle in front, the absolute position error of the target vehicle, the relative distance error between the target vehicle and the lane line, and whether the target vehicle senses non-existent obstacles.

[0109] Preferably, different perception problems may be determined by the same measurable parameter. For example, the two perception problems of not sensing the speed of the vehicle in front and sensing that the speed of the vehicle in front is too high can be measured based on the measurable parameter of the relative speed error between the target vehicle and the vehicle in front.

[0110] Preferably, the testable parameters for Scenario 2 include: whether pedestrians can be detected, the relative distance error between the target vehicle and the pedestrian, the relative position error between the target vehicle and the pedestrian, the pedestrian's speed error, the absolute position error of the target vehicle, and the relative distance error between the target vehicle and the lane line.

[0111] Preferably, the testable parameters for scenario three include: whether traffic lights and their colors can be perceived, whether stop lines can be perceived, the relative distance error between the target vehicle and the stop line, whether pedestrians can be perceived, the relative distance error between the target vehicle and pedestrians, the relative position information error between the target vehicle and pedestrians, pedestrian speed error, the absolute position information error of the target vehicle, and the relative distance error between the target vehicle and the lane line.

[0112] Preferably, the testable parameters for scenario four include: whether stationary obstacles can be detected, whether the relative position error between the target vehicle and the obstacle is too high, whether the absolute position error of the target vehicle is too high, whether lane lines can be detected, and whether the relative distance error between the target vehicle and the lane lines is too high.

[0113] Determining the occurrence level and severity level of an accident based on perception issues:

[0114] Preferably, based on GB / T 34402-2017, Guidelines for Safety Risk Assessment and Risk Control of Automobile Products, and in conjunction with relevant literature: Zha Zhiyong. Research on Reverse Construction of ICV Accident Scenarios and Driving Risk Evaluation Based on In-Depth Investigation [D]. Xihua University, 2023, the grading standards for accident occurrence level and accident severity level are determined.

[0115] Preferably, the accident occurrence level includes:

[0116] Accident Occurrence Level 5: Accident occurrence ≥ 80 / 100 times;

[0117] Accident Occurrence Level 4: 80 / 100 times > Accident Occurrence Level ≥ 45 / 100 times;

[0118] Accident Occurrence Level 3: 45 / 100 times > Accident Occurrence Rate ≥ 17.5 / 100 times;

[0119] Accident Occurrence Level 2: 17.5 / 100 times > Accident Occurrence Level ≥ 0.1 / 100 times;

[0120] Accident Occurrence Level 1: Accident occurrence ≤ 0.1 / 100 times;

[0121] Accident occurrence rate indicates the likelihood that a perceived problem will lead to an accident; the higher the accident occurrence rate, the higher the likelihood that a perceived problem will lead to an accident.

[0122] Preferably, the severity levels of the accident include:

[0123] Accident severity level 5: Fatalities and vehicle damage or fatalities;

[0124] Accident severity level 4: Personal injury and vehicle damage or personal injury;

[0125] Accident severity level 3: Vehicle damaged, no injuries;

[0126] Accident Severity Level 2: Damage to other facilities, violation of traffic rules, fright of persons, or damage to the vehicle;

[0127] Accident severity level 1: Disruption of normal driving;

[0128] Accident severity is classified into five levels, from 1 to 5, based on the number of casualties, vehicle damage, and traffic violations caused. The higher the severity level, the more serious the accident.

[0129] Determine the scenario risk number for all perceived problems based on the accident occurrence level and accident severity level:

[0130] Preferably, the product of the accident occurrence level and the accident severity level is used as the scenario risk number, that is, the risk number of a single perception problem. The scenario risk number is used to measure the magnitude of the risk of the perception problem.

[0131] Preferably, taking Scenario 1 as an example, a collision occurs due to "failure to perceive the vehicle in front," with an accident occurrence level of 4 and an accident severity level of 4. The corresponding scenario risk number is 16. Similarly, the scenario risk number corresponding to perception problems in other scenarios can be calculated. The scenario risk number for Scenario 1: Following another vehicle on a long straight lane is shown in Table 1.

[0132] Table 1. Risk Count of Perception Issues in Long Straight Lane Following Scenarios

[0133]

[0134] Scenario 2: Pedestrians crossing the road on a long straight road. The number of scenario risks is shown in Table 2.

[0135] Table 2. Risk Count of Scenarios Related to Electric Vehicles Crossing Long Straight Roads.

[0136]

[0137] Scenario 3: Electric vehicles crossing the road at an intersection - scenario perception issues. The number of scenario risks is shown in Table 3.

[0138] Table 3: Risk Count of Scenarios Related to Electric Vehicles Crossing the Road at Intersections.

[0139]

[0140]

[0141] Scenario 4: Perception issues in scenarios involving stationary obstacles on long, straight multi-lane roads. The number of scenario risks is shown in Table 4.

[0142] Table 4. Risk Count of Scene Perception Issues on Long Straight Roads with Stationary Obstacles (Multi-lane)

[0143]

[0144]

[0145] Establish a security evaluation table for the perception system based on scenario risk numbers and accident severity levels:

[0146] Preferably, the risk number of the perception system in a given scenario is obtained by summing the risk numbers of individual perception problems in the scenario, and then the risk numbers of the four scenarios are further summed to obtain the total risk number of the perception system.

[0147] Preferably, a security evaluation table for the sensing system is established, which specifically includes:

[0148] The total number of risks is obtained by summing the risk numbers for all scenarios.

[0149] Establish a safety evaluation table for the perception system based on the total number of risks and the severity level of accidents;

[0150] The security evaluation form for the sensing system includes:

[0151] Low safety level: Total risk number ≥ 43 or accident severity level is 5;

[0152] In the security level: 10 ≤ Total risk number < 43;

[0153] High security level: Total risk number <10.

[0154] Preferably, the safety evaluation form for the perception system is determined by an expert group based on the scenario risk numbers in different scenarios listed in Tables 1-4, combined with the severity level of the accident. The safety of the perception system is categorized into three levels: high, medium, and low. It is worth noting that if a perception problem with a severity level of 5 occurs during the test, indicating that the problem could lead to fatalities, the safety of the perception system will be rated as low, regardless of the risk number.

[0155] The security of the sensing system under test is evaluated based on the testable parameters and the security evaluation table of the sensing system.

[0156] Preferably, the safety evaluation specifically includes:

[0157] Based on the perception system under test, tests are conducted in autonomous driving scenarios according to testable parameters to obtain test results;

[0158] Based on the test results, identify the corresponding perception problems in different autonomous driving scenarios;

[0159] The number of scenario risks is obtained based on the perception problems;

[0160] The total number of risks for the system under test is obtained by summing the risk numbers from all scenarios.

[0161] Based on the total number of risks in the system under test, the security evaluation table of the perception system is queried to obtain the security evaluation results.

[0162] Example 3

[0163] Practical application of the method of this invention:

[0164] The autonomous driving perception system under test produced by a certain manufacturer includes cameras, lidar, millimeter-wave radar, ultrasonic radar, IMU, RTK, and fusion algorithms.

[0165] The autonomous driving perception system under test was installed on a regular commercial vehicle and tested in the four autonomous driving scenarios mentioned above. In the testable parameters of the system, the relative speed of the vehicle in front was not detected in scenario 1; the speed of a pedestrian or electric bicycle was detected with an error of 5 km / h in scenario 2; the stop line was not detected in scenario 3, and the pedestrian speed was detected at 0.2 m / s; no perception fault occurred in scenario 4.

[0166] The analysis revealed the following perception problems in the tested autonomous driving perception system: failure to detect the relative speed of the vehicle in front, excessive error in detecting the speed of pedestrians or electric vehicles, and malfunctions in traffic light or stop line detection, particularly excessive error in detecting the speed of pedestrians. Referring to Tables 1 to 4, the test results are shown in Table 5.

[0167] Table 5 Test Results

[0168]

[0169] As shown in Table 5, the total risk number of the autonomous driving perception system under test is 14. According to the perception system safety evaluation table, the safety of the perception system is medium.

[0170] Analysis of the testable parameter measurement results reveals that the perception system still requires improvement or optimization in speed perception. In actual testing, the higher the risk score of a detected perception problem, the greater its impact on the safety of the tested perception system. Further testing can be conducted to determine the specific cause, improve the corresponding hardware and software performance, and thus enhance the safety level of the perception system. In this example, for the speed and error issues related to the vehicle, pedestrian, and electric bicycle in scenarios 1 and 2, the performance of the speed measuring radar should be prioritized for improvement. For scenario 3, the detection failure of traffic lights or stop lines should focus on testing the camera performance, and the excessively large error in perceiving pedestrian speed further illustrates the necessity of testing radar performance. Simultaneously, for multimodal intelligent perception sensors, the accuracy of their fusion algorithm should be improved to provide more precise input to the decision-making system. A more prudent decision-making approach should be adopted based on the speed perception results during the development of the decision-making system.

[0171] Example 4

[0172] like Figure 10 As shown in the figure, this invention discloses an on-site evaluation system for the safety of an autonomous vehicle perception system, including: a scene acquisition module, a problem perception module, a test parameter determination module, a scene risk number determination module, an evaluation table determination module, and a result output module;

[0173] The scene acquisition module is used to acquire autonomous driving scenarios for testing.

[0174] The problem perception module is used to identify the perceived problems that lead to accidents based on autonomous driving scenarios.

[0175] The test parameter determination module is used to determine testable parameters based on the perception problem;

[0176] The scenario risk determination module is used to determine the accident occurrence level and accident severity level based on the perceived problem; and to determine the scenario risk number for all perceived problems based on the accident occurrence level and accident severity level.

[0177] The evaluation form determination module is used to establish a safety evaluation form for the perception system based on the number of scenario risks and the severity level of accidents;

[0178] The results output module is used to evaluate the security of the sensing system under test based on testable parameters and the sensing system security evaluation form.

[0179] Preferably, the methods implemented by the system modules of the present invention correspond one-to-one with the corresponding methods described above, and will not be repeated here.

[0180] Example 5

[0181] Based on the same inventive concept, the present invention also provides a computer device, including a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus;

[0182] Memory, used to store computer programs;

[0183] When the processor executes a program stored in the memory, it is able to implement a field evaluation method for the safety of an autonomous vehicle perception system, as shown in Embodiment 1 or 2.

[0184] The electronic device may include a processor, a communications interface, a memory, and a communication bus, wherein the processor, communications interface, and memory communicate with each other via the communication bus. The processor can invoke logical instructions in the memory to execute a field evaluation method for the safety of an autonomous vehicle perception system, as described in Embodiment 1 or 2.

[0185] Furthermore, when the logical instructions in the aforementioned memory can be implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, essentially, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0186] As can be seen from the above technical solution, the present invention discloses a method and system for on-site evaluation of the safety of an autonomous vehicle perception system. Based on a systematic analysis of perception problems in driving scenarios, it proposes quantitative evaluation indicators for perception risks in different scenarios, and comprehensively evaluates the safety of the perception system based on the results of multi-scenario risk analysis, which has the following beneficial effects:

[0187] 1. By conducting field tests in different scenarios, the limitations of using whole-vehicle testing and single-sensor testing for autonomous vehicles have been overcome. Potential problems in the perception system have been clearly identified, enabling targeted optimization and improvement of the perception system under test, thus providing convenience for perception system R&D companies.

[0188] 2. The safety evaluation method proposed in this invention directly evaluates the perception system of autonomous vehicles. The testing method is convenient and overcomes the shortcomings of high testing costs for autonomous vehicles, poor interpretability of test results, and the inability of single sensor testing to reflect overall safety.

[0189] 3. This invention proposes a method for quantitatively analyzing the perception risks of autonomous driving, using risk numbers and safety levels, and provides identification methods for safety evaluation indicators in different scenarios and perception safety evaluation criteria. This guides perception system manufacturers in effectively evaluating the safety of their products.

[0190] 4. The method proposed in this invention is based on on-site testing, which overcomes the shortcomings of questionable testing results and lack of real feedback in virtual environments, thus ensuring the validity of the test results.

[0191] 5. The method proposed in this invention is a safety evaluation method for perception systems based on quantitative risk indicators. The results obtained can provide necessary input for the upgrading and optimization of intelligent driving vehicle perception systems and the research and development of decision-making systems.

[0192] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the apparatus disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the description is relatively simple; relevant parts can be referred to the method section.

[0193] The above description of the disclosed embodiments enables those skilled in the art to make or use the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for on-site evaluation of the safety of an autonomous vehicle's perception system, characterized in that, include: Obtain autonomous driving scenarios for testing; Based on the aforementioned autonomous driving scenario, identify the perception issues that led to the accident; Based on the aforementioned perception problem, testable parameters are determined; Based on the aforementioned perception problem, an accident occurrence level and an accident severity level are determined, where the accident occurrence level represents the likelihood that the perception problem will lead to an accident. The scenario risk number for all the aforementioned perception problems is determined based on the accident occurrence level and the accident severity level. A safety evaluation table for the perception system is established based on the scenario risk number and the accident severity level; Establish a security evaluation table for the sensing system, specifically including: The risk number of the perception system in a given scenario is obtained by summing the risk numbers of individual perception problems in the scenario. The risk numbers of the four scenarios are then summed to obtain the total risk number of the perception system. A safety evaluation table for the perception system is established based on the total number of risks and the severity level of the accidents. The security evaluation table for the sensing system includes: Low safety level: Total risk number ≥ 43 or accident severity level is 5; In the security level: 10 ≤ Total risk number < 43; High security level: Total risk count < 10; The security of the sensing system under test is evaluated based on the testable parameters and the sensing system security evaluation table. The higher the risk score of a detected perception problem, the greater its impact on the security of the perception system under test. Further testing should be conducted to determine the specific cause and improve the corresponding software and hardware performance. The autonomous driving scenarios include: Scenario 1: Following another car on a long, straight road; Scene 2: Pedestrians are crossing the road on a long, straight road; Scene 3: Pedestrians are crossing the road at a traffic light intersection; Scenario 4: There is a stationary obstacle on a long, straight multi-lane road; Fault tree analysis was used to identify the possible accidents and the perception problems that caused the corresponding accidents in the above scenarios. The perception problems include: perception problems in scenario one, scenario two, scenario three, and scenario four. The testable parameters for Scenario 1 include: whether the vehicle in front can be detected, the relative distance error between the target vehicle and the vehicle in front, the relative speed error between the target vehicle and the vehicle in front, the time it takes for the target vehicle to detect the information of the vehicle in front, the absolute position error of the target vehicle, the relative distance error between the target vehicle and the lane line, and whether non-existent obstacles are detected. The testable parameters for Scenario 2 include: whether pedestrians can be detected, the relative distance error between the target vehicle and the pedestrian, the relative position error between the target vehicle and the pedestrian, the pedestrian's speed error, the absolute position error of the target vehicle, and the relative distance error between the target vehicle and the lane line. The testable parameters for scenario three include: whether traffic lights and their colors can be perceived, whether stop lines can be perceived, the relative distance error between the target vehicle and the stop line, whether pedestrians can be perceived, the relative distance error between the target vehicle and pedestrians, the relative position error between the target vehicle and pedestrians, pedestrian speed error, the absolute position error of the target vehicle, and the relative distance error between the target vehicle and the lane line. The testable parameters for scenario four include: whether stationary obstacles can be detected, the relative position error between the target vehicle and the obstacle, the absolute position error of the target vehicle, whether lane lines can be detected, and the relative distance error between the target vehicle and the lane lines.

2. The on-site evaluation method for the safety of an autonomous vehicle perception system according to claim 1, characterized in that, The perception problems in scenario one include: not sensing the vehicle in front, not sensing the position information of the vehicle in front, sensing that the vehicle in front is too far away, not sensing the speed of the vehicle in front, sensing that the speed of the vehicle in front is too high, reacting slowly to changes in the position and speed of the vehicle in front, being unable to sense one's own position information, sensing that one's own position information has too large an error, not sensing lane line information, having too large an error in lane line positioning, and mistakenly sensing obstacles.

3. The on-site evaluation method for the safety of an autonomous vehicle perception system according to claim 1, characterized in that, The perception problems in Scenario 2 include: not detecting pedestrians or electric bicycles, not detecting the distance to pedestrians or electric bicycles, detecting pedestrians or electric bicycles at excessive distances, detecting pedestrians or electric bicycles with excessive lateral deviations, not detecting the speed of pedestrians or electric bicycles, detecting pedestrians or electric bicycles with excessive speed errors, perceiving changes in the lateral position and speed of pedestrians or electric bicycles too slowly, being unable to perceive one's own position information, perceiving one's own position information with excessive errors, not perceiving lane line information, and lane line positioning errors being excessive.

4. The on-site evaluation method for the safety of an autonomous vehicle perception system according to claim 1, characterized in that, The three perception problems described include: pedestrian detection is normal even when traffic lights are not detected; pedestrian detection is normal even when traffic light color is not detected; pedestrian detection is normal even when traffic light color is incorrect; pedestrian detection is normal even when traffic light color is incorrect; pedestrian detection is normal even when stop lines are not detected; pedestrian detection is normal even when stop line distance is too large; pedestrian detection is normal even when traffic light or stop line detection is faulty and no pedestrian is detected; pedestrian detection is faulty and pedestrian distance is too large even when traffic light or stop line detection is faulty; pedestrian lateral position deviation is too large even when traffic light or stop line detection is faulty; pedestrian speed is not detected even when traffic light or stop line detection is faulty; pedestrian speed detection error is too large even when traffic light or stop line detection is faulty; pedestrian lateral position and speed changes are perceived too slowly even when traffic light or stop line detection is faulty; pedestrian crossing a solid line into another lane at an intersection is not perceived; pedestrian crossing a solid line into another lane at an intersection has too large lane line positioning error; pedestrian crossing a solid line into another lane at an intersection is not perceived; and pedestrian crossing a solid line into another lane at an intersection has too large error in perceiving its own position information.

5. The on-site evaluation method for the safety of an autonomous vehicle perception system according to claim 1, characterized in that, The four perception problems in the scenario include: not detecting obstacles, not detecting obstacle location information, detecting obstacles at too great a distance, being unable to perceive one's own location information, perceiving location information with excessive error, not perceiving lane line information, and perceiving lane line positioning error with excessive error.

6. The on-site evaluation method for the safety of an autonomous vehicle perception system according to claim 1, characterized in that, The accident occurrence level includes: Accident Occurrence Level 5: Accident occurrence ≥ 80 / 100 times; Accident Occurrence Level 4: 80 / 100 times > Accident Occurrence Level ≥ 45 / 100 times; Accident Occurrence Level 3: 45 / 100 times > Accident Occurrence Rate ≥ 17.5 / 100 times; Accident Occurrence Level 2: 17.5 / 100 times > Accident Occurrence Level ≥ 0.1 / 100 times; Accident Occurrence Level 1: Accident occurrence ≤ 0.1 / 100 times; The higher the accident occurrence level, the higher the likelihood that the perception problem will lead to an accident.

7. The on-site evaluation method for the safety of an autonomous vehicle perception system according to claim 1, characterized in that, The severity levels of the accidents include: Accident severity level 5: Fatalities and vehicle damage or fatalities; Accident severity level 4: Personal injury and vehicle damage or personal injury; Accident severity level 3: Vehicle damaged, no injuries; Accident Severity Level 2: Damage to other facilities, violation of traffic rules, fright of persons, or damage to the vehicle; Accident severity level 1: Disruption of normal driving; The higher the severity level of the accident, the more serious the accident.

8. A field testing system for the safety of an autonomous vehicle perception system, applied to the field testing method for the safety of an autonomous vehicle perception system as described in any one of claims 1-7, characterized in that, include: The module includes a scenario acquisition module, a problem perception module, a test parameter determination module, a scenario risk number determination module, an evaluation table determination module, and a result output module. The scene acquisition module is used to acquire autonomous driving scenarios for testing; The problem perception module is used to determine the perceived problem that leads to the accident based on the autonomous driving scenario; The test parameter determination module is used to determine testable parameters based on the perception problem; The scenario risk determination module is used to determine the accident occurrence level and accident severity level based on the perception problem; The scenario risk number for all perceived problems is determined based on the accident occurrence level and accident severity level; The evaluation table determination module is used to establish a safety evaluation table for the perception system based on the number of scenario risks and the severity level of accidents. The result output module is used to evaluate the security of the sensing system under test based on testable parameters and a sensing system security evaluation table.

Images