A multi-dimensional quantitative evaluation method and system for an automatic driving simulation test system

Abstract

The application discloses a kind of multi-dimension quantitative evaluation method and system of automatic driving simulation test system, belong to the field of automatic driving technology.The present application is to solve the technical problem that existing evaluation system cannot effectively integrate multi-source physical parameters, leading to difficulty in quantitative evaluation of the comprehensive performance of automatic driving system, by obtaining the motion state parameters of the measured vehicle to calculate the comprehensive comfort degree;Obtain the relative position parameters of vehicle and environmental elements and calculate the comprehensive risk degree combined with the comprehensive comfort degree;Obtain the driving speed parameter and calculate the comprehensive speed combined with the comprehensive risk degree;Based on the above three indexes, the comprehensive performance evaluation result of automatic driving is generated.The present application can realize the objective quantitative comparison of the performance of different automatic driving systems, and intuitively show the comprehensive performance and strategy preference of the system.

Description

Technical Field

[0001] This invention belongs to the field of autonomous driving technology, specifically relating to a multi-dimensional quantitative evaluation method and system for autonomous driving simulation testing systems. Background Technology

[0002] Since its inception, autonomous driving technology has garnered widespread attention, with driverless cars frequently appearing in science fiction literature and films. In recent years, due to the surge in interest in artificial intelligence, autonomous driving, closely related to it, has received even greater attention and entered the public eye. More and more researchers are dedicating themselves to research related to autonomous vehicles, with many envisioning a future of intelligent transportation where vehicles can replace all human driving operations, creating a zero-accident travel environment. Within this framework, assessing the safety and reliability of autonomous driving systems is crucial, such as rehearsing vehicle responses under extreme conditions in simulation environments or quantifying the performance of algorithms in complex traffic flows. Many autonomous driving manufacturers primarily rely on reproducing real-world scenario logs or building virtual scenario libraries for testing. However, existing testing and evaluation systems have significant limitations.

[0003] First, there is a lack of unified quantitative evaluation standards. Different manufacturers and research institutions often use their own defined evaluation metrics, making it difficult to conduct a horizontal and objective comparison of the performance of different autonomous driving algorithms. Second, the evaluation dimensions are too narrow. Existing evaluations often focus on a single indicator, such as whether a collision occurs, while neglecting other dimensions crucial to passenger experience, such as ride comfort and driving efficiency. Some algorithms, while avoiding collisions, may employ extremely conservative strategies leading to inefficiency, or take actions such as sudden braking that severely impact comfort. Therefore, it is necessary to establish a quantitative evaluation model that incorporates multiple dimensions, including comfort, risk level, and driving efficiency, to achieve a comprehensive and objective analysis of the performance of autonomous driving systems. Summary of the Invention

[0004] The purpose of this invention is to address the technical problem that existing evaluation systems cannot effectively integrate multi-source physical parameters, making it difficult to conduct comprehensive quantitative evaluation of the performance of autonomous driving systems. This invention proposes a multi-dimensional quantitative evaluation method and system for autonomous driving simulation testing systems. By quantifying and comprehensively mapping multi-source parameters in the dimensions of comfort, risk, and speed, the invention achieves an objective and comprehensive quantitative evaluation of the performance of autonomous driving systems.

[0005] To achieve the above objectives, the present invention adopts the following technical solution.

[0006] A multi-dimensional quantitative evaluation method for an autonomous driving simulation test system includes the following steps: The motion state parameters of the vehicle under test are obtained, and the overall comfort level is calculated. The relative position parameters of the tested vehicle and traffic elements in the environment are obtained, and the comprehensive risk level is calculated by combining the comprehensive comfort level. The driving speed parameters of the vehicle under test are obtained, and the comprehensive speed is calculated by combining the comprehensive risk level. Based on the overall comfort level, the overall risk level, and the overall speed, a comprehensive performance evaluation result for autonomous driving is generated.

[0007] Furthermore, the motion state parameters of the tested vehicle are obtained, and the overall comfort level is calculated, including: Acceleration comfort is calculated based on lateral and longitudinal acceleration. The yaw rate comfort level is calculated based on the yaw rate. The acceleration comfort and the yaw rate comfort are weighted and summed to obtain the overall comfort.

[0008] Furthermore, based on lateral and longitudinal acceleration, acceleration comfort is calculated, including: Calculate the sum of the squares of the lateral acceleration and the longitudinal acceleration; The acceleration comfort level is obtained by integrating the sum of squares over time and calculating the time mean.

[0009] Furthermore, the relative positional parameters of the tested vehicle and traffic elements in the environment are obtained, and combined with the comprehensive comfort level, the comprehensive risk level is calculated, including: Extract the Euclidean distance between the measured vehicle and the nearest traffic element at each time point; Obtain the minimum value of the Euclidean distance over the entire test period, as the minimum distance risk; Based on the overall comfort level, the inverse average of the minimum distance risk under all test scenarios is calculated to obtain the overall risk level.

[0010] Furthermore, the driving speed parameters of the tested vehicle are obtained, and combined with the comprehensive risk level, the comprehensive speed is calculated, including: Calculate the average speed of the tested vehicle over the total travel time; Calculate the deviation of the tested vehicle from the target speed relative to the set speed during the total driving time; By combining the comprehensive risk level, the deviation between the average speed and the target speed is weighted and the difference is calculated to obtain the comprehensive speed.

[0011] Furthermore, the deviation of the tested vehicle from the target speed relative to the set speed during the total driving time is calculated, including: Calculate the absolute value of the difference between the real-time speed of the vehicle being tested and the set speed; The target velocity deviation is obtained by integrating the absolute value over time and calculating the time average.

[0012] Furthermore, based on the overall comfort level, the overall risk level, and the overall speed, a comprehensive performance evaluation result for autonomous driving is generated, including: The overall comfort, overall risk, and overall speed are positively evaluated and normalized to obtain positive evaluation scores for the three dimensions. The positive evaluation scores of the three dimensions are projected onto a radar chart coordinate system with three vertices to generate a closed geometric figure. Based on the characteristics of the closed geometric figure, a comprehensive performance evaluation result is obtained.

[0013] Furthermore, the overall comfort, overall risk, and overall speed are positively normalized and processed to obtain positive evaluation scores in three dimensions, including: For positive indicators where a larger value represents better performance, the ratio of the original value to the maximum value of all tested systems is calculated to obtain the corresponding positive evaluation score. For negative indicators where smaller values ​​represent better performance, the ratio of the minimum value to the original value of all tested systems is calculated to obtain the corresponding positive evaluation score.

[0014] Furthermore, based on the characteristics of the closed geometric figure, a comprehensive performance evaluation result is obtained, including: The overall performance of the autonomous driving system is evaluated by the area of ​​the closed geometric figure. The strategy preferences of the autonomous driving system are identified by the direction of the center of gravity offset and the concave and convex features of the closed geometric figure.

[0015] A multi-dimensional quantitative evaluation system for autonomous driving simulation testing systems includes: The comfort assessment module is used to acquire the motion state parameters of the vehicle under test and calculate the overall comfort level. The risk assessment module is used to obtain the relative position parameters of the tested vehicle and traffic elements in the environment, and calculate the comprehensive risk level in combination with the comprehensive comfort level. The speed assessment module is used to obtain the driving speed parameters of the tested vehicle and calculate the comprehensive speed by combining the comprehensive risk level. The comprehensive evaluation module is used to generate a comprehensive performance evaluation result for autonomous driving based on the comprehensive comfort level, the comprehensive risk level, and the comprehensive speed.

[0016] The present invention has achieved the following beneficial effects.

[0017] 1. This invention solves the problem of difficulty in horizontally comparing algorithm performance caused by inconsistent standards in autonomous driving testing by establishing a multi-dimensional quantitative model covering comfort, risk, and speed, and using a unified physical parameter processing logic.

[0018] 2. This invention uses time integration to process acceleration and yaw rate. Compared with instantaneous value determination, it can more accurately measure the cumulative oscillation of the system throughout the entire operation and reflect a more realistic sense of ride smoothness.

[0019] 3. This invention can accurately capture instantaneous high-risk approach states during driving by extracting the minimum Euclidean distance over the entire time period and summing the reciprocals, thus avoiding the defect that mean calculation may mask extreme dangerous conditions.

[0020] 4. By introducing a target speed deviation as a penalty and combining it with the average speed, this invention effectively balances driving efficiency and speed control capability, and can identify inefficiencies caused by excessive conservatism or frequent speed fluctuations.

[0021] 5. This invention utilizes index positiveization, normalization processing, and radar chart mapping mechanisms to transform heterogeneous physical data into intuitive geometric figures. Through the characteristics of the figure area and centroid distribution, it enables rapid identification and evaluation of the system's overall performance and strategy tendencies. Attached Figure Description

[0022] Figure 1 This is a flowchart of the multi-dimensional quantitative evaluation method for the autonomous driving simulation test system in the embodiment. Figure 2 A scoring radar chart of a common autonomous driving system generated using the method of this invention; Figure 3 This is a block diagram of the multi-dimensional quantitative evaluation system of the autonomous driving simulation test system in the embodiment. Detailed Implementation

[0023] The present invention will be further described below with reference to the accompanying drawings and specific embodiments, but this does not constitute any limitation on the present invention.

[0024] This invention provides a multi-dimensional quantitative evaluation method for an autonomous driving simulation testing system, the process of which is as follows: Figure 1 As shown, it includes the following steps: Step S1: Obtain the motion state parameters of the vehicle under test and calculate the overall comfort level.

[0025] Specifically, overall comfort is used to evaluate the smoothness of system operation. It considers not only instantaneous values ​​but also measures the cumulative oscillations over time by integrating acceleration and yaw rate, reflecting the passenger's true experience. Comfort assesses the vehicle's stability during driving. When the vehicle experiences frequent rapid acceleration, deceleration, or sharp turns, the amplitude of acceleration or yaw rate increases, resulting in a higher integrated comfort value, indicating a decrease in comfort. Conversely, when the vehicle travels smoothly, with gentle acceleration and deceleration and smooth steering, the amplitude of acceleration or yaw rate decreases, resulting in a lower integrated comfort value, indicating an increase in comfort.

[0026] In an optional embodiment of the present invention, step S1 may include: Step S11: Calculate acceleration comfort based on lateral and longitudinal acceleration.

[0027] In an optional embodiment of the present invention, step S11 may include: Step S111: Calculate the sum of the squares of the lateral acceleration and the longitudinal acceleration.

[0028] Step S112: Integrate the sum of squares over time and calculate the time mean to obtain the acceleration comfort level.

[0029] Specifically, acceleration comfort The definition is as follows: in, and These respectively indicate that the vehicle is in shaft and Acceleration in the axial direction; This indicates the total travel time.

[0030] Step S12: Calculate the yaw rate comfort level based on the yaw rate.

[0031] Specifically, in an optional embodiment of the present invention, the yaw rate comfort The definition is as follows: in, This indicates the yaw rate of the vehicle. This indicates the total travel time.

[0032] Step S13: Perform a weighted summation of the acceleration comfort and the yaw rate comfort to obtain the overall comfort.

[0033] Specifically, overall comfort From acceleration comfort and yaw rate comfort We perform a weighted summation, defined as follows: in, and These are weighting coefficients, which can be adjusted according to actual needs. For overall comfort, a comfort evaluation model is constructed by calculating the average of the square integrals of longitudinal acceleration and lateral acceleration, as well as the average of the square integrals of yaw rate, and then summing them using the weighting coefficients.

[0034] Step S2: Obtain the relative position parameters of the tested vehicle and traffic elements in the environment, and calculate the comprehensive risk level by combining the comprehensive comfort level.

[0035] Specifically, the risk level is used to assess the safety of the system in complex traffic situations. It is calculated based on the minimum distance between the vehicle and obstacles, and can capture even a fleeting moment of high risk during operation. The risk level assesses the performance of the autonomous driving system in complex traffic situations or potential hazards, particularly capturing the most dangerous moments during operation. When the autonomous vehicle approaches an obstacle or pedestrian at close range, or even nearly collides, the minimum distance risk value decreases, resulting in an increase in the overall risk value calculated by reciprocal calculation, representing a higher risk. Conversely, when the autonomous vehicle maintains a large safe distance from surrounding obstacles, the minimum distance value increases, resulting in a decrease in the overall risk value calculated by reciprocal calculation, representing a lower risk.

[0036] In an optional embodiment of the present invention, step S2 may include: Step S21: Extract the Euclidean distance between the measured vehicle and the nearest traffic element at each time point.

[0037] Step S22: Obtain the minimum value of the Euclidean distance during the entire test period, as the minimum distance risk.

[0038] Specifically, minimum distance risk$ The definition is as follows: in, and These represent the autonomous vehicle and the traffic element closest to the vehicle at time [time]. The position; the min function is used to extract from 0 to... The minimum value within the time period.

[0039] Step S23: Based on the overall comfort level, calculate the inverse average of the minimum distance risk under all test scenarios to obtain the overall risk level.

[0040] Specifically, comprehensive risk level The definition is as follows: in, This represents the total number of test scenarios; For the first Minimum distance risk in each scenario. To assess risk level, the minimum Euclidean distance between the vehicle and the nearest traffic element is extracted throughout the entire test period. The reciprocal of this distance is then taken for all test scenarios, and the average is calculated. The closer the distance, the higher the risk score, thus constructing a risk level assessment model.

[0041] Step S3: Obtain the driving speed parameters of the vehicle under test, and calculate the comprehensive speed by combining the comprehensive risk level.

[0042] Specifically, driving speed is used to evaluate the system's driving efficiency. Taking into account both the average driving speed and the deviation from the target speed, smooth and efficient driving receives a positive evaluation, while excessively slow or fluctuating speeds receive a negative evaluation. The purpose of constructing the driving speed evaluation model is to assess the driving efficiency of the autonomous driving system from its starting point to its destination, as well as its ability to maintain the preset target speed. When the autonomous vehicle can maintain a high driving speed while ensuring safety, the average speed value increases, resulting in a higher overall speed value, representing improved efficiency. Conversely, when the autonomous vehicle drives too slowly or experiences drastic speed fluctuations that deviate from the target speed, such as frequent sudden stops or speeding, the speed deviation value increases. Since this is a penalty factor, it reduces the overall speed value, representing decreased efficiency.

[0043] In an optional embodiment of the present invention, step S3 may include: Step S31: Calculate the average speed of the tested vehicle during the total driving time.

[0044] Specifically, average speed The definition is as follows: in, Indicates the vehicle at time speed; This represents the total time.

[0045] Step S32: Calculate the deviation of the tested vehicle from the target speed relative to the set speed during the total driving time.

[0046] In an optional embodiment of the present invention, step S32 may include: Step S321: Calculate the absolute value of the difference between the real-time speed of the vehicle under test and the set speed.

[0047] Step S322: Integrate the absolute value over time and calculate the time average to obtain the target speed deviation.

[0048] Specifically, target speed deviation The definition is as follows: in, Indicates the vehicle at time speed; This indicates the set target cruising speed; This represents the total time.

[0049] Step S33: Based on the comprehensive risk level, perform a weighted difference calculation on the deviation between the average speed and the target speed to obtain the comprehensive speed.

[0050] Specifically, overall speed The definition is as follows: in, and The weighting coefficients can be adjusted according to actual needs. For driving speed, efficiency is reflected by calculating the integral of the deviation between the average speed and the target speed and then performing weighted difference processing.

[0051] Step S4: Based on the overall comfort level, the overall risk level, and the overall speed, generate an overall performance evaluation result for autonomous driving.

[0052] Specifically, to intuitively compare the overall performance of different autonomous driving systems, radar charts are used for visualization, such as... Figure 2 As shown, by normalizing the original calculated values ​​of the above three dimensions and mapping the results onto a radar chart, the performance of the tested system can be intuitively displayed, and the advantages and disadvantages between different systems can be analyzed.

[0053] In an optional embodiment of the present invention, step S4 may include: Step S41: The overall comfort, overall risk, and overall speed are positiveized and normalized to obtain positive evaluation scores for the three dimensions.

[0054] Specifically, the positiveization and normalization of indicators aim to eliminate the differences among the three evaluation dimensions of overall comfort, overall risk, and overall speed, and to uniformly convert all indicators into positive scores where larger values ​​represent better performance, so as to facilitate comparison within the same coordinate system. Through normalization, the scores of each dimension are mapped to the range [0, 100].

[0055] In an optional embodiment of the present invention, step S41 may include: Step S411: For positive indicators where a larger value represents better performance, calculate the ratio of the original value to the maximum value of all tested systems to obtain the corresponding positive evaluation score.

[0056] Specifically, let the first The original value of the system is The maximum value in all systems is Then the normalized score The calculation formula is: Step S412: For negative indicators where smaller values ​​represent better performance, calculate the ratio of the minimum value to the original value for all tested systems to obtain the corresponding positive evaluation score.

[0057] Specifically, let the original value of the j-th system be... The minimum value in all systems is Then the normalized score The calculation formula is: Step S42: Project the positive evaluation scores of the three dimensions onto a radar chart coordinate system with three vertices to generate a closed geometric figure.

[0058] Specifically, the graphic mapping process involves constructing a radar chart coordinate system with three vertices. The three axes are defined as high comfort, low risk, and high driving speed, respectively. The positive evaluation scores for the three dimensions calculated in the previous step are then... The data points are projected onto their respective coordinate axes to determine their specific positions in the coordinate system. Then, these three data points are connected sequentially to form a closed triangular region, thus completing the mapping process from abstract data to a visual geometric figure.

[0059] Step S43: Based on the characteristics of the closed geometric figure, obtain the comprehensive performance evaluation result.

[0060] In an optional embodiment of the present invention, step S43 may include: Step S431: Evaluate the overall performance of the autonomous driving system by measuring the area of ​​the closed geometric figure.

[0061] Specifically, the larger the area of ​​the triangle, the stronger the overall performance of the system in all three dimensions.

[0062] Step S432: Identify the strategy preference of the autonomous driving system by the direction of the center of gravity offset and the concave and convex features of the closed geometric figure.

[0063] Specifically, the strategy preference of the autonomous driving system is evaluated by analyzing the shape characteristics of closed triangles in the radar chart. If the center of gravity of the triangle shifts towards the high speed axis and is concave on the low risk axis, it indicates that the system has adopted an aggressive strategy; if the shape is full on the high comfort and low risk axes but shrinks on the high speed axis, it indicates that the system has adopted a conservative strategy.

[0064] This invention also provides a multi-dimensional quantitative evaluation system for autonomous driving simulation testing systems, such as... Figure 3 As shown, it includes: The comfort assessment module is used to acquire the motion state parameters of the vehicle under test and calculate the overall comfort level. The risk assessment module is used to obtain the relative position parameters of the tested vehicle and traffic elements in the environment, and calculate the comprehensive risk level in combination with the comprehensive comfort level. The speed assessment module is used to obtain the driving speed parameters of the tested vehicle and calculate the comprehensive speed by combining the comprehensive risk level. The comprehensive evaluation module is used to generate a comprehensive performance evaluation result for autonomous driving based on the comprehensive comfort level, the comprehensive risk level, and the comprehensive speed.

[0065] Method performance testing: To verify the effectiveness of the proposed method, a comparative test was conducted on two autonomous driving systems, Apollo 6.0 and Autoware, using the proposed method on a CARLA-based simulation platform.

[0066] Test results are as follows Figure 2 As shown in the radar chart, the Apollo system scores highly in the speed dimension, with its triangle's center of gravity shifting significantly towards the speed axis, indicating a more aggressive driving strategy. However, its score in the risk level dimension is slightly lower than Autoware. In contrast, the Autoware system performs better in the comfort and risk levels dimensions. Its graph is fuller along the safety and stability axis, but contracts along the speed axis, reflecting a low-risk but inefficient conservative driving strategy.

[0067] This quantitative evaluation mechanism can clearly identify and distinguish the underlying logic of different algorithms: Apollo tends to be efficient but risky, while Autoware tends to be safe but inefficient. This test result not only objectively quantifies the performance differences between the two systems but also fully verifies the effectiveness of the evaluation method of this invention in identifying the overall performance and strategy preferences of autonomous driving systems.

[0068] The above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit them. Those skilled in the art can modify or make equivalent substitutions to the technical solutions of the present invention. The scope of protection of the present invention is defined by the claims.

Claims

1. A multi-dimensional quantitative evaluation method for an autonomous driving simulation testing system, characterized in that, Includes the following steps: The motion state parameters of the vehicle under test are obtained, and the overall comfort level is calculated. The relative position parameters of the tested vehicle and traffic elements in the environment are obtained, and the comprehensive risk level is calculated by combining the comprehensive comfort level. The driving speed parameters of the vehicle under test are obtained, and the comprehensive speed is calculated by combining the comprehensive risk level. Based on the overall comfort level, the overall risk level, and the overall speed, a comprehensive performance evaluation result for autonomous driving is generated.

2. The method as described in claim 1, characterized in that, The motion state parameters of the vehicle under test are obtained, and the overall comfort level is calculated, including: Acceleration comfort is calculated based on lateral and longitudinal acceleration. The yaw rate comfort level is calculated based on the yaw rate. The acceleration comfort and the yaw rate comfort are weighted and summed to obtain the overall comfort.

3. The method as described in claim 2, characterized in that, Acceleration comfort is calculated based on lateral and longitudinal acceleration, including: Calculate the sum of the squares of the lateral acceleration and the longitudinal acceleration; The acceleration comfort level is obtained by integrating the sum of squares over time and calculating the time mean.

4. The method as described in claim 1, characterized in that, The relative position parameters of the tested vehicle and traffic elements in the environment are obtained, and combined with the comprehensive comfort level, the comprehensive risk level is calculated, including: Extract the Euclidean distance between the measured vehicle and the nearest traffic element at each time point; Obtain the minimum value of the Euclidean distance over the entire test period, as the minimum distance risk; Based on the overall comfort level, the inverse average of the minimum distance risk under all test scenarios is calculated to obtain the overall risk level.

5. The method as described in claim 1, characterized in that, The vehicle's speed parameters are obtained, and combined with the overall risk level, the overall speed is calculated, including: Calculate the average speed of the tested vehicle over the total driving time; Calculate the deviation of the tested vehicle from the target speed relative to the set speed during the total driving time; By combining the comprehensive risk level, a weighted difference is calculated on the deviation between the average speed and the target speed to obtain the comprehensive speed.

6. The method as described in claim 5, characterized in that, Calculate the deviation of the tested vehicle from the target speed relative to the set speed over the total driving time, including: Calculate the absolute value of the difference between the real-time speed of the vehicle being tested and the set speed; The target velocity deviation is obtained by integrating the absolute value over time and calculating the time average.

7. The method as described in claim 1, characterized in that, Based on the overall comfort level, the overall risk level, and the overall speed, a comprehensive performance evaluation result for autonomous driving is generated, including: The overall comfort, overall risk, and overall speed are positively evaluated and normalized to obtain positive evaluation scores for the three dimensions. The positive evaluation scores of the three dimensions are projected onto a radar chart coordinate system with three vertices to generate a closed geometric figure. Based on the characteristics of the closed geometric figure, a comprehensive performance evaluation result is obtained.

8. The method as described in claim 7, characterized in that, The overall comfort, overall risk, and overall speed are positively normalized and processed to obtain positive evaluation scores in three dimensions, including: For positive indicators where a larger value represents better performance, the ratio of the original value to the maximum value of all tested systems is calculated to obtain the corresponding positive evaluation score. For negative indicators where smaller values ​​represent better performance, the ratio of the minimum value to the original value of all tested systems is calculated to obtain the corresponding positive evaluation score.

9. The method as described in claim 7, characterized in that, Based on the characteristics of the closed geometric figure, a comprehensive performance evaluation result is obtained, including: The overall performance of the autonomous driving system is evaluated by the area of ​​the closed geometric figure. The strategy preferences of the autonomous driving system are identified by the direction of the center of gravity offset and the concave and convex features of the closed geometric figure.

10. A multi-dimensional quantitative evaluation system for an autonomous driving simulation testing system, characterized in that, include: The comfort assessment module is used to acquire the motion state parameters of the vehicle under test and calculate the overall comfort level. The risk assessment module is used to obtain the relative position parameters of the tested vehicle and traffic elements in the environment, and calculate the comprehensive risk level in combination with the comprehensive comfort level. The speed evaluation module is used to obtain the driving speed parameters of the tested vehicle and calculate the comprehensive speed by combining the comprehensive risk level. The comprehensive evaluation module is used to generate a comprehensive performance evaluation result for autonomous driving based on the comprehensive comfort level, the comprehensive risk level, and the comprehensive speed.

Images