Method for determining road test mileage and method for evaluating autonomous driving performance

Abstract

The present disclosure provides a method for determining road test mileage and a method for evaluating automatic driving performance, and relates to the technical field of computers, in particular to the technical field of automatic driving, big data and cloud computing. The specific implementation scheme is: determining a driving scene set of a road network region according to historical road test data of the road network region; determining a corresponding relationship between vehicle driving mileage and the number of driving scenes and a scene coverage rate relationship between the number of driving scenes and the driving scene set according to the historical road test data; and determining a reference road test mileage of the road network region based on a preset scene coverage rate, the corresponding relationship and the scene coverage rate relationship. According to the scheme of the present disclosure, the reference road test mileage of the road network region can be accurately obtained, so that the automatic driving vehicle can quantitatively evaluate the test sufficiency in the road network region according to the reference road test mileage.

Description

Technical Field

[0001] This disclosure relates to the field of computer technology, and in particular to the fields of autonomous driving, big data and cloud computing. Background Technology

[0002] Autonomous driving testing of vehicles generally consists of three phases: simulation offline testing, closed-course testing, and road testing. Road testing, in particular, requires interaction with real traffic participants in a real traffic flow environment to verify the vehicle's autonomous driving capabilities. Summary of the Invention

[0003] This disclosure provides a method for determining road test mileage and a method for evaluating autonomous driving performance.

[0004] According to one aspect of this disclosure, a method for determining road test mileage is provided, comprising:

[0005] Based on historical road test data of the road network area, determine the set of driving scenarios for the road network area;

[0006] Based on historical road test data, the correlation between vehicle mileage and the number of driving scenarios was determined, as well as the relationship between the number of driving scenarios and the scenario coverage of the driving scenario set; and

[0007] Based on the preset scene coverage, the benchmark road test mileage of the road network area is determined according to the correspondence and scene coverage relationship.

[0008] According to another aspect of this disclosure, a method for evaluating autonomous driving performance is provided, comprising:

[0009] Based on the autonomous driving road test data of the target test vehicle in the road network area, determine the road test mileage of the target test vehicle;

[0010] Using the method for determining road test mileage according to any embodiment of this disclosure, a reference road test mileage for a road network area is determined; and

[0011] Provided that the road test mileage is not less than the benchmark road test mileage, the sufficiency of the test to evaluate the autonomous driving performance of the target test vehicle meets the requirements.

[0012] According to another aspect of this disclosure, a road test mileage determination apparatus is provided, comprising:

[0013] The first determining module is used to determine the set of driving scenarios in the road network area based on historical road test data of the road network area;

[0014] The second determining module is used to determine, based on historical road test data, the correspondence between vehicle mileage and the number of driving scenarios, as well as the relationship between the number of driving scenarios and the scenario coverage of the driving scenario set; and

[0015] The third determination module is used to determine the benchmark road test mileage of the road network area based on the preset scene coverage rate and the corresponding relationship and scene coverage rate relationship.

[0016] According to another aspect of this disclosure, an evaluation apparatus for autonomous driving performance is provided, comprising:

[0017] The sixth determination module is used to determine the road test mileage of the target test vehicle based on the autonomous driving road test data of the target test vehicle in the road network area;

[0018] The seventh determining module is used to determine the reference road test mileage of the road network area using the road test mileage determination method of any embodiment of the present disclosure;

[0019] The evaluation module is used to assess whether the autonomous driving performance of the target test vehicle meets the testing adequacy requirements, provided that the road test mileage is not less than the benchmark road test mileage.

[0020] According to another aspect of this disclosure, an electronic device is provided, comprising:

[0021] At least one processor; and

[0022] The memory is communicatively connected to the at least one processor; wherein,

[0023] The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the methods of any embodiment of the present disclosure.

[0024] According to another aspect of this disclosure, a non-transitory computer-readable storage medium is provided storing computer instructions, wherein the computer instructions are used to cause the computer to perform a method according to any embodiment of this disclosure.

[0025] According to another aspect of this disclosure, a computer program product is provided, including a computer program that, when executed by a processor, implements a method according to any embodiment of this disclosure.

[0026] According to the scheme disclosed herein, the baseline road test mileage of the road network area can be accurately obtained.

[0027] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0028] The accompanying drawings are provided to better understand this solution and do not constitute a limitation of this disclosure. Wherein:

[0029] Figure 1 This is a schematic flowchart of a method for determining road test mileage according to an embodiment of this disclosure;

[0030] Figure 2 This is a schematic diagram of the functional relationship of the method for determining road test mileage according to an embodiment of the present disclosure;

[0031] Figure 3 This is a flowchart illustrating an autonomous driving performance evaluation method according to an embodiment of the present disclosure;

[0032] Figure 4 This is the application intent of the method for determining road test mileage and / or the method for evaluating autonomous driving performance according to the embodiments of this disclosure;

[0033] Figure 5 This is a schematic diagram of the structure of a road test mileage determination device according to an embodiment of the present disclosure;

[0034] Figure 6 This is a schematic diagram of the structure of an autonomous driving performance evaluation device according to an embodiment of the present disclosure;

[0035] Figure 7 This is a block diagram of an electronic device used to implement the method for determining road test mileage and / or the method for evaluating autonomous driving performance according to embodiments of the present disclosure. Detailed Implementation

[0036] The exemplary embodiments of this disclosure are described below with reference to the accompanying drawings, including various details of the embodiments to aid understanding, and should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope of this disclosure. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0037] This disclosure provides a method for determining road test mileage, such as... Figure 1 As shown, the method includes:

[0038] Step S101: Determine the set of driving scenarios for the road network area based on historical road test data of the road network area.

[0039] Step S102: Based on historical road test data, determine the correspondence between vehicle mileage and the number of driving scenarios, as well as the relationship between the number of driving scenarios and the scenario coverage of the driving scenario set.

[0040] Step S103: Based on the preset scene coverage rate, determine the benchmark road test mileage of the road network area according to the correspondence and scene coverage relationship.

[0041] Based on the above embodiments of this disclosure, it should be noted that:

[0042] A road network area can be understood as any area where vehicle driving tests can be conducted. This road network area may contain one or more road networks. The traffic environment of this road network area must include at least road elements (e.g., wide roads, narrow roads, roundabouts, zebra crossings, turning lanes, lane markings, etc.) and traffic facilities (e.g., fences, bollards, safety islands, traffic lights, etc.).

[0043] Historical road test data can be understood as data obtained from multiple vehicles conducting road tests (including autonomous driving tests and manually controlled tests) within a road network area at the same or different time periods. This data can be data collected by sensors during vehicle operation, autonomous driving data generated by the vehicle based on the surrounding environment, autonomous driving decision data, etc., without specific limitations here.

[0044] The driving scenario set can include all driving scenarios a vehicle might encounter while traveling within a road network area. These driving scenarios can include one or more of the following: time elements, weather elements, traffic environment elements, and traffic participant elements. For example, a driving scenario could be: encountering a pedestrian crossing the road (traffic participant element) during the morning rush hour on a snowy day (weather element) while driving on a wide road (traffic environment element). Another example is: encountering a bus changing lanes during the off-peak hour on a rainy day (weather element) while navigating a roundabout (traffic environment element). The driving scenarios in this set can contain coarse-grained information, meaning they may not include detailed information about a particular element; for example, a bus changing lane scenario might not distinguish between changing lane distance and changing lane time.

[0045] The relationship between vehicle mileage and the number of driving scenarios can be understood as the specific number of driving scenarios encountered during a vehicle's journey of a certain distance. For example, if the target vehicle has traveled 10 kilometers, it has encountered 2 driving scenarios during that time. If the target vehicle has traveled 100 kilometers, it has encountered 20 driving scenarios during that time.

[0046] The relationship between the number of driving scenarios and the scenario coverage of the driving scenario set can be understood as the percentage of driving scenarios corresponding to different vehicle mileages within the total number of driving scenarios in the driving scenario set. For example, if the driving scenario set includes 50 driving scenarios, and the target vehicle travels 10 kilometers, encountering 2 driving scenarios during this period, then the relationship between the number of driving scenarios and the scenario coverage of the driving scenario set is 2 ÷ 50 = 4%, meaning that the target vehicle travels 10 kilometers within the road network area, and the corresponding scenario coverage of the road network area is 4%. Similarly, if the driving scenario set includes 50 driving scenarios, and the target vehicle travels 100 kilometers, encountering 20 driving scenarios during this period, then the relationship between the number of driving scenarios and the scenario coverage of the driving scenario set is 20 ÷ 50 = 40%, meaning that the target vehicle travels 10 kilometers within the road network area, and the corresponding scenario coverage of the road network area is 40%.

[0047] The preset scenario coverage rate can be set based on experience. Alternatively, it can be determined based on the correlation between vehicle mileage and the number of driving scenarios in the road network area, as well as the relationship between the number of driving scenarios and the scenario coverage rate of the driving scenario set. For example, the preset scenario coverage rate can be determined based on this correlation. If the driving scenario set includes 50 driving scenarios, and it is found that when the vehicle travels 500 kilometers, 45 driving scenarios are encountered in the road network area (90% scenario coverage), and when the vehicle travels 1000 kilometers, 47 driving scenarios are encountered (94% scenario coverage), then the preset scenario coverage rate is set to 90%. This is because traveling an additional 500 kilometers to encounter an additional 4% of driving scenarios is less efficient and wastes resources compared to encountering 90% of driving scenarios after only 500 kilometers.

[0048] The correspondence and scenario coverage relationship can be understood as the proportion of driving scenarios corresponding to different vehicle mileages in the total number of driving scenarios in the driving scenario set, and the relationship between the number of driving scenarios corresponding to different vehicle mileages and the vehicle mileage itself. For example, if the driving scenario set includes 50 driving scenarios, and the target vehicle has traveled 10 kilometers, encountering 2 driving scenarios during this period, then the relationship between the number of driving scenarios and the scenario coverage of the driving scenario set is 2 ÷ 50 = 4%. That is, if the target vehicle travels 10 kilometers within the road network area, the scenario coverage of the corresponding road network area is 4%.

[0049] The baseline road test mileage in a road network area can be understood as the mileage a vehicle travels under a preset scenario coverage rate. For example, when the preset scenario coverage rate is 90% and the driving scenario set includes 50 driving scenarios, based on historical road test data, the correspondence between vehicle mileage and the number of driving scenarios, and the relationship between the number of driving scenarios and the scenario coverage rate of the driving scenario set, it can be known that when a vehicle travels 500 kilometers, it will encounter 45 driving scenarios in the road network area, that is, the scenario coverage rate reaches 90%. At this point, 500 kilometers is the baseline road test mileage in the road network area.

[0050] According to embodiments of this disclosure, research has found that when continuously testing a road network area, the traffic flow environment within that area is essentially fixed, making the major categories of autonomous driving test scenarios largely predictable. Therefore, determining the required testing distance (in kilometers) to demonstrate sufficient testing sufficiency and how to quantify this sufficiency is a key technical problem. However, existing methods cannot quantify the sufficiency of autonomous driving performance testing; they rely solely on subjective evaluation and lack a theoretical basis for providing targeted testing and verification methods based on data distribution. The method for determining road test mileage in embodiments of this disclosure effectively solves this technical problem. When testing and delivering autonomous driving capabilities in a new road network area, the method of this disclosure, by combining historical road test data of the road network area and employing scenario mining, can accurately determine the set of driving scenarios for that area. Then, by utilizing the correspondence between vehicle mileage and the number of driving scenarios, as well as the relationship between the number of driving scenarios and the scenario coverage of the driving scenario set, the baseline road test mileage for the road network area can be accurately obtained. This allows autonomous vehicles to quantitatively assess the sufficiency of testing in the road network area based on the baseline road test mileage, providing a strong basis for subsequent autonomous vehicle testing and delivery. The method in this disclosure equates road test mileage to a combination of continuous driving scenarios, observing the adequacy of vehicle testing in the road network area from a scenario perspective. Because the scenarios occur continuously and the interactions are realistic, it ensures the advantage of a rich and comprehensive variety of driving scenarios within the defined set of driving scenarios.

[0051] In one embodiment, the method for determining road test mileage according to this disclosure includes steps S101 to S103, wherein step S101: determining the set of driving scenarios for the road network area based on historical road test data of the road network area may include:

[0052] Step S1011: Based on the historical road test data of the road network area, obtain the road test data collected in the first time period.

[0053] Step S1012: Based on the preset scenario mining rules and the road test data collected in the first time period, determine the multiple first driving scenarios contained in the road network area.

[0054] Step S1013: Determine the set of driving scenarios for the road network area based on multiple first driving scenarios.

[0055] Based on the above embodiments of this disclosure, it should be noted that:

[0056] Historical road test data can include road test data collected from different vehicles at different time periods. The road test data collected in the first time period can be understood as a portion of the historical road test data.

[0057] The pre-defined scenario mining rules may include one or more of the following: time elements, weather elements, traffic environment elements, and traffic participant elements. Based on the pre-defined scenario mining rules and the information represented by each data point in the road test data collected in the first time period, multiple first driving scenarios are determined within the road network area during the first time period. Each first driving scenario includes at least one information related to one or more of the following: time elements, weather elements, traffic environment elements, and traffic participant elements. For example, a first driving scenario could be: encountering a pedestrian crossing the road (traffic participant element) while driving on a wide road (traffic environment element) during the morning rush hour (time element) on a snowy day (weather element). Another example is: encountering a bus cutting in during the off-peak hour (time element) while navigating a roundabout (traffic environment element) on a rainy day (weather element).

[0058] According to embodiments of this disclosure, a set of driving scenarios for a road network area can be accurately extracted from historical road test data based on preset scenario mining rules.

[0059] In one example, based on multiple first driving scenarios, the set of driving scenarios for the road network area is determined. This includes: studying the test data of each vehicle per day within the first time period as a unit, recording the number of driving scenarios and test mileage for each vehicle, accumulating and deduplicating the number of driving scenarios encountered by different vehicles as the vertical axis, and accumulating the test mileage of different vehicles as the horizontal axis. Based on the comprehensive determination of each vehicle, the relationship between the test mileage and the number of driving scenarios in the road network area is determined. As the test mileage increases, the number of driving scenarios gradually stabilizes until the mileage increases but the number of driving scenarios no longer increases. At this point, the set of driving scenarios for the road network area can be considered to have been determined.

[0060] It should be noted that, in order to minimize the impact of randomness and uncertainty in driving scenarios, the order of vehicle processing is shuffled. If the relationship between test mileage and the number of scenarios remains basically consistent when using the above method to plot the coordinate system of vehicle data from different time series, then the driving scenario set is considered accurate.

[0061] In one example, a large-scale test is first conducted on a selected road network area, covering different weather conditions, including sunny days, rainy days, and backlighting scenarios. The test also covers all traffic elements within the selected road network area, including roundabouts, main and auxiliary roads, single lanes, multi-lane roads, pedestrian crossings, and school zones. It also covers delivery times, such as off-peak and morning / evening rush hours. The number of vehicles online simultaneously must be no less than the road network mileage divided by 10. For a 100km road network, 10 vehicles are needed for coverage (the rationale is that with an average vehicle interval of 10km, and an average speed of 20km / h, vehicles can reach each other within half an hour, and traffic flow is not expected to change significantly within that time). The specific number of test vehicles required will be adjusted based on the mileage of the road network area and is not specifically limited here. Secondly, offline simulation tools with preset scenario mining rules were used to conduct scenario mining on the test data. The granularity of the scenarios was coarser compared to the construction of the simulation scenario library (for example, vehicle switching scenarios only distinguished between bus switching, private vehicle switching, and non-motorized vehicle switching, without distinguishing between switching distance and switching time). One day's test data for one vehicle was used as a unit for research, recording the number of driving scenarios and test mileage. The number of driving scenarios encountered by different vehicles was accumulated and removed, used as the ordinate, and the test mileage of different vehicles was accumulated, used as the abscissa, to determine the relationship between test mileage and the number of driving scenarios. As test mileage increased, the number of driving scenarios gradually stabilized until the number of scenarios no longer increased with increasing mileage, at which point the complete scenario set was considered formed. Finally, to minimize the impact of randomness and uncertainty in scenario occurrences, the order of vehicle processing was shuffled, and the relationship between test mileage and the number of scenarios was recorded for vehicle data recorded in different orders. If the trend was generally consistent, the scenario set Q was considered accurate.

[0062] In one embodiment, the method for determining road test mileage according to this disclosure includes steps S101 to S103 and steps S1011 to S1013, wherein step S1011: obtaining road test data collected in a first time period based on historical road test data of the road network area, may include:

[0063] Based on historical road test data of the road network area, obtain the first driving data generated by multiple vehicles in the first time period.

[0064] The data containing weather information, traffic environment information, and driving interaction information in the first driving data is identified as the road test data collected in the first time period.

[0065] Based on the above embodiments of this disclosure, it should be noted that:

[0066] Weather information can include: sunny, rainy, backlit, cloudy, rainy / snowy, foggy / hazy, windy, etc.

[0067] Traffic environment information may include: wide roads, narrow roads, fences, guardrails, safety islands, roundabouts, main and auxiliary roads, single lanes, multi-lane roads, school zones, off-peak hours, morning and evening peak hours, etc.

[0068] Driving interaction information may include: aggressive driving, delivery vehicles illegally crossing roads, pedestrians crossing the road, buses changing lanes, private vehicles changing lanes, and non-motorized vehicles changing lanes.

[0069] According to embodiments of this disclosure, by utilizing data related to weather information, traffic environment information, and driving interaction information, road test data related to driving scenario mining can be accurately found, providing a foundation for constructing a set of driving scenarios in the road network area.

[0070] In one embodiment, the method for determining road test mileage according to this disclosure includes steps S101 to S103, wherein step S102: determining the correspondence between vehicle mileage and the number of driving scenarios, and the relationship between the number of driving scenarios and the scenario coverage of the driving scenario set, based on historical road test data, may include:

[0071] Based on historical road test data, obtain the road test data collected in the second time period.

[0072] Based on the road test data collected in the second time period, which includes weather information, traffic environment information, and driving interaction information, the number of second driving scenarios corresponding to the road test data collected in the second time period is determined.

[0073] Based on the road test data collected in the second time period, the correspondence between vehicle mileage and the number of second driving scenarios was determined.

[0074] Based on the relationship between the number of second driving scenarios and the number of multiple first driving scenarios in the driving scenario set, the relationship between the number of second driving scenarios and the scenario coverage of the driving scenario set is determined.

[0075] Based on the above embodiments of this disclosure, it should be noted that:

[0076] Historical road test data can include road test data collected by different vehicles at different time periods. Road test data collected in the second time period can be understood as a portion of the historical road test data. The second time period is a different time period from the first time period; for example, the second time period can be later than the first time period.

[0077] To determine the correspondence between vehicle mileage and the number of second driving scenarios, please refer to the description of "correspondence between vehicle mileage and the number of driving scenarios" in the above embodiment, which will not be repeated here.

[0078] The specific scenario of the second driving scenario can be understood as a scenario that is consistent with the driving scenarios in the driving scenario set. Because the driving scenario set already includes the complete set of driving scenarios in the road network area when it is constructed, the second driving scenario encountered by the vehicle during the second time period is basically consistent with the driving scenarios in the driving scenario set.

[0079] To determine the relationship between the number of second driving scenarios and the scene coverage of the driving scenario set, please refer to the description of "the relationship between the number of driving scenarios and the scene coverage of the driving scenario set" in the above embodiment, which will not be repeated here.

[0080] According to the embodiments of this disclosure, road test data collected in the second time period is used to verify the driving scenario set constructed based on the road test data collected in the first time period. By using two sets of data from different time dimensions, the correspondence between vehicle mileage and the number of driving scenarios, as well as the relationship between the number of driving scenarios and the scenario coverage of the driving scenario set, can be determined more accurately. Furthermore, since the road test data collected in the second time period includes weather information, traffic environment information, and driving interaction information, the uniformity of the road test data collected in the second time period is ensured. This uniformity is reflected in full coverage of the time period, full coverage of the weather environment, full coverage of traffic elements, and uniform road network coverage within the road network area.

[0081] In one example, after determining the set of driving scenarios, acceptance testing officially begins. The ratio of the number of deduplicated driving scenarios P ​​mined from the acceptance data (i.e., road test data collected in the second time period) to the number of driving scenarios Q in the driving scenario set is defined as the scenario coverage rate f, and used as the ordinate. The acceptance mileage L (obtained based on the road test data collected in the second time period) is used as the abscissa, and the relationship between the scenario coverage rate f and the acceptance mileage L is observed (e.g., ...). Figure 2 (As shown).

[0082] In one embodiment, the method for determining road test mileage according to this disclosure includes steps S101 to S103, wherein step S102: determining the correspondence between vehicle mileage and the number of driving scenarios, and the relationship between the number of driving scenarios and the scenario coverage of the driving scenario set, based on historical road test data, may include:

[0083] Based on the road test data collected in the first time period, the correspondence between vehicle mileage and the number of first driving scenarios was determined.

[0084] Based on the relationship between the number of first driving scenarios and the number of multiple first driving scenarios in the driving scenario set, the relationship between the number of first driving scenarios and the scenario coverage of the driving scenario set is determined.

[0085] According to embodiments of this disclosure, by utilizing road test data collected in the first time period for constructing a driving scenario dataset, it is possible to quickly and accurately determine the correspondence between vehicle mileage and the number of driving scenarios, as well as the relationship between the number of driving scenarios and the scenario coverage of the driving scenario set.

[0086] In one embodiment, the method for determining road test mileage according to this disclosure includes steps S101 to S103, wherein step S103: based on a preset scene coverage rate, determining the baseline road test mileage of the road network area according to the correspondence relationship and the scene coverage rate relationship, may include:

[0087] Step S1031: Based on the correspondence and scene coverage relationship, construct the functional relationship between vehicle mileage and scene coverage.

[0088] Step S1032: Determine the scene coverage relationship in the function relationship that matches the preset scene coverage.

[0089] Step S1033: Based on the function relationship, determine the vehicle mileage corresponding to the scene coverage relationship that matches the preset scene coverage as the benchmark road test mileage of the road network area.

[0090] Based on the above embodiments of this disclosure, it should be noted that:

[0091] The functional relationship can be understood as a deterministic relationship between vehicle mileage and scene coverage.

[0092] According to embodiments of this disclosure, the baseline road test mileage for a road network area can be quickly and accurately determined by utilizing the functional relationship between vehicle mileage and scene coverage.

[0093] In one example, such as Figure 2 As shown, the ratio of the number of deduplicated driving scenarios P ​​mined from the acceptance data (i.e., the road test data collected in the second time period) to the number of driving scenarios Q in the driving scenario set is defined as the scenario coverage rate f, which is used as the vertical axis. The acceptance mileage L (obtained based on the road test data collected in the second time period) is used as the horizontal axis to observe the relationship between the scenario coverage rate f and the acceptance mileage L. After obtaining the fL relationship, a convex curve is presented. At the beginning of the test, the scenario coverage rate increases significantly with the cumulative increase of driving mileage. However, in the later stage, the increase in scenario coverage rate becomes very insignificant, indicating that the test efficiency is decreasing. In order to find a balance between test sufficiency and test efficiency, the mileage corresponding to the preset scenario coverage rate is generally selected as the reference mileage of the baseline road test mileage.

[0094] Specifically, such as Figure 2As shown, with a set of 100 driving scenarios, it was found that when the vehicle traveled 500 kilometers, 95 driving scenarios were encountered in the road network area, representing a scenario coverage of 95%. When the vehicle traveled 1000 kilometers, 97 driving scenarios were encountered in the road network area, representing a scenario coverage of 97%. Since the vehicle had to travel an additional 500 kilometers to encounter an additional 2% of driving scenarios, the testing efficiency was lower compared to encountering 95% of driving scenarios after only 500 kilometers. To find a balance between testing sufficiency and efficiency, the mileage corresponding to a preset scenario coverage of 95% was selected as the initial road test mileage for the baseline road test.

[0095] In one embodiment, the method for determining road test mileage according to this disclosure includes steps S101 to S103 and steps S1031 to S1033, wherein step S1033: based on a functional relationship, the vehicle mileage corresponding to the scene coverage relationship that matches the preset scene coverage rate is determined as the benchmark road test mileage for the road network area, including:

[0096] Based on the functional relationship, the vehicle mileage corresponding to the scene coverage relationship that matches the preset scene coverage rate is determined as the initial road test mileage.

[0097] The initial road test mileage is adjusted using a mileage correction factor to obtain the baseline road test mileage for the road network area.

[0098] Based on the above embodiments of this disclosure, it should be noted that:

[0099] The mileage correction factor can be determined and adjusted as needed. To avoid unexpected situations during road network testing and to ensure that the vehicle can encounter all driving scenarios corresponding to the initial road test mileage during road testing, a mileage correction factor is set to extend the initial road test mileage.

[0100] According to embodiments of this disclosure, by expanding the initial road test mileage through a mileage correction factor, it can be ensured that the vehicle can encounter various driving scenarios with scene coverage corresponding to the initial road test mileage during road testing.

[0101] In one example, to include scenarios with a certain probability of occurrence, the mileage correction factor is set to 3. Based on the initial road test mileage Y, the obtained baseline road test mileage is 3*Y, that is, the baseline road test mileage is 3 times the initial road test mileage.

[0102] In one embodiment, the method for determining road test mileage according to this disclosure includes steps S101 to S103, and may further include:

[0103] Based on the driving scenarios and driving scenario set corresponding to the baseline road test mileage, identify the driving scenarios in the driving scenario set that are not covered by the baseline road test mileage.

[0104] Uncovered driving scenarios are identified as target driving scenarios for the road network area.

[0105] Based on the above embodiments of this disclosure, it should be noted that:

[0106] If the driving scenarios corresponding to the baseline road test mileage cover 95% of the driving scenario set, then there are 5% of driving scenarios that are not covered. These 5% of driving scenarios are the target driving scenarios. These target driving scenarios can be tested separately, and can be supplemented by road test data from closed courses, simulation tests, or other sites to identify and fill gaps, ensuring that the testing is sufficient and that these 5% of safety risk driving scenarios are not missed.

[0107] According to the embodiments of this disclosure, target driving scenarios not covered by the benchmark road test mileage in the road network area can be accurately determined, so that vehicles can be tested separately, and gaps can be filled by relying on closed track, simulation test or other site road test data to ensure that the test is sufficient and that these full-risk driving scenarios are not missed.

[0108] This disclosure provides a method for evaluating the performance of autonomous driving, such as... Figure 3 As shown, the method includes:

[0109] Step S301: Determine the road test mileage of the target test vehicle based on the autonomous driving road test data of the target test vehicle in the road network area.

[0110] Step S302: Determine the baseline road test mileage for the road network area using the road test mileage determination method.

[0111] Step S303: Under the condition that the road test mileage is not less than the benchmark road test mileage, evaluate whether the test adequacy of the autonomous driving performance of the target test vehicle meets the requirements.

[0112] Based on the above embodiments of this disclosure, it should be noted that:

[0113] If the road test mileage is less than the baseline road test mileage, it indicates that the testing of the autonomous driving performance of the target test vehicle is insufficient and does not cover the driving scenarios of the road network area covered by the baseline road test mileage. The target test vehicle needs to continue to conduct autonomous driving performance testing in the road network area.

[0114] The method for determining the road test mileage can be any of the methods for determining the road test mileage described in the above embodiments of this disclosure.

[0115] According to embodiments of this disclosure, by utilizing the benchmark road test mileage in the road network area, the adequacy of testing of the target test vehicle in the road network area can be quantitatively evaluated, providing a strong basis for judgment on the testing and delivery of autonomous vehicles.

[0116] In one embodiment, the autonomous driving performance evaluation method provided in this disclosure includes steps S301 to S303, and further includes the step:

[0117] If the autonomous driving road test sufficiency of the target test vehicle meets the requirements, supplementary test instructions are generated according to the target driving scenario in the road network area. The supplementary test instructions are used to conduct supplementary tests on the autonomous driving performance of the target test vehicle according to the target driving scenario.

[0118] According to the embodiments of this disclosure, the target test vehicle can conduct separate specialized tests on target driving scenarios in road network areas not covered by the benchmark road test mileage. It can also use closed-course, simulation tests, or other site road test data to identify and fill gaps, ensuring that the tests are sufficient and do not miss any of these full-risk driving scenarios.

[0119] In one embodiment, the method for determining road test mileage and / or the method for evaluating autonomous driving performance provided by the embodiments of this disclosure can be applied to, for example... Figure 4 Within the scene framework shown. Figure 4 In this document, 10 represents a terminal, 20 represents a server, 30 represents a distributed computer system, and 40 represents historical road test data obtained from multiple vehicles in a target area. The method for determining road test mileage and / or the method for evaluating autonomous driving performance disclosed herein can be executed by server 20 or distributed computer system 30. Terminal 10 is used to report / send historical road test data to server 20 or distributed computer system 30. After server 20 or distributed computer system 30 completes the method for determining road test mileage and / or the method for evaluating autonomous driving performance disclosed herein, it can feed the results back to terminal 10.

[0120] This disclosure provides a device for determining road test mileage, such as... Figure 5 As shown, the device includes:

[0121] The first determining module 510 is used to determine the set of driving scenarios in the road network area based on historical road test data of the road network area.

[0122] The second determining module 520 is used to determine the correspondence between vehicle mileage and the number of driving scenarios, as well as the relationship between the number of driving scenarios and the scenario coverage of the driving scenario set, based on historical road test data.

[0123] as well as

[0124] The third determination module 530 is used to determine the benchmark road test mileage of the road network area based on the preset scene coverage rate and the correspondence and scene coverage relationship.

[0125] In one implementation, the first determining module 510 includes:

[0126] The first acquisition submodule is used to acquire the road test data collected in the first time period based on the historical road test data of the road network area.

[0127] The first determination submodule is used to determine multiple first driving scenarios contained in the road network area based on the road test data collected in the first time period, according to the preset scenario mining rules.

[0128] The second determining submodule is used to determine the set of driving scenarios in the road network area based on multiple first driving scenarios.

[0129] In one implementation, the first acquisition submodule is used to:

[0130] Based on historical road test data of the road network area, obtain the first driving data generated by multiple vehicles in the first time period.

[0131] The data containing weather information, traffic environment information, and driving interaction information in the first driving data is identified as the road test data collected in the first time period.

[0132] In one implementation, the second determining module 520 is used to:

[0133] Based on historical road test data, obtain the road test data collected in the second time period.

[0134] Based on the road test data collected in the second time period, which includes weather information, traffic environment information, and driving interaction information, the number of second driving scenarios corresponding to the road test data collected in the second time period is determined.

[0135] Based on the road test data collected in the second time period, the correspondence between vehicle mileage and the number of second driving scenarios was determined.

[0136] Based on the relationship between the number of second driving scenarios and the number of multiple first driving scenarios in the driving scenario set, the relationship between the number of second driving scenarios and the scenario coverage of the driving scenario set is determined.

[0137] In one implementation, the second determining module 520 is used to:

[0138] Based on the road test data collected in the first time period, the correspondence between vehicle mileage and the number of first driving scenarios was determined.

[0139] Based on the relationship between the number of first driving scenarios and the number of multiple first driving scenarios in the driving scenario set, the relationship between the number of first driving scenarios and the scenario coverage of the driving scenario set is determined.

[0140] In one implementation, the third determining module 530 includes:

[0141] A submodule is built to construct a functional relationship between vehicle mileage and scene coverage based on the correspondence and scene coverage relationships.

[0142] The third determination submodule is used to determine the scene coverage relationship in the function relationship that matches the preset scene coverage.

[0143] The fourth determination submodule is used to determine the vehicle mileage corresponding to the scene coverage relationship that matches the preset scene coverage rate as the benchmark road test mileage of the road network area based on the function relationship.

[0144] In one implementation, the fourth determining submodule is used to:

[0145] Based on the functional relationship, the vehicle mileage corresponding to the scene coverage relationship that matches the preset scene coverage rate is determined as the initial road test mileage.

[0146] The initial road test mileage is adjusted using a mileage correction factor to obtain the baseline road test mileage for the road network area.

[0147] In one embodiment, the road test mileage determination device further includes:

[0148] The fourth determination module is used to determine the driving scenarios not covered by the benchmark road test mileage in the driving scenario set, based on the driving scenarios corresponding to the benchmark road test mileage and the driving scenario set.

[0149] The fifth determination module is used to identify uncovered driving scenarios as target driving scenarios within the road network area.

[0150] The specific functions and examples of each module and submodule of the apparatus in this disclosure can be found in the relevant descriptions of the corresponding steps in the above method embodiments, and will not be repeated here.

[0151] This disclosure provides an evaluation device for autonomous driving performance, such as... Figure 6 As shown, the device includes:

[0152] The sixth determining module 610 is used to determine the road test mileage of the target test vehicle based on the autonomous driving road test data of the target test vehicle in the road network area.

[0153] The seventh determining module 620 is used to determine the reference road test mileage of the road network area using the road test mileage determination method of any embodiment of the present disclosure.

[0154] Evaluation module 630 is used to evaluate the adequacy of the test requirements for the autonomous driving performance of the target test vehicle, provided that the road test mileage is not less than the benchmark road test mileage.

[0155] In one embodiment, the autonomous driving performance evaluation device further includes:

[0156] The instruction module is used to generate supplementary test instructions based on the target driving scenario in the road network area, provided that the autonomous driving road test sufficiency of the target test vehicle meets the requirements. The supplementary test instructions are used to conduct supplementary tests on the autonomous driving performance of the target test vehicle based on the target driving scenario.

[0157] The specific functions and examples of each module and submodule of the apparatus in this disclosure can be found in the relevant descriptions of the corresponding steps in the above method embodiments, and will not be repeated here.

[0158] The acquisition, storage, and application of user personal information involved in the technical solution disclosed herein comply with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0159] According to embodiments of this disclosure, this disclosure also provides an electronic device, a readable storage medium, and a computer program product.

[0160] Figure 7 A schematic block diagram of an example electronic device 700 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0161] like Figure 7As shown, device 700 includes a computing unit 701, which can perform various appropriate actions and processes based on a computer program stored in read-only memory (ROM) 702 or a computer program loaded from storage unit 708 into random access memory (RAM) 703. RAM 703 may also store various programs and data required for the operation of device 700. The computing unit 701, ROM 702, and RAM 703 are interconnected via bus 704. Input / output (I / O) interface 705 is also connected to bus 704.

[0162] Multiple components in device 700 are connected to I / O interface 705, including: input unit 706, such as keyboard, mouse, etc.; output unit 707, such as various types of monitors, speakers, etc.; storage unit 708, such as disk, optical disk, etc.; and communication unit 709, such as network card, modem, wireless transceiver, etc. Communication unit 709 allows device 700 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0163] The computing unit 701 can be various general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 701 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 701 performs the various methods and processes described above, such as methods for determining road test mileage and / or methods for evaluating autonomous driving performance. For example, in some embodiments, the methods for determining road test mileage and / or evaluating autonomous driving performance can be implemented as computer software programs tangibly contained in a machine-readable medium, such as storage unit 708. In some embodiments, part or all of the computer program can be loaded and / or installed on device 700 via ROM 702 and / or communication unit 709. When the computer program is loaded into RAM 703 and executed by the computing unit 701, one or more steps of the methods for determining road test mileage and / or evaluating autonomous driving performance described above can be performed. Alternatively, in other embodiments, the computing unit 701 may be configured by any other suitable means (e.g., by means of firmware) to perform methods for determining road test mileage and / or evaluating autonomous driving performance.

[0164] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0165] The program code used to implement the methods of this disclosure may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0166] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0167] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0168] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as a data server), or computing systems that include middleware components (e.g., an application server), or computing systems that include frontend components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., a communication network). Examples of communication networks include local area networks (LANs), wide area networks (WANs), and the Internet.

[0169] Computer systems can include clients and servers. Clients and servers are generally located far apart and typically interact via communication networks. Client-server relationships are created by computer programs running on the respective computers and having a client-server relationship with each other. Servers can be cloud servers, servers in distributed systems, or servers incorporating blockchain technology.

[0170] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this disclosure can be achieved, and this is not limited herein.

[0171] The specific embodiments described above do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the principles of this disclosure should be included within the scope of protection of this disclosure.

Claims

1. A method for determining a road test mileage, comprising: determining a driving scene set of a road network region according to historical road test data of the road network region; determining a correspondence between a vehicle driving mileage and a driving scene quantity and a scene coverage rate relationship between the driving scene quantity and the driving scene set according to the historical road test data; and determining a reference road test mileage of the road network region according to the correspondence and the scene coverage rate relationship based on a preset scene coverage rate. The determining of the driving scene set of the road network region according to the historical road test data of the road network region comprises: obtaining road test data collected in a first time period from the historical road test data of the road network region; determining a plurality of first driving scenes contained in the road network region based on the road test data collected in the first time period according to a preset scene mining rule; and determining the driving scene set of the road network region according to the plurality of first driving scenes. The determining of the reference road test mileage of the road network region according to the correspondence and the scene coverage rate relationship based on the preset scene coverage rate comprises: constructing a functional relationship between the vehicle driving mileage and the scene coverage rate relationship according to the correspondence and the scene coverage rate relationship; determining a scene coverage rate relationship matching the preset scene coverage rate in the functional relationship; and determining a vehicle driving mileage corresponding to the scene coverage rate relationship matching the preset scene coverage rate as the reference road test mileage of the road network region according to the functional relationship. The obtaining of the road test data collected in the first time period from the historical road test data of the road network region comprises: obtaining first driving data generated by a plurality of vehicles in the road network region in a first time period from the historical road test data of the road network region; and determining data containing weather information, traffic environment information and driving interaction information in the first driving data as the road test data collected in the first time period. The determining of the correspondence between the vehicle driving mileage and the driving scene quantity and the scene coverage rate relationship between the driving scene quantity and the driving scene set according to the historical road test data comprises: obtaining road test data collected in a second time period from the historical road test data; determining a second driving scene quantity corresponding to the road test data collected in the second time period based on data containing weather information, traffic environment information and driving interaction information in the road test data collected in the second time period; determining a correspondence between a vehicle driving mileage and the second driving scene quantity according to the road test data collected in the second time period; and determining a scene coverage rate relationship between the second driving scene quantity and the driving scene set according to a quantity relationship between the second driving scene quantity and the plurality of first driving scenes in the driving scene set. The determining of the correspondence between the vehicle driving mileage and the driving scene quantity and the scene coverage rate relationship between the driving scene quantity and the driving scene set according to the historical road test data comprises: determining a correspondence between a vehicle driving mileage and a first driving scene quantity according to the road test data collected in the first time period; and determining a scene coverage rate relationship between the first driving scene quantity and the driving scene set according to a quantity relationship between the first driving scene quantity and the plurality of first driving scenes in the driving scene set. ​ ​ ​ ​ ​ ​ ​ 2. The method of claim 1, wherein, ​ ​ ​ 3. The method of claim 1, wherein, ​ ​ ​ ​ ​ 4. The method of claim 1 or 2, wherein, ​ ​ According to the quantity relationship between the first driving scene quantity and the quantity of the plurality of first driving scenes in the driving scene set, a scene coverage rate relationship between the first driving scene quantity and the driving scene set is determined.

5. The method of claim 1, wherein, The function relationship between the vehicle driving mileage and the scene coverage rate relationship is determined according to the preset scene coverage rate. According to the function relationship, the vehicle driving mileage corresponding to the scene coverage rate relationship matching the preset scene coverage rate is determined as the initial road test mileage. The initial road test mileage is adjusted by using a mileage correction coefficient to obtain the reference road test mileage of the road network region.

6. The method of claim 1, further comprising: determining, according to the driving scene corresponding to the reference road test mileage and the driving scene set, a driving scene in the driving scene set that is not covered by the reference road test mileage; determining the uncovered driving scene as a target driving scene of the road network region.

7. An automatic driving performance evaluation method, comprising: determining a road test driving mileage of a target test vehicle according to automatic driving road test data of the target test vehicle in a road network region; determining a reference road test mileage of the road network region by using the method of any one of claims 1 to 6; and in a case where the road test driving mileage is not less than the reference road test mileage, evaluating that a test sufficiency of the target test vehicle in automatic driving performance meets a requirement.

8. The method of claim 7, further comprising: in a case where the automatic driving road test sufficiency of the target test vehicle meets a requirement, generating a supplementary test instruction according to a target driving scene of the road network region, the supplementary test instruction being used to cause the target test vehicle to perform a supplementary test on automatic driving performance according to the target driving scene.

9. A road test mileage determination apparatus, comprising: a first determination module configured to determine a driving scene set of a road network region according to historical road test data of the road network region; a second determination module configured to determine a corresponding relationship between a vehicle driving mileage and a driving scene quantity and a scene coverage rate relationship between the driving scene quantity and the driving scene set according to the historical road test data; and a third determination module configured to determine a reference road test mileage of the road network region based on a preset scene coverage rate according to the corresponding relationship and the scene coverage rate relationship. The first determination module comprises: a first acquisition sub-module configured to acquire road test data collected in a first time period from the historical road test data of the road network region; a first determination sub-module configured to determine a plurality of first driving scenes contained in the road network region based on the road test data collected in the first time period according to a preset scene mining rule; a second determination sub-module configured to determine a driving scene set of the road network region according to the plurality of first driving scenes; The third determination module comprises: a construction sub-module configured to construct a function relationship between the vehicle driving mileage and the scene coverage rate relationship according to the corresponding relationship and the scene coverage rate relationship. ​ ​ a third determining sub-module, configured to determine a scenario coverage rate relationship in the function relationship that matches a preset scenario coverage rate; a fourth determining sub-module, configured to determine, according to the function relationship, a vehicle driving mileage corresponding to the scenario coverage rate relationship that matches the preset scenario coverage rate, as a reference road test mileage of the road network region.

10. The apparatus of claim 9, wherein, The first obtaining sub-module is configured to: obtain, according to historical road test data of a road network region, first driving data generated by a plurality of vehicles in the road network region in a first time period; determine, as road test data collected in the first time period, data containing weather information, traffic environment information and driving interaction information in the first driving data.

11. The apparatus of claim 9, wherein, The second determining module is configured to: obtain, according to the historical road test data, road test data collected in a second time period; determine, based on data containing weather information, traffic environment information and driving interaction information in the road test data collected in the second time period, a second driving scenario quantity corresponding to the road test data collected in the second time period; determine, according to the road test data collected in the second time period, a corresponding relationship between a vehicle driving mileage and the second driving scenario quantity; and determine, according to a quantity relationship between the second driving scenario quantity and the plurality of first driving scenarios in the driving scenario set, a scenario coverage rate relationship between the second driving scenario quantity and the driving scenario set. The second determining module is configured to:

12. The apparatus of claim 9 or 10, wherein, determine, according to the road test data collected in the first time period, a corresponding relationship between a vehicle driving mileage and a first driving scenario quantity; and determine, according to a quantity relationship between the first driving scenario quantity and the plurality of first driving scenarios in the driving scenario set, a scenario coverage rate relationship between the first driving scenario quantity and the driving scenario set. The fourth determining sub-module is configured to:

13. The apparatus of claim 9, wherein, determine, according to the function relationship, a vehicle driving mileage corresponding to the scenario coverage rate relationship that matches the preset scenario coverage rate, as an initial road test mileage; and adjust the initial road test mileage by using a mileage correction coefficient to obtain the reference road test mileage of the road network region.

14. The apparatus according to claim 9, further comprising: a fourth determining module, configured to determine, according to a driving scenario corresponding to the reference road test mileage and the driving scenario set, a driving scenario in the driving scenario set that is not covered by the reference road test mileage; a fifth determining module, configured to determine the uncovered driving scenario as a target driving scenario of the road network region.

15. An apparatus for evaluating automatic driving performance, comprising: a sixth determining module, configured to determine, according to automatic driving road test data of a target test vehicle in a road network region, a road test driving mileage of the target test vehicle; a seventh determining module, configured to determine, by using the method according to any one of claims 1 to 6, a reference road test mileage of the road network region; an evaluation module, configured to evaluate, in a case where the road test driving mileage is not less than the reference road test mileage, that a test sufficiency of the automatic driving performance of the target test vehicle meets a requirement.

16. The apparatus according to claim 15, further comprising: ​ ​ The instruction module is configured to generate a supplementary test instruction for causing the target test vehicle to perform a supplementary test on the automatic driving performance according to the target driving scene, if the automatic driving road test sufficiency of the target test vehicle meets the requirement. 17.An electronic device, comprising: at least one processor; and a memory connected to the at least one processor in communication; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1 to 8.

18. A non-transitory computer readable storage medium having stored thereon computer instructions, wherein, The computer instructions are configured to cause the computer to perform the method of any one of claims 1 to 8. 19.A computer program product comprising a computer program which, when executed by a processor, implements the method of any one of claims 1 to 8.

Images