Method and device for determining coverage degree of automatic driving test data

Abstract

The invention provides a method and a device for determining the coverage degree of automatic driving test data. The method comprises the following steps: acquiring a road acquisition data set of a target area; performing scene element analysis on the road acquisition data set, and determining a target scene element analysis result including scene element types related to the automatic driving test, the occurrence frequency of each scene element type and the total number of samples; according to the target scene element analysis result, determining a coverage estimation value, and according to the determined coverage estimation value, determining a confidence interval of the coverage estimation value under a specified confidence coefficient; and determining an evaluation result of the automatic driving test data coverage according to the determined confidence interval. Therefore, the scene coverage degree is estimated according to the analysis result of each scene element in the road acquisition data set, and the confidence interval under the specified confidence degree is determined according to the estimated scene coverage degree, so that the situation that no scene appears in the road acquisition data can be estimated, and the coverage degree of the automatic driving test data is further evaluated.

Description

Technical Field

[0001] This application relates to the field of autonomous driving technology, and in particular to a method and apparatus for determining the coverage of autonomous driving test data. Background Technology

[0002] With the rapid development of autonomous driving technology, testing and verifying the safety and reliability of algorithms has become a core challenge in ensuring the safe deployment of autonomous driving. A key challenge in testing and verifying autonomous driving systems is how to handle unknown scenarios or how to evaluate the weight of unknown scenarios to better analyze the safety of the autonomous driving system.

[0003] However, existing commonly used coverage evaluation schemes include: First, evaluating the coverage of the test set for combined scenarios, where each type of element in a scenario has several possible values, and a scenario is formed by combinations of different values ​​of different elements. This method cannot evaluate completely unknown scenarios, i.e., it cannot evaluate participants / events / elements that have never been seen before. Second, evaluating the coverage of logical scenarios through generalization, but since generalization cannot generalize to unknown elements, this type of evaluation method is obviously different from our proposed requirements, i.e., this method cannot evaluate completely unknown scenarios. Summary of the Invention

[0004] In view of this, the purpose of this application is to provide a method and apparatus for determining the coverage of autonomous driving test data. By analyzing the analysis results of various scene elements in the road-collected data, the scene coverage is estimated, and a confidence interval at a specified confidence level is determined based on the estimated scene coverage. Thus, the scenes that do not appear in the road-collected data can be estimated based on the determined confidence interval, thereby determining the evaluation result of the coverage of autonomous driving test data.

[0005] This application provides a method for determining the coverage of autonomous driving test data, the method comprising:

[0006] Obtain the road survey dataset for the target area;

[0007] Scene element analysis is performed on the road sampling dataset to determine the target scene element analysis results in the road sampling dataset; wherein, the target scene element analysis results include the scene element types related to autonomous driving testing, the number of times each scene element type appears, and the total number of samples;

[0008] Based on the analysis results of the target scene elements, scene coverage is estimated, a coverage estimate is determined, and a confidence interval for the coverage estimate at a specified confidence level is determined based on the determined coverage estimate.

[0009] Based on the determined confidence interval, the evaluation results of the autonomous driving test data coverage are determined.

[0010] Optionally, the step of estimating scene coverage based on the analysis results of the target scene elements, determining the coverage estimate, and determining the confidence interval of the coverage estimate at a specified confidence level based on the determined coverage estimate includes:

[0011] The coverage estimate is determined based on the total number of samples and the number of scene element types that appear a preset number of times.

[0012] Determine the upper bound of the sample variance based on the total sample size;

[0013] Substitute the aforementioned upper bound of variance into Chebyshev's inequality to calculate the significance level;

[0014] Based on the significance level and the coverage estimate, determine the confidence interval of the coverage estimate at a specified confidence level.

[0015] Optionally, the coverage estimate can be determined using the following formula, based on the total number of samples and the number of scene element types that appear a preset number of times:

[0016]

[0017] in, This is the coverage estimate, where n is the total number of samples, and M is the total number of samples. x,n The number of scene element types that appear a preset number of times, where x is the preset number of times.

[0018] Optionally, the step of estimating scene coverage based on the analysis results of the target scene elements, determining a coverage estimate, and determining a confidence interval for the coverage estimate at a specified confidence level includes:

[0019] Based on the analysis results of the target scene elements, the corresponding Bayesian nonparametric inference method is used to determine the first hyperparameter of the Poisson-Dirichlet process and the second hyperparameter of the generalized standard gamma process, respectively.

[0020] Based on the first hyperparameter and the analysis results of the target scene elements, the first likelihood value corresponding to the Poisson-Dirichlet process is determined, and based on the second hyperparameter and the analysis results of the target scene elements, the second likelihood value corresponding to the generalized standard gamma process is determined.

[0021] The target prior value is determined from the first likelihood value and the second likelihood value;

[0022] Based on the target reasoning process corresponding to the target prior value, the hyperparameters corresponding to the target reasoning process are used and combined with the target scene element analysis results to perform coverage estimation and determine the coverage estimate value corresponding to the target reasoning process.

[0023] Based on the target reasoning process, determine the quantile corresponding to the target reasoning process;

[0024] Based on the quantiles corresponding to the target reasoning process and the corresponding coverage estimate, determine the confidence interval of the coverage estimate at a specified confidence level.

[0025] Optionally, determining the target prior value from the first likelihood value and the second likelihood value includes:

[0026] The maximum value between the first likelihood value and the second likelihood value is determined as the target prior value.

[0027] Optionally, determining the quantile corresponding to the target reasoning process based on the target reasoning process includes:

[0028] When the target reasoning process is a Poisson-Dirichlet process, the quantiles corresponding to the target reasoning process are determined according to the beta distribution;

[0029] When the target reasoning process is a generalized standard gamma process, Monte Carlo sampling is used to determine the quantile corresponding to the target reasoning process.

[0030] Optionally, the step of determining the coverage estimate value corresponding to the target inference process based on the target prior value, using the hyperparameters corresponding to the target inference process and combining the target scene element analysis results to perform coverage estimation, includes:

[0031] When the target inference process is a Poisson-Dirichlet process, the coverage estimate is calculated using the following formula:

[0032]

[0033] in, This is a coverage estimate. These are estimates of the lumped parameters in the first hyperparameter set. Here, K represents the estimated value of the discount parameter in the first hyperparameter set, where i is the i-th sample, n is the total number of samples, and K is the value of the discount parameter. n The number of different scene element types in n samples;

[0034] When the target inference process is a generalized standard gamma process, the coverage estimate is calculated using the following formula:

[0035]

[0036] in, This is an estimate of the steady-state exponent in the second hyperparameter. Here, represents the estimated scaling parameter in the second hyperparameter set, where i is the i-th sample, n is the total number of samples, and K is the value of the scale parameter. n This represents the number of different scene element types in the n samples.

[0037] This application embodiment also provides a device for determining the coverage of autonomous driving test data, the device comprising:

[0038] The acquisition module is used to acquire road sampling datasets for the target area;

[0039] The analysis module is used to perform scene element analysis on the road sampling dataset and determine the target scene element analysis results in the road sampling dataset; wherein, the target scene element analysis results include the scene element types related to autonomous driving testing, the number of times each scene element type appears, and the total number of samples;

[0040] The estimation module is used to estimate the scene coverage based on the analysis results of the target scene elements, determine the estimated coverage value, and determine the confidence interval of the estimated coverage value at a specified confidence level based on the determined estimated coverage value.

[0041] The determination module is used to determine the evaluation results of autonomous driving test data coverage based on the determined confidence interval.

[0042] This application also provides an electronic device, including: a processor, a memory, and a bus. The memory stores machine-readable instructions executable by the processor. When the electronic device is running, the processor communicates with the memory via the bus. When the machine-readable instructions are executed by the processor, the steps of the determination method described above are performed.

[0043] This application also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, performs the steps of the determination method described above.

[0044] This application provides a method and apparatus for determining the coverage of autonomous driving test data. The method includes: acquiring a road sampling dataset of a target area; performing scene element analysis on the road sampling dataset to determine the target scene element analysis results in the road sampling dataset; wherein the target scene element analysis results include scene element types related to autonomous driving testing, the frequency of occurrence of each scene element type, and the total number of samples; estimating scene coverage based on the target scene element analysis results to determine a coverage estimate, and determining a confidence interval for the coverage estimate at a specified confidence level based on the determined coverage estimate; and determining an evaluation result of the autonomous driving test data coverage based on the determined confidence interval.

[0045] Thus, this application uses the analysis results of various scene elements in the road-collected data to estimate scene coverage using multiple inference methods, and determines a confidence interval at a specified confidence level based on the estimated scene coverage. This allows for the estimation of scenes not appearing in the road-collected data based on the determined confidence interval, thereby determining the evaluation result of the autonomous driving test data coverage. Therefore, the method provided in this application can estimate the probability of scene elements not appearing in the road-collected data in an autonomous driving scenario, thereby inferring the probability of unknown scenarios, and estimating the safety of the tested autonomous driving algorithm based on the estimated probability of unknown scenarios.

[0046] To make the above-mentioned objectives, features and advantages of this application more apparent and understandable, preferred embodiments are described below in detail with reference to the accompanying drawings. Attached Figure Description

[0047] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of this application and should not be regarded as a limitation of the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0048] Figure 1 A flowchart illustrating a method for determining autonomous driving test data coverage as provided in an embodiment of this application;

[0049] Figure 2 A schematic diagram illustrating the process of a method for determining the confidence interval of an estimated scene coverage at a specified confidence level, as provided in this application;

[0050] Figure 3 A schematic diagram of the structure of an autonomous driving test data coverage determination device provided in an embodiment of this application;

[0051] Figure 4 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0052] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. The components of the embodiments of this application described and shown in the accompanying drawings can generally be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of this application provided in the accompanying drawings is not intended to limit the scope of the claimed application, but merely represents selected embodiments of this application. Based on the embodiments of this application, every other embodiment obtained by those skilled in the art without inventive effort falls within the scope of protection of this application.

[0053] With the rapid development of autonomous driving technology, testing and verifying the safety and reliability of algorithms has become a core challenge in ensuring the safe deployment of autonomous driving. A key challenge in testing and verifying autonomous driving systems is how to handle unknown scenarios or how to evaluate the weight of unknown scenarios to better analyze the safety of the autonomous driving system.

[0054] However, existing commonly used coverage evaluation schemes include: First, evaluating the coverage of the test set for combined scenarios, where each element in a scenario has several possible values, and a scenario is formed by combinations of different values ​​of different elements. This method cannot evaluate completely unknown scenarios, i.e., it cannot evaluate participants / events / elements that have never been seen before. Second, using Bayesian methods for evaluation. This method estimates coverage by verifying the number of new scenarios / participants / elements / events over a sufficiently long distance or time (usually very, very long) for an existing test set. However, this method generally assumes a uniform distribution of coverage in the 0-1 interval, so it is not accurate for evaluating unknown scenarios. Third, evaluating the coverage of logical scenarios by generalizing scenarios, but since generalization cannot generalize to unknown elements, this type of evaluation method is obviously different from our proposed requirements, i.e., this method cannot evaluate completely unknown scenarios.

[0055] Furthermore, it should be noted that in verifying the safety of autonomous driving algorithms / systems, a core challenge lies in evaluating unknown scenarios or unknown participants / behaviors / elements, given the impossibility of traversing an infinite number of real-world scenarios / elements / events / participants. Since we cannot verify the performance of autonomous driving systems in unknown scenarios (because it is unknown, neither simulation nor real-world scenarios can verify it), a fallback approach is to study the probability of unknown scenarios / events / elements / participants themselves, which can also help verify the safety of autonomous driving (because the probability of verifying the unknown itself needs to be sufficiently low).

[0056] Based on this, embodiments of this application provide a method and apparatus for determining the coverage of autonomous driving test data. By analyzing the analysis results of various scene elements in the road-collected data, the scene coverage is estimated, and a confidence interval at a specified confidence level is determined based on the estimated scene coverage. Thus, the scenes that do not appear in the road-collected data can be estimated based on the determined confidence interval, thereby determining the evaluation result of the coverage of autonomous driving test data.

[0057] Please see Figure 1 , Figure 1 This is a flowchart illustrating a method for determining autonomous driving test data coverage as provided in an embodiment of this application. Figure 1 As shown in the embodiments of this application, the determination method includes:

[0058] S101. Obtain the road survey dataset for the target area.

[0059] S102. Perform scene element analysis on the road survey dataset to determine the target scene element analysis results in the road survey dataset.

[0060] S103. Based on the analysis results of the target scene elements, perform scene coverage estimation, determine the coverage estimate value, and determine the confidence interval at the specified confidence level of the coverage estimate value.

[0061] S104. Based on the determined confidence interval, determine the evaluation result of the autonomous driving test data coverage.

[0062] The exemplary steps of the embodiments of this application are described below:

[0063] Regarding step S101, the road sampling dataset is a dataset composed of random road sampling data obtained from the target area, which can be obtained through simple random sampling.

[0064] For example, if you want to evaluate the coverage of a dataset to a certain city, the dataset should be a random sample taken from that city, that is, it should include various different types of working conditions, weather and other scene elements, and should not simply use data from a single road.

[0065] Regarding step S102, the target scene element analysis results include the scene element types related to autonomous driving testing, the number of times each scene element type appears, and the total number of samples.

[0066] Here, when performing scene element analysis, the types of scene elements to be divided can be determined in advance. For example, scene element types may include: different scenes, different events, different participants, different elements, etc.

[0067] The results of the target scene element analysis in the determined road sampling dataset can be represented by the following symbols.

[0068] Let X1, X2, ..., X n The samples obtained from the road survey dataset (which can be scenes, events, traffic participants, or elements, etc., depending on the requirements); K n This represents the number of different scene element types in the above n samples (for example, if we want to consider traffic participants, then X1, X2, ..., X...). n (Indicates encountering traffic participants); use N j,n M represents the number of samples belonging to the j-th class among the above n samples; r,n M represents the number of scene element types that appear exactly r times in the above n samples (for example, if only regular bicycles and regular taxis appear 100 times in the data above, then M). 100,n =2).

[0069] Regarding step S103, this application provides various methods for estimating scene coverage, as well as technical means for determining the corresponding confidence intervals for these methods.

[0070] For an example, please refer to Figure 2 , Figure 2 A schematic diagram illustrating a method for determining the confidence interval of estimated scene coverage at a specified confidence level, as provided in this application, is shown below. Figure 2 As shown, the step of estimating scene coverage based on the analysis results of the target scene elements, determining the coverage estimate, and determining the confidence interval of the coverage estimate at a specified confidence level based on the determined coverage estimate includes:

[0071] S10311. Determine the coverage estimate based on the total number of samples and the number of scene element types that appear a preset number of times.

[0072] S10312. Determine the upper bound of the sample variance based on the total sample size.

[0073] S10313. Substitute the upper bound of the variance into the Chebyshev inequality to calculate the significance level.

[0074] S10314. Based on the significance level and the coverage estimate, determine the confidence interval of the coverage estimate at a specified confidence level.

[0075] Among them, steps S10311-S10314 determine the coverage estimate of the autonomous driving test scenario in this application and the confidence interval in which it is located based on the estimation method of Good-Turing statistics.

[0076] Regarding step S10311, in one embodiment provided in this application, the coverage estimate is determined using the following formula, based on the total number of samples and the number of scene element types that appear a preset number of times:

[0077]

[0078] in, This is the coverage estimate, where n is the total number of samples, and M is the total number of samples. x,n The number of scene element types that appear a preset number of times, where x is the preset number of times.

[0079] In this step, x can be selected once.

[0080] The basic idea of ​​this method is that when the sample size is large enough, the selection of a scene element type that appears only once in the sample can be considered a mere coincidence. Therefore, these scene element types can be used as samples of rare scene element types in the population, and further estimated as the probability of the observed category. This method has good robustness. If C represents the true value of the coverage, it can be proven that the estimator is asymptotically unbiased, i.e., as n→∞:

[0081]

[0082] Regarding step S10312, when determining the upper bound of the sample variance based on the total sample size, the specific formula can be used:

[0083]

[0084] This is the upper bound of the variance, where the value of n is taken as a larger number.

[0085] For step S10313, in this step, the upper bound of variance is substituted into the Chebyshev inequality to calculate the significance level, which is achieved through the following formula:

[0086]

[0087] ε is the significance level. 1-α represents the specified confidence level. The value of α can be set adaptively, for example, to 5%.

[0088] Regarding step S10314, in this step, based on the significance level and the coverage estimate, the confidence interval of the coverage estimate at a specified confidence level is determined as follows:

[0089] Furthermore, in another embodiment provided in this application, the step of estimating scene coverage based on the analysis results of the target scene elements, determining a coverage estimate, and determining a confidence interval for the coverage estimate at a specified confidence level based on the determined coverage estimate includes:

[0090] S10321. Based on the analysis results of the target scene elements, the corresponding Bayesian nonparametric inference method is used to determine the first hyperparameter of the Poisson-Dirichlet process and the second hyperparameter of the generalized standard gamma process, respectively.

[0091] S10322. Based on the first hyperparameter and the analysis results of the target scene elements, determine the first likelihood value corresponding to the Poisson-Dirichlet process, and based on the second hyperparameter and the analysis results of the target scene elements, determine the second likelihood value corresponding to the generalized standard gamma process.

[0092] S10323. Determine the target prior value from the first likelihood value and the second likelihood value.

[0093] S10324. Based on the target inference process corresponding to the target prior value, use the hyperparameters corresponding to the target inference process and combine them with the target scene element analysis results to perform coverage estimation and determine the coverage estimate value corresponding to the target inference process.

[0094] S10325. Based on the target reasoning process, determine the quantile corresponding to the target reasoning process.

[0095] S10326. Based on the quantiles corresponding to the target reasoning process and the corresponding coverage estimate, determine the confidence interval of the coverage estimate at a specified confidence level.

[0096] Steps S10321-S10326 determine the coverage estimate of the autonomous driving test scenario in this application and its confidence interval based on a Bayesian nonparametric inference method. This method exemplarily uses two of the most common Gibbs-type nonparametric priors: the Poisson–Dirichlet (PD) process and the normalized generalized Gamma (GG) process as priors. Specific technical means are described below:

[0097] For S10321, the formula for calculating the first hyperparameter in this step is as follows:

[0098]

[0099] The formula for calculating the second hyperparameter is as follows:

[0100]

[0101] in, These are estimates of the lumped parameters in the first hyperparameter set. This is an estimate of the discount parameter in the first hyperparameter set. This is an estimate of the steady-state exponent in the second hyperparameter. Let (x) be the estimated value of the scaling parameter in the second hyperparameter. n :=∏ 0≤i≤n-1 (x+i).

[0102] For S10321, in this step, a better prior is selected according to the likelihood function, wherein the formula for calculating the first likelihood value is as follows:

[0103]

[0104] The formula for calculating the second likelihood value is as follows:

[0105]

[0106] Where j is the j-th scene element type, K n This represents the number of different scene element types in the above n samples.

[0107] Regarding step S10323, in one embodiment provided in this application, determining the target prior value from the first likelihood value and the second likelihood value includes: determining the maximum value among the first likelihood value and the second likelihood value as the target prior value.

[0108] Therefore, the determined prior value of the target is L. GG or L PD .

[0109] For step S10324, the calculation method for the corresponding coverage estimate will be different depending on the different verification values ​​determined in this step.

[0110] In one embodiment provided in this application, the step of determining the coverage estimate value corresponding to the target inference process based on the target prior value, using the hyperparameters corresponding to the target inference process and combining the target scene element analysis results to perform coverage estimation, includes:

[0111] When the target inference process is a Poisson-Dirichlet process, the coverage estimate is calculated using the following formula:

[0112]

[0113] in, This is a coverage estimate. These are estimates of the lumped parameters in the first hyperparameter set. Here, K represents the estimated value of the discount parameter in the first hyperparameter set, where i is the i-th sample, n is the total number of samples, and K is the value of the discount parameter. n The number of different scene element types in n samples;

[0114] When the target inference process is a generalized standard gamma process, the coverage estimate is calculated using the following formula:

[0115]

[0116] in, This is an estimate of the steady-state exponent in the second hyperparameter. Here, represents the estimated scaling parameter in the second hyperparameter set, where i is the i-th sample, n is the total number of samples, and K is the value of the scale parameter. n This represents the number of different scene element types in the n samples.

[0117] Regarding step S10325, in one embodiment provided in this application, determining the quantile corresponding to the target inference process based on the target inference process includes:

[0118] When the target reasoning process is a Poisson-Dirichlet process, the quantiles corresponding to the target reasoning process are determined according to the beta distribution;

[0119] When the target reasoning process is a generalized standard gamma process, Monte Carlo sampling is used to determine the quantile corresponding to the target reasoning process.

[0120] Regarding step S10326, in this step, based on the quantiles corresponding to the target inference process and the corresponding coverage estimate, the confidence interval of the coverage estimate at a specified confidence level is determined, which can be specifically as follows:

[0121] The lower limit of the confidence interval is determined by subtracting the corresponding quantile from the coverage estimate, and the upper limit of the confidence interval is determined by adding the corresponding quantile to the coverage estimate.

[0122] For example, when prior to using L PD hour, In X1,...,X n The prior distribution follows a beta distribution Beta(θ+σK). n ,θ-σK n Therefore, the confidence intervals satisfying the specified confidence level 1-α are generated as follows:

[0123] For example, when prior to using LGG At that time, the situation was more complicated. In X1,...,X n Given prior knowledge, it is identically distributed with the following random variables:

[0124]

[0125] Among them G a,1 ~Gamma(a,1) can be sampled directly; R σ,b It is an exponentially tilted σ-stable random variable with an unparalleled b parameter. Its sampling is implemented using a special algorithm, so it can be estimated through sampling using the Monte Carlo method. The confidence interval.

[0126] Among them, regarding the determination through the Monte Carlo method The confidence interval is explained below:

[0127] In the determination through the Monte Carlo method Before determining the confidence interval, the following information is required:

[0128] 1. Positive σ-stable distribution

[0129] f σ (x), σ∈(0,1) is a density function supported by positive real numbers, if it satisfies the following for any t>0:

[0130]

[0131] Then f σ (x) is the density function of a positive σ-stable distribution. Generally, f σ (x) does not have an explicit representation, therefore it cannot be sampled directly. Using S... σ Indicates obedience to f σ A random variable (x).

[0132] 2. Zolotarev's integral representation

[0133] Define the Zolotarev function

[0134]

[0135] but

[0136]

[0137] Where U is a uniform distribution on [0,π], E ~ exp(1), and U and E are independent. Therefore, S can be obtained by sampling U and E respectively. σ The sample.

[0138] 3. Exponentially tilted

[0139] f(x) is a density function supported by positive real numbers, representing an exponentially tilted random variable X. λ Density function and e -λx f(x) is proportional. Theoretically, X can be obtained by accepting rejection samples. λ ,Right now:

[0140] Step 1. Independently generate a random number X that follows f and a uniformly distributed random number U on the interval 0-1;

[0141] Step 2. If U≤e -λX If X is included in the sample set, then X is included; otherwise, step 1 is repeated.

[0142] Obviously, the acceptance probability P(U≤e) in the above sampling process is... -λX ) = E f [e -λX If λ is very large, the acceptance probability will be very small, resulting in low efficiency of the sampling method described above. For example, X = S σ hour Leading to average Only one acceptable sample will be obtained after multiple samplings.

[0143] If using S σ,λ This indicates that the density is proportional to e. -λX f σ The random variable (x) has the following actual density function: Then Y = (S σ,λ ) -σ / (1-σ) The density function is in:

[0144]

[0145] Therefore, we can consider (Y,U) to be a binary random variable derived from [0,∞)×[0,π], where the marginal distribution of U is a uniform distribution on [0,π]. If we can sample according to h, we can discard U and pass through S. σ,λ =(Y) -(1-σ) / σ Get S σ,λ The sample.

[0146] 4. Double Rejection Sampling

[0147] If a curve g ≥ h exists, define g * (u)=∫g(x,u)dx / G * G * These are the normalization coefficients. If for g... * (u) If sampling is relatively easy, the following steps can be performed:

[0148] Step 1, according to g * (u) Generate U;

[0149] Step 2: Generate X based on the distribution proportional to g(x,U);

[0150] Step 3: Generate a uniformly random number V in the range of 0-1;

[0151] Step 4: If h(X,U) / g(X,U)≥V, accept sample X; otherwise, return to step 1.

[0152] It can be seen that under such sampling, U, X, V follow a joint density proportional to g. * (u)·g(x,u) / g * (u)·I(h(x,u) / g(x,u)≥v). Integrating with respect to V, we can see that the sample (X,U) obtained by this sampling follows a sequence h. It can be seen that the average number of iterations for each accepted sample in the above algorithm is G. * .

[0153] If g * (u) is still difficult to sample; further methods such as accept-rejection sampling can be used, i.e., searching for g. ** ≥g * First, generate U, then continue with the above method.

[0154] 5. R distribution (R cumulative distribution function)

[0155] Define the density function r(y) = a·c·exp(-λy) -b -ay), then there exists a density function r * (y) satisfies r(y)≤r * (y). Follows log(r) * The random variable Y can be composed of the following three parts.

[0156]

[0157] in L′(x)=bλ / x b+1 -a,L″(x)=-b(b+1) / x b+2 ,N~N(0,1),V~U(0,1),E~exp(1),

[0158] a2 = ace -am(1+1 / b) δ,a3=a·c·e -am(1+1 / b) / |L′(m+δ)|.

[0159] Simultaneously, it can calculate

[0160]

[0161] Therefore, based on the above, relying on Monte Carlo sampling can specifically include the following:

[0162] It is possible to construct h(y,u)≤r * (y). Therefore, it can be determined according to R. * Construct g * (u) is as follows:

[0163]

[0164] Where α = σ, γ = λ σ σ(1-σ),B(0)=1 / (σ σ (1-σ) 1-σ ),B(u)=

[0165] A(u) -1+σ .

[0166] This can also be verified.

[0167]

[0168] in Therefore, it obeys g. ** The random variable U of (u) can be represented as a mixture of the following random variables:

[0169] When γ≥1:

[0170]

[0171] When γ < 1

[0172]

[0173] Among them N~N(0,1), W~U(0,1), w3 = ξπ;

[0174] The specific algorithm logic is as follows:

[0175] According to the above explanation, h(y,u) = r(y), where a = A(u). That is, h(y,u)≤r * (y,u)and∫r *(y,u)du=R * (u)=g * (u)≤g ** (u). Therefore, g is used first. ** (u) is the proposal distribution, generated using the accept / reject sampling method, which follows a density proportional to g. * (u) of U, then using density proportional to r * The proposal distribution for (y,U) generates samples satisfying h(y,u) using an accept / reject sampling method. It's important to note that in practice, calculating log(r)... * (y,U)) and log(h(y,U)) are more efficient, so we directly use exp(1) random number instead of uniform random number.

[0176] The specific algorithm is as follows:

[0177] Therefore, for S σ,λ The double rejection sampling algorithm is as follows:

[0178] Step 1: Define the following parameters

[0179] γ=λ σ σ(1-σ),

[0180] w3=ξπ,b=(1-σ) / σ;

[0181] According to g ** (u) Samples are obtained that obey g * (u) Distribution of U.

[0182] Step 1: Independently generate uniform random numbers V, W, and W′ on the interval 0-1.

[0183] Step 2, if γ ≥ 1;

[0184] if

[0185] else: U = W

[0186] else:

[0187] if U = πW

[0188] else: U = W

[0189] step 3.ifU * <π and g * (U) / g ** (U)>W′:

[0190] B(0)=1 / (σσ (1-σ) 1-σ ),B(u)=A(u) -1+σ

[0191]

[0192] return U,z

[0193] else:

[0194] Back to step 1

[0195] Step 2: Define the following parameters:

[0196] a = A(U), m = (bλ / a) σ , a2=δ, a3=z / a, s=a1+a2+a3

[0197] Generate random numbers X whose density function is proportional to g(x,U).

[0198] Step 1. Independently generate uniform random numbers V' on the interval 0-1.

[0199] ifV′≤a1 / s:X=m-δ|N|,N~N(0,1)

[0200] elifV′≤(a1+a2) / s:X~uniform(m,m+δ)

[0201] elseV′>(a1+a2) / s:X=m+δ+a3E′,E′~exp(1)

[0202] Step 2. Generate E ~ exp(1) and determine whether to accept.

[0203] if X≥0 and

[0204] accept X

[0205] else:

[0206] Back to step 1

[0207] returnS σ,λ =X -b

[0208] Therefore, R(λ,σ) can be used to represent the average number of iterations required to obtain a sample using the above algorithm, as follows:

[0209]

[0210] Thus, the Monte Carlo method can be used to estimate through sampling. The confidence interval.

[0211] For step S104, the evaluation result of the autonomous driving test data coverage is determined based on the determined confidence interval. Specifically, the evaluation result of the autonomous driving test data coverage is determined based on the lower limit of the determined confidence interval.

[0212] Thus, this application uses the analysis results of various scene elements in the road-collected data to estimate scene coverage using multiple inference methods, and determines a confidence interval at a specified confidence level based on the estimated scene coverage. This allows for the estimation of scenes not appearing in the road-collected data based on the determined confidence interval, thereby determining the evaluation result of the autonomous driving test data coverage. Therefore, the method provided in this application can estimate the probability of scene elements not appearing in the road-collected data in an autonomous driving scenario, thereby inferring the probability of unknown scenarios, and estimating the safety of the tested autonomous driving algorithm based on the estimated probability of unknown scenarios.

[0213] Based on the same inventive concept, this application also provides a determining device corresponding to the determining method. Since the principle of the device in this application is similar to the determining method described above, the implementation of the device can refer to the implementation of the method, and the repeated parts will not be described again.

[0214] Please see Figure 3 , Figure 3 This is a schematic diagram of a device for determining the coverage of autonomous driving test data provided in an embodiment of this application. Figure 3 As shown, the determining device 300 includes:

[0215] Module 310 is used to acquire road sampling datasets for the target area;

[0216] The analysis module 320 is used to perform scene element analysis on the road sampling dataset and determine the target scene element analysis results in the road sampling dataset; wherein, the target scene element analysis results include the scene element types related to autonomous driving testing, the number of times each scene element type appears, and the total number of samples;

[0217] The estimation module 330 is used to estimate the scene coverage based on the analysis results of the target scene elements, determine the coverage estimate value, and determine the confidence interval of the coverage estimate value at a specified confidence level based on the determined coverage estimate value.

[0218] The determination module 340 is used to determine the evaluation result of the coverage of autonomous driving test data based on the determined confidence interval.

[0219] Optionally, when the estimation module 330 is used to perform scene coverage estimation based on the target scene element analysis results, determine the coverage estimate value, and determine the confidence interval of the coverage estimate value at a specified confidence level, the estimation module 330 is used to:

[0220] The coverage estimate is determined based on the total number of samples and the number of scene element types that appear a preset number of times.

[0221] Determine the upper bound of the sample variance based on the total sample size;

[0222] Substitute the aforementioned upper bound of variance into Chebyshev's inequality to calculate the significance level;

[0223] Based on the significance level and the coverage estimate, determine the confidence interval of the coverage estimate at a specified confidence level.

[0224] Optionally, the estimation module 330 is further configured to determine the coverage estimate using the following formula, based on the total number of samples and the number of scene element types that occur a preset number of times:

[0225]

[0226] in, This is the coverage estimate, where n is the total number of samples, and M is the total number of samples. x,n The number of scene element types that appear a preset number of times, where x is the preset number of times.

[0227] Optionally, when the estimation module 330 is used to perform scene coverage estimation based on the target scene element analysis results, determine the coverage estimate value, and determine the confidence interval of the coverage estimate value at a specified confidence level, the estimation module 330 is used to:

[0228] Based on the analysis results of the target scene elements, the corresponding Bayesian nonparametric inference method is used to determine the first hyperparameter of the Poisson-Dirichlet process and the second hyperparameter of the generalized standard gamma process, respectively.

[0229] Based on the first hyperparameter and the analysis results of the target scene elements, the first likelihood value corresponding to the Poisson-Dirichlet process is determined, and based on the second hyperparameter and the analysis results of the target scene elements, the second likelihood value corresponding to the generalized standard gamma process is determined.

[0230] The target prior value is determined from the first likelihood value and the second likelihood value;

[0231] Based on the target reasoning process corresponding to the target prior value, the hyperparameters corresponding to the target reasoning process are used and combined with the target scene element analysis results to perform coverage estimation and determine the coverage estimate value corresponding to the target reasoning process.

[0232] Based on the target reasoning process, determine the quantile corresponding to the target reasoning process;

[0233] Based on the quantiles corresponding to the target reasoning process and the corresponding coverage estimate, determine the confidence interval of the coverage estimate at a specified confidence level.

[0234] Optionally, when determining the target prior value from the first likelihood value and the second likelihood value, the estimation module 330 is configured to:

[0235] The maximum value between the first likelihood value and the second likelihood value is determined as the target prior value.

[0236] Optionally, when the estimation module 330 is used to determine the quantile corresponding to the target reasoning process based on the target reasoning process, the estimation module 330 is used to:

[0237] When the target reasoning process is a Poisson-Dirichlet process, the quantiles corresponding to the target reasoning process are determined according to the beta distribution;

[0238] When the target reasoning process is a generalized standard gamma process, Monte Carlo sampling is used to determine the quantile corresponding to the target reasoning process.

[0239] Optionally, when the estimation module 330 is used to perform coverage estimation based on the target inference process corresponding to the target prior value, using the hyperparameters corresponding to the target inference process and combining the target scene element analysis results, and determining the coverage estimate value corresponding to the target inference process, the estimation module 330 is used to:

[0240] When the target inference process is a Poisson-Dirichlet process, the coverage estimate is calculated using the following formula:

[0241]

[0242] in, This is a coverage estimate. These are estimates of the lumped parameters in the first hyperparameter set. Here, K represents the estimated value of the discount parameter in the first hyperparameter set, where i is the i-th sample, n is the total number of samples, and K is the value of the discount parameter. n The number of different scene element types in n samples;

[0243] When the target inference process is a generalized standard gamma process, the coverage estimate is calculated using the following formula:

[0244]

[0245] in, This is an estimate of the steady-state exponent in the second hyperparameter. Here, represents the estimated scaling parameter in the second hyperparameter set, where i is the i-th sample, n is the total number of samples, and K is the value of the scale parameter. n This represents the number of different scene element types in the n samples.

[0246] Please see Figure 4 , Figure 4 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Figure 4 As shown, the electronic device 400 includes a processor 410, a memory 420, and a bus 430.

[0247] The memory 420 stores machine-readable instructions executable by the processor 410. When the electronic device 400 is running, the processor 410 communicates with the memory 420 via the bus 430. When the machine-readable instructions are executed by the processor 410, they can perform the operations described above. Figure 1 as well as Figure 2 The steps in the method embodiment shown are specifically implemented in the method embodiment and will not be repeated here.

[0248] This application also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, can perform the above-described actions. Figure 1 as well as Figure 2 The steps in the method embodiment shown are specifically implemented in the method embodiment and will not be repeated here.

[0249] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0250] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. The apparatus embodiments described above are merely illustrative. For example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. Furthermore, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Additionally, the shown or discussed mutual couplings, direct couplings, or communication connections may be through some communication interfaces; indirect couplings or communication connections between devices or units may be electrical, mechanical, or other forms.

[0251] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0252] In addition, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0253] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a processor-executable, non-volatile, computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0254] Finally, it should be noted that the above-described embodiments are merely specific implementations of this application, used to illustrate the technical solutions of this application, and not to limit them. The scope of protection of this application is not limited thereto. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that any person skilled in the art can still modify or easily conceive of changes to the technical solutions described in the foregoing embodiments, or make equivalent substitutions for some of the technical features, within the scope of the technology disclosed in this application. Such modifications, changes, or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be covered within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A method for determining the coverage of autonomous driving test data, characterized in that, The determination method includes: Obtain the road survey dataset for the target area; Scene element analysis is performed on the road sampling dataset to determine the target scene element analysis results in the road sampling dataset; wherein, the target scene element analysis results include the scene element types related to autonomous driving testing, the number of times each scene element type appears, and the total number of samples; Based on the analysis results of the target scene elements, scene coverage is estimated, a coverage estimate is determined, and a confidence interval for the coverage estimate at a specified confidence level is determined based on the determined coverage estimate. Based on the determined confidence interval, the evaluation results of the autonomous driving test data coverage are determined.

2. The determination method according to claim 1, characterized in that, The step of estimating scene coverage based on the analysis results of the target scene elements, determining the coverage estimate, and determining the confidence interval of the coverage estimate at a specified confidence level based on the determined coverage estimate includes: The coverage estimate is determined based on the total number of samples and the number of scene element types that appear a preset number of times. Determine the upper bound of the sample variance based on the total sample size; Substitute the aforementioned upper bound of variance into Chebyshev's inequality to calculate the significance level; Based on the significance level and the coverage estimate, determine the confidence interval of the coverage estimate at a specified confidence level.

3. The determination method according to claim 2, characterized in that, The coverage estimate is determined using the following formula, based on the total number of samples and the number of scene element types that appear a preset number of times: in, This is the coverage estimate, where n is the total number of samples, and M is the total number of samples. x,n The number of scene element types that appear a preset number of times, where x is the preset number of times.

4. The determination method according to claim 1, characterized in that, The step of estimating scene coverage based on the analysis results of the target scene elements, determining the estimated coverage value, and determining the confidence interval of the estimated coverage value at a specified confidence level includes: Based on the analysis results of the target scene elements, the corresponding Bayesian nonparametric inference method is used to determine the first hyperparameter of the Poisson-Dirichlet process and the second hyperparameter of the generalized standard gamma process, respectively. Based on the first hyperparameter and the analysis results of the target scene elements, the first likelihood value corresponding to the Poisson-Dirichlet process is determined, and based on the second hyperparameter and the analysis results of the target scene elements, the second likelihood value corresponding to the generalized standard gamma process is determined. The target prior value is determined from the first likelihood value and the second likelihood value; Based on the target reasoning process corresponding to the target prior value, the hyperparameters corresponding to the target reasoning process are used and combined with the target scene element analysis results to perform coverage estimation and determine the coverage estimate value corresponding to the target reasoning process. Based on the target reasoning process, determine the quantile corresponding to the target reasoning process; Based on the quantiles corresponding to the target reasoning process and the corresponding coverage estimate, determine the confidence interval of the coverage estimate at a specified confidence level.

5. The determination method according to claim 4, characterized in that, Determining the target prior value from the first likelihood value and the second likelihood value includes: The maximum value between the first likelihood value and the second likelihood value is determined as the target prior value.

6. The determination method according to claim 4, characterized in that, The step of determining the quantile corresponding to the target reasoning process based on the target reasoning process includes: When the target reasoning process is a Poisson-Dirichlet process, the quantiles corresponding to the target reasoning process are determined according to the beta distribution; When the target reasoning process is a generalized standard gamma process, Monte Carlo sampling is used to determine the quantile corresponding to the target reasoning process.

7. The determination method according to claim 4, characterized in that, The step of determining the coverage estimate value corresponding to the target inference process based on the target prior value, using the hyperparameters corresponding to the target inference process and combining the target scene element analysis results, includes: When the target inference process is a Poisson-Dirichlet process, the coverage estimate is calculated using the following formula: in, This is a coverage estimate. These are estimates of the lumped parameters in the first hyperparameter set. Here, K represents the estimated value of the discount parameter in the first hyperparameter set, where i is the i-th sample, n is the total number of samples, and K is the value of the discount parameter. n The number of different scene element types in n samples; When the target inference process is a generalized standard gamma process, the coverage estimate is calculated using the following formula: in, This is an estimate of the steady-state exponent in the second hyperparameter. Here, represents the estimated scaling parameter in the second hyperparameter set, where i is the i-th sample, n is the total number of samples, and K is the value of the scale parameter. n This represents the number of different scene element types in the n samples.

8. A device for determining the coverage of autonomous driving test data, characterized in that, The determining device includes: The acquisition module is used to acquire road sampling datasets for the target area; The analysis module is used to perform scene element analysis on the road sampling dataset and determine the target scene element analysis results in the road sampling dataset; wherein, the target scene element analysis results include the scene element types related to autonomous driving testing, the number of times each scene element type appears, and the total number of samples; The estimation module is used to estimate the scene coverage based on the analysis results of the target scene elements, determine the estimated coverage value, and determine the confidence interval of the estimated coverage value at a specified confidence level based on the determined estimated coverage value. The determination module is used to determine the evaluation results of autonomous driving test data coverage based on the determined confidence interval.

9. An electronic device, characterized in that, include: The device includes a processor, a memory, and a bus, wherein the memory stores machine-readable instructions executable by the processor, and when the electronic device is running, the processor communicates with the memory via the bus, and the machine-readable instructions are executed by the processor to perform the steps of the determination method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, performs the steps of the determination method as described in any one of claims 1 to 7.

Images