A Neuron Coverage Guided Testing Method for Autonomous Driving Systems

Abstract

The present invention discloses a neuron coverage-guided testing method for an autonomous driving system, belonging to the field of artificial intelligence testing. Specifically, it relates to the problem of severe scarcity of large-scale high-quality test samples in the testing of autonomous driving systems. Its feature is to cluster the original test samples collected by the autonomous driving system according to image spot information, screen out an equal number of seed samples, and then, with the goal of maximizing neuron coverage and making the system predictions inconsistent, transform the modification on the original test cases into an optimization problem and solve it using the gradient ascent method to automatically generate a test case set that is likely to cause the system to output errors and has a high neuron coverage. Finally, it is re-input into the autonomous driving model for training, mainly solving the problems of insufficient test cases, low neuron coverage, and long time consumption in the current autonomous driving system, thereby uncovering more unreasonable system defects in autonomous driving software, reducing test losses, and enhancing the prediction accuracy of the system model.

Description

Technical Field

[0001] The present invention belongs to the field of artificial intelligence testing, and particularly relates to a neuron coverage-guided testing method for an autonomous driving system. Based on an existing image dataset of the autonomous driving system, test cases are augmented to detect and repair potential defects and unreasonable behaviors of the system. Background Art

[0002] With the rapid development of deep neural network technology, it has been widely applied to many safety-sensitive fields such as computer vision and natural language processing. The research and development of autonomous driving systems have also made great breakthroughs and become one of the core driving forces for the automotive industry to embrace the intelligent era. However, in recent years, there have been continuous accidents related to autonomous driving. Accident analysis points out that the causes are all misjudgments of the autonomous driving system to varying degrees. Like any system relying on software algorithms, the autonomous driving system also has usage risks and may encounter errors or unexpected extreme situations. Once a behavior prediction problem occurs, it may lead to serious consequences such as fatal collisions. Therefore, before autonomous driving is deployed, the autonomous driving system must be fully tested to eliminate potential safety hazards as much as possible.

[0003] The classic method for testing deep neural network models is to collect enough labeled test data to evaluate the prediction accuracy of the models. However, the input samples of the autonomous driving system are obtained by different sensors, and the input space is very large. It is difficult to manually collect all possible inputs to trigger every feasible logic of the autonomous driving system. Moreover, there are problems such as class imbalance, annotation uncertainty, and low neuron coverage rate in the small amount of test samples collected manually, which seriously hinder the testing of the autonomous driving system.

[0004] For the above reasons, the present invention proposes a neuron coverage-guided testing method to solve such testing problems faced by the autonomous driving system, thereby automatically generating an autonomous driving test case set and improving the testing efficiency. Summary of the Invention

[0005] The problem solved by the present invention is that in the testing of autonomous driving systems, large-scale and high-quality test samples are severely scarce, and the efficiency of manually collecting and screening test samples is low, resulting in insufficient test cases for autonomous driving systems, problems such as low neuron coverage rate and long time consumption. How to fully test and verify safety-sensitive autonomous driving systems has become the main challenge currently faced. The present invention introduces neuron coverage into autonomous driving testing, automatically generates test cases with high neuron coverage rate and easy to be mispredicted, augments the original test dataset, and discovers potential defects and unreasonable behaviors existing in the autonomous driving system to improve the model robustness of the autonomous driving system under deep neural network technology. Its characteristics mainly include the following steps:

[0006] Step 1: Obtain the optical sensor image data collected by the autonomous driving system and the corresponding label information as the original test samples, and detect the number of speckle features in the images.

[0007] Step 2: Conduct cluster analysis on the test samples according to the number of image speckle features and the label information, and select an equal number of test samples from each category and save them into the seed sample set.

[0008] Step 3: Select the target autonomous driving prediction model, input the quantitative seed samples into the pre-trained model, record the model prediction values. If the error between the output value of the steering angle and the original label of the corresponding seed sample exceeds the set range value, discard this seed sample; otherwise, proceed to Step 4.

[0009] Step 4: For the selected seed samples, select the targeted neurons that are often and rarely covered in past tests, calculate the loss value, and update the neuron coverage rate.

[0010] Step 5: With the goal of maximizing the neuron coverage rate and making the predictions of the autonomous driving system inconsistent, convert the modification on the seed samples into an optimization problem and solve it using the gradient ascent method.

[0011] Step 6: Add three different types of constraint conditions to the seed samples, set the width of the image part, occlusion, and dirt to simulate different environments of the image data, and constrain the solution of the optimization problem.

[0012] Step 7: Set the number of gradient iterations, start to execute the local search guided by gradient ascent, find the new input that maximizes the required goal, and make the seed samples enter the iterative process.

[0013] Step 8: Repeat Step 7 until the maximum number of iterations or the error between the predicted steering value and the original label of the corresponding seed sample reaches the preset condition, and generate new test samples for the target model.

[0014] Step 9: Loop through Steps 3 - 7 until the test sample set of the target model is generated, and jointly form the neuron coverage-guided test sample set for the autonomous driving system with the test samples of different models.

[0015] Step 10: Retrain the autonomous driving prediction model using the finally generated test sample set, and calculate and evaluate the prediction accuracy of the autonomous driving system.

[0016] Among them, in Step 1, the speckle information is used as an important feature of the image, and the LoG algorithm is used to detect the number of speckle features of the test samples. In Step 2, the K-Means algorithm is used to cluster the test sample set according to the speckle features and label information, so that an equal number of seed samples are selected from each category. The optimization problem described in Step 5 is defined as objjoint =max((c i -c)+λ(f n (x)), c i is the probability that the target model predicts the seed sample x as category i, c is the true label of the seed sample x, and f n (x) is the output value of neuron n under the seed sample x, and λ is the balance parameter used to balance the two objective optimization problems.

[0017] The present invention is characterized in that:

[0018] 1. Combine image blob detection technology with the neuron coverage-based testing method and apply it to the test case generation of autonomous driving systems.

[0019] 2. Automatically generate new test cases for autonomous driving systems and apply them to the training of autonomous driving models to enhance model robustness.

[0020] Compared with the prior art, the present invention has the following beneficial effects:

[0021] The neuron coverage-guided testing method of the present invention eliminates the process of cross-validation of multiple models with similar functions, shortens the test case generation time, and improves the neuron coverage more significantly. Compared with the method based on adversarial generation of test cases, it is easier to implement and ensures the diversity of the seed sample queue. The experiment uses the same autonomous driving model and data set as the existing coverage-based method DeepXplore, and uses two methods to generate 100 test samples on the test data set HMB3. The average neuron coverage improvement, average generation time, and number of samples with prediction errors are compared to judge the quality of the test cases. The experimental results are shown in Table 1, which confirms the effectiveness of this method.

[0022] Table 1 Comparison results of DeepXplore method and the method of the present invention

[0023]

[0024] The above examples and analysis show that, under the same parameter settings and constraints, the method of the present invention improves the neuron coverage by 1.3% on average compared to the existing coverage-based test method, and the average generation time is only 1 / 12 of that, and the number of test samples that cause the model to misjudge increases from 20 to 52, which can discover more erroneous behaviors of the model. This shows that the method of the present invention has great advantages in improving neuron coverage, generating test samples in a shorter time, and discovering more system erroneous behaviors under the same original test sample constraints. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Attached Figure 1Schematic diagram of the neuron coverage guided test method for autonomous driving systems. Detailed implementation manners

[0026] To make the objectives, technical solutions and advantages of the present invention more clear, the following will further describe the present invention in detail with specific examples and attached Figure 1 which specifically includes the following steps:

[0027] 1. Obtain the prediction model and image dataset of the autonomous driving system, specifically including:

[0028] 1.1 Collect the image dataset, obtain the Udacity autonomous driving dataset HMB3 and save it;

[0029] 1.2 Collect the prediction models of the autonomous driving system, and obtain three driving system models based on NVIDIA DAVE-2, namely DAVE-orig, DAVE-norminit, and DAVE-dropout.

[0030] 2. Preprocess, specifically including:

[0031] 2.1 Select a certain dataset collected in step 1.1 and process the picture format into the format required by the model input;

[0032] 2.2 Select the processed dataset in step 2.2 and label the corresponding label values for the pictures according to the label list.

[0033] 3. Obtain the seed samples, specifically including:

[0034] 3.1 Perform image blob detection on the processed dataset and record the number of blobs in each picture. The blob detection is achieved by calculating the similarity between the image and the convolution function. Assume the image to be tested as the density function I(x,y) of a random variable x and perform convolution operation with the Laplacian of Gaussian function. When the distribution of the two functions is similar, a larger function response value is obtained. When the Laplacian response of the image to be tested is larger, the detected image pixel points are the image feature blobs. The calculation formula is: where G σ (x,y) is the standard deviation and is the two-dimensional Gaussian function of σ.

[0035] 3.2 Perform clustering analysis on the dataset according to the number of image blobs and the corresponding labels. Set the number of categories to six, and select 50 samples in each category. A total of 300 seed samples form the seed sample set T. The clustering analysis is based on the expectation-maximization algorithm, and continuously iterates the mean distance between the samples and the feature centers. The mean error formula is Σ k i＝1 Σ x∈ci |d(x,c i )|2 , where d is the Euclidean distance between the sample x and the clustering center c i . When the number of iterations is reached or the mean vector no longer changes, the model construction is completed and the clustering result is output.

[0036] 4. Screen seed samples, specifically including:

[0037] 4.1 Select a certain autonomous driving model collected in step 1.2 as the target model F, and input the seed sample set T into the model for predicting the steering angle of the vehicle, and record the predicted values;

[0038] 4.2 Initialize the neuron coverage table, record the initial neuron coverage of each seed sample, and the calculation formula is where N represents the number of neurons in the deep neural network model, t is the activation function threshold, T is the seed sample set, and f(x, n) represents the output of the seed sample x on the neuron n.

[0039] 4.3 Compare the predicted value of the seed sample with the original label value, record the predicted difference value of the seed sample. If the difference exceeds the preset range of 0.2, it is regarded as a test case with a prediction error, and the sample is discarded; if it does not exceed the preset range, the sample is retained in the new batch.

[0040] 5. Establish an optimization problem, specifically including:

[0041] 5.1 Select the neurons that have been frequently and rarely covered in the past from the new batch of seed samples according to the neuron selection strategy as the target neurons and calculate their loss values;

[0042] 5.2 Combine the predicted difference value described in step 4.3 and the neuron coverage rate described in step 4.2 to construct the target optimization problem of the seed sample, defined as obj joint = max((c i - c)+ λ(f n (x)), where c i is the probability that the target model predicts the seed sample x as class i, c is the true label of the seed sample x, f n (x) is the output value of the neuron n under the seed sample x, and λ is the balance parameter used to balance the two target optimization problems, set to 0.1.

[0043] 6. Solve the optimization problem, specifically including:

[0044] 6.1 Add three different types of constraint conditions to the seed samples, set the width of the image part, occlusion, and dirt to simulate different environments of the image data, and constrain the solution gradient. Specifically, simulate the intensity of light by restricting the addition and subtraction of image pixels; simulate the occluded situation by adding a small rectangle R at any position of the seed sample; complete the constraint by adding one or more fragments S to any part of the seed sample to simulate the image taken under a dirty lens.

[0045] 6.2 Set the number of gradient iterations, start performing local search guided by gradient ascent, find a new input that maximizes the required objective, and make the seed sample enter the iterative process. The gradient ascent method explores the maximum value along the gradient direction of the function, and its solution formula is: w := w + α▽wf(w), where α is the learning rate and ▽ is the gradient of w.

[0046] 6.3 Repeat step 6.2 until the maximum number of iterations or the error between the model prediction value of the test sample and its original label reaches the preset condition, obtain a new test sample of the target model, and calculate and record the model prediction value and neuron coverage information of the new test sample.

[0047] 7. Loop through steps 4 - 6 until a test sample set of the target model is generated, and jointly form a neuron coverage-guided test sample set for the autonomous driving system with the test samples of different models.

[0048] 8. Retrain the autonomous driving prediction model using the finally generated test sample set, and calculate and evaluate the prediction accuracy of the autonomous driving system.

[0049] 9. The above examples have elaborated on the technical solutions of the present invention in detail. It should be understood that the above are only specific examples of the present invention and are not used to limit the present invention. Any modifications, supplements, equivalent replacements, etc. made within the scope of the principles of the present invention shall be included within the protection scope of the present invention.

Claims

1. A neuron coverage-guided testing method for an autonomous driving system, characterized in that The method includes the following steps: Step 1: Obtain the optical sensor image data and corresponding label information collected by the autonomous driving system as the original test samples, and detect the number of speckle features in the images; Step 2: Perform clustering analysis on the test samples according to the number of image speckle features and label information, and select an equal number of test samples from each category and save them to the seed sample set; Step 3: Select a target autonomous driving prediction model, input the quantitative seed samples into the pre-trained model, record the model prediction values. If the error between the output value of the steering angle and the original label of the corresponding seed sample exceeds the set range value, discard this seed sample, otherwise proceed to Step 4; Step 4: For the selected seed samples, select the target neurons that are often and rarely covered in past tests, calculate the loss value, and update the neuron coverage rate; Step 5: With the goal of maximizing the neuron coverage rate and making the predictions of the autonomous driving system inconsistent, convert the modification on the seed samples into an optimization problem and solve it using the gradient ascent method; Step 6: Add three different types of constraint conditions to the seed samples, set the width, occlusion, and dirt of the image part to simulate different environments of the image data, and constrain the solution of the optimization problem; Step 7: Set the number of gradient iterations, start performing local search guided by gradient ascent, find the new input that maximizes the required goal, and make the seed samples enter the iterative process; Step 8: Repeat Step 7 until the maximum number of iterations or the error between the predicted steering value and the original label of the corresponding seed sample reaches the preset condition, and generate new test samples for the target model; Step 9: Loop through Steps 3 - 7 until a test sample set for the target model is generated, and jointly form a neuron coverage-guided test sample set for the autonomous driving system with the test samples of different models; Step 10: Use the finally generated test sample set to retrain the autonomous driving prediction model, and calculate and evaluate the prediction accuracy of the autonomous driving system.

2. The neuron coverage guided test method for an autonomous driving system according to claim 1, characterized in that: In Step 2, the speckle information is used as an important feature of the image. The LoG algorithm is used to detect the number of speckle features of the test samples, and then the K-Means algorithm is used to cluster the test sample set according to the speckle features and label information, so that an equal number of seed samples are selected from each category.

3. The neuron coverage-guided testing method for an autonomous driving system according to claim 1, wherein: The optimization problem described in step 5 is defined as obj joint = max((c i - c) + λ(f n (x)), c i is the probability that the target model predicts the seed sample x as class i, c is the true label of the seed sample x, f n (x) is the output value of neuron n under the seed sample x, and λ is the balance parameter used to balance the two objective optimization problems.

Images