A test method for automatic driving multi-dimensional perception fusion target detection

Abstract

A kind of test method for multi-dimensional perception fusion target detection of automatic driving, comprising: extracting common instance in each traffic scene by instance segmentation, reasonably inserting instance into original image according to the design of extreme traffic scene, generating extreme sample with more real visual effect;Then a set of lightweight test indicators based on scene perception is calculated for the generated sample, and a set of test samples with high quality is screened out;Finally, it is sent into the multi-dimensional perception fusion target detection model of automatic driving for testing.The present application provides a kind of test method for multi-dimensional perception fusion target detection system of automatic driving, the present application solves the problems of existing automatic driving test generation sample reality, lack of diversity, extreme scene difficult to cover, provides a kind of lightweight test indicator for automatic driving target detection, perception scene, efficiently measures the comprehensive quality of test sample, improves the test efficiency of automatic driving.

Description

Technical Field

[0001] This invention relates to autonomous driving, computer vision, object detection, and metamorphic testing, and in particular to a testing method for multi-dimensional perception fusion object detection in autonomous driving. Background Technology

[0002] In recent years, Automated Driving Systems (ADS) have made significant progress. A typical ADS consists of multiple modules, including sensing, perception, planning, and control, incorporating the latest advancements from various fields. While ADS has achieved considerable development, ensuring system safety remains paramount, as unsafe behavior can lead to catastrophic consequences and substantial economic and social losses. Testing is a crucial method for validating systems in practice; however, due to the complexity and multidisciplinary nature of ADS, testing is extremely challenging.

[0003] Multidimensional perception fusion target detection is a crucial foundation for visual perception, motion prediction, and trajectory planning in many mainstream autonomous systems (such as ADS and UAVs). The perception module senses the external environment by receiving and processing sensor data. Therefore, before deploying the target detection module to an ADS, it is necessary to thoroughly test its robustness to improve system reliability. This invention primarily focuses on security testing research for a multidimensional perception fusion target detection module in an ADS.

[0004] However, existing ADS testing efforts primarily focus on generating test cases using methods such as style transfer, adversarial generation, and image manipulation. These generated test cases lack realism and diversity, and fail to cover extreme traffic scenarios that are more likely to trigger ADS erroneous behavior. Existing ADS testing metrics are mostly direct transplants or simple variations of DNN testing metrics, without considering the semantic characteristics of the ADS perception module.

[0005] Chinese patent CN202211118613.6 discloses an autonomous driving testing system, method, electronic device, and storage medium. It utilizes a traffic flow generation module, a simulation module, an injection device, and an intelligent test bench to build a testing system for automated testing of the vehicle under test, thereby improving testing efficiency. Chinese patent CN202211086736.6 discloses an autonomous driving simulation testing method, device, equipment, and computer-readable medium. This method combines real and simulated vehicle testing simultaneously, which can improve the accuracy of autonomous driving simulation test results. However, these methods test vehicles in conventional traffic scenarios, and their test sample diversity is insufficient, making it difficult to cover extreme traffic scenarios that are more likely to induce erroneous behavior.

[0006] Chinese patent CN202211003321.8 discloses a method and system for generating autonomous driving test data, an electronic device, and a storage medium. This invention obtains scene data through genetic algorithms and brute-force search algorithms, and can generate a large amount of dangerous scene data. However, genetic algorithms and brute-force search algorithms are inefficient, computationally expensive, and cannot guarantee the authenticity of the generated scenes, resulting in poor test effectiveness. Summary of the Invention

[0007] This invention aims to overcome the aforementioned shortcomings of existing technologies and provides a testing method for multi-dimensional perception fusion target detection systems for autonomous driving. This method has a simple algorithm, is easy to implement in software, and covers more extreme traffic scenarios while ensuring the authenticity and diversity of test samples, thereby improving the efficiency of autonomous driving safety testing.

[0008] The technical solution adopted by this invention to solve its technical problem is as follows:

[0009] A testing method for multi-dimensional perception fusion target detection in autonomous driving includes the following steps:

[0010] S1: Generate an instance pool: On the test data of the publicly available autonomous driving dataset, use the YOLACT instance segmentation algorithm to extract various types of instances such as pedestrians and vehicles, and filter the extracted results according to size and quality;

[0011] S2: Generate test cases: Based on the designed extreme traffic scenarios such as pedestrian crossing, traffic congestion, illegal parking, and uncivilized driving, select the corresponding instances from the instance pool, scale the instances according to the vanishing point estimate, and paste them onto the original test data;

[0012] S3: Filter test cases based on test metrics: Calculate test metrics on each generated test case and filter out a set of optimal test cases;

[0013] S4: Test the target detection model: Input the selected extreme scene examples into the multidimensional perception fusion target detection model for transformation test, obtain the average accuracy of the model, and analyze the test results.

[0014] Furthermore, in step S1, the YOLACT model is loaded, the original traffic data is input, and the binocular image matching is used to perform instance segmentation based on the category, angle and other information labeled in the dataset to obtain pedestrians, animals, non-motorized vehicles and cars (forward, reverse and lateral), and incomplete and excessively small or large instances are removed to obtain an instance pool.

[0015] Furthermore, step S2 includes the following steps:

[0016] S2-1: Select the instance to be inserted based on the four defined extreme traffic scenarios: pedestrian crossing scenario selects pedestrian and animal instances, traffic congestion scenario selects forward-moving vehicle instances, illegal parking scenario selects non-motorized vehicle instances, and uncivilized driving scenario selects lateral and reverse-moving vehicle instances.

[0017] S2-2: Estimate the vanishing point on the original test image. Use the Canny and HoughLinesP functions of the OpenCV library to search for all straight lines in the image, filter out line segments with a length of less than 100 pixels, calculate the intersection of the extended two lines based on the coordinates of any two points on the line, calculate all lines pairwise, find the coordinate region where most intersection points appear, and take the average to obtain the approximate coordinates as the vanishing point.

[0018] S2-3: For car instances, an insertion region (i.e., an open ground area) is defined based on the dataset annotation information to prevent detection errors caused by large-area occlusion. The coordinates of the instance to be inserted are randomly selected within the insertion region. Each time, a car instance is randomly selected from the instance pool. Then, based on the vanishing point coordinates and the size of existing instances in the original test image, the car instance to be inserted is scaled proportionally, with the pixels of the car instance covering the corresponding background pixels, thus obtaining examples of traffic congestion and uncivilized driving scenarios. For pedestrian, animal, and non-motorized vehicle instances, the division of the insertion region does not consider occlusion of existing instances. After random selection from the instance pool, the instance to be inserted is scaled based on the height of vehicles near the insertion coordinates, thus obtaining examples of pedestrian crossing and illegal parking scenarios. Furthermore, because car instances are larger, only one instance is inserted per example, while pedestrian, animal, and non-motorized vehicle instances are smaller, and depending on the complexity of the scene, 1-4 instances can be inserted per example.

[0019] Furthermore, step S3 includes the following steps:

[0020] S3-1: Based on the distribution pattern of all instances in the original dataset image, the image is divided into 6 regions of 3*2 in both horizontal and vertical dimensions, namely left side, front side, right side, left front side, front far side, and right front side.

[0021] S3-2: Calculate the metric score for each generated test case. The metrics are divided into 4 items, and the formula is as follows:

[0022]

[0023] Where F(S) refers to the overall score of a scenario S, F k (S) refers to the score of scenario S on the k-th metric, λ k λ is the weight of the k-th indicator in the overall score. In the absence of any scenario-specific bias, λ is always taken as 1 / 4.

[0024] F1 scores based on the total number of instances in scenario S, measuring the abundance of instances in extreme scenarios; F2 scores based on the number of instances in 6-point scenario S, measuring the density of instances in the scenario; F3 scores based on the number of categories in 6-point scenario S, measuring the diversity of instance types in the scenario; F4 scores based on the standard deviation of the number of instances in 6-point scenario S, measuring the difference between the instance distribution in the test scenario and the training set scenario. The formulas are as follows:

[0025]

[0026]

[0027]

[0028]

[0029] In formula (2), Refers to the j-th category c that originally existed in the scene. j The total number of instances, n insertion The total number of instances inserted during the generation process is th1, which is the maximum number of instances contained in a single scene in the dataset; in formula (3), n i th2 refers to the total number of instances of a scene in the i-th partition, and th2 is the average number of scene instances in a partition of the dataset; similarly, in formula (4), C i th3 is the average number of scene instance categories in the i-th partition of the dataset; finally, in formula (5), th4 is the maximum standard deviation of the distribution of scene instance numbers in the dataset, and std() refers to the sample standard deviation, the formula of which is as follows:

[0030]

[0031]

[0032] in, It is the sample average.

[0033] After calculating the overall score, for different extreme scenarios, a set of test cases with the highest index scores are selected.

[0034] In step S4, the selected extreme scene examples are input into the multidimensional perception fusion target detection model and a transformation test is performed on the 'Car' class. The transformation relationship is that the detection results of the model on the original test examples and the generated test examples remain unchanged, that is, the original real labels are not modified.

[0035] The beneficial effects of this invention are: it solves the problems of insufficient realism and diversity in existing autonomous driving test sample generation, and difficulty in covering extreme scenarios. It provides a lightweight test metric for autonomous driving object detection, perceiving the scene and efficiently measuring the overall quality of test samples. This improves the testing efficiency of autonomous driving, exposes faulty behaviors of the model, and thus helps to repair model defects. Attached Figure Description

[0036] Figure 1 This is a schematic diagram illustrating an extreme scenario of the method of the present invention;

[0037] Figure 2 This is a flowchart illustrating the method of the present invention;

[0038] Figure 3 This is a partial result diagram of the method of the present invention. Detailed Implementation

[0039] The specific embodiments of the present invention will be further described in detail below with reference to the accompanying drawings.

[0040] Reference Figure 1-3 A testing method for multi-dimensional perception fusion target detection in autonomous driving includes the following steps:

[0041] S1: Generate an instance pool: On the test data of the publicly available autonomous driving dataset, use the YOLACT instance segmentation algorithm to extract various types of instances such as pedestrians and vehicles, and filter the extracted results according to size and quality;

[0042] S2: Generate test cases: Based on the designed extreme traffic scenarios such as pedestrian crossing, traffic congestion, illegal parking, and uncivilized driving, select the corresponding instances from the instance pool, scale the instances according to the vanishing point estimate, and paste them onto the original test data;

[0043] S3: Filter test cases based on test metrics: Calculate test metrics on each generated test case and filter out a set of optimal test cases;

[0044] S4: Test the target detection model: Input the selected extreme scene examples into the multidimensional perception fusion target detection model for transformation test, obtain the average accuracy of the model, and analyze the test results.

[0045] Furthermore, in step S1, the YOLACT model is loaded, the original traffic data is input, and the binocular image matching is used to perform instance segmentation based on the category, angle and other information labeled in the dataset to obtain pedestrians, animals, non-motorized vehicles and cars (forward, reverse and lateral), and incomplete and excessively small or large instances are removed to obtain an instance pool.

[0046] Furthermore, step S2 includes the following steps:

[0047] S2-1: Select the instance to be inserted based on the four defined extreme traffic scenarios: pedestrian crossing scenario selects pedestrian and animal instances, traffic congestion scenario selects forward-moving vehicle instances, illegal parking scenario selects non-motorized vehicle instances, and uncivilized driving scenario selects lateral and reverse-moving vehicle instances.

[0048] S2-2: Estimate the vanishing point on the original test image. Use the Canny and HoughLinesP functions of the OpenCV library to search for all straight lines in the image, filter out line segments with a length of less than 100 pixels, calculate the intersection of the extended two lines based on the coordinates of any two points on the line, calculate all lines pairwise, find the coordinate region where most intersection points appear, and take the average to obtain the approximate coordinates as the vanishing point.

[0049] S2-3: For car instances, an insertion region (i.e., an open ground area) is defined based on the dataset annotation information to prevent detection errors caused by large-area occlusion. The coordinates of the instance to be inserted are randomly selected within the insertion region. Each time, a car instance is randomly selected from the instance pool. Then, based on the vanishing point coordinates and the size of existing instances in the original test image, the car instance to be inserted is scaled proportionally, with the pixels of the car instance covering the corresponding background pixels, thus obtaining examples of traffic congestion and uncivilized driving scenarios. For pedestrian, animal, and non-motorized vehicle instances, the division of the insertion region does not consider occlusion of existing instances. After random selection from the instance pool, the instance to be inserted is scaled based on the height of vehicles near the insertion coordinates, thus obtaining examples of pedestrian crossing and illegal parking scenarios. Furthermore, because car instances are larger, only one instance is inserted per example, while pedestrian, animal, and non-motorized vehicle instances are smaller, and depending on the complexity of the scene, 1-4 instances can be inserted per example.

[0050] Furthermore, step S3 includes the following steps:

[0051] S3-1: Based on the distribution pattern of all instances in the original dataset image, the image is divided into 6 regions of 3*2 in both horizontal and vertical dimensions, namely left side, front side, right side, left front side, front far side, and right front side.

[0052] S3-2: Calculate the metric score for each generated test case. The metrics are divided into 4 items, and the formula is as follows:

[0053]

[0054] Where F(S) refers to the overall score of a scenario S, F k (S) refers to the score of scenario S on the k-th metric, λ k λ is the weight of the k-th indicator in the overall score. In the absence of any scenario-specific bias, λ is always taken as 1 / 4.

[0055] F1 scores based on the total number of instances in scenario S, measuring the abundance of instances in extreme scenarios; F2 scores based on the number of instances in 6-point scenario S, measuring the density of instances in the scenario; F3 scores based on the number of categories in 6-point scenario S, measuring the diversity of instance types in the scenario; F4 scores based on the standard deviation of the number of instances in 6-point scenario S, measuring the difference between the instance distribution in the test scenario and the training set scenario. The formulas are as follows:

[0056]

[0057]

[0058]

[0059]

[0060] In formula (2), Refers to the j-th category c that originally existed in the scene. j The total number of instances, n insertion The total number of instances inserted during the generation process is th1, which is the maximum number of instances contained in a single scene in the dataset; in formula (3), n i th2 refers to the total number of instances of a scene in the i-th partition, and th2 is the average number of scene instances in a partition of the dataset; similarly, in formula (4), C i th3 is the average number of scene instance categories in the i-th partition of the dataset; finally, in formula (5), th4 is the maximum standard deviation of the distribution of scene instance numbers in the dataset, and std() refers to the sample standard deviation, the formula of which is as follows:

[0061]

[0062]

[0063] in, It is the sample average.

[0064] After calculating the overall score, for different extreme scenarios, a set of test cases with the highest index scores are selected.

[0065] In step S4, the selected extreme scene examples are input into the multidimensional perception fusion target detection model and a transformation test is performed on the 'Car' class. The transformation relationship is that the detection results of the model on the original test examples and the generated test examples remain unchanged, that is, the original real labels are not modified.

[0066] Example: An implementation plan for a testing method for a multi-dimensional perception fusion target detection system for autonomous driving is as follows:

[0067] (1) Selecting experimental data and models

[0068] The experimental data selected in this invention is the publicly available autonomous driving dataset KITTI, which contains 3769 binocular test images. The label categories include Car, Van, Truck, Pedestrain, Person (sitting), Cyclist, Tram, and Misc. This experiment focuses on analyzing the test results for the "Car" category.

[0069] The experimental models selected in this invention are the 3D object detection models Stereo RCNN, IDA-3D, and DSGN. The 3D object detection models detect the position, orientation, and distance of vehicles in the input binocular images.

[0070] like Figure 1 The top and bottom images are the left and right eyes, respectively. This experiment uses a pedestrian crossing scene as an example. First, we randomly select several pedestrian instances from the segmented images, assign random coordinates to them within the insertion region of the scene, scale the pedestrian instances to a suitable size, and then insert them into the original image to obtain pedestrian crossing examples, such as... Figure 1 The rectangle is shown. Then, the index is calculated on all generated samples, and the group with the highest index score is selected. The filtered test samples are input into the experimental model to obtain the model's average precision (AP). For example... Figure 3 The figures show the average accuracy of the Stereo R-CNN, IDA-3D, and DSGN models on different extreme traffic scenario examples, with specific values ​​shown in Table 1. The experimental results show that the AP of the models on our test examples significantly decreases compared to the AP on the original test set, including issues with missed detections, orientation errors, and distance errors.

[0071] The above description is merely a preferred embodiment of the present invention and is illustrative rather than restrictive. Any equivalent substitutions or modifications made by those skilled in the art within the scope disclosed in this invention, based on the technical solution and inventive concept of the present invention, shall fall within the protection scope of this invention.

[0072] Table 1:

[0073]

[0074] This invention provides a testing method for a multi-dimensional perception fusion target detection system for autonomous driving, involving autonomous driving, computer vision, target detection, and metamorphic testing. This invention leverages the phenomenon that autonomous driving perception modules are more prone to erroneous behavior in extreme traffic scenarios. It extracts common instances from various traffic scenarios through instance segmentation, and rationally inserts these instances into the original image based on the design of extreme traffic scenarios, generating more visually realistic extreme examples. Then, it calculates a set of lightweight test metrics based on scene perception for the generated examples, selecting a high-quality set of test examples. Finally, these are fed into a multi-dimensional perception fusion target detection model for autonomous driving for testing. This invention provides a testing method for a multi-dimensional perception fusion target detection system for autonomous driving, solving the problems of insufficient realism and diversity of generated test examples and difficulty in covering extreme scenarios in existing autonomous driving tests. It provides a lightweight test metric for autonomous driving target detection, perceiving the scene and efficiently measuring the overall quality of test examples, thus improving the testing efficiency of autonomous driving.

Claims

1. A testing method for multi-dimensional perception fusion target detection in autonomous driving, characterized in that, Includes the following steps: S1: Generate an instance pool: On the test data of the publicly available autonomous driving dataset, use the YOLACT instance segmentation algorithm to extract various types of instances of pedestrians and vehicles, and filter the extracted results according to size and quality; S2: Generate test cases: Based on the designed extreme traffic scenarios such as pedestrian crossing, traffic congestion, illegal parking, and uncivilized driving, select the corresponding instances from the instance pool, scale the instances according to the vanishing point estimate, and paste them onto the original test data; S3: Filter test cases based on test metrics: Calculate test metrics for each generated test case and filter out a set of optimal test cases; including the following steps: S3-1: Based on the distribution pattern of all instances in the original dataset image, the image is divided into 6 regions with a horizontal and vertical dimension of 3*2, namely the left side, front side, right side, left front side, front far side, and right front side. S3-2: Calculate the metric score for each generated test case. The metrics are divided into 4 items, and the formula is as follows: Where F(S) refers to the overall score of a scenario S, F k (S) refers to the score of scenario S on the k-th metric, λ k λ is the weight of the kth indicator in the overall score. In the absence of any scene-specific bias, λ is always taken as 1 / 4. F1 scores based on the total number of instances in scenario S, measuring the abundance of instances in extreme scenarios; F2 scores based on the number of instances in 6-point scenario S, measuring the density of instances in the scenario; F3 scores based on the number of categories in 6-point scenario S, measuring the diversity of instance types in the scenario; F4 scores based on the standard deviation of the number of instances in 6-point scenario S, measuring the difference between the instance distribution in the test scenario and the training set scenario; the formulas are as follows: In formula (2), Refers to the j-th category c that originally existed in the scene. j The total number of instances, n insertion The total number of instances inserted during the generation process is th1, which is the maximum number of instances contained in a single scene in the dataset; in formula (3), n i th2 refers to the total number of instances of a scene in the i-th partition, and th2 is the average number of scene instances in a partition of the dataset; similarly, in formula (4), C i th3 is the average number of scene instance categories in a partition of the dataset; finally, in formula (5), th4 is the maximum standard deviation of the distribution of scene instance numbers in partitions of the dataset, and std() refers to the sample standard deviation, the formula of which is as follows: in, It is the sample mean; After calculating the comprehensive score, for different extreme scenarios, a set of test cases with the highest index scores are selected respectively; S4: Test the target detection model: Input the selected extreme scene examples into the multidimensional perception fusion target detection model for transformation test, obtain the average accuracy of the model, and analyze the test results.

2. The testing method for multi-dimensional perception fusion target detection in autonomous driving as described in claim 1, characterized in that: In step S1, the YOLACT model is loaded, the original traffic data is input, and the binocular images are matched to perform instance segmentation based on the category and angle information labeled in the dataset to obtain pedestrians, animals, non-motorized vehicles, and cars. Incomplete instances and instances that are too small or too large are removed to obtain an instance pool.

3. A test method for multi-dimensional perception fusion target detection for autonomous driving as described in claim 1 or 2, characterized in that: Step S2 includes the following steps: S2-1: Select the instance to paste based on the four defined extreme traffic scenarios: pedestrian crossing scenario, select pedestrian and animal instances; traffic congestion scenario, select forward-moving vehicle instances; illegal parking scenario, select non-motorized vehicle instances; and uncivilized driving scenario, select lateral and reverse-moving vehicle instances. S2-2: Estimate the vanishing point on the original test image. Use the Canny and HoughLinesP functions of the OpenCV library to search for all straight lines in the image, filter out straight line segments with a length of less than 100 pixels, calculate the intersection of the extended two lines based on the coordinates of any two points on the line, calculate all lines pairwise, find the coordinate region where most intersection points appear, and take the average to obtain the approximate coordinates as the vanishing point. S2-3: For car instances, the insertion region is divided based on the dataset annotation information, i.e., an open ground area, to prevent detection errors caused by large-area occlusion. The coordinates to be inserted are randomly selected in the insertion region. Each time, a car instance is randomly selected from the instance pool. Then, the car instance to be inserted is scaled proportionally according to the vanishing point coordinates and the size of the existing instances in the original test image. The pixels of the car instance cover the corresponding background pixels, thus obtaining examples of traffic congestion and uncivilized driving scenarios. For pedestrian, animal, and non-motorized vehicle instances, the division of the insertion region does not consider the occlusion of the original instances. After randomly selecting from the instance pool, the insertion instance is scaled according to the height of the vehicles near the coordinates to be inserted, thus obtaining examples of pedestrian crossing and illegal parking scenarios. In addition, at most one car instance is inserted in each example, while 1-4 pedestrian, animal, and non-motorized vehicle instances can be inserted in each example depending on the complexity of the scene.

4. A test method for multi-dimensional perception fusion target detection in autonomous driving as described in claim 1 or 2, characterized in that, In step S4, the selected extreme scene examples are input into the multidimensional perception fusion target detection model and a transformation test is performed on the 'Car' class. The transformation relationship is that the detection results of the model on the original test examples and the generated test examples remain unchanged, that is, the original real labels are not modified.

Images