Method and related device for generating dangerous test cases for visual perception algorithms

Abstract

The application discloses a kind of dangerous test case generation methods for visual perception algorithm and related equipment, the method includes: obtaining the image captured by automatic driving car vision sensor, clean and mark the collected image, according to the observation data set of no-reference image quality evaluation feature;According to observation data set, and fusion field knowledge, the causal structure diagram of the influence of environmental factor on perception performance is constructed;Key environmental factors are separated from causal structure diagram, and the influence degree of each environmental factor on visual perception performance is quantitatively calculated;Based on the qualitative analysis of the causal relationship between key environmental variables and visual perception performance and the quantitative estimation of causal effect, challenge index is proposed to generate perception-oriented dangerous test cases in batches.The present application has significant advantages in causal relationship mining, challenge index measurement, dangerous scene identification, search space modeling and other aspects, and more accurately evaluates and verifies the performance of automatic driving visual algorithm in complex environment.

Description

Technical Field

[0001] This invention relates to the fields of artificial intelligence and autonomous driving technology, and in particular to a method, system, terminal, and computer-readable storage medium for generating hazardous test cases for visual perception algorithms. Background Technology

[0002] Visual perception is a core foundational module in autonomous driving systems, serving as a crucial component for autonomous vehicles to acquire environmental information and make decisions. Through visual perception, autonomous driving systems can identify road signs, vehicles, pedestrians, and more. Therefore, robust visual perception capabilities are essential for achieving safe and reliable autonomous driving.

[0003] Existing methods typically employ discrete, manually selected test scenarios to evaluate the robustness of visual perception algorithms. These methods rely on experience or expert judgment to select challenging scenarios, such as varying weather conditions, road sign occlusion, and changes in lighting, and then evaluate them through offline testing or simulation. These scenarios are often fixed or limited, failing to comprehensively cover all possible situations. Furthermore, existing technical solutions lack systematic and quantitative evaluation metrics for testing environments, making it difficult to accurately assess the challenges of real-world, complex environments.

[0004] Therefore, existing technologies still need to be improved and developed. Summary of the Invention

[0005] The main objective of this invention is to provide a method, system, terminal, and computer-readable storage medium for generating hazardous test cases for visual perception algorithms. This aims to solve the problem in the prior art that there is a lack of systematic and quantitative test environment evaluation indicators for assessing visual perception capabilities, making it difficult to accurately assess real and complex environments.

[0006] To achieve the above objectives, the present invention provides a method for generating hazardous test cases for visual perception algorithms, the method comprising the following steps:

[0007] Acquire images captured by the vision sensors of autonomous vehicles, clean and label the collected images, and obtain the observation dataset based on the quality assessment features of no-reference images;

[0008] Based on the observed dataset and by integrating domain knowledge, a causal structure diagram of the impact of environmental factors on perception performance is constructed.

[0009] Key environmental factors are obtained from the causal structure diagram, and the degree of influence of each environmental factor on visual perception performance is quantitatively calculated.

[0010] Qualitative analysis of the causal relationship between key environmental variables and visual perception performance, and quantitative estimation of causal effects, are used to generate hazardous test cases for perception in batches based on the challenge index.

[0011] Optionally, the method for generating hazardous test cases for visual perception algorithms, wherein acquiring images captured by the vision sensors of autonomous vehicles, cleaning and labeling the collected images, and obtaining an observation dataset based on no-reference image quality assessment features, specifically includes:

[0012] The environmental variables to be examined are combined into conditions, and a certain number of images are captured for each environmental combination. These images are acquired by the vision sensors of the autonomous vehicle.

[0013] Clean and label the collected images, and extract no-reference image quality assessment features from the images;

[0014] The observation dataset was obtained based on the quality assessment features of the no-reference image.

[0015] The obtained observation dataset is represented as follows: ,in, N It is the number of cases in the observation dataset. It is one of the cases. It is a covariate vector. It is a vector of environmental factors to be examined. It is the performance metric of the visual perception algorithm being tested.

[0016] Optionally, the method for generating hazardous test cases for visual perception algorithms, wherein constructing a causal structure graph of the impact of environmental factors on perception performance based on the observation dataset and incorporating domain knowledge, specifically includes:

[0017] Through a directed acyclic graph This represents a causal relationship between a set of variables, where, Represents a random variable. Indicates causal links between variables;

[0018] Starting from an empty graph, a fast greedy equivalence search algorithm is used to iteratively add or delete edges through a scoring function until convergence, incorporating domain knowledge to construct a causal structure graph.

[0019] Optionally, the method for generating hazardous test cases for visual perception algorithms, wherein the step of employing a fast greedy equivalence search algorithm to start from an empty graph and iteratively add or delete edges through a scoring function until convergence specifically includes:

[0020] In the observation dataset Get an empty image

[0021] Iterate through each pair of variables: , as well as Add an edge between the variables;

[0022] For each potential edge, test whether adding the potential edge to the current graph results in a cycle graph. If it does, skip the potential edge.

[0023] For each potential edge that will not lead to a cycle graph, test whether adding the potential edge improves the scoring metric; if it does, then add the potential edge.

[0024] Iteratively process all edges in the current graph, check whether potential edges can be removed and the scoring metric improved without opening the graph; if so, perform the edge removal operation.

[0025] Repeat the iterative process until the scoring metric no longer improves;

[0026] Specifically, checking whether to remove potential edges is done by calculating the Bayesian information criterion. Judged by whether the value increases:

[0027]

[0028] in, Represents a directed acyclic graph The parameters, It is a constant that takes the value 1. It is the number of parameters in a directed acyclic graph. n Indicates the sample size of the observed data;

[0029] If the current causal structure graph An increase in the value indicates the removal of potential edges.

[0030] The process of acquiring key environmental factors from the observed dataset and quantitatively calculating the impact of each environmental factor on visual perception performance specifically includes:

[0031] The difference between potential intervention outcomes and control outcomes indicates the individual treatment effect. :

[0032]

[0033] in,, This indicates the potential intervention outcome for this record in the observation dataset when the fog intensity is set to light. This indicates the control results under conditions without fog; Indicates the first in the observation data iThe record shows a fog intensity of 0. Indicates the first i The recorded fog intensity was light;

[0034] The average causal effect of light fog on the mean of average precision. Represented as:

[0035]

[0036] in, Represents the mathematical expectation.

[0037] Selection bias exists when the treatment and control groups are not randomly assigned. :

[0038]

[0039] in, and These are the sample sizes for the treatment group and the control group, respectively.

[0040] Estimating causal effects using specialized statistical learning models:

[0041] The results of the treatment group and the control group were estimated using arbitrary statistical regression methods, and are expressed as follows: and ;

[0042] In the control group, the difference was defined as: , recorded as In the treatment group, the difference was defined as: , recorded as ; and The therapeutic effect is referred to as filling;

[0043] Use any regression method based on the new data set ( )and( Calculate the therapeutic effect ,in, This represents the estimated results for the control group. This represents the estimation results for the control group. and These are covariates for the control group and the treatment group, respectively, by... and Combining these, we obtain the final individual estimate of causal effect. :

[0044]

[0045] in, It is a weighting function, usually taken as the score of the tendency value;

[0046] Through calculation The mathematical expectation is used to determine this.

[0047] Optionally, the method for generating hazardous test cases for visual perception algorithms, wherein the qualitative analysis of the causal relationship between key environmental variables and visual perception performance, and the quantitative estimation of causal effects, and the generation of hazardous test cases for perception in batches based on a challenge index, specifically includes:

[0048] Define the challenge index:

[0049]

[0050] in, It is the relative weight among multiple key environmental impact factors after multiple screenings, reflecting the quantitative contribution of different environmental factors to the perception results; It is the number of key environmental factors; It is corresponding The normalized causal effect, if Applying intervention This makes the causal effect expressed as ,but:

[0051]

[0052] in, It is an environmental node Possible types of intervention;

[0053] The generated driving score is below the hazard score threshold. For dangerous test cases, the algorithm test score is calculated as follows:

[0054]

[0055] in, This indicates the test score of the system under test in this scenario; It is the first The percentage of the driving path of the tested system relative to the planned path in each test scenario; Corresponding to the violation penalty items of the tested system;

[0056] By combining proxy models and causal knowledge to maximize the acquisition function, and using the combination of test parameters that maximize the acquisition function as a single test case:

[0057]

[0058] in, and These are the predicted driving scores and variance of the proxy model, respectively; This represents the lowest driving score predicted by the proxy model; It is the probability density function of the standard normal distribution; It is the defined challenge index; It is the minimum Euclidean space distance between candidate test cases and explored test cases;

[0059] If the current experiment count Greater than the preset number of search iterations At the same time, generate batch sets of risk test cases for awareness. .

[0060] Optionally, in the method for generating hazardous test cases for visual perception algorithms, the hazardous test cases are a test environment with a set of defined values.

[0061] Furthermore, to achieve the above objectives, the present invention also provides a hazardous test case generation system for visual perception algorithms, wherein the hazardous test case generation system for visual perception algorithms includes:

[0062] The observation dataset collection module is used to acquire images captured by the vision sensors of autonomous vehicles, clean and label the collected images, and obtain the observation dataset based on the no-reference image quality assessment features;

[0063] The causal structure graph construction module is used to construct a causal structure graph of the influence of environmental factors on perception performance based on the observation dataset and by integrating domain knowledge.

[0064] The causal effect estimation module is used to obtain key environmental factors in the causal structure diagram and quantitatively calculate the degree of influence of each environmental factor on visual perception performance.

[0065] The hazardous test case generation module is used for qualitative analysis of the causal relationship between key environmental variables and visual perception performance, as well as quantitative estimation of causal effects, and to generate hazardous test cases for perception in batches based on the challenge index.

[0066] Furthermore, to achieve the above objectives, the present invention also provides a terminal, wherein the terminal includes: a memory, a processor, and a dangerous test case generation program for visual perception algorithms stored in the memory and executable on the processor, wherein when the dangerous test case generation program for visual perception algorithms is executed by the processor, it implements the steps of the dangerous test case generation method for visual perception algorithms as described above.

[0067] Furthermore, to achieve the above objectives, the present invention also provides a computer-readable storage medium, wherein the computer-readable storage medium stores a dangerous test case generation program for visual perception algorithms, and when the dangerous test case generation program for visual perception algorithms is executed by a processor, it implements the steps of the dangerous test case generation method for visual perception algorithms as described above.

[0068] This invention acquires images captured by the vision sensors of an autonomous vehicle, cleans and labels the collected images, and obtains an observation dataset based on no-reference image quality assessment features. Based on the observation dataset and incorporating domain knowledge, a causal structure diagram of the impact of environmental factors on perception performance is constructed. Key environmental factors are identified from the causal structure diagram, and the degree of influence of each environmental factor on visual perception performance is quantitatively calculated. Based on the qualitative analysis of the causal relationship between key environmental variables and visual perception performance, and the quantitative estimation of causal effects, dangerous test cases for perception are generated in batches based on a challenge index. This invention, based on the quantitative estimation and normalization calculation of causal effects, can effectively measure the degree of challenge posed by environmental conditions to the vision algorithm. By introducing causal knowledge from the challenge index, it guides the search strategy to generate more challenging test cases. Combining the challenge index and driving scores, it can effectively determine whether the performance of the autonomous driving system in a specific scenario is below a predetermined threshold, thereby identifying potential dangerous situations. Attached Figure Description

[0069] Figure 1 This is a flowchart of a preferred embodiment of the method for generating hazardous test cases for visual perception algorithms according to the present invention;

[0070] Figure 2 This is a schematic diagram of the principle framework for the analysis, screening, and test scenario generation of key environmental elements for autonomous driving visual perception algorithms in a preferred embodiment of the present invention's method for generating hazardous test cases for visual perception algorithms.

[0071] Figure 3 This is a schematic diagram of the observation dataset construction process in a preferred embodiment of the dangerous test case generation method for visual perception algorithms of the present invention;

[0072] Figure 4 This is a schematic diagram of the causal structure graph construction process in a preferred embodiment of the dangerous test case generation method for visual perception algorithms of the present invention;

[0073] Figure 5 This is a schematic diagram of the challenge index-guided search process in a preferred embodiment of the dangerous test case generation method for visual perception algorithms of the present invention;

[0074] Figure 6 This is a schematic diagram of a preferred embodiment of the hazardous test case generation system for visual perception algorithms of the present invention;

[0075] Figure 7 This is a schematic diagram of the operating environment of a preferred embodiment of the terminal of the present invention. Detailed Implementation

[0076] Autonomous vehicles are of great significance for improving traffic safety, enhancing traffic efficiency, and achieving low-carbon travel, and have become a strategic direction for the transformation and upgrading of the global automotive industry. However, the frequent accidents involving autonomous vehicles in recent years have severely damaged public confidence and have become a major obstacle to the commercialization of the autonomous driving industry, especially since the robustness testing of visual perception algorithms has long been neglected.

[0077] Existing testing methods for visual perception can be roughly divided into the following four categories. (1) Data augmentation, which is a technique that attempts to expand the training set of perception algorithms. For example, a car manufacturer adopted a crowdsourcing-based data collection strategy to obtain training data from actual road driving. The car manufacturer's vehicles are equipped with a variety of sensors that can record a variety of data during vehicle driving. After anonymization, the car owners choose whether to participate in the data sharing plan and upload it to the car manufacturer's server, forming a large and diverse dataset. However, dataset expansion is essentially a blind exploration of sparse dangerous scenarios in a high-dimensional test space, which is inefficient. (2) Dangerous scenario search based on neuron coverage: When the test input passes through a neuron and causes a specific output, the neuron is considered to be covered. For example, a white-box testing framework called DeepXplore uses neuron coverage to detect behavioral inconsistencies between tested algorithms. Based on the research of DeepXplore, a greedy search method is provided to increase neuron coverage, further promoting the concept of neuron coverage and defining a series of neuron coverage indicators. Recent studies have combined fuzzy concepts and optimization algorithms with these specific neuron coverage metrics to establish various testing frameworks, such as DLFuzz, DeepHunter, TensorFuzz, and Test4Deep. However, neuron coverage testing is still in the early stages of research, especially since it is questioned due to its significant deviation from traditional software testing concepts. (3) Image transformation generation: Generative adversarial networks are used to transform the original input to adverse weather conditions, with representative schemes such as DeepRoad and style transfer algorithms based on neural networks. However, image transformation techniques lack clear interpretability, and the generation efficiency of synthetic images is low. (4) Adversarial attacks: This is a malicious image attack method that affects the output of the model by adding perturbations to the visual input. Early research on adversarial attacks mainly focused on attacks that perturb all pixels. Since perturbations do not occur on all pixels in real-world scenarios, the current research focus is on patch attacks targeting specific regions of the image. For example, a method for generating printable adversarial billboards has been proposed, which cause visual perception to produce incorrect results under dynamically changing driving conditions. It is worth noting that adversarial attacks lack mapping to the real environment.

[0078] In general, most existing methods focus on inducing erroneous behavior in deep neural network-based vision algorithms, while neglecting the environmental factors that cause these errors.

[0079] Existing testing technologies for autonomous driving vision modules have the following drawbacks:

[0080] (1) Lack of interpretability: Some methods, such as neuron overlay and adversarial attacks, cannot reasonably explain the mapping relationship between the discovered vulnerabilities and the real environment, thus failing to effectively guide algorithm iteration.

[0081] (2) High test resource consumption: Image conversion is an inefficient and resource-intensive testing method, which usually requires a lot of computing resources to extract diverse adversarial weather features. In addition, due to the lack of screening for environmental factors that actually cause a decline in visual perception performance, only exhaustive environmental combinations can be enumerated, which exacerbates the computational burden of image synthesis.

[0082] For a long time, the industry has lacked a good solution to identify key environmental factors in autonomous driving visual perception, and existing technical solutions cannot effectively generate high-risk perception test scenarios.

[0083] This invention proposes a method for analyzing, screening, and generating test scenarios for key environmental elements in visual perception algorithms for autonomous driving. The method includes: (1) collecting observation datasets; (2) constructing a structural causal graph to screen key environmental variables; (3) quantitatively estimating the degree of influence of each environmental factor on visual perception performance and proposing an alternative metric for evaluating the deployment environment of the algorithm based on the challenge index; and (4) generating perception risk test scenarios online based on the challenge index.

[0084] To make the objectives, technical solutions, and advantages of this invention clearer and more explicit, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.

[0085] The method for generating hazardous test cases for visual perception algorithms described in the preferred embodiment of the present invention, such as... Figure 1 and Figure 2 As shown, the method for generating hazardous test cases for visual perception algorithms includes the following steps:

[0086] Step S10: Acquire images captured by the vision sensors of the autonomous vehicle, clean and label the collected images, and obtain the observation dataset based on the no-reference image quality assessment features.

[0087] This invention uses causal inference theory to isolate key environmental factors from a high-dimensional parameter space. Typically, this requires obtaining randomized controlled trial data, where the environmental variables to be examined are randomly assigned to either the control or analysis group. However, obtaining randomized controlled trial data is costly or even impractical.

[0088] Therefore, this invention proposes using the principle of combinatorial testing to collect observation datasets. Specifically, the environmental variables to be examined are combined into conditions. Combinatorial testing, also known as combinatorial coverage testing, is a software testing technique aimed at efficiently testing the interaction between different input parameters or configuration settings in a system. In this invention, the principle of combinatorial testing is used to randomly obtain as many possible environmental combinations as possible. The advantage of doing so is that it balances the coverage of collected data with cost savings.

[0089] like Figure 3 As shown, a certain number of images (images captured by the vision sensors of autonomous vehicles) are then captured for each combination. Subsequently, the collected data is cleaned and labeled, and no-reference image quality assessment features are extracted from the images to measure the impact of various environmental factors on image quality. No-reference image quality assessment features are a metric used to measure image quality without the need for reference images or subjective evaluation. They are based on the features and statistical information of the image itself and estimate the image quality by analyzing the low-level features, perceptual features, or structural features of the image.

[0090] Extracting quality metrics from no-reference images is an intermediate step in the construction of the entire observational dataset. Its purpose is to help better understand how environmental nodes affect the quality of the images in subsequent causal analysis, ultimately compromising the performance of visual algorithms.

[0091] The process of constructing the observation dataset is as follows: Figure 3 As shown, the obtained observation dataset is represented as follows: ,in, N It is the number of cases in the observation dataset. It is one of the cases. It is a covariate vector. It is a vector of environmental factors to be examined. It is the performance metric of the visual perception algorithm being tested.

[0092] Step S20: Based on the observation dataset and by integrating domain knowledge, construct a causal structure diagram of the impact of environmental factors on perception performance.

[0093] Specifically, based on the observation dataset collected in step S10, this invention uses a causal structure discovery method (a method for inferring causal relationships between variables; it aims to identify causal relationships between variables by observing statistical patterns and dependencies in data, revealing the causal structure between phenomena. The goal of the causal structure discovery method is to discover causal models from data, not just observed correlations. It is based on the core principle of causality, that is, the occurrence of an event is caused by other events, not just related or simultaneous. Commonly used causal structure discovery methods typically use statistical methods and machine learning algorithms to infer causal relationships between variables by modeling and reasoning about data. Currently, causal structure discovery methods have wide applications in many fields, including medical research, social sciences, and economics. However, its application in the field of autonomous driving perception testing has not yet been found) to build a structural causal graph represented by a Bayesian diagram. The causal structure discovery method focuses on describing the causal relationships between a set of variables through a directed acyclic graph. To indicate, among which Represents a random variable. To represent the causal link between variables, there are many ways to express causal structures. One of the most commonly used types is to use graph structures. In the graph, the nodes correspond to variables in the observational data (including environmental variables, image quality index variables, and visual perception detection accuracy), and the edge is a directed arrow. A→B can be simply understood as A having a direct causal effect on B, and A being the cause of B.

[0094] For example, This indicates that fog has a direct causal impact on visual perception algorithms. Indicates the performance metrics of visual algorithms; In this invention, the Fast Greedy Equivalence Search (FGES) algorithm is employed. This algorithm is a parallelized and optimized version of the Greedy Equivalence Search (GES) algorithm. The FGES algorithm starts with an empty graph and iteratively adds or removes edges using a scoring function until convergence, as shown below. Figure 4 As shown.

[0095] like Figure 4 As shown, the detailed steps for building a cause-and-effect diagram are as follows:

[0096] Step (1): In the observation dataset Get an empty image .

[0097] Step (2): Iterate through each pair of variables: , as well as Add an edge between variables (the edge represents a causal relationship).

[0098] Step (3): For each potential edge (a possible causal relationship, but further confirmation is needed in subsequent steps; if the potential edge improves the scoring metric)... If a potential edge is found to be a valid edge and does not violate the rules of the graph structure, then it is considered a valid edge. The test is conducted to see if adding the potential edge to the current graph results in a cycle graph. If it results in a cycle graph (a cycle graph is a graph structure in which one or more loops exist), then the potential edge is skipped.

[0099] Step (4): For each potential edge that will not lead to a cycle graph, test whether adding the potential edge will improve the scoring metric. If it will improve the scoring metric, then add the potential edge.

[0100] In the scoring-based causal graph structure discovery method, scoring metrics are used to measure the fit or quality of potential causal graph models. They are used to compare the relative merits of different models and help determine the optimal causal graph structure.

[0101] Determining whether adding latent edges improves the scoring metrics typically involves model comparison and analysis. In causal graph structure discovery, the merits of different graph structures can be judged by comparing their scoring metrics. If adding latent edges improves the scoring metrics, then the addition of those edges can be considered to have improved the model's fit or quality.

[0102] Step (5): Iteratively process all edges in the current graph, check whether potential edges are removed and the scoring metric is improved without opening the graph, and if so, perform the edge removal operation;

[0103] Repeat steps (2)-(5) until the scoring indicators no longer improve.

[0104] In this invention, the determination of whether to remove potential edges is achieved by calculating the Bayesian Information Criterion (BIC). Judged by whether the value increases:

[0105]

[0106] in, Represents a directed acyclic graph The parameters, It is a constant that takes the value 1. It is the number of parameters in a directed acyclic graph. n This indicates the sample size of the observed data.

[0107] If the current causal structure graph An increase in the value indicates the removal of potential edges.

[0108] Generally speaking, higher The value indicates the identified graphical causal model. It can explain causal relationships more effectively without being overly complex (the construction of a causal graph structure is an iterative process, equivalent to finding a structure in which...) (The highest score among previously discovered structures is sufficient). It is worth noting that this invention does not rely entirely on data-driven causal structure discovery paradigms, such as... Figure 4 As shown, this invention incorporates domain knowledge to construct a causal structure diagram of the impact of environmental factors on perception performance.

[0109] Step S30: Obtain key environmental factors from the observation dataset and quantitatively calculate the degree of influence of each environmental factor on visual perception performance.

[0110] Specifically, this step aims to estimate the perceived performance caused by key environmental factors. The degree of degradation, or "causal effect," is defined in the research paradigm of the potential outcome framework. The term "causal effect" refers to the relationship between "treatment" and "no treatment" in a given observational dataset regarding the outcome. The differences in their effects. For example, taking "fog" in the environment as an example, where "treatment" refers to different fog intensity levels, the possible different treatment levels are... express, It represents all possible values ​​of a variable in the graph structure. As illustrated in this example, fog nodes have different intensities, for example, they can be divided into 5 intensities from none to one.

[0111] in, Indicates the first in the observation data The record shows a fog intensity of 0. Indicates the first The recorded fog intensity was light; This indicates the potential intervention outcome for this record in the observation dataset when the fog intensity is set to light. This indicates the control results under conditions without fog; here... The performance of visual perception algorithms is measured, for example, by the mean average precision (mAP). The difference between the two groups (control and control) is the individual treatment effect (ITE).

[0112]

[0113] According to the causal graph model described in step 20 of this invention, non-critical environmental factors can be considered as a vector of covariates: These are unaffected by treatment. Image quality metrics, however, are post-treatment variables, and their outcomes are influenced by the treatment. The average causal effect (ACE) of mild fog on mAP can be further expressed as:

[0114]

[0115] in, It represents the mathematical expectation.

[0116] However, in practice, it is not possible to retrieve the results of both the intervention and control groups simultaneously from the observational data. Therefore, it is not possible to simply subtract the mAP of the intervention group from the mAP of the control group, which may lead to selection bias.

[0117] That is, when the treatment group and the control group are not randomly assigned, selection bias exists. :

[0118]

[0119] in, and These are the sample sizes for the treatment group and the control group, respectively.

[0120] When the treatment and control groups are not randomly assigned, selection bias can occur, leading to systematic differences in covariates between the two groups, which may result in adverse reactions. The estimates are inaccurate. Therefore, this invention proposes using a dedicated statistical learning model to estimate causal effects, taking the meta-learning algorithm X-learner as an example. This algorithm has been proven to produce accurate inferences even in the presence of confounding variables and unobserved heterogeneity. X-learner uses a three-stage process to estimate the treatment effect, as follows:

[0121] Step (11), Simulate treatment / control outcomes: Use any statistical regression method, such as random forest, to estimate the outcomes for the treatment group and the control group, as shown below. and .

[0122] Step (12), Filling in the treatment effect: Filling in the treatment effect between the treated / control group individuals. In the control group, the difference is defined as: , recorded as In the treatment group, the difference was defined as: , recorded as ; and This is known as the filling therapeutic effect.

[0123] Step (13), combine treatment effects to generate a final estimate: use any regression method based on the new data set ( )and( Calculate the therapeutic effect ,in, This represents the estimated results for the control group. This represents the estimation results for the control group. and These are covariates for the control group and the treatment group, respectively, by... and Combining these, we obtain the final individual estimate of causal effect. :

[0124]

[0125] in,, It is a weighting function, usually taken as the propensity score; thus, Through calculation The mathematical expectation is used to determine this.

[0126] Step S40: Qualitative analysis of the causal relationship between key environmental variables and visual perception performance, and quantitative estimation of causal effects, to generate batch of perception-oriented hazardous test cases based on the challenge index.

[0127] Specifically, based on the qualitative analysis of the causal relationship between key environmental variables and visual perception performance, and the quantitative estimation of the causal effect, this invention further designs a novel alternative evaluation index for assessing the challenging environmental conditions for deploying visual algorithms, which this invention refers to as the "Challenge Index." This Challenge Index is defined by the following equation:

[0128]

[0129] in, It is the relative weight among multiple key environmental impact factors after multiple screenings, reflecting the quantitative contribution of different environmental factors to the perception results; It is the number of key environmental factors; It is corresponding The normalized causal effect, if Applying intervention This makes the causal effect express as ,but:

[0130]

[0131] in, It is an environmental node Possible types of intervention.

[0132] The "Challenge Index" is the first in the industry to offer a novel method for predicting the deployment difficulty of visual algorithms based on causal knowledge. By collecting training and testing samples under key environmental combinations, it can significantly accelerate algorithm validation and iteration. It's important to note that the vectors in the Challenge Index... This reflects the quantitative contribution of different environmental factors to the perceived results. To determine these relative weights, the entropy weighting method is recommended.

[0133] Furthermore, this invention proposes a "challenge index-guided search" method for generating hazardous test scenarios to verify the safety of vision-based end-to-end autonomous driving solutions. Any search-based scenario generation method needs to address the following key questions: what is the search space, and what search strategy is used to generate test cases? Considering that the vast search space makes existing search technologies inefficient and indiscriminate, this invention first selects key environmental factors filtered through causal inference as dimensions of the search space, reducing numerous non-causal test dimensions. Second, a surrogate model is used to probabilistically model the search space, utilizing spatial correlation to capture the similarity between explored test samples and nearby unexplored areas, and establishing result prediction and uncertainty estimation. In addition, by integrating discovered causal knowledge (challenge index), the search strategy is further guided towards more challenging key areas, thereby generating perception-oriented hazardous test scenarios (i.e., hazardous test cases) in batches. The online scenario search process is as follows: Figure 5 As shown.

[0134] Figure 5 middle, This is the current experiment count; It is the pre-set number of search iterations; and These are the test score and hazard score threshold of the tested system in this scenario, respectively. It should be noted that the goal of this invention is to generate a driving score as low as possible below a specific threshold (…). In dangerous scenarios, the algorithm test score is calculated as follows:

[0135]

[0136] in, It is the first i The percentage of the driving path of the tested system relative to the planned path in each test scenario; The penalties for violations by the tested system include route deviation, collision, and simulation timeout. For example... Figure 5As shown, the search method of the present invention dynamically adjusts the exploration area in the test space online based on the test results of the previous test case. Specifically, it combines the surrogate model and causal knowledge to maximize the following acquisition function, and combines the test parameters with the largest acquisition function as a test case (a test case is a set of test environments with certain values, such as the intensity of sunlight, the concentration of fog, etc., and formula (9) only selects the one with the largest acquisition from the possible possibilities as a test case):

[0137]

[0138] in, and These are the predicted driving scores and variance of the alternative model, respectively; This represents the lowest driving score predicted by the proxy model; It is the probability density function of the standard normal distribution; It is the challenge index, calculated using formulas (6) and (7); It is the minimum Euclidean space distance between candidate test cases and explored test cases; if the current experiment count Greater than the preset number of search iterations At the same time, generate batch sets of risk test cases for awareness. .

[0139] The key points of this invention are: proposing a testing method for visual perception algorithms in autonomous driving, specifically including qualitative research on causal relationships and quantitative estimation of causal effects; proposing a new evaluation index, namely the challenge index, to measure the degree of challenge posed by environmental conditions to the visual algorithm; and proposing a challenge index-guided search method to generate risk test scenarios to verify the performance of end-to-end visual algorithms under various challenging environments. In summary, this invention can help accelerate the verification and iteration of visual perception algorithms.

[0140] The main protection points of this invention are as follows:

[0141] (1) Calculation method of challenge index: The method of calculating challenge index proposed in this invention is based on quantitative estimation and normalization calculation of causal effect, which can effectively measure the degree of challenge of environmental conditions to visual algorithms.

[0142] (2) Dangerous scene identification method: Protect the method of using challenge index and driving score to identify dangerous scenes. By combining challenge index and driving score, this method can effectively determine whether the performance of autonomous driving system in a specific scene is lower than a predetermined threshold, thereby identifying potential dangerous situations.

[0143] (3) Algorithm flow and related algorithm details: Protect the search space modeling method based on the surrogate model, which uses spatial correlation to establish a potential test sample space model and guides the search strategy to generate more challenging test cases by introducing causal knowledge of the challenge index; Protect the algorithm flow and related algorithm details proposed in this invention, including the specific steps of the challenge index to guide the search, the construction and updating methods of the surrogate model, etc. These details are of great technical and innovative significance for implementing the method of this invention.

[0144] Compared with the prior art, the present invention has the following advantages (i.e., beneficial effects):

[0145] (1) Deeply explore causal relationships: This invention is the first in the industry to propose the application of causal inference theory to deeply explore the causal relationship between environmental conditions and visual algorithm performance, so as to more accurately evaluate the impact of different environmental factors on the algorithm, which is more scientific and reliable than existing technologies.

[0146] (2) Innovative measurement of challenge index: This invention proposes a challenge index as a new indicator for evaluating the degree of challenge of an algorithm under various environmental conditions. This indicator combines the normalization calculation of causal effects and weight allocation, which can more comprehensively measure the challenge of the environment to the algorithm and more accurately reflect the needs of actual application scenarios compared with existing technologies.

[0147] (3) Search space modeling and test case generation: The search space modeling method based on the surrogate model in this invention can generate challenging test cases more efficiently, which helps to more comprehensively verify and evaluate the robustness and performance of the visual algorithm. It is more efficient and feasible than existing technologies.

[0148] (4) Effective Hazardous Scene Identification: This invention utilizes causal reasoning to rationally screen key factors affecting visual perception algorithms and pioneers the introduction of the concept of a "challenge index" to quantify the causal effects of key influencing factors. The efficient hazardous scene generation method designed in this invention, by combining the proposed challenge index and driving scores, can quickly and accurately identify the performance of autonomous driving systems in hazardous scenes and provide timely warnings of potential dangers. This method can generate more challenging test scenarios than existing technologies, providing crucial assurance for the safety and reliability of autonomous driving systems.

[0149] In summary, compared with existing technologies, this invention has significant advantages in causal relationship mining, challenge index measurement, dangerous scene identification, and search space modeling, and can more accurately evaluate and verify the performance of visual algorithms in complex environments.

[0150] Furthermore, the method of this invention has undergone extensive experimentation, and compared to random search and random neighborhood search, the method of this invention has discovered 3.75 to 12.3 times and 1.5 to 9.25 times more dangerous scenarios, respectively. Moreover, the generated scenarios maintain a balance between test coverage and hazard, posing a greater threat to the system under test.

[0151] Furthermore, such as Figure 6 As shown, based on the above-described method for generating hazardous test cases for visual perception algorithms, this invention also provides a system for generating hazardous test cases for visual perception algorithms, wherein the system includes:

[0152] The observation dataset collection module 51 is used to acquire images captured by the vision sensors of autonomous vehicles, clean and label the collected images, and obtain the observation dataset based on the no-reference image quality assessment features.

[0153] The causal structure graph construction module 52 is used to construct a causal structure graph of the influence of environmental factors on perception performance based on the observation dataset and by integrating domain knowledge.

[0154] The causal effect estimation module 53 is used to obtain key environmental factors from the observation dataset and quantitatively calculate the degree of influence of each environmental factor on visual perception performance.

[0155] The hazardous test case generation module 54 is used for qualitative analysis of the causal relationship between key environmental variables and visual perception performance, as well as quantitative estimation of causal effects, and to generate hazardous test cases for perception in batches based on the challenge index.

[0156] Furthermore, such as Figure 7 As shown, based on the above-mentioned dangerous test case generation method and system for visual perception algorithms, the present invention also provides a terminal, which includes a processor 10, a memory 20 and a display 30. Figure 7 Only some of the terminal components are shown; however, it should be understood that it is not required to implement all of the components shown, and more or fewer components may be implemented instead.

[0157] In some embodiments, the memory 20 may be an internal storage unit of the terminal, such as a hard disk or memory. In other embodiments, the memory 20 may be an external storage device of the terminal, such as a plug-in hard disk, smart media card (SMC), secure digital card (SD), flash card, etc. Further, the memory 20 may include both internal and external storage devices. The memory 20 is used to store application software and various types of data installed on the terminal, such as program code installed on the terminal. The memory 20 can also be used to temporarily store data that has been output or will be output. In one embodiment, the memory 20 stores a hazardous test case generation program 40 for visual perception algorithms, which can be executed by the processor 10 to implement the hazardous test case generation method for visual perception algorithms in this application.

[0158] In some embodiments, the processor 10 may be a central processing unit (CPU), a microprocessor, or other data processing chip, used to run program code stored in the memory 20 or process data, such as executing the dangerous test case generation method for visual perception algorithms.

[0159] In some embodiments, the display 30 may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, or an OLED (Organic Light-Emitting Diode) touchscreen. The display 30 is used to display information on the terminal and to display a visual user interface. The components 10-30 of the terminal communicate with each other via a system bus.

[0160] In one embodiment, when the processor 10 executes the hazardous test case generation program 40 for visual perception algorithms in the memory 20, it implements the steps of the hazardous test case generation method for visual perception algorithms as described above.

[0161] The present invention also provides a computer-readable storage medium, wherein the computer-readable storage medium stores a dangerous test case generation program for visual perception algorithms, and the dangerous test case generation program for visual perception algorithms implements the steps of the dangerous test case generation method for visual perception algorithms as described above when executed by a processor.

[0162] In summary, this invention provides a method and related equipment for generating hazardous test cases for visual perception algorithms. The method includes: acquiring images captured by the visual sensors of an autonomous vehicle; cleaning and labeling the collected images; obtaining an observation dataset based on no-reference image quality assessment features; constructing a causal structure graph of the influence of environmental factors on perception performance based on the observation dataset and incorporating domain knowledge; identifying key environmental factors in the causal structure graph and quantitatively calculating the degree of influence of each environmental factor on visual perception performance; and generating hazardous test cases for perception in batches based on a challenge index, based on a qualitative analysis of the causal relationship between key environmental variables and visual perception performance, and a quantitative estimation of causal effects. This invention, based on the quantitative estimation and normalization calculation of causal effects, can effectively measure the degree of challenge posed by environmental conditions to visual algorithms. By introducing causal knowledge from the challenge index, it guides search strategies to generate more challenging test cases. Combining the challenge index and driving scores, it can effectively determine whether the performance of an autonomous driving system in a specific scenario is below a predetermined threshold, thereby identifying potential hazardous situations.

[0163] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or terminal that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or terminal. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or terminal that includes that element.

[0164] Of course, those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware (such as a processor, controller, etc.). The program can be stored in a computer-readable storage medium, and when executed, it can include the processes described in the above method embodiments. The computer-readable storage medium can be a memory, magnetic disk, optical disk, etc.

[0165] It should be understood that the application of the present invention is not limited to the examples above. Those skilled in the art can make improvements or modifications based on the above description, and all such improvements and modifications should fall within the protection scope of the appended claims.

Claims

1. A method for generating test cases for visual perception algorithm oriented hazards, characterized in that, The method for generating hazardous test cases for visual perception algorithms includes: Acquire images captured by the vision sensors of autonomous vehicles, clean and label the collected images, and obtain the observation dataset based on the quality assessment features of no-reference images; Based on the observed dataset and by integrating domain knowledge, a causal structure diagram of the impact of environmental factors on perception performance is constructed. Key environmental factors are obtained from the causal structure diagram, and the degree of influence of each environmental factor on visual perception performance is quantitatively calculated. Qualitative analysis of the causal relationship between key environmental variables and visual perception performance, along with quantitative estimation of causal effects, is used to generate batches of hazardous test cases for perception based on a challenge index. Specifically, this includes: Define the challenge index: wherein, is the relative weight between the plurality of key environmental impact factors after screening, reflecting the quantitative contribution of different environmental factors to the perception result; is the number of key environmental factors; is the normalized causal effect of , if intervention is applied , so that the causal effect is expressed as , then: wherein, is an environmental node Possible intervention categories; Generating a driving score below a dangerous score threshold a dangerous test case whose test score of the measured autonomous driving perception algorithm under the given test scenario is: wherein, represents a test score of the measured system under the scenario; is the percentage of the driving path of the measured system in the planning path in the th test scenario; corresponds to the violation penalty term of the measured system; By combining proxy models and causal knowledge, the acquisition function is maximized, and the combination of test parameters that maximizes the acquisition function is used as a test case. The acquisition function is defined as follows: wherein, and are the predicted driving score and variance of the proxy model, respectively; denotes the lowest driving score predicted by the proxy model; is the probability density function of the standard normal distribution; is the defined challenge index; is the minimum Euclidean spatial distance between the candidate test case and the already explored test cases; Current experiment count greater than a preset search iteration number a batch of perception-oriented dangerous test case sets are generated .

2. The visual perception algorithm oriented hazard test case generation method according to claim 1, characterized in that, The process of acquiring images captured by the vision sensors of autonomous vehicles, cleaning and labeling the collected images, and obtaining an observation dataset based on no-reference image quality assessment features specifically includes: The environmental variables to be examined are combined into conditions, and a certain number of images are captured for each environmental combination. These images are acquired by the vision sensors of the autonomous vehicle. Clean and label the collected images, and extract no-reference image quality assessment features from the images; The observation dataset was obtained based on the quality assessment features of the no-reference image. wherein the obtained observation dataset is denoted as wherein, N is the number of cases in the observation dataset, is a certain one of the number of cases, is a covariate vector, is an environmental factor vector to be investigated, is a visual perception algorithm performance indicator to be measured.

3. The visual perception algorithm oriented hazard test case generation method according to claim 2, characterized in that, The step of constructing a causal structure diagram of the impact of environmental factors on perception performance based on the observed dataset and integrating domain knowledge specifically includes: through a directed acyclic graph representing causal relationships between a set of variables, wherein, representing random variables, representing causal links between variables; Starting from an empty graph, a fast greedy equivalence search algorithm is used to iteratively add or delete edges through a scoring function until convergence, and a causal structure graph is constructed by combining domain knowledge.

4. The visual perception algorithm oriented hazard test case generation method according to claim 3, characterized in that, The method employs a fast greedy equivalence search algorithm, starting from an empty graph and iteratively adding or deleting edges using a score function until convergence. Specifically, this includes: In the observed data set an empty graph is obtained Iterate over each pair of variables: , and add an edge between the variables; For each potential edge, test whether adding the potential edge to the current graph results in a cycle graph. If it does, skip the potential edge. For each potential edge that will not lead to a cycle in the graph, test whether adding each potential edge that will not lead to a cycle in the graph improves the scoring metric. If it improves the scoring metric, then add each potential edge that will not lead to a cycle in the graph. Iteratively process all edges in the current graph, check whether potential edges can be removed and the scoring metric improved without opening the graph; if so, perform the edge removal operation. Repeat the iterative process until the scoring metric no longer improves; where the check whether a potential edge is removed is by computing the Bayesian information criterion whether the value is increased to judge: ； wherein denotes a directed acyclic graph of parameters, is a constant taking 1, is the number of directed acyclic graph parameters, n denotes the sample size of the observation data; If the value of the current causal graph's increases, it indicates that a potential edge is removed.

5. The visual perception algorithm oriented hazard test case generation method of claim 4, wherein, The process of obtaining key environmental factors from the causal structure diagram and quantitatively calculating the impact of each environmental factor on visual perception performance specifically includes: The difference between the potential intervention outcome and the control outcome represents the individual treatment effect : wherein, indicates the potential intervention result for the i th record in the observation data set when the fog intensity is set to light; indicates the control result in the absence of fog; indicates the i th record in the observation data has a fog intensity of 0, indicates the fog intensity for the i th record is light; The average causal effect of light mist on visual perception performance is represented as: wherein denotes the mathematical expectation; When treatment groups and control groups are not randomly assigned, there is selection bias : wherein, and n is the sample size of the treatment group and control group, respectively; Estimating causal effects using specialized statistical learning models: Estimates of treatment and control outcomes were obtained using any statistical regression method, represented as and ; In the control group, the difference is defined as: , denoted by ; in the treatment group, the difference is defined as: , denoted by ; and is called the filled treatment effect; Calculate the treatment effect based on the new data set (Y, X, A) using any regression method ) and ( ) where is the estimate of the control group, is the estimate of the control group, is the estimate of the control group, and are the covariates for the control and treatment groups, respectively, and the final individual estimate of the causal effect is obtained by combining and : wherein, is a weighting function that takes as an argument the propensity score; is determined by calculating the mathematical expectation of .

6. The visual perception algorithm oriented hazard test case generation method of claim 1, wherein, The hazardous test cases are test environments with a set of defined values.

7. A visual perception algorithm oriented hazard test case generation system, characterized by, The hazardous test case generation system for visual perception algorithms is used to implement the hazardous test case generation method for visual perception algorithms according to any one of claims 1-6, and the hazardous test case generation system for visual perception algorithms includes: An observation dataset collection module is configured to acquire images captured by a visual sensor of an autonomous vehicle, clean and label the acquired images, and obtain an observation dataset according to a no-reference image quality assessment feature; A causal structure diagram construction module is configured to construct a causal structure diagram of the influence of environmental factors on perception performance according to the observation dataset and by fusing domain knowledge; A causal effect estimation module is configured to acquire key environmental factors in the causal structure diagram and quantitatively calculate the influence of each environmental factor on visual perception performance; A dangerous test case generation module is configured to generate perception-oriented dangerous test cases in batches based on qualitative analysis of the causal relationship between key environmental variables and visual perception performance and quantitative estimation of causal effects based on a challenge index.

8. A terminal, characterized by comprising: The terminal comprises a memory, a processor, and a visual perception algorithm-oriented dangerous test case generation program stored on the memory and executable on the processor, and the visual perception algorithm-oriented dangerous test case generation program, when executed by the processor, implements the steps of the visual perception algorithm-oriented dangerous test case generation method according to any one of claims 1-6.

9. A computer-readable storage medium, characterized in that, The computer readable storage medium stores a visual perception algorithm-oriented dangerous test case generation program, and the visual perception algorithm-oriented dangerous test case generation program, when executed by the processor, implements the steps of the visual perception algorithm-oriented dangerous test case generation method according to any one of claims 1-6.

Images