Danger test case generation method for visual perception algorithm, and related device

Abstract

Disclosed in the present invention are a danger test case generation method for a visual perception algorithm, and a related device. The method comprises: acquiring images captured by a visual sensor of a self-driving vehicle, cleaning and marking the collected images, and obtaining an observation data set according to no-reference image quality evaluation features; according to the observation data set and by means of performing fusion with domain knowledge, constructing a causal structure diagram that indicates the influence of environmental factors on perception performance; separating key environmental factors from the causal structure diagram, and quantitatively calculating the degree of influence of each environmental factor on visual perception performance; and on the basis of qualitative analysis of a causal relationship between a key environment variable and the visual perception performance, and quantitative estimation of a causal effect, proposing a challenge index, so as to generate, in batches, danger test cases for perception. The present invention has significant advantages in multiple aspects such as causal relationship mining, challenge index measurement, dangerous-scenario recognition and search space modeling, such that the performance of a visual algorithm for self-driving in a complex environment is evaluated and verified more accurately.

Description

Dangerous test case generation method and related equipment for visual perception algorithm Technical Field

[0001] The present invention relates to the field of artificial intelligence and autonomous driving technology, and in particular to a method, system, terminal, and computer-readable storage medium for generating dangerous test cases for visual perception algorithms. Background Art

[0002] Visual perception is a core building block in autonomous driving systems. It's a key component for autonomous vehicles to acquire environmental information and make decisions. Through visual perception, autonomous driving systems can identify road signs, vehicles, pedestrians, and more. Therefore, strong visual perception is crucial for achieving safe and reliable autonomous driving.

[0003] Existing methods typically use discrete, manually selected test scenarios to evaluate the robustness of visual perception algorithms. These methods rely on experience or expert judgment to select challenging scenarios, such as varying weather conditions, occlusion of road signs, and varying lighting conditions. These scenarios are then evaluated through offline testing or simulation. However, these scenarios are often fixed or limited, failing to fully capture every possible scenario. Furthermore, existing technical solutions lack systematic, quantitative test environment evaluation metrics, making it difficult to accurately assess the challenges of complex, real-world environments.

[0004] Therefore, the existing technology still needs to be improved and developed.

[0005] Summary of the Invention

[0006] The main purpose of the present invention is to provide a method, system, terminal and computer-readable storage medium for generating dangerous test cases for visual perception algorithms, aiming to solve the problem in the existing technology that there is a lack of systematic and quantitative test environment evaluation indicators for evaluating visual perception capabilities, making it difficult to accurately evaluate real complex environments.

[0007] To achieve the above object, the present invention provides a method for generating dangerous test cases for a visual perception algorithm, the method comprising the following steps:

[0008] Obtain images captured by the autonomous vehicle's visual sensor, clean and label the collected images, and obtain an observation dataset based on the no-reference image quality assessment features;

[0009] Based on the observation data set and integrating domain knowledge, a causal structure diagram of the impact of environmental factors on perception performance is constructed;

[0010] Obtain key environmental factors in the causal structure diagram, and quantitatively calculate the degree of influence of each environmental factor on visual perception performance;

[0011] Based on the qualitative analysis of the causal relationship between key environmental variables and visual perception performance, and the quantitative estimation of the causal effect, the challenge index is used to batch generate perception-oriented dangerous test cases.

[0012] Optionally, the method for generating dangerous test cases for visual perception algorithms, wherein acquiring images captured by a visual sensor of an autonomous vehicle, cleaning and labeling the collected images, and obtaining an observation dataset based on a no-reference image quality assessment feature, specifically includes:

[0013] The environmental variables to be examined are formed into combination conditions, and a certain number of images are captured for each environmental combination, where the images are collected by a visual sensor of the autonomous vehicle;

[0014] Clean and label the collected images and extract no-reference image quality assessment features from them;

[0015] Obtaining an observation dataset based on no-reference image quality assessment features;

[0016] Among them, the obtained observation data set is expressed as Where N is the number of cases in the observation dataset, i is one of the cases, X is the covariate vector, W is the environmental factor vector to be examined, and Y is the performance indicator of the visual perception algorithm being tested.

[0017] Optionally, the method for generating dangerous test cases for visual perception algorithms, wherein the step of constructing a causal structure diagram of the impact of environmental factors on perception performance based on the observation dataset and integrating domain knowledge, specifically includes:

[0018] The causal relationship between a set of variables is represented by a directed acyclic graph G(ν,ε), where ν represents a random variable and ε represents the causal link between the variables;

[0019] A fast greedy equivalent search algorithm is used to start from an empty graph, and edges are iteratively added or deleted through a scoring function until convergence. Domain knowledge is integrated to construct a causal structure graph.

[0020] Optionally, the method for generating dangerous test cases for visual perception algorithms, wherein the method starts from an empty graph using a fast greedy equivalent search algorithm and iteratively adds or deletes edges using a scoring function until convergence, specifically includes:

[0021] In the observation dataset Get an empty graph G init ;

[0022] Iterate over each pair of variables:<W,Y> ,<W,X> as well as<X,Y> , add an edge between variables;

[0023] For each potential edge, test whether adding the potential edge to the current graph would result in a cyclic graph. If so, skip the potential edge.

[0024] For each potential edge that does not lead to a cyclic graph, test whether adding the potential edge will improve the scoring metric. If so, add the potential edge.

[0025] Iterate over all edges in the current graph, checking whether removing a potential edge would improve the score without breaking the graph, and if so, remove the edge.

[0026] Repeat the iterative process until the scoring indicator no longer improves;

[0027] The check of whether to remove the potential edge is to determine whether the Bayesian Information Criterion (BIC) value is improved: BIC = 2·ln P(data|θ,G)-c·k·ln(n)

[0028] Where θ represents the parameters of the directed acyclic graph G(ν,ε), c is a constant equal to 1, k is the number of directed acyclic graph parameters, and n represents the sample size of the observed data;

[0029] If the BIC value of the current causal structure graph increases, it means that the potential edge is removed.

[0030] 5. The method for generating dangerous test cases for visual perception algorithms according to claim 4, wherein obtaining key environmental factors from the observation dataset and quantitatively calculating the degree of influence of each environmental factor on visual perception performance specifically comprises:

[0031] The difference between the potential intervention outcome and the control outcome represents the individual treatment effect ITE: i =Y i (W i =1)-Y i (W i =0)

[0032] Among them, Y i (W i =1) indicates the potential intervention result of this record in the observation dataset when the fog intensity is set to light; Y i (W i =0) represents the control result in the absence of fog; W i =0 means that the fog intensity of the i-th record in the observation data is 0, W i =1 means the fog intensity of the i-th record is light;

[0033] The average causal effect ACE of light fog on the average accuracy is expressed as:

[0034] in, represents the mathematical expectation,

[0035] Selection bias exists when treatment and control groups are not randomly assigned

[0036] Among them, N T and N c are the sample sizes of the treatment and control groups, respectively;

[0037] Use specialized statistical learning models to estimate causal effects:

[0038] Any statistical regression method is used to estimate the results of the treatment group and the control group, which are expressed as and

[0039] In the control group, the difference was defined as: Denoted as D c ; Within treatment groups, the difference was defined as: Denoted as D T ;D c and D T This is known as the therapeutic effect of filling;

[0040] Use any regression method based on a new data set (X c ,D c ) and (X T ,D T ) calculates the treatment effect τ0(x) / τ1(x), where τ0(x) represents the estimated result of the control group, τ1(x) represents the estimated result of the control group, and X c and X T are the covariates of the control group and the treatment group respectively. By combining τ0(x) and τ1(x), we can get the final individual estimated causal effect τ X (x): τ X (x)=g(x)τ0(x)+(1-g(x))τ1(x)

[0041] Among them, g(x)∈[0,1] is a weighting function, which is usually taken as the propensity score;

[0042] ACE calculates τ X (x) is determined by the mathematical expectation.

[0043] Optionally, the method for generating dangerous test cases for visual perception algorithms, wherein the qualitative analysis of the causal relationship between key environmental variables and visual perception performance, and the quantitative estimation of the causal effect, and the batch generation of dangerous test cases for perception based on the challenge index, specifically include:

[0044] Define challenge index: challenge index=[r1,r2,…,r i ,r n ]·c

[0045] Where c=[c1,c2,…,c i ,c m ] T is the relative weight among multiple key environmental factors after multiple screenings, reflecting the quantitative contribution of different environmental factors to the perceived results; m is the number of key environmental factors; r i is the corresponding c i The normalized causal effect of c i Apply intervention w so that the causal effect is represented by ACE w ,but:

[0046] Among them, N i Is the environment node c i possible types of interventions;

[0047] Generate a driving score below the danger score threshold d thr The algorithm test score is calculated as follows: score =100×R i P i

[0048] Among them, d score Indicates the test score of the tested system in this scenario; R i is the percentage of the driving path of the tested system in the i-th test scenario to the planned path; P i The violation penalty item corresponding to the system under test;

[0049] Combine the agent model and causal knowledge to maximize the acquisition function, and use the test parameter combination with the maximum acquisition function as a test case:

[0050] in, and s are the predicted driving score and variance of the surrogate model, respectively; represents the minimum driving score predicted by the surrogate model; φ is the probability density function of the standard normal distribution; ci is the defined challenge index; dis is the minimum Euclidean space distance between the candidate test case and the explored test case;

[0051] If the current experiment count k is greater than the preset search iteration number N limit When the perception-oriented dangerous test case set T is generated in batches set .

[0052] Optionally, in the method for generating dangerous test cases for visual perception algorithms, the dangerous test cases are a set of test environments with certain values.

[0053] In addition, to achieve the above-mentioned purpose, the present invention further provides a dangerous test case generation system for visual perception algorithms, wherein the dangerous test case generation system for visual perception algorithms comprises:

[0054] The observation dataset collection module is used to obtain images captured by the autonomous vehicle's visual sensor, clean and label the collected images, and obtain the observation dataset based on the no-reference image quality assessment features;

[0055] A causal structure diagram construction module is used to construct a causal structure diagram of the impact of environmental factors on perception performance based on the observation data set and integrating domain knowledge;

[0056] A causal effect estimation module, configured to obtain key environmental factors from the causal structure diagram and quantitatively calculate the degree of influence of each environmental factor on visual perception performance;

[0057] The dangerous test case generation module is used to generate dangerous test cases for perception in batches based on the challenge index, based on the qualitative analysis of the causal relationship between key environmental variables and visual perception performance, and the quantitative estimation of the causal effect.

[0058] In addition, to achieve the above-mentioned purpose, the present invention also provides a terminal, wherein the terminal includes: a memory, a processor, and a danger test case generation program for visual perception algorithms stored on the memory and runnable on the processor, and when the danger test case generation program for visual perception algorithms is executed by the processor, the steps of the danger test case generation method for visual perception algorithms as described above are implemented.

[0059] In addition, to achieve the above-mentioned purpose, the present invention also provides a computer-readable storage medium, wherein the computer-readable storage medium stores a dangerous test case generation program for visual perception algorithms, and when the dangerous test case generation program for visual perception algorithms is executed by a processor, it implements the steps of the dangerous test case generation method for visual perception algorithms as described above.

[0060] In the present invention, images captured by the autonomous vehicle's visual sensor are obtained, cleaned and labeled, and an observation dataset is obtained based on the no-reference image quality assessment feature. Based on the observation dataset and integrated with domain knowledge, a causal structure diagram is constructed to illustrate the impact of environmental factors on perception performance. Key environmental factors are obtained from the causal structure diagram, and the degree of impact of each environmental factor on visual perception performance is quantitatively calculated. Based on a qualitative analysis of the causal relationship between key environmental variables and visual perception performance, as well as a quantitative estimation of the causal effect, a challenge index is used to batch generate perception-oriented hazard test cases. Based on the quantitative estimation and normalized calculation of the causal effect, the present invention can effectively measure the degree of challenge posed by environmental conditions to the visual algorithm. By introducing the causal knowledge of the challenge index, it guides the search strategy to generate more challenging test cases. Combined with the challenge index and driving score, it can effectively determine whether the performance of the autonomous driving system in a specific scenario is below a predetermined threshold, thereby identifying potential dangerous situations. BRIEF DESCRIPTION OF THE DRAWINGS

[0061] FIG1 is a flow chart of a preferred embodiment of a method for generating dangerous test cases for a visual perception algorithm according to the present invention;

[0062] FIG2 is a schematic diagram illustrating the principle framework of analyzing, screening, and generating test scenarios for key environmental factors of an autonomous driving visual perception algorithm in a preferred embodiment of a method for generating dangerous test cases for a visual perception algorithm according to the present invention;

[0063] FIG3 is a schematic diagram of the observation data set construction process in a preferred embodiment of the method for generating dangerous test cases for visual perception algorithms of the present invention;

[0064] FIG4 is a schematic diagram of a causal structure diagram construction process in a preferred embodiment of a method for generating dangerous test cases for a visual perception algorithm according to the present invention;

[0065] FIG5 is a schematic diagram of a challenge index guided search process in a preferred embodiment of the method for generating dangerous test cases for visual perception algorithms of the present invention;

[0066] FIG6 is a schematic diagram showing the principles of a preferred embodiment of a system for generating dangerous test cases for visual perception algorithms according to the present invention;

[0067] FIG7 is a schematic diagram of an operating environment of a preferred embodiment of the terminal of the present invention. DETAILED DESCRIPTION

[0068] Autonomous vehicles hold significant significance for improving traffic safety, traffic efficiency, and low-carbon mobility, and have become a strategic direction for the transformation and upgrading of the global automotive industry. However, the frequent accidents involving autonomous vehicles in recent years have severely undermined public confidence and become a major obstacle to the commercialization of the autonomous driving industry. In particular, robustness testing of visual perception algorithms has long received insufficient attention.

[0069] Existing testing methods for visual perception can be roughly divided into the following four categories. (1) Data augmentation, which is a technology that attempts to expand the training set of perception algorithms. For example, a certain automobile manufacturer adopted a crowdsourcing-based data collection strategy to obtain training data from actual road driving. The automobile manufacturer's vehicles are equipped with a variety of sensors that can record a variety of data during vehicle driving. After these data are anonymized, the car owners choose whether to participate in the data sharing program and upload them to the automobile manufacturer's server, forming a large and diverse data set. However, data set augmentation is essentially a blind exploration of sparse dangerous scenarios in a high-dimensional test space, which is inefficient. (2) Dangerous scenario search based on neuron coverage: When the test input passes through a neuron and leads to a specific output, the neuron is considered to be covered. For example, a white box testing framework called DeepXplore uses neuron coverage to detect behavioral inconsistencies between tested algorithms. Based on the research of DeepXplore, a greedy search method is provided to increase neuron coverage, further generalize the concept of neuron coverage, and define a series of neuron coverage indicators. Recent studies have combined fuzzy concepts and optimization algorithms with these specific neuron coverage metrics to establish a variety of testing frameworks, such as DLFuzz, DeepHunter, TensorFuzz, and Test4Deep. However, neuron coverage testing is still in its early stages of research, especially because it deviates significantly from traditional software testing concepts and is questioned. (3) Image conversion generation: Generative adversarial networks are used to convert original inputs into harsh weather conditions. Representative solutions include DeepRoad and neural network-based style transfer algorithms. However, image conversion technology lacks clear interpretability and the generation efficiency of synthetic images is low. (4) Adversarial attack: This is a malicious image attack method that affects the output of the model by adding perturbations to the visual input. Early research on adversarial attacks focused on attacks that perturb all pixels. Since perturbations do not occur on all pixels in real scenes, the current research focus is on patch attacks targeting specific areas of the image. For example, a method is proposed to generate printable adversarial billboards that cause visual perception to output incorrect results under dynamically changing driving conditions. It is worth noting that adversarial attacks lack mapping to real-world environments.

[0070] In general, most existing methods focus on inducing erroneous behaviors in deep neural network-based vision algorithms, but lack sufficient attention to the environmental factors that cause these errors.

[0071] Existing testing technologies for autonomous driving vision modules have the following shortcomings:

[0072] (1) Lack of explainability: Some methods, such as neuron coverage and adversarial attacks, cannot reasonably explain the mapping relationship between the discovered vulnerabilities and the real environment, and thus cannot effectively guide algorithm iteration.

[0073] (2) High testing resource consumption: Image conversion is an inefficient and resource-intensive testing method, typically requiring a large amount of computing resources to extract diverse adversarial weather features. Furthermore, due to the lack of screening for environmental factors that actually cause visual perception performance degradation, only an exhaustive list of environmental combinations can be used, exacerbating the computational burden of image synthesis.

[0074] For a long time, the industry has not had a good solution to identify key environmental factors in autonomous driving visual perception, and existing technical solutions cannot effectively generate high-risk perception test scenarios.

[0075] The present invention proposes a method for analyzing, screening, and generating test scenarios for key environmental factors of autonomous driving visual perception algorithms. The method comprises the following steps: (1) collecting observation data sets; (2) constructing a structural causal diagram to screen key environmental variables; (3) quantitatively estimating the degree of influence of various environmental factors on visual perception performance, and proposing an alternative measurement indicator of the challenge index for evaluating the algorithm deployment environment; and (4) generating perception risk test scenarios online based on the challenge index.

[0076] In order to make the purpose, technical solutions and advantages of the present invention more clear and distinct, the present invention is further described in detail below with reference to the accompanying drawings and examples. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0077] The method for generating dangerous test cases for visual perception algorithms according to a preferred embodiment of the present invention, as shown in FIG1 and FIG2 , comprises the following steps:

[0078] Step S10: Obtain images captured by the autonomous driving vehicle's visual sensor, clean and label the collected images, and obtain an observation dataset based on the no-reference image quality assessment features.

[0079] The present invention is based on causal inference theory to isolate key environmental factors from a high-dimensional parameter space. Typically, this requires obtaining data from randomized controlled trials, where the environmental variables to be investigated are randomly assigned to either a control or analysis group. However, obtaining data from randomized controlled trials is expensive or even impractical.

[0080] To this end, the present invention proposes using combinatorial testing principles to collect observational data sets. Specifically, combinatorial testing, also known as combinatorial testing or combinatorial coverage testing, is a software testing technique designed to efficiently test the interactions between different input parameters or configuration settings in a system. In the present invention, combinatorial testing principles are first used to randomly obtain as many possible environmental combinations as possible. This approach has the advantage of balancing data coverage and cost savings.

[0081] As shown in Figure 3, a certain number of images (images captured by the autonomous vehicle's visual sensor) are then captured for each combination. Subsequently, the collected data is cleaned and labeled, and no-reference image quality assessment features are extracted from the images to measure the impact of various environmental factors on image quality. No-reference image quality assessment features are a type of metric used to measure image quality without the need for a reference image or subjective evaluation. They are based on the characteristics and statistical information of the image itself and estimate the image quality by analyzing the low-level features, perceptual features, or structural features of the image.

[0082] The extraction of no-reference image quality indicators is an intermediate step in the construction of the entire observational dataset. Its purpose is to help better understand how environmental nodes affect the quality of images in subsequent causal analysis, which ultimately undermines the performance of visual algorithms.

[0083] The process of constructing the observation dataset is shown in Figure 3. The obtained observation dataset is expressed as Where N is the number of cases in the observation dataset, i is one of the cases, X is the covariate vector, W is the environmental factor vector to be examined, and Y is the performance indicator of the visual perception algorithm being tested.

[0084] Step S20: Based on the observation data set and integrating domain knowledge, a causal structure diagram of the impact of environmental factors on perception performance is constructed.

[0085] Specifically, based on the observation data set collected in step S10, the present invention uses a causal structure discovery method (the causal structure discovery method is a method for inferring causal relationships between variables. It aims to identify causal relationships between variables by observing statistical patterns and dependencies in the data, and reveal the causal structure between phenomena. The goal of the causal structure discovery method is to discover causal models from the data, not just observed correlations. It is based on the core principle of causality, that is, the occurrence of an event is caused by other events, not just related or simultaneous occurrences. Commonly used causal structure discovery methods usually use statistical methods and machine learning algorithms to infer causal relationships between variables by modeling and reasoning on the data. At present, the causal structure discovery method has been widely used in many fields, including medical research, social sciences, and economics. However, its application in the field of autonomous driving perception testing has not yet been found) to establish a structural causal graph represented by a Bayesian graph. The causal structure discovery method focuses on describing the causal relationship between a set of variables, represented by a directed acyclic graph G(ν,ε), where ν represents a random variable and ε represents the causal link between variables. There are many ways to express causal structures, one of the most commonly used of which is to use a graph structure, where the nodes in the graph correspond to variables in the observational data (including environmental variables, image quality indicator variables, and visual perception detection accuracy), and the edge is a directed arrow, A→B, which can be simply understood as A has a direct causal effect on B, and A is the cause of B.

[0086] For example, W fog →Y indicates that fog has a direct causal impact on the visual perception algorithm, and Y represents the performance indicator of the visual algorithm; W fog Represents a fog node in a structural causal graph. In this paper, the Fast Greedy Equivalence Search (FGES) algorithm is employed, which is a parallelized and optimized version of the Greedy Equivalence Search (GES) algorithm. The FGES algorithm starts with an empty graph and iteratively adds or removes edges using a scoring function until convergence, as shown in Figure 4.

[0087] As shown in Figure 4, the detailed steps for establishing a causal structure diagram are as follows:

[0088] Step (1), in the observation data set Get an empty graph G init .

[0089] Step (2), iteratively process each pair of variables:<W,Y> ,<W,X> as well as<X,Y> , add an edge between variables (edges represent causal relationships).

[0090] Step (3): For each potential edge (possible causal relationship, but further confirmation is required in subsequent steps. If the potential edge improves the scoring indicator BIC and does not violate the rules of the graph structure, it becomes a confirmed edge), test whether adding the potential edge to the current graph will lead to a cyclic graph. If it does, skip the potential edge.

[0091] Step (4): For each potential edge that does not lead to a cyclic graph, test whether adding the potential edge will improve the scoring indicator. If so, add the potential edge.

[0092] Among them, in the scoring-based causal graph structure discovery method, scoring indicators are used to measure the degree of fit or quality of the potential causal graph model. They are used to compare the relative advantages and disadvantages between different models and help determine the optimal causal graph structure.

[0093] Determining whether adding a potential edge improves a scoring metric typically involves comparing and analyzing models. In causal graph structure discovery, the pros and cons of different graph structures can be determined by comparing their scoring metrics. If adding a potential edge improves the scoring metric, it can be assumed that the addition of that edge has improved the model's fit or quality.

[0094] Step (5), iteratively process all edges in the current graph, check whether removing potential edges can improve the scoring index without breaking the graph, and if so, perform the edge removal operation;

[0095] Repeat steps (2)-(5) until the scoring indicator no longer improves.

[0096] The present invention checks whether to remove potential edges by calculating whether the Bayesian Information Criterion (BIC) value is improved: BIC = 2·ln P(data|θ,G)-c·k·ln(n)

[0097] Where θ represents the parameters of the directed acyclic graph G(ν,ε), c is a constant equal to 1, k is the number of directed acyclic graph parameters, and n represents the sample size of the observed data.

[0098] If the BIC value of the current causal structure graph increases, it means that the potential edge is removed.

[0099] Generally speaking, a higher BIC value indicates that the identified graphical causal model G(ν,ε) can more effectively explain causal relationships without being overly complex. (Constructing a causal graph structure is an iterative process, equivalent to finding a structure with the highest BIC score among all previously discovered structures.) It is worth noting that the present invention does not rely entirely on the data-driven causal structure discovery paradigm. As shown in Figure 4, the present invention incorporates domain knowledge to construct a causal structure diagram that illustrates the impact of environmental factors on perceptual performance.

[0100] Step S30: Acquire key environmental factors from the observation data set, and quantitatively calculate the degree of influence of each environmental factor on visual perception performance.

[0101] Specifically, in this step, we aim to estimate the degree of degradation of the perceived performance Y caused by key environmental factors, namely the "causal effect". According to the research paradigm of the potential outcome framework, the term "causal effect" refers to the difference in the impact of "treatment" and "no treatment" on the outcome Y in a given observation data set. For example, taking "fog" in the environment as an example, where "treatment" refers to different fog intensity levels, the possible different treatment levels are represented by {0,1,2,…,N w} means, N w It is all possible values ​​of a variable in the graph structure. This example has shown that fog nodes have different intensities. For example, they can be divided into 5 intensities from zero to strong.

[0102] Among them, W i =0 means that the fog intensity of the i-th record in the observation data is 0, W i =1 means the fog intensity of the i-th record is light; Y i (W i =1) indicates the potential intervention result of this record in the observation dataset when the fog intensity is set to light; Y i (W i =0) represents the control result in the absence of fog; Y here is used to measure the performance of the visual perception algorithm, such as mean average precision (mAP), and the difference between the two (control group and control group) is the individual treatment effect (ITE): ITE i =Y i (W i =1)-Y i (W i =0)

[0103] According to the causal graph model introduced in step 20 of the present invention, non-critical environmental factors can be considered as covariate vectors: X, which are not affected by the treatment. Image quality indicators are post-treatment variables, and their results are affected by the treatment. The average causal effect (ACE) of mild fog on mAP can be further expressed as:

[0104] in, represents the mathematical expectation.

[0105] However, in practice, it is impossible to retrieve the results of the intervention group and the control group at the same time in the observational data, so it is not possible to simply subtract the mAP of the intervention group from the mAP of the control group, that is, there may be a problem of selection bias:

[0106] That is, when the treatment and control groups are not randomly assigned, there is selection bias

[0107] Among them, N T and N c are the sample sizes of the treatment group and the control group, respectively.

[0108] When treatment and control groups are not randomly assigned, selection bias can occur, leading to systematic differences in the covariates between the two groups and potentially inaccurate estimates of ACE. To this end, this paper proposes the use of dedicated statistical learning models to estimate causal effects. For example, the meta-learning algorithm X-learner has been shown to produce accurate inferences in the presence of confounding variables and unobserved heterogeneity. X-learner uses a three-stage process to estimate treatment effects, as follows:

[0109] Step (11), simulate the treatment / control results: use any statistical regression method, such as random forest, to estimate the treatment group and control group results, expressed as and

[0110] Step (12), imputing treatment effect: imputing treatment effect for treated / control individuals. In the control group, the difference is defined as: Denoted as D c ; Within treatment groups, the difference was defined as: Denoted as D T ;D c and D T This is called the treatment effect of filling.

[0111] Step (13), combine treatment effects to generate final estimates: Use any regression method based on the new data set (Xc ,D c ) and (X T ,D T ) calculates the treatment effect τ0(x) / τ1(x), where τ0(x) represents the estimated result of the control group, τ1(x) represents the estimated result of the control group, and X c and X T are the covariates of the control group and the treatment group respectively. By combining τ0(x) and τ1(x), we can get the final individual estimated causal effect τ X (x): τ X (x)=g(x)τ0(x)+(1-g(x))τ1(x)

[0112] Among them, g(x)∈[0,1] is a weighting function, usually taken as a propensity score; so far, ACE calculates τ X (x) is determined by the mathematical expectation.

[0113] Step S40: Based on the qualitative analysis of the causal relationship between key environmental variables and visual perception performance, and the quantitative estimation of the causal effect, perception-oriented hazard test cases are generated in batches based on the challenge index.

[0114] Specifically, based on the qualitative analysis of the causal relationship between key environmental variables and visual perception performance, and the quantitative estimation of the causal effect, this paper further designs a new alternative evaluation metric for evaluating the challenging environment conditions for visual algorithm deployment, which the paper calls the "challenge index." The challenge index is defined by the following equation: challenge index = [r1, r2, ..., r i ,r n ]·c

[0115] Where c=[c1,c2,…,c i ,c m ] T is the relative weight among multiple key environmental factors after multiple screenings, reflecting the quantitative contribution of different environmental factors to the perceived results; m is the number of key environmental factors; r i is the corresponding c i The normalized causal effect of c i Apply intervention w so that the causal effect is represented by ACE w ,but:

[0116] Among them, N i Is the environment node c i Possible types of intervention.

[0117] The "Challenge Index" provides an industry-first approach to estimating the deployment difficulty of visual algorithms based on causal knowledge. By collecting training and test samples in key environmental combinations, it can significantly accelerate algorithm verification and iteration. It's important to note that the vector c in the Challenge Index reflects the quantitative contribution of different environmental factors to the perception results. To determine these relative weights, we recommend using the entropy weighting method.

[0118] Furthermore, the present invention proposes a "challenge index-guided search" method for generating dangerous test scenarios to verify the safety of vision-based end-to-end autonomous driving solutions. Any search-based scenario generation method needs to address the following key issues: what is the search space, and what search strategy is used to generate test cases. Considering that the huge search space makes existing search technologies very inefficient and blind, the present invention first selects key environmental factors that have been screened by causal inference as the dimensions of the search space, reducing many non-causal test dimensions. Secondly, a surrogate model is used to probabilistically model the search space, using spatial correlation to capture the similarity between explored test samples and neighboring unexplored areas, and establish result predictions and uncertainty estimates for them. In addition, by integrating the discovered causal knowledge (challenge index), the search strategy is further guided to search towards more challenging key areas, thereby batch generating perception-oriented dangerous test scenarios (i.e., dangerous test cases). The online scenario search process is shown in Figure 5.

[0119] In Figure 5, k is the current experiment count; N limit is the preset number of search iterations; d score and d thr are the test score of the tested system in this scenario and the risk score threshold. It should be noted that the goal of the present invention is to generate as many driving scores as possible below a specific threshold (d thr ) dangerous scenario, the algorithm test score is calculated as follows: d score =100×R i P i

[0120] Among them, R i is the percentage of the driving path of the tested system in the i-th test scenario to the planned path; P iThe violation penalty items corresponding to the system under test include route deviation, collision, and simulation timeout. As shown in Figure 5, the search method of the present invention dynamically adjusts the exploration area in the test space online based on the test results of the previous test case. Specifically, it combines the agent model and causal knowledge to maximize the following acquisition function, and takes the test parameter combination with the largest acquisition function as a test case (a test case is a set of test environments with certain values, such as the sunlight intensity, fog concentration, etc. Formula (9) simply selects the one with the largest acquisition from the possible options as a test case):

[0121] in, and s are the predicted driving score and variance of the alternative model, respectively; represents the lowest driving score predicted by the agent model; φ is the probability density function of the standard normal distribution; ci is the challenge index, calculated using formulas (6) and (7); dis is the minimum Euclidean space distance between the candidate test case and the explored test case; if the current experiment count k is greater than the preset number of search iterations N limit When the perception-oriented dangerous test case set T is generated in batches set .

[0122] The key point of the present invention is to propose a testing method for autonomous driving visual perception algorithms, which specifically includes qualitative research on causal relationships and quantitative estimation of causal effects; a new evaluation indicator, namely the challenge index, is proposed to measure the degree of challenge of environmental conditions to visual algorithms; in addition, the present invention also proposes a challenge index-guided search method to generate risk test scenarios to verify the performance of visual end-to-end algorithms in various challenging environments. In summary, the present invention can help accelerate the verification and iteration of visual perception algorithms.

[0123] The main protection points of the present invention are as follows:

[0124] (1) Challenge Index Calculation Method: The present invention provides a challenge index calculation method, which is based on quantitative estimation and normalized calculation of causal effects and can effectively measure the degree of challenge posed by environmental conditions to visual algorithms.

[0125] (2) Dangerous scenario identification method: A method for identifying dangerous scenarios using a challenge index and a driving score. By combining the challenge index and the driving score, this method can effectively determine whether the performance of the autonomous driving system in a specific scenario is below a predetermined threshold, thereby identifying potential dangerous situations.

[0126] (3) Algorithm flow and related algorithm details: Protect the search space modeling method based on the proxy model, which uses spatial correlation to establish a potential test sample space model, and guides the search strategy to generate more challenging test cases by introducing causal knowledge of the challenge index; protect the algorithm flow and related algorithm details proposed in the present invention, including the specific steps of the challenge index-guided search, the construction and update method of the proxy model, etc. These details are of great technical and innovative significance for implementing the method of the present invention.

[0127] Compared with the best existing technology, the present invention has the following advantages (i.e. beneficial effects):

[0128] (1) In-depth exploration of causal relationships: This invention is the first in the industry to apply causal inference theory to deeply explore the causal relationship between environmental conditions and visual algorithm performance, thereby enabling a more accurate assessment of the impact of different environmental factors on the algorithm, which is more scientific and reliable than existing technologies.

[0129] (2) Innovative measurement of challenge index: This paper proposes a challenge index as a new indicator for evaluating the degree of challenge of an algorithm under various environmental conditions. This index combines the normalized calculation and weight distribution of causal effects to more comprehensively measure the challenge of the environment to the algorithm and can more accurately reflect the needs of actual application scenarios than existing technologies.

[0130] (3) Search space modeling and test case generation: The search space modeling method based on the proxy model of the present invention can generate challenging test cases more efficiently, which helps to more comprehensively verify and evaluate the robustness and performance of visual algorithms. It is more efficient and feasible than existing technologies.

[0131] (4) Effective identification of dangerous scenarios: The present invention uses causal reasoning methods to rationally screen the key factors that affect visual perception algorithms, and is the first to introduce the concept of "challenge index" to quantify the causal effects of key influencing factors. The efficient dangerous scenario generation method designed by the present invention, by combining the proposed challenge index and driving score, can quickly and accurately identify the performance of the autonomous driving system in dangerous scenarios and promptly warn of potential dangerous situations. This method can generate challenging test scenarios more reasonably than existing technologies, providing important guarantees for the safety and reliability of autonomous driving systems.

[0132] In summary, compared with the best existing technologies, the present invention has significant advantages in causal relationship mining, challenge index measurement, dangerous scene identification, search space modeling, etc., and can more accurately evaluate and verify the performance of visual algorithms in complex environments.

[0133] Furthermore, extensive experiments have shown that our method, compared to random search and random neighborhood search, discovers 3.75 to 12.3 times and 1.5 to 9.25 times more dangerous scenarios, respectively. Furthermore, the generated scenarios maintain a balance between test coverage and risk, posing a greater threat to the system under test.

[0134] Furthermore, as shown in FIG6 , based on the above-mentioned dangerous test case generation method for visual perception algorithms, the present invention also provides a dangerous test case generation system for visual perception algorithms, wherein the dangerous test case generation system for visual perception algorithms includes:

[0135] an observation dataset collection module 51 for acquiring images captured by the autonomous vehicle's visual sensor, cleaning and labeling the collected images, and obtaining an observation dataset based on a no-reference image quality assessment feature;

[0136] A causal structure diagram construction module 52 is used to construct a causal structure diagram of the impact of environmental factors on perception performance based on the observation data set and integrating domain knowledge;

[0137] a causal effect estimation module 53 for obtaining key environmental factors from the observation data set and quantitatively calculating the degree of influence of each environmental factor on visual perception performance;

[0138] The dangerous test case generation module 54 is used to generate perception-oriented dangerous test cases in batches based on the challenge index based on the qualitative analysis of the causal relationship between key environmental variables and visual perception performance and the quantitative estimation of the causal effect.

[0139] Furthermore, as shown in FIG7 , based on the above-described method and system for generating dangerous test cases for visual perception algorithms, the present invention also provides a terminal, which includes a processor 10, a memory 20, and a display 30. FIG7 illustrates only some components of the terminal, but it should be understood that implementation of all illustrated components is not required, and more or fewer components may be implemented instead.

[0140] In some embodiments, the memory 20 may be an internal storage unit of the terminal, such as a hard disk or memory of the terminal. In other embodiments, the memory 20 may also be an external storage device of the terminal, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. equipped on the terminal. Furthermore, the memory 20 may also include both an internal storage unit of the terminal and an external storage device. The memory 20 is used to store application software and various types of data installed on the terminal, such as the program code of the installation terminal, etc. The memory 20 may also be used to temporarily store data that has been output or is to be output. In one embodiment, a hazard test case generation program 40 for a visual perception algorithm is stored on the memory 20, and the hazard test case generation program 40 for a visual perception algorithm can be executed by the processor 10, thereby realizing the hazard test case generation method for a visual perception algorithm in the present application.

[0141] In some embodiments, the processor 10 can be a central processing unit (CPU), a microprocessor or other data processing chip, used to run the program code or process data stored in the memory 20, such as executing the dangerous test case generation method for the visual perception algorithm.

[0142] In some embodiments, the display 30 may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, or an OLED (Organic Light-Emitting Diode) touchscreen. The display 30 is used to display information on the terminal and to display a visual user interface. The components 10-30 of the terminal communicate with each other via a system bus.

[0143] In one embodiment, when the processor 10 executes the dangerous test case generation program 40 for visual perception algorithms in the memory 20 , the steps of the dangerous test case generation method for visual perception algorithms described above are implemented.

[0144] The present invention also provides a computer-readable storage medium, wherein the computer-readable storage medium stores a dangerous test case generation program for visual perception algorithms, and when the dangerous test case generation program for visual perception algorithms is executed by a processor, the steps of the dangerous test case generation method for visual perception algorithms as described above are implemented.

[0145] In summary, the present invention provides a method and related equipment for generating dangerous test cases for visual perception algorithms. The method includes: obtaining images captured by a visual sensor of an autonomous vehicle, cleaning and labeling the collected images, and obtaining an observation dataset based on a no-reference image quality assessment feature; constructing a causal structure diagram of the impact of environmental factors on perception performance based on the observation dataset and integrating domain knowledge; obtaining key environmental factors from the causal structure diagram and quantitatively calculating the degree of impact of each environmental factor on visual perception performance; and generating batches of dangerous test cases for perception based on a challenge index based on a qualitative analysis of the causal relationship between key environmental variables and visual perception performance and a quantitative estimation of the causal effect. Based on the quantitative estimation and normalized calculation of the causal effect, the present invention can effectively measure the degree of challenge posed by environmental conditions to visual algorithms. By introducing the causal knowledge of the challenge index, it guides the search strategy to generate more challenging test cases. Combined with the challenge index and driving score, it can effectively determine whether the performance of the autonomous driving system in a specific scenario is below a predetermined threshold, thereby identifying potential dangerous situations.

[0146] It should be noted that, in this document, the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, article, or terminal comprising a series of elements includes not only those elements but also other elements not explicitly listed, or elements inherent to such process, method, article, or terminal. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of other identical elements in the process, method, article, or terminal comprising the element.

[0147] Of course, those skilled in the art will appreciate that all or part of the processes in the above-described method embodiments can be implemented by instructing related hardware (such as a processor, controller, etc.) through a computer program. The program can be stored in a computer-readable storage medium that can be read by a computer. When the program is executed, it can include the processes in the above-described method embodiments. The computer-readable storage medium can be a memory, a magnetic disk, an optical disk, etc.

[0148] It should be understood that the application of the present invention is not limited to the above examples. For those skilled in the art, improvements or changes can be made based on the above description. All these improvements and changes should fall within the scope of protection of the claims attached to the present invention.

Claims

1. A method for generating dangerous test cases for visual perception algorithms, characterized in that: The method for generating dangerous test cases for visual perception algorithms comprises: Obtain images captured by the autonomous vehicle’s visual sensor, clean and label the collected images, and obtain an observation dataset based on the no-reference image quality assessment features; Based on the observation data set and integrating domain knowledge, a causal structure diagram of the impact of environmental factors on perception performance is constructed; Obtain key environmental factors in the causal structure diagram, and quantitatively calculate the degree of influence of each environmental factor on visual perception performance; Based on the qualitative analysis of the causal relationship between key environmental variables and visual perception performance, and the quantitative estimation of the causal effect, perception-oriented hazard test cases are batch generated based on the challenge index.

2. The method for generating dangerous test cases for visual perception algorithms according to claim 1, characterized in that: The method of obtaining images captured by the visual sensor of the autonomous driving vehicle, cleaning and marking the collected images, and obtaining an observation data set based on the no-reference image quality assessment features specifically includes: The environmental variables to be examined are combined into combination conditions, and a certain number of images are captured for each environmental combination, where the images are collected by a visual sensor of an autonomous driving vehicle; Clean and label the collected images and extract features from them for no-reference image quality assessment; Obtaining an observation dataset based on a no-reference image quality assessment feature; Among them, the obtained observation data set is expressed as Among them, N is the number of cases in the observation data set, i is one of the number of cases, X is the covariate vector, W is the environmental factor vector to be examined, and Y is the performance indicator of the visual perception algorithm being tested.

3. The method for generating dangerous test cases for visual perception algorithms according to claim 2, characterized in that: The process of constructing a causal structure diagram of the impact of environmental factors on perception performance based on the observation data set and integrating domain knowledge specifically includes: The causal relationship between a set of variables is represented by a directed acyclic graph G(v,ε), where v represents a random variable and ε represents the causal link between the variables; A fast greedy equivalent search algorithm is used to start from an empty graph, and edges are iteratively added or deleted through a score function until convergence. The necessary domain knowledge is combined to construct a causal structure graph.

4. The method for generating dangerous test cases for visual perception algorithms according to claim 3, characterized in that: The fast greedy equivalent search algorithm starts from an empty graph and iteratively adds or deletes edges through a score function until convergence, specifically including: In the observational dataset Get an empty graph G from init ; This iterates over each pair of variables:<W,Y> ,<W,X> as well as<X,Y> , add an edge between variables; For each potential edge, test whether adding the potential edge to the current graph would result in a cyclic graph. If so, skip the potential edge. For each potential edge that does not lead to a cyclic graph, test whether adding the potential edge will improve the scoring metric, and if so, add the potential edge; Iterate over all edges in the current graph, check if removing a potential edge will improve the score without breaking the graph, and if so, remove the edge. Repeat the iterative process until the scoring metric stops improving; Among them, checking whether to remove potential edges is to determine whether the Bayesian Information Criterion BIC value is improved: BIC=2·ln P(data|θ,G)-c·k·ln(n) Where θ represents the parameter of the directed acyclic graph G(v,ε), c is a constant equal to 1, k is the number of directed acyclic graph parameters, and n represents the sample size of the observed data; If the BIC value of the current causal structure graph increases, it means that the potential edge is removed.

5. The method for generating dangerous test cases for visual perception algorithms according to claim 4, characterized in that: The step of obtaining key environmental factors from the observation data set and quantitatively calculating the influence of each environmental factor on visual perception performance specifically includes: The difference between the potential intervention outcome and the control outcome represents the individual treatment effect ITE: ITE i =Y i (W i =1)-Y i (W i =0) Among them, Y i (W i =1) indicates the potential intervention result of this record in the observation data set when the fog intensity is set to light; Y i (W i =0) represents the control result in the absence of fog; W i =0 means that the fog intensity of the i-th record in the observation data is 0, W i =1 means the fog intensity of the i-th record is light; The average causal effect ACE of light fog on visual perception performance is expressed as: in, represents mathematical expectation; Selection bias exists when treatment and control groups are not randomly assigned Among them, N T and N c are the sample sizes of the treatment and control groups, respectively; Use specialized statistical learning models to estimate causal effects: The results of the treatment and control groups are estimated using any statistical regression method and are expressed as and In the control group, the difference was defined as: Denoted as D c ; Within treatment groups, differences were defined as: Denoted as D T ;D c and D T The therapeutic effect is called filling; Use any regression method based on a new data set (X c ,D c ) and (X T ,D T ) calculates the treatment effect τ0(x) / τ1(x), where τ0(x) represents the estimated result for the control group, τ1(x) represents the estimated result for the control group, and X c and X T are the covariates of the control group and the treatment group respectively. By combining τ0(x) and τ1(x), we get the final individual estimated causal effect τ X (x): t X (x)=g(x)τ0(x)+(1-g(x))τ1(x) Among them, g(x)∈[0,1] is a weighting function, which is usually taken as the propensity score; ACE calculates τ X (x) is determined by the mathematical expectation.

6. The method for generating dangerous test cases for visual perception algorithms according to claim 5, characterized in that: The qualitative analysis of the causal relationship between key environmental variables and visual perception performance, as well as the quantitative estimation of causal effects, and batch generation of perception-oriented hazard test cases based on the challenge index specifically include: Defining the Challenge Index: challenge index=[r1,r2,…,r i ,r n ]·c Where c=[c1,c2,…,c i ,c m ] T is the relative weight between multiple key environmental factors after multiple screenings, reflecting the quantitative contribution of different environmental factors to the perceived results; m is the number of key environmental factors; r i is the corresponding c i The normalized causal effect of c i Apply intervention w so that the causal effect is represented by ACE w ,but: Among them, N i Is the environment node c i possible types of interventions; Generate driving score below the danger score threshold d thr The test scores of the tested autonomous driving perception algorithm in the given test scenario are: d score =100×R i P i Among them, d score Indicates the test score of the tested system in this scenario; R i is the percentage of the driving path of the tested system in the i-th test scenario to the planned path; P i The violation penalty item corresponding to the system under test; The agent model and causal knowledge are combined to maximize the acquisition function, and the combination of test parameters with the largest acquisition function is used as a test case. The acquisition function is defined as: in, and s are the predicted driving score and variance of the surrogate model, respectively; represents the minimum driving score predicted by the agent model; φ is the probability density function of the standard normal distribution; ci is the defined challenge index; dis is the minimum Euclidean space distance between the candidate test case and the explored test case; If the current experiment count k is greater than the preset search iteration number N limit When the perception-oriented dangerous test case set T is generated in batches set .

7. The method for generating dangerous test cases for visual perception algorithms according to claim 6, characterized in that: The dangerous test case is a test environment with a set of determined values.

8. A dangerous test case generation system for visual perception algorithms, characterized in that: The dangerous test case generation system for visual perception algorithm comprises: The observation dataset collection module is used to obtain images captured by the autonomous vehicle visual sensor, clean and label the collected images, and obtain the observation dataset based on the no-reference image quality assessment features; A causal structure diagram construction module is used to construct a causal structure diagram of the impact of environmental factors on perception performance based on the observation data set and integrating domain knowledge; A causal effect estimation module, used to obtain key environmental factors in the causal structure diagram and quantitatively calculate the influence of each environmental factor on visual perception performance; The dangerous test case generation module is used to generate dangerous test cases for perception in batches based on the challenge index based on the qualitative analysis of the causal relationship between key environmental variables and visual perception performance and the quantitative estimation of the causal effect.

9. A terminal, characterized in that: The terminal includes: a memory, a processor, and a danger test case generation program for a visual perception algorithm stored in the memory and executable on the processor. When the danger test case generation program for a visual perception algorithm is executed by the processor, the steps of the danger test case generation method for a visual perception algorithm as described in any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a dangerous test case generation program for visual perception algorithms, and when the dangerous test case generation program for visual perception algorithms is executed by a processor, the steps of the dangerous test case generation method for visual perception algorithms as described in any one of claims 1-7 are implemented.

Images