Visual perception algorithm failure mode formalized test method and device

Abstract

The invention relates to a visual perception algorithm failure mode formal test method and device, and belongs to the technical field of visual perception. The method comprises the following steps: mining failure modes possibly occurring in an execution process of a visual perception algorithm, and generating a description text of each failure mode; identifying keywords in the description text, associating each keyword with one or more nodes in the knowledge graph, and constructing a failure mode tree of the failure mode; generating a formalized description grammar system structure of the failure mode; for each node in the failure mode tree, generating a nested object in a corresponding formalized description statement based on the formalized description grammar system structure so as to obtain the formalized description statement; and generating a test case based on the formalized description statement. According to the method, the failure test cases corresponding to different failure modes can be generated, so that the visual perception algorithm is tested in an instructive, more accurate and faster manner, and all potential failure modes are effectively covered.

Description

Technical Field

[0001] The invention belongs to the technical field of visual perception, and proposes a formalized testing method and device for failure modes of a visual perception algorithm. Background Art

[0002] Visual perception algorithms are key technologies in the fields of autonomous driving, robotics, security systems, etc. The purpose of visual perception algorithm testing technology is to ensure that these algorithms can correctly recognize and understand visual information in various environments and conditions. Existing technologies for testing visual perception algorithms mainly include dataset testing, simulation testing, and adversarial sample testing.

[0003] Dataset testing is to benchmark the algorithm using standardized public datasets (such as ImageNet, COCO, KITTI, etc.). By increasing the diversity and complexity of the test dataset, various situations that may be encountered in the real world are simulated. However, aimless optimization on the dataset can easily cause the algorithm to overfit to a specific dataset and have weak generalization ability. In addition, it is difficult for the dataset to cover many types of failure situations in the real world.

[0004] Simulation testing uses computer graphics and virtual reality technology to create visual scenes in a controlled environment. It can precisely control environmental conditions and parameters and is suitable for testing the response of algorithms to specific variables. However, simulation testing consumes too much resources and manpower, and the scenes generated in the simulation environment are very different from the real world, so it is not possible to test the performance of the algorithm in the real world.

[0005] Adversarial sample testing is a specially designed input sample to mislead visual perception algorithms into making wrong judgments. It can test the robustness of the algorithm against abnormal or malicious attacks. Although adversarial sample testing can well discover the weaknesses of the algorithm, the generated data is far from the failure conditions in the real world and cannot simulate the failure conditions in the real world in daily life.

[0006] In addition, due to the black-box nature of visual perception algorithms, manual non-targeted testing consumes a lot of time and resources, and is prone to missing certain failure conditions, making it impossible to achieve comprehensive testing.

[0007] Therefore, there is an urgent need for a technical solution to formally test the failure modes of visual perception algorithms. Summary of the invention

[0008] The present invention provides a method and device for formalizing the failure mode testing of a visual perception algorithm, which can generate failure test cases corresponding to different failure modes, so as to test the visual perception algorithm in a guided, more accurate and faster manner, and ensure that all potential failure modes of the visual perception algorithm have been effectively covered.

[0009] According to a first aspect of an embodiment of the present disclosure, a method for formal testing of failure modes of a visual perception algorithm is provided, comprising:

[0010] Mining possible failure modes that may occur during the execution of visual perception algorithms and generating description text for each failure mode;

[0011] Identify keywords in the description text, and associate each keyword with one or more nodes in the knowledge graph to construct a failure mode tree for the failure mode;

[0012] Generate a formal description grammar architecture of the failure mode;

[0013] For each node in the failure mode tree, generating a nested object in a corresponding formal description statement based on the formal description grammar architecture, thereby obtaining a formal description statement;

[0014] Based on the formal description statement, a test case is generated.

[0015] Preferably, the identifying keywords in the description text includes:

[0016] Preprocessing the description text;

[0017] Decomposing the preprocessed description text into individual words or phrases;

[0018] Convert each word or phrase into a numeric vector;

[0019] Based on the numerical vector, keywords in the description text are obtained.

[0020] Preferably, the knowledge graph consists of several key nodes V i With this key node V i The key node V i Includes: failure labels, algorithm classification, failure stages, deployment platforms, application scenarios and data sets.

[0021] Preferably, associating each keyword with one or more nodes in the knowledge graph to construct a failure mode tree of the failure mode includes:

[0022] Using the failure mode name as the root node of the failure mode tree;

[0023] Use the depth-first search algorithm to locate the nodes corresponding to the extracted keywords in the knowledge graph;

[0024] The located node is the key node V in the knowledge graph i In the case ofi , and based on the key node V i The properties of the key node U i Next, generate an attribute node;

[0025] The node being located is the key node V in the knowledge graph. i Child nodes of In the case of i The child node below And based on the child node The attributes of the child node Generate an attribute node; where i and j are natural numbers, and t is the depth of the child node.

[0026] Preferably, at the key node V i In the case of the failure stage, the formal description grammar architecture of the failure mode is generated, including:

[0027] Using the failure mode name as the root node of the formal description grammar architecture;

[0028] The failure stage is used as a child node of the root node of the formal description grammar architecture; wherein the failure stage includes: data set stage, training stage, reasoning stage, deployment stage, update stage and destruction stage;

[0029] According to the scenario of mining the visual perception algorithm, a multi-level dimension of each failure stage is obtained, and a branch of the failure stage is generated based on the multi-level dimension; wherein each dimension includes the attributes of the dimension and the failure condition that triggers the failure of the visual perception algorithm under the dimension.

[0030] Preferably, for each node in the failure mode tree, generating a corresponding nested object in a formal description statement based on the formal description grammar architecture includes:

[0031] Conduct formal analysis of descriptive texts;

[0032] Generate the corresponding nested format in the formal description statement based on the formal description grammar architecture;

[0033] The nested format is filled with content using the nodes in the failure mode tree and the parameter type and parameter range of the failure triggering condition to obtain the nested objects in the formal description statement.

[0034] Preferably, the nested format is filled with content using the nodes in the failure mode tree and the parameter type and parameter range of the failure trigger condition to obtain the nested objects in the formal description statement, including:

[0035] When a node in the failure mode tree is a failure stage, the failure stage is presented as a nested object in the formal statement, and all child nodes under the failure stage are As a key value within a nested object;

[0036] The nodes in the failure mode tree are child nodes. In the case of and the parameter type and parameter range of the failure trigger condition as the key node U i An object nested in is presented;

[0037] When a node in the failure mode tree is an attribute node, the attribute node is used as an attribute of an upper-level child node.

[0038] Preferably, generating a test case based on the formal description statement includes:

[0039] Determining the algorithm version and preliminary parameters of the visual perception algorithm;

[0040] Applying the formal description statement in combination with the algorithm version of the visual perception algorithm and preliminary parameters to generate an initial test case;

[0041] Train a machine learning model using the initial test cases to predict failure regions in parameter space.

[0042] An optimal parameter combination is explored in the failure region, and the optimal parameter combination is applied to the formal description statement to obtain a final test case.

[0043] A visual perception algorithm failure mode formalized testing device, the device comprising:

[0044] The description text generation module is used to mine the possible failure modes that may occur during the execution of the visual perception algorithm and generate description text for each failure mode;

[0045] A failure mode tree construction module, used to identify keywords in the description text, and associate each keyword with one or more nodes in the knowledge graph to construct a failure mode tree for the failure mode;

[0046] A grammar architecture generation module, used for generating a formal description grammar architecture of the failure mode;

[0047] A formal description module, for generating, for each node in the failure mode tree, a nested object in a corresponding formal description statement based on the formal description grammar architecture, thereby obtaining a formal description statement;

[0048] The test case generation module is used to generate test cases based on the formal description statements.

[0049] An electronic device, characterized in that the electronic device comprises: a processor and a memory storing computer program instructions; when the processor executes the computer program instructions, the above-mentioned formal testing method for failure modes of visual perception algorithms is implemented.

[0050] The technical solution provided by the embodiments of the present disclosure includes at least the following beneficial effects:

[0051] 1. The failure mode analysis method proposed in the present invention can systematically analyze the failure of the visual perception algorithm, obtain the failure mode of the visual perception algorithm, and provide guidance for testing it.

[0052] 2. The formal description framework constructed by the present invention can describe the failure mode in a standardized manner, avoiding the ambiguity caused by traditional language description, making the recording and expression of failure mode more standardized and comprehensive. And combined with the knowledge graph, this method can make the generation of formal description language automatic, faster and more convenient.

[0053] 3. The present invention realizes automatic generation of failure test cases corresponding to different failure modes based on a formal description language. The automation process implemented in this step can test the visual perception algorithm in a guided, more accurate and faster manner. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] Figure 1 It is a flowchart of a method for constructing a data-driven intelligent algorithm capability boundary according to an exemplary embodiment.

[0055] Figure 2 The figure is an algorithm failure knowledge graph shown according to an exemplary embodiment.

[0056] Figure 3 FIG. 4 is a failure mode tree of abnormal image exposure according to an exemplary embodiment.

[0057] Figure 4 It is a block diagram of a formal description according to an exemplary embodiment.

[0058] Figure 5 The present invention is a flowchart of generating a failure test case according to an exemplary embodiment.

[0059] Figure 6 The figure is a block diagram of a visual perception algorithm failure mode formalization test device according to an exemplary embodiment.

[0060] Figure 7 is a block diagram of an electronic device 700 according to an exemplary embodiment. DETAILED DESCRIPTION

[0061] The present invention is further described in detail below in conjunction with specific embodiments. The given examples are only for illustrating the present invention, but not for limiting the scope of the present invention.

[0062] Visual perception algorithms are a core component of today's technology field. They play a key role in many scenarios, such as autonomous driving and safety monitoring. However, even the most advanced algorithms may experience unexpected performance degradation in certain situations. This phenomenon is called algorithm failure, and a class of failures with common properties can be called failure modes. This paper proposes a formalized test method for visual perception algorithm failure modes. Figure 1 A flowchart of the method is shown in FIG.

[0063] Step 1: Mining possible failure modes that may occur during the execution of the visual perception algorithm and generating a description text for each failure mode.

[0064] Unlike traditional hardware or software failures, the failure modes of visual perception algorithms usually result in performance degradation rather than complete loss of functionality. These failure modes can be caused by a variety of factors, such as algorithm design flaws, data bias, or the complexity of the external environment.

[0065] In order to further describe the failure of visual perception algorithms, this paper will describe the failure modes from several dimensions:

[0066] (1) Algorithm classification: the category of perception algorithms involved.

[0067] (2) Failure mode name: Provide a concise name for each failure.

[0068] (3) Failure description: Describe the failure situation in detail.

[0069] (4) Failure stage: Indicates the algorithm stage involved in the failure, such as training or inference.

[0070] (5) Cause of failure: List the causes of failure, such as technical defects.

[0071] The present invention proposes a method for mining failure modes of visual perception algorithms, which includes the following key steps.

[0072] a. Target Identification and Scope Definition

[0073] Determine the application scenarios, types, and functional scope of the visual perception algorithm, for example, from autonomous driving to medical imaging diagnosis. Further clarify its input and output data types, such as grayscale image input and target box output.

[0074] b. Analysis of Visual Perception Algorithm Components and Stages

[0075] Analyze the key components of visual perception algorithms, such as data preprocessing and feature extraction. At the same time, identify the life cycle stages of the algorithm, such as data collection, training, reasoning, and deployment, and pay attention to the failure mode characteristics of each stage.

[0076] c. Failure Mode Identification

[0077] Explore possible anomalies in algorithm execution, covering issues with the algorithm's structure, external environmental factors such as lighting changes, and the quality and characteristics of the input data.

[0078] d. Failure Cause Analysis

[0079] We delve into the core drivers behind each failure mode, combining the components and lifecycle of the algorithm. We analyze issues such as input errors, feature extraction differences, and decision outputs, as well as environmental, real-time, and hardware risks. We emphasize the interrelationships between the various parts of the algorithm to ensure its robust performance.

[0080] Step 2: Identify keywords in the description text, and associate each keyword with one or more nodes in the knowledge graph to construct a failure mode tree for the failure mode.

[0081] The present invention proposes a formal description framework for the failure modes of visual perception algorithms based on knowledge graphs, and its goal is to realize the automatic generation of formal description languages ​​through knowledge graphs.

[0082] Step 2.1: Acquisition of knowledge graph of visual perception algorithm failure

[0083] In order to realize the automatic generation process of formal description language, the present invention obtains a manually annotated algorithm failure knowledge graph. The notable features of this knowledge graph are the hierarchical and detailed structure. It is built around the core issue of algorithm failure, including several key nodes: failure label, algorithm classification, failure stage, deployment platform, application scenario, data set, etc. Each key node classifies the key elements of algorithm failure, which are further subdivided into multiple sub-nodes. For example, the algorithm classification node is divided into more specific categories such as target detection and image segmentation, and the application scenario node includes specific application fields such as autonomous driving and road monitoring. For each sub-node, there are various attributes that are further subdivided, such as specific algorithm attributes under the algorithm classification sub-node or trigger condition attributes under the failure stage sub-node, etc. These attributes provide detailed information for in-depth insight into various aspects of algorithm failure.

[0084] The nodes are linked by “is_a” type relationships, forming an interconnected, clear and explicit network structure, which enables analysis and understanding of the reasons for algorithm failure from multiple perspectives. The following is a detailed description of the knowledge graph.

[0085] Introduction to Algorithm Failure Knowledge Graph

[0086] Key points:

[0087] Failure Label: Failure mode category.

[0088] Algorithm classification: Types of visual perception algorithms.

[0089] Failure Phase: The phase in the algorithm where the failure mode occurs.

[0090] Deployment platform: refers to the platform on which the algorithm runs.

[0091] Application scenario: the specific scenario in which the algorithm is applied.

[0092] Dataset: refers to the data set used to train and test the algorithm.

[0093] Child nodes:

[0094] Child nodes of the dataset node: such as video, image, etc., indicating the dataset type.

[0095] Subnodes of the algorithm classification node: such as object detection, image segmentation, etc., represent specific algorithm categories. Under this node, "specific algorithms" are subdivided to represent specific visual perception algorithms.

[0096] Subnodes of the failure stage node: such as the dataset stage, algorithm training stage, etc., represent the specific activities or steps of the algorithm stage. Each stage has an attribute "trigger condition", such as one of the attributes of the dataset stage is "image exposure value greater than 1.5".

[0097] Subnodes of the deployment platform node: such as PC, mobile device, etc., representing the deployment platform of the algorithm.

[0098] Subnodes of the application scenario node: such as autonomous driving, road monitoring, etc., represent specific application scenarios.

[0099] Relationship:

[0100] In this knowledge graph, the relationship between the key node and the child node is an is_a relationship.

[0101] property:

[0102] Attributes of each node: such as the sample size of the data set, performance indicators of the algorithm, etc.

[0103] For each failure label, the present invention selects key failure trigger condition method parameters and determines the parameter range based on a large number of experiments. By determining these parameter values, the present invention can trigger specific failure modes under controlled conditions. On this basis, fuzzy testing technology can be used to determine the optimal key parameters from these parameter ranges and generate corresponding failure data sets accordingly. Such a method not only enhances the representativeness of the data set, but also improves the comprehensiveness and accuracy of the algorithm test.

[0104] 2.2 Linking process between failure modes and knowledge graph

[0105] The present invention links the failure modes of visual perception algorithms with knowledge graphs by using natural language processing related technologies in machine learning. This process includes the following steps:

[0106] 0) Get the failure mode description text

[0107] Use the failure mode mining method proposed in step 1 to conduct a comprehensive analysis of specific failure cases and failure scenarios to obtain failure modes. The description text contains the name of the failure mode, detailed description, occurrence stage and failure cause.

[0108] 1) Parse the failure mode description text:

[0109] When processing the description text of the algorithm failure mode, the text needs to be preprocessed first, the text needs to be standardized, such as removing irrelevant special characters, unifying the case, and performing word segmentation, that is, breaking the text into separate words or phrases. This process is crucial for cleaning data and eliminating noise, laying the foundation for deeper analysis. Next, word embedding techniques, such as Word2Vec, are used to process these words. In this way, each word is converted into a numerical vector, which can effectively capture the semantic relationship between words and provide processable input data for subsequent deep learning models. TF-IDF (term frequency-inverse document frequency) or deep learning-based methods (such as BERT's attention mechanism) are then used to extract keywords from the text. These technologies can identify key words that are closely related to failure modes, such as "algorithm classification", "failure stage", "deployment platform", etc., to identify and highlight the most critical information in the text. Finally, named entity recognition (NER) technology can be applied to identify specific entities in the text, such as specific algorithm names or deployment platforms, to provide a deeper understanding of the text. In this way, the failure mode description text is parsed.

[0110] 2) Construct failure mode tree:

[0111] The keywords extracted in 1) are used to locate the corresponding nodes in the knowledge graph, thereby realizing the linking process between the failure mode and the knowledge graph. This linking process matches each keyword with a predefined node in the knowledge graph, and associates each keyword with one or more nodes of the knowledge graph, thereby forming a direct link between the failure mode and the graph. Through these links, a relationship tree between the failure mode and the graph nodes is constructed. This tree reveals the multi-dimensional characteristics of the failure mode and its link relationship in the knowledge graph.

[0112] The core of this process is to use NLP technology to automatically identify and extract key information from text, and then effectively link this information to the knowledge graph to achieve a systematic and structured analysis of failure modes. In this way, the efficiency of analyzing algorithm failure modes can be greatly improved, providing effective support for the subsequent automatic generation of formal description languages.

[0113] For the failure mode "abnormal image exposure", the detailed description is as follows:

[0114] Failure mode name: Abnormal image exposure

[0115] Algorithm classification: target detection algorithm, face recognition algorithm, image classification algorithm.

[0116] Failure description: This failure mode is common in fields such as autonomous driving and safety monitoring. It usually occurs in the Pytorch framework on Windows or Linux platforms. The algorithm in which this failure mode occurs can be run using the CPU or GPU. In this failure mode, the algorithm may not be able to correctly handle overexposed or underexposed images. This may result in loss of image details, color distortion, or important features in the image cannot be correctly identified. For example, in the case of overexposure, the image may be too bright and lose the necessary contrast and details; in the case of underexposure, the image may be too dim and difficult to recognize.

[0117] Failure stage: This type of failure usually occurs in the dataset stage and the algorithm training stage.

[0118] Failure reasons: This failure mode may have multiple reasons. If the training data set lacks over-exposed or under-exposed images or videos, the algorithm will not be able to adapt to abnormal exposure conditions that may occur in actual applications. If the model itself has too few parameters or the model structure is too simple, the algorithm may also fail.

[0119] For this failure mode, this method uses the BERT model to extract text features and extracts keywords such as "target detection algorithm", "face recognition algorithm", "image classification algorithm", "autonomous driving", "safety monitoring" and "exposure". Then the BFS (breadth-first search) method is used to locate the nodes corresponding to the extracted keywords in the knowledge graph. For example, "target detection algorithm" is associated with the "algorithm classification" node in the knowledge graph, and "exposure" is associated with the "failure label" node.

[0120] According to the association between the extracted keywords and knowledge graph nodes, such as Figure 3 As shown, the present invention uses a tree structure data model to construct a failure mode tree. In the constructed tree, nodes represent extracted keywords and their corresponding knowledge graph nodes, and edges represent the association relationship between these keywords and nodes.

[0121] Step 3: Generate a formal description grammar architecture for the failure mode.

[0122] The present invention develops a formal language description framework, which adopts a comprehensive and highly extensible multi-dimensional description strategy. This framework comprehensively includes the logical structure and grammatical system of formal description. By adopting a standardized description language, the failure mode of the visual perception algorithm can be deeply analyzed from multiple angles and dimensions, which also provides a solid foundation for automatically generating test cases.

[0123] In the process of establishing the logical structure of the description, the present invention conducts in-depth exploration of the diversity of failure causes and adopts a comprehensive and scalable multi-dimensional description strategy. Taking the failure stage as an example, the strategy covers six core stages, namely: data set, training, reasoning, deployment, algorithm update and destruction. Each stage is further subdivided into several sub-dimensions, and deeper subdivisions are set up when necessary to ensure that potential influencing factors are captured in all aspects, while also providing sufficient flexibility to cope with various algorithm application backgrounds. For example, when a failure mode is specific to a detail of the data set, training, or reasoning, this description system allows selective description in related fields to ensure the accuracy and pertinence of the overall description.

[0124] The formal framework follows a tree-like grammatical architecture. In this system, the failure mode is placed at the top level as the root node. Subsequently, according to the stages and specific dimensions involved in the failure mode, the present invention sets up hierarchical sub-nodes for it. These sub-nodes not only include failure stages and specific dimensions, but may also be further split into more detailed sub-dimensions to ensure that technicians can describe failure modes from multiple angles and levels. In order to store these formal descriptions of failure modes in a standardized manner, the present invention uses the JSON format to define each failure mode and its corresponding triggering conditions. In this description, each failure mode consists of one or more attributes and corresponding triggering conditions. For example, a failure mode may be triggered by both an image exposure problem and a collection problem.

[0125] The formal description grammar architecture provides the ability to fully describe the failure modes of visual perception algorithms, allowing for more targeted analysis. Using this grammar system, software developers and analysts can describe and classify failure modes in a more systematic and standardized manner, while also laying a solid foundation for automatic test case generation and providing precise guidance for subsequent fault analysis and solution development. Figure 4 A flowchart of the formal description framework is given below.

[0126] Step 4: For each node in the failure mode tree, a nested object in a corresponding formal description statement is generated based on the formal description grammar architecture, thereby obtaining a formal description statement.

[0127] The present invention develops a method for automatically generating a formal description language based on knowledge graph, focusing on accurately and quickly describing the failure modes of visual perception algorithms. The core of this method is to automatically generate a formal description of the algorithm failure mode by traversing the failure mode tree.

[0128] Specifically, the generation of the formal description statement includes the following steps 4.1 to 4.3.

[0129] Step 4.1: Perform formal analysis on the description text.

[0130] In the formal analysis of failure modes, the present invention adopts a multi-step strategy. First, a preliminary classification and identification is performed according to the stages and causes involved in the failure modes. Next, each failure mode is analyzed and described in detail with reference to the previously established logical structure.

[0131] In order to ensure the accuracy and depth of the description, the present invention also determines the specific triggering conditions for each failure mode, ensuring that these conditions are specific, measurable and closely related to the actual application scenario. In the process of determining the triggering conditions, different parameters and conditions may affect the occurrence and manifestation of the failure mode. In order to ensure the comprehensiveness and accuracy of the description, the present invention considers the relationship between these triggering conditions and failure modes, and sets a suitable type and value range for each parameter.

[0132] Step 4.2: Generate the corresponding nested format in the formal description statement based on the formal description grammar architecture.

[0133] The root node of the failure mode tree is the failure mode name, which represents a specific failure mode, such as "abnormal image exposure failure". Next, the system will traverse each node under the root node in turn. These nodes cover various failure information of the algorithm, including algorithm classification, failure stage, deployment platform, etc.

[0134] The system further traverses and analyzes each child node until all leaf nodes are traversed to obtain more detailed information about the failure mode. For example, under the node of the failure stage, there may be child nodes such as algorithm training, algorithm reasoning, etc. Each child node may also contain more specific trigger condition attributes to reveal different levels of the failure mode.

[0135] In the process of traversing the failure mode tree and automatically generating formal description statements, for each traversed node, if it is at a key node position in the knowledge graph, it is presented as a nested object in the formal statement, and its child nodes are presented as key values ​​within the nested object; if it is a child node, it is presented as an object nested in the key node of the previous level; if it is an attribute node, it exists as an attribute of the child node of the previous level. Based on this hierarchical structure, the system automatically constructs a formal description language. The language follows clear grammatical rules and logical frameworks to ensure that the generated description is not only complete, but also logically rigorous. The formal description starts from the root node and goes deeper layer by layer until all relevant child nodes and subdivision dimensions are covered, thereby describing the failure mode in an all-round way.

[0136] Step 4.3: Using the nodes in the failure mode tree and the parameter type and parameter range of the failure trigger condition, the nested format is filled with content to obtain a nested object in a formal description statement.

[0137] The following is a formal description statement of the "abnormal image exposure" failure mode.

[0138]

[0139]

[0140] Step 5: Generate test cases based on the formal description statements.

[0141] Failure test cases for visual perception algorithms are designed to trigger and detect failure behaviors of algorithms under specific conditions. In order to more accurately observe the failure nodes of algorithm performance, this method generates failure data sets that cause the algorithm to degrade to different degrees as test cases. These test cases not only evaluate the performance of the algorithm under non-ideal conditions, but also help to discover its potential defects and guide optimization. When generating test cases, it is necessary to ensure that various failure scenarios are fully covered and use quantitative evaluation indicators, such as recognition confidence. Figure 5 Generate a flowchart for the failure test case, including steps 3.1 to 3.4.

[0142] Step 5.1: Determine the algorithm version and preliminary parameters.

[0143] Before generating test cases, it is necessary to identify the specific version of the visual perception algorithm being tested, and in order to generate test cases for failure modes, it is necessary to select specific failure parameters and preliminarily give their ranges to ensure that when generating test samples, they are within a reasonable range and no errors such as out-of-bounds errors occur. For example, when the exposure of the test image is abnormal, it is necessary to set the parameter range of the exposure level to ensure that the test case can correctly and accurately simulate the exposure problems that may be encountered in practice.

[0144] Step 5.2: Automatically generate test cases based on formal description.

[0145] The present invention constructs a parsing system of a formal description language. Through this description language, the present invention can provide the necessary information for generating failure mode test cases in a standardized manner. The parsing system receives the formal description language as input, and according to the information contained therein, adopts the strategy of fuzzy testing to automatically output test cases.

[0146] Step 5.3: Fuzz testing.

[0147] The fuzz testing of this method includes but is not limited to structural disturbances of input data, format changes, or other factors that may interfere with the normal operation of the algorithm.

[0148] This paper improves fuzz testing and proposes an automatic parameter tuning method to guide parameter adjustment in fuzz testing. Automatic parameter tuning predicts and optimizes input parameter combinations through machine learning models and automatically identifies conditions that may cause algorithm failure. This method reduces the tedious process of manual parameter adjustment and improves test efficiency and coverage.

[0149] Step 5.3.1: Randomly sample the parameters.

[0150] The method first generates initial parameter samples using random sampling within the parameter range given in step 5.1. These samples are used to explore the parameter space and provide basic data for subsequent optimization. Next, an objective function is defined to evaluate the effectiveness of each set of parameter combinations.

[0151] Step 5.3.2: Use the initial failure parameter sample for training to obtain the optimal parameter combination.

[0152] In order to better optimize the parameters, the present invention uses the initial sample to train the machine learning model, which is used to predict the failure area in the parameter space. The model improves the accuracy of the prediction by continuous learning and updating. Iterative optimization algorithms such as Bayesian optimization are used to search for the optimal parameter combination in the parameter space. In each round of iteration, the optimization algorithm selects a new parameter combination for testing, updates the model, and gradually approaches the failure area of ​​the specified failure degree. When the optimization algorithm no longer significantly improves the objective function value after multiple rounds of iterations, or reaches the preset maximum number of iterations, the optimization process is stopped. At this point, the optimal parameter combination has been found and can be used to generate further failure test cases.

[0153] Step 5.3.3: Next, the optimal parameter combination will be passed into the code file generated by the corresponding failure test case. Based on this parameter combination, a test data set with different degrees of failure of this failure mode will be systematically generated to further test and verify the robustness of the algorithm.

[0154] Taking the failure mode of abnormal image exposure as an example, the test model selected by the present invention is YOLOv5, and its formal description statement has been given in step 2. The process of generating test cases is as follows:

[0155] 1) Model selection YOLOv5, a widely used real-time object detection model, is used to process images with its efficient performance and good generalization ability. The failure parameter brightness_value is selected as the failure parameter, and its value range is set to 0 to 9. The sample points are selected from the set brightness range using a random sampling method. Samples are taken every 0.5 units from the range 0 to 9 to obtain a set of 20 images with different brightness levels. Different brightness adjustments are applied to the sampled images to generate exposure abnormal images.

[0156] 2) Use YOLOv5 to identify targets in images with abnormal exposure and record the target detection confidence of each image. Pair the brightness parameter with the confidence of the corresponding image as the input data of the machine learning model. The model uses the logistic regression model because it is simple and effective in dealing with binary classification problems. The goal here is to classify images where the brightness adjustment causes detection failure.

[0157] 3) Use the data generated in the previous step to train the model and determine the relationship between the brightness adjustment value and the confidence failure. Through analysis, it is found that when the brightness value is between 3 and 7, the confidence of target detection decreases significantly. According to the model output, the most sensitive brightness failure interval is determined to be 3 to 7. This means that within this range, the exposure level of the image is most likely to cause YOLOv5 recognition failure.

[0158] 4) Use the Bayesian optimization algorithm to optimize the brightness parameters, with the goal of maximizing the target detection failure rate. First, based on the initial test results, a prior probability model is constructed to predict the impact of brightness adjustment on detection confidence. 5% of the images in the dataset are selected for failure test case generation to ensure sufficient test coverage while controlling the consumption of computing resources. Next, iterative optimization is performed. In each iteration, new brightness parameters are selected for testing based on the failure probability predicted by the current model, and then the model is updated based on the actual test results.

[0159] 5) Based on the feedback from the Bayesian optimization algorithm, the brightness parameters are continuously adjusted until the preset failure test target is reached.

[0160] 6) Find the brightness parameter that can cause the specified failure level, and record this parameter value and the corresponding test data set.

[0161] 7) The finalized failure parameters and data set are passed into a dedicated test case generation script to automatically generate test cases with a specified degree of failure for the entire data set.

[0162] Step 5.4: Result evaluation and feedback.

[0163] After using the automatically generated test cases, the present invention will collect the performance data and response results of the algorithm. Based on these data, the present invention will analyze the failure parameters of the visual perception algorithm, observe the distribution of its failure parameters, the correlation between the algorithm performance and the failure parameters, etc. The results can guide the adjustment and improvement of the test case generation strategy, optimize the failure parameter variation range, use new test cases and the adjusted algorithm to test again, collect and analyze performance data to verify the effect of the adjustment, and ensure that all potential failure modes have been effectively covered.

[0164] In summary, the present invention can realize the automatic generation of a formal description language through the knowledge graph of failure modes. The formal description language can provide a comprehensive and standardized description of failure modes, avoid ambiguity, and lay the foundation for the subsequent automatic generation of test cases.

[0165] The present invention realizes the automatic generation of test cases based on the formal description language of failure modes. The failure test cases of visual perception algorithms are intended to trigger and detect the failure behavior of the algorithm under specific conditions. In order to more accurately observe the failure nodes of the algorithm performance, this method will generate failure data sets that cause the algorithm to decline to different degrees as test cases. These test cases not only evaluate the performance of the algorithm under non-ideal conditions, but also help to discover its potential defects, thereby guiding optimization. When generating test cases, it is necessary to ensure that various failure scenarios are fully covered, and to use quantitative evaluation indicators, such as recognition confidence.

[0166] Exemplary Devices

[0167] Figure 6 A visual perception algorithm failure mode formalization test device is shown according to an exemplary embodiment. Figure 6 The device includes: a description text generation module 610, a failure mode tree construction module 620, a grammar architecture generation module 630, a formal description module 640 and a test case generation module 650.

[0168] Optionally, the description text generation module 610 is further used to: preprocess the description text; decompose the preprocessed description text into separate words or phrases; convert each word or phrase into a numerical vector; and obtain keywords in the description text based on the numerical vector.

[0169] Optionally, the knowledge graph consists of several key nodes V i With this key node V i The key node V i Includes: failure labels, algorithm classification, failure stages, deployment platforms, application scenarios and data sets.

[0170] Optionally, the failure mode tree construction module 620 is further used to: use the failure mode name as the root node of the failure mode tree; use a depth-first search algorithm to locate a node corresponding to the extracted keyword in the knowledge graph; and when the located node is a key node V in the knowledge graph, i In the case of i , and based on the key node V i The properties of the key node U i Generate an attribute node; the located node is the key node V in the knowledge graph i Child nodes of In the case of i The child node below And based on the child node The attributes of the child node Generate an attribute node; where i and j are natural numbers, and t is the depth of the child node.

[0171] Optionally, at the key node V i In the case of a failure stage, the grammar architecture generation module 630 is also used to: use the failure mode name as the root node of the formal description grammar architecture; use the failure stage as the child node of the root node of the formal description grammar architecture; wherein the failure stage includes: data set stage, training stage, reasoning stage, deployment stage, update stage and destruction stage; according to the scenario of mining the visual perception algorithm, obtain the multi-level dimensions of each failure stage, and generate the branch of the failure stage based on the multi-level dimensions; wherein each dimension contains the attributes of the dimension and the failure condition that triggers the failure of the visual perception algorithm under the dimension.

[0172] Optionally, the formal description module 640 is also used to: perform formal analysis on the description text; generate a corresponding nested format in the formal description statement based on the formal description grammar architecture; use the nodes in the failure mode tree and the parameter type and parameter range of the failure trigger condition to fill the nested format with content to obtain a nested object in the formal description statement.

[0173] Optionally, the formal description module 640 is further configured to: when a node in the failure mode tree is a failure stage, present the failure stage as a nested object in the formal statement, and present all child nodes under the failure stage as nested objects. As a key value in a nested object; a node in the failure mode tree is a child node In the case of and the parameter type and parameter range of the failure trigger condition as the key node U i An object nested in is presented; when the node in the failure mode tree is an attribute node, the attribute node is used as the attribute of the upper-level child node.

[0174] Optionally, the test case generation module 650 is also used to: determine the algorithm version and preliminary parameters of the visual perception algorithm; apply the formal description statement in combination with the algorithm version and preliminary parameters of the visual perception algorithm to generate an initial test case; use the initial test case to train the machine learning model to predict the failure area in the parameter space; explore the optimal parameter combination in the failure area, and apply the optimal parameter combination to the formal description statement to obtain the final test case.

[0175] The exemplary device is an embodiment of the device corresponding to the above exemplary method. The specific operations of each module can be understood by referring to the description of the method embodiment, which will not be repeated here.

[0176] Exemplary Electronic Devices

[0177] Figure 7 1 is a block diagram of an electronic device 700 according to an exemplary embodiment. The electronic device 700 may be a terminal, a notebook computer, a desktop computer, a server, a computer cluster or other types of electronic devices.

[0178] Reference Figure 7 , the electronic device 700 may include at least one processor 710 and a memory 720. The processor 710 may execute instructions stored in the memory 720. The processor 710 is communicatively connected to the memory 720 via a data bus. In addition to the memory 720, the processor 710 may also be communicatively connected to an input device 730, an output device 740, and a communication device 750 via a data bus.

[0179] The processor 710 may be any conventional processor, such as a central processing unit (CPU), a graphic processing unit (GPU), a field programmable gate array (FPGA), a system on chip (SOC), an application specific integrated circuit (ASIC), or a combination thereof.

[0180] The memory 720 may be implemented by any type of volatile or nonvolatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk.

[0181] In the embodiment of the present disclosure, executable instructions are stored in the memory 720, and the processor 710 can read the executable instructions from the memory 720 and execute the instructions to implement all or part of the steps of the formal testing method for the failure mode of the visual perception algorithm in the above exemplary embodiment.

[0182] Exemplary computer-readable storage media

[0183] In addition to the above methods and devices, the exemplary embodiments of the present disclosure also include a computer program product or a computer-readable storage medium storing the computer program product. The computer product includes computer program instructions that can be executed by a processor to implement all or part of the steps described in the above exemplary embodiments.

[0184] The computer program product may be written in any combination of one or more programming languages ​​to write program codes for performing the operations of the embodiments of the present application, including object-oriented programming languages, such as Java, C++, etc., and also conventional procedural programming languages, such as "C" language or similar programming languages ​​and scripting languages ​​(e.g., Python). The program code may be executed entirely on the user computing device, partially on the user computing device, as an independent software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0185] Computer readable storage media can adopt any combination of one or more readable media. The readable medium can be a readable signal medium or a readable storage medium. The readable storage medium can include, for example, but is not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices or devices, or any combination of the above. More specific examples of readable storage media include: static random access memory (SRAM) with one or more wires electrically connected, electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk or optical disk, or any suitable combination of the above.

[0186] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the present disclosure. The present disclosure is intended to cover any variations, uses or adaptations of the present disclosure, which follow the general principles of the present disclosure and include common knowledge or customary technical means in the art that are not disclosed in the present disclosure. The specification and examples are intended to be exemplary only, and the present disclosure is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes may be made without departing from the scope thereof.

Claims

1. A formalized testing method for failure modes of visual perception algorithms, characterized in that: The method comprises: Mining possible failure modes that may occur during the execution of visual perception algorithms and generating description text for each failure mode; Identify keywords in the description text, and associate each keyword with one or more nodes in the knowledge graph to construct a failure mode tree for the failure mode; Generate a formal description grammar architecture of the failure mode; For each node in the failure mode tree, generating a nested object in a corresponding formal description statement based on the formal description grammar architecture, thereby obtaining a formal description statement; Based on the formal description statement, a test case is generated.

2. The method according to claim 1, characterized in that The identifying keywords in the description text includes: Preprocessing the description text; Decomposing the preprocessed description text into individual words or phrases; Convert each word or phrase into a numeric vector; Based on the numerical vector, keywords in the description text are obtained.

3. The method according to claim 1, characterized in that The knowledge graph consists of several key nodes V i With this key node V i The key node V i Includes: failure labels, algorithm classification, failure stages, deployment platforms, application scenarios and data sets.

4. The method according to claim 3, characterized in that: The step of associating each keyword with one or more nodes in the knowledge graph to construct a failure mode tree of the failure mode includes: Using the failure mode name as the root node of the failure mode tree; Use the depth-first search algorithm to locate the nodes corresponding to the extracted keywords in the knowledge graph; The located node is the key node V in the knowledge graph i In the case of i , and based on the key node V i The properties of the key node U i Next, generate an attribute node; The node being located is the key node V in the knowledge graph. i Child nodes of In the case of i The child node below And based on the child node The attributes of the child node Generate an attribute node; where i and j are natural numbers, and t is the depth of the child node.

5. The method according to claim 4, characterized in that At the key node V i In the case of the failure stage, the formal description grammar architecture of the failure mode is generated, including: Using the failure mode name as the root node of the formal description grammar architecture; The failure stage is used as a child node of the root node of the formal description grammar architecture; wherein the failure stage includes: data set stage, training stage, reasoning stage, deployment stage, update stage and destruction stage; According to the scenario of mining the visual perception algorithm, a multi-level dimension of each failure stage is obtained, and a branch of the failure stage is generated based on the multi-level dimension; wherein each dimension includes the attributes of the dimension and the failure condition that triggers the failure of the visual perception algorithm under the dimension.

6. The method according to claim 4, characterized in that For each node in the failure mode tree, generating a corresponding nested object in a formal description statement based on the formal description grammar architecture includes: Conduct formal analysis of descriptive texts; Generate the corresponding nested format in the formal description statement based on the formal description grammar architecture; The nested format is filled with content using the nodes in the failure mode tree and the parameter type and parameter range of the failure triggering condition to obtain the nested objects in the formal description statement.

7. The method according to claim 5, characterized in that The nested format is filled with content using the nodes in the failure mode tree and the parameter type and parameter range of the failure trigger condition to obtain the nested objects in the formal description statement, including: When a node in the failure mode tree is a failure stage, the failure stage is presented as a nested object in the formal statement, and all child nodes under the failure stage are As a key value within a nested object; The nodes in the failure mode tree are child nodes. In the case of and the parameter type and parameter range of the failure trigger condition as the key node U i An object nested in is presented; When a node in the failure mode tree is an attribute node, the attribute node is used as an attribute of an upper-level child node.

8. The method according to claim 1, characterized in that The generating of test cases based on the formal description statement includes: Determining the algorithm version and preliminary parameters of the visual perception algorithm; Applying the formal description statement in combination with the algorithm version of the visual perception algorithm and preliminary parameters to generate an initial test case; Train a machine learning model using the initial test cases to predict failure regions in parameter space. An optimal parameter combination is explored in the failure region, and the optimal parameter combination is applied to the formal description statement to obtain a final test case.

9. A visual perception algorithm failure mode formalization test device, characterized in that: The device comprises: The description text generation module is used to mine the possible failure modes that may occur during the execution of the visual perception algorithm and generate a description text for each failure mode; A failure mode tree construction module is used to identify keywords in the description text and associate each keyword with one or more nodes in the knowledge graph to construct a failure mode tree for the failure mode; A grammar architecture generation module, used for generating a formal description grammar architecture of the failure mode; A formal description module, for generating, for each node in the failure mode tree, a nested object in a corresponding formal description statement based on the formal description grammar architecture, thereby obtaining a formal description statement; The test case generation module is used to generate test cases based on the formal description statements.

10. An electronic device, characterized in that: The electronic device comprises: a processor and a memory storing computer program instructions; when the processor executes the computer program instructions, the formalized testing method for failure modes of a visual perception algorithm as described in any one of claims 1 to 8 is implemented.

Images