Performance evaluation method of visual algorithm, electronic equipment and program product

Abstract

The invention relates to the technical field of computers, in particular to a visual algorithm performance evaluation method, electronic equipment and a program product, and the method comprises the steps: obtaining a test demand of a to-be-tested visual algorithm; determining a target test item corresponding to the test demand according to the life cycle stage to which the to-be-tested visual algorithm belongs, and generating a target test case according to the test input of the target test item; and executing the test case to obtain a performance evaluation result of the to-be-tested visual algorithm. According to the method and the device, the target test item and the target test case corresponding to the visual intelligent algorithm test requirement can be designed from the visual intelligent algorithm test requirement, so that the test efficiency and the test quality can be improved.

Description

Technical Field

[0001] The present disclosure relates to the field of computer technology, and in particular to a performance evaluation method for a visual algorithm, an electronic device, and a program product. Background Art

[0002] In the 21st century, an era of rapid technological development, the advancement of artificial intelligence technology has made the digital economy the dominant force in global economic development. However, as digital transformation deepens, artificial intelligence security faces increasing challenges and risks. Data flows and is applied in various forms among various entities and in different scenarios. Artificial intelligence security issues involve confidentiality, integrity, availability, and compliance, as well as the security of algorithms, models, applications, and systems. Traditional information security risk management methods can no longer adapt to the rapid development and changes of digital business, and cannot achieve risk control throughout the entire life cycle of artificial intelligence.

[0003] Although deep neural networks have shown effectiveness in solving complex problems, their application in safety-critical environments is still limited, mainly because neural network models are often viewed as black boxes whose predictive behaviors cannot be reasonably explained, and their properties are difficult to prove in theory. According to the "2022-2027 Global and China Big Data Market Business Opportunities and Investment Strategy Research Report" released by China Business Industry Research Institute, my country's artificial intelligence technology is mainly concentrated at the perception level, and computer vision artificial intelligence accounts for about 52.55% of the artificial intelligence industry. Against this background, it is urgent to establish an evaluation method for visual intelligent algorithms. Summary of the invention

[0004] In order to solve the above technical problems or at least partially solve the above technical problems, the present disclosure provides a performance evaluation method of a visual algorithm, an electronic device and a program product.

[0005] In a first aspect, the present disclosure provides a performance evaluation method for a visual algorithm, comprising:

[0006] Obtain the test requirements of the visual algorithm to be tested;

[0007] According to the life cycle stage of the visual algorithm to be tested, determine the target test items corresponding to the test requirements, and generate target test cases based on the test input of the target test items;

[0008] Execute the test cases to obtain the performance evaluation results of the visual algorithm to be tested.

[0009] In a second aspect, the present disclosure provides an electronic device comprising: a processor, a memory, and a computer program stored in the memory and executable on the processor, wherein when the computer program is executed by the processor, the performance evaluation method of the visual algorithm as described in the first aspect or any one of its optional embodiments is implemented.

[0010] In a third aspect, the present disclosure provides a computer program product, including: the computer program product includes a computer program, and when the computer program runs on a computer, the computer implements the performance evaluation method of the visual algorithm as described in the first aspect or any optional embodiment thereof.

[0011] Compared with the prior art, the technical solution provided by the embodiments of the present disclosure has the following advantages:

[0012] The performance evaluation method of the visual algorithm provided by the embodiment of the present disclosure first obtains the test requirements of the visual algorithm to be tested, then determines the target test items corresponding to the test requirements according to the life cycle stage of the visual algorithm to be tested, generates the target test case according to the test input of the target test item and then executes the test case to obtain the performance evaluation result of the visual algorithm to be tested. In this way, the embodiment of the present disclosure provides a method for evaluating the comprehensive performance of the visual algorithm, starting from the test requirements, applicable to each stage of the life cycle of the visual algorithm, designing the target test items to generate the target test cases, and the performance evaluation results obtained in this way can have a high degree of matching with the test requirements, which is conducive to improving the test efficiency and test quality. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present disclosure and, together with the description, serve to explain the principles of the present disclosure.

[0014] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0015] Figure 1 Schematic diagram of the performance evaluation method of the visual algorithm provided in the embodiment of the present disclosure Figure 1 ;

[0016] Figure 2 A schematic diagram of the life cycle and testing activities of a visual intelligent algorithm provided in an embodiment of the present disclosure;

[0017] Figure 3 Schematic diagram of the performance evaluation method of the visual algorithm provided in the embodiment of the present disclosure Figure 2 ;

[0018] Figure 4 A schematic diagram of evaluation indicators provided in an embodiment of the present disclosure;

[0019] Figure 5 A schematic diagram of the structure of an electronic device provided in an embodiment of the present disclosure. DETAILED DESCRIPTION

[0020] In order to more clearly understand the above-mentioned objectives, features and advantages of the present disclosure, the scheme of the present disclosure will be further described below. It should be noted that the embodiments of the present disclosure and the features in the embodiments can be combined with each other without conflict.

[0021] In the following description, many specific details are set forth to facilitate a full understanding of the present disclosure, but the present disclosure may also be implemented in other ways different from those described herein; it is obvious that the embodiments in the specification are only part of the embodiments of the present disclosure, rather than all of the embodiments.

[0022] The focus of testing and evaluation methods for visual intelligent algorithms is on understanding requirements and selecting evaluation methods. However, current applications often ignore the process of understanding requirements and lack the transition from testing requirements to selecting evaluation methods.

[0023] From the perspective of testing requirements, the objects processed by visual intelligent algorithms include not only data, but also knowledge, as well as the ability to represent, acquire, access and process knowledge;

[0024] Visual intelligent algorithms often use artificial intelligence problem-solving models to obtain results. The solution methods are mainly divided into three categories: search, reasoning and planning. Compared with the traditional solution model, there are three obvious characteristics: its problem-solving algorithm is often non-deterministic or heuristic, its problem solving relies heavily on knowledge, and the problems of visual intelligent algorithms often have exponential computational complexity; visual intelligent algorithms have the ability to sense the scene (environmental adaptation) and adapt to the scene. This sensing ability includes perception, learning, reasoning, judgment and making corresponding actions.

[0025] In summary, since visual intelligent algorithms need to learn from data or experience, they have characteristics such as uncertain output results and unexplainable decision-making processes. Therefore, whether visual intelligent algorithms can be successfully executed not only involves the algorithm itself, but also the algorithm application. All stages of the life cycle of visual intelligent algorithms face execution risks at the algorithm, data and environment levels.

[0026] It can be seen that the inherent ability of visual intelligent algorithms to solve learning problems and uncertain behavior make human users doubt whether visual intelligent algorithms can cope with the uncertainty of situation changes. More importantly, they ignore the process of understanding test requirements and lack a complete process from test requirements to evaluation methods, resulting in low test efficiency and quality.

[0027] Based on this, the embodiments of the present disclosure provide a performance evaluation method for a visual algorithm, which starts from the testing requirements of visual intelligent algorithms, can reduce the work difficulty of testers, improve testing efficiency, and help improve testing quality.

[0028] Reference Figure 1 The method provided by the embodiment of the present disclosure includes the following steps S101 to S103:

[0029] Step S101, obtaining the test requirements of the visual algorithm to be tested, wherein the visual algorithm to be tested is a visual intelligent algorithm whose performance needs to be evaluated.

[0030] In some embodiments, the test requirements input by the user for the visual algorithm to be tested are obtained. The test requirements are a series of descriptions and constraint information about the function, performance, etc. of the object to be tested (the visual algorithm to be tested) proposed by the user. The test requirements are closely related to the user and are the system inputs expected by the user, while the test items and test cases are the system outputs expected by the user.

[0031] Testing requirements generally include but are not limited to at least one of the following: test level, test type and software to be tested, as well as descriptions of test contents such as functional points and performance points.

[0032] Test requirements are generally obtained from the following four types of data: The first type is the intelligent algorithm development task book, which describes the requirements for the operating environment, technology, design, and data set of the intelligent algorithm. The second type is the intelligent algorithm requirement specification, which points out the capability requirements, interface requirements, internal data requirements, etc. that the intelligent algorithm needs to meet. The third type is the intelligent algorithm design description, which points out and clarifies the requirements for the program module, data structure, data flow, and processing flow of the intelligent algorithm. The fourth type is the intelligent algorithm user manual, which guides the operation and use of the intelligent algorithm.

[0033] Step S102, determining a target test item corresponding to the test requirement according to the life cycle stage of the visual algorithm to be tested, and generating a target test case according to the test input of the target test item.

[0034] The test input is the input data of the target test item, which is used as the constraint condition of the target test item to ensure the feasibility and practicality of the target test case. The target test case generated according to the constraint condition of the target test item is sufficient.

[0035] It should be noted that the process of developing a visual intelligent algorithm includes: determining the target scene, selecting a suitable data set or labeling a data set that is suitable for the target scene, and then selecting a suitable algorithm model based on the determined target scene. Further use the processed data set to train the optimized model, and finally deploy the optimized model to the system to complete the development of the visual intelligent algorithm. Based on this, refer to Figure 2 The life cycle of intelligent vision algorithm projects includes the following five stages: problem definition stage, data collection stage, feature engineering stage, model training stage, and deployment and application stage. Different stages correspond to different test items. For example, the test items corresponding to the problem definition stage are used to review test requirements; the test items corresponding to the data collection stage are used to evaluate the quality of the data set in terms of content, annotation criteria, data quality, etc.; the test items corresponding to the feature engineering stage are used to evaluate whether the test data meets the test requirements; the test items corresponding to the model training stage are used to test the code and test the model; the test items corresponding to the deployment and application stage are used to test the system and monitor the operation of the system.

[0036] The disclosed embodiment determines the target test item corresponding to the test requirement according to the life cycle stage to which the visual algorithm to be tested belongs, and generates a target test case based on the test input of the target test item; it can be understood that the test requirement indicates the object / thing that needs to be tested, the target test item indicates how to test, and the target test case is a refinement of the test item. The target test item serves as a bridge connecting the test requirements and the target test case, covering the test requirements, and designing the target test case based on the target test item. It should be noted that after the target test case is automatically generated, the tester can consider the interactive impact of various factors and improve the target test case to formulate and design a more comprehensive and reasonable test case.

[0037] Step S103, executing the test case to obtain a performance evaluation result of the visual algorithm to be tested.

[0038] This embodiment constructs a comprehensive performance test and evaluation method for visual intelligent algorithms. Starting from the target test requirements, a universal evaluation method is proposed at a higher level. When conducting comprehensive performance test and evaluation of specific visual intelligent algorithms, this method can be used as a guide, and specific evaluation details, including evaluation indicators, methods, etc., are implemented under the method. The performance evaluation results obtained in this way can have a high degree of match with the test requirements, and can assist testers to scientifically and effectively consider the interactive effects of various factors, formulate and design more comprehensive and reasonable test cases, and thus help improve test efficiency and test quality.

[0039] For ease of understanding, the above method is described in detail in the embodiments of the present disclosure.

[0040] In some embodiments, when the life cycle stage of the visual algorithm to be tested is the problem definition stage, the target test item corresponding to the test requirement is determined, and a test case corresponding to the target test item is generated. At this time, the target test item indicates that the test requirement is tested based on the algorithm design requirement table. Among them, the algorithm design requirement table is the requirement based on which the visual algorithm to be tested is designed and developed. It can be understood that the visual algorithm to be tested is developed and designed in accordance with the algorithm design requirement. During testing, it is necessary to compare the algorithm design requirement table to evaluate whether the test requirement is reasonable.

[0041] In some embodiments, when the life cycle stage of the visual algorithm to be tested is the data collection stage, the target test item corresponding to the test requirement is determined, and a test case corresponding to the target test item is generated. At this time, the target test item indicates a data set collected based on the algorithm design requirement, and the test data set corresponding to the test requirement is tested.

[0042] In some embodiments, when the life cycle stage of the visual algorithm to be tested is the feature engineering stage, the target test item corresponding to the test requirement is determined, and a test case corresponding to the target test item is generated. At this time, the target test item indicates a feature constructed based on the algorithm design requirement, and the test feature corresponding to the test requirement is tested.

[0043] In some embodiments, when the life cycle stage of the visual algorithm to be tested is the model training stage, or the deployment application stage, such as Figure 3 As shown, the embodiment of the present disclosure describes the above step S102, including the following steps S301-S302:

[0044] S301. Select evaluation indicators corresponding to the test requirements according to the model training stage or the deployment and application stage;

[0045] The evaluation indicators corresponding to the test requirements include but are not limited to at least one of the following: (1) algorithm correctness indicator, (2) algorithm convergence indicator, (3) algorithm robustness indicator, (4) algorithm efficiency indicator, (5) code standardization indicator, (6) algorithm dependency indicator, (7) algorithm enhanceability indicator, (8) algorithm computing power indicator and (9) algorithm recoverability indicator.

[0046] At different life cycle stages, different testing requirements have different corresponding evaluation indicators.

[0047] S302: Determine the target test items corresponding to the evaluation indicators and generate target test cases.

[0048] Each evaluation indicator corresponds to a certain test content, based on which the target test item is determined. The target test item includes the test content corresponding to one or more evaluation indicators.

[0049] The target test case is a strategy for executing the target test items in a certain test environment and in a certain execution order and verifying the results.

[0050] This embodiment describes the evaluation indicators in step S301 above. Figure 4 , the evaluation indicators corresponding to the test requirements are as follows:

[0051] (1) Algorithm Correctness Index

[0052] Correctness indicates the process of evaluating or verifying the accuracy and effectiveness of the machine learning model and system in running the specified tasks and user goals and the results produced. It includes test elements such as data accuracy satisfaction, functional correctness of model design, correctness of code implementation, correctness of calculation results, etc. It is used to evaluate whether the functions implemented by the deep learning algorithm meet the algorithm design requirements, and should include but not be limited to algorithm capability evaluation indicators and code implementation correctness indicators. The specific contents are as follows:

[0053] 1) Algorithm capability evaluation indicators

[0054] In the model training phase or deployment application phase, task-related basic indicators can be selected based on the actual application scenario to evaluate the algorithm's ability to complete the function. It can be understood that the test requirements include application scenarios and / or tasks.

[0055] For example, different types of tasks correspond to different algorithm capability evaluation indicators, such as: word error rate, sentence error rate, etc. in speech recognition tasks; average accuracy rate, etc. in target detection tasks; the impact of the degree of error deviation in the use of the algorithm, etc.

[0056] ① Regression task: Use the mean absolute error (Mean AbsoluteError), mean square error (Mean Square Error), root mean square error (Root Mean Square Error) and other parameters of the machine learning model prediction results for testing. The specific parameter selection depends on the specific application scenario.

[0057] ② Classification task: Use the precision and recall of the machine learning model to retrieve the results. You can also use accuracy, F1 value, receiver operating characteristic (ROC) curve, mean average precision, etc. for comprehensive evaluation.

[0058] ③ Reinforcement learning tasks: Different from traditional machine learning algorithms, reinforcement learning is difficult to evaluate reproducible results due to the diversity of output space. Therefore, in the embodiment, evaluation indicators such as goal-oriented task completion rate, cumulative rewards, average rewards, and strategy quality can be used.

[0059] ④Speech recognition tasks: word error rate, sentence error rate, etc.

[0060] ⑤Target detection task: average accuracy, etc.

[0061] The algorithm capability evaluation indicators of the visual intelligent algorithm in the embodiment of the present disclosure are shown in Table 1 below:

[0062] Table 1 Algorithm capability evaluation indicators

[0063]

[0064]

[0065]

[0066]

[0067]

[0068]

[0069] The target test items corresponding to the algorithm capability evaluation indicators are used to calculate the indicator values ​​according to the calculation method of the corresponding indicators.

[0070] 2) Code implementation correctness indicators

[0071] Algorithm correctness indicators also include code implementation correctness indicators, which are used to evaluate the correctness of code implementation functions. They may include but are not limited to code standardization indicators and code vulnerability indicators, as follows:

[0072] ① Code standardization indicators: code declaration and definition, boundary condition processing, pointer use, branch control, jump control, operation processing, function call, statement use, type conversion, initialization, code logic, comparison judgment and variable use, code comments and readability, etc. The target test items corresponding to the code standardization indicators are used to test whether the above indicators meet the programming requirements in the relevant standards or specifications;

[0073] ② Code vulnerability indicators: The target test items corresponding to the code vulnerability indicators are used to test whether there are vulnerabilities in the code, such as stack overflow vulnerabilities, stack overflow vulnerabilities, integer overflows, array out of bounds, buffer overflows, etc.

[0074] (2) Algorithm convergence index

[0075] The algorithm convergence index is used to evaluate whether an algorithm can gradually approach or reach the desired goal or optimal solution during the iteration process. Convergence is used to define and evaluate whether the training process of the machine learning model can quickly converge to achieve the expected performance, and should include two parts: loss function convergence and parameter space convergence. Due to its own characteristics, the reinforcement learning algorithm is used as a special case in the application embodiment to supplement the agent convergence index of the reinforcement learning algorithm. Algorithm convergence indicators include loss function, parameter space and agent convergence indicators.

[0076] 1) Loss Function

[0077] The loss function is an indicator that measures the difference between the predicted value and the true value of the intelligent model during the training process. Loss function convergence refers to the process in which the model gradually reduces the loss function by adjusting parameters during the training process until a stable state is reached. The loss function convergence evaluation includes the following:

[0078] ① Stability of algorithm convergence: whether the algorithm can reach a convergent state under the condition of a given training data set.

[0079] ②Algorithm convergence time: Under a given hardware and software environment, the time required for the algorithm to operate on given data and achieve convergence.

[0080] ③Algorithm convergence value: Under the condition of a given training data set, the numerical value of the convergence value when the algorithm reaches the convergence state.

[0081] 2) Parameter Space

[0082] On the basis of the convergence of the loss function, evaluate whether the hyperspace where the parameters of the visual intelligent algorithm are located is converged. Different parameter spaces may lead to the same loss convergence value. Under a given experimental environment, calculate the norm of the model parameter space under two adjacent training batches, draw the difference curve, and judge whether the parameter space is converged.

[0083] 3) The agent convergence indicators include reward convergence value and reward convergence time.

[0084] Unlike loss function convergence, reinforcement learning algorithms do not use loss function minimization as the convergence goal, but instead continuously evaluate the agent's behavior and provide reward values. Reinforcement learning is the process of continuously optimizing the agent itself through the rewards it receives during training. Agent convergence evaluation should include the following:

[0085] ① Reward convergence value: Test the reward value that the reinforcement learning algorithm can obtain after completing the task after convergence. This reward value needs to be compared with the expected reward value. If it is too high, it means that the intelligent agent may have "cheated rewards".

[0086] ② Reward convergence time: Test the training time required for the reward to stabilize as the time step changes during the training process of the reinforcement learning algorithm. The longer the convergence time, the worse the performance of the current reinforcement learning algorithm.

[0087] (3) Algorithm robustness index

[0088] Robustness refers to the ability of a system or algorithm to remain stable in the face of uncertainty or abnormal situations. A robust system or algorithm can adapt to changes in external conditions, including changes in input, interference from noise, missing data, erroneous data, etc. This ability can ensure the reliability and stability of the system or algorithm.

[0089] Artificial intelligence algorithms are trained and predicted based on sample data. The model enhances the robustness of the algorithm model by learning features that are not easily disturbed in natural scenes or reducing the degree of dependence on features that are easily disturbed. For example, in the scene of image recognition, robustness can help the algorithm correctly classify different objects, even if these objects may have changes in light, angles, or occlusions. Another example is in the field of natural language processing, where robustness can help the algorithm adapt to abnormal situations such as language variants, colloquial language, or spelling errors.

[0090] The algorithm robustness indicators of artificial intelligence models should include but are not limited to algorithm robustness indicators including data enhancement indicators, anti-adversarial indicators and environmental data impact indicators, as follows:

[0091] 1) Data enhancement indicators:

[0092] The target test items corresponding to the data enhancement indicators are used to support the algorithm model to learn relevant features from the data and improve the robustness of the algorithm by simulating various situations that may occur in natural or malicious scenes. For example, training data can be generated by translation and rotation, adding natural noise, and using data style transfer to generate scene data that is difficult to collect, lossy data, and interference data to improve the robustness of the model.

[0093] 2) Anti-confrontation index

[0094] The target test items corresponding to the anti-adversarial index include adversarial sample generation algorithms, which can be divided into white-box and black-box methods according to the generation method. The target test items are used to generate adversarial test inputs through white-box and black-box adversarial sample algorithms, and then the initial test inputs and adversarial test inputs are given to the neural network for testing, and the robustness of the neural network is evaluated based on the test results.

[0095] Alternatively, the target test items corresponding to the anti-adversarial index use noise pollution to generate noise data to test the machine learning model and system, and determine whether it can operate normally and return the calculation results that meet the agreed format. In this way, the machine learning model and system can be evaluated for their normal operation, performance index changes, and recovery capabilities in the event of failure, and can test hazard detection, anti-attack, and anti-interference.

[0096] The anti-adversarial performance index of the visual intelligent algorithm of the embodiment of the present disclosure is shown in Table 2 below:

[0097] Table 2 Anti-antagonism index

[0098]

[0099] As shown in Table 2, the target test item corresponding to the anti-adversarial index can be to verify the model by adding noise and outliers to the verification set, and then calculate the change value of the model fit goodness of fit.

[0100] 3) Environmental data impact indicators are used to evaluate the impact of the actual operating environment on the algorithm, including interference data, data set distribution migration, and outlier data.

[0101] ① Interference data: refers to unexpected real data generated due to the complexity of the environment, which may affect the reliability of the algorithm;

[0102] ② Dataset distribution migration: Algorithms usually assume that training data samples and real data samples follow the same distribution, but in actual use of the algorithm, the data set distribution may migrate, that is, there is a difference between the distribution of the real data set and the distribution of the training data set;

[0103] ③Outlier data: refers to some extreme observation values. In a set of data, there may be a few data that are quite different from the rest of the data, also known as abnormal observation values.

[0104] The target test items of the environmental data impact indicators are used to test whether there is interference data, whether data distribution migration occurs, or whether there is outlier data.

[0105] (4) Algorithm efficiency index

[0106] Efficiency is used to define and evaluate the ability of visual intelligence algorithms to process large amounts of data and produce results within a reasonable time relative to the resources used, under specified conditions and within a specified time, to meet the requirements of practical applications, and the ability of machine learning models and systems to complete their work. Algorithm efficiency indicators include time consumption indicators, resource consumption indicators, durability indicators, etc.

[0107] 1) Time consumption indicators:

[0108] The time consumption characteristic is used to define and evaluate the computational amount or computational complexity of intelligent model training and testing under the same software and hardware environment.

[0109] The target test item corresponding to the time consumption indicator is to train the machine learning model and system multiple times with training sets of different sizes under specified experimental conditions, and to calculate the average time for training convergence under different data volumes, the average time for a single training round, and the curve of running time changing with data size. In the model training phase, the time consumption indicator includes test elements such as model convergence time and model training single round time. It can measure the time required for the model to complete reasoning on a given input. The time test method uses multiple measurements and removes the average value of the lowest and highest test data.

[0110] 2) Resource consumption indicators

[0111] Resource consumption indicators refer to the amount of storage or space computing complexity of intelligent model training. Smaller models can be loaded into memory faster and have advantages in storage and transmission.

[0112] The target test item corresponding to the resource consumption indicator is to conduct multiple system trainings on visual intelligent algorithms under specified test conditions using training sets of different sizes, and to count the maximum storage occupancy, average storage occupancy, maximum bandwidth occupancy, average bandwidth occupancy, maximum computing resource (CPU, GPU, etc.) occupancy, average computing resource occupancy, etc. of the training process under different data volumes, as well as the curves of the above indicators changing with data scale.

[0113] 3) Durability index

[0114] In some application scenarios, multiple input samples need to be processed simultaneously. Batch inference can be used to infer multiple samples at the same time to improve the efficiency of the algorithm.

[0115] The target test item corresponding to the durability index is to measure the inference time of the model under different batch sizes, so that the visual algorithm can choose the appropriate batch size to balance efficiency and accuracy. This reduces the severity and frequency of failures that occur when the visual intelligent algorithm runs continuously under specified conditions within a specified time, improves the fault tolerance of the algorithm, and the longer the continuous running time, the better the safety).

[0116] (5) Code Standardization Indicators

[0117] Code standardization indicators can evaluate the structure, naming, format, comments and consistency of the code to ensure that the code conforms to a consistent coding style and best practices. It includes specific code standardization indicators, readability and maintainability indicators, and code security audit indicators.

[0118] 1) Code standardization indicators

[0119] Code standardization indicators include: code declaration and definition, layout writing, pointer use, branch control, jump control, operation processing, function call, statement use, loop control, type conversion, initialization, comparison judgment and variable use, etc. The target test items corresponding to the code standardization indicators are used to determine whether the above indicators meet the programming requirements in the relevant standards or specifications.

[0120] 2) Readability and maintainability indicators

[0121] The readability and maintainability index can evaluate the readability and maintainability of the code. The code should be easy to understand, clearly structured, and have an appropriate level of abstraction. Reasonable modularization and function decomposition can improve the maintainability of the code.

[0122] 3) Code security audit indicators

[0123] The target test items corresponding to the code security audit indicators are used to audit and analyze the source code, and to find security risks and vulnerabilities by using analysis tools to scan the security defects in the code, and by analyzing the code specifications, business logic, business implementation methods, data flows, etc. through manual analysis methods. For example, stack overflow vulnerabilities, stack overflow vulnerabilities, integer overflows, array out-of-bounds, buffer overflows, etc.

[0124] (6) Algorithm dependency indicators

[0125] Algorithm dependency is used to define and evaluate the migration capabilities of machine learning models and systems, and to assess the impact of the software and hardware platforms that run the algorithms on the dependencies. Algorithm dependency indicators include: framework dependency indicators, operating system dependency indicators, and hardware architecture dependency indicators.

[0126] 1) Framework dependency index: Framework dependency refers to the impact of differences in different frameworks on the reliability of algorithms in terms of supported programming languages, model design, interface design, distributed performance, etc. The target test items corresponding to the framework dependency index are used to test the reliability of algorithms under different frameworks.

[0127] 2) Operating system dependency index: Operating system dependency refers to the impact of differences in operating system user operability, device independence, portability, system security, etc. on algorithm reliability. The target test items corresponding to the operating system dependency index are used to test the reliability of the algorithm under different operating systems.

[0128] 3) Hardware architecture dependency index: Hardware architecture dependency refers to the impact of different hardware architectures and their differences in computing power, processing accuracy, etc. on the reliability of the algorithm. The target test items corresponding to the hardware architecture dependency index are used to test the reliability of the algorithm under different hardware architectures.

[0129] (7) Algorithm Enhanceability Index

[0130] Visual intelligence algorithms have the potential to improve performance through learning, optimization or adaptive adjustment in different tasks and environments. The algorithm enhanceability indicators of visual intelligence algorithms include offline learning indicators and online learning indicators.

[0131] 1) Offline learning indicators

[0132] Offline learning is to collect a certain amount of data before model training and updating, and then perform offline training on this data.

[0133] The target test items corresponding to offline learning include: dividing the data into training sets and validation sets; adjusting model parameters through the training set; using the validation set to evaluate model performance; adjusting the scenario of offline data; modifying the task objectives and retraining; and evaluating the functional performance of visual intelligent algorithms in new scenarios.

[0134] 2) Online learning indicators

[0135] Online learning is different from collecting data in advance to train visual intelligent algorithms. Online learning is a dynamic learning method. Visual intelligent algorithms continuously receive new data and continuously update themselves while learning. Unlike offline learning, the target test items corresponding to online learning are used to adapt to changing scenarios, receive new samples for learning and adapt to new situations.

[0136] (8) Algorithm computing power indicators

[0137] The demand for hardware computing resources when visual intelligent algorithms are running and producing effective results. Hardware computing resources include the usage of CPU, GPU and other memory resources. Algorithm computing power indicators include model parameter quantity, computing amount, model memory usage, optimizer memory usage, gradient memory usage, training time, and training delay.

[0138] The algorithm computing power indicators of the visual intelligent algorithm in the embodiment of the present disclosure are as follows:

[0139] Table 3 Algorithm computing power indicators

[0140]

[0141]

[0142] (9) Algorithm recoverability index

[0143] Algorithm recoverability indicators are used to evaluate the time and investment required to recover algorithms and data from a disaster state to an operational state. Algorithm recoverability indicators include exception handling indicators, logging indicators, recoverability indicators, and recovery time indicators.

[0144] The target test items corresponding to the algorithm recoverability index may include at least one of the following: 1) Evaluate the time required for the algorithm and data to recover from a disaster state to an operational state; 2) Evaluate whether there are appropriate error handling mechanisms in the code, such as using exception handling, error code return, logging, etc. These mechanisms can capture and handle errors, helping the program to recover when abnormal situations occur without causing the algorithm to crash or stop running; 3) Evaluate whether appropriate recovery mechanisms are implemented in the code to return to the normal state as soon as possible after an error occurs. For example, save program status, back up data, rollback operations, etc. These mechanisms can help the program recover from errors and maintain consistency.

[0145] 1) Exception handling indicators

[0146] Exception handling can capture and handle exceptions that occur during program runtime, such as input errors, file read and write problems, network connection interruptions, etc. The target test items corresponding to the exception handling indicators are used to check whether the appropriate exception handling mechanism is used in the code.

[0147] Evaluate the accuracy and completeness of exception handling to ensure that the program can correctly handle various exceptional situations.

[0148] 2) Logging metrics

[0149] Logs can record the program's running process and status information, including error messages, warnings, exception stack traces, etc. The target test items corresponding to the logging indicators are used to evaluate whether there is an appropriate logging mechanism in the code. Detailed logging provides a basis for recovery.

[0150] The log record indicators of the visual intelligent algorithm of the embodiment of the present disclosure are shown in Table 4 below:

[0151] Table 4 Logging indicators

[0152]

[0153]

[0154]

[0155] 3) Resilience index

[0156] Recoverability refers to the degree of effectiveness of recovery from algorithm failure to repair after a security incident occurs. The recoverability indicator can be the proportion of algorithms that recover to full functionality by simulating security incidents under specified time and conditions.

[0157] The target test items corresponding to the recoverability index include: first, make extreme or abnormal interference to the data input, software and hardware operating environment, etc. of the intelligent model. Interference methods include interference with operating resource restrictions, such as forced restrictions on memory usage, reducing the operating frequency of the central processing unit, limiting the number of processor cores, etc.; hardware failure interference, such as forced failure of data acquisition equipment, etc.; for machine learning models that need to save status, use forced deletion of status files and other methods to interfere. Then test whether the machine learning model and system can prompt or internally digest errors that occur at the system level, and the ability of the machine learning model and system to operate normally, so as to test the avoidance of failure of the model system, the resistance to misoperation, and the harmfulness of misoperation.

[0158] Exemplarily, the recoverability index is shown in Table 5 below.

[0159] Table 5 Recoverability index

[0160]

[0161]

[0162] 4) Recovery time indicator

[0163] Recovery time is the average time between the moment the algorithm fails and business is suspended after a security incident occurs and the moment the algorithm recovers to support the operation of the intelligent system.

[0164] The target test item corresponding to the recovery time indicator is to simulate security incidents under specified time and conditions, and to statistically calculate the average, maximum, mode and median of the recovery time of the algorithm. The average time required to repair the software system and data, the average time to restart, the correctness of recovery, the average failure time, etc. under the software and hardware error environment are tested by using software fault and hardware fault detection machine learning models and systems, so as to test the average recovery time, ease of restart, ease of recovery, and effectiveness of recovery.

[0165] Exemplarily, the recovery time index is shown in Table 6 below.

[0166] Table 6 Recovery time indicators

[0167]

[0168]

[0169] In one embodiment, the present disclosure provides an electronic device, which may be a terminal, and its internal structure diagram may be as follows: Figure 5As shown. The electronic device includes a processor, a memory, a communication interface, a display screen and an input device connected by a system bus. Among them, the processor of the electronic device is used to provide computing and control capabilities. The memory of the electronic device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The communication interface of the electronic device is used to communicate with an external terminal in a wired or wireless manner, and the wireless manner can be implemented through WIFI, an operator network, near field communication (NFC) or other technologies. When the computer program is executed by the processor, a method for detecting a jam is implemented. The display screen of the electronic device can be a liquid crystal display screen or an electronic ink display screen, and the input device of the electronic device can be a touch layer covered on the display screen, or a button, a trackball or a touchpad provided on the housing of the electronic device, or an external keyboard, touchpad or mouse, etc.

[0170] Those skilled in the art will understand that Figure 5 The structure shown in the figure is merely a block diagram of a partial structure related to the scheme of the present disclosure, and does not constitute a limitation on the electronic device to which the scheme of the present disclosure is applied. The specific electronic device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0171] The embodiments of the present disclosure provide a computer program product, which stores a computer program. When the computer program is executed by a processor, each process of the performance evaluation method of the visual algorithm in the above method embodiment is implemented, and the same technical effect can be achieved. To avoid repetition, it will not be described here.

[0172] Those skilled in the art will appreciate that the embodiments of the present disclosure may be provided as methods, systems, or computer program products. Therefore, the present disclosure may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Furthermore, the present disclosure may take the form of a computer program product implemented on one or more computer-usable storage media containing computer-usable program code.

[0173] In several embodiments provided by the present disclosure, it should be understood that the disclosed method can also be implemented in other ways. In this regard, each box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and a part of a module, a program segment or a code includes one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a sequence different from that marked in the accompanying drawings. For example, two continuous boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flow chart, and the combination of the boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs the specified function or action, or can be implemented with a combination of dedicated hardware and computer instructions.

[0174] In the present disclosure, the processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0175] In the present disclosure, memory may include non-permanent memory in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. Memory is an example of a computer-readable medium.

[0176] It should be noted that, in this article, relational terms such as "first" and "second" are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0177] The above are only specific embodiments of the present disclosure, so that those skilled in the art can understand or implement the present disclosure. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein can be implemented in other embodiments without departing from the spirit or scope of the present disclosure. Therefore, the present disclosure will not be limited to these embodiments herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A performance evaluation method for a visual algorithm, characterized in that: include: Obtain the test requirements of the visual algorithm to be tested; Determine the target test item corresponding to the test requirement according to the life cycle stage of the visual algorithm to be tested, and generate a target test case according to the test input of the target test item; Execute the test case to obtain a performance evaluation result of the visual algorithm to be tested.

2. The method according to claim 1, characterized in that The life cycle stage of the visual algorithm to be tested includes any one of the following: problem definition stage, data collection stage, feature engineering stage, model training stage, and deployment and application stage; At different life cycle stages, the target test items corresponding to the test requirements are different.

3. The method according to claim 2, characterized in that The life cycle stage to which the visual algorithm to be tested belongs is the problem definition stage; The target test item indicates that the test requirement is tested based on the algorithm design requirement table.

4. The method according to claim 2, characterized in that The life cycle stage of the visual algorithm to be tested is the data collection stage, or the feature engineering stage; The target test item indicates a data set collected based on the algorithm design requirements, and tests the test data set corresponding to the test requirements; or, The target test item indicates a feature constructed based on the algorithm design requirement, and tests the test feature corresponding to the test requirement.

5. The method according to claim 2, characterized in that: The life cycle stage of the visual algorithm to be tested is the model training stage, or the deployment and application stage; The step of determining a target test item corresponding to the test requirement according to the life cycle stage to which the visual algorithm to be tested belongs, and generating a target test case according to the test input of the target test item includes: According to the model training stage or the deployment and application stage, selecting an evaluation indicator corresponding to the test requirement; Determine the target test item corresponding to the evaluation indicator, and generate the target test case according to the test input of the target test item.

6. The method according to claim 5, characterized in that The evaluation indicators corresponding to the test requirements include at least one of the following: Algorithm correctness index, algorithm convergence index, algorithm robustness index, algorithm efficiency index, code standardization index, algorithm dependency index, algorithm enhanceability index, algorithm computing power index and algorithm recoverability index.

7. The method according to claim 6, characterized in that The algorithm correctness index includes at least one of an algorithm capability evaluation index and a code implementation correctness index; different types of tasks correspond to different algorithm capability evaluation indicators, and the code implementation correctness index is used to evaluate the correctness of the code implementation function; The algorithm convergence index includes at least one of a loss function, a parameter space, and an agent convergence index, wherein the agent convergence index includes a reward convergence value and a reward convergence time; The algorithm robustness index includes at least one of a data enhancement index, an anti-adversarial index, and an environmental data impact index; The algorithm efficiency index includes at least one of a time consumption index, a resource consumption index, and a durability index; The code standardization index includes at least one of a readability and maintainability index and a code security audit index; The algorithm dependency index includes at least one of a framework dependency index, an operating system dependency index, and a hardware architecture dependency index; The algorithm enhanceability index includes offline learning index and online learning index; The algorithm computing power index includes at least one of the following: model parameter quantity, model calculation quantity, model memory occupancy, optimizer memory occupancy, gradient memory occupancy, training time, and training delay; The algorithm recoverability index includes at least one of an exception handling index, a log recording index, a recoverability index, and a recovery time index.

8. The method according to claim 7, characterized in that The algorithm capability evaluation index includes at least one of the following: Precision, cross precision, average precision, cross average precision, average precision, mean cross average precision, recall, cross recall, F1 value, cross F1 value, accuracy, cross accuracy, error rate, intersection over union ratio, receiver operating characteristic curve, number of target identity matching errors, number of target identity repetitions, tracking stability rate, tracking accuracy, mean square error, normalized mean square error, task completion rate, cumulative reward, average reward, policy quality, reward variance.

9. An electronic device, characterized in that: include: A processor, a memory, and a computer program stored in the memory and executable on the processor, wherein the computer program, when executed by the processor, implements the performance evaluation method of a visual algorithm as claimed in any one of claims 1 to 8.

10. A computer program product, characterized in that include: The computer program product comprises a computer program, and when the computer program is run on a computer, the computer is enabled to implement the performance evaluation method of a visual algorithm according to any one of claims 1 to 8.

Images