Evaluation method for artificial intelligence algorithm performance

Abstract

The invention discloses a method for evaluating the performance of an artificial intelligence algorithm, belongs to the technical field of image processing, and mainly relates to a method for testing and evaluating the performance of an intelligent algorithm by using a multi-dimensional evaluation system which is uniform in standard and wide in application range. According to the method, the performance of an algorithm and an algorithm model system of various specified application environments and various data source types is tested and evaluated according to multiple specified capability dimensions. And determining the weight of each evaluation index and each capability by using an analytic hierarchy process, calculating to obtain a comprehensive performance score, and dividing the grade according to a set threshold. Different from a traditional evaluation method, the method comprehensively evaluates the performance of the measured algorithm from core performance, flexibility, robustness, variability and coordination, objectively and comprehensively reflects the comprehensive performance of the measured algorithm, and solves the problem that an existing evaluation method is single in evaluation index.

Description

Technical Field

[0001] The present invention belongs to the technical field of image processing, and in particular relates to a method for evaluating the performance of an artificial intelligence algorithm. Background Art

[0002] In the field of artificial intelligence, a leading topic is the comprehensive and standardized evaluation of intelligent algorithms. The core task of this research is to measure and confirm the effectiveness and accuracy of various intelligent perception algorithms in different specific application scenarios. This evaluation method is suitable for testing the performance of intelligent perception algorithms in many application domains, including but not limited to object detection, object tracking, image registration, and speech recognition. Through such evaluation, developers can optimize the performance of the algorithm and provide the necessary data support for making more informed decisions.

[0003] At present, some common measurement indicators, such as precision and recall, are widely used in the evaluation process of many artificial intelligence algorithms. However, the applicability of these standard indicators in different application scenarios may vary. In view of this, it is necessary to conduct in-depth research and development on evaluation indicators that are more suitable for various application environments. In addition, the current evaluation methods can usually only evaluate the performance of the algorithm in a specific aspect, and lack a multi-dimensional evaluation strategy that can comprehensively reflect the performance of the algorithm from multiple angles. Summary of the invention

[0004] In view of the defects of the prior art, the present invention provides a method for evaluating the performance of an artificial intelligence algorithm, which is used to solve the problems in the prior art such as single testing dimension, high difficulty in algorithm adaptation and unintuitive results.

[0005] To achieve the above objectives and other related objectives, the present invention provides a method for evaluating the performance of an artificial intelligence algorithm, comprising: Step 1: configure the algorithm to be tested. After loading the algorithm to be tested in the form of a local file or a Docker package, set the band of the data set and transfer the storage location to the algorithm to be tested. Adjust the execution command according to the configuration requirements and operating environment of the algorithm to be tested.

[0006] Step 2, test the core performance of the algorithm under test, run the algorithm under test on the data set, obtain the results of the algorithm under test on the samples without adding noise, calculate the scores of various evaluation indicators based on the results, and then weight them to obtain the core performance score and divide the grades according to the set threshold.

[0007] Step 3, test the flexibility of the algorithm under test in different environments, by combining different interference noises to simulate the noise conditions that may occur in actual application scenarios. Use the samples with added noise for testing and calculate the flexibility score by the evaluation index and divide the level by the set threshold. Specifically, the process involves mixing multiple interference noises according to specific combination rules to create a series of composite noise samples that can effectively simulate the noise conditions encountered in the actual application environment. These noisy samples are used to test the performance of intelligent algorithms, which can more accurately evaluate the performance of the algorithm under real-world noise interference. In order to accurately quantify the adaptability and robustness of the algorithm, this method introduces a special evaluation index to calculate the flexibility score of the algorithm. This score is based on the performance of the algorithm in response to various noise challenges and can provide a reliable numerical measure of the flexibility of the algorithm.

[0008] In step 4, the adversarial attack samples generated by the adversarial attack algorithm are added to the data set described in step 1, and the algorithm under test is called again to calculate the robustness score and classify it by the set threshold. Specifically, this method details a process for testing and quantifying the robustness of an algorithm using adversarial attack samples. The process first involves using an advanced adversarial attack algorithm to generate a series of adversarial samples, which are designed to contain diverse perturbation patterns, different perturbation intensities, and perturbation ranges to simulate various interference situations that may be encountered in the real world.

[0009] These adversarial examples are then added to the dataset and provided as input to the algorithm to be evaluated. In this way, the response and output of the algorithm when dealing with adversarial examples can be systematically observed and analyzed. During this process, the performance of the algorithm is carefully recorded and the algorithm's robustness score is calculated using established evaluation metrics. This score is a key indicator of the algorithm's ability to resist adversarial attacks and can reflect the stability and reliability of the algorithm in the face of malicious perturbations.

[0010] Finally, the robustness of the algorithm can be graded according to the predetermined threshold criteria. This grading system not only helps to objectively evaluate the performance of the algorithm, but also provides a clear goal for further optimization of the algorithm. The evaluation method provided by this scheme can significantly improve the security performance of the algorithm, which is crucial for safety-critical applications, and also promotes the development of adversarial sample defense technology in the field of artificial intelligence.

[0011] Step 5, testing the variability of the algorithm under test on new samples, by downloading additional data sets of the same type, which are used to further train the algorithm under test, so as to simulate the impact of the dynamic data environment in the real world on the algorithm. In this way, the algorithm's plasticity and ability to update after receiving new information can be tested.

[0012] After the additional training is completed, the weight file loaded by the algorithm is replaced to ensure that the algorithm model contains the latest learning results. Then, the algorithm's score is calculated using the established evaluation indicators. This step is crucial for quantifying the learning effect of the algorithm. It is worth noting that the self-learning ability test uses the same evaluation logic as the basic performance test and uses the same test set to ensure consistency and fairness of the evaluation.

[0013] Step 6, testing the fusion processing capability of the algorithm under test for multimodal data, passing the data samples corresponding to the set band to the algorithm under test, and calling the multi-band collaborative interface of the algorithm under test, after obtaining the output results, calculating various performance indicators, and dividing the levels according to the set thresholds.

[0014] Step 7: Calculate the comprehensive performance by combining various capabilities. Specifically, this method uses multiple evaluation dimensions, including but not limited to computing resource consumption, fit, manual participation, sensitivity to abnormal samples, and intuitiveness. These dimensions reflect the performance of the algorithm in different aspects, such as resource efficiency, ability to adapt to specific tasks, convenience of user interaction, ability to identify abnormal situations, and comprehensibility of results.

[0015] Through the hierarchical analysis method, a weight is first set for each evaluation dimension, which reflects the relative importance of the dimension in the entire evaluation process. Then, the corresponding score is calculated based on the specific performance of the algorithm in each dimension. Finally, the score of each dimension is combined with its weight, and the comprehensive score of the algorithm is weighted and divided into grades accordingly.

[0016] Compared with the prior art, the present invention has the following beneficial effects: The present invention provides a method for evaluating the performance of an artificial intelligence algorithm. Different from traditional evaluation methods, this method comprehensively evaluates the performance of the algorithm under test from the perspective of core performance, flexibility, robustness, variability, and coordination, objectively and comprehensively reflects the comprehensive performance of the algorithm under test, and solves the problem of single evaluation indicators in existing evaluation methods. This method can effectively measure and improve the long-term performance and reliability of intelligent algorithms in practical applications. This not only helps algorithm developers optimize their models, but also provides end users with more powerful and flexible artificial intelligence solutions. The advantage of this evaluation method lies in its systematicity and comprehensiveness, which can provide a multi-angle and comprehensive performance evaluation result. It not only helps developers understand the overall performance of the algorithm, but also points out in which aspects the algorithm performs well and in which aspects there is room for improvement. In addition, through the classification of levels, this method also provides decision makers with a concise and clear performance reference, so that algorithms can be selected or optimized more effectively.

[0017] The present invention also proposes an experimental device for intelligent algorithm hierarchical evaluation, comprising: Storage unit, used to store the tested samples, tested algorithms, intermediate result files and historical evaluation results required during the test process; The calculation unit transmits the tested samples to the tested algorithm in sequence through the set test environment parameters and calculates the scores of various performance indicators and the comprehensive scores and their levels; A visualization unit to display samples of various noise combinations and the results of various tests; The control unit is used to adjust various operating commands involved in the test, including but not limited to the data set band, call instruction form and noise combination form.

[0018] The present invention also provides a computer device, including a processor, a register, a display and a memory, and a computing program executed by the processor in the register, wherein the computing program implements the steps of any of the above-mentioned methods for evaluating the performance of an artificial intelligence algorithm. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] The accompanying drawings are used to provide a further understanding of the embodiments of the present invention and constitute a part of the specification. Together with the following specific implementations, they are used to explain the embodiments of the present invention, but do not constitute a limitation on the embodiments of the present invention. In the accompanying drawings: Figure 1 A system workflow diagram provided for Example 1 of the present invention; Figure 2 Schematic diagram of the experimental setup. DETAILED DESCRIPTION

[0020] In order to make the technical solutions and advantages of the present invention more clearly understood, the present invention is described in detail below in conjunction with the accompanying drawings and specific embodiments. The described embodiments are part of the embodiments of the present invention, rather than all of the embodiments. The terms used in the following embodiments of the present application are only for the purpose of describing specific embodiments and are not intended to be limiting of the present application.

[0021] The present invention provides a method for evaluating the performance of artificial intelligence algorithms. The system consists of an information transmission component, a data processing unit, a command input device, and a display terminal. The flow chart is as follows: Figure 1 As shown, the specific steps include the following steps.

[0022] Step S10: Arrangement of experimental device.

[0023] Step S20: Set the algorithm to be tested and the data set used in the test, and check and determine the samples of the selected data set and their annotations.

[0024] Step S30, in the initialization phase of the process, the computing unit first determines the loading method of the algorithm. If the algorithm is loaded through the Docker package, the computing unit will call the test script designed for the Docker environment. This script can be seamlessly integrated with the Docker container to ensure that the algorithm runs in an isolated environment, thereby avoiding potential dependency conflicts and environmental differences. On the other hand, if the algorithm is loaded through a local model file, the computing unit will call a test script suitable for the local environment. This type of script allows the model parameters to be read directly from the file and the performance evaluation of the algorithm to be performed on the host operating system.

[0025] Step S40, this embodiment relates to a method for core performance testing, which first requires the user or system to automatically select the algorithm to be evaluated and the type of the corresponding data set. This step is a key part of the test preparation stage, which ensures that the test environment matches the expected application scenario of the algorithm. After the selection is completed, the computing unit is responsible for verifying the configuration. This is an important pre-step, the purpose of which is to ensure that all necessary parameters and settings are correctly configured and meet the requirements of the performance test. The verification process may include checking the integrity and consistency of the algorithm parameters and data sets, and ensuring that the settings of the test environment meet the predetermined standards. Once the configuration verification is correct, the computing unit immediately begins to perform the core performance test. The test at this stage is intended to evaluate the performance of the algorithm on the selected data set, including but not limited to key indicators such as the accuracy, processing speed, and resource consumption of the algorithm. The results of the performance test will provide a basis for the optimization of the algorithm and provide users with detailed information about the performance of the algorithm.

[0026] Step S50, test the flexibility of the algorithm under test, and examine the adaptability of the algorithm to key scenes by combining different interference signals. This embodiment proposes a method for evaluating the flexibility of an algorithm for a specific environment, which is applicable to intelligent algorithms in the fields of image processing and sound analysis. First, the technical solution described in this embodiment involves creating a series of composite noise samples, which test the robustness of the algorithm by simulating interference conditions that may be encountered in the real world. For image samples, the added noise includes but is not limited to occlusion, deformation, illumination changes, cloud effects, and traditional image noise. These noise types can appear alone or be added to the image in different combinations and intensities to examine the performance of the algorithm under different interferences. For sound data, this embodiment considers a variety of interference types including white noise, explosion sound, and data loss. These noises simulate various problems that may be encountered in the transmission and processing of sound signals, thereby providing an experimental basis for evaluating the stability of sound processing algorithms.

[0027] After generating the samples containing noise, these samples are used to test the algorithm under test. In this way, the performance of the algorithm in dealing with interference that may be encountered in the actual environment can be systematically evaluated. In order to quantify the flexibility of the algorithm, this embodiment introduces a special evaluation index to calculate the flexibility score of the algorithm. This score is based on the performance of the algorithm in dealing with various noise challenges, and can provide a reliable numerical measurement for the overall performance of the algorithm.

[0028] Step S60, the computing unit accesses the model architecture and parameters, and then determines the specific goal of the attack, such as causing misclassification, mislocalization, or triggering a specific response of the algorithm. The adversarial attack method is selected according to the attack goal. Optional methods include, but are not limited to: fast gradient sign method (FGSM), iterative gradient sign method, Kalini-Wagner attack, pixel attack, evolutionary algorithm, and generative adversarial network (GAN). The maximum allowable perturbation of the input data is defined according to the control unit to ensure that the generated adversarial examples are visually indistinguishable from the original image. According to the attack method, the optimization process is set to find the perturbation that maximizes the loss function while respecting the perturbation boundary. Finally, the success of the attack is measured according to indicators such as misclassification rate, detection failure, or other relevant criteria.

[0029] Step S70, obtain the current scenario and test set type, and use the HTTP GET request method to request the backend, the backend obtains the current scenario and test set type at the same time, and assigns different data sets in the storage unit according to the type of the current scenario and test set, and provides them for download. After the algorithm under test completes the self-learning training, the trained weights are transmitted to the specified location of the storage unit, and the calculation unit calculates the scores of various evaluation indicators again.

[0030] Step S80, perform a multi-band collaborative capability test, the task is to carry out a performance evaluation of a multi-modal fusion detection algorithm. It mainly covers three bands: visible light, infrared and radar. The control unit selects two or three of the bands to test the algorithm under test that supports fusion multi-modality to reflect its performance. Specifically, during the test, the control unit is responsible for selecting two or three of the bands to simulate different detection environments and task requirements. This flexibility ensures that the test can cover various actual situations that the algorithm may face, thereby comprehensively evaluating its performance. Subsequently, the algorithm under test that supports fusion multi-modality will process the selected band information. In this process, the system will record the processing results of the algorithm, including key indicators such as the accuracy of the fused information, processing speed and stability. These indicators not only reflect the performance of the algorithm when processing a single band of information, but also reflect its synergistic effect when integrating multiple bands of information. Finally, according to the test results, this embodiment provides a set of evaluation indicators to quantify the multi-band collaborative capability of the algorithm. These evaluation indicators provide a reliable numerical measurement for the comprehensive performance of the algorithm and provide a clear direction for further optimization of the algorithm.

[0031] Step S90, evaluate the comprehensive performance of the algorithm under test. First, the technical solution in this embodiment takes into account multiple evaluation dimensions, including but not limited to computing resource consumption, fit, manual participation, sensitivity to abnormal samples and intuitiveness. These indicators reflect the performance of the algorithm in different aspects, such as resource efficiency, ability to adapt to specific tasks, convenience of user interaction, ability to identify abnormal situations, and comprehensibility of results. In the application process of the hierarchical analysis method, a weight is first set for each evaluation dimension, and this weight reflects the relative importance of the dimension in the entire evaluation process. Then, according to the specific performance of the algorithm in each dimension, the corresponding score is calculated. Finally, the score of each dimension is combined with its weight, and the comprehensive score of the algorithm is calculated by weighted summation, and its level is divided accordingly.

[0032] In summary, the above are only preferred embodiments of the present invention and are not intended to limit the protection scope of the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A method for evaluating the performance of an artificial intelligence algorithm, characterized in that: The steps include: Step 1: Configure the running algorithm, including loading the algorithm to be tested, setting the data set, and modifying the execution command; Step 2: testing the core performance, using a data set without adding noise to test the algorithm under test, and converting the test results into a core performance score according to the evaluation index; Step 3 tests the flexibility of the algorithm under test for different environments, processes the data set without adding noise according to the set environmental parameters, obtains sample data of the corresponding environment, uses the environmental sample data to test the algorithm under test and obtains a flexibility score; Step 4 tests the robustness of the algorithm under test by using an adversarial attack algorithm to conduct an adversarial attack on the algorithm under test, and calculates the flexibility score of the algorithm under test after being subjected to the adversarial attack; Step 5: training the algorithm under test using an additional data set, and testing the algorithm under test using new weights after training to test the variability of the algorithm under test; Step 6: testing the multi-band coordination of the algorithm under test using visible light, infrared and radar data samples; Step 7 evaluates the various levels and overall level of the algorithm under test by comprehensively evaluating the scores of various capabilities.

2. The method for evaluating the performance of an artificial intelligence algorithm according to claim 1, characterized in that: The step 1 refers to, after loading the algorithm under test in the form of a local file or a Docker package, setting the band of the data set and transmitting the storage location to the algorithm under test, and adjusting the execution command according to the configuration requirements and operating environment of the algorithm under test.

3. The method for evaluating the performance of an artificial intelligence algorithm according to claim 1, characterized in that: The step 2 refers to calling the algorithm under test according to the configuration information in the step 1, obtaining the results of the algorithm under test on the samples without adding noise, calculating the scores of various evaluation indicators according to the results, and then weighting them to obtain the core performance score and its level.

4. The method for evaluating the performance of an artificial intelligence algorithm according to claim 1, characterized in that: The step 3 refers to examining the flexibility of the algorithm under test in a specific environment by combining different interference noises. The noise added to the image samples includes occlusion, deformation, illumination, fog and image noise. For the sound data, interference types including white noise, explosion sound and data loss are added. The samples with added noise are used for testing and the flexibility score and its level are calculated by the evaluation index.

5. The method for evaluating the performance of an artificial intelligence algorithm according to claim 1, characterized in that: The step 4 means that the adversarial attack samples generated by the adversarial attack algorithm will be added to the data set. These samples include various perturbation modes, perturbation intensities and perturbation ranges. Then, these samples are input into the algorithm to be evaluated, the output results of the algorithm are observed, and the robustness score and its level are calculated.

6. The method for evaluating the performance of an artificial intelligence algorithm according to claim 1, characterized in that: The step 5 refers to testing the learning ability of the algorithm under test on new samples. The variability test provides the function of downloading additional data sets of the same type to continue training the algorithm. After the training is completed, the weight file loaded by the algorithm under test is replaced and the score of the evaluation index is calculated. The variability test adopts the same evaluation logic as the core performance and uses the same test set.

7. The method for evaluating the performance of an artificial intelligence algorithm according to claim 1, characterized in that: The step 6 refers to passing the data samples corresponding to the set band to the algorithm under test, calling the multi-band coordination interface of the algorithm under test, and after obtaining the output result of the algorithm under test, calculating various performance indicators and obtaining the coordination score and its level.

8. The method for evaluating the performance of an artificial intelligence algorithm according to claim 1, characterized in that: The step 7 refers to using the analytic hierarchy process to calculate the weights of various capabilities based on the calculation of resource consumption, fit, manual participation, sensitivity to abnormal samples and intuitiveness, and then weighting them to obtain the final score and its level.

Images