Power transmission and transformation image algorithm evaluation system and method based on sandbox mechanism

Abstract

The invention discloses a power transmission and transformation image algorithm evaluation system and method based on a sandbox mechanism, and the system comprises an SDK management module which is used for managing SDK packets of a test algorithm submitted by a test manufacturer, specifying a unified SDK communication protocol and specification, enabling all submitted SDK packets to interact in the same frame, and enabling the SDK packets to interact in the same frame. Standardized data exchange and function calling are realized; the data set management module is used for checking the format and integrity of the test data set, ensuring the consistency and integrity of the data and unifying the data and labels of the test data set; a performance evaluation module; the sandbox mechanism module is used for constructing a standardized test environment for the performance evaluation module; and a result export module. And the performance evaluation module is used for mapping the to-be-tested algorithm processed by the SDK management module and the to-be-tested data set processed by the data set management module to the performance evaluation module configured with the test environment for algorithm test, and importing an evaluation result into the result export module.

Description

Technical Field

[0001] The present invention relates to image and video algorithm evaluation technology, and in particular to a power transmission and transformation image algorithm evaluation system and method based on a sandbox mechanism. Background Art

[0002] With the rapid development of image and video processing technology, a variety of algorithms have emerged to meet different application needs, such as image recognition, video analysis, etc. Most of the existing image and video algorithm evaluation methods rely on manually set data sets and evaluation indicators, and have the following major problems: lack of data set management, inconsistent evaluation standards, data tampering, leakage of annotation files, low degree of automation, and non-intuitive evaluation results. Therefore, there is an urgent need for a new image and video algorithm evaluation method that can overcome the above limitations and achieve a more comprehensive, safe, fair, objective, and automated evaluation of the performance of these algorithms. Summary of the invention

[0003] The purpose of the present invention is to solve the above-mentioned deficiencies in the prior art and provide a power transmission and transformation image algorithm evaluation system and method based on a sandbox mechanism. The present invention can provide a fair and safe comparison environment for developers and users of image and video processing algorithms, helping to improve the quality and credibility of the algorithms.

[0004] To achieve the purpose of the present invention, the technical solution adopted by the present invention is: a power transmission and transformation image algorithm evaluation system based on a sandbox mechanism, comprising: SDK management module, which is used to manage SDK packages of test algorithms submitted by test vendors, and stipulate unified SDK communication protocols and specifications so that all submitted SDK packages can interact within the same framework and realize standardized data exchange and function calls; The data set management module is used to check the format and integrity of the test data set, ensure the consistency and integrity of the data, and unify the data and labels of the test data set; Performance evaluation module; Sandbox mechanism module, used to build a standardized test environment for the performance evaluation module; and result export module, used to export test reports based on evaluation results; The performance evaluation module is used to map the algorithm to be tested processed by the SDK management module and the data set to be tested processed by the data set management module to the performance evaluation module configured with the test environment to perform algorithm testing, and import the evaluation results into the result export module.

[0005] It also includes a database, which is used to store the algorithms to be tested processed by the SDK management module and the data sets to be tested processed by the data set management module.

[0006] Sandbox mechanism module, specifically used for: Based on the Linux system's control group Cgroup and namespace technology, fixed CPU, memory, video memory, disk, and network limits are set to simulate the test environment of the input and output scenarios that need to be tested.

[0007] The following steps are involved: Step 1: System startup and scene selection; Step 2: Configure the performance evaluation module operating environment through the sandbox mechanism module; Step 3: Process the test data set and SDK package of the test algorithm; Step 4: Map the processed test data set and the SDK package of the test algorithm to the performance evaluation module with the configured operating environment; Step 5: Execute the SDK startup script . / run.sh in the performance evaluation module, and send the path of the test dataset to the SDK service through HTTP request to trigger the algorithm reasoning process for testing; Step 6: Monitor the performance indicators of the SDK service in real time, record the performance indicator test results into the database through the result export module, and generate a detailed performance analysis report.

[0008] Step 1 is as follows: The user starts the performance test system through the command line interface or graphical user interface; The user selects a specific test scenario based on the interactive menu provided by the performance testing system.

[0009] Step 2, specifically: Run the unshare -n command to create a new network namespace and isolate the network environment. Use os.chroot to set the root directory for running the sandbox environment program to ensure file system isolation in the test environment.

[0010] Create a Cgroup group: Create a control group through cgcreate and set limits on CPU, memory, video memory, disk, and network resources.

[0011] Create a control group through cgcreate and set limits on CPU, memory, video memory, disk, and network resources. Specifically: CPU limit: echo 1 > cpu.cores; Memory limit: echo 1024M > memory.limit_in_bytes; Video memory limit: nvidia-smi --gpu-reset -i GPU_ID --mem-percent=LIMIT; Disk limit: configure disk quota or use cgset -r blkio.weight=1000 my_cgroup; Network Limitation: Use the tc tool to limit network bandwidth.

[0012] Step 3, specifically: Import the predefined test data set into the data set management module for processing; Import the SDK package of the algorithm to be tested into the SDK management module for processing.

[0013] Step 4, specifically: Use the mount command to map the processed test data set and the SDK package of the test algorithm to the sandbox module with the configured running environment to ensure the isolation of the test data and SDK.

[0014] The following steps are also included: Step 7: After completing the performance test, the test system automatically cleans up the resources allocated to the test environment, including deleting Cgroups and releasing mount points.

[0015] The beneficial effect of the present invention is that, compared with the prior art, the present invention can provide a fair and safe comparison environment for developers and users of image and video processing algorithms, helping to improve the quality and credibility of the algorithms. Based on the sandbox mechanism, it can ensure that the algorithm evaluation is carried out in a completely independent environment. This isolation technology not only prevents the potential threat of malware, but also ensures the integrity and security of the data during the evaluation process. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 It is a block diagram of the power transmission algorithm evaluation system based on the sandbox mechanism of the present invention; Figure 2 It is a flow chart of the image and video algorithm evaluation method based on the sandbox mechanism of the present invention. DETAILED DESCRIPTION

[0017] The following is combined with Figure 1-2 The following examples are only used to more clearly illustrate the technical solution of the present invention, and cannot be used to limit the protection scope of the present application.

[0018] like Figure 1 The power transmission algorithm evaluation system based on the sandbox mechanism of the present invention includes a data set management module, an SDK management module, a sandbox mechanism module, a performance evaluation module, and a result export module.

[0019] The data set management module performs format and integrity checks on the test data set to ensure data consistency and integrity, and unifies the data and labels of the test data set. The labels of the same data set cannot be changed or deleted. When testing the performance of algorithms from different manufacturers, the data and labels of the same data set can be guaranteed to be consistent, ensuring the fairness of the test.

[0020] The SDK management module is used to manage the SDK packages of the test algorithms submitted by the test manufacturers. It supports the test manufacturers to submit their own SDK packages. The system will conduct evaluations on this basis to ensure that all algorithms run in a consistent system environment and eliminate the impact of hardware and environmental differences on performance evaluation. It also stipulates a unified SDK communication protocol and specification (QGDW 12319-2023, Inspection Specification for Intelligent Recognition Algorithm of Defect Images of Power Transmission and Transformation Equipment), ensuring that all submitted SDK packages can interact within the same framework, realize standardized data exchange and function calls, and thus ensure the consistency and comparability of the evaluation process.

[0021] Through the data set management module and SDK management module, all manufacturers can be uniformly specified to use specific data sets in the same test scenarios to ensure the fairness and comparability of the evaluation results.

[0022] The sandbox mechanism module is used to build a standardized test environment for the performance evaluation module. It uses advanced container isolation technology to build a standardized container environment to ensure that each evaluation task is executed under the same configuration of hardware resources (such as CPU, memory, and graphics card), thereby eliminating evaluation deviations caused by environmental differences. This consistency not only improves the fairness of the evaluation, but also makes the evaluation results more comparable. During the evaluation process, the system strictly restricts access to evaluation files and only opens read-only permissions to the evaluation algorithm to prevent any possible data tampering. In addition, the annotated files will be completely shielded and invisible in the evaluation environment to ensure the security of sensitive information and avoid the risk of leakage.

[0023] The performance evaluation module supports flexible adjustment of the weight of the evaluation algorithm. For example, if you value accuracy more in some scenarios, you can increase the weight of accuracy; in other scenarios, if you pay more attention to the recognition performance of a specific category (such as fire recognition), you can increase the weight of that category to achieve a more targeted evaluation. During the execution of the test task, the system will record the execution time and failure rate of the algorithm to comprehensively evaluate the performance of the algorithm.

[0024] The result export module will automatically generate a report containing detailed performance data based on the evaluation results, covering the scores of various indicators, the specific circumstances of recognition errors, and frame recognition problems. The report will be presented in the form of charts and tables to ensure that the results are intuitive and easy to understand, so that users can quickly understand the performance of each algorithm. The final test results will be presented in an intuitive way, including which data set types have recognition errors, which data frame recognition has problems, etc. The scores of each indicator will be displayed in a graphical table, so that users can easily compare the performance of algorithms from different manufacturers.

[0025] The performance evaluation module is used to map the algorithm to be tested processed by the SDK management module and the data set to be tested processed by the data set management module to the performance evaluation module with a configured test environment for algorithm testing, and import the evaluation results into the result export module.

[0026] It also includes a database for storing the algorithms to be tested processed by the SDK management module and the data sets to be tested processed by the data set management module.

[0027] The sandbox mechanism module is specifically used to set fixed CPU, memory, video memory, disk and network limits based on the Linux system's control group Cgroup and namespace technology to simulate the test environment of the input and output scenarios that need to be tested.

[0028] like Figure 2 The image and video algorithm evaluation method based on the sandbox mechanism of the present invention comprises the following steps: Step 1: System startup and scene selection; The user starts the performance test system through the command line interface or graphical user interface; The user selects a specific test scenario based on the interactive menu provided by the performance testing system.

[0029] Step 2: Configure the performance evaluation module operating environment through the sandbox mechanism module; Run the unshare -n command to create a new network namespace and isolate the network environment. Use os.chroot to set the root directory for running the sandbox environment program to ensure file system isolation in the test environment.

[0030] Create Cgroup group: Create a control group through cgcreate and set limits on CPU, memory, video memory, disk and network resources; CPU limit: echo 1 > cpu.cores; Memory limit: echo 1024M > memory.limit_in_bytes; Video memory limit: nvidia-smi --gpu-reset -i GPU_ID --mem-percent=LIMIT; Disk limit: configure disk quota or use cgset -r blkio.weight=1000 my_cgroup; Network restriction: Use the tc tool to limit the network bandwidth, such as tc qdisc add dev eth0 root handle1: htb rate 100kbps.

[0031] Step 3: Process the test data set and SDK package of the test algorithm; Import the predefined test data set into the data set management module for processing; Import the SDK package of the algorithm to be tested into the SDK management module for processing.

[0032] Step 4: Use the mount command to map the processed test data set and the SDK package of the test algorithm to the sandbox module with the configured running environment to ensure the isolation of the test data and SDK; Step 5: Execute the SDK startup script . / run.sh in the performance evaluation module, and send the path of the test dataset to the SDK service through HTTP request to trigger the algorithm reasoning process for testing; Step 6: Monitor the performance indicators of the SDK service in real time, record the performance indicator test results into the database through the result export module, and generate a detailed performance analysis report; Step 7: After completing the performance test, the test system automatically cleans up the resources allocated to the test environment, including deleting Cgroups and releasing mount points.

[0033] The final test results will be presented in an intuitive way, including which data set types have recognition errors, which data frame recognition has problems, etc. The scores of each indicator will be displayed in a graphical table, so that users can easily compare the performance of algorithms from different manufacturers.

[0034] The implementation examples of the present invention are illustrated and described in detail in conjunction with the drawings in the specification. However, those skilled in the art should understand that the above implementation examples are only preferred implementation schemes of the present invention, and the detailed description is only to help readers better understand the spirit of the present invention, but not to limit the scope of protection of the present invention. On the contrary, any improvements or modifications based on the inventive spirit of the present invention should fall within the scope of protection of the present invention.

Claims

1. A power transmission and transformation image algorithm evaluation system based on a sandbox mechanism, characterized in that: include: SDK management module, which is used to manage SDK packages of test algorithms submitted by test vendors, and stipulate unified SDK communication protocols and specifications so that all submitted SDK packages can interact within the same framework and realize standardized data exchange and function calls; The data set management module is used to check the format and integrity of the test data set, ensure the consistency and integrity of the data, and unify the data and labels of the test data set; Performance evaluation module; Sandbox mechanism module, used to build a standardized test environment for the performance evaluation module; and result export module, used to export test reports based on evaluation results; The performance evaluation module is used to map the algorithm to be tested processed by the SDK management module and the data set to be tested processed by the data set management module to the performance evaluation module configured with the test environment to perform algorithm testing, and import the evaluation results into the result export module.

2. According to the sandbox mechanism-based power transmission and transformation image algorithm evaluation system of claim 1, it is characterized in that: It also includes a database, which is used to store the algorithms to be tested processed by the SDK management module and the data sets to be tested processed by the data set management module.

3. A power transmission and transformation image algorithm evaluation system based on a sandbox mechanism according to claim 1 or 2, characterized in that: Sandbox mechanism module, specifically used for: Based on the Linux system's control group Cgroup and namespace technology, fixed CPU, memory, video memory, disk, and network limits are set to simulate the test environment of the input and output scenarios that need to be tested.

4. A method for evaluating a power transmission and transformation image algorithm using the above-mentioned power transmission and transformation image algorithm evaluation system based on the sandbox mechanism, characterized in that: The following steps are involved: Step 1: System startup and scene selection; Step 2: Configure the performance evaluation module operating environment through the sandbox mechanism module; Step 3: Process the test data set and SDK package of the test algorithm; Step 4: Map the processed test data set and the SDK package of the test algorithm to the performance evaluation module with the configured operating environment; Step 5: Execute the SDK startup script . / run.sh in the performance evaluation module, and send the path of the test dataset to the SDK service through HTTP request to trigger the algorithm reasoning process for testing; Step 6: Monitor the performance indicators of the SDK service in real time, record the performance indicator test results into the database through the result export module, and generate a detailed performance analysis report.

5. According to claim 4, a method for evaluating a power transmission and transformation image algorithm using the power transmission and transformation image algorithm evaluation system based on the sandbox mechanism is characterized in that: Step 1 is as follows: The user starts the performance test system through the command line interface or graphical user interface; The user selects a specific test scenario based on the interactive menu provided by the performance testing system.

6. According to claim 4, a method for evaluating a power transmission and transformation image algorithm using the power transmission and transformation image algorithm evaluation system based on the sandbox mechanism is characterized in that: Step 2, specifically: Run the unshare -n command to create a new network namespace and isolate the network environment. Use os.chroot to set the root directory for running the sandbox environment program to ensure file system isolation in the test environment; Create a Cgroup group: Create a control group through cgcreate and set limits on CPU, memory, video memory, disk, and network resources.

7. A method for evaluating a power transmission and transformation image algorithm using the power transmission and transformation image algorithm evaluation system based on the sandbox mechanism according to claim 6, characterized in that: Create a control group through cgcreate and set limits on CPU, memory, video memory, disk, and network resources. Specifically: CPU limit: echo 1 > cpu.cores; Memory limit: echo 1024M > memory.limit_in_bytes; Video memory limit: nvidia-smi --gpu-reset -i GPU_ID --mem-percent=LIMIT; Disk limit: configure disk quota or use cgset -r blkio.weight=1000 my_cgroup; Network Limitation: Use the tc tool to limit network bandwidth.

8. According to claim 4, a method for evaluating a power transmission and transformation image algorithm using the power transmission and transformation image algorithm evaluation system based on the sandbox mechanism is characterized in that: Step 3, specifically: Import the predefined test data set into the data set management module for processing; Import the SDK package of the algorithm to be tested into the SDK management module for processing.

9. The method for evaluating power transmission and transformation image algorithms using the power transmission and transformation image algorithm evaluation system based on the sandbox mechanism according to claim 4 is characterized in that: Step 4, specifically: Use the mount command to map the processed test data set and the SDK package of the test algorithm to the sandbox module with the configured running environment to ensure the isolation of the test data and SDK.

10. The method for evaluating a power transmission and transformation image algorithm using the power transmission and transformation image algorithm evaluation system based on the sandbox mechanism according to claim 4 is characterized in that: The following steps are also included: Step 7: After completing the performance test, the test system automatically cleans up the resources allocated to the test environment, including deleting Cgroups and releasing mount points.

Images