A method and device for evaluating artificial intelligence solutions

Abstract

The present invention relates to an evaluation method and device for artificial intelligence solutions, which can automatically evaluate business-level artificial intelligence solutions by supporting heterogeneous hardware and eliminating hardware differences, and visually present comprehensive and objective evaluation and analysis results. The method of comprehensive evaluation of business-level artificial intelligence solutions in the present invention will make the evaluation results closer to actual application scenarios, help users choose the best artificial intelligence solutions on different hardware platforms, improve overall business efficiency, and provide important guidance for the development and optimization of artificial intelligence solutions.

Description

Technical Field

[0001] The present invention belongs to the field of artificial intelligence technology, and specifically relates to a method and device for evaluating artificial intelligence solutions. Background Art

[0002] With the rapid development of artificial intelligence technology, artificial intelligence solutions have been widely used in different business fields.

[0003] In the field of artificial intelligence, solving practical problems usually requires a comprehensive artificial intelligence solution, which consists of multiple processing components, including one or more models / algorithms, and includes pre-processing, post-processing and other business processing related components. These components work together to form a complete artificial intelligence system to solve specific business problems.

[0004] The purpose of the evaluation system is to evaluate and perform performance tests on the overall business-level AI solution. This evaluation system needs to consider the interaction of multiple components to simulate real business scenarios and provide comprehensive performance indicators to reflect the overall effect of the AI ​​solution.

[0005] The design goal of the evaluation system is to ensure that the performance of artificial intelligence solutions in real business scenarios is consistent with the expected goals, while providing a means to quantify and compare the performance of different components and the overall system. Such an evaluation system can help developers and users understand the performance of their artificial intelligence solutions in actual applications in advance and guide the further optimization and improvement of artificial intelligence solutions.

[0006] The evaluation system relies on the support of heterogeneous hardware. Heterogeneous hardware mainly refers to the hardware components of different types, different architectures or different functions in the system. Common hardware categories include CPU, GPU, DSP, FPGA, etc. These hardware components can work together in the same system to meet different computing needs and tasks. Heterogeneous hardware design achieves the purpose of improving system performance by assigning tasks to suitable hardware components. For example, for graphics-intensive tasks, deep learning training and inference tasks, computing tasks can be assigned to the GPU and general computing tasks can be assigned to the CPU. In this way, the system can execute multiple tasks at the same time, improving the overall computing speed and energy efficiency.

[0007] The existing technical solution "CN114116481A: Testing method and system for artificial intelligence algorithm model based on Kubernetes system" aims to solve the problems of low efficiency, serious performance loss, affected testing process and inaccurate test results in the performance testing of traditional artificial intelligence algorithm models. However, this patent is only an evaluation system for algorithm models and cannot complete the evaluation of artificial intelligence solutions at the overall business level. It also cannot realize the visualization of evaluation results, and users cannot analyze the evaluation results more effectively and conveniently.

[0008] The existing technical solution "CN111897730A-Artificial Intelligence Algorithm Evaluation Method, System, Equipment and Medium in Urban Brain Construction" involves the evaluation of artificial intelligence algorithms in urban brain construction, but the patent does not involve the support of heterogeneous hardware. It only mentions the release of evaluation reports, but does not mention the visualization of evaluation results.

[0009] The main problems and shortcomings of existing evaluation methods are:

[0010] 1) Not oriented to the overall business-level scenario. Only a single model / algorithm is evaluated, and there is a lack of evaluation for business-level application scenarios;

[0011] 2) Lack of support for heterogeneous hardware. The performance of AI solutions for specific business scenarios on different hardware varies significantly, and the evaluation methods lack support and adaptation for different hardware;

[0012] 3) The evaluation results of artificial intelligence solutions cannot be visualized.

[0013] Therefore, a system is needed that can adapt to different hardware and evaluate the overall business-level artificial intelligence solution. Specifically, the evaluation system needs to be oriented to the overall business-level artificial intelligence solution and support multiple hardware types, including CPU, GPU and other acceleration cards, and provide clear evaluation indicators and visual presentation of results, so as to provide a comprehensive evaluation and optimization basis for improving the performance and efficiency of artificial intelligence solutions.

[0014] Specifically speaking of the heterogeneous hardware running on the evaluation system for artificial intelligence solutions, the focus of heterogeneous hardware is to support different types of hardware devices (such as CPU, GPU and other accelerator cards) to automatically run the test of artificial intelligence solutions for the overall business level, and eliminate hardware differences by adapting to different hardware. Such an evaluation system can more comprehensively evaluate the performance of artificial intelligence solutions in diverse hardware environments and provide more comprehensive and objective evaluation results. Furthermore, by supporting heterogeneous hardware, the evaluation system can conduct comparative experiments in different hardware environments, so as to better understand the adaptability and stability of the evaluated artificial intelligence solutions under different hardware conditions. This is of great significance for developers and users in choosing appropriate hardware platforms and optimizing artificial intelligence solutions.

[0015] In summary, what is needed now is an automated evaluation system for artificial intelligence solutions that can support heterogeneous hardware, eliminate hardware differences, evaluate business-level solutions, and visually present comprehensive and objective evaluation and analysis results, providing important guidance for the development and optimization of artificial intelligence solutions. Summary of the invention

[0016] The technical problem to be solved by the present invention is to provide an evaluation method and device for artificial intelligence solutions, so as to solve the problems that the existing evaluation system cannot eliminate hardware differences, cannot evaluate business-level artificial intelligence solutions, fails to output comprehensive and objective evaluation and analysis results in a visual manner, and is insufficient to provide guidance and support for the development and optimization of artificial intelligence solutions.

[0017] In order to solve the above technical problems, the technical solution adopted by the present invention is: the business scene video data is processed by the input processing module, the artificial intelligence analysis module, the result output module, the evaluation analysis module, the result integration module and the evaluation result visualization module in sequence, and then the visual evaluation result is output, which mainly goes through four steps: automatic processing of video data, automatic artificial intelligence analysis, evaluation analysis of results, and integration and visualization of evaluation results;

[0018] The specific process of the automatic processing step of the video data is as follows: taking the video data as input, automatically converting the video data into a data format supported by the artificial intelligence analysis module in the input processing module;

[0019] The specific process of the automated artificial intelligence analysis step is: the artificial intelligence analysis module automatically infers and analyzes the data processed by the input processing module, and outputs the processing results of the artificial intelligence solution to be tested on the video data;

[0020] The specific process of the result evaluation and analysis step is as follows: the evaluation and analysis module evaluates and analyzes the processing result of the previous step;

[0021] The specific process of the evaluation result integration and visualization step is: the result integration module integrates the effect analysis results, performance analysis results, and resource analysis results output by the evaluation analysis module, and transmits them to the evaluation result visualization module to output the visualized evaluation results.

[0022] Furthermore, the artificial intelligence analysis module uses a combination of algorithm model reasoning and business logic processing of the current artificial intelligence solution to be evaluated to reason and analyze the video data.

[0023] Furthermore, the evaluation and analysis module includes an effect analysis module, a performance analysis module and a resource analysis module.

[0024] Furthermore, the effect analysis module is used to analyze the output results of the result output module and evaluate the performance of the artificial intelligence solution in the current video simulation scenario.

[0025] Furthermore, the performance analysis module is used to evaluate the processing speed and resource consumption of the artificial intelligence solution, and to deeply analyze the performance within the artificial intelligence analysis module.

[0026] Furthermore, the resource analysis module is used to evaluate the resource usage of the artificial intelligence solution and analyze the resource usage within the artificial intelligence analysis module.

[0027] Furthermore, the evaluation result visualization module presents the effect analysis results in the form of pictures but not limited to pictures, outputs the performance analysis results in the form of tables / reports but not limited to tables / reports, and outputs the resource analysis results in the form of tables / reports but not limited to tables / reports.

[0028] Furthermore, a device for evaluating artificial intelligence solutions is provided, which is composed of an input processing module, an artificial intelligence analysis module, a result output module, an evaluation analysis module, a result integration module and an evaluation result visualization module. The artificial intelligence analysis module supports the architecture of heterogeneous hardware, and the selection and adaptation of the heterogeneous hardware includes the following steps:

[0029] 1) Available hardware query: The AI ​​analysis module actively queries the available hardware in the system;

[0030] 2) Hardware information query: obtain the hardware information of the system based on the available hardware in the query, including but not limited to the device model and video memory size;

[0031] 3) Predefined hardware list: The system will predefine the hardware list that the current artificial intelligence analysis module is adapted and supported by;

[0032] 4) Hardware matching: By querying the available hardware information and matching it with the predefined hardware list, confirm whether the current hardware is adapted to the accelerator card supported by the system. If the accelerator card supported by the system does not exist in the current available hardware, an error message will be prompted to the user, and other calculation methods will be provided as alternative solutions for the user;

[0033] 5) Hardware selection: The hardware judged as suitable in the previous step will be selected as the operating device of the artificial intelligence analysis module;

[0034] 6) Hardware driver adaptation: After the AI ​​analysis module’s operating device selection is completed, the corresponding driver is matched from the system’s predefined hardware drivers for automatic adaptation.

[0035] The advantages of the present invention are specifically as follows:

[0036] 1) Implement the evaluation of business-level artificial intelligence solutions. Unlike the existing technology that mainly focuses on the evaluation of a single algorithm or model, the method of comprehensive evaluation of the overall business-level artificial intelligence solution in this invention will make the evaluation results closer to the actual application scenario, help users choose the best artificial intelligence solution on different hardware platforms, and improve overall business efficiency. It is not limited to the evaluation of a single algorithm model, but is aimed at the overall solution, which is more in line with the actual use scenario.

[0037] 2) Support for multiple heterogeneous hardware types. The AI ​​solution evaluation system in the prior art is usually targeted at specific hardware types and lacks support for heterogeneous hardware. The evaluation system in the present invention supports the capabilities of multiple hardware types (CPU, GPU and other accelerator cards, etc.), allowing users to evaluate and optimize on a wider range of hardware platforms to better meet different application requirements. Support for different hardware types facilitates users to compare and select appropriate hardware for AI solution deployment based on actual conditions.

[0038] 3) Automated evaluation: Compared with the existing technology that relies on manual evaluation, the automated evaluation function in the present invention will greatly save evaluation time and labor costs, and can quickly evaluate artificial intelligence solutions on a variety of hardware platforms, thereby improving evaluation efficiency and reducing statistical errors in evaluation indicators caused by human factors.

[0039] 4) Visual presentation of evaluation results: The result visualization analysis function displays the evaluation results to users in the form of at least pictures / tables / reports. This visualization method allows users to intuitively compare the performance differences of solutions on different hardware platforms, better understand the evaluation results, and thus optimize and improve more conveniently. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Figure 1 The flowchart is a process flow diagram of the evaluation method for artificial intelligence solutions.

[0041] Figure 2 This is the processing flow chart of the artificial intelligence analysis module.

[0042] Figure 3 This is a composition diagram of the evaluation and analysis module.

[0043] Figure 4 Flowchart for heterogeneous hardware selection and adaptation for artificial intelligence analysis modules.

[0044] Explanation of the accompanying drawings: 11-input processing module, 21-artificial intelligence analysis module, 22-result output module, 31-evaluation and analysis module, 41-result integration module, 42-evaluation result visualization module, S1-automatic processing of video data, S2-automated artificial intelligence analysis, S3-evaluation and analysis of results, S4-integration and visualization of evaluation results, 311-effect analysis module, 312-performance analysis module, 313-resource analysis module. DETAILED DESCRIPTION

[0045] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the specific implementation modes of the present invention are further described below in conjunction with specific drawings.

[0046] In the following description, many specific details are set forth to facilitate a full understanding of the present invention, but the present invention may also be implemented in other ways different from those described herein, and those skilled in the art may make similar generalizations without violating the connotation of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed below.

[0047] In the power industry, there are high-risk work scenarios where operators operate high-voltage electrical equipment, so standardized clothing is required to ensure the safety of operators, such as standardized wearing of safety helmets and protective clothing. The artificial intelligence solution in this business scenario is to use the artificial intelligence model of computer vision to analyze whether the operator's clothing in the video meets the specifications. The present invention can automatically analyze and evaluate the artificial intelligence solution, and finally visually output multi-dimensional evaluation results, improve evaluation efficiency, and help users gain a deeper understanding of the performance of the artificial intelligence solution in business scenarios.

[0048] The artificial intelligence solution evaluation system for high-voltage electricity and high-risk dress code business scenarios in the power industry processes the dress video data in sequence through the input processing module 11, the artificial intelligence analysis module 21, the result output module 22, the evaluation analysis module 31, the result integration module 41 and the evaluation result visualization module 42, and then outputs the visualized evaluation results. It mainly goes through four steps: automatic processing of video data S1, automatic artificial intelligence analysis S2, evaluation analysis of results S3, and integration and visualization of evaluation results S4;

[0049] 1) Automated processing step S1 of video data: The evaluation system receives video data such as video / video stream of high-voltage electrical equipment operators wearing safety helmets and protective clothing as input. In the input processing module 11, the input video / video stream is automatically converted into an input format supported by the artificial intelligence analysis module 21, and the processed data is sent to the artificial intelligence analysis module 21.

[0050] 2) Automated artificial intelligence analysis step S2: After automatic reasoning and analysis by the artificial intelligence analysis module 21, the processing results of the video data of whether the high-voltage electrical equipment operators wear safety helmets and protective clothing in accordance with regulations are output, and the processing results are output to the evaluation and analysis module 31 via the result output module 22. The processing results also include performance information processed by the artificial intelligence solution, including runtime latency, number of frames processed per second, system access latency, and resource occupancy of the artificial intelligence solution, etc. The resource occupancy of the artificial intelligence solution includes video memory occupancy, CPU occupancy, memory occupancy, etc.

[0051] The video data is inferred and analyzed by combining the algorithm model reasoning and business logic processing of the current AI solution to be evaluated, such as Figure 2 As shown; the algorithm model reasoning refers to the ability to analyze the current operator's helmet wearing category and protective clothing wearing category; the business logic processing refers to the ability to analyze whether the operator is dressed in standard clothing, and the processing results are output to the result output module 22.

[0052] 3) Result evaluation and analysis step S3: The data output by the result output module 22 is processed by the evaluation and analysis module 31 and then output to the result integration module 41 .

[0053] The evaluation and analysis module 31 is composed of three modules: an effect analysis module 311, a performance analysis module 312 and a resource analysis module 313. Figure 3 shown.

[0054] The effect analysis module 311 is used to analyze the output results of the result output module 22 to evaluate the performance of the artificial intelligence solution in the current video simulation scenario, specifically by counting the number of frames judged as standard clothing and the number of frames judged as non-standard clothing in the current clothing evaluation video, and analyzing the analysis accuracy, recall rate and other indicators of the artificial intelligence solution for the video according to the preset GT (Ground Truth) file.

[0055] The performance analysis module 312 is used to evaluate the processing speed and resource consumption of the entire artificial intelligence solution, and to deeply analyze the performance of the artificial intelligence analysis module 21, thereby helping developers and users to find the performance bottlenecks that may exist in the current artificial intelligence solution;

[0056] The resource analysis module 313 is used to evaluate the resource usage of the entire artificial intelligence solution, including the artificial intelligence accelerator card video memory, internal memory, and other computing resources. It will also analyze the resource usage within the artificial intelligence analysis module 21 to guide the subsequent optimization of resource allocation for the artificial intelligence solution.

[0057] 4) Evaluation result integration and visualization step S4: The result integration module 41 integrates the effect analysis results, performance analysis results, and resource analysis results of the artificial intelligence solution for the power dress code recognition scenario in which high-voltage electrical equipment operators wear safety helmets and protective clothing, and further transmits the integration results to the evaluation result visualization module 42; the evaluation result visualization module 42 is used to visualize the evaluation results, present the effect analysis results of the artificial intelligence solution in the form of pictures, output the performance analysis results in the form of tables / reports, and output the resource analysis results in the form of tables / reports. Provide intuitive and comprehensive evaluation information to help users gain a deep understanding of the performance of the artificial intelligence solution in the simulation scenario, and provide valuable reference for further improvement and optimization.

[0058] The artificial intelligence analysis module 21 is the core processing component of the present invention, which is highly flexible and supports the architecture of heterogeneous hardware, such as heterogeneous GPUs, artificial intelligence acceleration cards, etc., so that the artificial intelligence analysis module 21 can adapt to different types of hardware for artificial intelligence analysis and processing, and realize automatic evaluation and analysis of input video data.

[0059] By supporting heterogeneous hardware, the artificial intelligence analysis module 21 can be deployed in different computing environments, which gives users more choices. According to their budget and needs, they can choose accelerator cards of different models or manufacturers and deploy them according to the actual hardware configuration without worrying about compatibility issues. This flexibility enables the evaluation system proposed by the present invention to provide users with a more comprehensive evaluation of artificial intelligence solutions.

[0060] The specific implementation plan of the heterogeneous hardware of the artificial intelligence analysis module 21 is as follows:

[0061] Since GPUs and AI accelerator cards of different models or manufacturers have different hardware architectures and characteristics, the AI ​​analysis module 21 needs to query available hardware before the evaluation task starts, obtain hardware information, and match it with the predefined hardware list to select and adapt the hardware device, such as Figure 4 As shown, the selection and adaptation process of the heterogeneous hardware of the artificial intelligence analysis module 21 is divided into the following six steps:

[0062] 1) Available hardware query: The artificial intelligence analysis module 21 will actively query the available hardware in the system.

[0063] 2) Hardware information query: Get the hardware information based on the available hardware in the queried system. Common hardware information includes: device model, video memory size, etc.

[0064] 3) Predefined hardware list: The system will predefine the hardware information that the current artificial intelligence analysis module 21 is adapted and supported.

[0065] 4) Hardware matching: By querying the available hardware information and matching it with the predefined hardware list, you can confirm whether the current hardware is adapted to the accelerator card supported by the system. If there is no accelerator card supported by the system in the currently available hardware, an error message will be prompted to the user, and other calculation methods will be provided as alternatives, such as using the CPU to perform artificial intelligence analysis tasks.

[0066] 5) Hardware selection: The hardware determined to be suitable in the previous step will be selected as the operating device of the artificial intelligence analysis module 21.

[0067] 6) Hardware driver adaptation. Since different hardware devices usually require different drivers to communicate with the hardware, after the hardware selection of the artificial intelligence analysis module 21 is completed, the corresponding driver will be matched from the system predefined hardware driver according to the current hardware device for automatic adaptation, thereby supporting the use of the current hardware. After completing the hardware driver adaptation, the artificial intelligence analysis module 21 can analyze and process the input data according to the hardware, thereby realizing the heterogeneous hardware function of the artificial intelligence analysis module 21.

[0068] The above is a detailed introduction to a method and device for evaluating artificial intelligence solutions provided by the present application. Specific examples are used in this article to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea; at the same time, for general technical personnel in this field, according to the idea of ​​the present application, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.

Claims

1. A method for evaluating artificial intelligence solutions, characterized in that: The video data is processed in sequence by an input processing module (11), an artificial intelligence analysis module (21), a result output module (22), an evaluation analysis module (31), a result integration module (41) and an evaluation result visualization module (42), and then a visual evaluation result is output. The process mainly includes four steps: automatic processing of video data (S1), automatic artificial intelligence analysis (S2), evaluation analysis of the results (S3), and integration and visualization of the evaluation results (S4). The specific process of the automatic processing of video data (S1) step is: taking the video data as input, and automatically converting the video data into a data format supported by the artificial intelligence analysis module (21) in the input processing module (11); The specific process of the automated artificial intelligence analysis (S2) step is as follows: the artificial intelligence analysis module (21) automatically infers and analyzes the data processed by the input processing module (11), and outputs the processing result of the artificial intelligence solution to be tested on the video data; The specific process of the result evaluation and analysis step (S3) is as follows: the evaluation and analysis module (31) evaluates and analyzes the processing result of the previous step; The specific process of the evaluation result integration and visualization (S4) step is as follows: the result integration module (41) integrates the effect analysis results, performance analysis results, and resource analysis results output by the evaluation analysis module (31), and transmits them to the evaluation result visualization module (42), and outputs the visualized evaluation results; The evaluation and analysis module (31) includes an effect analysis module (311), a performance analysis module (312) and a resource analysis module (313); The effect analysis module (311) is used to analyze the output results of the result output module (22) to evaluate the effect performance of the artificial intelligence solution in the current video simulation scene; The performance analysis module (312) is used to evaluate the processing speed and resource consumption of the artificial intelligence solution, and to deeply analyze the performance of the artificial intelligence analysis module (21); The resource analysis module (313) is used to evaluate the resource usage of the artificial intelligence solution and analyze the resource usage within the artificial intelligence analysis module (21); The artificial intelligence analysis module (21) supports the architecture of heterogeneous hardware, and the selection and adaptation of the heterogeneous hardware includes the following steps: 1) Available hardware query: The artificial intelligence analysis module (21) actively queries the available hardware in the system; 2) Hardware information query: obtain the hardware information of the system based on the available hardware in the query, including but not limited to the device model and video memory size; 3) Predefined hardware list: The system will predefine the hardware list that the current artificial intelligence analysis module (21) is adapted and supported; 4) Hardware matching: By querying the available hardware information and matching it with the predefined hardware list, confirm whether the current hardware is adapted to the accelerator card supported by the system. If the accelerator card supported by the system does not exist in the current available hardware, an error message will be prompted to the user, and other calculation methods will be provided as alternative solutions for the user; 5) Hardware selection: The hardware determined to be suitable in the previous step will be selected as the operating device of the artificial intelligence analysis module (21); 6) Hardware driver adaptation: After the artificial intelligence analysis module (21) completes the operation device selection, it automatically adapts by matching the corresponding driver from the system predefined hardware drivers.

2. The method for evaluating artificial intelligence solutions according to claim 1, characterized in that: The artificial intelligence analysis module (21) uses the algorithm model reasoning and business logic processing combination of the current artificial intelligence solution to be evaluated to reason and analyze the video data.

3. The method for evaluating artificial intelligence solutions according to claim 1, characterized in that: The evaluation result visualization module (42) presents the effect analysis results in the form of pictures but not limited to pictures, outputs the performance analysis results in the form of tables / reports but not limited to tables / reports, and outputs the resource analysis results in the form of tables / reports but not limited to tables / reports.

Images