AI framework testing method, device, equipment and storage medium

By testing the operators and models of the AI ​​framework, the problems of low efficiency and poor accuracy of domestic artificial intelligence frameworks are solved, and a comprehensive evaluation of the framework functions and performance is achieved, the accuracy and efficiency of the test are improved, and domestic artificial intelligence is supported. Development of intelligent frameworks.

CN118626402BActive Publication Date: 2024-11-01CHINA IND INTERNET RES INST +1
View PDF 5 Cites 0 Cited by

Patent Information

Application Number
CN202411112285.8
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-08-14
Publication Date
2024-11-01
Estimated Expiration
2044-08-14

AI Technical Summary

Technical Problem

The domestic artificial intelligence framework lacks standardized and automated testing tools and processes, resulting in low testing efficiency and poor accuracy, and the inability to comprehensively evaluate the model performance and correctness of the framework.

Method used

A method of testing AI framework is proposed. By performing operator tests and model tests on operators, obtaining operator capability test results and model capability test results, comprehensively evaluating the functions, performance and correctness of the AI ​​framework, and improving the accuracy of the test and Comprehensive.

Benefits of technology

A comprehensive evaluation of the AI ​​framework has been achieved, testing efficiency and accuracy have been improved, the functions and performance of the framework have been ensured, and the development and promotion of domestic artificial intelligence frameworks have been supported.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118626402B_ABST
    Figure CN118626402B_ABST
Patent Text Reader

Abstract

The present invention discloses an AI framework testing method, device, equipment and storage medium, which belongs to the field of artificial intelligence technology. According to the AI ​​framework to be tested, an operator to be tested is obtained; an operator test is performed on the operator to be tested to obtain an operator capability test result of the AI ​​framework to be tested; a model capability test is performed on the AI ​​framework to be tested to obtain a model capability test result of the AI ​​framework to be tested; and an evaluation result of the AI ​​framework to be tested is obtained according to the operator capability test result and the model capability test result. The above method realizes that through the two parts of operator testing and model testing, the function, performance and correctness of the AI ​​framework can be comprehensively evaluated, and the recognition of the actual performance level of the model can be improved, thereby providing a basis for further performance optimization, resource allocation and application deployment.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to an AI framework testing method, device, equipment and storage medium. Background Art

[0002] In today's era of rapid technological development, artificial intelligence technology is evolving rapidly and showing increasingly broad application prospects in industry and academia. In particular, as deep learning algorithms continue to mature, artificial intelligence frameworks (AI frameworks) have gradually emerged as important infrastructure to support the development of this technology. These frameworks not only provide a wealth of APIs and development kits, but also provide developers with convenient tools and resources for building neural network models.

[0003] However, in the process of developing domestic artificial intelligence frameworks, some technical problems have gradually surfaced. These problems not only hinder the comprehensive development of the framework, but also restrict its performance in applications. At present, the diversity of test tools for artificial intelligence development frameworks has become a prominent problem. These test tools are often written in different scripting languages, which leads to difficulties in management and maintenance, and also affects the efficiency of testing. At the same time, the current field lacks comprehensive, accurate, and efficient testing methods for domestic artificial intelligence development frameworks, and cannot fully verify the correctness and performance of the framework. In addition, the lack of functional and performance testing tools for operators and models also highlights the difficulty of verifying the correctness of the framework in dynamic and static graph modes and aligning the results with international mainstream frameworks.

[0004] The existence of these problems has seriously affected the development and promotion of domestic artificial intelligence frameworks. Therefore, this application proposes a set of automated artificial intelligence testing methods and processes to improve the testing efficiency and accuracy of domestic artificial intelligence frameworks. Summary of the invention

[0005] The main purpose of the present invention is to propose an AI framework testing method, device, equipment and storage medium, aiming to solve the technical problem that the current artificial intelligence framework lacks a set of standardized and automated testing tools and processes, and cannot comprehensively evaluate the performance of various models of the framework.

[0006] To achieve the above object, the present invention provides an AI framework testing method, which comprises the following steps:

[0007] According to the AI ​​framework to be tested, the operator to be tested is obtained;

[0008] Performing an operator test on the operator to be tested to obtain an operator capability test result of the AI ​​framework to be tested;

[0009] Performing a model capability test on the AI ​​framework to be tested to obtain a model capability test result of the AI ​​framework to be tested;

[0010] According to the operator capability test results and the model capability test results, an evaluation result of the AI ​​framework to be tested is obtained.

[0011] Optionally, performing an operator test on the operator to be tested to obtain an operator capability test result of the AI ​​framework to be tested includes:

[0012] Acquire standard input data and a preset test framework, wherein the preset test framework includes a functional test framework and a performance test framework;

[0013] Performing an operator function test on the operator to be tested according to the standard input data and the function test framework to obtain a function test result;

[0014] Performing an operator performance test on the operator to be tested according to the standard input data and the performance test framework to obtain a performance test result;

[0015] According to the functional test results and the performance test results, the operator capability test results are obtained.

[0016] Optionally, performing an operator test on the operator to be tested to obtain an operator capability test result of the AI ​​framework to be tested includes:

[0017] In the functional test framework, the standard input data is processed according to the operator to be tested to obtain a standard output result of the operator to be tested;

[0018] In the AI ​​framework to be tested, the standard input data is processed according to the operator to be tested to obtain a test output result of the operator to be tested;

[0019] Obtaining a correctness index of the operator to be tested according to the standard output result and the test output result;

[0020] According to the correctness index, an operator function test result of the operator to be tested is obtained.

[0021] Optionally, performing an operator performance test on the operator to be tested according to the standard input data and the performance test framework to obtain a performance test result includes:

[0022] In the performance test framework, the standard input data is processed according to the operator to be tested to obtain a standard processing time of the operator to be tested;

[0023] In the AI ​​framework to be tested, the standard input data is processed according to the operator to be tested to obtain the test processing time of the operator to be tested;

[0024] Obtaining a performance indicator of the operator to be tested according to the standard processing time and the test processing time;

[0025] According to the performance indicator, an operator performance test result of the operator to be tested is obtained.

[0026] Optionally, performing a model capability test on the AI ​​framework to be tested to obtain a model capability test result of the AI ​​framework to be tested includes:

[0027] Performing a model accuracy test on the AI ​​framework to be tested to obtain a model correctness test result;

[0028] Performing a model performance test on the AI ​​framework to be tested to obtain a model performance test result;

[0029] According to the model correctness test results and the model performance test results, the model capability test results of the AI ​​framework to be tested are obtained.

[0030] Optionally, performing a model accuracy test on the AI ​​framework to be tested to obtain a model correctness test result includes:

[0031] Acquire a test data set and preset test parameters, wherein the preset test parameters include a preset number of iterations and a model accuracy benchmark value;

[0032] The AI ​​framework to be tested is trained according to a preset number of iterations using the test data set to obtain a trained AI framework to be tested;

[0033] According to the trained AI framework to be tested and the test data set, the training accuracy on the test data set is obtained;

[0034] According to the training accuracy and the accuracy benchmark value, a model correctness test result is obtained.

[0035] Optionally, performing a model performance test on the AI ​​framework to be tested to obtain a model performance test result includes:

[0036] Obtain the performance indicators and test data sets to be tested;

[0037] Determine the performance test items according to the performance indicators to be tested;

[0038] Training the AI ​​framework to be tested to obtain a trained AI framework to be tested;

[0039] The trained AI framework to be tested is tested according to the performance test items to obtain the model performance test results.

[0040] In addition, to achieve the above-mentioned purpose, the present invention also proposes an AI framework testing device, the AI ​​framework testing device comprising:

[0041] The data processing module is used to obtain the operator to be tested according to the AI ​​framework to be tested;

[0042] A testing module, used to perform operator testing on the operator to be tested, and obtain the operator capability test result of the AI ​​framework to be tested;

[0043] The testing module is further used to perform a model capability test on the AI ​​framework to be tested, and obtain a model capability test result of the AI ​​framework to be tested;

[0044] The data processing module is also used to obtain an evaluation result of the AI ​​framework to be tested based on the operator capability test result and the model capability test result.

[0045] In addition, to achieve the above-mentioned purpose, the present invention also proposes an AI framework testing device, which includes: a memory, a processor, and an AI framework testing program stored in the memory and executable on the processor, and the AI ​​framework testing program is configured to implement the AI ​​framework testing method described above.

[0046] In addition, to achieve the above-mentioned purpose, the present invention also proposes a storage medium, on which an AI framework testing program is stored. When the AI ​​framework testing program is executed by a processor, the AI ​​framework testing method as described above is implemented.

[0047] The AI ​​framework testing method proposed in the present invention obtains the operator to be tested through the AI ​​framework to be tested; performs operator testing on the operator to be tested to obtain the operator capability test result of the AI ​​framework to be tested; performs model capability testing on the AI ​​framework to be tested to obtain the model capability test result of the AI ​​framework to be tested; and obtains the evaluation result of the AI ​​framework to be tested based on the operator capability test result and the model capability test result. The above method realizes that the function, performance and correctness of the AI ​​framework can be comprehensively evaluated through the two parts of operator testing and model testing, thereby improving the accuracy and comprehensiveness of the test and the efficiency of the test. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] Figure 1 It is a schematic diagram of the structure of an AI framework test device of a hardware operating environment involved in an embodiment of the present invention;

[0049] Figure 2 This is a flowchart of the first embodiment of the AI ​​framework testing method of the present invention;

[0050] Figure 3 This is a flow chart of the second embodiment of the AI ​​framework testing method of the present invention;

[0051] Figure 4 This is a flow chart of the third embodiment of the AI ​​framework testing method of the present invention;

[0052] Figure 5 This is a schematic diagram of the functional modules of the first embodiment of the AI ​​framework testing device of the present invention.

[0053] The realization of the purpose, functional features and advantages of the present invention will be further explained in conjunction with embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0054] It should be understood that the specific embodiments described herein are only used to explain the present invention, and are not used to limit the present invention.

[0055] Reference Figure 1 , Figure 1 The figure is a schematic diagram of the device structure of the hardware operating environment involved in the embodiment of the present invention.

[0056] like Figure 1 As shown, the AI ​​framework test device may include a processing device 1001 (such as a central processing unit, a graphics processor, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM: Read Only Memory) 1002 or a program loaded from a storage device 1003 to a random access memory (RAM: Random Access Memory) 1004. In RAM1004, various programs and data required for the operation of the AI ​​framework test device are also stored. The processing device 1001, ROM1002 and RAM1004 are connected to each other through a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Generally, the following systems can be connected to the I / O interface 1006: an input device 1007 including, for example, a touch screen, a touchpad, a keyboard, a mouse, an image sensor, a microphone, an accelerometer, a gyroscope, etc.; an output device 1008 including, for example, a liquid crystal display (LCD: Liquid Crystal Display), a speaker, a vibrator, etc.; a storage device 1003 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 1009. The communication device 1009 can allow the AI ​​framework test device to communicate with other devices wirelessly or by wire to exchange data. Although the figure shows an AI framework test device with various systems, it should be understood that it is not required to implement or have all the systems shown. More or fewer systems can be implemented or provided instead.

[0057] In particular, according to the embodiments disclosed in the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, the embodiments disclosed in the present application include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication device, or installed from a storage device 1003, or installed from a ROM 1002. When the computer program is executed by the processing device 1001, the above-mentioned functions defined in the method of the embodiment disclosed in the present application are executed.

[0058] Those skilled in the art will understand that Figure 1 The structure of the AI ​​framework test device shown in the figure does not constitute a limitation of the AI ​​framework test device, and may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently.

[0059] Based on the above hardware structure, an embodiment of the AI ​​framework testing method of the present invention is proposed.

[0060] It should be noted that the implementation of the AI ​​framework testing method of the present invention can be applied to a variety of testing environments, such as testing on a single server with 8 GPU cards and testing on dual servers with 16 GPU cards. Among them, the single-server 8-card GPU configuration allows a large number of parallel calculations to be performed simultaneously, which can be applied to test the throughput or other data indicators of the AI ​​framework when processing large-scale data sets; the dual-server 16-card GPU configuration can simulate a distributed computing environment, which is used to simulate the performance indicators of the AI ​​framework when it spans multiple processing nodes. The joint use of the test configurations of a single server with 8 GPU cards and dual servers with 16 GPU cards can help researchers gain an in-depth understanding of the performance of the AI ​​framework to be tested under different computing scales and distributed conditions. This configuration provides a comprehensive testing platform.

[0061] Reference Figure 2 , Figure 2 Schematic diagram of the first embodiment of the AI ​​framework testing method of the present invention.

[0062] In a first embodiment, the AI ​​framework testing method comprises the following steps:

[0063] Step S10: Obtain the operator to be tested according to the AI ​​framework to be tested.

[0064] It should be noted that in the fields of computer science and mathematics, an operator refers to a symbol or function that operates on one or more operands to produce a new value. An operator can be unary, binary, or multi-ary, depending on the number of operands operated by the operator itself. In the field of artificial intelligence, an operator usually refers to a specific computing operation used to build and perform various operations in neural network models. These operations include matrix multiplication, convolution operations, pooling operations, etc. They are the basic components of neural networks. By combining different operators, various types of neural networks can be constructed to solve various artificial intelligence tasks, such as image recognition, speech recognition, natural language processing, etc. In essence, an operator is a symbol or function that operates on operands. It is of great significance in both computer science and artificial intelligence, and is the basis for building complex computing models.

[0065] It is understandable that an artificial intelligence (AI) framework is a software tool designed to simplify and accelerate the development, training, and deployment process of artificial intelligence applications. These frameworks usually provide a series of APIs, tools, and libraries for implementing various artificial intelligence tasks, such as machine learning, deep learning, natural language processing, etc. AI frameworks usually contain the implementation and encapsulation of various operators, and developers can directly use these operators to build their own artificial intelligence models. For example, in frameworks such as TensorFlow or PyTorch, various commonly used operators are built-in, and developers can directly call these operators to build their own neural network models. In addition, the framework provides high-level interfaces and tools, allowing developers to more conveniently build, train, and deploy models, and operators are one of the basic components for implementing these functions within the framework. They are encapsulated and optimized by the framework, allowing developers to develop and execute artificial intelligence models more efficiently.

[0066] It should be understood that since operators are part of the framework, on the one hand, you can consult the source code of the framework to be tested to understand the operators that implement various operations. Most of the AI ​​frameworks currently in the application stage on the market provide detailed documentation, which includes a list of operators supported by the framework and how to use them. You can find the required list of operators by consulting the official documentation of the framework to be tested. On the other hand, some frameworks may have built-in operator query APIs, and use the relevant APIs built into the framework to obtain operator information. Many AI frameworks currently provide functions or classes to create and operate neural network models. You can understand the operators supported by the framework by viewing these APIs. The above two methods can obtain the operators to be tested in the framework to be tested.

[0067] Step S20: Perform an operator test on the operator to be tested to obtain an operator capability test result of the AI ​​framework to be tested.

[0068] It should be noted that operator testing includes two aspects: correctness testing and performance testing. Among them, the correctness test of the operator is a test conducted on a specific operator, which aims to verify whether the operator produces correct output under various input conditions. When conducting the correctness test of the operator, the following tests are mainly focused on: functional verification test, boundary condition test, numerical stability test and benchmark framework difference. The actual operator correctness test can include one or more of the above tests. The overall number of test items can be adjusted according to the actual test needs.

[0069] It can be understood that functional verification testing refers to testing whether the operator produces correct output as expected under various input conditions. Testing in this area requires the design of various test cases to cover various situations that the operator may encounter, including normal situations, boundary situations, and abnormal situations. By executing these test cases, it can be verified whether the operator conforms to the behavior specified in its design and documentation.

[0070] It can be understood that boundary condition testing refers to testing the performance of the operator under boundary conditions for the input data. For example, when the input is zero, maximum value, minimum value, etc., whether the operator can correctly process and produce reasonable output, which helps to verify whether the operator's behavior in extreme cases is as expected.

[0071] It can be understood that numerical stability testing refers to the need to test the numerical stability performance of operators involved in numerical calculations. In this test, it is necessary to verify whether the operator is numerically unstable when processing large or small numbers, so as to ensure the accuracy and reliability of numerical calculations.

[0072] It is understandable that benchmark framework difference testing refers to comparing the output of an operator with the calculation results of other verified frameworks (such as PyTorch, TensorFlow, etc.) to verify the correctness of the operator. By designing consistent test cases and comparing the operator output with the output of the benchmark framework, it is possible to verify whether the operator produces the correct results as expected. Combining these verification steps can comprehensively evaluate the correctness of the operator and ensure that its behavior is as expected in different situations, thereby improving the reliability and accuracy of the entire artificial intelligence system.

[0073] It should be noted that the above test methods all belong to the correctness test of the operator, while the performance test of the operator mainly tests the speed of the operator's calculation and training for the framework, memory and video memory usage, device hardware utilization and other indicators. Such indicators usually involve the following aspects: test the calculation speed of the operator on different hardware devices, such as the time spent on processing a certain size of data on the CPU and GPU, to evaluate its performance on different platforms; memory usage refers to the evaluation of the memory usage required by the operator when processing data, to ensure that the operator will not cause errors due to insufficient memory when processing large-scale data; parallel performance refers to the performance of the operator in a parallel computing environment, including multi-threading, distributed computing, etc., to evaluate its scalability and efficiency in a parallel environment; data type support: evaluate the operator's support for different data types (such as floating point numbers, integers, etc.) to determine its performance when processing different data types. In practical applications, AI framework operator performance testing can be carried out through performance benchmarking tools, performance analysis tools, etc. By comprehensively evaluating and testing the performance of operators, developers and researchers can choose appropriate operators and optimization strategies, thereby improving the training and reasoning efficiency of AI models.

[0074] It should be understood that by performing the above-mentioned multiple tests on operators, the functions and performance of the operators to be tested can be comprehensively evaluated, which can not only evaluate the quality of the operators themselves, but also provide important reference and support for evaluating the stability and performance optimization of the framework to which the operators belong.

[0075] Step S30: Perform a model capability test on the AI ​​framework to be tested to obtain a model capability test result of the AI ​​framework to be tested.

[0076] It should be noted that the model is a specific application implemented in the framework. It defines the model structure, loss function, optimizer, etc. through the interfaces and functions provided by the framework, and performs training and inference through the computing power provided by the framework. The model can be various machine learning models (such as linear regression, support vector machine, etc.) or deep learning models (such as neural networks, convolutional neural networks, etc.). A framework usually supports multiple types of models, including traditional machine learning models and deep learning models.

[0077] It is understandable that model capability testing includes two categories: model correctness testing and model performance testing. Among them, model correctness testing and operator correctness testing have similar testing purposes. Model correctness testing refers to the verification process of whether a machine learning or deep learning model produces correct output under given input conditions. This process aims to confirm that the model can accurately perform the required tasks under various circumstances, such as classification, regression, generation, etc., and can make reasonable predictions or generation of input data. On the other hand, model performance testing is mainly to obtain a number of test indicator data for each model in the framework. The test indicators include: throughput, GPU utilization, video memory occupancy, acceleration ratio, number of floating-point operations per second, model parameter scale, and total end-to-end complete training time. Performance testing includes inference performance testing, training performance testing, scalability testing, concurrency performance testing, energy consumption testing, and memory occupancy testing. Inference performance testing refers to evaluating the speed and resource occupancy of the model during the inference process, including inference time, memory occupancy, CPU / GPU utilization and other indicators.

[0078] It should be understood that the model correctness test ensures that the machine learning or deep learning model can accurately perform tasks under various input conditions, including data set verification, accuracy evaluation, and precision-recall evaluation. The model performance test aims to evaluate the speed, resource usage, stability and scalability of the model during reasoning and training, including reasoning performance, training performance, concurrency performance, energy consumption, memory usage and other indicators. The test results of these two parts can be used to obtain the test results of the overall model capability of the AI ​​framework to be tested.

[0079] Step S40: Obtaining an evaluation result of the AI ​​framework to be tested according to the operator capability test result and the model capability test result.

[0080] It should be noted that in actual operation, appropriate weights can be assigned to each type of test result based on the importance of the application scenario and performance indicators. For example, if the application scenario has high requirements for inference speed, the inference performance may be the main indicator of the evaluation framework, which can increase the weight of the model capability test results. If training is performed on a large-scale data set, the training performance may be more critical, which can increase the weight of the operator capability test results. Normalize the values ​​of different test results so that they are within the same range of magnitude, which can eliminate errors caused by different magnitudes. Use weighted averaging or other appropriate methods to synthesize the normalized test results according to the weights to obtain a comprehensive test evaluation result.

[0081] It is understandable that in addition to the above indicators, the scalability and flexibility of the framework in addition to performance can also be considered. The user experience in these aspects has a significant impact on the promotion of new AI frameworks. A good AI framework should have good scalability, be able to support models of different sizes and types, and flexible deployment methods. Therefore, these factors can also be considered when integrating the results. In addition, the maintenance and support of the framework can also be considered, including factors such as documentation, community support, and bug fixing speed. Frameworks with active community support can usually solve problems and improve performance faster, so these factors should also be considered when integrating the results. By considering the above content, the performance and applicability of the AI ​​framework to be tested can be more comprehensively evaluated, providing users with more valuable evaluation results and selection guidance.

[0082] In this embodiment, the operator to be tested is obtained through the AI ​​framework to be tested; the operator to be tested is tested to obtain the operator capability test result of the AI ​​framework to be tested; the model capability test is performed on the AI ​​framework to be tested to obtain the model capability test result of the AI ​​framework to be tested; and the evaluation result of the AI ​​framework to be tested is obtained according to the operator capability test result and the model capability test result. The above method realizes that the function, performance and correctness of the AI ​​framework can be comprehensively evaluated through the two parts of operator testing and model testing, thereby improving the accuracy and comprehensiveness of the test and the efficiency of the test.

[0083] Reference Figure 3 , Figure 3 Schematic diagram of the second embodiment of the AI ​​framework testing method of the present invention.

[0084] Based on the above first embodiment, step S20 in the AI ​​framework testing method of this embodiment includes:

[0085] Step S201: Acquire standard input data and a preset test framework, wherein the preset test framework includes a functional test framework and a performance test framework.

[0086] It should be noted that standard input data is a standard data set or standard input sample used to test and evaluate the AI ​​framework to be tested. These data usually represent the data distribution and characteristics in actual applications and are used to verify the performance and accuracy of the framework in different scenarios. The selection of standard input data should be representative and cover various situations and boundary conditions that the framework may encounter. Moreover, due to the characteristics of standard input data, standard input data can be used as input data for multiple operator tests. A preset test framework refers to a test framework designed in advance for testing and evaluating the AI ​​framework to be tested.

[0087] It is understandable that the preset test framework usually includes two parts: a functional test framework and a performance test framework: the functional test framework is used to verify whether the functions of the AI ​​framework to be tested meet expectations, including tests on operator correctness and model correctness. The functional test framework usually includes various test cases to test whether the various functions of the framework work properly. The performance test framework is used to evaluate the performance of the AI ​​framework to be tested under different conditions, including indicators such as computing speed, memory usage, and resource utilization. The performance test framework usually includes various performance test cases and evaluation indicators to quantify the performance of the framework in different scenarios.

[0088] Step S202: performing an operator function test on the operator to be tested according to the standard input data and the function test framework to obtain a function test result.

[0089] Furthermore, the operator test is performed on the operator to be tested to obtain the operator capability test result of the AI ​​framework to be tested, including: within the functional test framework, processing the standard input data according to the operator to be tested to obtain the standard output result of the operator to be tested; within the AI ​​framework to be tested, processing the standard input data according to the operator to be tested to obtain the test output result of the operator to be tested; obtaining the correctness index of the operator to be tested based on the standard output result and the test output result; and obtaining the operator function test result of the operator to be tested based on the correctness index.

[0090] It should be noted that the correctness test of the operator requires input data first. The input data here can be to use the operator to perform calculations on the preset framework to obtain the standard output results under the input configuration, and then use the above input parameters to execute the operator on the framework to be tested to obtain the corresponding test output results. The test output results are compared with the standard output results, and the relative error and absolute error are calculated to obtain the error index. These error indicators can help determine whether the operator of the framework to be tested produces output that is consistent with the expected results under different input conditions. If the error is small, it means that the implementation of the operator of the framework to be tested is more accurate; conversely, if the error is large, there may be problems in the algorithm implementation or poor performance.

[0091] It is understandable that the process and significance of the above two steps are explained through a specific case. Assuming that an AI framework for image recognition is being developed and needs to verify the correctness and performance of its operator, the convolution operator (conv2d) commonly used in image recognition is selected as the object to be tested, and the standard image dataset MNIST is decided to be used as the standard input data. The MNIST dataset contains a large number of handwritten digital images, which are grayscale images that have been standardized to 28x28 pixels. This part of the MNIST dataset is used as the standard input data, and two preset test frameworks are selected: the functional test framework and the performance test framework. The functional test framework includes a series of test cases to verify whether the convolution operator can correctly identify digital edges and features when processing MNIST images; the performance test framework focuses on the speed and resource consumption of the operator processing images. Entering the test process, first use the PyTorch framework (as a known benchmark) in the functional test framework to perform convolution operations on the MNIST image to obtain a set of standard output results. For example, a certain number "3" in the image is recognized and a set of feature maps are obtained. Then perform the same convolution operation on the AI ​​framework to be tested and record the output results. By comparing the two sets of results, the relative error and absolute error can be calculated. For example, if the error index is less than the preset threshold (such as 5%), it is considered that the convolution operator of the tested framework has achieved the expected function. Through this operation, not only the functional correctness of the convolution operator in the tested AI framework can be verified, but also its performance in actual application scenarios can be evaluated, ensuring the accuracy and efficiency of the AI ​​framework in processing image recognition tasks. This process is crucial to improving the reliability of the framework and optimizing the algorithm implementation.

[0092] It is understandable that within the functional testing framework, the operator to be tested is used to process the standard input data to obtain the standard output result of the operator to be tested under given input conditions. This step is to establish a reference standard for subsequent evaluation of the performance of the operator to be tested on the AI ​​framework to be tested. In practical applications, AI framework operator correctness testing is an important part of ensuring the accuracy and stability of AI models. The correctness of the operator can be ensured through automated testing, continuous integration, and other methods. At the same time, it is also necessary to design test cases in combination with domain knowledge and actual scenarios to cover various possible situations.

[0093] Step S203: performing an operator performance test on the operator to be tested according to the standard input data and the performance test framework to obtain a performance test result.

[0094] Furthermore, within the performance testing framework, the standard input data is processed according to the operator to be tested to obtain the standard processing time of the operator to be tested; within the AI ​​framework to be tested, the standard input data is processed according to the operator to be tested to obtain the test processing time of the operator to be tested; based on the standard processing time and the test processing time, the performance index of the operator to be tested is obtained; based on the performance index, the operator performance test result of the operator to be tested is obtained.

[0095] It can be understood that operators are tools for model operations, and the model as a whole is composed of a large number of operators that perform data processing to finally obtain the required data. Therefore, the choice of operators will directly affect the performance of the model. For example, when the model needs to perform convolution-type calculations during operation, the effects that can be achieved by the convolution-type operators vary depending on the input data. For example, two-dimensional convolution, one-dimensional convolution, deep convolution, etc. are used to calculate different types of data. Two-dimensional convolution is often used to process image data. It uses a convolution kernel (or filter) to slide on the input image to calculate the weighted sum of the local area; one-dimensional convolution is often used to process sequence data, such as time series, audio signals, etc.; each input channel of deep convolution has an independent convolution kernel, which is used to independently filter each channel in the spatial dimension to extract the deep features of the data. Therefore, adaptively setting the most suitable operator at the appropriate operation position will help improve the model's capabilities. Assume that some convolution operations are involved in the framework's data processing process. Then replace the original operator of this operation with an operator that has not been tested. By comparing the performance before and after the replacement, the actual performance of this operator can be tested.

[0096] It can be understood that, taking the above content as an example, this process requires first understanding how the convolution operation in the current framework is implemented, including its interface, parameters, and how it is called, and then ensuring that the new convolution operator has the same interface and functionality as the original operator to facilitate replacement, and then adaptively adjusting the code of the new operator to implement the required convolution operation, and finally integrating the new operator into the framework, which may involve modifying the framework's code base to replace the original operator with the new convolution operator so that subsequent testing can be carried out.

[0097] It should be noted that, first, within the performance test framework, the operator to be tested is used to process the standard input data, and the standard processing time of the operator to be tested under given input conditions is recorded. This step is to establish a benchmark for subsequent evaluation of the performance of the operator to be tested on the AI ​​framework to be tested. Next, within the AI ​​framework to be tested, the same standard input data and the operator to be tested are used for processing, and the test processing time of the operator to be tested is recorded. Based on the standard processing time and the test processing time, performance indicators such as processing speed, resource usage, etc. can be calculated to evaluate the performance of the operator to be tested under different frameworks.

[0098] It is understandable that the operator performance test results of the operator to be tested are obtained based on the calculated performance indicators. If the processing time of the operator to be tested under the AI ​​framework to be tested is similar to that under the preset framework, and the performance indicators are good, it means that the performance of the operator to be tested is good and has passed the performance test; otherwise, there may be performance bottlenecks or inefficiencies, which require further optimization.

[0099] Step S204: Obtain operator capability test results according to the function test results and the performance test results.

[0100] It can be understood that the operator capability test result is a comprehensive result of the functional test results and the performance test results. Prior to this, the functional test results and the performance test results need to be quantified separately. For example, the functional test results are expressed as the correctness index of the operator, and the performance test results are expressed as the processing time or other performance indicators. Then, these indicators are weighted and integrated to comprehensively consider the operator's performance in function and performance.

[0101] It should be noted that the above steps involve performance and capability testing of operators in the AI ​​framework. Here, a real-time case is given for illustration. It is assumed that the performance and capability testing of the matrix multiplication operator (matmul) in an AI framework for data processing and analysis is performed. A standardized large-scale data set is preferred as the standard input data, which contains randomly generated floating-point number matrices. Then, an industry-recognized AI framework (such as pytorch) is used as a benchmark in the performance test framework, and its dynamic computation graph features can flexibly handle various operations, including matrix multiplication. PyTorch is used to perform matrix multiplication operations on a large-scale standardized data set, and the processing time is recorded, assuming it is 500 milliseconds. This time reflects the efficiency of PyTorch in performing such operations, and also reflects its support for real-time operations in dynamic graph mode. Then, the same matrix multiplication operation is performed on the AI ​​framework to be tested, and the processing time is recorded, assuming it is 600 milliseconds. By comparing these two durations, we can get the data processing performance indicators of the framework (the ratio of PyTorch's 500 milliseconds to the tested framework's 600 milliseconds is 0.83, which is equivalent to the tested framework having 83% of the performance level of the reference framework). This comparison can indirectly evaluate the performance of the tested framework. Finally, when the functional test results (such as relative error and absolute error) are combined with the performance test results for comprehensive evaluation, a weight is assigned to each indicator, for example, the functional test results account for 60% and the performance test results account for 40%. If the comprehensive operator capability score is higher than the preset threshold (such as 80 points), the operator is considered to have met the requirements in both function and performance.

[0102] It is understandable that the above process not only evaluates the performance of the operators in the AI ​​framework to be tested, but also considers the reliability and efficiency of the operators in actual applications. This comprehensive evaluation method provides important feedback for the development and optimization of AI frameworks, and helps to improve the characteristics of domestic AI frameworks in dynamic computing and flexibility, so as to better meet the needs of actual applications.

[0103] In this embodiment, by obtaining standard input data and a preset test framework, the preset test framework includes a functional test framework and a performance test framework; according to the standard input data and the functional test framework, the operator to be tested is subjected to an operator functional test to obtain a functional test result; according to the standard input data and the performance test framework, the operator to be tested is subjected to an operator functional test to obtain a performance test result; according to the functional test result and the performance test result, the operator capability test result is obtained. Through the two test contents of functional testing and performance testing, the performance of the operator to be tested in different aspects can be comprehensively evaluated. It helps to improve the credibility, performance and stability of the operator, and provides important guarantees for the development, testing and deployment of the AI ​​framework.

[0104] Reference Figure 4 , Figure 4 Schematic diagram of the third embodiment of the AI ​​framework testing method of the present invention.

[0105] Step S301: Obtain a data set for testing the capability of the AI ​​framework model to be tested.

[0106] It should be noted that the model capability test includes but is not limited to the model accuracy test and the model performance test. Among them, the test data categories for the model accuracy test include: standard test sets, which are widely recognized and commonly used to evaluate model performance, such as ImageNet in image recognition, MIMIC in medical image analysis, etc., and also include diversified data, which have different categories, features, conditions, etc., to test whether the model is robust to various situations. On the other hand, the test data categories for the model capability test include high-dimensional data sets, which are data sets with large feature spaces, used to test the performance of the model in high-dimensional data processing; stress test data sets, which are specially designed data sets that can be used to test the performance limit and stability of the model under extreme conditions; in addition, sparse data sets are also included. Since data may be sparse in many practical applications, this data set can be used to test the model's processing ability for sparse inputs. In actual testing, not all types of data sets mentioned above will participate in the test. Developers can independently select appropriate test data sets based on model types and test items to understand the performance of the model in different application scenarios and guide further optimization.

[0107] Step S302: Perform a model accuracy test on the AI ​​framework to be tested to obtain a model correctness test result.

[0108] It should be noted that model accuracy testing is a test of machine learning or deep learning models, which aims to evaluate the prediction or classification accuracy of the model on a given task or data set. When performing model accuracy testing, the main focus is on evaluating the performance of machine learning or deep learning models on a given task or data set. This test aims to determine the prediction accuracy of the model, that is, the degree of difference between the model's prediction results for the input data and the actual results.

[0109] Furthermore, the model accuracy test of the AI ​​framework to be tested is performed to obtain a model correctness test result, including: obtaining a test data set and preset test parameters, the preset test parameters including a preset number of iterations and a model accuracy reference value; training the AI ​​framework to be tested according to the test data set according to a preset number of iterations to obtain a trained AI framework to be tested; obtaining the training accuracy on the test data set based on the trained AI framework to be tested and the test data set; and obtaining the model correctness test result based on the training accuracy and the accuracy reference value.

[0110] It should be noted that, specifically, the model accuracy test usually includes the following steps: Data preparation: Select an appropriate data set as test data. This step divides an independent test set from the existing data, or uses an existing standard data set to ensure that the selected data set is representative and covers various situations and samples that the model may encounter. Then deploy the pre-trained model to the test environment to load model parameters, set model input and output interfaces, etc., to ensure that the deployment environment is as close as possible to the actual application scenario. Use the test data set to predict the deployed model. Provide input data to the model and obtain the model's prediction results. Compare these prediction results with the actual labels or results in the data set to determine the model's prediction accuracy. Finally, calculate various evaluation indicators based on the prediction results and actual labels. Common evaluation indicators include accuracy, precision, recall, F1 score, etc. These indicators provide a quantitative evaluation of the performance of the model in different aspects, helping to understand the advantages and limitations of each model in the framework to be tested.

[0111] It is understandable that when using a real data set to actually train this model, it is expected that in the nth training round, the model's accuracy on the training set, validation set, and test set will be less than 1% from the benchmark value to be qualified. For example, the ResNet50 model can achieve the target accuracy of 76% for the top1 classification accuracy of the 50th training round of the Image Net data set. In other words, the test data set in the above steps is the real data set in the example. The preset number of iterations means that the model will perform a preset number of complete training cycles, and each training cycle contains multiple iterations to finally reach the preset number of iterations. The model accuracy benchmark value is usually determined by previous research or practical experience. In deep learning, different models are usually trained and evaluated on specific data sets, and their indicators on the training set, validation set, and test set are recorded, such as accuracy, precision, recall, etc. By analyzing these indicators, a suitable benchmark value can be determined as a reference standard for model performance.

[0112] Step S303: Perform a model performance test on the AI ​​framework to be tested to obtain a model performance test result.

[0113] It should be noted that model performance testing includes several performance tests, such as reasoning performance testing, training performance testing, scalability testing, concurrency performance testing, energy consumption testing, memory usage testing, etc. That is, the actual performance test is related to the model indicators that need to be obtained. For example, when it is necessary to measure the reasoning speed, latency and throughput of the model, the reasoning performance test can be used to evaluate the performance of the model in the reasoning stage, that is, the speed and efficiency of predicting or classifying input data; when indicators such as training speed, convergence speed and resource utilization are required, training performance testing can be used to conduct; when it is necessary to determine the scalability of the model in processing large-scale data or under different hardware configurations, scalability testing can be selected to evaluate the performance of the model in different scale data sets or different hardware environments; when it is necessary to obtain indicators such as the processing speed of concurrent requests, resource utilization and system stability, relevant data can be obtained through concurrent performance testing.

[0114] It is understandable that since the specific model indicators are too scattered, it is difficult to have a more comprehensive ability judgment on the model with a few test items. Therefore, in the actual process, it is necessary to first determine the indicator types that the evaluation model should understand, and then determine the test items based on the indicator types. Finally, the models in the framework to be tested are tested one by one according to the test items to finally obtain the model performance test results.

[0115] Furthermore, the model performance test is performed on the AI ​​framework to be tested to obtain the model performance test result, including: obtaining the performance indicators to be tested and the test data set; determining the performance test items according to the performance indicators to be tested; training the AI ​​framework to be tested to obtain a trained AI framework to be tested; testing the trained AI framework to be tested according to the performance test items to obtain the model performance test result.

[0116] It is understandable that to test the model performance of the domestic AI framework, it is necessary to define performance indicators first. The performance indicators of this embodiment include inference speed, accuracy, memory usage and GPU utilization to comprehensively evaluate its performance in practical applications. Taking the image recognition model ResNet50 as an example, in the inference performance test, the average time required for the model to be tested to classify an image is recorded, assuming it is 20 milliseconds, and the number of images processed per second (QPS) is measured, assuming it is 15 frames / second. In the training performance test, the focus is on counting the time required for the model to train an epoch on the ImageNet dataset. In addition, it is also necessary to count the difference between the average operating temperature of the GPU and the temperature under normal operation, and the normal operating temperature is 72 degrees Celsius.

[0117] It should be understood that different test contents will also show different types of model capabilities. The scalability test evaluates the performance of the model on data sets of different sizes. For example, on a data set containing 100,000 images, the model's reasoning speed drops slightly, but still remains above 10 frames per second, which shows that the AI ​​framework under test shows good scalability and stable performance when processing large-scale data sets. Despite the increase in data volume, the model's reasoning efficiency is only slightly affected, and it can still quickly classify or recognize images, showing its efficiency and reliability in practical applications. This shows that the AI ​​framework and model design can adapt to the needs of large data volumes and provide users with predictable performance in large-scale image processing tasks. Assuming that in the concurrent performance test, 100 concurrent requests are simulated, and the model's response time only increases by 5%, it can show the model's good concurrent processing capabilities. Finally, in the energy consumption test, the power consumption of the model during reasoning is monitored. The baseline value is assumed to be 200 watts, and the equivalent increase or decrease in energy consumption is calculated based on this.

[0118] It should be understood that through these tests, performance test results at all levels of the model can be obtained. They not only reflect the efficiency and stability of the model in practical applications, but also provide researchers with a basis for further optimizing model performance.

[0119] It should be noted that model performance testing of the AI ​​framework to be tested is a systematic process that includes multiple steps and considerations. First, it is necessary to clearly define the performance indicators to be tested and prepare appropriate test data sets. Then, performance test items are determined based on these performance indicators, which may involve tests on reasoning performance, training performance, scalability, concurrency, energy consumption, and memory usage. Next, the AI ​​framework is trained to obtain a trained model. Finally, according to the determined performance test items, the trained model is tested to obtain the model performance test results. This process requires comprehensive consideration of the actual application scenarios and requirements of the model to ensure the accuracy and reliability of the test results.

[0120] It should be understood that each performance test item has its specific indicators and test methods, so when conducting performance tests, it is necessary to select an appropriate test solution based on the actual situation. For example, when evaluating inference performance, the inference speed and accuracy of the model can be considered; when evaluating training performance, the convergence speed and training efficiency of the model can be considered. At the same time, the setting of the test environment and the selection of experimental parameters also need to be considered to ensure the comparability and credibility of the test results.

[0121] Step S304: Obtain the model capability test result of the AI ​​framework to be tested according to the model correctness test result and the model performance test result.

[0122] It is understandable that the correctness test results reflect the prediction accuracy of the model on a given task or data set, while the performance test results reflect the performance of the model in different aspects, such as inference speed, training efficiency, etc. Combining these test results, we can fully understand the model capabilities of the AI ​​framework to be tested, including its strengths and weaknesses in accuracy and performance.

[0123] It is understandable that the trade-off between correctness test results and performance test results depends on the specific application scenarios and requirements. Some scenarios may focus more on the accuracy of the model, while others may focus more on the performance of the model. Therefore, when obtaining the model capability test results, it is necessary to make trade-offs and comprehensive considerations based on the actual situation to ensure the practicality and reliability of the evaluation results.

[0124] In this embodiment, the model accuracy test of the AI ​​framework to be tested is performed to obtain the model correctness test result; the model performance test of the AI ​​framework to be tested is performed to obtain the model performance test result; and the model capability test result of the AI ​​framework to be tested is obtained according to the model correctness test result and the model performance test result. Through the above steps, a systematic test process covering multiple key steps such as model accuracy test, model performance test and model capability test is realized. Such a process design helps to ensure the comprehensiveness and accuracy of the test, thereby improving the comprehensive evaluation capability of the performance of the AI ​​framework to be tested.

[0125] like Figure 5 As shown, the AI ​​framework testing device proposed in the embodiment of the present invention includes:

[0126] The data processing module 10 is used to obtain the operator to be tested according to the AI ​​framework to be tested;

[0127] A testing module 20 is used to perform operator testing on the operator to be tested to obtain the operator capability test result of the AI ​​framework to be tested;

[0128] The testing module 20 is further used to perform a model capability test on the AI ​​framework to be tested, and obtain a model capability test result of the AI ​​framework to be tested;

[0129] The data processing module 10 is also used to obtain an evaluation result of the AI ​​framework to be tested based on the operator capability test result and the model capability test result.

[0130] In one embodiment, the test module 20 is also used to obtain standard input data and a preset test framework, and the preset test framework includes a functional test framework and a performance test framework; according to the standard input data and the functional test framework, the operator to be tested is subjected to an operator functional test to obtain a functional test result; according to the standard input data and the performance test framework, the operator to be tested is subjected to an operator performance test to obtain a performance test result; according to the functional test result and the performance test result, an operator capability test result is obtained.

[0131] In one embodiment, the test module 20 is also used to process the standard input data according to the operator to be tested within the functional test framework to obtain a standard output result of the operator to be tested; to process the standard input data according to the operator to be tested within the AI ​​framework to be tested to obtain a test output result of the operator to be tested; to obtain a correctness index of the operator to be tested based on the standard output result and the test output result; and to obtain an operator function test result of the operator to be tested based on the correctness index.

[0132] In one embodiment, the test module 20 is also used to process the standard input data according to the operator to be tested within the performance test framework to obtain the standard processing time of the operator to be tested; to process the standard input data according to the operator to be tested within the AI ​​framework to be tested to obtain the test processing time of the operator to be tested; to obtain the performance index of the operator to be tested based on the standard processing time and the test processing time; and to obtain the operator performance test result of the operator to be tested based on the performance index.

[0133] In one embodiment, the testing module 20 is further used to perform a model accuracy test on the AI ​​framework to be tested to obtain a model correctness test result; perform a model performance test on the AI ​​framework to be tested to obtain a model performance test result; and obtain a model capability test result of the AI ​​framework to be tested based on the model correctness test result and the model performance test result.

[0134] In one embodiment, the test module 20 is also used to obtain a test data set and preset test parameters, wherein the preset test parameters include a preset number of iterations and a model accuracy reference value; the AI ​​framework to be tested is trained according to the test data set according to a preset number of iterations to obtain a trained AI framework to be tested; based on the trained AI framework to be tested and the test data set, the training accuracy on the test data set is obtained; based on the training accuracy and the accuracy reference value, the model correctness test result is obtained.

[0135] In one embodiment, the test module 20 is also used to obtain the performance indicators to be tested and the test data set; determine the performance test items according to the performance indicators to be tested; train the AI ​​framework to be tested to obtain the trained AI framework to be tested; test the trained AI framework to be tested according to the performance test items to obtain the model performance test results.

[0136] In addition, to achieve the above-mentioned purpose, the present invention also proposes an AI framework testing device, which includes: a memory, a processor, and an AI framework testing program stored in the memory and executable on the processor, and the AI ​​framework testing program is configured to implement the AI ​​framework testing method described above.

[0137] Since the present AI framework testing device adopts all the technical solutions of all the above-mentioned embodiments, it has at least all the beneficial effects brought by the technical solutions of the above-mentioned embodiments, which will not be described one by one here.

[0138] In addition, an embodiment of the present invention further proposes a storage medium, on which an AI framework test program is stored. When the AI ​​framework test program is executed by a processor, the AI ​​framework test method as described above is implemented.

[0139] Since the storage medium adopts all the technical solutions of all the above embodiments, it has at least all the beneficial effects brought by the technical solutions of the above embodiments, which will not be described one by one here.

[0140] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or system including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or system. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or system including the element.

[0141] The serial numbers of the above embodiments of the present invention are only for description and do not represent the advantages or disadvantages of the embodiments.

[0142] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which is stored in a computer-readable storage medium (such as ROM / RAM, magnetic disk, optical disk) as described above, and includes a number of instructions for enabling an intelligent terminal (which can be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in each embodiment of the present invention.

[0143] The above are only preferred embodiments of the present invention, and are not intended to limit the patent scope of the present invention. Any equivalent structure or equivalent process transformation made using the contents of the present invention specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present invention.

Claims

1. An AI framework testing method, characterized in that: The AI ​​framework testing method comprises: According to the AI ​​framework to be tested, the operator to be tested is obtained; Performing an operator test on the operator to be tested to obtain an operator capability test result of the AI ​​framework to be tested; The performing an operator test on the operator to be tested to obtain an operator capability test result of the AI ​​framework to be tested includes: Acquire standard input data and a preset test framework, wherein the preset test framework includes a functional test framework and a performance test framework; Performing an operator function test on the operator to be tested according to the standard input data and the function test framework to obtain a function test result; Performing an operator performance test on the operator to be tested according to the standard input data and the performance test framework to obtain a performance test result; Obtaining operator capability test results according to the functional test results and the performance test results; The obtaining of operator capability test results according to the functional test results and the performance test results includes: In the functional test framework, the standard input data is processed according to the operator to be tested to obtain a standard output result of the operator to be tested; In the AI ​​framework to be tested, the standard input data is processed according to the operator to be tested to obtain a test output result of the operator to be tested; Obtaining a correctness index of the operator to be tested according to the standard output result and the test output result; According to the correctness index, obtaining the operator function test result of the operator to be tested; Performing a model capability test on the AI ​​framework to be tested to obtain a model capability test result of the AI ​​framework to be tested; According to the operator capability test results and the model capability test results, an evaluation result of the AI ​​framework to be tested is obtained.

2. The AI ​​framework testing method according to claim 1, characterized in that: The performing of an operator performance test on the operator to be tested according to the standard input data and the performance test framework to obtain a performance test result includes: In the performance test framework, the standard input data is processed according to the operator to be tested to obtain a standard processing time of the operator to be tested; In the AI ​​framework to be tested, the standard input data is processed according to the operator to be tested to obtain the test processing time of the operator to be tested; Obtaining a performance indicator of the operator to be tested according to the standard processing time and the test processing time; According to the performance indicator, an operator performance test result of the operator to be tested is obtained.

3. The AI ​​framework testing method according to claim 1, characterized in that: The performing a model capability test on the AI ​​framework to be tested to obtain a model capability test result of the AI ​​framework to be tested includes: Performing a model accuracy test on the AI ​​framework to be tested to obtain a model correctness test result; Performing a model performance test on the AI ​​framework to be tested to obtain a model performance test result; According to the model correctness test results and the model performance test results, the model capability test results of the AI ​​framework to be tested are obtained.

4. The AI ​​framework testing method according to claim 3, characterized in that: The step of performing a model accuracy test on the AI ​​framework to be tested to obtain a model correctness test result includes: Acquire a test data set and preset test parameters, wherein the preset test parameters include a preset number of iterations and a model accuracy benchmark value; The AI ​​framework to be tested is trained according to a preset number of iterations using the test data set to obtain a trained AI framework to be tested; According to the trained AI framework to be tested and the test data set, the training accuracy on the test data set is obtained; According to the training accuracy and the accuracy benchmark value, a model correctness test result is obtained.

5. The AI ​​framework testing method according to claim 3, characterized in that: The performing of a model performance test on the AI ​​framework to be tested to obtain a model performance test result includes: Obtain the performance indicators and test data sets to be tested; Determine the performance test items according to the performance indicators to be tested; Training the AI ​​framework to be tested to obtain a trained AI framework to be tested; The trained AI framework to be tested is tested according to the performance test items to obtain the model performance test results.

6. An AI framework testing device, characterized in that: The AI ​​framework testing device comprises: The data processing module is used to obtain the operator to be tested according to the AI ​​framework to be tested; A testing module, used to perform operator testing on the operator to be tested, and obtain the operator capability test result of the AI ​​framework to be tested; The test module is further used to obtain standard input data and a preset test framework, wherein the preset test framework includes a function test framework and a performance test framework; perform an operator function test on the operator to be tested according to the standard input data and the function test framework to obtain a function test result; perform an operator performance test on the operator to be tested according to the standard input data and the performance test framework to obtain a performance test result; and obtain an operator capability test result according to the function test result and the performance test result; The data processing module is further used to process the standard input data according to the operator to be tested within the functional test framework to obtain the standard output result of the operator to be tested; process the standard input data according to the operator to be tested within the AI ​​framework to be tested to obtain the test output result of the operator to be tested; obtain the correctness index of the operator to be tested according to the standard output result and the test output result; obtain the operator function test result of the operator to be tested according to the correctness index; The testing module is further used to perform a model capability test on the AI ​​framework to be tested, and obtain a model capability test result of the AI ​​framework to be tested; The data processing module is also used to obtain an evaluation result of the AI ​​framework to be tested based on the operator capability test result and the model capability test result.

7. An AI framework testing device, characterized in that: The AI ​​framework testing device comprises: a memory, a processor, and an AI framework testing program stored in the memory and executable on the processor, wherein the AI ​​framework testing program is configured to implement the steps of the AI ​​framework testing method according to any one of claims 1 to 5.

8. A storage medium, characterized in that: The storage medium stores an AI framework test program, and when the AI ​​framework test program is executed by the processor, the steps of the AI ​​framework test method according to any one of claims 1 to 5 are implemented.

Citation Information

Patent Citations

  • Deep learning framework determination method and device, and readable storage medium

    CN112257856A

  • Method and system for evaluating artificial intelligence model

    CN116483697A

  • Test method and device of AI software stack, computer readable storage medium and terminal

    CN117520205A

  • Chip testing method and device, computer equipment and storage medium

    CN117825912A

  • Method and apparatus for testing ai chip computing performance, and non-transitory computer-readable storage medium

    US20230029359A1