Model evaluation method, device, computer equipment and storage medium

Abstract

The embodiment of the present invention discloses a model evaluation method, device, computer equipment and storage medium, including the following steps: obtaining an artificial intelligence model to be evaluated; calculating the artificial intelligence model by a preset model evaluation method to obtain evaluation parameters; judging whether the artificial intelligence model is suitable for the product to be evaluated based on the preset performance parameters of the product to be evaluated and the evaluation parameters, so as to obtain the evaluation result of the artificial intelligence model. The artificial intelligence model is calculated by a preset model evaluation method to obtain the evaluation parameters, and then the artificial intelligence model is evaluated according to the preset performance parameters and evaluation parameters of the product to be evaluated to obtain the evaluation result of the artificial intelligence model. This method obtains the evaluation result by using the performance parameters and evaluation parameters of the product to be evaluated, so as to select the preferred artificial intelligence model, facilitate resource allocation, and effectively reduce the difficulty of development.

Description

Technical Field

[0001] Embodiments of the present invention relate to the field of artificial intelligence technology, and in particular to a model evaluation method, device, computer equipment and storage medium. Background Art

[0002] With the advancement of science and technology, artificial intelligence (AI) has developed rapidly. The specific implementation of AI requires the deployment of theoretical algorithms in actual application scenarios. However, due to the different details of each scenario, such as temperature, price, power consumption, size, etc., AI chips containing different deep learning models need to be selected for different application scenarios. Therefore, it is necessary to evaluate the deep learning models in AI chips.

[0003] At present, the existing evaluation tools are basically model optimization tools developed by chip companies for their own chips. They are mainly aimed at optimizing the deployment of models to hardware after the selection is determined. This method cannot analyze the performance of the model and cannot predict the difficulty of later development. Summary of the invention

[0004] Embodiments of the present invention provide a model evaluation method, apparatus, computer equipment, and storage medium.

[0005] In order to solve the above technical problems, a technical solution adopted by an embodiment of the present invention is: to provide a model evaluation method, comprising the following steps:

[0006] Obtain the AI ​​model to be evaluated;

[0007] Calculating the artificial intelligence model using a preset model evaluation method to obtain evaluation parameters;

[0008] Whether the artificial intelligence model is suitable for the product to be evaluated is determined based on the preset performance parameters of the product to be evaluated and the evaluation parameters, so as to obtain the evaluation result of the artificial intelligence model.

[0009] Optionally, the artificial intelligence model is calculated by a preset model evaluation method to obtain evaluation parameters, including:

[0010] Obtaining structural unit information of the artificial intelligence model, wherein the structural unit information includes the percentage of the number of each type of structural units to the total number of all structural units;

[0011] Find the friendliness score of each type of structural unit to the product in a preset scoring information table;

[0012] The percentage of each type of structural unit in the artificial intelligence model and the corresponding friendliness score are weighted and calculated to obtain the friendliness of the artificial intelligence model to the hardware of the product.

[0013] Optionally, the artificial intelligence model is calculated by a preset model evaluation method to obtain evaluation parameters, including:

[0014] compressing the artificial intelligence model;

[0015] Quantize the compressed AI model according to the quantization type selected by the user;

[0016] Compare the loss accuracy of each layer of the quantized AI model with that of the AI ​​model before compression to obtain the accuracy loss of each layer;

[0017] The precision loss of each layer is averaged to obtain the precision loss of the artificial intelligence model.

[0018] Optionally, judging whether the artificial intelligence model is suitable for the product to be evaluated according to preset performance parameters of the product to be evaluated and the evaluation parameters to obtain an evaluation result of the artificial intelligence model includes:

[0019] Comparing the evaluation parameters with the performance parameters of the product to be evaluated to obtain a score of whether the artificial intelligence model is suitable for the product to be evaluated;

[0020] Determine whether the score is greater than a preset score;

[0021] When the score is greater than the preset score, it is determined that the artificial intelligence model is suitable for the product to be evaluated, and the score is used as the evaluation result.

[0022] Optionally, comparing the evaluation parameters with the performance parameters of the product to be evaluated to obtain a score of whether the artificial intelligence model is suitable for the product to be evaluated includes:

[0023] Obtaining a performance scoring information table for indicating the degree of operation of the product, wherein the performance scoring information table includes a plurality of performance parameter ranges and a fitness corresponding to each performance parameter range;

[0024] Determine the target performance parameter range that the evaluation parameter is suitable for, and search the performance score information table for the degree of fitness corresponding to the target performance parameter range;

[0025] A weighted calculation is performed on multiple fitness levels to obtain a score for whether the artificial intelligence model is suitable for the product to be evaluated.

[0026] Optionally, after determining whether the score is greater than a preset score, the method further includes:

[0027] When the score is less than or equal to the preset score, it is determined that the artificial intelligence model is not suitable for the product to be evaluated.

[0028] Optionally, the evaluation parameters also include: computing power and memory bandwidth of the artificial intelligence model. To solve the above technical problems, an embodiment of the present invention further provides a model evaluation device, including:

[0029] An acquisition module, used to acquire the artificial intelligence model to be evaluated;

[0030] A processing module, used to calculate the artificial intelligence model through a preset model evaluation method to obtain evaluation parameters;

[0031] The execution module is used to determine whether the artificial intelligence model is suitable for the product to be evaluated based on the preset performance parameters of the product to be evaluated and the evaluation parameters, so as to obtain the evaluation result of the artificial intelligence model.

[0032] Optionally, the processing module includes:

[0033] A first acquisition submodule is used to acquire structural unit information of the artificial intelligence model, wherein the structural unit information includes the percentage of the number of each type of structural units to the total number of all structural units;

[0034] A first processing submodule is used to search for a friendliness score of each type of structural unit to the product in a preset scoring information table;

[0035] The first execution submodule is used to perform a weighted operation on the percentage of each type of structural unit in the artificial intelligence model and the corresponding friendliness score to obtain the friendliness of the artificial intelligence model to the hardware of the product.

[0036] Optionally, the processing module includes:

[0037] A second processing submodule, used for compressing the artificial intelligence model;

[0038] A third processing submodule, for quantizing the compressed artificial intelligence model according to a quantization type selected by a user;

[0039] A fourth processing submodule is used to compare the loss accuracy of each layer of the quantized artificial intelligence model with that of the artificial intelligence model before compression to obtain the accuracy loss of each layer;

[0040] The second execution submodule is used to perform an average operation on the precision loss of each layer to obtain the precision loss of the artificial intelligence model.

[0041] Optionally, the execution module includes:

[0042] A fifth processing submodule, configured to compare the evaluation parameters with the performance parameters of the product to be evaluated to obtain a score of the artificial intelligence model being suitable for the product to be evaluated;

[0043] A sixth processing submodule, used to determine whether the score is greater than a preset score;

[0044] The third execution submodule is used to determine that the artificial intelligence model is suitable for the product to be evaluated when the score is greater than the preset score, and use the score as the evaluation result.

[0045] Optionally, the fifth processing submodule includes:

[0046] A second acquisition submodule is used to acquire a performance score information table for indicating the degree of operation of the product, wherein the performance score information table includes a plurality of performance parameter ranges and a fitness corresponding to each performance parameter range;

[0047] A seventh processing submodule, configured to determine a target performance parameter range suitable for the evaluation parameter, and to search the performance scoring information table for a degree of fitness corresponding to the target performance parameter range;

[0048] The fourth execution submodule is used to perform weighted calculation on multiple fitness levels to obtain a score for whether the artificial intelligence model conforms to the product to be evaluated.

[0049] Optionally, it also includes:

[0050] The eighth processing submodule is used to determine that the artificial intelligence model is not suitable for the product to be evaluated when the score is less than or equal to the preset score.

[0051] Optionally, the evaluation parameters also include: computing power and memory bandwidth of the artificial intelligence model.

[0052] To solve the above technical problems, an embodiment of the present invention further provides a computer device, comprising a memory and a processor, wherein the memory stores computer-readable instructions, and when the computer-readable instructions are executed by the processor, the processor executes the steps of the above-mentioned model evaluation method.

[0053] In order to solve the above technical problems, an embodiment of the present invention further provides a storage medium storing computer-readable instructions, and when the computer-readable instructions are executed by one or more processors, the one or more processors execute the steps of the above-mentioned model evaluation method.

[0054] The beneficial effect of the embodiment of the present invention is that the artificial intelligence model is calculated by a preset model evaluation method to obtain evaluation parameters, and then the artificial intelligence model is evaluated according to the preset performance parameters and evaluation parameters of the product to be evaluated to obtain the evaluation result of the artificial intelligence model. The method obtains the evaluation result by using the performance parameters and evaluation parameters of the product to be evaluated, and can select the preferred artificial intelligence model, which is convenient for resource allocation and effectively reduces the difficulty of development. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative work.

[0056] Figure 1 A schematic diagram of a basic flow chart of a model evaluation method according to an embodiment of the present invention;

[0057] Figure 2 A basic flow chart of a method for calculating an artificial intelligence model to obtain evaluation parameters by a preset model evaluation method is provided for an embodiment of the present invention;

[0058] Figure 3 A basic flow chart of a method for calculating an artificial intelligence model to obtain evaluation parameters by a preset model evaluation method is provided for an embodiment of the present invention;

[0059] Figure 4 A basic flow chart of a method for determining whether an artificial intelligence model is suitable for a product to be evaluated based on preset performance parameters and evaluation parameters of the product to be evaluated provided by an embodiment of the present invention;

[0060] Figure 5 A basic flow chart of a method for comparing evaluation parameters with performance parameters of a product to be evaluated to obtain a fitness score of an artificial intelligence model for the product to be evaluated provided by an embodiment of the present invention;

[0061] Figure 6 A basic structural block diagram of a model evaluation device provided by an embodiment of the present invention;

[0062] Figure 7 A basic structural block diagram of a computer device provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0063] In order to enable those skilled in the art to better understand the solutions of the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings in the embodiments of the present invention.

[0064] In some of the processes described in the specification and claims of the present invention and the above-mentioned figures, multiple operations that appear in a specific order are included, but it should be clearly understood that these operations may not be executed in the order in which they appear in this article or executed in parallel. The serial numbers of the operations, such as 101, 102, etc., are only used to distinguish different operations, and the serial numbers themselves do not represent any execution order. In addition, these processes may include more or fewer operations, and these operations may be executed in sequence or in parallel. It should be noted that the descriptions of "first", "second", etc. in this article are used to distinguish different messages, devices, modules, etc., do not represent the order of precedence, and do not limit the "first" and "second" to be different types.

[0065] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.

[0066] Example

[0067] It will be understood by those skilled in the art that the "terminal" and "terminal device" used herein include both devices with wireless signal receivers, which are devices with only wireless signal receivers without transmission capabilities, and devices with receiving and transmitting hardware, which are devices with receiving and transmitting hardware capable of performing two-way communications on a two-way communication link. Such devices may include: cellular or other communication devices, which have a single-line display or a multi-line display or a cellular or other communication device without a multi-line display; PCS (Personal Communications Service), which may combine voice, data processing, fax and / or data communication capabilities; PDA (Personal Digital Assistant), which may include a radio frequency receiver, a pager, Internet / intranet access, a web browser, a notepad, a calendar and / or a GPS (Global Positioning System) receiver; conventional laptop and / or palmtop computers or other devices, which have and / or include a conventional laptop and / or palmtop computer or other device with a radio frequency receiver. The "terminal" or "terminal device" used herein may be portable, transportable, installed in a vehicle (air, sea and / or land), or adapted and / or configured to operate locally, and / or in a distributed form, at any other location on the earth and / or in space. The "terminal" or "terminal device" used herein may also be a communication terminal, an Internet terminal, a music / video playing terminal, such as a PDA, a MID (Mobile Internet Device) and / or a mobile phone with a music / video playing function, or a smart TV, a set-top box and other devices.

[0068] The client terminal in this embodiment is the terminal mentioned above.

[0069] Specifically, see Figure 1 , Figure 1 Schematic diagram of the basic flow of the model evaluation method of this embodiment.

[0070] like Figure 1 As shown, the model evaluation method includes the following steps:

[0071] S1100, obtaining an artificial intelligence model to be evaluated;

[0072] In practical applications, the application of artificial intelligence is involved in multiple scenarios, such as face recognition at the entry and exit gates, voice recognition during voice calls, etc. However, due to the different details of different scenarios, when applying artificial intelligence in different scenarios, the requirements for the deep learning models involved in artificial intelligence are different. When developing artificial intelligence, in order to predict the difficulty of later development products (scenarios of artificial intelligence applications), resources are prepared and allocated in advance. In this embodiment, the artificial models to be evaluated are a variety of deep learning algorithms, such as convolutional neural network algorithms, linear regression algorithms, logistic regression algorithms, decision tree algorithms, support vector machine algorithms, naive Bayesian algorithms, K nearest neighbor algorithms, K mean algorithms, random forest algorithms, and dimensionality reduction algorithms, etc.

[0073] In some embodiments, there are artificial intelligence chips containing different deep learning algorithms on the market, and multiple artificial intelligence chips are selected, and artificial intelligence algorithms are extracted from the various artificial intelligence chips as the artificial intelligence models to be evaluated.

[0074] It should be noted that since artificial intelligence is applied in different scenarios, the evaluation of artificial intelligence models focuses on the current application scenario, that is, the product to be developed.

[0075] S1200, calculating the artificial intelligence model using a preset model evaluation method to obtain evaluation parameters;

[0076] Evaluation parameters are used to evaluate whether the artificial intelligence model is suitable for the product. The evaluation parameters include the computing power, memory bandwidth, precision loss, and friendliness to the product's hardware of the artificial intelligence model.

[0077] In some embodiments, when the evaluation parameter is computing power, computing power refers to the number of floating point operations per second, which can be understood as computing speed. When the artificial intelligence model is a convolutional neural network model, when calculating the computing power (Floating Point Operations Per Second, FLOPs) of the artificial intelligence model, it is assumed that the size of a convolutional layer is h*w*c*n, where h and w represent the size of the convolution layer, c represents the output channel, n represents the output channel, and the size of the output feature map is H'*W', then the computing power of the convolution layer FLOPs = H'*W'*[n*(h*w*c)+1]. When calculating the computing power of the convolutional neural network model, the computing power of multiple convolution layers can be obtained by multiplying them.

[0078] In some embodiments, the evaluation parameter is the memory bandwidth required by the artificial intelligence model. In practical applications, the parameters for measuring the model operation speed are mainly computing power and required memory bandwidth, wherein memory bandwidth is usually expressed by operational intensity (or arithmetic intensity), and the unit is OPs / byte, that is, how many operations can be supported per unit data read in the artificial intelligence model on average. Among them, the greater the operational intensity, the more operations can be supported per unit data, that is, the lower the requirement of the artificial intelligence model for memory bandwidth.

[0079] In this embodiment, an example is given to illustrate the method of calculating the memory bandwidth required for the artificial intelligence model: for a 3×3 convolution operation with a stride of 1, it is assumed that the input data plane size is 64×64. For simplicity, it is assumed that both the input and output features are 1. At this time, a total of 62×62 convolution operations are required, and each convolution requires 3×3=9 multiplication and addition operations, so the total number of calculations is 34596, and the amount of data is (assuming that both the data and the convolution kernel use single-precision floating point numbers 2 bytes): 64x64x2 (input data) + 3x3x2 (convolution kernel data) = 8210 bytes, so the computing intensity, that is, the required memory bandwidth, is 34596 / 8210 = 4.21.

[0080] In some embodiments, the evaluation parameter is precision loss. When calculating the precision loss, the artificial intelligence model is compressed by a quantization tool and quantized according to a mode selected by the user, for example, int8, fp16, or fp32, and the precision loss of each layer of the quantized artificial intelligence model is compared with the precision loss of the artificial intelligence model before compression to obtain the precision loss of each layer of the artificial intelligence model, and then the average precision loss of the artificial intelligence model is obtained through an averaging algorithm.

[0081] In some embodiments, the evaluation parameter is the friendliness of the product's hardware, wherein the artificial intelligence model can be decomposed to obtain structural units, the proportion of different types of structural units in all structural units can be counted, and the scores of the support of the above-mentioned different types of structural units for the product framework can be found in the scoring information table, and the proportions and scores of different types of structural units can be weighted to obtain the friendliness of the artificial intelligence model to the product.

[0082] S1300. Determine whether the artificial intelligence model is suitable for the product to be evaluated based on preset performance parameters and evaluation parameters of the product to be evaluated, so as to obtain an evaluation result of the artificial intelligence model.

[0083] In some embodiments, the evaluation parameters can be compared with the performance parameters of the product to be evaluated. When the comparison is consistent, it is determined that the artificial intelligence model is suitable for the product to be evaluated; when the comparison is inconsistent, it is determined that the artificial intelligence model is not suitable for the product to be evaluated.

[0084] In some embodiments, when there are multiple performance parameters, some performance parameters may be consistent with the evaluation parameters while some may be inconsistent. In this case, weights are set according to the importance of the performance parameters, and weighted operations are performed on the consistent evaluation parameters to obtain the evaluation result of the artificial intelligence model applied to the product to be evaluated.

[0085] It should be noted that the performance parameters of the product to be evaluated are the computing power, bandwidth, applied hardware framework, loss precision, etc. determined according to the scenario to which it is applied. For example, the application scenario of the product to be evaluated is face recognition at entry and exit. Due to the large flow of people, the required computing power and bandwidth are large, and the loss precision is also high.

[0086] The above model evaluation method calculates the artificial intelligence model through a preset model evaluation method to obtain evaluation parameters, and then evaluates the artificial intelligence model according to the preset performance parameters and evaluation parameters of the product to be evaluated to obtain the evaluation result of the artificial intelligence model. This method can select the preferred artificial intelligence model by obtaining the evaluation result through the performance parameters and evaluation parameters of the product to be evaluated, which is convenient for resource allocation and effectively reduces the difficulty of development.

[0087] In the embodiment of the present invention, each artificial intelligence model has different friendliness to the hardware framework, wherein the more the artificial intelligence model supports the hardware framework, the smoother the operation in the hardware. Therefore, when the evaluation parameter is the friendliness to the hardware of the product, the embodiment of the present invention provides a method for calculating the artificial intelligence model through a preset model evaluation method to obtain the evaluation parameter, such as Figure 2 As shown, Figure 2 A basic flow chart of a method for calculating evaluation parameters of an artificial intelligence model using a preset model evaluation method is provided for an embodiment of the present invention.

[0088] Specifically, Figure 2 As shown, step S1200 includes the following steps:

[0089] S1211. Obtain structural unit information of the artificial intelligence model, wherein the structural unit information includes the percentage of the number of each type of structural units to the total number of all structural units;

[0090] The structural unit information is used to represent the information of the basic calculation formula of the artificial intelligence model. For example, when the artificial intelligence model is a convolutional neural network model, the structural unit includes: convolution kernel. In practical applications, each artificial intelligence model includes multiple types of structural units. In this embodiment, when obtaining the structural unit information, the percentage of each type of structural unit in the total number is counted.

[0091] S1212, searching for the friendliness score of each type of structural unit to the product in a preset scoring information table;

[0092] The scoring information table is used to represent the evaluation table of the running performance of various types of structural units under different hardware frameworks, wherein the friendliness score is used in the scoring information table to measure the degree of running of the structural unit under the hardware framework. For example, if the structural unit of type A runs fast and efficiently in the first hardware framework, then the friendliness score is high, and if the structural unit of type B runs slowly and inefficiently in the second hardware framework, then the friendliness score is low.

[0093] In this embodiment, the hardware framework is a hardware and software operating system existing in the prior art, and its friendliness score can be set according to actual experience.

[0094] S1213. Perform a weighted calculation on the percentage of each type of structural unit in the artificial intelligence model and the corresponding friendliness score to obtain the friendliness of the artificial intelligence model to the hardware of the product.

[0095] Taking this embodiment as an example, it is assumed that the product hardware framework is the first hardware framework, and the artificial intelligence model includes: 10 Class A structural units, 30 Class B structural units, and 10 Class C structural units. Then, the proportion of Class A structural units is 20%, the proportion of Class B structural units is 60%, and the proportion of Class C structural units is 20%. Among them, by consulting the scoring information table, the friendliness scores of Class A, Class B and Class C structural units to the first hardware framework are 1 point, 5 points, and 3 points respectively. Then, the friendliness score of the artificial intelligence model to the first hardware framework is 1*20%+5*60%+3*20%=3.8 points.

[0096] When the evaluation parameter is precision loss, an embodiment of the present invention provides a method for calculating the evaluation parameter of the artificial intelligence model by a preset model evaluation method, such as Figure 3 As shown, Figure 3 A basic flow chart of a method for calculating evaluation parameters of an artificial intelligence model using a preset model evaluation method is provided for an embodiment of the present invention.

[0097] Specifically, Figure 3 As shown, step S1200 includes the following steps:

[0098] S1221. compressing the artificial intelligence model;

[0099] In some implementations, compression can be performed by reducing the parameters of the model, for example, using SqueezeNet, MobileNet, etc.

[0100] In some embodiments, the model can also be pruned, for example, pruning, weight sharing or neural network binarization. Among them, since the neural network in the artificial intelligence model is connected by layers of nodes through edges, each edge has a weight, pruning is to delete the edge with a smaller weight. Weight sharing is to let some edges share a weight to achieve the purpose of reducing the number of parameters. Assuming that the two adjacent layers are fully connected and each layer has 1000 nodes, there are 1000*1000=1 million weight parameters between the two layers. These one million weights can be clustered, and the mean of each class is used to replace the size of each weight in this class, so that many edges belonging to the same class share the same weight. Assuming that one million weights are clustered into one thousand classes, the number of parameters can be reduced from one million to one thousand. Neural network binarization is to use binary numbers instead of floating point numbers to represent all weights, either +1 or -1, and to represent them in binary. The original 32-bit weight now only needs one bit to represent it, which can greatly reduce the model size.

[0101] S1222. quantize the compressed artificial intelligence model according to the quantization type selected by the user;

[0102] Generally, quantization is to reduce the space occupied by each weight by reducing the precision. For example, the parameters of the neural network model are all represented by 32-bit floating-point numbers. In fact, when it is not necessary to retain such high precision, it can be achieved through quantization, such as using 0 to 255 to represent the precision represented by the original 32 bits. In this embodiment, quantization can be performed according to the type selected by the user. For example, the user can choose three types: int8, fp16 or fp32. Among them, when the user selects the quantization type, he can refer to the details of the product for selection.

[0103] S1223, comparing the loss accuracy of each layer of the quantized artificial intelligence model with that of the artificial intelligence model before compression, and obtaining the accuracy loss of each layer;

[0104] S1224. Perform an average calculation on the precision loss of each layer to obtain the precision loss of the artificial intelligence model.

[0105] It should be noted that the accuracy loss of the model is obtained after the artificial intelligence model is adapted to the product. Usually, after the above processing, some models have a smaller accuracy loss, while some models have a larger accuracy loss. Usually, a larger accuracy loss will affect the operation of the model.

[0106] The embodiment of the present invention provides a method for determining whether an artificial intelligence model is suitable for a product to be evaluated based on preset performance parameters and evaluation parameters of the product to be evaluated. Figure 4 As shown, Figure 4 A basic flow chart of a method for determining whether an artificial intelligence model is suitable for a product to be evaluated based on preset performance parameters and evaluation parameters of the product to be evaluated provided in an embodiment of the present invention.

[0107] Specifically, Figure 4 As shown, step S1300 includes the following steps:

[0108] S1310, comparing the evaluation parameters with the performance parameters of the product to be evaluated to obtain a score of the artificial intelligence model suitable for the product to be evaluated;

[0109] The embodiment of the present invention provides a method for comparing the evaluation parameters with the performance parameters of the product to be evaluated to obtain the fitness score of the artificial intelligence model for the product to be evaluated, such as Figure 5 As shown, Figure 5 A basic flow chart of a method for comparing evaluation parameters with performance parameters of a product to be evaluated to obtain a fitness score of an artificial intelligence model for the product to be evaluated provided in an embodiment of the present invention.

[0110] Specifically, Figure 5 , step S1310 includes the following steps:

[0111] S1311, obtaining a performance scoring information table for indicating the degree of operation of the product, wherein the performance scoring information table includes a plurality of performance parameter ranges and a fitness corresponding to each performance parameter range;

[0112] Generally, due to different product application scenarios, the requirements for their operating performance are different. Among them, there will be a performance parameter range that is very suitable for the product. Therefore, in this embodiment, a performance score information table is preset, wherein multiple parameter ranges are preset for each performance parameter, and score values ​​are preset according to the size of the suitability of the product. The performance parameters of the product also include: computing power, memory bandwidth, precision loss, and hardware friendliness. Among them, taking computing power as an example, three ranges are preset for computing power in the performance score information table, for example, A is 100-1000, B1000-10000, and C is greater than 10000. Since the application range of the product is small and does not require a lot of computing power, 1000-10000 is the most suitable range. Therefore, B has a higher fitness of 3, and A and B have lower fitness of 1 and 2 respectively.

[0113] S1312, determining a target performance parameter range suitable for the evaluation parameter, and searching for the degree of fitness corresponding to the target performance parameter range in the performance scoring information table;

[0114] In this embodiment, the evaluation parameters of artificial intelligence are calculated by the above method, and the range corresponding to the evaluation parameters is found in the performance score list according to the type. For example, the evaluation parameter is computing power and the obtained computing power value is 20000. The computing power 20000 is compared with the preset computing power range in the performance score list. If it meets the C range, the fitness corresponding to the C range is determined to be the fitness of the artificial intelligence model and the computing power in the product.

[0115] S1313. Perform weighted calculations on multiple fitness levels to obtain a score for whether the artificial intelligence model is suitable for the product to be evaluated.

[0116] The fitness of multiple scoring parameters is determined according to the above method. The importance of each product performance parameter to the product is different. Therefore, a weight can be set according to the importance of each performance parameter to the product. In this embodiment, when calculating the score of the artificial intelligence model and the product, the weight of the performance parameter is weighted with the referenced score to finally obtain the evaluation result of the artificial intelligence model.

[0117] S1320, determining whether the score is greater than a preset score;

[0118] S1330. When the score is greater than a preset score, it is determined that the artificial intelligence model is suitable for the product to be evaluated, and the score is used as the evaluation result.

[0119] The preset score can be set according to the conditions of different artificial intelligence models in actual applications when running in various hardware frameworks. In some embodiments, when there are multiple artificial intelligence models with scores greater than the preset score, the artificial intelligence model with the highest score can be selected as the standby model for the product.

[0120] In some embodiments, when the score is less than or equal to a preset score, it is determined that the artificial intelligence model is not suitable for the product to be evaluated. In this embodiment, since there are various artificial intelligence model chips on the market, some models can be excluded through this method, which facilitates the study of the remaining models, thereby saving financial and material resources.

[0121] To solve the above technical problems, the present invention also provides a model evaluation device. Figure 6 , Figure 6 This is a basic structural block diagram of the model evaluation device of this embodiment.

[0122] like Figure 6 As shown, a model evaluation device includes: an acquisition module 2100, a processing module 2200 and an execution module 2300. Among them, the acquisition module 2100 is used to obtain the artificial intelligence model to be evaluated; the processing module 2200 is used to calculate the artificial intelligence model through a preset model evaluation method to obtain evaluation parameters; the execution module 2300 is used to determine whether the artificial intelligence model is suitable for the product to be evaluated based on the preset performance parameters of the product to be evaluated and the evaluation parameters, so as to obtain the evaluation result of the artificial intelligence model.

[0123] The above-mentioned model evaluation device calculates the artificial intelligence model through a preset model evaluation method to obtain evaluation parameters, and then evaluates the artificial intelligence model according to the preset performance parameters and evaluation parameters of the product to be evaluated to obtain the evaluation result of the artificial intelligence model. This method obtains the evaluation result by using the performance parameters and evaluation parameters of the product to be evaluated, and can select the preferred artificial intelligence model, which is convenient for resource allocation and effectively reduces the difficulty of development.

[0124] In some embodiments, the processing module includes: a first acquisition submodule, used to acquire structural unit information of the artificial intelligence model, wherein the structural unit information includes the percentage of the number of each type of structural units to the total number of all structural units; a first processing submodule, used to search for the friendliness score of each type of structural unit to the product in a preset scoring information table; and a first execution submodule, used to perform a weighted operation on the percentage of each type of structural unit in the artificial intelligence model and the corresponding friendliness score to obtain the friendliness of the artificial intelligence model to the hardware of the product.

[0125] In some embodiments, the processing module includes: a second processing sub-module, used to compress the artificial intelligence model; a third processing sub-module, used to quantize the compressed artificial intelligence model according to the quantization type selected by the user; a fourth processing sub-module, used to compare the loss accuracy of each layer of the quantized artificial intelligence model with that of the artificial intelligence model before compression, to obtain the accuracy loss of each layer; and a second execution sub-module, used to average the accuracy loss of each layer to obtain the accuracy loss of the artificial intelligence model.

[0126] In some embodiments, the execution module includes: a fifth processing sub-module, used to compare the evaluation parameters with the performance parameters of the product to be evaluated to obtain a score for the artificial intelligence model to be suitable for the product to be evaluated; a sixth processing sub-module, used to determine whether the score is greater than a preset score; and a third execution sub-module, used to determine that the artificial intelligence model is suitable for the product to be evaluated when the score is greater than the preset score, and to use the score as the evaluation result.

[0127] In some embodiments, the fifth processing submodule includes: a second acquisition submodule, used to obtain a performance score information table used to represent the degree of operation of the product, wherein the performance score information table includes multiple performance parameter ranges and the fitness corresponding to each performance parameter range; a seventh processing submodule, used to determine the target performance parameter range that the evaluation parameter is suitable for, and to search the fitness corresponding to the target performance parameter range in the performance score information table; and a fourth execution submodule, used to perform weighted operations on multiple fitnesses to obtain a score for the artificial intelligence model that conforms to the product to be evaluated.

[0128] In some embodiments, it also includes: an eighth processing submodule, which is used to determine that the artificial intelligence model is not suitable for the product to be evaluated when the score is less than or equal to the preset score.

[0129] In some embodiments, the evaluation parameters also include: computing power and memory bandwidth of the artificial intelligence model.

[0130] To solve the above technical problems, the embodiment of the present invention also provides a computer device. Figure 7 , Figure 7 This is a basic structural block diagram of the computer device in this embodiment.

[0131] like Figure 7 As shown in Figure 1, the internal structure diagram of a computer device. Figure 7As shown, the computer device includes a processor, a non-volatile storage medium, a memory and a network interface connected via a system bus. Among them, the non-volatile storage medium of the computer device stores an operating system, a database and computer-readable instructions, and the database may store a control information sequence. When the computer-readable instructions are executed by the processor, the processor can implement a model evaluation method. The processor of the computer device is used to provide computing and control capabilities to support the operation of the entire computer device. The memory of the computer device may store computer-readable instructions. When the computer-readable instructions are executed by the processor, the processor can execute a model evaluation method. The network interface of the computer device is used to connect and communicate with a terminal. Those skilled in the art can understand that Figure 7 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0132] In this embodiment, the processor is used to execute Figure 6 The specific contents of the acquisition module 2100, the processing module 2200 and the execution module 2300 are stored in the memory, and the program code and various data required to execute the above modules are stored. The network interface is used to transmit data between the user terminal or the server. The memory in this embodiment stores the program code and data required to execute all sub-modules in the model evaluation method, and the server can call the program code and data of the server to execute the functions of all sub-modules.

[0133] The computer device calculates the artificial intelligence model through a preset model evaluation method to obtain evaluation parameters, and then evaluates the artificial intelligence model according to the preset performance parameters and evaluation parameters of the product to be evaluated to obtain the evaluation result of the artificial intelligence model. This method obtains the evaluation result by using the performance parameters and evaluation parameters of the product to be evaluated, and can select the preferred artificial intelligence model, which is convenient for resource allocation and effectively reduces the difficulty of development.

[0134] The present invention also provides a storage medium storing computer-readable instructions, which, when executed by one or more processors, enables the one or more processors to execute the steps of the model evaluation method described in any of the above embodiments.

[0135] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program, and the computer program can be stored in a computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, the aforementioned storage medium can be a non-volatile storage medium such as a disk, an optical disk, a read-only memory (ROM), or a random access memory (RAM).

[0136] It should be understood that, although the steps in the flowchart of the accompanying drawings are displayed in sequence as indicated by the arrows, these steps are not necessarily executed in sequence in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least a part of the steps in the flowchart of the accompanying drawings may include multiple sub-steps or multiple stages, and these sub-steps or stages are not necessarily executed at the same time, but can be executed at different times, and their execution order is not necessarily sequential, but can be executed in turn or alternately with other steps or at least a part of the sub-steps or stages of other steps.

[0137] The above descriptions are only some embodiments of the present invention. It should be pointed out that, for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A model evaluation method, characterized in that: The steps include: Obtain the AI ​​model to be evaluated; The artificial intelligence model is calculated by a preset model evaluation method to obtain evaluation parameters, wherein the evaluation parameters include computing power, memory bandwidth, precision loss, and friendliness to product hardware of the artificial intelligence model; Determining whether the artificial intelligence model is suitable for the product to be evaluated based on the preset performance parameters of the product to be evaluated and the evaluation parameters, so as to obtain an evaluation result of the artificial intelligence model, wherein the product to be evaluated is a product to be developed and deployed in an actual artificial intelligence application scenario; If the evaluation parameter is the friendliness of the hardware of the product, the artificial intelligence model is calculated by a preset model evaluation method to obtain the evaluation parameter, including: Obtaining structural unit information of the artificial intelligence model, wherein the structural unit information includes the percentage of the number of each type of structural units to the total number of all structural units; Find the friendliness rating of each type of structural unit to the product in the preset rating information table; The percentage of each type of structural unit in the artificial intelligence model and the corresponding friendliness score are weighted and calculated to obtain the friendliness of the artificial intelligence model to the hardware of the product.

2. The model evaluation method according to claim 1, characterized in that: The evaluation parameter is the precision loss; the artificial intelligence model is calculated by a preset model evaluation method to obtain the evaluation parameter, including: compressing the artificial intelligence model; Quantize the compressed AI model according to the quantization type selected by the user; Compare the loss accuracy of each layer of the quantized AI model with that of the AI ​​model before compression to obtain the accuracy loss of each layer; The precision loss of each layer is averaged to obtain the precision loss of the artificial intelligence model.

3. The model evaluation method according to claim 1, characterized in that: The step of judging whether the artificial intelligence model is suitable for the product to be evaluated based on the preset performance parameters of the product to be evaluated and the evaluation parameters to obtain the evaluation result of the artificial intelligence model includes: Comparing the evaluation parameters with the performance parameters of the product to be evaluated to obtain a score of whether the artificial intelligence model is suitable for the product to be evaluated; Determine whether the score is greater than a preset score; When the score is greater than the preset score, it is determined that the artificial intelligence model is suitable for the product to be evaluated, and the score is used as the evaluation result.

4. The model evaluation method according to claim 3, characterized in that: The step of comparing the evaluation parameters with the performance parameters of the product to be evaluated to obtain a score of the artificial intelligence model suitable for the product to be evaluated includes: Obtaining a performance scoring information table for indicating the degree of operation of the product, wherein the performance scoring information table includes a plurality of performance parameter ranges and a fitness corresponding to each performance parameter range; Determine the target performance parameter range that the evaluation parameter is suitable for, and search the performance score information table for the degree of fitness corresponding to the target performance parameter range; A weighted calculation is performed on multiple fitness levels to obtain a score for whether the artificial intelligence model is suitable for the product to be evaluated.

5. The model evaluation method according to claim 3, characterized in that: After determining whether the score is greater than a preset score, the method further includes: When the score is less than or equal to the preset score, it is determined that the artificial intelligence model is not suitable for the product to be evaluated.

6. The model evaluation method according to claim 1, characterized in that: The evaluation parameters also include: computing power and memory bandwidth of the artificial intelligence model.

7. A model evaluation device, characterized in that: include: An acquisition module, used to acquire the artificial intelligence model to be evaluated; A processing module, used to calculate the artificial intelligence model by a preset model evaluation method to obtain evaluation parameters, wherein the evaluation parameters include computing power, memory bandwidth, precision loss, and friendliness to product hardware of the artificial intelligence model; An execution module, used to determine whether the artificial intelligence model is suitable for the product to be evaluated according to preset performance parameters of the product to be evaluated and the evaluation parameters, so as to obtain an evaluation result of the artificial intelligence model, wherein the product to be evaluated is a product to be developed and deployed in an actual artificial intelligence application scenario; Wherein, the processing module includes: A first acquisition submodule is used to acquire structural unit information of the artificial intelligence model, wherein the structural unit information includes the percentage of the number of each type of structural units to the total number of all structural units; The first processing submodule is used to search for the friendliness score of each type of structural unit to the product in a preset scoring information table; The first execution submodule is used to perform a weighted operation on the percentage of each type of structural unit in the artificial intelligence model and the corresponding friendliness score to obtain the friendliness of the artificial intelligence model to the hardware of the product.

8. The model evaluation device according to claim 7, characterized in that: The processing module comprises: A second processing submodule, used for compressing the artificial intelligence model; A third processing submodule, for quantizing the compressed artificial intelligence model according to a quantization type selected by a user; A fourth processing submodule is used to compare the loss accuracy of each layer of the quantized artificial intelligence model with that of the artificial intelligence model before compression to obtain the accuracy loss of each layer; The second execution submodule is used to perform an average operation on the precision loss of each layer to obtain the precision loss of the artificial intelligence model.

9. The model evaluation device according to claim 7, characterized in that: The execution module includes: A fifth processing submodule, configured to compare the evaluation parameters with the performance parameters of the product to be evaluated to obtain a score of the artificial intelligence model being suitable for the product to be evaluated; A sixth processing submodule, used to determine whether the score is greater than a preset score; The third execution submodule is used to determine that the artificial intelligence model is suitable for the product to be evaluated when the score is greater than the preset score, and use the score as the evaluation result.

10. The model evaluation device according to claim 9, characterized in that The fifth processing submodule comprises: A second acquisition submodule is used to acquire a performance score information table for indicating the degree of operation of the product, wherein the performance score information table includes a plurality of performance parameter ranges and a fitness corresponding to each performance parameter range; A seventh processing submodule, configured to determine a target performance parameter range suitable for the evaluation parameter, and to search the performance scoring information table for a degree of fitness corresponding to the target performance parameter range; The fourth execution submodule is used to perform weighted calculation on multiple fitness levels to obtain a score for whether the artificial intelligence model conforms to the product to be evaluated.

11. The model evaluation device according to claim 9, characterized in that: Also includes: The eighth processing submodule is used to determine that the artificial intelligence model is not suitable for the product to be evaluated when the score is less than or equal to the preset score.

12. The model evaluation device according to claim 7, characterized in that: The evaluation parameters also include: computing power and memory bandwidth of the artificial intelligence model.

13. A computer device comprising a memory and a processor, wherein the memory stores computer-readable instructions, and when the computer-readable instructions are executed by the processor, the processor performs the steps of the model evaluation method according to any one of claims 1 to 6.

14. A storage medium storing computer-readable instructions, wherein when the computer-readable instructions are executed by one or more processors, the one or more processors execute the steps of the model evaluation method according to any one of claims 1 to 6.

Images