Algorithm model performance evaluation method and device, electronic equipment and storage medium

Abstract

The invention discloses an algorithm model performance evaluation method and device, electronic equipment and a storage medium, and relates to the technical field of artificial intelligence, and the method comprises the steps: obtaining a function list of an algorithm model; obtaining mapping scenes corresponding to the functions according to the function list; determining behavior characteristics corresponding to the plurality of scenes and influence weights corresponding to the behavior characteristics according to the mapping scene; and performing performance evaluation on the algorithm model according to each behavior feature and the influence weight corresponding to each behavior feature. The mapping scene of each function of the algorithm model is obtained, the scene of each function is disassembled into each feature behavior, the algorithm model is comprehensively and accurately evaluated according to the feature behaviors and the influence weights of the feature behaviors, and compared with the existing algorithm model evaluation by using a specific data set, the algorithm model evaluation efficiency is improved. According to the invention, the problem of one-sided evaluation caused by imbalance of the test data set can be effectively solved.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to an algorithm model performance evaluation method, device, electronic device and storage medium. Background Art

[0002] The existing algorithm model performance test scheme is to decompose the data set into a training data set and a test data set, and use the test data set to evaluate the performance of the algorithm model. This is a method often used in the product development stage. However, when the product is put into actual use, the performance effect is often significantly different from that in the development stage, and the effect obviously does not meet expectations. Therefore, before actual production, the algorithm model needs to be tested with a real production data set to verify the performance indicators of the algorithm model. However, in the construction of this test set, there is a human subjective influence. In the performance indicator evaluation of the algorithm model, the industry's performance indicator evaluation formula is simply and directly used to calculate as the evaluation result of its performance indicator. Such results lack a scientific basis for the performance verification of the algorithm model and cannot play the reverse guidance role that the testing work should play. Summary of the invention

[0003] The main purpose of the embodiments of the present application is to propose an algorithm model performance evaluation method, device, electronic device and storage medium to accurately evaluate the performance of the algorithm model.

[0004] To achieve the above purpose, an embodiment of the present application provides a method for evaluating the performance of an algorithm model, the method comprising the following steps:

[0005] Get the function list of the algorithm model;

[0006] Acquire mapping scenarios corresponding to various functions according to the function list;

[0007] Determine, according to the mapping scenario, each behavior feature corresponding to a plurality of scenarios and an influence weight corresponding to each behavior feature;

[0008] The performance of the algorithm model is evaluated according to each of the behavioral features and the influence weight corresponding to each of the behavioral features.

[0009] In some embodiments, obtaining a function list of the algorithm model comprises the following steps:

[0010] The function list is determined according to the product requirement functions of the algorithm model.

[0011] In some embodiments, the step of obtaining a mapping scenario corresponding to each function according to the function list includes the following steps:

[0012] Determine the scene corresponding to each function in the function list;

[0013] Determining each element associated with each of the scenes;

[0014] determining each characteristic associated with each of said elements;

[0015] Determining each characteristic behavior associated with each of the characteristics;

[0016] A data set is constructed according to each of the characteristic behaviors as the mapping scenario.

[0017] In some embodiments, the determining, according to the mapping scenario, each behavior feature corresponding to a plurality of scenarios and an influence weight corresponding to each behavior feature comprises the following steps:

[0018] The influence weights of each of the scenes, each of the elements, each of the features, and each of the feature behaviors are set in sequence using the hierarchical analysis method.

[0019] In some embodiments, the performance evaluation of the algorithm model according to each of the behavioral features and the influence weights corresponding to each of the behavioral features comprises the following steps:

[0020] Calculate a first performance score corresponding to each of the behavioral features according to the influence weight of each of the behavioral features;

[0021] Accumulating the first performance scores corresponding to the functions to obtain the second performance scores;

[0022] Accumulate the second performance scores of the various functions to obtain a total performance score;

[0023] A performance evaluation is performed on the algorithm model according to the total performance score.

[0024] To achieve the above purpose, another aspect of the embodiment of the present application provides an algorithm model performance evaluation device, the device comprising:

[0025] A function acquisition unit, used to obtain a function list of an algorithm model;

[0026] A function mapping unit, used to obtain a mapping scenario corresponding to each function according to the function list;

[0027] A feature and weight determination unit, used to determine each behavior feature corresponding to a plurality of scenarios and an influence weight corresponding to each behavior feature according to the mapping scenario;

[0028] A performance evaluation unit is used to perform performance evaluation on the algorithm model according to each of the behavioral features and the influence weight corresponding to each of the behavioral features.

[0029] In some embodiments, the function mapping unit includes:

[0030] A scene association unit, used to determine the scene corresponding to each function in the function list;

[0031] An element association unit, used to determine each element associated with each of the scenes;

[0032] A feature association unit, used to determine each feature associated with each of the elements;

[0033] A feature behavior association unit, used to determine each feature behavior associated with each of the features;

[0034] The data set construction unit is used to construct a data set as the mapping scenario according to each of the characteristic behaviors.

[0035] In some embodiments, the feature and weight determination unit includes:

[0036] The weight setting unit is used to sequentially set the influence weights of each of the scenes, each of the elements, each of the features, and each of the feature behaviors by using a hierarchical analysis method.

[0037] To achieve the above objective, another aspect of an embodiment of the present application provides an electronic device, the electronic device comprising a memory and a processor, the memory storing a computer program, and the processor implementing the above method when executing the computer program.

[0038] To achieve the above objective, another aspect of an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and the computer program implements the above method when executed by a processor.

[0039] The embodiments of the present application include at least the following beneficial effects:

[0040] This application can obtain a list of functions of the algorithm model; obtain mapping scenarios corresponding to each function based on the list of functions; determine each behavioral feature corresponding to multiple scenarios and the impact weight corresponding to each behavioral feature based on the mapping scenarios; and perform performance evaluation on the algorithm model based on each behavioral feature and the impact weight corresponding to each behavioral feature. This application obtains mapping scenarios for each function of the algorithm model, breaks down the scenarios of each function into each characteristic behavior, and then conducts a comprehensive and accurate evaluation of the algorithm model based on the characteristic behaviors and the impact weights of the characteristic behaviors. Compared with the existing use of specific data sets to evaluate algorithm models, this application can effectively solve the problem of one-sided evaluation caused by imbalanced test data sets. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0042] Figure 1 A flowchart of an algorithm model performance evaluation method provided in an embodiment of the present application;

[0043] Figure 2 An example diagram of a mapping scenario provided in an embodiment of the present application;

[0044] Figure 3 An example flow chart of an algorithm model performance evaluation method provided in an embodiment of the present application;

[0045] Figure 4 Example diagrams of disassembly tests for various scenarios provided in the embodiments of the present application;

[0046] Figure 5 A schematic diagram of the structure of an algorithm model performance evaluation device provided in an embodiment of the present application;

[0047] Figure 6 A schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0048] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application is further described in detail below in conjunction with the accompanying drawings and examples. It should be understood that the specific embodiments described herein are only used to explain the present application and are not intended to limit the present application. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the embodiments of the present application. They are only examples of devices and methods consistent with some aspects of the embodiments of the present application as detailed in the attached claims.

[0049] It is understood that the terms "first", "second", etc. used in this application can be used to describe various concepts in this article, but unless otherwise specified, these concepts are not limited by these terms. These terms are only used to distinguish one concept from another concept. For example, without departing from the scope of the embodiment of the present application, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Depending on the context, the words "if" and "if" as used herein can be interpreted as "at the time of" or "when" or "in response to determination".

[0050] The terms "at least one", "multiple", "each", "any", etc. used in this application, at least one includes one, two or more, multiple includes two or more, each refers to each of the corresponding multiple, and any refers to any one of the multiple.

[0051] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0052] Before describing the embodiments of the present application in detail, some related technologies involved in the embodiments of the present application are first described as follows:

[0053] The performance evaluation of artificial intelligence algorithm models mainly focuses on the performance indicators of the algorithm models, such as accuracy, recall rate, precision, intersection-over-union ratio, missed detection rate, false recognition rate, rejection rate, macro-average accuracy, micro-average accuracy, etc.

[0054] IoU (Intersection over Union) is a standard for measuring the accuracy of detecting corresponding objects in a specific data set. It is used in detection and positioning models to evaluate the area intersection over union ratio between the marking result and the model result.

[0055] The performance testing of artificial intelligence algorithm models can generally be divided into two types: one is the testing in the algorithm model R&D stage; the other is the testing and verification of the algorithm model product when it is put into production. The corresponding test data sets for the two are different. The former is a small part of the verification test set divided from the data set in the R&D stage, and the latter is a test data set reconstructed based on the actual production scenario.

[0056] There are some commonalities in the methods of constructing data sets for different types of algorithms or artificial intelligence deep learning models. According to the data set construction logic of this application, a test data set can be based on the "application scenario-element construction in the scenario-feature classification of the elements-behavior formed by the tested target" to finally form a test data set under the corresponding scenario-element-feature-behavior, which is used to evaluate the algorithm performance feedback under this indicator.

[0057] For example: 1) In a large model applied to macroeconomic analysis, there are query scenarios for statistical indicators - elements such as region / time / industry - core region / high-growth region / recent / last year / last month / industrial total / construction total / wholesale and retail total / financial total, etc. - development of core businesses in core regions / statistics of key enterprises this month / growth analysis of a certain industry, etc. 2) Another example is in an algorithm for a regional intrusion application model, in a certain period of time during a certain day, the tested area - vehicle - small car - slowly drives into the inspected area.

[0058] After building the data set, the traditional algorithm model test evaluation method is to input the data set into the algorithm model and perform statistical analysis based on the performance evaluation index calculation method selected according to the type of algorithm model. For example, if a confusion matrix is ​​used in a binary classification algorithm to evaluate the accuracy of its algorithm model, the corresponding calculation method is as follows:

[0059] Confusion matrix: In a binary classification task, a second-order matrix is ​​formed by the sample labels and the model classification results, where the true value and the model prediction value are both positive, recorded as a true sample (TP); the true value is negative and the model prediction value is positive, recorded as a false positive sample (FP); the true value is positive and the model prediction value is negative, recorded as a false negative sample (FN); the true value and the model prediction value are both negative, recorded as a true negative sample (TN). See Table 1 for details:

[0060] Table 1 Binary classification confusion matrix

[0061]

[0062] The accuracy rate is equal to the ratio of the number of correctly identified and classified samples to the total number of samples in a given data set. See the following formula for details:

[0063]

[0064] In the past, traditional algorithm model evaluation methods directly used formula calculations to obtain a value to evaluate the accuracy of the algorithm model.

[0065] This type of test quality analysis cannot truly analyze the pain points of the algorithm model. To address the problem that the test set construction in the past was too subjective and the test analysis was too general, this application proposes two optimization technologies to make the algorithm model testing work objective and scientific. The analysis results can provide guidance for optimizing the algorithm model:

[0066] 1) This application proposes an algorithm model test data set to apply scene element feature behavior classification management mechanism innovation;

[0067] 2) This application proposes that the characteristic behaviors of application scenario elements in the test data set of the algorithm model all have weights of their actual application impact, and thus proposes a calculation method for optimizing the performance of the algorithm model under this test data set.

[0068] Related technology 1 proposes to use an online system to select one or more evaluation indicators and corresponding indicator data sets for the algorithm under test to evaluate the algorithm performance. Related technology 1 reflects the test response performance of traditional software, rather than the evaluation of the performance indicators of the algorithm model itself. Moreover, the invention does not mention the problem of constructing the data set required for the testing of artificial intelligence algorithm models, nor does it mention the scientific evaluation of the performance indicators of artificial intelligence algorithm models.

[0069] Related technology 2 proposes that the data set is composed of scene attribute characteristics, which is different from the scene element feature behavior in this application. The data set mentioned in this application is constructed more based on the behaviors that may be triggered by the element characteristics in all application scenarios of the algorithm model to construct a performance data set for evaluating the algorithm model.

[0070] The purpose of weight calculation mentioned in Related Technology 3 is to propose a more objective comprehensive evaluation. Different from the way of using weights in this application, the behavioral scenarios of the data set and the application scenarios of the algorithm model in this application have associated weights and are related to the features and elements of the previous level. Therefore, the weights in this application will involve multiple associated weights, which is significantly different from the way of using them in the comparative patents.

[0071] In summary, the shortcomings of the prior art include:

[0072] 1) The existing test data sets for testing artificial intelligence algorithm models have personal subjective factors, and the selected test data sets cannot objectively and scientifically evaluate the performance of their models.

[0073] 2) When specifying a test data set for algorithm model performance evaluation, the application scenario is not disassembled in the data set construction, and only a test data set with diverse styles is achieved. When feedback on the algorithm model performance, only the performance feedback of the algorithm model under the test data set can be explained, which cannot reflect the objectivity and scientificity of the test;

[0074] 3) When evaluating the performance of an algorithm model, directly calling the calculation formulas of relevant performance indicators, such as the accuracy formula, missed detection rate formula, recall rate formula, IoU (Intersection over Union) formula, etc., cannot provide feedback on the performance of the algorithm model in different scenarios and with different features.

[0075] 4) Data set imbalance problem: If the test data set contains too much low-quality data that has little impact on the product, the previous algorithm model performance evaluation method will get a very poor result.

[0076] In response to the above-mentioned shortcomings of the prior art, the present application proposes a data set classification management mechanism for a test data set constructed based on scene element features, and an objective and scientific evaluation of the performance indicators of the algorithm model by setting weights based on the degree of recognition influence of scene element features on the actual production of the algorithm model.

[0077] This application can solve the scientific problems of algorithm model performance evaluation verification and result analysis in the field of artificial intelligence. In this application, the following technical solutions are proposed to improve the objectivity and scientificity of algorithm model performance evaluation testing work:

[0078] 1) This application proposes an algorithm model test data set to apply scene element feature behavior classification management mechanism innovation;

[0079] 2) This application proposes that the behavior characteristics of application scenario elements in the test data set of the algorithm model all have their actual application impact weights, and proposes a calculation method for optimizing the performance of the algorithm model under this test data set;

[0080] 3) The algorithm model evaluation method of this application effectively solves the impact problem caused by the imbalance of test data sets.

[0081] The embodiments of the present application provide a method, device, electronic device and storage medium for evaluating the performance of an algorithm model. The technical solution of the present application includes: obtaining a function list of the algorithm model; obtaining mapping scenarios corresponding to each function according to the function list; determining each behavior feature corresponding to multiple scenarios and the influence weight corresponding to each behavior feature according to the mapping scenarios; and evaluating the performance of the algorithm model according to each behavior feature and the influence weight corresponding to each behavior feature. The present application obtains the mapping scenarios of each function of the algorithm model, decomposes the scenarios of each function into each characteristic behavior, and then comprehensively and accurately evaluates the algorithm model according to the characteristic behaviors and the influence weights of the characteristic behaviors. Compared with the existing method of evaluating the algorithm model using a specific data set, the present application can effectively solve the problem of one-sided evaluation caused by the imbalance of the test data set.

[0082] The embodiments of the present application provide an algorithm model performance evaluation method, device, electronic device and storage medium, which relate to the field of artificial intelligence technology. The algorithm model performance evaluation method, device, electronic device and storage medium provided in the embodiments of the present application can be applied to a terminal, can also be applied to a server, and can also be software running in a terminal or a server. In some embodiments, the terminal can be a smart phone, a tablet computer, a laptop computer, a desktop computer, a smart speaker, a smart watch, and a car terminal, etc., but is not limited to this; the server side can be configured as an independent physical server, or a server cluster or distributed system composed of multiple physical servers, and can also be configured to provide cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms and other basic cloud computing services. The server can also be a node server in a blockchain network; the software can be an application that implements a knowledge extraction method, etc., but is not limited to the above forms.

[0083] The present application can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, etc. The present application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present application can also be practiced in distributed computing environments, in which tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media including storage devices.

[0084] Reference Figure 1 , the embodiment of the present application provides an algorithm model performance evaluation method, which may include but is not limited to S100 to S130, as follows:

[0085] S100: Obtain a function list of the algorithm model.

[0086] Exemplarily, the function list of this embodiment may be in the form of a list, in which the various functions that can be implemented by the algorithm model may be listed separately by type.

[0087] Further, S100 may include S101:

[0088] S101: Determine the function list according to the product requirement functions of the algorithm model.

[0089] S110: Acquire a mapping scenario corresponding to each function according to the function list.

[0090] Furthermore, S110 may include S111 to S115:

[0091] S111: Determine the scene corresponding to each function in the function list;

[0092] S112: Determine each element associated with each of the scenes;

[0093] S113: Determine each feature associated with each of the elements;

[0094] S114: Determine each characteristic behavior associated with each characteristic;

[0095] S115: Constructing a data set as the mapping scenario according to each of the characteristic behaviors.

[0096] For example, Figure 2 This is an example diagram of a mapping scenario provided in this embodiment.

[0097] S120: Determine, according to the mapping scenario, each behavior feature corresponding to a plurality of scenarios and an influence weight corresponding to each behavior feature.

[0098] Further, S120 may include S121:

[0099] S121: using the analytic hierarchy process to sequentially set the influence weights of the scenes, the elements, the features, and the feature behaviors.

[0100] S130: Performing a performance evaluation on the algorithm model according to each of the behavioral features and the impact weight corresponding to each of the behavioral features.

[0101] Further, S130 may include S131 to S134:

[0102] S131: Calculating a first performance score corresponding to each of the behavior features according to the influence weight of each of the behavior features;

[0103] S132: accumulating the first performance scores corresponding to the functions to obtain the second performance scores;

[0104] S133: accumulating the second performance scores of the functions to obtain a total performance score;

[0105] S134: Performing a performance evaluation on the algorithm model according to the total performance score.

[0106] Next, the solution of the embodiment of the present application will be introduced and explained in detail with reference to specific application examples.

[0107] Reference Figure 3 , this embodiment provides an example flowchart of an algorithm model performance evaluation method.

[0108] This embodiment proposes an algorithm model performance evaluation method to address the problem of inaccurate performance evaluation of artificial intelligence algorithm models, which can perform performance evaluation based on feature weight factors and realize classification management of test data sets.

[0109] In the construction of the test data set, in order to objectively and scientifically evaluate the performance of the algorithm model, the method of decomposing the characteristic behaviors of the scene elements according to the functions of the algorithm model product is adopted, and the corresponding test data set is constructed according to the element characteristics to evaluate the performance of the element characteristics.

[0110] Specifically, this embodiment may include the following technical solutions:

[0111] 1) Import the algorithm model to be tested, and form a function list of the algorithm model through the product requirement function selection of the algorithm model, which is recorded as It means that there are m product requirement functions in this algorithm model.

[0112] 2) Cyclic reading Get each feature F m , (m=1,2,3,…m) corresponding mapping scenarios. Associate the product's demand functions with the application scenarios, and use the hierarchical analysis method to build custom weights for product requirements and application scenarios:

[0113] F m →scn m =[sc1,sc2,…,sc n ];

[0114] Indicates that the mth function item includes a set of n application scenarios, denoted as scn m When determining the scenario, the weight of each scenario on its function can be constructed by using the hierarchical analysis method according to the application scenario of its function, which is expressed as follows:

[0115] F m →scn m →wscn m =[wsc1,wsc2,…,wsc n ];

[0116] For example, in an algorithm model for regional intrusion, its application scenarios include intrusion of the object under test during the day, intrusion of the object under test during the evening, intrusion of the object under test in the dark before dawn, obstruction of the object under test in the intrusion area, movement of non-living equipment in the intrusion area, etc. Combined with the relevance of the scenarios of its actual production application, its relevance weight can be assigned, such as:

[0117] F→scn m →wscn m =[0.3,0.3,0.2,0.15,0.05];

[0118] 3) Get scn m After the list is obtained, query to obtain each sc n The associated element set is recorded as:

[0119] Elemc=[elc1,elc2,…,elc l ];

[0120] The above formula represents the scene scn m A collection of l elements, where elc l It is a set formed by this element. When the algorithm model product is actually put into production, the elements of the test data set have different influences on the scenario. The weight values ​​of the corresponding element features are recorded as follows:

[0121] W ele =[w ele1 ,w ele2 ,…,w elel ];

[0122] It indicates the degree of influence of the element on the scene. For example, if the scene of the algorithm for regional intrusion is the recognition of people entering the detection area, then when the elements in the scene are ambient light, people, tables, inanimate mobile devices, occlusion, etc., the weight of the element-scene can be obtained according to the hierarchical analysis method, such as:

[0123] W ele =[0.4,0.4,0.05,0.05,0.1];

[0124] 4) In step 3, the corresponding scene number and the corresponding element-feature set have been obtained. Each element has a corresponding feature. Similarly, the corresponding influence weight can be obtained in step 3):

[0125] EleC ij =[sc i1 ,sc i2 ,…,sc il ];

[0126] For example, the characteristics of ambient light include daytime, evening, and pre-dawn darkness, and their weights can be:

[0127] EleC ij =[0.4,0.4,0.2];

[0128] 5) By looping the characteristic behavior corresponding to each element, the constructed test data set is passed to the trigger interface of the algorithm model, and the function method f is calculated based on the obtained annotation results and the results of the algorithm model, as well as the different performance indicators. i (represents the performance test of the algorithm model in item i) Calculate the performance test result of the characteristic behavior of the element in item i of the algorithm model and record it as:

[0129] Case mi =[sca i1 ,sca i2 ,…,sca il ];

[0130] The weight of the corresponding characteristic behavior on its characteristics, such as the behavior of pedestrians under the light illumination characteristics, the posture, clothing, cover, occlusion, gender, age, etc. of the person, such as:

[0131] wcase i =[0.2,0.15,0.2,0.15,0.2,0.1];

[0132] When the mth scenario completes steps 2, 3, and 4, the algorithm model is tested for the i-th performance, such as Figure 4 shown.

[0133] The Hadamard product of the first-order matrix of the performance test result and the corresponding weight in step 4 is recorded as:

[0134]

[0135] The above formula represents the real impact feedback of the performance of the feature behavior test set corresponding to the overall application scenario, S cnm A first performance score may be represented.

[0136] 6) Based on the results of step 5, the performance results of each function item are statistically analyzed and summed with the total number of scene element feature sets for evaluation, and recorded as:

[0137] Score m =∑S cnm ;

[0138] S total =∑S fm ;

[0139] Where S fm=[Score1, Score2, …, Score m ], representing the mth function F m The performance scores of all mapped scenarios. m can represent the second performance score, s total The total performance score can be expressed.

[0140] S total It represents the scores of all functions of the artificial intelligence algorithm model in other scenario sets. The closer the value is to 1, the better the performance of the algorithm model.

[0141] In addition, the basis for generating the influence weight of the feature in this embodiment may include: one is to directly set it through the results of business analysis; the other is to generate it through the correlation recognition calculation of the scene and element features, such as through the semantic understanding model to identify the correlation between the scene and element features.

[0142] In summary, this embodiment includes the following technical features:

[0143] 1) This embodiment proposes an evaluation method based on feature weight calculation for the performance evaluation of artificial intelligence algorithm models, which effectively solves the problem of one-sided evaluation caused by imbalanced test data sets.

[0144] 2) This embodiment proposes the idea of ​​decomposing the element features based on functional scenarios when constructing a test data set for evaluating an artificial intelligence algorithm model, and constructing a test data set covering the features based on the composition of the element features.

[0145] Therefore, this embodiment has the following beneficial effects:

[0146] 1) The performance evaluation method of the feature weight factor proposed in this embodiment effectively solves the imbalance problem of the test data set, so that the evaluation result can obtain a more objective and scientific analysis result.

[0147] 2) The test data set construction method proposed in this embodiment improves the coverage of the test of the artificial intelligence algorithm model based on the functional decomposition element characteristics, and reduces the adverse effects of the product after it is put into production.

[0148] Reference Figure 5 The embodiment of the present application further provides an algorithm model performance evaluation device, which can implement the above-mentioned algorithm model performance evaluation method, and the device includes:

[0149] A function acquisition unit, used to obtain a function list of an algorithm model;

[0150] A function mapping unit, used to obtain a mapping scenario corresponding to each function according to the function list;

[0151] A feature and weight determination unit, used to determine each behavior feature corresponding to a plurality of scenarios and an influence weight corresponding to each behavior feature according to the mapping scenario;

[0152] A performance evaluation unit is used to perform performance evaluation on the algorithm model according to each of the behavioral features and the influence weight corresponding to each of the behavioral features.

[0153] Furthermore, the function mapping unit includes:

[0154] A scene association unit, used to determine the scene corresponding to each function in the function list;

[0155] An element association unit, used to determine each element associated with each of the scenes;

[0156] A feature association unit, used to determine each feature associated with each of the elements;

[0157] A feature behavior association unit, used to determine each feature behavior associated with each of the features;

[0158] The data set construction unit is used to construct a data set as the mapping scenario according to each of the characteristic behaviors.

[0159] Furthermore, the feature and weight determination unit includes:

[0160] The weight setting unit is used to sequentially set the influence weights of each of the scenes, each of the elements, each of the features, and each of the feature behaviors by using a hierarchical analysis method.

[0161] It can be understood that the contents of the above method embodiments are all applicable to the present device embodiments, the functions specifically implemented by the present device embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0162] The embodiment of the present application also provides an electronic device, the electronic device includes a memory and a processor, the memory stores a computer program, and the processor implements the above algorithm model performance evaluation method when executing the computer program. The electronic device can be any smart terminal including a tablet computer, a car computer, etc.

[0163] It can be understood that the contents of the above method embodiments are all applicable to the present device embodiments, the functions specifically implemented by the present device embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0164] See also Figure 6 , Figure 6The hardware structure of an electronic device of another embodiment is illustrated, and the electronic device includes:

[0165] The processor 601 may be implemented by a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present application;

[0166] The memory 602 can be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 602 can store an operating system and other application programs. When the technical solution provided in the embodiment of this specification is implemented by software or firmware, the relevant program code is stored in the memory 602, and the processor 601 calls and executes an algorithm model performance evaluation method of the embodiment of this application;

[0167] Input / output interface 603, used to implement information input and output;

[0168] Communication interface 604, used to realize communication interaction between the device and other devices, which can be realized by wired mode (such as USB, network cable, etc.) or wireless mode (such as mobile network, WIFI, Bluetooth, etc.);

[0169] Bus 605 , which transmits information between various components of the device (e.g., processor 601 , memory 602 , input / output interface 603 , and communication interface 604 );

[0170] The processor 601 , the memory 602 , the input / output interface 603 and the communication interface 604 are connected to each other in communication within the device via a bus 605 .

[0171] An embodiment of the present application also provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, it implements the above-mentioned algorithm model performance evaluation method.

[0172] It can be understood that the contents of the above method embodiments are all applicable to the present storage medium embodiments, the functions specifically implemented by the present storage medium embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0173] The memory, as a non-transient computer-readable storage medium, can be used to store non-transient software programs and non-transient computer executable programs. In addition, the memory may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory may optionally include a memory remotely disposed relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0174] The embodiments described in the embodiments of the present application are intended to more clearly illustrate the technical solutions of the embodiments of the present application and do not constitute a limitation on the technical solutions provided in the embodiments of the present application. Those skilled in the art will appreciate that with the evolution of technology and the emergence of new application scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0175] Those skilled in the art will appreciate that the technical solutions shown in the figures do not constitute a limitation on the embodiments of the present application, and may include more or fewer steps than shown in the figures, or a combination of certain steps, or different steps.

[0176] The device embodiments described above are merely illustrative, and the units described as separate components may or may not be physically separated, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0177] Those skilled in the art will appreciate that all or some of the steps in the methods disclosed above, and the functional modules / units in the systems and devices may be implemented as software, firmware, hardware, or a suitable combination thereof.

[0178] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0179] It should be understood that in the present application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0180] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the above units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0181] The units described above as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0182] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0183] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including multiple instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, referred to as ROM), random access memory (Random Access Memory, referred to as RAM), disk or optical disk and other media that can store programs.

[0184] The preferred embodiments of the present invention are described above with reference to the accompanying drawings, but the scope of the rights of the present invention is not limited thereto. Any modification, equivalent substitution and improvement made by a person skilled in the art without departing from the scope and essence of the present invention should be within the scope of the rights of the present invention.

Claims

1. A method for evaluating algorithm model performance, characterized in that: The method comprises the following steps: Get the function list of the algorithm model; Acquire mapping scenarios corresponding to various functions according to the function list; Determine, according to the mapping scenario, each behavior feature corresponding to a plurality of scenarios and an influence weight corresponding to each behavior feature; The performance of the algorithm model is evaluated according to each of the behavioral features and the influence weight corresponding to each of the behavioral features.

2. The algorithm model performance evaluation method according to claim 1, characterized in that: The step of obtaining the function list of the algorithm model comprises the following steps: The function list is determined according to the product requirement functions of the algorithm model.

3. The algorithm model performance evaluation method according to claim 1, characterized in that: The step of obtaining the mapping scenarios corresponding to the various functions according to the function list includes the following steps: Determine the scene corresponding to each function in the function list; Determining each element associated with each of the scenes; determining each characteristic associated with each of said elements; Determining each characteristic behavior associated with each of the characteristics; A data set is constructed according to each of the characteristic behaviors as the mapping scenario.

4. The algorithm model performance evaluation method according to claim 3, characterized in that: The step of determining, according to the mapping scenario, each behavior feature corresponding to a plurality of scenarios and an influence weight corresponding to each behavior feature comprises the following steps: The influence weights of each of the scenes, each of the elements, each of the features, and each of the feature behaviors are set in sequence using the hierarchical analysis method.

5. The algorithm model performance evaluation method according to claim 1, characterized in that: The step of performing performance evaluation on the algorithm model according to each of the behavioral features and the influence weights corresponding to each of the behavioral features comprises the following steps: Calculate a first performance score corresponding to each of the behavioral features according to the influence weight of each of the behavioral features; Accumulating the first performance scores corresponding to the functions to obtain the second performance scores; Accumulate the second performance scores of the various functions to obtain a total performance score; A performance evaluation is performed on the algorithm model according to the total performance score.

6. An algorithm model performance evaluation device, characterized in that: The device comprises: A function acquisition unit, used to obtain a function list of an algorithm model; A function mapping unit, used to obtain a mapping scenario corresponding to each function according to the function list; A feature and weight determination unit, used to determine each behavior feature corresponding to a plurality of scenarios and an influence weight corresponding to each behavior feature according to the mapping scenario; A performance evaluation unit is used to perform performance evaluation on the algorithm model according to each of the behavioral features and the influence weight corresponding to each of the behavioral features.

7. The algorithm model performance evaluation device according to claim 6, characterized in that: The function mapping unit comprises: A scene association unit, used to determine the scene corresponding to each function in the function list; An element association unit, used to determine each element associated with each of the scenes; A feature association unit, used to determine each feature associated with each of the elements; A feature behavior association unit, used to determine each feature behavior associated with each of the features; The data set construction unit is used to construct a data set as the mapping scenario according to each of the characteristic behaviors.

8. The algorithm model performance evaluation device according to claim 7, characterized in that: The feature and weight determination unit comprises: The weight setting unit is used to sequentially set the influence weights of each of the scenes, each of the elements, each of the features, and each of the feature behaviors by using a hierarchical analysis method.

9. An electronic device, characterized in that: The electronic device comprises a memory and a processor, the memory stores a computer program, and the processor implements the method according to any one of claims 1 to 5 when executing the computer program.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 5 is implemented.

Images