Methods, apparatus, equipment and storage media for assessing the maturity of computer vision models

Abstract

This application discloses a method, apparatus, device, and storage medium for evaluating the maturity of computer vision models, belonging to the field of deep learning. The technical solution provided by this application includes: evaluating machine algorithm models in the field of computer vision from the dimensions of data, algorithm model, and runtime framework to obtain a first evaluation value corresponding to the data dimension, a second evaluation value corresponding to the algorithm model dimension, and a third evaluation value corresponding to the runtime framework dimension; obtaining the weight values ​​corresponding to the data dimension, algorithm model dimension, and runtime framework dimension respectively; and performing a weighted summation of the first evaluation value, second evaluation value, and third evaluation value based on the obtained weight values ​​to obtain the maturity score corresponding to the machine algorithm model. The technical solution provided by this application can evaluate machine algorithm models in the field of computer vision.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of deep learning, and in particular to a computer vision model maturity evaluation method and device, equipment and a storage medium. BACKGROUND

[0002] With the continuous development of deep learning algorithms, the technology in the field of computer vision in deep learning algorithms is also continuously landed, such as face recognition, medical image recognition, unmanned driving, intelligent robots, etc., and the corresponding industrial scale is continuously growing, therefore, how to measure the maturity of the computer vision algorithm is of great significance for the future development of computer vision algorithm.

[0003] Currently, how to evaluate the machine algorithm model in the field of computer vision has become a problem to be solved. SUMMARY

[0004] Based on this, the embodiments of the present application provide a computer vision model maturity evaluation method, device, equipment and storage medium, which can evaluate the machine algorithm model in the field of computer vision.

[0005] In a first aspect, a computer vision model maturity evaluation method is provided, the method comprising:

[0006] Evaluating the machine algorithm model in the field of computer vision from the dimensions of data, algorithm model and running framework to obtain a first evaluation value corresponding to the dimension of data, a second evaluation value corresponding to the dimension of algorithm model and a third evaluation value corresponding to the dimension of running framework; obtaining weight values respectively corresponding to the dimension of data, the dimension of algorithm model and the dimension of running framework; performing weighted summation processing on the first evaluation value, the second evaluation value and the third evaluation value according to the obtained weight values to obtain a maturity score corresponding to the machine algorithm model; wherein the data includes a training set of the machine algorithm model, the algorithm model includes a network structure and network parameters of the machine algorithm model, and the running framework includes software and hardware facilities relied on by the machine algorithm model.

[0007] In one of the embodiments, the first evaluation value includes a data authenticity measurement index value, a data specification measurement index value and a data balance measurement index value; the data authenticity measurement index value is used to indicate whether the training set has authenticity; the data specification measurement index value is used to indicate whether the training set conforms to the specification; and the data balance measurement index value is used to indicate whether the training set has representativeness.

[0008] In one of the embodiments, the second evaluation value includes a generalization capability metric value, a robustness metric value and a performance efficiency metric value; the generalization capability metric value is used to represent the prediction capability of the machine algorithm model on unknown data; the robustness metric value is used to represent the robustness of the machine algorithm model on abnormal data; and the performance efficiency metric value is used to represent the running speed of the machine algorithm model.

[0009] In one of the embodiments, the third evaluation value includes an availability metric value, a portability metric value and an information security metric value; the availability metric value is used to represent the ability of the computing resource to operate or maintain functions when needed; the portability metric value is used to represent the workload and effectiveness required for migration from the current running environment to another running environment; and the information security metric value is used to represent the degree of protection of information and data during the running of the machine algorithm model.

[0010] In one of the embodiments, the first evaluation value, the second evaluation value and the third evaluation value are weighted and summed according to the obtained weight value to obtain the maturity score corresponding to the machine algorithm model, including:

[0011] The final evaluation value corresponding to the dimension of the data is calculated according to the index value in the first evaluation value; the final evaluation value corresponding to the dimension of the algorithm model is calculated according to the index value in the second evaluation value; the final evaluation value corresponding to the dimension of the running framework is calculated according to the index value in the third evaluation value; and the final evaluation value corresponding to the dimension of the data, the final evaluation value corresponding to the dimension of the algorithm model and the final evaluation value corresponding to the dimension of the running framework are weighted and summed according to the obtained weight value to obtain the maturity score corresponding to the machine algorithm model.

[0012] In one of the embodiments, the method further includes:

[0013] The maturity score corresponding to the machine algorithm model is used to query a rating table, the rating table including a plurality of corresponding relationships between maturity scores and machine algorithm model levels; and the machine algorithm model level corresponding to the machine algorithm model is determined according to the query result.

[0014] In a second aspect, a computer vision model maturity evaluation device is provided, which includes:

[0015] An evaluation module is configured to perform evaluation processing on the machine algorithm model in the field of computer vision from the dimension of data, the dimension of algorithm model and the dimension of running framework, to obtain a first evaluation value corresponding to the dimension of data, a second evaluation value corresponding to the dimension of algorithm model and a third evaluation value corresponding to the dimension of running framework.

[0016] A first obtaining module is configured to obtain a weight value corresponding to the dimension of data, the dimension of algorithm model and the dimension of running framework respectively.

[0017] A second obtaining module is configured to perform weighted summation processing on the first evaluation value, the second evaluation value and the third evaluation value according to the obtained weight value, to obtain a maturity score corresponding to the machine algorithm model.

[0018] The data includes a training set of the machine algorithm model, the algorithm model includes a network structure and a network parameter of the machine algorithm model, and the running framework includes a software and hardware facility relied on by the machine algorithm model.

[0019] In one of the embodiments, the first evaluation value includes a data authenticity measurement indicator value, a data specification measurement indicator value and a data balance measurement indicator value; the data authenticity measurement indicator value is used to indicate whether the training set has authenticity; the data specification measurement indicator value is used to indicate whether the training set conforms to the specification; and the data balance measurement indicator value is used to indicate whether the training set has representativeness.

[0020] In one of the embodiments, the second evaluation value includes a generalization ability measurement indicator value, a robustness measurement indicator value and a performance efficiency measurement indicator value; the generalization ability measurement indicator value is used to represent the prediction ability of the machine algorithm model on unknown data; the robustness measurement indicator value is used to represent the robustness of the machine algorithm model on abnormal data; and the performance efficiency measurement indicator value is used to represent the running speed of the machine algorithm model.

[0021] In one of the embodiments, the third evaluation value includes an availability measurement indicator value, a portability measurement indicator value and an information security measurement indicator value; the availability measurement indicator value is used to represent the ability of the computing resource to operate or maintain functions when needed; the portability measurement indicator value is used to represent the workload and effectiveness required for migration from a current running environment to another running environment; and the information security measurement indicator value is used to represent the degree of protection of information and data of the machine algorithm model in the running process.

[0022] In one of the embodiments, the second obtaining module is specifically configured to:

[0023] The final evaluation value corresponding to the dimension of the data is calculated according to each index value in the first evaluation value; the final evaluation value corresponding to the dimension of the algorithm model is calculated according to each index value in the second evaluation value; the final evaluation value corresponding to the dimension of the running framework is calculated according to each index value in the third evaluation value; and the final evaluation value corresponding to the dimension of the data, the final evaluation value corresponding to the dimension of the algorithm model and the final evaluation value corresponding to the dimension of the running framework are weighted and summed according to the obtained weight value, so as to obtain the maturity score corresponding to the machine algorithm model.

[0024] In one of the embodiments, the apparatus further comprises a querying module and a determining module.

[0025] The querying module is configured to query a rating table according to the maturity score corresponding to the machine algorithm model, wherein the rating table comprises a plurality of corresponding relationships between maturity scores and machine algorithm model levels.

[0026] The determining module is configured to determine the machine algorithm model level corresponding to the machine algorithm model according to the query result.

[0027] In a third aspect, a computer device is provided, comprising a memory and a processor, the memory stores a computer program, and the processor implements the computer vision model maturity evaluation method of any one of the first aspect when executing the computer program.

[0028] In a fourth aspect, a computer readable storage medium is provided, which stores a computer program, and the computer program is executed by a processor to implement the computer vision model maturity evaluation method of any one of the first aspect.

[0029] The technical scheme provided by the embodiments of the present application has at least the following beneficial effects:

[0030] In the technical scheme provided by the embodiments of the present application, the machine algorithm model in the field of computer vision is evaluated and processed from the dimension of data, the dimension of algorithm model and the dimension of running framework, to obtain the first evaluation value corresponding to the dimension of data, the second evaluation value corresponding to the dimension of algorithm model and the third evaluation value corresponding to the dimension of running framework, obtain the weight value corresponding to the dimension of data, the dimension of algorithm model and the dimension of running framework respectively, perform weighted sum processing on the first evaluation value, the second evaluation value and the third evaluation value according to the obtained weight value, and obtain the maturity score corresponding to the machine algorithm model. The technical scheme provided by the embodiments of the present application proposes a comprehensive machine learning algorithm maturity evaluation system for the field of computer vision, to ensure that the algorithm is mature enough and further ensure the quality of the software applying the machine learning algorithm, and provides an important reference for the continuous development and landing application of the machine learning algorithm. BRIEF DESCRIPTION OF DRAWINGS

[0031] Figure 1 A flowchart of a computer vision model maturity evaluation method provided for an embodiment of the present application;

[0032] Figure 2 A flowchart of a computer vision model maturity evaluation method provided for an embodiment of the present application;

[0033] Figure 3 A flowchart of a computer vision model maturity evaluation method provided for an embodiment of the present application

[0034] Figure 4 A block diagram of a computer vision model maturity evaluation device provided for an embodiment of the present application;

[0035] Figure 5 A block diagram of a computer vision model maturity evaluation device provided for an embodiment of the present application;

[0036] Figure 6 A schematic diagram of a computer device provided for an embodiment of the present application. DETAILED DESCRIPTION

[0037] In order to make the purpose, technical scheme and advantages of the present application clearer, the embodiments of the present application will be further described in detail below with reference to the drawings.

[0038] With the continuous improvement of the precision of machine algorithm models and the continuous improvement of software and hardware infrastructure, machine algorithm models have been widely landed in the field of computer vision, such as face recognition, medical image recognition, unmanned driving, intelligent robots, etc., and the corresponding industry scale is growing.

[0039] At present, the research on machine algorithm models is limited to the level of algorithm and prospect, and the academic field only evaluates the performance of a machine algorithm model by the accuracy on the test set. Secondly, most of the software applications involved in machine algorithm models are built by experienced data scientists using open source frameworks, and few people pay attention to software engineering principles. Unlike traditional software programs that are hard-coded to solve specific tasks, machine algorithm models learn from training data containing input and corresponding output examples by building complex networks, so that traditional maturity evaluation methods and measurement indicators cannot be directly transplanted and used. In practical applications, machine algorithm models will face a series of targeted attacks, which will affect the output results of machine algorithm models, and environmental changes may also produce data noise and unexpected situations. Therefore, how to measure the maturity of machine algorithm models, especially in the field of computer vision, has great significance for the wide application of machine algorithm models in the future.

[0040] Therefore, the embodiment of the present application provides a method for evaluating the maturity of a machine algorithm model in the field of computer vision. The method can be applied to a computer vision model maturity evaluation device, which can be implemented by software, hardware or a combination of software and hardware to become part of or all of a computer device, which can be a server or a terminal. The server can be a single server or a server cluster composed of multiple servers. The terminal can be a desktop computer, a tablet computer, a notebook computer or a smart phone, and the embodiment of the present application does not limit the specific type of terminal. In the following method embodiment, the execution subject is taken as an example of a computer device.

[0041] Please refer to Figure 1 which shows a flowchart of a computer vision model maturity evaluation method provided by the embodiment of the present application. As shown in Figure 1 , the computer vision model maturity evaluation method comprises the following steps:

[0042] Step 101, the computer device performs evaluation processing on the machine algorithm model in the field of computer vision from the dimensions of data, algorithm model and running framework, to obtain a first evaluation value corresponding to the dimension of data, a second evaluation value corresponding to the dimension of algorithm model and a third evaluation value corresponding to the dimension of running framework.

[0043] Among them, the data includes the training set of the machine algorithm model, the algorithm model includes the network structure and network parameters of the machine algorithm model, and the running framework includes the software and hardware facilities on which the machine algorithm model depends.

[0044] It should be pointed out that the running framework can include a machine learning algorithm framework, computing power resources and intelligent sensors.

[0045] Step 102, the computer device obtains the weight value corresponding to the dimension of data, the dimension of algorithm model and the dimension of running framework respectively.

[0046] Step 103, the computer device performs weighted summation processing on the first evaluation value, the second evaluation value and the third evaluation value according to the obtained weight value, to obtain a maturity score corresponding to the machine algorithm model.

[0047] In the optional embodiment of the present application, the first evaluation value includes a data authenticity measurement index value, a data specification measurement index value and a data balance measurement index value.

[0048] The data authenticity measurement index value is used to indicate whether the training set is authentic, the data specification measurement index value is used to indicate whether the training set conforms to the specification, and the data balance measurement index value is used to indicate whether the training set is representative.

[0049] In the following, the application embodiment will describe the calculation method of the data authenticity measurement index value, the data specification measurement index value and the data balance measurement index value:

[0050] 1. The data authenticity measurement index value:

[0051] Authenticity refers to ensuring that the data has real meaning and conforms to the data distribution in the real application scenario.

[0052] The data authenticity measurement index value can be calculated by the following formula:

[0053]

[0054]

[0055] The data authenticity measurement index value is R d1 , TP is the number of true positives obtained by the binary classifier for the training set and the data set in the real scene, TN is the number of true negatives obtained by the binary classifier for the training set and the data set in the real scene, FP is the number of false positives obtained by the binary classifier for the training set and the data set in the real scene, and FN is the number of false negatives obtained by the binary classifier for the training set and the data set in the real scene. If the MCC coefficient is greater than 0.2, it indicates that the distribution of the training set and the data set in the real scene is inconsistent, and the data authenticity is low. If the MCC coefficient is less than or equal to 0.2, it indicates that the distribution of the training set and the data set in the real scene is consistent, and the data has authenticity.

[0056] 2. The data specification measurement index value:

[0057] Specification refers to the data in the data set conforming to the corresponding specification in the data acquisition, data labeling and other data processing processes.

[0058] A. For image classification tasks, the calculation formula of the data specification measurement index value is:

[0059]

[0060] R d2For data specification metric value, the data quality of the data set is evaluated by comparing the annotation results of two versions. Version A is the standard of data annotation, and whether version B conforms to the specification is evaluated. A (i) and C B (i) respectively represents the category of the i-th image classification annotation in the data set in version A and version B, and N is the number of images included in the data set.

[0061] B. For the target detection task, the calculation formula of the data specification metric value is:

[0062]

[0063] bbox A (i), bbox B (i) represents the bounding box of the target annotation in the i-th image in version A and version B, and IOU calculates the intersection over union of the two bounding boxes. Greater than 0.9 is in line with the specification.

[0064] C. For the image segmentation task, the calculation formula of the data specification metric value is:

[0065]

[0066] Mask A (i) and Mask B (i) represents the mask obtained in the i-th image in version A and version B. IOU calculates the intersection over union of the two masks. Greater than 0.9 is in line with the specification.

[0067] It should be noted that the final data specification metric value is a percentage value.

[0068] 3. Data balance metric value:

[0069] Balance ensures that the data in the data set is representative and avoids biased results. Balance can be defined as:

[0070]

[0071] where R d3 is the data balance metric value, and for N training set sample quantities, n i is the data quantity of the i-th sample in the training set, n min is the minimum data quantity of the training set sample, is the average value of the training set sample data quantity.

[0072] For the image classification task, the data volume refers to the data volume of the picture, for the target detection task, the data volume refers to the number of bounding boxes, and for the image segmentation task, the data volume refers to the number of masks.

[0073] In an optional embodiment of the present application, the second evaluation value includes a generalization capability metric value, a robustness metric value and a performance efficiency metric value.

[0074] The generalization capability metric value is used to represent the prediction capability of the machine algorithm model for unknown data, the robustness metric value is used to represent the robustness of the machine algorithm model for abnormal data, and the performance efficiency metric value is used to represent the running speed of the machine algorithm model.

[0075] In the following, the embodiments of the present application will describe the calculation method of the generalization capability metric value, the robustness metric value and the performance efficiency metric value:

[0076] 1. Generalization capability metric value:

[0077] The generalization capability refers to the prediction capability of the machine algorithm model for unknown data, which is the basic requirement of the machine algorithm model.

[0078] The generalization capability metric value can be calculated by F1 score:

[0079]

[0080]

[0081]

[0082] wherein, R a1 is the generalization capability metric value, Precision (precision) is the ratio of true positive TP in all positive examples identified by the machine algorithm model, and Recall (recall) is the ratio of true positive TP in all positive examples in the test set.

[0083] 2. Robustness metric value:

[0084] Robustness refers to the robustness of the machine algorithm model to abnormal data, which is the ability not to be affected.

[0085] R a2 = 1-max(ASR w , ASR b )

[0086]

[0087]

[0088] wherein R a2 is a robustness metric value, f() is a given model, N adversarial samples x w are generated for the model using a white-box attack algorithm, and N adversarial attack samples x b are generated by a black-box attack algorithm. i y w is a set value, and the adversarial success rates ASR b and ASR a3 are calculated, respectively, and the maximum value of the two is finally taken, the lower the adversarial attack success rate, the higher the robustness of the machine algorithm model, and the higher the adversarial attack success rate, the lower the robustness of the machine algorithm model.

[0089] 3. Performance efficiency metric value:

[0090] wherein the calculation formula of the performance efficiency metric value is as follows:

[0091]

[0092]

[0093] wherein R a3 is a performance efficiency metric value, and the average of multiple continuous running times is taken, from t start to t end , the machine algorithm model has processed a total of N images, the frame rate fps of the machine algorithm model is calculated, and it is generally considered that the frame rate of the machine algorithm model greater than 30 can meet the demand of real-time image processing.

[0094] In an optional embodiment of the present application, the third evaluation value includes an availability metric value, a portability metric value, and an information security metric value.

[0095] wherein the availability metric value is used to represent the ability of the computing resource to operate or maintain its function when needed, the portability metric value is used to represent the amount of work and effectiveness required for migration from the current running environment to another running environment, and the information security metric value is used to represent the degree of protection of information and data during the running of the machine algorithm model.

[0096] In the following, the present application embodiment will explain the calculation method of the availability metric value, the portability metric value, and the information security metric value:

[0097] 1. Availability metric value:

[0098] The availability refers to the ability of the computing resource to operate or maintain its function when needed.

[0099]

[0100] Among them, R f1 These are availability metrics, with MTBF being Mean Time Between Failure and MTTR being Mean Time To Repair.

[0101] 2. Portability metrics:

[0102] Portability refers to the amount of work and efficiency required to migrate from the current operating environment to another environment.

[0103]

[0104] Among them, R f2 The usability metric is the amount of work T required to port the entire algorithm while ensuring its effectiveness.

[0105] 3. Information security metrics:

[0106] Information security metrics are used to characterize the degree to which information and data are protected during the operation of machine algorithm models. Data protection methods can include differential privacy methods, homomorphic encryption methods, etc., and this application does not specifically limit these methods.

[0107]

[0108] Please refer to Figure 2 Based on the above embodiments, step 103 above may optionally include the following:

[0109] Step 201: The computer equipment calculates the final evaluation value corresponding to the dimension of the data based on the index values ​​in the first evaluation value.

[0110] The formula for calculating the final evaluation value corresponding to the dimension of the data can be:

[0111]

[0112] Among them, R d R represents the final evaluation value corresponding to the dimensions of the data. di ω is the i-th index value in the first evaluation value. di This is the weight value corresponding to the i-th indicator value in the first evaluation value.

[0113] Step 202: The computer device calculates the final evaluation value corresponding to the dimension of the algorithm model based on the index values ​​in the second evaluation value.

[0114] The formula for calculating the final evaluation value corresponding to the dimension of the algorithm model can be:

[0115]

[0116] Among them, R a R represents the final evaluation value corresponding to the dimension of the algorithm model. ai ω is the i-th index value in the second evaluation value. ai This is the weight value corresponding to the i-th indicator value in the second evaluation value.

[0117] Step 203: The computer equipment calculates the final evaluation value corresponding to the dimension of the operating framework based on the values ​​of each indicator in the third evaluation value.

[0118] The formula for calculating the final evaluation value corresponding to the dimension of the runtime framework can be:

[0119]

[0120] Among them, R f R represents the final evaluation value corresponding to the dimensions of the running framework. fi ω is the i-th indicator value in the third evaluation value. fi This is the weight value corresponding to the i-th indicator value in the third evaluation value.

[0121] Step 204: The computer device performs a weighted summation of the final evaluation values ​​corresponding to the dimensions of the data, the dimensions of the algorithm model, and the dimensions of the running framework based on the obtained weight values, to obtain the maturity score corresponding to the machine algorithm model.

[0122] The formula for the maturity score corresponding to the computer algorithm model can be:

[0123]

[0124] Where R is the maturity score corresponding to the machine algorithm model, and ω d ω represents the weight value corresponding to the final evaluation value for each dimension of the data. a ω represents the weight value corresponding to the final evaluation value of the algorithm model's dimension. f The weight values ​​corresponding to the final evaluation values ​​of the dimensions of the running framework.

[0125] Please refer to Figure 3 Based on the above embodiments, optionally, after calculating the maturity score corresponding to the machine algorithm model, the computer vision model maturity assessment method provided in this application embodiment further includes the following steps:

[0126] Step 301: The computer device queries the rating table based on the maturity score corresponding to the machine algorithm model. The rating table includes multiple correspondences between maturity scores and machine algorithm model levels.

[0127] Please refer to Table 1, which shows an exemplary rating table provided in an embodiment of this application.

[0128] Table 1

[0129] Serial number Grade Explanation 1 A 0.9 < R, the algorithm maturity is very high, the rating level is A 2 B 0.75 < R≤0.9, the algorithm maturity is higher, the rating level is B 3 C 0.5 < R≤0.75, the algorithm maturity is lower, the rating level is C 4 D 0 < R≤0.5, the algorithm maturity is very low, the rating level is D

[0130] Step 302: The computer equipment determines the machine algorithm model level corresponding to the machine algorithm model based on the query results.

[0131] Based on the same inventive concept, this application also provides a computer vision model maturity assessment device for implementing the aforementioned computer vision model maturity assessment method. The solution provided by this device is similar to the implementation described in the above method; therefore, the specific limitations of one or more computer vision model maturity assessment device embodiments provided below can be found in the limitations of the computer vision model maturity assessment method described above, and will not be repeated here.

[0132] In one embodiment, such as Figure 4 As shown, a computer vision model maturity assessment device 400 is provided, including: an assessment module 401, a first acquisition module 402 and a second acquisition module 403.

[0133] The evaluation module 401 is used to evaluate machine algorithm models in the field of computer vision from the dimensions of data, algorithm model, and operating framework to obtain a first evaluation value corresponding to the dimension of data, a second evaluation value corresponding to the dimension of algorithm model, and a third evaluation value corresponding to the dimension of operating framework.

[0134] The first acquisition module 402 is used to acquire the weight values ​​corresponding to the dimensions of the data, the dimensions of the algorithm model, and the dimensions of the running framework.

[0135] The second acquisition module 403 is used to perform weighted summation of the first evaluation value, the second evaluation value and the third evaluation value based on the acquired weight values ​​to obtain the maturity score corresponding to the machine algorithm model.

[0136] The data includes the training set of the machine algorithm model, the algorithm model includes the network structure and network parameters of the machine algorithm model, and the operating framework includes the hardware and software facilities on which the machine algorithm model depends for operation.

[0137] In an optional embodiment of this application, the first evaluation value includes a data authenticity metric, a data standardization metric, and a data balance metric; the data authenticity metric is used to indicate whether the training set is authentic; the data standardization metric is used to indicate whether the training set conforms to standards; and the data balance metric is used to indicate whether the training set is representative.

[0138] In an optional embodiment of this application, the second evaluation value includes a generalization ability metric, a robustness metric, and a performance efficiency metric; wherein the generalization ability metric is used to characterize the machine algorithm model's predictive ability for unknown data; the robustness metric is used to characterize the machine algorithm model's robustness for abnormal data; and the performance efficiency metric is used to characterize the machine algorithm model's running speed.

[0139] In an optional embodiment of this application, the third evaluation value includes an availability metric, a portability metric, and an information security metric; wherein, the availability metric is used to characterize the ability of computing resources to operate or maintain functionality when needed; the portability metric is used to characterize the workload and effectiveness required to migrate from the current operating environment to another operating environment; and the information security metric is used to characterize the degree to which information and data are protected during the operation of the machine algorithm model.

[0140] In an optional embodiment of this application, the second acquisition module 403 is specifically used for:

[0141] The final evaluation value corresponding to the dimension of the data is calculated based on the indicator values ​​in the first evaluation value; the final evaluation value corresponding to the dimension of the algorithm model is calculated based on the indicator values ​​in the second evaluation value; the final evaluation value corresponding to the dimension of the operating framework is calculated based on the indicator values ​​in the third evaluation value; the final evaluation value corresponding to the dimension of the data, the final evaluation value corresponding to the dimension of the algorithm model, and the final evaluation value corresponding to the dimension of the operating framework are weighted and summed according to the obtained weight values ​​to obtain the maturity score corresponding to the machine algorithm model.

[0142] Please refer to Figure 5 This application embodiment also provides another computer vision model maturity assessment device 500, which, in addition to the modules of the computer vision model maturity assessment device 400, also includes a query module 404 and a determination module 405.

[0143] The query module 404 is used to query a rating table based on the maturity score corresponding to the machine algorithm model. The rating table includes multiple correspondences between maturity scores and machine algorithm model levels.

[0144] The determination module 405 is used to determine the machine algorithm model level corresponding to the machine algorithm model based on the query results.

[0145] In one embodiment, a computer device is provided, which may be a server or a terminal, and its internal structure diagram may be as follows: Figure 6 As shown, the computer device includes a processor, memory, and a network interface connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and a database. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The database stores data. The network interface communicates with external computer devices via a network connection. When the computer program is executed by the processor, it implements a computer vision model maturity assessment method.

[0146] Those skilled in the art will understand that, Figure 6 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0147] In one embodiment, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to perform the following steps:

[0148] This evaluation process assesses machine learning algorithms in computer vision from the perspectives of data, algorithm model, and runtime framework. The process yields a first evaluation value corresponding to the data dimension, a second evaluation value corresponding to the algorithm model dimension, and a third evaluation value corresponding to the runtime framework dimension. Weight values ​​are then assigned to each of these dimensions. Finally, a weighted sum of these three evaluation values ​​is calculated to obtain a maturity score for the machine learning algorithm model. The data includes the training set of the machine learning algorithm model, the algorithm model includes its network structure and parameters, and the runtime framework includes the hardware and software infrastructure upon which the machine learning algorithm model depends.

[0149] In one embodiment of this application, the first evaluation value includes a data authenticity metric, a data normalization metric, and a data balance metric; the data authenticity metric is used to indicate whether the training set is authentic; the data normalization metric is used to indicate whether the training set conforms to the normalization; and the data balance metric is used to indicate whether the training set is representative.

[0150] In one embodiment of this application, the second evaluation value includes a generalization ability metric, a robustness metric, and a performance efficiency metric; wherein the generalization ability metric is used to characterize the machine algorithm model's predictive ability for unknown data; the robustness metric is used to characterize the machine algorithm model's robustness for abnormal data; and the performance efficiency metric is used to characterize the machine algorithm model's running speed.

[0151] In one embodiment of this application, the third evaluation value includes an availability metric, a portability metric, and an information security metric; wherein, the availability metric is used to characterize the ability of computing resources to operate or maintain functionality when needed; the portability metric is used to characterize the workload and effectiveness required to migrate from the current operating environment to another operating environment; and the information security metric is used to characterize the degree to which information and data are protected during the operation of the machine algorithm model.

[0152] In one embodiment of this application, when the processor executes the computer program, it further implements the following steps: calculating the final evaluation value corresponding to the dimension of the data based on each indicator value in the first evaluation value; calculating the final evaluation value corresponding to the dimension of the algorithm model based on each indicator value in the second evaluation value; calculating the final evaluation value corresponding to the dimension of the running framework based on each indicator value in the third evaluation value; and performing a weighted summation of the final evaluation value corresponding to the dimension of the data, the final evaluation value corresponding to the dimension of the algorithm model, and the final evaluation value corresponding to the dimension of the running framework based on the obtained weight values ​​to obtain the maturity score corresponding to the machine algorithm model.

[0153] In one embodiment of this application, when the processor executes the computer program, it further performs the following steps: querying a rating table based on the maturity score corresponding to the machine algorithm model, the rating table including multiple correspondences between maturity scores and machine algorithm model levels; and determining the machine algorithm model level corresponding to the machine algorithm model based on the query results.

[0154] In one embodiment of this application, a computer-readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, it performs the following steps:

[0155] This evaluation process assesses machine learning algorithms in computer vision from the perspectives of data, algorithm model, and runtime framework. The process yields a first evaluation value corresponding to the data dimension, a second evaluation value corresponding to the algorithm model dimension, and a third evaluation value corresponding to the runtime framework dimension. Weight values ​​are then assigned to each of these dimensions. Finally, a weighted sum of these three evaluation values ​​is calculated to obtain a maturity score for the machine learning algorithm model. The data includes the training set of the machine learning algorithm model, the algorithm model includes its network structure and parameters, and the runtime framework includes the hardware and software infrastructure upon which the machine learning algorithm model depends.

[0156] In one embodiment of this application, the first evaluation value includes a data authenticity metric, a data normalization metric, and a data balance metric; the data authenticity metric is used to indicate whether the training set is authentic; the data normalization metric is used to indicate whether the training set conforms to the normalization; and the data balance metric is used to indicate whether the training set is representative.

[0157] In one embodiment of this application, the second evaluation value includes a generalization ability metric, a robustness metric, and a performance efficiency metric; wherein the generalization ability metric is used to characterize the machine algorithm model's predictive ability for unknown data; the robustness metric is used to characterize the machine algorithm model's robustness for abnormal data; and the performance efficiency metric is used to characterize the machine algorithm model's running speed.

[0158] In one embodiment of this application, the third evaluation value includes an availability metric, a portability metric, and an information security metric; wherein, the availability metric is used to characterize the ability of computing resources to operate or maintain functionality when needed; the portability metric is used to characterize the workload and effectiveness required to migrate from the current operating environment to another operating environment; and the information security metric is used to characterize the degree to which information and data are protected during the operation of the machine algorithm model.

[0159] In one embodiment of this application, when the computer program is executed by the processor, it further performs the following steps: calculating the final evaluation value corresponding to the dimension of the data based on each indicator value in the first evaluation value; calculating the final evaluation value corresponding to the dimension of the algorithm model based on each indicator value in the second evaluation value; calculating the final evaluation value corresponding to the dimension of the running framework based on each indicator value in the third evaluation value; and performing a weighted summation of the final evaluation value corresponding to the dimension of the data, the final evaluation value corresponding to the dimension of the algorithm model, and the final evaluation value corresponding to the dimension of the running framework based on the obtained weight values ​​to obtain the maturity score corresponding to the machine algorithm model.

[0160] In one embodiment of this application, when the computer program is executed by the processor, it further performs the following steps: querying a rating table based on the maturity score corresponding to the machine algorithm model, the rating table including multiple correspondences between maturity scores and machine algorithm model levels; and determining the machine algorithm model level corresponding to the machine algorithm model based on the query results.

[0161] The computer-readable storage medium provided in this embodiment is similar in principle and technical effect to the method embodiment described above, and will not be repeated here.

[0162] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in M ​​forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), memory bus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0163] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0164] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this patent application should be determined by the appended claims.

Claims

1. A computer vision model maturity assessment method, comprising: The method is applied to a computer device and comprises the following steps: Evaluating a machine algorithm model in the field of computer vision from the dimensions of data, algorithm model and running framework to obtain a first evaluation value corresponding to the dimension of data, a second evaluation value corresponding to the dimension of algorithm model and a third evaluation value corresponding to the dimension of running framework; Obtaining weight values corresponding to the dimensions of data, algorithm model and running framework respectively; Performing weighted summation processing on the first evaluation value, the second evaluation value and the third evaluation value according to the obtained weight values to obtain a maturity score corresponding to the machine algorithm model; The data includes a training set of the machine algorithm model, the algorithm model includes a network structure and network parameters of the machine algorithm model, and the running framework includes software and hardware facilities relied on by the machine algorithm model for running; The first evaluation value includes a data authenticity measurement index value, a data specification measurement index value and a data balance measurement index value; the data authenticity measurement index value is used to indicate whether the training set is authentic; the data specification measurement index value is used to indicate whether the training set is in line with specifications; and the data balance measurement index value is used to indicate whether the training set is representative; The calculation formula of the data authenticity measurement index value is: wherein R d1 is the data authenticity measure indicator value, TP is the number of true positives obtained by the binary classifier for the training set and the data set under the real scenario, TN is the number of true negatives obtained by the binary classifier for the training set and the data set under the real scenario, FP is the number of false positives obtained by the binary classifier for the training set and the data set under the real scenario, and FN is the number of false negatives obtained by the binary classifier for the training set and the data set under the real scenario. For an image classification task, the calculation formula of the data specification measurement index value is: wherein R d2 is the data specification metric value, the data quality of the data set is evaluated by comparing the annotation results of two versions, version A is taken as the standard of data annotation, and whether version B conforms to the specification is evaluated; C A (i) and C B (i) respectively represent the categories of the i-th image classification annotation in the data set by version A and version B, and N is the number of images included in the data set; For a target detection task, the calculation formula of the data specification measurement index value is: wherein bbox A (i) represents the bounding box of the target in the A / B version for the i-th image in the data set, and IOU represents the intersection over union of the two bounding boxes. B (i) represents the bounding box of the target in the A / B version for the i-th image in the data set, and IOU represents the intersection over union of the two bounding boxes. For an image segmentation task, the calculation formula of the data specification measurement index value is: Mask A (i) and Mask B (i) represents the mask map obtained for the i-th image in the data set in version A, B, and IOU represents the intersection over union of the two mask maps. The calculation formula of the data balance measurement index value is: wherein R d3 is the data balance measure indicator value, for the number of samples in the N training sets, n i is the data amount of the i-th sample in the training set, n min is the minimum data amount of the samples in the training set, is the average value of the sample data amount in the training set; The second evaluation value includes a generalization ability measurement index value, a robustness measurement index value and a performance efficiency measurement index value; The generalization ability measurement index value is used to represent the prediction ability of the machine algorithm model for unknown data; The robustness measurement index value is used to represent the robustness of the machine algorithm model for abnormal data; The performance efficiency measurement index value is used to represent the running speed of the machine algorithm model; The calculation formula of the generalization ability measurement index value is as follows: wherein R a1 is the generalization capability metric value, Precision is the ratio of true positives TP in all positive examples identified by the machine algorithm model, and Recall is the ratio of true positives TP in all positive examples in the test set. The calculation formula of the robustness measurement index value is as follows: R a2 = 1 - max(ASR w , ASR b ) wherein R a2 is the robustness metric value, f() is a given model, for which N adversarial samples x w are generated using a white-box attack algorithm b , y i is a set value, and the adversarial success rates ASR w and ASR b are calculated respectively; The calculation formula of the performance efficiency measurement index value is as follows: wherein R a3 is the performance efficiency metric value, calculated by averaging over multiple runs from t start to t end , when the machine algorithm model has processed a total of N images, calculating the frame rate fps of the machine algorithm model. The third evaluation value includes an availability measurement index value, a portability measurement index value and an information security measurement index value; The availability measurement index value is used to represent the ability of a computing resource to operate or maintain functions when needed; The portability measurement index value is used to represent the amount of work and effectiveness required for migration from a current running environment to another running environment; The information security measurement index value is used to represent the degree of protection of information and data of the machine algorithm model during running; The calculation formula of the availability measurement index value is as follows: wherein R f1 is the availability metric value, MTBF is the mean time between failures, and MTTR is the mean time to repair. The calculation formula of the portability measurement index value is as follows: wherein R f2 The evaluation of whether the portability measurement criterion is met is based on how much effort T is required as a cost when the entire algorithm is ported while ensuring the effectiveness of the algorithm. The calculation formula of the information security measurement index value is as follows: wherein R f3 is the information security metric indicator value.

2. The method of claim 1, wherein, The weighting sum processing of the first evaluation value, the second evaluation value and the third evaluation value according to the obtained weight value obtains the maturity score corresponding to the machine algorithm model, and the maturity score corresponding to the machine algorithm model is obtained by the weighting sum processing of the final evaluation value corresponding to the dimension of the data, the final evaluation value corresponding to the dimension of the algorithm model and the final evaluation value corresponding to the dimension of the running framework according to the obtained weight value. The final evaluation value corresponding to the dimension of the data is calculated according to each index value in the first evaluation value; The final evaluation value corresponding to the dimension of the algorithm model is calculated according to each index value in the second evaluation value; The final evaluation value corresponding to the dimension of the running framework is calculated according to each index value in the third evaluation value; The final evaluation value corresponding to the dimension of the data, the final evaluation value corresponding to the dimension of the algorithm model and the final evaluation value corresponding to the dimension of the running framework are processed by the weighting sum processing according to the obtained weight value, and the maturity score corresponding to the machine algorithm model is obtained.

3. The method of claim 1, wherein, The method further comprises: According to the maturity score corresponding to the machine algorithm model, a rating table is queried, and the rating table comprises a plurality of corresponding relationships between maturity scores and machine algorithm model levels; According to the query result, the machine algorithm model level corresponding to the machine algorithm model is determined.

4. A computer vision model maturity assessment apparatus, comprising: Applied to a computer device, the device comprises: An evaluation module is configured to evaluate a machine algorithm model in the field of computer vision from the dimension of data, the dimension of an algorithm model and the dimension of a running framework to obtain a first evaluation value corresponding to the dimension of the data, a second evaluation value corresponding to the dimension of the algorithm model and a third evaluation value corresponding to the dimension of the running framework; A first obtaining module is configured to obtain weight values corresponding to the dimension of the data, the dimension of the algorithm model and the dimension of the running framework respectively; A second obtaining module is configured to process the first evaluation value, the second evaluation value and the third evaluation value by the weighting sum processing according to the obtained weight value to obtain the maturity score corresponding to the machine algorithm model; The data comprises a training set of the machine algorithm model, the algorithm model comprises a network structure and a network parameter of the machine algorithm model, and the running framework comprises software and hardware facilities relied on by the machine algorithm model; The first evaluation value comprises a data authenticity measurement index value, a data specification measurement index value and a data balance measurement index value; the data authenticity measurement index value is used to indicate whether the training set has authenticity; the data specification measurement index value is used to indicate whether the training set conforms to the specification; and the data balance measurement index value is used to indicate whether the training set has representativeness; The calculation formula of the data authenticity measurement index value is: wherein R d1 is the data authenticity measure indicator value, TP is the number of true positives obtained by the binary classifier for the training set and the data set under the real scenario, TN is the number of true negatives obtained by the binary classifier for the training set and the data set under the real scenario, FP is the number of false positives obtained by the binary classifier for the training set and the data set under the real scenario, and FN is the number of false negatives obtained by the binary classifier for the training set and the data set under the real scenario. For an image classification task, the calculation formula of the data specification measurement index value is: Among them, R d2 The data quality of the dataset is evaluated by comparing the annotation results of two versions, with version A used as the standard for data annotation, and version B used to evaluate whether it conforms to the specifications; C A (i) and C B (i) represent the categories labeled by version A and version B for the i-th image in the dataset, respectively, and N is the number of images included in the dataset; For a target detection task, the calculation formula of the data specification measurement index value is: wherein bbox A (i) and bbox B (i) represents the bounding box of the target labeled in the i-th image in the data set in version A and B, and IOU represents the intersection over union of two bounding boxes. For an image segmentation task, the calculation formula of the data specification measurement index value is: Mask A (i) and Mask B (i) represents the mask map obtained for the i-th image in the data set in version A, B, and IOU represents the intersection over union of two mask maps. The calculation formula of the data balance measurement index value is: wherein R d3 is the data balance measure indicator value, for the number of samples in the N training sets, n i is the data amount of the i-th sample in the training set, n min is the minimum data amount of the samples in the training set, is the average value of the sample data amount in the training set; The second evaluation value comprises a generalization ability measurement index value, a robustness measurement index value and a performance efficiency measurement index value; The generalization ability measurement index value is used to represent the prediction ability of the machine algorithm model to unknown data; The robustness metric value is used to represent the robustness of the machine algorithm model to abnormal data; The performance efficiency metric value is used to represent the running speed of the machine algorithm model; The generalization capability metric value is calculated according to the following formula: wherein R a1 is the generalization capability metric value, Precision is the ratio of true positives TP in all positive examples identified by the machine algorithm model, and Recall is the ratio of true positives TP in all positive examples in the test set. The robustness metric value is calculated according to the following formula: R a2 = 1 - max(ASR w , ASR b ) wherein R a2 is the robustness metric value, f() is a given model, for which N adversarial samples x w are generated using a white-box attack algorithm b , y i is a set value, and the adversarial success rates ASR w and ASR b are calculated respectively; The performance efficiency metric value is calculated according to the following formula: wherein R a3 is the performance efficiency metric value, which is averaged by continuously running multiple times from t start moment to t end moment, the machine algorithm model has totally processed N images, and the frame rate fps of the machine algorithm model is calculated. The third evaluation value includes an availability metric value, a portability metric value and an information security metric value; The availability metric value is used to represent the ability of the computing resource to operate or maintain functions when needed; The portability metric value is used to represent the workload and effectiveness required for migration from the current running environment to another running environment; The information security metric value is used to represent the degree of protection of information and data during the running of the machine algorithm model; The availability metric value is calculated according to the following formula: wherein R f1 is the availability metric value, MTBF is the mean time between failures, and MTTR is the mean time to repair. The portability metric value is calculated according to the following formula: wherein R f2 The evaluation of whether the portability measurement criterion is met is based on how much effort T is required as a cost when the entire algorithm is ported while ensuring the effectiveness of the algorithm. The information security metric value is calculated according to the following formula: wherein R f3 is the information security metric indicator value.

5. A computer device, comprising: A memory and a processor are included, the memory stores a computer program, and the computer program is executed by the processor to realize the computer vision model maturity evaluation method according to any one of claims 1 to 3.

6. A computer-readable storage medium, characterized in that, A computer program is stored thereon, and the computer program is executed by a processor to realize the computer vision model maturity evaluation method according to any one of claims 1 to 3.

Images