Deep learning algorithm comprehensive evaluation method and system based on game theory combination empowerment-TOPSIS

Abstract

The invention discloses a deep learning algorithm comprehensive evaluation method and system based on game theory combination weighting-TOPSIS, and the method comprises the steps: constructing a deep learning algorithm evaluation index system, calculating different evaluation index values of each candidate algorithm, constructing an evaluation matrix, and carrying out the range standardization processing according to a positive or negative index type; obtaining importance weights of four groups of indexes based on an analytic hierarchy process, an optimal sequence diagram method, an entropy weight method and a CRITIC weight method; on the basis of the four groups of index importance weights, a final index importance weight is obtained based on a game theory combination weighting method; and according to the final index importance weight, in combination with a weighted TOPSIS method, obtaining a comprehensive evaluation and selection sorting result of all candidate algorithms. According to the method, the comprehensiveness and accuracy of evaluation are improved, and a decision maker can be better supported to make scientific judgment in the deep learning algorithm selection process.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence, and in particular to a comprehensive evaluation method and system for a deep learning algorithm based on game theory combined weighting-TOPSIS. Background Art

[0002] With the rapid development of artificial intelligence technology, deep learning algorithms are increasingly used in various fields, especially in tasks such as image recognition and natural language processing. These technologies not only improve efficiency, but also create new business value for enterprises. However, with the increase in algorithm complexity and the diversification of application scenarios, the traditional single evaluation index can no longer accurately reflect the performance of the algorithm, ignoring its adaptability and robustness in different environments and tasks. In addition, the lack of standardization of existing evaluation methods makes it difficult to compare evaluation results in different scenarios, which increases the difficulty of algorithm selection and optimization. This limitation makes it difficult for R&D teams to compare the advantages and disadvantages of algorithms in actual applications, affecting decision-making and resource allocation. Therefore, it is necessary to develop an effective comprehensive evaluation method to evaluate deep learning algorithms from multiple dimensions.

[0003] The comprehensive evaluation of deep learning algorithms belongs to a multi-criteria decision-making problem. When evaluating the performance of an algorithm, it is often necessary to consider multiple performance indicators, such as correctness, robustness, and security. These indicators may be interrelated and have different importance, so a comprehensive analysis is required during the evaluation process in order to accurately judge the overall performance and applicability of the algorithm. There are currently dozens of comprehensive evaluation methods, which can be roughly divided into subjective weighting methods and objective weighting methods according to the different methods of determining the evaluation weights. Among them, the subjective weighting method relies on the knowledge and experience of experts, has high flexibility, and can capture non-quantitative factors in complex problems, but may bring subjective bias and consistency problems; the objective weighting method is based on actual data and statistical analysis, has the advantages of objectivity and data-driven, and is more suitable for processing large-scale data, but is highly dependent on data quality and availability, and may ignore domain-specific expertise. Summary of the invention

[0004] The purpose of the present invention is to propose a comprehensive evaluation method and system for deep learning algorithms based on game theory combined weighting-TOPSIS.

[0005] The technical solution to achieve the purpose of the present invention is: a comprehensive evaluation method of deep learning algorithms based on game theory combined weighting-TOPSIS, comprising the following steps:

[0006] Step 1: Build a deep learning algorithm evaluation index system, calculate different evaluation index values ​​for each candidate algorithm, build an evaluation matrix, and perform range normalization according to the positive or negative index type;

[0007] Step 2, obtain the importance weights of four groups of indicators based on the analytic hierarchy process, priority diagram method, entropy weight method and CRITIC weight method;

[0008] Step 3: Based on the importance weights of the four groups of indicators, the final indicator importance weights are obtained based on the game theory combined weighting method;

[0009] Step 4: Based on the final indicator importance weights and combined with the weighted TOPSIS method, the comprehensive evaluation and selection ranking results of all candidate algorithms are obtained.

[0010] Furthermore, in step 1, a deep learning algorithm evaluation index system is constructed, different evaluation index values ​​of each candidate algorithm are calculated, an evaluation matrix is ​​constructed, and range normalization is performed according to the positive or negative index type. The deep learning algorithm evaluation index system includes:

[0011] Algorithm correctness indicators evaluate whether the functions implemented by the deep learning algorithm meet the requirements. Classification tasks include accuracy, precision, recall, F-measure, and area under the PR / ROC curve AUC value; target detection tasks also include single-class average precision (AP), multi-class average precision (mAP), and intersection over union (IoU);

[0012] Robustness indicators evaluate the ability of deep learning models to maintain their performance level in the face of changes in input data. They can be divided into two main types: natural robustness and adversarial robustness. Natural robustness refers to the model's ability to resist unintentional data variations or noise, such as changes in the image caused by natural factors such as lighting changes, rotation, scaling, cropping, weather conditions (such as fog, snow), etc. Adversarial robustness refers to the model's ability to resist deliberately designed small perturbations (including white-box adversarial attacks and black-box adversarial attacks);

[0013] Security indicators evaluate the ability of deep learning algorithms to resist security threats, including poisoning attacks, model extraction attacks, model inversion attacks, and backdoor attacks. The security of the algorithm is measured by calculating the success rate of different attack methods in the algorithm to be evaluated.

[0014] Algorithm efficiency indicators evaluate the resource consumption and running speed of the algorithm, including the number of model parameters, memory usage, inference speed, and model throughput.

[0015] Further, step 1, constructs a deep learning algorithm evaluation index system, calculates different evaluation index values ​​of each candidate algorithm, constructs an evaluation matrix, and performs range standardization processing according to the positive or negative index type, wherein different evaluation index values ​​of each candidate algorithm are calculated, an evaluation matrix is ​​constructed, and range standardization processing is performed according to the positive or negative index type, and the specific method is:

[0016] Calculate the evaluation index value of each candidate algorithm. Assuming there are n candidate algorithms and m evaluation indexes, the evaluation matrix is ​​constructed as follows:

[0017]

[0018] In the evaluation indicators of deep learning algorithms shown in Table 1, correctness uses positive indicators, that is, the larger the indicator value, the better the correctness, and robustness and security use negative indicators, that is, the larger the indicator value, the worse the robustness and security. In terms of algorithm efficiency, the number of model parameters and memory usage are negative indicators, while model inference speed and model throughput are positive indicators. According to the positive or negative indicator type, the evaluation indicator value of the deep learning algorithm is normalized by the range, and the calculation formula is as follows:

[0019] Positive indicators:

[0020] Negative indicators:

[0021] Among them, x ij represents the value of the i-th candidate algorithm on the j-th indicator, is the maximum value of all candidate algorithms on the jth index, and It is the minimum value;

[0022] After range standardization, we get:

[0023]

[0024] Furthermore, in step 2, four groups of indicator importance weights are obtained based on the analytic hierarchy process, the priority diagram method, the entropy weight method and the CRITIC weight method. The indicator importance weights are obtained based on the analytic hierarchy process. The specific method is as follows:

[0025] The algorithm evaluation problem is decomposed into a hierarchical structure using the analytic hierarchy process, including the target layer, the criterion layer and the indicator layer.

[0026] The relative importance ratio scale of nine quartiles was used to compare the importance of each indicator pairwise, and all judgment matrices at each level were constructed;

[0027] Perform consistency check on all experts' judgment matrices. If a judgment matrix fails the consistency check, adjust the element values ​​of the judgment matrix until all judgment matrices pass the consistency check.

[0028] Calculate the geometric mean of the relative importance values ​​of all experts for any two indicators, aggregate multiple judgment matrices into one judgment matrix, take the eigenvector corresponding to the maximum eigenvalue, and normalize it to obtain the ranking weight of the relative importance of the corresponding indicators in the same layer to the indicators in the previous layer;

[0029] The composite weight of each layer's indicators to the target is calculated layer by layer from top to bottom to obtain the importance weight of the indicator layer to the target layer.

[0030] 5. The comprehensive evaluation method of deep learning algorithm based on game theory combined weighting-TOPSIS according to claim 1 is characterized in that, in step 2, four groups of indicator importance weights are obtained based on the hierarchical analysis method, the priority diagram method, the entropy weight method and the CRITIC weight method, respectively, wherein the indicator importance weights are obtained based on the priority diagram method, and the specific method is:

[0031] The importance of each indicator was scored using a five-point Likert scale, and the average score of each indicator was calculated;

[0032] A precedence diagram matrix is ​​constructed based on the average score of each indicator. The precedence diagram matrix is ​​in the form of a square matrix, with rows and columns corresponding to each evaluation indicator. If indicator i is more important than indicator j, the matrix element (i, j) is filled in as 1; if the two are equally important, fill in 0.5; if indicator j is more important than indicator i, fill in 0;

[0033] Each row in the priority diagram matrix is ​​summed up to obtain the comprehensive score of each indicator. The larger the comprehensive score, the higher the importance of the indicator. The comprehensive scores of the indicators are normalized to obtain the importance weight of each indicator.

[0034] Furthermore, in step 2, four groups of indicator importance weights are obtained based on the analytic hierarchy process, priority diagram method, entropy weight method and CRITIC weight method, among which the indicator importance weights are obtained based on the entropy weight method. The specific method is as follows:

[0035] Calculate the proportional coefficient of each candidate algorithm under each indicator, and calculate the information entropy of each indicator. If the difference of a certain indicator in different candidate algorithms is large, the information entropy of the indicator is low;

[0036] The importance weight of each evaluation indicator is determined according to the information entropy of the indicator. The calculation formula is as follows:

[0037]

[0038] where e jis the information entropy of the jth indicator, and m is the number of evaluation indicators.

[0039] Furthermore, in step 2, four groups of indicator importance weights are obtained based on the analytic hierarchy process, the priority diagram method, the entropy weight method and the CRITIC weight method. The indicator importance weights are obtained based on the CRITIC weight method. The specific method is as follows:

[0040] The importance of the indicators is comprehensively measured based on the comparison strength of the evaluation indicators and the conflict between the indicators. The comparison strength refers to the difference in the values ​​of the same indicator among the candidate algorithms, which is expressed in the form of standard deviation. The standard deviation calculation formula of the indicator is as follows:

[0041]

[0042] in, is the mean value of each indicator data. The larger the standard deviation of the indicator, the greater its fluctuation, that is, the greater the value difference between the candidate algorithms, x ij represents the value of the i-th candidate algorithm on the j-th indicator, and n is the number of candidate algorithms;

[0043] The conflict between indicators is expressed by the correlation coefficient. First, the correlation matrix between each indicator is calculated as follows:

[0044]

[0045] Then the conflict between the indicators is calculated, and the conflict degree between indicator j and the remaining indicators is defined as f j :

[0046]

[0047] Among them, r ij It is the element in the i-th row and j-th column of the correlation matrix, that is, the Pearson correlation coefficient between the i-th indicator and the j-th indicator. If there is a strong positive correlation between the indicators, it means that their conflict is small;

[0048] Define information carrying capacity C j , and normalize the information carrying capacity of the indicator to obtain its importance weight:

[0049] C j =σ j ×f j (11)

[0050]

[0051] Furthermore, in step 3, based on the importance weights of the four groups of indicators, the final indicator importance weights are obtained based on the game theory combined weighting method. The specific method is:

[0052] Minimizing the deviation between the weighted evaluation methods is modeled as a cooperative game problem, and the indicator weights of the t evaluation methods are denoted as A i (i=1,2,3,…,t), in m-dimensional Euclidean space, the point represented by the combined weight vector B and the weight vector A of each evaluation method i The sum of the squared distances of the points represented is Q i , its mathematical expression is as follows:

[0053] Q i =||BA i || 2 =(BA i )(BA i ) T (13)

[0054] The optimization goal is min Q i ,i=1,2,3,…,t, let The following equation is obtained:

[0055]

[0056] By solving the Nash equilibrium solution, we get the combined weight vector B of each indicator, and then normalize it to get the final indicator weight E.

[0057] Furthermore, in step 4, based on the final indicator importance weights, combined with the weighted TOPSIS method, the comprehensive evaluation and selection ranking results of all candidate algorithms are obtained. The specific method is as follows:

[0058] Construct positive ideal solution Z + and negative ideal solution Z - , by calculating the weighted Euclidean distance between each candidate algorithm and the positive and negative ideal solutions, where the positive and negative ideal solutions are composed of the maximum and minimum values ​​of each column in the evaluation matrix:

[0059] X + =[max{x 11 ,x 21 ,…,x n1},max{x 12 ,x 22 ,…,x n2},…,max{x 1m ,x 2m ,…}]

[0060] X - =[min{x 11 ,x 21 ,…,x n1},min{x 12 ,x 22,…,x n2},…,min{x 1m ,x 2m ,…}]

[0061] Define the i-th (i=1,2,…,n) candidate algorithm and the positive ideal solution X + The weighted Euclidean distance of :

[0062]

[0063] Similarly, define the i-th (i=1,2,…,n) candidate algorithm and the negative ideal solution X - The weighted Euclidean distance of :

[0064]

[0065] Then calculate the closeness of the i-th (i=1,2,…,n) candidate algorithm to the ideal solution, the closer to 1 the better:

[0066]

[0067] The degree of closeness between each candidate algorithm and the ideal solution is used as the comprehensive evaluation result of the algorithm. The results of the closeness ranking are sorted from large to small, which is the recommended order of multiple candidate deep learning algorithms.

[0068] A comprehensive evaluation system for deep learning algorithms based on game theory combined weighting-TOPSIS implements the comprehensive evaluation method for deep learning algorithms based on game theory combined weighting-TOPSIS, realizes comprehensive evaluation of deep learning algorithms based on game theory combined weighting-TOPSIS, and is divided into four modules, executing steps 1 to 4 respectively.

[0069] A computer device comprises a memory, a processor and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the method for comprehensive evaluation of deep learning algorithms based on game theory combined weighting-TOPSIS is implemented to achieve comprehensive evaluation of deep learning algorithms based on game theory combined weighting-TOPSIS.

[0070] A computer-readable storage medium stores a computer program. When the computer program is executed by a processor, the method for comprehensive evaluation of deep learning algorithms based on game theory combined weighting-TOPSIS is implemented to achieve comprehensive evaluation of deep learning algorithms based on game theory combined weighting-TOPSIS.

[0071] Compared with the prior art, the present invention has the following significant advantages: 1) It no longer only considers algorithm evaluation based on a single dimension, but also considers multiple dimensions such as correctness, robustness, and security. In addition, since different tasks and scenarios have different performance requirements for deep learning algorithms, the emphasis of algorithm evaluation is also different. The present invention combines expert experience and adopts a specific problem-by-specific analysis approach to assign different weights to different evaluation indicators, so that the algorithm evaluation method can be applied to different tasks and scenarios. 2) Considering that the subjective weighted evaluation method cannot explore the intrinsic connection between indicators, the present invention adopts a variety of subjective and objective weighted evaluation methods, and then draws on the idea of ​​cooperative game, effectively combining the respective advantages of subjective and objective weighted evaluation methods, ensuring that the evaluation results can reflect professional judgments without losing the objectivity of data-driven, improving the comprehensiveness and accuracy of the evaluation, and better supporting decision makers to make scientific judgments in the process of deep learning algorithm selection. BRIEF DESCRIPTION OF THE DRAWINGS

[0072] Figure 1 It is a comprehensive evaluation flow chart of deep learning algorithms based on game theory combined weighting-TOPSIS. DETAILED DESCRIPTION

[0073] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0074] One purpose of the present invention is to comprehensively evaluate the performance of deep learning algorithms, propose a series of evaluation indicators from the dimensions of algorithm correctness, robustness, security, efficiency, etc., and form a set of deep learning algorithm evaluation indicator system. Among them, the algorithm correctness indicator mainly evaluates whether the functions implemented by the deep learning algorithm meet the requirements. The classification tasks include accuracy, precision, recall, F-measure, PR / ROC curve area under the AUC value, etc.; the target detection tasks also include single-class average precision (Average-Precision, AP), multi-class average precision (meanAverage Precision, mAP), intersection over Union (IoU), etc. The robustness index mainly evaluates the ability of the algorithm to maintain its performance level in the face of natural interference and malicious attacks, including natural robustness and adversarial robustness. Natural interference mainly includes noise disturbance and image transformation, and adversarial robustness includes white-box adversarial robustness and black-box adversarial robustness. The security index mainly evaluates the algorithm's ability to resist security threats, mainly including poisoning attacks, model extraction attacks, model inversion attacks, and backdoor attacks. The security is measured by calculating the attack success rate of different attack methods in the algorithm to be evaluated. The algorithm efficiency index mainly evaluates the resource consumption and running speed of the algorithm, including the number of model parameters, memory usage, inference speed, and model throughput. Through the above analysis, the deep learning algorithm evaluation index system shown in Table 1 is formed.

[0075] Table 1 Evaluation index system of deep learning algorithm

[0076]

[0077]

[0078] Another purpose of the present invention is to integrate the advantages of subjective and objective weighted evaluation methods and achieve coordination and balance between the two, introduce the idea of ​​game theory, and propose a comprehensive evaluation method based on game theory combined weighting, so as to find a reasonable balance point between multiple weighting methods, so that the conflicts between different weighting evaluation methods can be effectively reconciled. The premise of optimizing the algorithm evaluation process by using the game method is to realize the subjective weighting method and the objective weighting method. In terms of the subjective weighting method, the present invention adopts the hierarchical analysis method and the priority diagram method; in terms of the objective weighting evaluation method, the present invention adopts the entropy weight method and the CRITIC weight method. Four different groups of indicator weighting schemes can be obtained by solving the above methods. Then, the idea of ​​game theory is introduced, and the four weighting schemes are used as the participants of the game. By minimizing the deviation between different weighting schemes, a comprehensive balance of multiple weighting methods is achieved. Subsequently, based on the indicator importance weights obtained by the game theory combined weighting method, the TOPSIS method is used to reflect the gap between the algorithms, and a comprehensive ranking and selection of multiple candidate algorithms is achieved.

[0079] Based on the above purpose, the present invention provides a comprehensive evaluation method for deep learning algorithms based on game theory combined weighting-TOPSIS, comprising the following steps:

[0080] Step 1: Calculate the evaluation index value of each candidate algorithm based on the deep learning algorithm evaluation index system shown in Table 1. Assuming there are n candidate algorithms and m evaluation indexes, the evaluation matrix is ​​as follows:

[0081]

[0082] According to the positive or negative indicator type, the range standardization is performed, and the calculation formula is as follows:

[0083] Positive indicators:

[0084] Negative indicators:

[0085] Among them, x ij represents the value of the i-th candidate algorithm on the j-th indicator, is the maximum value of all candidate algorithms on the jth index, and Then it is the minimum value. The original evaluation matrix is ​​obtained after the range standardization:

[0086]

[0087] Step 2, invite multiple experts to conduct independent scoring, and obtain the importance weights of the indicators based on the hierarchical analysis method. The hierarchical analysis method is a subjective assignment evaluation method that decomposes the original problem into a hierarchical structure. This method requires inviting multiple experts and asking them to use a nine-point relative importance ratio scale to compare the importance of each indicator in pairs to obtain a judgment matrix. Subsequently, the judgment matrices given by all experts are subjected to a consistency test. This process may require multiple adjustments until all judgment matrices pass the consistency test. The hierarchical analysis method used in the present invention does not rely on the experience judgment of a certain expert, but integrates the experience of multiple experts, and uses the geometric mean method to summarize the judgment matrices of multiple experts into one judgment matrix. Subsequently, the eigenvector corresponding to the maximum eigenvalue of the judgment matrix is ​​taken, and it is normalized to obtain the ranking weight of the relative importance of the corresponding indicator at the same level to the indicator at the previous level. Finally, the composite weight of the indicators at each level to the target is calculated layer by layer from top to bottom to obtain the importance weight of the indicator layer to the target layer. Specific details are as follows:

[0088] The analytic hierarchy process decomposes the algorithm evaluation problem into a hierarchical structure, as shown in Table 2, which is divided into the target layer, the criterion layer, the indicator layer, and the solution layer consists of all candidate algorithms. Subsequently, multiple experts are invited to compare and judge the factors of the next level with those of the previous level, and construct all judgment matrices in each level. Considering that it is difficult for experts to directly weight the indicators, the present invention adopts the relative importance ratio scale of the nine quartiles shown in Table 1 to form a judgment matrix. Among them, the element a in the matrix ij The meaning of is the importance of the i-th indicator relative to the j-th indicator.

[0089] Table 2 Relative importance scale of the quintiles

[0090] Index i is better than index j Quantization value Equally important 1 Slightly important 3 Strong and important 5 Strongly important 7 Extremely important 9 The middle value of two adjacent judgments 2，4，6，8 reciprocal <![CDATA[a ij =1 / a ji ]]>

[0091] For the elements on each layer of the hierarchical structure, a series of judgment matrices can be established by comparing them with the elements on the previous layer. A judgment matrix is ​​usually a square matrix, and the elements in the matrix represent the relative importance between the corresponding row indicators and column indicators. In actual operation, due to the complexity of objective things and the ambiguity of people's judgment and comparison of things, it is difficult to construct a completely consistent judgment matrix. For this reason, it is necessary to conduct a consistency test on the judgment matrices given by all experts. Consistency index λmax is the maximum eigenvalue of the judgment matrix, n is the order of the matrix. The random consistency index RI is a constant obtained by looking up the table according to the matrix order n (see Table 3).

[0092] Table 3 Random consistency index table

[0093]

[0094] The consistency ratio is defined as CR = CI / RI. If CR < 0.1, the judgment matrix passes the consistency test, otherwise the element values ​​of the judgment matrix need to be adjusted until all experts' judgment matrices pass the consistency test. Calculate the geometric mean of the relative importance values ​​of all experts for any two indicators, and summarize multiple judgment matrices into one judgment matrix. For example, 3 experts independently gave judgment matrices for 5 evaluation indicators (all passed the consistency test):

[0095]

[0096] The above three judgment matrices can be summarized by geometric mean to obtain the following judgment matrix (retain two decimal places), that is, the elements h at the corresponding positions of all judgment matrices ij Multiply them and find the cube root of the product (3 is the number of judgment matrices).

[0097]

[0098] Take the eigenvector corresponding to the maximum eigenvalue λmax=5.14 [0.89 0.39 0.19 0.11 0.07], and normalize it to get [0.54 0.24 0.12 0.06 0.04], which is the ranking weight of the relative importance of the corresponding indicators of the same layer to the indicators of the previous layer. Finally, calculate the composite weight of the indicators of each layer to the target layer from top to bottom, and get the importance weight of the indicator layer to the target layer. For the convenience of explanation, the above weight vector is directly used as the indicator importance weight obtained by the hierarchical analysis method.

[0099] Step 3, invite multiple experts to independently score, and obtain the indicator importance weights based on the priority diagram method. The priority diagram method obtains the subjective preference information of the importance of each indicator through expert scoring, and then calculates the average score of each indicator, and eliminates individual differences by integrating the preferences of multiple experts. Then, a priority diagram matrix is ​​constructed based on the average score of each indicator, and the priority diagram matrix is ​​used to determine the importance order and weight of each indicator. The details are as follows:

[0100] The priority diagram method uses a matrix form to compare the relative importance of each evaluation indicator to the target. To this end, experts in related fields are first invited to score the importance of each indicator using the Likert five-point scale. Each expert will score independently, and finally an importance score table as shown below is obtained.

[0101] Table 4 Quantification of the importance of each indicator using the Likert five-level scale

[0102] Expert No. Index 1 Importance Index 2 Importance Index 3 Importance Index 4 Importance Index 5 Importance Expert 1 5 4 3 2 1 Expert 2 5 5 3 3 1 Expert 3 4 4 3 2 1

[0103] The average of the values ​​in the same column of the table is the average importance score of the indicator among all experts. The average importance score vector obtained in the above table is [4.67 4.33 3 1.67 1]. By integrating the preferences of multiple experts and eliminating individual differences, a more representative indicator importance evaluation is obtained. Based on this, a priority graph matrix is ​​constructed. The priority graph matrix is ​​in the form of a square matrix, with rows and columns corresponding to each evaluation indicator. If indicator i is more important than indicator j, the matrix element (i, j) is filled in with 1; if the two are equally important, fill in 0.5; if indicator j is more important than indicator i, fill in 0. The priority graph matrix in this example is shown in Table 5:

[0104] Table 5 Priority diagram matrix

[0105] Comparison Items Index 1 Importance Index 2 Importance Index 3 Importance Index 4 Importance Index 5 Importance Index 1 Importance 0.5 1 1 1 1 Index 2 Importance 0 0.5 1 1 1 Index 3 Importance 0 0 0.5 1 1 Index 4 Importance 0 0 0 0.5 1 Index 5 Importance 0 0 0 0 0.5

[0106] Add up each row in the table to get the comprehensive score of each indicator. The larger the comprehensive score, the more important the indicator is. Normalize the score to get the importance weight of each indicator. The indicator importance weight list is as follows:

[0107] Table 6 Indicator importance weights calculated from the priority diagram matrix

[0108] index Indicator 1 Indicator 2 Indicator 3 Indicator 4 Indicator 5 Index score 4.5 3.5 2.5 1.5 0.5 Indicator weight 0.36 0.28 0.2 0.12 0.04

[0109] Step 4, based on the entropy weight method, calculate the information entropy of each indicator to obtain the importance weight of the indicator. The entropy weight method is based on the concept of information entropy and uses information entropy to reflect the distribution of an indicator. If the difference between an indicator in different evaluation objects is large, the information entropy of the indicator is low, and its impact on the overall evaluation result is greater. Therefore, the entropy weight method calculates the information entropy of each indicator and distributes the importance weight of the indicator. The details are as follows:

[0110] The entropy weight method measures the degree of dispersion of the indicator value by calculating the information entropy. The larger the information entropy, the more evenly the distribution of the indicator is, and the smaller the contribution to the decision-making is. The importance weight of each indicator in the evaluation is determined by calculating the information entropy of each indicator. Assuming there are 10 candidate algorithms and 5 evaluation indicators, the evaluation matrix after range normalization is as follows:

[0111]

[0112]

[0113] The entropy weight method first calculates the proportional coefficient of each candidate algorithm under each indicator:

[0114]

[0115] Among them, x ijIt represents the element in the i-th row and j-th column of the evaluation matrix, that is, the value of the i-th candidate algorithm on the j-th indicator, and n is the number of candidate algorithms.

[0116] The information entropy formula is then used to calculate the information entropy of each indicator:

[0117]

[0118] Among them, e j is the information entropy of the jth indicator, and n is the number of candidate algorithms. ij = 0, define p ij ln(p ij )=0. Taking the above matrix as an example, the information entropy of each indicator is [0.85 0.87 0.88 0.93 0.89].

[0119] The importance weight of each evaluation indicator is determined according to the information entropy of the indicator. The calculation formula is as follows:

[0120]

[0121] The importance weight vector determined by the above indicator information entropy vector is [0.26 0.22 0.20 0.12 0.20].

[0122] Step 5, obtain the importance weight of the indicator based on the CRITIC weight method. The CRITIC weight method comprehensively measures the importance of the indicator based on the contrast strength of the evaluation indicator and the conflict between the indicators. Among them, the contrast strength refers to the size of the difference between the values ​​of each evaluated object of the same indicator, expressed in the form of standard deviation; the conflict between indicators is expressed by the correlation coefficient. By defining the information carrying capacity of the indicator, that is, the product of the contrast strength and the conflict degree, and then normalizing the information carrying capacity of the indicator to obtain its importance weight. The details are as follows:

[0123] The CRITIC weighting method is also an objective weighting evaluation method. It comprehensively measures the importance of indicators based on the comparative strength of evaluation indicators and the conflict between indicators. The comparative strength refers to the size of the difference between the values ​​of the same indicator and the objects to be evaluated, which is expressed in the form of standard deviation. The standard deviation calculation formula of the indicator is as follows:

[0124]

[0125] in, is the mean value of each indicator (column). The larger the standard deviation of an indicator, the greater its fluctuation, that is, the greater the value difference between objects, and the higher the importance weight of the indicator. Taking the above evaluation matrix as an example, the standard deviation vector composed of the standard deviations of the five indicators is [0.32, 0.38, 0.34, 0.27, 0.34].

[0126] The conflict between indicators is expressed by the correlation coefficient. First, the correlation matrix between each indicator is calculated as follows:

[0127]

[0128] Then the conflict between the indicators is calculated, and the conflict degree between indicator j and the remaining indicators is defined as f j :

[0129]

[0130] Among them, r ij is the element in the i-th row and j-th column of the correlation matrix, that is, the Pearson correlation coefficient between the i-th indicator and the j-th indicator. If there is a strong positive correlation between the indicators, it means that their conflict is small and they should be given a lower weight. In this implementation case, the conflict degrees of the five indicators are 2.79, 3.24, 3.00, 3.27, and 5.47 respectively.

[0131] Define information carrying capacity C j , and normalize the information carrying capacity of the indicator to obtain its importance weight:

[0132] C j =σ j ×f j (11)

[0133]

[0134] Based on the above comparison intensity and conflict degree results, we further calculated the information carrying capacity of these five indicators. Finally, we normalized the information carrying capacity results of all indicators and obtained the indicator weights as [0.15 0.21 0.17 0.150.32].

[0135] Step 6. Based on the importance weights of the above four groups of indicators, the final indicator importance weights are calculated based on the game theory combined weighting method to achieve a comprehensive balance of multiple evaluation methods. The game theory combined weighting method introduces the idea of ​​game theory, and models the minimization of the deviation between each weighted evaluation method as a cooperative game problem. Then, by solving the cooperative game problem, the comprehensive balance of multiple weighted evaluation methods is achieved. The core idea of ​​the game theory combined weighting method is to take multiple weighted evaluation methods as participants in the game. The process of interaction and mutual game between these evaluation methods can adopt an idea similar to Nash equilibrium, and find the balance point between each weighted method through multiple rounds of games. The weight combination of Nash equilibrium points can ensure that the advantages of different weighted evaluation methods are retained while minimizing the deviation between each scheme. After the game process is over, the combined weight of each indicator is obtained. Specifically as follows:

[0136] The game theory combined weighting method models the deviation minimization problem between different weighting schemes as a cooperative game problem, and achieves the trade-off of multiple weighting evaluation methods by solving the Nash equilibrium solution. Here, we denote the indicator weights of t evaluation methods as A i (i=1,2,3,…,t), in m-dimensional Euclidean space, the point represented by the combined weight vector B and the weight vector A of each evaluation method i The sum of the squared distances of the points represented is Q i , its mathematical expression is as follows:

[0137]

[0138] The optimization goal is min Q i ,i=1,2,3,…,t, let The following equation can be obtained:

[0139]

[0140] Solving this equation yields C, At this time, B=A i , but there is no guarantee that the sum of squared deviations between B and other rows of A is minimal. i The sum of squared deviations between As small as possible, the present invention uses the idea of ​​multi-objective game to find all min Q i , i = 1, 2, 3, ..., t. To this end, the present invention regards each evaluation method as a game participant, each participant has t choices, and the choice of participant j is d j represents the vacancy selection corresponding to the jth equation in (14), d jOnly from the i-th choice of t participants, the more vacancies filled by participant j, the smaller the loss, and all participants hope to minimize their own losses. Assume that t vacancies constitute D = [d1, d2, d3, ..., d t ] T , for the kth result (k = 1, 2, 3, ..., t!), the following equation is satisfied:

[0141]

[0142]

[0143] B k =C k A=[(AA T ) -1 D k ] T A (17)

[0144] In the calculation process, the goal of the game theory combined weighting method is to find a solution that can make all Q i As small as possible, all components of B need to satisfy the non-negative condition, then B is the optimal solution of the multi-objective game. After obtaining the combined weight vector B, normalize it to obtain the final indicator weight E.

[0145] The indicator weight matrix obtained by the above four weighted evaluation methods is as follows:

[0146]

[0147] According to the calculation process of game theory combination weighting, the weighted vector B = [0.135 0.454 0.262 0.280 0.057] is obtained. Each element in this vector is non-negative. At this time, B and all A i The total square of the deviations between them is L = 0.664. Then the combined weight vector B is normalized to obtain the combined weight index weight E = [0.1140.3820.2210.2350.048].

[0148] Step 7, using the importance weight vector of each indicator, combined with the weighted TOPSIS method to obtain the comprehensive evaluation and selection ranking results of all candidate algorithms. The TOPSIS method is also called the approximate ideal solution ranking method or the superior and inferior solution distance method. This method can make full use of the information of the original data, and its results can accurately reflect the gap between the evaluation schemes. In order to give a ranking for a variety of deep learning algorithms, after obtaining the evaluation index data of all candidate algorithms, an ideal optimal solution and the worst solution in a system composed of all candidate algorithms can be constructed. The TOPSIS method will evaluate the comprehensive distance of each candidate algorithm from the ideal optimal solution and the worst solution. If an algorithm is closer to the ideal optimal solution and farther from the worst solution, the algorithm is better. In addition, the weighted TOPSIS method used in the present invention no longer assumes that all evaluation indicators are of the same importance, but changes the Euclidean distance calculation to the weighted Euclidean distance calculation, and the weights are obtained by the subjective and objective combined weighted evaluation method. The weights are as follows:

[0149] The TOPSIS method constructs a positive ideal solution Z + and negative ideal solution Z - , the evaluation is performed by calculating the weighted Euclidean distance between each candidate algorithm and the positive and negative ideal solutions. The positive and negative ideal solutions are composed of the maximum and minimum values ​​of each column in the matrix of formula (1):

[0150] X + =[max{x 11 ,x 21 ,…,x n1},max{x 12 ,x 22 ,…,x n2},…,max{x 1m ,x 2m ,…}]

[0151] X - =[min{x 11 ,x 21 ,…,x n1},min{x 12 ,x 22 ,…,x n2},…,min{x 1m ,x 2m ,…}]

[0152] Define the i-th (i=1,2,…,n) candidate algorithm and the positive ideal solution X + The weighted Euclidean distance of :

[0153]

[0154] Similarly, define the i-th (i=1,2,…,n) candidate algorithm and the negative ideal solution X- The weighted Euclidean distance of :

[0155]

[0156] Then calculate the closeness of the i-th (i=1,2,…,n) candidate algorithm to the ideal solution, the closer to 1 the better:

[0157]

[0158] The degree of closeness between each candidate algorithm and the ideal solution is used as the comprehensive evaluation result of the algorithm, and the result of the closeness sorting (from high to low) is the recommended order of multiple candidate deep learning algorithms.

[0159] Each weighted evaluation method has its own weight vector. Through the above game modeling, the multi-index comprehensive evaluation problem of deep learning algorithms is transformed into a cooperative game problem, that is, through cooperative games, the deviation between the combined weight vector and the index weight vectors obtained by various evaluation methods is minimized. The process of interaction and mutual game between subjective and objective weighting methods adopts the idea of ​​Nash equilibrium, and through multiple rounds of games, the balance point between each weighting method is found. The weight combination of Nash equilibrium points can ensure that the advantages of different weighted evaluation methods are retained while minimizing the deviation between the evaluation methods. After the game process is over, the final combined weight can take into account the rationality of both subjective and objective weighting.

[0160] The present invention also proposes a comprehensive evaluation system for deep learning algorithms based on game theory combined weighting-TOPSIS, implements the comprehensive evaluation method for deep learning algorithms based on game theory combined weighting-TOPSIS, realizes the comprehensive evaluation of deep learning algorithms based on game theory combined weighting-TOPSIS, and is divided into four modules, executing steps 1 to 4 respectively.

[0161] A computer device comprises a memory, a processor and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the method for comprehensive evaluation of deep learning algorithms based on game theory combined weighting-TOPSIS is implemented to achieve comprehensive evaluation of deep learning algorithms based on game theory combined weighting-TOPSIS.

[0162] A computer-readable storage medium stores a computer program. When the computer program is executed by a processor, the method for comprehensive evaluation of deep learning algorithms based on game theory combined weighting-TOPSIS is implemented to achieve comprehensive evaluation of deep learning algorithms based on game theory combined weighting-TOPSIS.

[0163] In summary, the present invention combines subjective weighting methods and objective weighting methods, so that it can adaptively adjust the weights of indicators according to the needs of specific tasks, and is used for comprehensive evaluation of deep learning algorithms. For example, in some tasks, the interpretability of the algorithm is more important, while in other tasks, accuracy may be the primary consideration. By combining these two methods, the weights of each indicator can be dynamically adjusted to ensure that the evaluation results can accurately reflect the performance of the deep learning algorithm in a specific scenario. In addition, this combination allows preliminary weights to be provided through objective analysis when there is sufficient data, and then fine-tuned through expert experience to balance data-driven and professional judgment. Such flexibility not only enables the algorithm evaluation method to adapt to different application scenarios, but also improves its effect in practical applications, provides a more scientific and reliable basis for decision makers, and can promote the optimal allocation of resources and the continuous progress of deep learning technology.

[0164] The technical features of the above embodiments may be arbitrarily combined. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0165] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A comprehensive evaluation method for deep learning algorithms based on game theory combined weighting-TOPSIS, characterized in that: The following steps are involved: Step 1: Build a deep learning algorithm evaluation index system, calculate different evaluation index values ​​for each candidate algorithm, build an evaluation matrix, and perform range normalization according to the positive or negative index type; Step 2, obtain the importance weights of four groups of indicators based on the analytic hierarchy process, priority diagram method, entropy weight method and CRITIC weight method; Step 3: Based on the importance weights of the four groups of indicators, the final indicator importance weights are obtained based on the game theory combined weighting method; Step 4: Based on the final indicator importance weights and combined with the weighted TOPSIS method, the comprehensive evaluation and selection ranking results of all candidate algorithms are obtained.

2. The comprehensive evaluation method of deep learning algorithm based on game theory combined weighting-TOPSIS according to claim 1 is characterized in that: Step 1: Construct a deep learning algorithm evaluation index system, calculate different evaluation index values ​​for each candidate algorithm, construct an evaluation matrix, and perform range normalization according to the positive or negative index type. The deep learning algorithm evaluation index system includes: Algorithm correctness indicators evaluate whether the functions implemented by the deep learning algorithm meet the requirements. Classification tasks include accuracy, precision, recall, F-measure, and area under the PR / ROC curve AUC value; target detection tasks also include single-class average precision (AP), multi-class average precision (mAP), and intersection over union (IoU); Robustness indicators evaluate the ability of deep learning algorithms to maintain their performance levels when facing changes in input data. There are two main types: natural robustness and adversarial robustness. Natural robustness refers to the ability to resist unintentional data variations or noise, including changes in lighting, rotation, scaling, cropping, and changes in weather conditions; adversarial robustness refers to the ability to resist deliberately designed small perturbations, including white-box adversarial attacks and black-box adversarial attacks. Security indicators evaluate the ability of deep learning algorithms to resist security threats, including poisoning attacks, model extraction attacks, model inversion attacks, and backdoor attacks. The security of the algorithm is measured by calculating the success rate of different attack methods in the algorithm to be evaluated. Algorithm efficiency indicators evaluate the resource consumption and running speed of the algorithm, including the number of model parameters, memory usage, inference speed, and model throughput.

3. The comprehensive evaluation method of deep learning algorithm based on game theory combined weighting-TOPSIS according to claim 1 is characterized in that: Step 1: construct a deep learning algorithm evaluation index system, calculate different evaluation index values ​​for each candidate algorithm, construct an evaluation matrix, and perform range standardization according to the positive or negative index type. The different evaluation index values ​​for each candidate algorithm are calculated, the evaluation matrix is ​​constructed, and the range standardization is performed according to the positive or negative index type. The specific method is as follows: Calculate the evaluation index value of each candidate algorithm. Assuming there are n candidate algorithms and m evaluation indexes, the evaluation matrix is ​​constructed as follows: According to the positive or negative indicator type, the evaluation index values ​​of the deep learning algorithm are normalized by range. The algorithm correctness index adopts positive indicators, that is, the larger the index value, the better the correctness. The robustness index and security index adopt negative indicators, that is, the larger the index value, the worse the robustness or security. Among the algorithm efficiency indicators, the model parameter quantity and memory usage are negative indicators, while the model inference speed and model throughput are positive indicators. Perform range standardization processing, and the calculation formula is as follows: Positive indicators: Negative indicators: Among them, x ij represents the value of the i-th candidate algorithm on the j-th indicator, is the maximum value of all candidate algorithms on the jth index, and is the minimum value, After range standardization, we get:

4. The comprehensive evaluation method of deep learning algorithm based on game theory combined weighting-TOPSIS according to claim 1 is characterized in that: Step 2: Based on the analytic hierarchy process, priority diagram method, entropy weight method and CRITIC weight method, four groups of indicator importance weights are obtained. The indicator importance weights are obtained based on the analytic hierarchy process. The specific method is as follows: The algorithm evaluation problem is decomposed into a hierarchical structure using the analytic hierarchy process, including the target layer, the criterion layer and the indicator layer. The relative importance ratio scale of nine quartiles was used to compare the importance of each indicator pairwise, and all judgment matrices at each level were constructed; Perform consistency check on all experts' judgment matrices. If a judgment matrix fails the consistency check, adjust the element values ​​of the judgment matrix until all judgment matrices pass the consistency check. Calculate the geometric mean of the relative importance values ​​of all experts for any two indicators, aggregate multiple judgment matrices into one judgment matrix, take the eigenvector corresponding to the maximum eigenvalue, and normalize it to obtain the ranking weight of the relative importance of the corresponding indicators in the same layer to the indicators in the previous layer; The composite weight of each layer's indicators to the target is calculated layer by layer from top to bottom to obtain the importance weight of the indicator layer to the target layer.

5. The comprehensive evaluation method of deep learning algorithm based on game theory combined weighting-TOPSIS according to claim 1 is characterized in that: Step 2: Based on the analytic hierarchy process, the priority diagram method, the entropy weight method and the CRITIC weight method, four groups of indicator importance weights are obtained. The indicator importance weights are obtained based on the priority diagram method. The specific method is as follows: The importance of each indicator was scored using a five-point Likert scale, and the average score of each indicator was calculated; A precedence diagram matrix is ​​constructed based on the average score of each indicator. The precedence diagram matrix is ​​in the form of a square matrix, with rows and columns corresponding to each evaluation indicator. If indicator i is more important than indicator j, the matrix element (i, j) is filled in as 1; If both are equally important, fill in 0.5; if indicator j is more important than indicator i, fill in 0; Each row in the priority diagram matrix is ​​summed up to obtain the comprehensive score of each indicator. The larger the comprehensive score, the higher the importance of the indicator. The comprehensive scores of the indicators are normalized to obtain the importance weight of each indicator.

6. The comprehensive evaluation method of deep learning algorithm based on game theory combined weighting-TOPSIS according to claim 1 is characterized in that: Step 2: Based on the analytic hierarchy process, priority diagram method, entropy weight method and CRITIC weight method, four groups of indicator importance weights are obtained. The entropy weight method is used to obtain the indicator importance weights. The specific method is as follows: Calculate the proportional coefficient of each candidate algorithm under each indicator, and calculate the information entropy of each indicator. If the difference of a certain indicator in different candidate algorithms is large, the information entropy of the indicator is low; The importance weight of each evaluation indicator is determined according to the information entropy of the indicator. The calculation formula is as follows: where e j is the information entropy of the jth indicator, and m is the number of evaluation indicators.

7. The comprehensive evaluation method of deep learning algorithm based on game theory combined weighting-TOPSIS according to claim 1 is characterized in that: Step 2: Based on the analytic hierarchy process, priority diagram method, entropy weight method and CRITIC weight method, four groups of indicator importance weights are obtained. The indicator importance weights are obtained based on the CRITIC weight method. The specific method is as follows: The importance of the indicators is comprehensively measured based on the comparison strength of the evaluation indicators and the conflict between the indicators. The comparison strength refers to the difference in the values ​​of the same indicator among the candidate algorithms, which is expressed in the form of standard deviation. The standard deviation calculation formula of the indicator is as follows: in, is the mean value of each indicator data. The larger the standard deviation of the indicator, the greater its fluctuation, that is, the greater the value difference between the candidate algorithms, x ij represents the value of the i-th candidate algorithm on the j-th indicator, and n is the number of candidate algorithms; The conflict between indicators is expressed by the correlation coefficient. First, the correlation matrix between each indicator is calculated as follows: Then the conflict between the indicators is calculated, and the conflict degree between indicator j and the remaining indicators is defined as f j : Among them, r ij It is the element in the i-th row and j-th column of the correlation matrix, that is, the Pearson correlation coefficient between the i-th indicator and the j-th indicator. If there is a strong positive correlation between the indicators, it means that their conflict is small; Define information carrying capacity C j , and normalize the information carrying capacity of the indicator to obtain its importance weight: C j =s j ×f j (11) 8. The comprehensive evaluation method of deep learning algorithm based on game theory combined weighting-TOPSIS according to claim 1 is characterized in that: Step 3: Based on the importance weights of the four groups of indicators, the final indicator importance weights are obtained based on the game theory combined weighting method. The specific method is: Minimizing the deviation between the weighted evaluation methods is modeled as a cooperative game problem, and the indicator weights of the t evaluation methods are denoted as A i (i=1,2,3,…,t), in m-dimensional Euclidean space, the point represented by the combined weight vector B and the weight vector A of each evaluation method i The sum of the squared distances of the points represented is Q i , its mathematical expression is as follows: Q i =||B-A i || 2 =(B-A i )(B-A i ) T (13) The optimization goal is min Q i ,i=1,2,3,…,t, let The following equation is obtained: By solving the Nash equilibrium solution, we get the combined weight vector t of each indicator, and then normalize it to get the final indicator weight E.

9. The comprehensive evaluation method of deep learning algorithm based on game theory combined weighting-TOPSIS according to claim 1 is characterized in that: Step 4: Based on the final indicator importance weights, combined with the weighted TOPSIS method, obtain the comprehensive evaluation and selection ranking results of all candidate algorithms. The specific method is as follows: Construct positive ideal solution Z + and negative ideal solution Z - , by calculating the weighted Euclidean distance between each candidate algorithm and the positive and negative ideal solutions, where the positive and negative ideal solutions are composed of the maximum and minimum values ​​of each column in the evaluation matrix: X + =[max{x 11 ,x 21 ,…,x n1 },max{x 12 ,x 22 ,…,x n2 },…,max{x 1m ,x 2m ,…}] X - =[min{x 11 ,x 21 ,…,x n1 },min{x 12 ,x 22 ,…,x n2 },…,min{x 1m ,x 2m ,…}] Define the i-th (i=1,2,…,n) candidate algorithm and the positive ideal solution X + The weighted Euclidean distance of : Similarly, define the i-th (i=1,2,…,n) candidate algorithm and the negative ideal solution X - The weighted Euclidean distance of : Then calculate the closeness of the i-th (i=1,2,…,n) candidate algorithm to the ideal solution, the closer to 1 the better: The degree of closeness between each candidate algorithm and the ideal solution is used as the comprehensive evaluation result of the algorithm. The results of the closeness ranking are sorted from large to small, which is the recommended order of multiple candidate deep learning algorithms.

10. A comprehensive evaluation system for deep learning algorithms based on game theory combined weighting-TOPSIS, characterized in that: Based on the comprehensive evaluation method of deep learning algorithm based on game theory combined weighting-TOPSIS as described in any one of claims 1 to 9, a comprehensive evaluation of deep learning algorithm based on game theory combined weighting-TOPSIS is implemented, which is divided into four modules and executes steps 1 to 4 respectively.

Images