Data-driven intelligent algorithm capability boundary construction method and device

Abstract

The invention relates to a data-driven intelligent algorithm capability boundary construction method and device, and belongs to the technical field of intelligent algorithm evaluation. The method comprises the following steps: acquiring a static attribute and a dynamic attribute which influence the capacity of an intelligent algorithm; after the influence score of the static attribute, the influence score of the dynamic attribute and the intelligent algorithm capability score of the intelligent algorithm are calculated respectively, triple data are formed, and the triple data are labeled according to the intelligent algorithm capability score; and constructing an intelligent algorithm capability boundary of the intelligent algorithm based on the labeled triple data. According to the invention, a more accurate intelligent algorithm capability boundary can be obtained.

Description

Technical Field

[0001] The present invention belongs to the technical field of intelligent algorithm evaluation, and proposes a data-driven intelligent algorithm capability boundary construction method and device. Background Art

[0002] With the rapid development of science and technology, intelligent algorithms have been widely used in various fields, greatly reducing human resource investment. However, as the complexity of various influencing factors faced by intelligent algorithms increases, effectively evaluating the ability boundaries of intelligent algorithms under multiple influencing factors has become an urgent problem to be solved.

[0003] The existing technologies for building the capability boundaries of intelligent algorithms mainly include the following:

[0004] Task-based intelligent algorithm capability boundary assessment method: According to the type of intelligent algorithm, such as target detection, target tracking, path planning, task planning, autonomous decision-making, etc., by constructing corresponding test tasks, conducting task tests, and assessing the capability boundary of the intelligent algorithm based on its performance under different tasks. This method targets specific tasks and can reflect the performance of intelligent algorithms in actual tasks, but usually only relies on a single test to derive the results of the intelligent algorithm capability boundary assessment, which has a large uncertainty and leads to inaccurate test results.

[0005] Three-dimensional intelligent algorithm capability boundary evaluation method based on task complexity, environmental complexity and human intervention: A three-axis representation is designed using task complexity, environmental complexity and human intervention, and each axis has a set of measurement indicators. Task complexity can be measured by indicators such as task level, decision-making and collaboration efficiency, perception accuracy, planning and execution performance. Environmental complexity can be measured by obstacle size, obstacle density, terrain type, background characteristics, etc. Human intervention can be measured by indicators such as interaction time and frequency, operator workload, skill level, and automatic startup. Combining the three aspects, tasks of different difficulties can be obtained, and then the capability boundary is obtained based on the task completion of the intelligent algorithm. This method is highly targeted and applicable to a variety of intelligent algorithms, but due to the existence of human intervention, it is easily affected by subjective influences, and the test results are not objective enough. At the same time, the correlation between the evaluation indicators of task complexity and environmental complexity and the capabilities of intelligent algorithms is not analyzed, and the degree of influence of the two on the capabilities of intelligent algorithms cannot be determined, which makes it difficult to obtain accurate algorithm capability boundaries based on the trend of the intelligent algorithm capabilities changing with the two.

[0006] In summary, existing intelligent algorithm capability boundary assessment methods have problems such as single task setting rules and difficulty in obtaining algorithm capability boundaries, which are mainly reflected in: (1) Existing intelligent algorithm capability boundary assessment methods do not analyze the correlation between various influencing factors and intelligent algorithm capabilities, and only draw intelligent algorithm capability assessment conclusions based on test task test results, resulting in the inability to obtain accurate intelligent algorithm capability boundaries based on changes in intelligent algorithm capability influencing factors. (2) Traditional intelligent algorithm capability boundary assessment methods usually rely on only a single test to draw conclusions. Due to the complexity and variability of intelligent algorithms, their performance often has large statistical uncertainties, and a single test is difficult to cover the scope of the intelligent algorithm capability boundary. Secondly, the results of a single test are easily affected by accidental factors and cannot represent the performance of the algorithm in a variety of practical applications. Summary of the invention

[0007] The present invention proposes a data-driven intelligent algorithm capability boundary construction method and system, which can obtain a more accurate intelligent algorithm capability boundary.

[0008] According to a first aspect of an embodiment of the present disclosure, a data-driven intelligent algorithm capability boundary construction method is provided, comprising:

[0009] Obtain static and dynamic properties that affect the capabilities of intelligent algorithms;

[0010] After respectively calculating the influence score of the static attribute, the influence score of the dynamic attribute and the intelligent algorithm capability score of the intelligent algorithm, triple data are formed, and the triple data is labeled according to the intelligent algorithm capability score;

[0011] Based on the labeled triple data, the intelligent algorithm capability boundary of the intelligent algorithm is constructed.

[0012] Optionally, when the intelligent algorithm is a planning algorithm, the static attributes include: path fitting degree, number of collisions, jerkiness and task planning rationality; wherein the planning algorithm includes: path planning algorithm, obstacle avoidance algorithm or task planning algorithm;

[0013] When the intelligent algorithm is a planning algorithm, the dynamic attributes include: path fitting degree, number of collisions, jerkiness, rationality of task planning, dynamic obstacle speed and noise interference.

[0014] Optionally, calculating the influence score of the static attribute includes:

[0015] For each influence evaluation index corresponding to a static attribute, obtain an influence evaluation index value; wherein the influence evaluation index includes: task time, road connectivity, road capacity and number of key points;

[0016] By comparing the influence evaluation index values ​​pairwise, a static attribute influence evaluation index comparison matrix A is constructed;

[0017] Based on the influence evaluation index comparison matrix A, the influence evaluation index weight W of each static attribute is obtained: i ;

[0018] Detect the influence evaluation index weight W of the static attribute i The accuracy of the impact evaluation indicator W i The accuracy of meets the set conditions, based on the influence evaluation index value corresponding to the static attribute and the index weight W of the influence evaluation index i , and obtain the influence score of the static attribute.

[0019] Optionally, obtaining the influence evaluation index value for each static attribute corresponding to the influence evaluation index includes:

[0020] Using the evaluation data, obtain the calculation results of the influence evaluation indicators corresponding to the static attributes;

[0021] The influence evaluation index calculation results are normalized to obtain the normalized influence evaluation index calculation results q i ;

[0022] For the influence evaluation index that is negatively correlated with the intelligent algorithm capability, the normalized influence evaluation index calculation result q i As the impact evaluation index value p i ;

[0023] For the influence evaluation index that is positively correlated with the intelligent algorithm capability, the influence evaluation index value p of the influence evaluation index is i =1-q i .

[0024] Optionally, based on the influence evaluation indicator comparison matrix A, the indicator weight W of each influence evaluation indicator is obtained: i ,include:

[0025] Standardize the elements of each column in the impact assessment index comparison matrix A;

[0026] Sum the standardized elements row by row and standardize the sum result to get the indicator weight W of each impact evaluation indicator. i .

[0027] Optionally, the weight W of the influence evaluation index for detecting the static attribute is i accuracy, including:

[0028] The influence evaluation index weight W based on the influence evaluation index comparison matrix A and the static attribute i , calculate the maximum eigenvalue λ of the influence evaluation index comparison matrix max ;

[0029] According to the maximum eigenvalue λ max , calculate the consistency value CI of the influence evaluation index weight matrix of the static attribute; wherein, the influence evaluation index weight matrix of the static attribute is based on the influence evaluation index weight W of the static attribute i Build;

[0030] According to Satty's simulation test of the static property several times, the random consistency index value RI is obtained;

[0031] Calculate the ratio of the consistency value CI to the random consistency index value RI, and when the ratio is less than a set threshold, determine the indicator weight W of the impact assessment indicator i The accuracy meets the requirements.

[0032] Optionally, constructing the intelligent algorithm capability boundary of the intelligent algorithm based on the labeled triple data includes:

[0033] Generate an intelligent capability boundary curve solution algorithm model, the intelligent capability boundary curve solution algorithm model is constructed based on the formula wx+y+b=0, w represents the boundary slope of the capability boundary curve, b represents the bias, x represents the influence score of the static attribute, and y represents the influence score of the dynamic attribute;

[0034] Solve for the optimal boundary slope w * and the optimal bias b * ;

[0035] The influence scores of static attributes and dynamic attributes are input into the optimal boundary slope w * and the optimal bias b * The intelligent capability boundary curve is used to solve the algorithm model to obtain the intelligent algorithm capability boundary of the intelligent algorithm.

[0036] Optionally, solve the formula wx+y+b=0 for the optimal boundary slope w corresponding to the boundary slope w * The optimal bias b corresponding to bias b * ,include:

[0037] Convert the formula wx+y+b=0 to the formula g=w(x+y); where g is the boundary intercept;

[0038] Take the boundary slope After making the boundary intercept g∈[0,1], the formula g=w(x+y) is converted into the formula

[0039] The formula After converting the test case influencing factor metric value to the linear function of the scenario influencing factor metric value y = -x + 2g, the optimal boundary slope w is obtained. * = -1;

[0040] Based on the labeled triple data, sample points whose intelligent algorithm capability scores exceed the threshold and sample points whose intelligent algorithm capability scores do not exceed the threshold are obtained;

[0041] In the case where the sample points whose intelligent algorithm capability scores exceed the threshold value and the sample points whose intelligent algorithm capability scores do not exceed the threshold value are completely separable, based on the formula Get the optimal bias And the bias The constraints that need to be met include: and Wherein, γ represents the minimum interval between the sample points where the intelligent algorithm capability score exceeds the threshold and the sample points where the intelligent algorithm capability score does not exceed the threshold, and x i is the influence score of the static attribute corresponding to the i-th sample point whose intelligent algorithm capability score does not exceed the threshold, y i is the influence score of the dynamic attribute corresponding to the i-th sample point whose intelligent algorithm capability score does not exceed the threshold, x j is the influence score of the static attribute corresponding to the jth sample point whose intelligent algorithm capability score exceeds the threshold, y i is the influence score of the dynamic attribute corresponding to the jth sample point whose intelligent algorithm capability score exceeds the threshold, x i* ,y i* Divided into a set of sample points C whose intelligent algorithm ability scores do not exceed the threshold -1 The influence scores of static attributes and dynamic attributes corresponding to the samples closest to the hyperplane, x j* ,y j* Divided into a set of sample points C whose intelligent algorithm ability scores exceed the threshold +1 The influence scores of static attributes and dynamic attributes corresponding to the samples closest to the hyperplane are given in the formula The constraints include: (x i ,y i ,-1)∈C -1 、(x j ,y j ,+1)∈C +1 ,x i +y i+b≥0 and x j +y j +b≤0;

[0042] In the case where the sample points whose intelligent algorithm capability scores exceed the threshold and the sample points whose intelligent algorithm capability scores do not exceed the threshold are not completely separable, based on the formula Get the optimal bias b * ; Among them, the formula The constraints include: (x i ,y i ,-1)∈C -1 、(x j ,y j ,+1)∈C +1 ,x i +y i +b≥0 and x j +y j +b≤0, C represents a constant.

[0043] Optionally, after constructing the intelligent algorithm capability boundary of the intelligent algorithm based on the labeled triple data, the method further includes:

[0044] In the sample point set C -1 Select the k sample points P closest to the capability boundary i ;

[0045] Based on the k sample points P closest to the capability boundary i The average distance to the capability boundary is used to calculate the volatility assessment indicator Volatile of the capability boundary of the intelligent algorithm.

[0046] According to a second aspect of an embodiment of the present disclosure, a data-driven intelligent algorithm capability boundary construction system is provided, characterized in that the system includes:

[0047] The attribute acquisition module is used to obtain static and dynamic attributes that affect the capabilities of intelligent algorithms;

[0048] A triplet generation module is used to calculate the influence score of the static attribute, the influence score of the dynamic attribute and the intelligent algorithm capability score of the intelligent algorithm, respectively, to form triplet data, and to label the triplet data according to the intelligent algorithm capability score;

[0049] The boundary construction module is used to construct the intelligent algorithm capability boundary of the intelligent algorithm based on the labeled triple data.

[0050] According to the third aspect of an embodiment of the present disclosure, an electronic device is provided, comprising: a processor and a memory storing computer program instructions; when the processor executes the computer program instructions, the data-driven intelligent algorithm capability boundary construction method provided by the first aspect of the present disclosure is implemented.

[0051] According to the fourth aspect of an embodiment of the present disclosure, a computer-readable storage medium is provided, on which computer program instructions are stored. When the computer program instructions are executed by a processor, the data-driven intelligent algorithm capability boundary construction method provided in the first aspect of the present disclosure is implemented.

[0052] The technical solution provided by the embodiments of the present disclosure includes at least the following beneficial effects:

[0053] (1) Analyze the correlation between the intelligent algorithm capability boundary and the static / dynamic attribute influencing factors of the test case, screen out the static / dynamic influencing factors with correlation to the intelligent algorithm capability, construct a two-dimensional coordinate system of the influencing factors of the intelligent algorithm capability boundary, and obtain the trend of the intelligent algorithm capability changing with the influencing factors.

[0054] (2) Based on the data-driven intelligent algorithm capability boundary calculation model, a large amount of test data is collected to construct a three-tuple data set (test case dynamic attribute influencing factor score - test case static attribute influencing factor score, whether the intelligent algorithm completes the test case). The data set is classified using the machine learning model data statistics and processing capabilities. The dividing line can represent the intelligent algorithm capability boundary. According to the distance between each sample point and the capability boundary, the volatility of the intelligent algorithm capability boundary is analyzed. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] Figure 1 It is a flowchart of a method for constructing a data-driven intelligent algorithm capability boundary according to an exemplary embodiment.

[0056] Figure 2 The figure is a visualization display diagram of a triple data set according to an exemplary embodiment.

[0057] Figure 3 The present invention is a visualization diagram showing the boundary line of intelligent algorithm capability according to an exemplary embodiment.

[0058] Figure 4 It is a block diagram of a device for constructing a data-driven intelligent algorithm capability boundary according to an exemplary embodiment.

[0059] Figure 5 is a block diagram of an electronic device 500 according to an exemplary embodiment. DETAILED DESCRIPTION

[0060] The present invention is further described in detail below in conjunction with specific embodiments. The given examples are only for illustrating the present invention, but not for limiting the scope of the present invention.

[0061] like Figure 1 As shown, the data-driven intelligent algorithm capability boundary construction method in the present invention may include the following steps (step 1 to step 3).

[0062] Step 1: Obtain static and dynamic properties that affect the capabilities of intelligent algorithms.

[0063] A large amount of data will be generated during the intelligent algorithm capability evaluation process, including but not limited to: test case data (images, description files, etc.), test environment information (terrain, weather, time, etc.), test object information (position, size, type, speed, direction, etc.), tested algorithm data (bounding box, path point, task sequence, semantic graph, etc.), evaluation index results (terrain undulation, detection target density, path length, target recognition accuracy, target recognition recall rate, IoU, path fit, number of collisions, use case completion rate, etc.), etc. The present invention uses a large amount of existing intelligent algorithm capability evaluation data to analyze the impact of static attributes and dynamic attributes in test cases on intelligent algorithm capabilities, and selects static attribute and dynamic attribute influence evaluation indicators based on the correlation between each attribute and intelligent algorithm capabilities.

[0064] Step 1.1 Analysis of factors affecting static properties.

[0065] The static attributes of a test case refer to those attributes that are defined when the test case is constructed and will not change during the test execution process. These attributes are used to describe the basic information and execution conditions of the test case. Static attributes generally include, but are not limited to: case time, image resolution, test scene terrain, test scene weather, target size, static target position, etc. The present invention uses a large amount of existing evaluation data to analyze the correlation between the influencing factors of each static attribute and the value of each intelligent algorithm capability evaluation index. If the value of the corresponding capability evaluation index of the intelligent algorithm continues to decrease with the increase of a certain attribute value, it is considered that the attribute is negatively correlated with the intelligent algorithm capability. If the value of the corresponding capability evaluation index of the intelligent algorithm continues to increase with the increase of a certain attribute value, it is considered that the attribute is positively correlated with the intelligent algorithm capability. Otherwise, the attribute has no correlation with the intelligent algorithm capability.

[0066] Taking planning algorithms such as path planning, obstacle avoidance, and task planning as an example, as attributes such as road connectivity and task time improve, the path fitting degree, number of collisions, jerkiness, and task planning rationality of planning algorithms improve, so the above attributes are positively correlated with the capabilities of planning algorithms. As attributes such as the number of key points, the number of static interference targets, and road restrictiveness continue to improve, the path fitting degree, number of collisions, jerkiness, and task planning rationality of planning algorithms continue to decrease, so the above attributes are negatively correlated with the capabilities of planning algorithms. However, attributes such as the location of static interference targets and the degree of ground undulation have no obvious correlation with the planning algorithm capability evaluation index values.

[0067] Step 1.2 Analysis of factors affecting dynamic attributes.

[0068] The dynamic attributes of a test case refer to those attributes that may change during the test execution process. Unlike static attributes, dynamic attributes reflect the real-time status of the test case. Dynamic attributes generally include but are not limited to: dynamic target speed, dynamic target orientation, dynamic target position, dynamic target density, noise intensity, target occlusion, etc. The present invention uses a large amount of existing evaluation data to analyze the correlation between the influencing factors of each dynamic attribute and the values ​​of different intelligent algorithm capability evaluation indicators. Similar to static attributes, if the value of the intelligent algorithm corresponding capability evaluation indicator continues to decrease with the increase of a certain attribute value, it is considered that the attribute is negatively correlated with the intelligent algorithm capability. If the value of the intelligent algorithm corresponding capability evaluation indicator continues to increase with the increase of a certain attribute value, it is considered that the attribute is positively correlated with the intelligent algorithm capability, otherwise the attribute has no correlation with the intelligent algorithm capability.

[0069] Taking planning algorithms such as path planning, obstacle avoidance, and task planning as an example, as visibility and dynamic obstacle visibility improve, the path fitting degree, number of collisions, jerkiness, and task planning rationality of planning algorithms improve, so the above attributes are positively correlated with the capabilities of planning algorithms. As the speed of dynamic obstacles and noise interference continue to improve, the path fitting degree, number of collisions, jerkiness, and task planning rationality of planning algorithms continue to decrease, so the above attributes are negatively correlated with the capabilities of planning algorithms. However, attributes such as the orientation of dynamic objects have no obvious correlation with the planning algorithm capability evaluation index values.

[0070] Step 2: After calculating the influence score of the static attribute, the influence score of the dynamic attribute and the intelligent algorithm capability score of the intelligent algorithm respectively, triple data are formed and the triple data is labeled according to the intelligent algorithm capability score.

[0071] Step 2.1: Calculate the influence score of the static attribute.

[0072] The following takes the planning algorithm as an example to calculate the static attribute influence score of the test case.

[0073] Step 2.1.1: For each influence evaluation indicator corresponding to a static attribute, obtain the influence evaluation indicator value.

[0074] According to the analysis results of the influencing factors of static attributes in step 1.1, the calculation results of the influence evaluation indicators corresponding to the static attributes that have a relevant impact on the intelligent algorithm capabilities are screened from a large amount of evaluation data. The calculation results of each influence evaluation indicator are normalized. The calculation results of the influence evaluation indicators that are negatively correlated with the intelligent algorithm capability evaluation indicators are directly retained. The calculation results of the indicators that are positively correlated are retained after the normalization operation of the corresponding indicator value is subtracted. The retained static attribute influence evaluation indicators are not limited to task time, road connectivity, road capacity, number of key points, etc. Table 1 includes the static attribute influence ranking based on the influence evaluation indicator value.

[0075] Table 1 Ranking of static attribute influence (influence decreases from top to bottom)

[0076]

[0077] Step 2.1.2: Construct the influence evaluation index comparison matrix A of static attributes by comparing the influence evaluation index values ​​pairwise.

[0078] First, after the influence evaluation index of the static attribute is determined, the importance comparison matrix between the indicators is given based on expert experience using Satty's 1-9 scaling method. Specifically, the present invention compares the influence evaluation indicators in pairs according to the indicator influence comparison scale in Table 2, and obtains the influence evaluation index comparison matrix a of the static attribute in Table 3. Among them, the results of the pairwise comparison are usually expressed on a scale of 1 to 9, where 1 means that the two elements are equally important, and 9 means that one element is extremely more important than another element, to construct the influence evaluation index comparison matrix of the static attribute.

[0079] Table 2 Comparison scale of indicator influence

[0080]

[0081] Table 3 Comparison matrix of influence evaluation indicators of static attributes

[0082]

[0083] Step 2.1.3: Based on the impact evaluation index comparison matrix A, obtain the impact evaluation index weight W of each static attribute i .

[0084] In this step, each column of the static attribute influence evaluation index comparison matrix A is first standardized, as shown in Table 4. Then, the standardized elements are summed row by row, and finally the summation result is standardized to obtain the static attribute influence evaluation index weight W as shown in Table 5. i .

[0085] Table 4 Test case static attribute influence index comparison matrix standardization

[0086]

[0087] Table 5 Weights of influence evaluation indicators of static attributes

[0088]

[0089]

[0090] Step 2.1.4 Detect the influence evaluation index weight W of the static attribute i The accuracy of the impact evaluation indicator W i The accuracy of meets the set conditions, based on the influence evaluation index value corresponding to the static attribute and the index weight W of the influence evaluation index i , and obtain the influence score of the static attribute.

[0091] The present invention determines the influence evaluation index weight W by comparing the consistency test of the matrix. i accuracy.

[0092] First, the present invention multiplies the influence evaluation index comparison matrix A in Table 4 by the influence evaluation index weight matrix W constructed based on Table 5, and then removes the corresponding index weight W i Finally, the obtained vectors are summed and divided by the comparison matrix order to obtain the maximum eigenvalue λ of the comparison matrix of the influence evaluation index max .

[0093]

[0094] Where n is the comparison matrix order. Based on the data in Table 4 and Table 5, it can be inferred that the maximum eigenvalue is 11.151.

[0095] Afterwards, the consistency index CI value is calculated based on the calculated maximum eigenvalue:

[0096]

[0097] Then, the random consistency index RI value table obtained by Satty simulation 1000 times is shown in Table 6. By looking up the RI table, it is known that RI(n)=RI(11)=1.52.

[0098] Solve the consistency ratio CR value to determine whether its consistency is passed.

[0099]

[0100] Table 6 Random consistency index RI table

[0101]

[0102] In actual operation, the construction of the influence comparison matrix is ​​often affected by subjective factors and external interference, resulting in inconsistencies. Usually, setting the CR value to be less than 0.1 can effectively control and reduce these errors. When the CR value is less than 0.1, it is considered that the judgment matrix passes the consistency test, otherwise, the influence comparison matrix needs to be reconstructed. In the above embodiment, since CR<0.1, the consistency test is passed.

[0103] Finally, after the comparison matrix passes the consistency check, the weight of the static attribute influence index calculated in step 2.1.3 can be determined, and the weighted sum of the static attribute influence index values ​​in a certain test case is performed, and the result is the static attribute influence score of the corresponding test case.

[0104] SPIC=∑w i *m i

[0105] Among them, w i is the weight of the ith indicator, m i The calculation result of the i-th indicator.

[0106] Assume that in a certain test, the static attribute influence evaluation indicators are: 0.8, 0.4, 0.9, 0.75, 0.35, 0.45, 0.9, 0.16, 0.9, 0.25, 0.41. The static attribute influence score of this test case is: 0.8*0.223+0.4*0.182+0.9*0.182+0.75*0.118+0.35*0.069+0.45*0.044+0.9*0.044+0.16*0.042+0.9*0.042+0.25*0.025+0.41*0.025=0.651.

[0107] Step 2.2: Calculate the influence score of the dynamic attribute.

[0108] The calculation process of the dynamic attribute influence score of the test case is the same as that of the static attribute influence score of the test case, so it will not be described in detail. According to the analysis results of the dynamic attribute influencing factors, the evaluation index calculation results corresponding to the dynamic attributes that have a correlation with the intelligent algorithm capability are screened from a large amount of evaluation data, and the calculation results of each indicator are normalized. The calculation results of the indicators that are negatively correlated with the intelligent algorithm capability evaluation indicators are directly retained, and the calculation results of the indicators that are positively correlated are retained after subtracting the corresponding indicator value. The retained static attribute influencing factor evaluation indicators include but are not limited to target speed, noise intensity, perceived target occlusion degree, target stability, target resolution, etc.

[0109] After the dynamic attribute influencing factor evaluation index is determined, according to the expert experience, the influence comparison matrix of the index is constructed according to Table 1, the weight of each index is calculated, and the matrix eigenvalue is calculated. Then, the CI value is calculated based on the calculated maximum eigenvalue:

[0110]

[0111] After looking up the RI table according to the size of n, the CR value is calculated:

[0112]

[0113] When the CR value is less than 0.1, the judgment matrix is ​​considered to pass the consistency test, otherwise, the influence comparison matrix needs to be reconstructed.

[0114] The weighted sum of each indicator is taken, and the result is the dynamic attribute influence score of the corresponding test case.

[0115] Taking the planning algorithm as an example, the dynamic attribute influence score of the test case is calculated. Referring to the expert experience, the dynamic attribute influence indicators are ranked, and the importance comparison matrix between the indicators is given using Satty's 1-9 scaling method.

[0116] Table 7 Ranking of dynamic attribute influence (decreasing from top to bottom)

[0117]

[0118]

[0119] Table 8 Test case dynamic attribute influence index comparison matrix

[0120]

[0121] Afterwards, the contrast matrix is ​​normalized column-wise.

[0122] Table 9 Test case dynamic attribute influence index comparison matrix standardization

[0123]

[0124] After summing up by rows, normalization is performed to obtain the weight of each indicator.

[0125] Table 10 Test case dynamic attribute influence index weight

[0126]

[0127]

[0128] The consistency check is performed according to the maximum eigenvalue calculation formula. First, the AW matrix is ​​calculated. The matrix A in Table 9 is multiplied by the weights in Table 10, and then the corresponding indicator weights are removed. Finally, the sum of the obtained vectors is divided by the comparison matrix order, and the maximum eigenvector is 7.034.

[0129] The calculated consistency index CI value is:

[0130]

[0131] By looking up the RI table, we know that RI(n)=RI(7)=1.36

[0132]

[0133] Because CR<0.1, the consistency test passes.

[0134] Assume that in a certain test, the dynamic attribute influence evaluation indicators are: 0.75, 0.56, 0.32, 0.48, 0.60, 0.55, 0.61. The static attribute influence score of this test case is: 0.75*0.349+0.56*0.244+0.32*0.178+0.48*0.098+0.60*0.069+0.55*0.037+0.61*0.020=0.579.

[0135] Step 2.3: Calculate the intelligent algorithm capabilities of the intelligent algorithm.

[0136] The calculation results of intelligent algorithm capability evaluation indicators that are relevant to the static and dynamic attributes of the test cases are selected from a large amount of evaluation data, and the calculation results of each indicator are normalized. The relevant evaluation indicators include but are not limited to target detection accuracy, target detection precision, path fitting degree, task planning rationality, decision efficiency, decision accuracy, etc. After the intelligent algorithm capability evaluation indicators are determined, the calculation process is consistent with the test case static attribute influence score calculation process, and will not be described in detail.

[0137] Based on expert experience, the influence comparison matrix of the indicators is constructed, the weights of the relevant indicators are calculated based on the comparison matrix, and the matrix eigenvalues ​​are calculated. Then, the CI value is calculated based on the calculated maximum eigenvalue:

[0138]

[0139] After looking up the RI table according to the size of n, the CR value is calculated:

[0140]

[0141] When the CR value is less than 0.1, the judgment matrix is ​​considered to pass the consistency test, otherwise, the influence comparison matrix needs to be reconstructed.

[0142] The weighted sum of each indicator is taken, and the result is the dynamic attribute influence score of the corresponding test case.

[0143] Taking planning algorithms as an example, the ability scores of intelligent algorithms are calculated, and the ability evaluation indicators of various intelligent algorithms are ranked with reference to expert experience. Satty's 1-9 scaling method is used to give the importance comparison matrix between the indicators.

[0144] Table 11 Ranking of dynamic attribute influence (decreasing from top to bottom)

[0145]

[0146] Table 12 Test case dynamic attribute influence index comparison matrix

[0147]

[0148] Afterwards, the contrast matrix is ​​normalized column-wise.

[0149] Table 13 Test case dynamic attribute influence index comparison matrix standardization

[0150]

[0151]

[0152] After summing up by rows, normalization is performed to obtain the weight of each indicator.

[0153] Table 14 Test case dynamic attribute influence index weight

[0154]

[0155] Based on the maximum eigenvalue calculation formula, the consistency check is performed. First, the AW matrix is ​​calculated, and the matrix A in Table 13 is multiplied by the weights in Table 14. Then the corresponding indicator weights are removed. Finally, the vectors obtained are summed and divided by the comparison matrix order. The maximum eigenvector is 7.312.

[0156] The calculated consistency index CI value is:

[0157]

[0158] By looking up the RI table, we know that RI(n)=RI(7)=1.36

[0159]

[0160] Because CR<0.1, the consistency test passes.

[0161] Assume that in a certain test, the intelligent algorithm capability evaluation indicators are: 0.8, 0.75, 0.51, 0.46, 0.60, 0.57, 0.32. The static attribute influence score of the test case is: 0.8*0.352+0.75*0.240+0.51*0.164+0.46*0.113+0.60*0.071+0.57*0.037+0.32*0.023=0.668.

[0162] Step 2.4: Based on the influence score of the static attribute, the influence score of the dynamic attribute and the intelligent algorithm capability score of the intelligent algorithm, triple data are formed, and the triple data are labeled according to the intelligent algorithm capability score.

[0163] This step combines the static influencing factors, dynamic influencing factors and intelligent algorithm capabilities of the test case to construct a test (test case static attribute influencing factor score - test case dynamic attribute influencing factor score, intelligent algorithm capability score) ternary data set to provide data support for the subsequent intelligent algorithm capability boundary calculation.

[0164] Arrange and calculate a large number of test case static attribute influencing factor scores, test case dynamic attribute influencing factor scores, and intelligent algorithm capability score data. Combine the test case static attribute influencing factor scores, test case dynamic attribute influencing factor scores, and intelligent algorithm capability scores generated by the same test case into a (test case static attribute influencing factor score, test case dynamic attribute influencing factor score, intelligent algorithm capability score) ternary data. After all the data are sorted and combined, annotate the ternary data. Set the intelligent algorithm capability score threshold according to the requirements for the intelligent algorithm capability, which is generally set to 0.6. If the intelligent algorithm capability score exceeds the threshold, it is considered that the intelligent algorithm has passed the test case, and the intelligent algorithm capability score in the corresponding ternary data is annotated as 1, otherwise it is annotated as -1. Form a ternary data set of ((test case static attribute influencing factor score, test case dynamic attribute influencing factor score), whether the intelligent algorithm has passed the test case (1 / -1)).

[0165] Figure 2 This is a visualization of a ternary data set of a certain intelligent algorithm. Among them, red represents the sample points of unfinished test cases, and blue represents the points of completed test cases. The sample points basically cover the complete effective range (0-1) of the test case static attribute influencing factor scores and the test case dynamic attribute influencing factor scores. The sample points are distributed relatively evenly. At the same time, as the static attribute and dynamic attribute influencing factors increase, the intelligent algorithm completes the case gradually worse, and the data set is constructed reasonably. If the sample point distribution cannot cover the complete effective range of the static attribute and dynamic attribute influencing factor scores, the number of test data needs to be supplemented until the constructed ternary data set is reasonable.

[0166] Step 3: Based on the labeled triple data, construct the intelligent algorithm capability boundary of the intelligent algorithm.

[0167] The construction of the ternary data set is completed. The present invention constructs a model for solving the capability boundary of the intelligent algorithm and solves the capability boundary of the intelligent algorithm according to the distribution of the ternary data set.

[0168] Step 3.1: Generate an intelligent capability boundary curve solving algorithm model.

[0169] The intelligent capability boundary curve solving algorithm model is generated based on the formula wx+y+b=0, where w is the boundary slope, b is the bias, and x and y are the test case static attribute influencing factor scores and the test case dynamic attribute influencing factor scores.

[0170] Step 3.2: Solve for the optimal boundary slope w * and the optimal bias b * .

[0171] The present invention uses wx+y+b=0 to classify the triplet data set and solves the optimal boundary slope k* With the optimal bias b * , so that the capacity boundary curve w*x+y+b * =0 has the highest degree of division of the data set.

[0172] For w * The optimization solution is obtained by referring to the existing intelligent algorithm capability evaluation method such as ALFUS. In the calculation method given, the present invention converts the calculation formula of the intelligent algorithm capability boundary into:

[0173] g=w(x+y)

[0174] Where g is the boundary intercept. To unify the scale, take Let g∈[0,1], that is:

[0175]

[0176] Use the test case static attribute influencing factor scores and the test case dynamic attribute influencing factor scores to replace them, and we get:

[0177] y=-x+2g

[0178] Take w * is -1.

[0179] For b * , assuming that there is such a dividing line that can linearly separate all sample points, and the unfinished use case sample points are on the same side of the positive normal vector of the boundary line, then there must be a minimum interval γ≥0, so that the shortest distance from the capability boundary in the two types of samples is γ, that is:

[0180]

[0181] st(x i ,y i ,-1)∈C -1 ,(x j ,y j ,+1)∈C +1 ,x i +y i +b≥0,x j +y j +b≤0

[0182] Among them, C -1 is the sample set of category -1; C +1 is a set of samples of category +1. The b that satisfies the above conditions is the optimal solution b * .

[0183] In the two types of sample points, w*x+y+b *= 0, completely separable, it can be seen that b * It only depends on the two sample points in the two sample sets that are closest to the hyperplane. And it is easy to know:

[0184]

[0185] Borrowing the concept of soft interval, when there is no w*x+y+b * =0, so that the two types of samples can be separated, then there must be a constant C>0, such that:

[0186]

[0187] st(x i ,y i ,-1)∈C -1 ,(x j ,y j ,+1)∈C +1 ,x i +y i +b≥0,x j +y j +b≤0

[0188] From a geometric point of view, This is equivalent to taking the sample point (x i ,y i ,-1) is translated by the distance C along the direction of the positive normal vector, The sample point (x i ,y i ,+1) is translated along the anti-normal vector by a distance of C. After the translation, the two types of sample points are completely separable by x+y+b=0. Therefore, in the case of inseparability, b can also be used. * The solution.

[0189] However, due to the existence of error samples, the results are greatly affected, so we can choose the point farthest from the hyperplane for b * Solve (default setting k=1).

[0190] Step 3.3: Input the influence scores of static attributes and dynamic attributes into the optimal boundary slope w * and the optimal bias b * The intelligent capability boundary curve is used to solve the algorithm model to obtain the intelligent algorithm capability boundary of the intelligent algorithm.

[0191] Input the triple data set into the intelligent algorithm capability boundary curve to solve the algorithm model and draw the capability boundary line. Figure 3 , where the black curve depicts the boundary line of the intelligent algorithm's capabilities.

[0192] Step 4: Calculate the volatility of the intelligent algorithm’s capability boundaries.

[0193] The capability boundary of the intelligent algorithm divides the triple data set. However, due to the instability of the intelligent algorithm, sample points that cannot be correctly classified appear, causing fluctuations in the capability boundary of the intelligent algorithm. Therefore, the present invention proposes a quantifiable method for calculating the volatility of the capability boundary of the intelligent algorithm, and selects the average distance of the k points closest to the capability boundary as the volatility evaluation index of the capability boundary of the intelligent algorithm, that is:

[0194]

[0195] Among them, P e yes The k points closest to the capability boundary in the error sample in , d(P i ) is the sample point P i The smaller the calculated result, the smaller the volatility of the intelligent algorithm's capability boundary, and vice versa.

[0196] To sum up, the existing intelligent algorithm capability evaluation methods do not analyze the correlation between various influencing factors and the intelligent algorithm capability, and only draw intelligent algorithm capability evaluation conclusions through test case test results, resulting in the problem that the intelligent algorithm capability boundary cannot be obtained according to the intelligent algorithm capability influencing factors. The present invention analyzes the influencing factors of test case static attributes and dynamic attributes on intelligent algorithm capability, and constructs a three-dimensional intelligent algorithm capability boundary evaluation method including test case static attribute influencing factor scores, test case dynamic attribute influencing factor scores, and intelligent algorithm capability scores. It realizes a full-process capability boundary evaluation method from test case dynamic and static attributes to intelligent algorithm case completion status, which helps intelligent algorithm developers to have a more comprehensive understanding of the capability boundary of intelligent algorithms when dealing with different test cases, and to perform targeted iterative optimization.

[0197] In view of the problem that traditional intelligent algorithm capability boundary evaluation methods usually rely only on a single test to obtain evaluation results, have large statistical uncertainty and are easily affected by accidental factors, the present invention is based on a large amount of intelligent algorithm evaluation data and carries out intelligent algorithm capability boundary evaluation in a data-driven manner. Specifically, the present invention constructs an intelligent algorithm capability evaluation triple data set, and uses a machine learning algorithm to perform binary classification on the triple data set, divide the intelligent algorithm capability boundary, effectively improve the reliability of intelligent algorithm capability boundary evaluation, and avoid the influence of sudden factors.

[0198] Exemplary Devices

[0199] Figure 4 is a data-driven intelligent algorithm capability boundary construction device according to an exemplary embodiment, referring to Figure 4The device includes: an attribute acquisition module 410, a triple generation module 420 and a boundary construction module 430.

[0200] The attribute acquisition module 410 is used to acquire static attributes and dynamic attributes that affect the capabilities of the intelligent algorithm;

[0201] A triplet generation module 420 is used to calculate the influence score of the static attribute, the influence score of the dynamic attribute and the intelligent algorithm capability score of the intelligent algorithm, respectively, to form triplet data, and to label the triplet data according to the intelligent algorithm capability score;

[0202] The boundary construction module 430 is used to construct the intelligent algorithm capability boundary of the intelligent algorithm based on the labeled triple data.

[0203] Optionally, when the intelligent algorithm is a planning type algorithm, the static attributes include: path fitting degree, number of collisions, jerkiness and task planning rationality; wherein, the planning type algorithm includes: path planning algorithm, obstacle avoidance algorithm or task planning algorithm; when the intelligent algorithm is a planning type algorithm, the dynamic attributes include: path fitting degree, number of collisions, jerkiness, task planning rationality, dynamic obstacle speed and noise interference.

[0204] Optionally, the triplet generation module 420 is used to: obtain the influence evaluation index value for each static attribute corresponding to the influence evaluation index; wherein the influence evaluation index includes: task time, road connectivity, road capacity and number of key points; construct the influence evaluation index comparison matrix A of the static attribute by comparing the influence evaluation index values ​​one by one; based on the influence evaluation index comparison matrix A, obtain the influence evaluation index weight W of each static attribute i ; Detect the influence evaluation index weight W of the static attribute i The accuracy of the impact evaluation indicator W i The accuracy of meets the set conditions, based on the influence evaluation index value corresponding to the static attribute and the index weight W of the influence evaluation index i , and obtain the influence score of the static attribute.

[0205] Optionally, the triplet generation module 420 is used to: use the evaluation data to obtain the influence evaluation index calculation result corresponding to the static attribute; normalize the influence evaluation index calculation result to obtain the normalized influence evaluation index calculation result q i ; For the influence evaluation index that is negatively correlated with the intelligent algorithm capability, the normalized influence evaluation index calculation result q i As the impact evaluation index value p i; For the influence evaluation index that is positively correlated with the intelligent algorithm capability, the influence evaluation index value p of the influence evaluation index is i =1-q i .

[0206] Optionally, the triplet generation module 420 is used to: standardize the elements of each column in the influence evaluation index comparison matrix A; sum the standardized elements by row, and standardize the summed results to obtain the index weight W of each influence evaluation index. i .

[0207] Optionally, the triplet generation module 420 is used to: compare the influence evaluation index matrix A with the influence evaluation index weight W of the static attribute i , calculate the maximum eigenvalue λ of the influence evaluation index comparison matrix max According to the maximum eigenvalue λ max , calculate the consistency value CI of the influence evaluation index weight matrix of the static attribute; wherein, the influence evaluation index weight matrix of the static attribute is based on the influence evaluation index weight W of the static attribute i Construct; perform several simulation tests on the static attribute according to Satty to obtain the random consistency index value RI; calculate the ratio of the consistency value CI to the random consistency index value RI, and when the ratio is less than a set threshold, determine the index weight W of the impact assessment index i The accuracy meets the requirements.

[0208] Optionally, the boundary construction module 430 is used to: generate an intelligent capability boundary curve solution algorithm model, the intelligent capability boundary curve solution algorithm model is constructed based on the formula wx+y+b=0, w represents the boundary slope of the capability boundary curve, b represents the bias, x represents the influence score of the static attribute, and y represents the influence score of the dynamic attribute; solve the optimal boundary slope w * and the optimal bias b * ; Input the influence scores of static attributes and dynamic attributes into the optimal boundary slope w * and the optimal bias b * The intelligent capability boundary curve is used to solve the algorithm model to obtain the intelligent algorithm capability boundary of the intelligent algorithm.

[0209] Optionally, the boundary construction module 430 is used to: convert the formula wx+y+b=0 into the formula g=w(x+y); wherein g is the boundary intercept; and take the boundary slope After making the boundary intercept g∈[0,1], the formula g=w(x+y) is converted into the formula The formula After converting the test case influencing factor metric value to the linear function of the scenario influencing factor metric value y = -x + 2g, the optimal boundary slope w is obtained. * = -1; based on the annotated triple data, obtain sample points whose intelligent algorithm capability scores exceed the threshold and sample points whose intelligent algorithm capability scores do not exceed the threshold; if the sample points whose intelligent algorithm capability scores exceed the threshold and the sample points whose intelligent algorithm capability scores do not exceed the threshold are completely separable, based on the formula Get the optimal bias x j* +y j* ), and the bias The constraints that need to be met include: and Wherein, γ represents the minimum interval between the sample points where the intelligent algorithm capability score exceeds the threshold and the sample points where the intelligent algorithm capability score does not exceed the threshold, and x i is the influence score of the static attribute corresponding to the i-th sample point whose intelligent algorithm capability score does not exceed the threshold, y i is the influence score of the dynamic attribute corresponding to the i-th sample point whose intelligent algorithm capability score does not exceed the threshold, x j is the influence score of the static attribute corresponding to the jth sample point whose intelligent algorithm capability score exceeds the threshold, y i is the influence score of the dynamic attribute corresponding to the jth sample point whose intelligent algorithm capability score exceeds the threshold, x i* ,y i* Divided into a set of sample points C whose intelligent algorithm ability scores do not exceed the threshold -1 The influence scores of static attributes and dynamic attributes corresponding to the samples closest to the hyperplane, x j* ,y j* Divided into a set of sample points C whose intelligent algorithm ability scores exceed the threshold +1 The influence scores of static attributes and dynamic attributes corresponding to the samples closest to the hyperplane are given in the formula The constraints include: (x i ,y i ,-1)∈C -1 、(x j ,y j ,+1)∈C +1 ,x i +y i +b≥0 and x j +y+b≤0; when the sample points whose scores of the intelligent algorithm capability exceed the threshold and the sample points whose scores of the intelligent algorithm capability do not exceed the threshold are not completely separable, based on the formula Get the optimal bias b* ; Among them, the formula The constraints include: (x i ,y i ,-1)∈C -1 、(x j ,y j ,+1)∈C +1 ,x i +y i +b≥0 and x j +y j +b≤0, C represents a constant.

[0210] Optionally, the device further comprises a boundary volatility calculation module 430, which is used to: -1 Select the k sample points P closest to the capability boundary i ; Based on the k sample points P closest to the capability boundary i The average distance to the capability boundary is used to calculate the volatility assessment indicator Volatile of the capability boundary of the intelligent algorithm.

[0211] The exemplary device is an embodiment of the device corresponding to the above exemplary method. The specific operations of each module can be understood by referring to the description of the method embodiment, which will not be repeated here.

[0212] Exemplary Electronic Devices

[0213] Figure 5 1 is a block diagram of an electronic device 500 according to an exemplary embodiment. The electronic device 500 may be a terminal, a notebook computer, a desktop computer, a server, a computer cluster or other types of electronic devices.

[0214] Reference Figure 5 , the electronic device 500 may include at least one processor 510 and a memory 520. The processor 510 may execute instructions stored in the memory 520. The processor 510 is communicatively connected to the memory 520 via a data bus. In addition to the memory 520, the processor 510 may also be communicatively connected to an input device 530, an output device 540, and a communication device 550 via a data bus.

[0215] The processor 510 may be any conventional processor, such as a central processing unit (CPU), a graphic processing unit (GPU), a field programmable gate array (FPGA), a system on chip (SOC), an application specific integrated circuit (ASIC), or a combination thereof.

[0216] The memory 520 may be implemented by any type of volatile or nonvolatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk.

[0217] In the embodiment of the present disclosure, executable instructions are stored in the memory 520, and the processor 510 can read the executable instructions from the memory 520 and execute the instructions to implement all or part of the steps of the data-driven intelligent algorithm capability boundary construction method in the above exemplary embodiment.

[0218] Exemplary computer-readable storage media

[0219] In addition to the above methods and devices, the exemplary embodiments of the present disclosure also include a computer program product or a computer-readable storage medium storing the computer program product. The computer product includes computer program instructions that can be executed by a processor to implement all or part of the steps described in the above exemplary embodiments.

[0220] The computer program product may be written in any combination of one or more programming languages ​​to write program codes for performing the operations of the embodiments of the present application, including object-oriented programming languages, such as Java, C++, etc., and also conventional procedural programming languages, such as "C" language or similar programming languages ​​and scripting languages ​​(e.g., Python). The program code may be executed entirely on the user computing device, partially on the user computing device, as an independent software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0221] Computer readable storage media can adopt any combination of one or more readable media. The readable medium can be a readable signal medium or a readable storage medium. The readable storage medium can include, for example, but is not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices or devices, or any combination of the above. More specific examples of readable storage media include: static random access memory (SRAM) with one or more wires electrically connected, electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk or optical disk, or any suitable combination of the above.

[0222] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the present disclosure. The present disclosure is intended to cover any variations, uses or adaptations of the present disclosure, which follow the general principles of the present disclosure and include common knowledge or customary technical means in the art that are not disclosed in the present disclosure. The specification and examples are intended to be exemplary only, and the present disclosure is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes may be made without departing from the scope thereof.

Claims

1. A data-driven intelligent algorithm capability boundary construction method, characterized in that: The method comprises: Obtain static and dynamic properties that affect the capabilities of intelligent algorithms; After respectively calculating the influence score of the static attribute, the influence score of the dynamic attribute and the intelligent algorithm capability score of the intelligent algorithm, triple data are formed, and the triple data is labeled according to the intelligent algorithm capability score; Based on the labeled triple data, the intelligent algorithm capability boundary of the intelligent algorithm is constructed.

2. The method according to claim 1, characterized in that In the case where the intelligent algorithm is a planning algorithm, the static attributes include: path fitting degree, number of collisions, jerkiness and task planning rationality; wherein the planning algorithm includes: path planning algorithm, obstacle avoidance algorithm or task planning algorithm; When the intelligent algorithm is a planning algorithm, the dynamic attributes include: path fitting degree, number of collisions, jerkiness, rationality of task planning, dynamic obstacle speed and noise interference.

3. The method according to claim 1, characterized in that The calculation of the influence score of the static attribute includes: For each influence evaluation index corresponding to a static attribute, obtain an influence evaluation index value; wherein the influence evaluation index includes: task time, road connectivity, road capacity and number of key points; By comparing the influence evaluation index values ​​pairwise, a static attribute influence evaluation index comparison matrix A is constructed; Based on the influence evaluation index comparison matrix A, the influence evaluation index weight W of each static attribute is obtained: i ; Detect the influence evaluation index weight W of the static attribute i The accuracy of the impact evaluation indicator W i The accuracy of meets the set conditions, based on the influence evaluation index value corresponding to the static attribute and the index weight W of the influence evaluation index i , and obtain the influence score of the static attribute.

4. The method according to claim 3, characterized in that The step of obtaining the influence evaluation index value for each static attribute corresponding to the influence evaluation index includes: Using the evaluation data, obtain the calculation results of the influence evaluation indicators corresponding to the static attributes; The influence evaluation index calculation results are normalized to obtain the normalized influence evaluation index calculation results q i ; For the influence evaluation index that is negatively correlated with the intelligent algorithm capability, the normalized influence evaluation index calculation result q i As the impact evaluation index value p i ; For the influence evaluation index that is positively correlated with the intelligent algorithm capability, the influence evaluation index value p of the influence evaluation index is i =1-q i .

5. The method according to claim 3, characterized in that: Based on the influence evaluation index comparison matrix A, obtain the index weight W of each influence evaluation index i ,include: Standardize the elements of each column in the impact assessment index comparison matrix A; Sum the standardized elements row by row and standardize the sum result to get the indicator weight W of each impact evaluation indicator. i .

6. The method according to claim 3, characterized in that The weight W of the influence evaluation index for detecting the static attribute i accuracy, including: The influence evaluation index weight W based on the influence evaluation index comparison matrix A and the static attribute i , calculate the maximum eigenvalue λ of the influence evaluation index comparison matrix max ; According to the maximum eigenvalue λ max , calculate the consistency value CI of the influence evaluation index weight matrix of the static attribute; wherein, the influence evaluation index weight matrix of the static attribute is based on the influence evaluation index weight W of the static attribute i Build; According to Satty's simulation test of the static property several times, the random consistency index value RI is obtained; Calculate the ratio of the consistency value CI to the random consistency index value RI, and when the ratio is less than a set threshold, determine the indicator weight W of the impact assessment indicator i The accuracy meets the requirements.

7. The method according to claim 1, characterized in that The intelligent algorithm capability boundary of the intelligent algorithm is constructed based on the labeled triple data, including: Generate an intelligent capability boundary curve solution algorithm model, the intelligent capability boundary curve solution algorithm model is constructed based on the formula wx+y+b=0, w represents the boundary slope of the capability boundary curve, b represents the bias, x represents the influence score of the static attribute, and y represents the influence score of the dynamic attribute; Solve for the optimal boundary slope w * and the optimal bias b * ; The influence scores of static attributes and dynamic attributes are input into the optimal boundary slope w * and the optimal bias b * The intelligent capability boundary curve is used to solve the algorithm model to obtain the intelligent algorithm capability boundary of the intelligent algorithm.

8. The method according to claim 7, characterized in that Solve the formula wx+y+b=0 for the optimal boundary slope w corresponding to the boundary slope w * The optimal bias b corresponding to bias b * ,include: Convert the formula wx+y+b=0 to the formula g=w(x+y); where g is the boundary intercept; Take the boundary slope After making the boundary intercept g∈[0,1], the formula g=w(x+y) is converted into the formula The formula After converting the test case influencing factor metric value to the linear function of the scenario influencing factor metric value y = -x + 2g, the optimal boundary slope w is obtained. * = -1; Based on the labeled triple data, sample points whose intelligent algorithm capability scores exceed the threshold and sample points whose intelligent algorithm capability scores do not exceed the threshold are obtained; In the case where the sample points whose intelligent algorithm capability scores exceed the threshold value and the sample points whose intelligent algorithm capability scores do not exceed the threshold value are completely separable, based on the formula Get the optimal bias And the bias The constraints that need to be met include: and Wherein, γ represents the minimum interval between the sample points where the intelligent algorithm capability score exceeds the threshold and the sample points where the intelligent algorithm capability score does not exceed the threshold, and x i is the influence score of the static attribute corresponding to the i-th sample point whose intelligent algorithm capability score does not exceed the threshold, y i is the influence score of the dynamic attribute corresponding to the i-th sample point whose intelligent algorithm capability score does not exceed the threshold, x j is the influence score of the static attribute corresponding to the jth sample point whose intelligent algorithm capability score exceeds the threshold, y i is the influence score of the dynamic attribute corresponding to the jth sample point whose intelligent algorithm capability score exceeds the threshold, Divided into a set of sample points C whose intelligent algorithm ability scores do not exceed the threshold -1 The influence scores of static attributes and dynamic attributes corresponding to the samples closest to the hyperplane, Divided into a set of sample points C whose intelligent algorithm ability scores exceed the threshold +1 The influence scores of static attributes and dynamic attributes corresponding to the samples closest to the hyperplane are given in the formula The constraints include: (x i ,y i ,-1)∈C -1 、(x j ,y j ,+1)∈C +1 ,x i +y i +b≥0 and x j +y j +b≤0; In the case where the sample points whose intelligent algorithm capability scores exceed the threshold and the sample points whose intelligent algorithm capability scores do not exceed the threshold are not completely separable, based on the formula Get the optimal bias b * ; Among them, the formula The constraints include: (x i ,y i ,-1)∈C -1 、(x j ,y i ,+1)∈C +1 ,x i +y i +b≥0 and x j +y j +b≤0, C represents a constant.

9. The method according to claim 8, characterized in that After constructing the intelligent algorithm capability boundary of the intelligent algorithm based on the labeled triple data, the following is further included: In the sample point set C -1 Select the k sample points P closest to the capability boundary i ; Based on the k sample points P closest to the capability boundary i The average distance to the capability boundary is used to calculate the volatility assessment indicator Volatile of the capability boundary of the intelligent algorithm.

10. A data-driven intelligent algorithm capability boundary construction device, characterized in that: The system comprises: The attribute acquisition module is used to obtain static and dynamic attributes that affect the capabilities of intelligent algorithms; A triplet generation module is used to calculate the influence score of the static attribute, the influence score of the dynamic attribute and the intelligent algorithm capability score of the intelligent algorithm, respectively, to form triplet data, and to label the triplet data according to the intelligent algorithm capability score; The boundary construction module is used to construct the intelligent algorithm capability boundary of the intelligent algorithm based on the labeled triple data.

Images