Intelligent agent performance evaluation method based on heterogeneous multi-dimensional index system fusion

Abstract

The invention discloses an agent performance evaluation method based on heterogeneous multi-dimensional index system fusion, which comprises the following steps of: firstly, constructing a heterogeneous multi-dimensional evaluation index system according to an evaluation target, designing a test scheme according to the heterogeneous multi-dimensional evaluation index system, collecting test data, performing index measurement and score assignment based on the test data, and performing evaluation on the evaluation target; and finally, calculating an efficiency evaluation result based on an improved analytic hierarchy process. According to the scheme, the accuracy and comparability of an evaluation result can be improved through quantitative processing and standardized mapping of test conditions in combination with evaluation target association degree and coverage of multi-dimensional feature factors influencing index performance, dependence on historical training data is not needed, and system deployment cost and early-stage investment are effectively reduced; according to the scheme, an interpretable digital calculation process is adopted in the whole process, transparency and traceability of an evaluation result are ensured, the high-credibility verification requirement is met, comprehensive analysis of the efficiency of the intelligent agent is achieved through a multi-dimensional index weight distribution mechanism, and the method is particularly suitable for a multi-target comprehensive evaluation scene of a complex system.

Description

Technical Field

[0001] This invention belongs to the field of performance evaluation, specifically relating to a method for evaluating the performance of intelligent agents based on the fusion of heterogeneous multidimensional indicator systems. Background Technology

[0002] As products in fields such as healthcare and services become increasingly feature-rich, the industry's talent pool is also showing a trend towards multi-skilled professionals. The performance indicators of intelligent agents (such as the performance of a single aspect of equipment, system performance, overall system performance, and the comprehensive quality of personnel) cannot be directly quantified, calculated, compared, or measured. Therefore, they need to be broken down into several measurable sub-indicators, and data must be obtained through experiments / evaluations. However, the performance of indicators under single experimental conditions lacks objectivity, requiring the design of multi-condition combinations to comprehensively cover real-world scenarios; furthermore, there is the challenge of integrating subjective and objective data.

[0003] To address the aforementioned issues, patent application number CN115409535A discloses a method for evaluating the performance of complex product sensory interaction by integrating multi-source heterogeneous data. It integrates subjective evaluation, eye-tracking and facial expression data to construct a comprehensive evaluation matrix and calculates weights through multiple correlation coefficients. However, it does not quantify the difficulty of experimental conditions, and the weights rely solely on expert-preset thresholds, failing to correct for environmental differences.

[0004] Patent application publication number 114254875A discloses a multi-dimensional performance evaluation method for tasks. It establishes a spatiotemporal integrated environmental impact model, quantifies the dynamic changes of the environment through a spatiotemporal coordinate system, and solves the shortest time cost based on a path search algorithm to evaluate task adaptability. However, the spatiotemporal prism solution adopts a black box decision-making approach, iteratively updates nodes through open lists and fixed lists, the node iteration logic is untraceable, and the environmental cost accumulation rate model needs to be manually defined, requiring high domain knowledge and large initial manual investment.

[0005] The patent application with publication number CN116821386A discloses a single feature evaluation and control method based on multi-source heterogeneous data structuring. It uses pure mathematical transformations such as nonlinear decoupling to process data, but does not correct for environmental suppression effects, such as the influence of interference intensity on index values.

[0006] Therefore, there is an urgent need for an evaluation method that combines the quantification of experimental conditions with numerical interpretability, and that can provide comprehensive and accurate evaluation results without prior accumulation, while being compatible with both subjective and objective data. Summary of the Invention

[0007] To address the technical bottlenecks of existing performance evaluation methods, such as data distortion in challenging scenarios due to the lack of quantification of experimental conditions, and the lack of interpretability, security, and credibility of black-box models, this invention provides a performance evaluation method for intelligent agents based on the fusion of heterogeneous multidimensional indicator systems. Starting from experimental design methods, a heterogeneous multidimensional indicator system is constructed, quantifying experimental conditions and integrating them into the performance evaluation numerical calculation method. This method requires no external training data or domain knowledge pre-sets, and the evaluation results are transparent, traceable, and more objective and comprehensive.

[0008] The specific technical solution for achieving the objective of this invention is as follows:

[0009] A method for evaluating the effectiveness of intelligent agents based on the fusion of heterogeneous multidimensional indicator systems includes the following steps:

[0010] Step 1: Define the evaluation objectives;

[0011] Step 2: Construct a heterogeneous, multidimensional evaluation index system based on the evaluation objectives;

[0012] Step 3: Design an experimental plan and set up experimental conditions based on the constructed heterogeneous multidimensional evaluation index system;

[0013] Step 4: Run the experiment and collect test data;

[0014] Step 5: Calculate and score the indicators based on the experimental data;

[0015] Step 6: Calculate the performance evaluation results based on the improved analytic hierarchy process.

[0016] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0017] The solution of this invention can combine the correlation of evaluation targets to cover multi-dimensional feature factors that affect the performance of indicators; through the quantitative processing and standardized mapping of experimental conditions, the accuracy and comparability of evaluation results are improved; the evaluation model built based on numerical calculation methods does not need to rely on historical training data, effectively reducing system deployment costs and upfront investment.

[0018] The entire process of the present invention adopts an interpretable digital calculation flow to ensure the transparency and traceability of the evaluation results and meet the requirements of high credibility verification.

[0019] The solution of this invention achieves a comprehensive analysis of the effectiveness of intelligent agents through a multi-dimensional index weight allocation mechanism, which is particularly suitable for multi-objective comprehensive evaluation scenarios of complex systems.

[0020] The present invention will be further described below with reference to specific embodiments. Attached Figure Description

[0021] Figure 1This is a schematic diagram of the process for evaluating the effectiveness of intelligent agents based on the fusion of heterogeneous multidimensional index systems, as described in this invention.

[0022] Figure 2 This is a schematic diagram of the heterogeneous multidimensional evaluation index system architecture of the present invention.

[0023] Figure 3 This is a schematic diagram of the heterogeneous multidimensional evaluation index system architecture constructed in an embodiment of the present invention.

[0024] Figure 4 This is a schematic diagram illustrating the process of calculating performance evaluation results based on the improved analytic hierarchy process in an embodiment of the present invention. Detailed Implementation

[0025] Example

[0026] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. The described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0027] As indicated in this application and claims, unless the context clearly indicates otherwise, the words "a," "an," "an," and / or "the" do not specifically refer to the singular and may also include the plural. Generally speaking, the terms "comprising" and "including" only indicate the inclusion of explicitly identified steps and elements, which do not constitute an exclusive list, and the method or apparatus may also include other steps or elements.

[0028] Unless otherwise specifically stated, the relative arrangement, numerical expressions, and values ​​of the components and steps described in these embodiments do not limit the scope of this application. It should also be understood that, for ease of description, the dimensions of the various parts shown in the drawings are not drawn to actual scale. Techniques, methods, and devices known to those skilled in the art may not be discussed in detail, but where appropriate, such techniques, methods, and devices should be considered part of the specification. In all examples shown and discussed herein, any specific values ​​should be interpreted as merely exemplary and not as limitations. Therefore, other examples of exemplary embodiments may have different values. It should be noted that similar reference numerals and letters in the following drawings denote similar items; therefore, once an item is defined in one drawing, it need not be further discussed in subsequent drawings.

[0029] Combination Figure 1 A method for evaluating the effectiveness of intelligent agents based on the fusion of heterogeneous multidimensional indicator systems includes the following steps:

[0030] Step 1: Define the evaluation objectives;

[0031] Generally, the evaluation methods in this scheme are more suitable for comprehensive indicators that are inconvenient to directly quantify, calculate, compare, and measure, such as the single-item effectiveness, comprehensive effectiveness, and system effectiveness of equipment, the various capabilities of personnel, and the management effectiveness of digital products. Taking drones as an example, the evaluation objective can be set as the reconnaissance effectiveness of drones; taking personnel evaluation as an example, the evaluation objective can be set as their knowledge and skills; taking digital management systems as an example, the evaluation objective can be set as resource management effectiveness and process efficiency.

[0032] The evaluation objective in this embodiment is the reconnaissance effectiveness of the intelligent agent (drone).

[0033] Step 2, Combining Figure 2 A heterogeneous, multidimensional evaluation index system was constructed based on the evaluation objectives:

[0034] Based on the evaluation objective T, and combined with experimental design methods, a heterogeneous multidimensional evaluation index system was constructed.

[0035] The heterogeneous multidimensional evaluation index system consists of two parts: index structure and condition structure.

[0036] Among them, the indicator structure takes the evaluation target T as the highest layer, that is, the target layer, which is used to characterize the determined evaluation target;

[0037] The target layer is decomposed hierarchically downwards, based on the capabilities required to evaluate the target, to obtain several levels of evaluation criteria for the target, denoted as the criterion layer, where the first level is G1 to G... n The next level after G1 is G 11 ~G 1n G m The next level is Gm1~G mn ;

[0038] The criteria layer is divided into several indicators at the end, which are set as indicator layer I. The indicator layer is obtained by decomposing the last level of evaluation criteria in the criteria layer.

[0039] The highest level of the condition structure is the end criterion of the criterion layer. Each end criterion requires the design of a corresponding condition structure. That is, the influencing factors are set according to the last level of evaluation criteria to form a factor layer. The setting idea is to list all the factors that affect the performance of the indicator layer I of the end criterion.

[0040] The last layer is the horizontal layer L, which divides each factor in the factor layer horizontally.

[0041] Taking the reconnaissance effectiveness against drones (evaluation target T) in this embodiment as an example:

[0042] The evaluation target of the target layer is the detection effectiveness of the agent, which can be evaluated using two criteria: "target recognition capability (G1)" and "target tracking capability (G2)". Therefore, the criterion layer includes target recognition capability and target tracking capability.

[0043] The metrics of the target recognition capability (G1) index layer include accuracy (I). 11 ), recall rate (I 12 ), and F1 score (I 13 The target tracking capability (G2) index layer includes the average center position error (I0). 21 ), average trajectory overlap rate (I 22 ) and multi-target tracking accuracy MOTA (I 23 );

[0044] In the conditional structure, the factors of the factor layer of the target recognition capability (G1), namely index I 11 ~I 13 Performance is affected by recognition distance (E) 11 ), Identification Category (E) 12 ), target speed (E) 13 Visibility (E) 14 ), occlusion range (E) 15 Factors influencing the target tracking capability (G2), namely, index I. 21 ~I 23 Performance is affected by target speed (E) 21 Tracking number (E) 22 ), occlusion range (E) 23 Factors affecting the outcome;

[0045] Then, different levels were assigned to each of the influencing factors {E}.

[0046] Step 3: Design an experimental plan and set up experimental conditions based on the constructed heterogeneous multidimensional evaluation index system;

[0047] Based on the factor levels defined by the last evaluation criteria of the criteria layer, different experimental schemes are designed and combined using orthogonal or uniform experimental methods to ensure that the level combinations of factors are uniformly comparable.

[0048] Establish experimental conditions and set different experimental difficulties according to the experimental plan to meet the level combination of all factors in the designed experimental plan.

[0049] Based on the above design, combined with Figure 3 In this embodiment, a heterogeneous multidimensional evaluation index system is established as follows:

[0050] First layer (target layer T): the reconnaissance effectiveness of drones;

[0051] The second layer (criteria layer G) consists of two end-point evaluation criteria: 1. Target identification capability (G1) and 2. Target tracking capability (G2).

[0052] Dimension 1 - Third Layer (Indicator Layer I): 1.1 Precision (I) 11 ), 1.2 recall rate (I 12 ), 1.3F1 score (I 13 ), 2.1 Average center position error (I 21 ), 2.2 Average trajectory overlap rate (I 22 ), 2.3 Multi-target tracking accuracy MOTA (I 23 );

[0053] Dimension 2 - Third Layer (Factor Layer E): 1.1 Recognition Distance (E) 11 ), 1.2 Identify Categories (E 12 ), 1.3 target velocity (E) 13 ), 1.4 visibility (E) 14 ), 1.5 occlusion range (E 15 ), 2.1 Target speed (E) 21 ), 2.2 Tracking Count (E) 22 ), 2.3 Obstruction range (E 23 );

[0054] Dimension 2 - Fourth Layer (Horizontal Layer L): 1.1.1 Near, 1.1.2 Slightly Near, 1.1.3 Slightly Far, 1.1.4 Far, 1.2.1 Single Target, 1.2.2 Multiple Targets, 1.3.1 Static Target, 1.3.2 Dynamic Target, 1.4.1 Good, 1.4.2 Smoke / Fog / Haze, 1.5.1 Unobstructed, 1.5.2 Obstructed (approximately 30%), 2.1.1 Slow, 2.1.2 Medium, 2.1.3 Fast, 2.2.1 Single Target, 2.2.2 2-5 Targets, 2.2.3 More Than 5 Targets, 2.3.1 Unobstructed, 2.3.2 Approximately 20% Obstructed, 2.3.3 Approximately 40% Obstructed.

[0055] In this embodiment, based on the orthogonal experimental design method and combined with the orthogonal design table L8(4×2) 4 ) for recognition distance (E 11 Divide the data into four levels, L1 to L4, and identify categories (E). 12 ), target speed (E) 13 Visibility (E) 14 ), occlusion range (E) 15 Divide the space into two levels, L1 and L2. Combine this with orthogonal design table L9(3). 3 ), for the target velocity (E) 21 Tracking number (E) 22 ), occlusion range (E)23 Divide the data into three levels, L1 to L3.

[0056] To ensure that the level combinations of factors are uniformly comparable, an orthogonal experimental method is used, combined with L8(4×2) 4 L9(3) 3 The orthogonal design table is used to design experimental conditions for the two end-point evaluation criteria (target recognition capability (G1) and target tracking capability (G2)). The experimental design scheme in this embodiment can be shown in Table 1:

[0057] Table 1 Examples of Experimental Design Schemes

[0058]

[0059]

[0060] Different levels of difficulty are designed by uniformly and neatly combining the levels of various factors. Real-world or simulated experimental conditions are then constructed based on the designed experimental plan.

[0061] Step 4: Run the experiment and collect test data;

[0062] Target recognition and target tracking experiments were conducted according to the experimental design scheme, and data required for calculating all indicators in the indicator layer were collected sequentially according to the experiment number. To facilitate rapid data collection, the target recognition and target tracking experiments could be conducted in a simulation environment.

[0063] The data collected in this embodiment includes:

[0064] 1.1 Accuracy (I) 11 True positives (TP) (number of correctly detected targets) and false positives (FP) (number of falsely detected background or incorrect targets).

[0065] 1.2 Recall Rate (I 12 ): True negatives (TP) (number of correctly detected targets), false negatives (FN) (number of true targets that were not detected).

[0066] 1.3 F1 score (I 13 ): Through accuracy (I 11 ) and recall (I 12 )calculate.

[0067] 2.1 Average center position error (I) 21 ): Frame index t for each frame of data, coordinates c of the predicted target center point in each frame of data. p,t and the coordinates of the actual target center point c g,t Total number of frames T.

[0068] 2.2 Average Trajectory Overlap Rate (I)22 ): Predicted bounding box region B in each frame of data p,t and the actual annotation area B g,t Total number of frames T.

[0069] 2.3 Multi-target tracking accuracy MOTA(I) 23 ): False positives (FPs) in each frame of data t False negative example FN t True target number GT t IDSW number of ID switching t .

[0070] Step 5: Calculate and score indicators based on experimental data:

[0071] Step 5-1: Determine the values ​​of each indicator based on the collected experimental data:

[0072] Accuracy:

[0073]

[0074] In this context, TP represents true positives and FP represents false positives.

[0075] Recall rate:

[0076]

[0077] F1 score:

[0078]

[0079] Where Precision is the accuracy rate and Recall is the recall rate;

[0080] Average center position error:

[0081]

[0082] Where t is the frame index, c p,t and c g,t Let be the predicted target center point coordinates and the actual target center point coordinates in each frame of data, respectively, and T be the total number of frames;

[0083] Average trajectory overlap rate:

[0084]

[0085] Among them, B p,t and B g,t These represent the predicted bounding box and the ground truth bounding box regions in each frame of data;

[0086] Multi-target tracking accuracy (MOTA):

[0087]

[0088] Among them, FP t 、FN t GT t IDSW t These represent the number of false positives, false negatives, true targets, and ID switching counts in each frame of data.

[0089] Step 5-2: Design the scoring rules for the performance indicators, assign scores to each factor, and obtain different indicator results under each experimental scheme;

[0090] For example: ① Segmented scoring: 95% or higher accuracy is 100 points, 90%–95% is 95 points, and so on. This scoring method can be used for subjective data;

[0091] ② Formula conversion: Precision score = indicator performance × 100, Precision score = indicator performance 2, etc.

[0092] The following are sample processing results from the target recognition experiment (out of 100 points, scored according to the formula "accuracy score = performance index × 100"):

[0093] Table 2 Example of data processing results

[0094]

[0095] Step 6, Combining Figure 4 The performance evaluation results were calculated based on the improved analytic hierarchy process.

[0096] Step 6-1: Construct the evaluation matrix and perform a consistency evaluation;

[0097] A rating matrix is ​​established based on the constructed heterogeneous multidimensional evaluation index system to characterize the relative importance of each element at each level of the heterogeneous multidimensional evaluation index system relative to a certain element or evaluation target at the next higher level.

[0098] When evaluating, the evaluation objective T should be considered. Elements with high difficulty do not necessarily have high weights; their difficulty and relevance to the evaluation objective T must be comprehensively considered. For example, in reconnaissance effectiveness, recall should be more important than precision in target identification, while in strike effectiveness, precision is more important than recall. Specifically, between the factor layer E and the criterion layer G, the evaluation matrix indicates the relative importance of the level of a factor to the relevant final criteria.

[0099] Table 3 shows the form of the evaluation matrix:

[0100] Table 3 Evaluation Matrix

[0101]

[0102] Among them, b ij It is relative to A k As for B i For B j The relative importance is represented numerically. Since the number of related elements in each level of the example is small, this example uses a 5-level scale, b ij The value can be 1 to 5 or their reciprocals: 1 represents B. i With B j Equally important; 3 represents B i B j One important point (slightly important); 5 represents B. i B j Important (obviously important); 2 and 4 are the intermediate scales between 1 and 3, and 3 and 5, respectively; the reciprocal represents B. j B i An important scale.

[0103] Then, a consistency check was performed on the evaluation matrix;

[0104] To verify whether the weight vector obtained from the evaluation matrix is ​​acceptable, the consistency index of the evaluation matrix needs to be checked. This is done by calculating the largest eigenvalue λ of the evaluation matrix. max The consistency index is calculated and compared with the average random consistency index RI. The evaluation matrix should have satisfactory consistency; otherwise, the evaluation matrix needs to be adjusted.

[0105] Step 6-2: Perform hierarchical single sorting, that is, determine the relative weight values ​​of elements in this level with respect to elements in the previous level:

[0106]

[0107] Among them, b ij,m qn represents the relative importance of elements at a certain level in the m-th rating matrix, and qn represents the total number of rating matrices, which is usually no less than 8.

[0108] The above formula can be used to take the geometric mean of the values ​​at the same position in q evaluation matrices as the result value of the final evaluation matrix.

[0109] Then, by normalizing each column of the final evaluation matrix, summing the rows, and normalizing again, the weights of the evaluation matrix at each level can be obtained. n represents the number of levels in the constructed heterogeneous multidimensional evaluation index system;

[0110] Step 6-3, the overall hierarchical ranking, utilizes the calculation results of the single ranking at each level, and calculates layer by layer from top to bottom to obtain the overall ranking of the final-level elements of the heterogeneous multidimensional indicator system relative to the evaluation target. In this embodiment, it is the overall ranking of the horizontal-level elements relative to the evaluation target, and determines the final effectiveness evaluation result.

[0111] Following the sequence of "Level L → Factor E → Indicator I → Criterion G → Target T", the process is divided into three steps:

[0112] Step 6-3-1: Calculate the index scores I at each level in the experiment. Ln The data was processed and calculated to obtain the index scores I for each factor. E :

[0113]

[0114] Where m is the total number of levels under this factor, p is the number of times each level appears in the designed experiment, and I Ln For the index scores corresponding to each level, w n Weights for each level;

[0115] Taking the data in Table 2 as an example, calculate factor E respectively. 11 The arithmetic mean of all indicators including L1, L2, L3, and L4 levels is used to calculate factor E. 12 E 13 E 14 E 15 The arithmetic mean of all indicators including L1 and L2 levels:

[0116] Table 4 Horizontal Level - Example of Indicator Scores

[0117]

[0118] By combining the scores of the level-level indicators with the weights of each level, a weighted calculation is performed to obtain the indicator scores of the factor level. Assume that for E... 11 The weights for L1 through L4 are 0.1, 0.15, 0.25, and 0.5, respectively; for E 12 ~E 15 The weights for L1 and L2 are 0.225 and 0.725, respectively.

[0119] Table 5. Factor Layer - Example of Indicator Scores

[0120]

[0121]

[0122] Step 6-3-2: Combine the score results of the factor layer indicators with the weight of each factor, perform a weighted calculation to obtain the score results of the indicator layer;

[0123] Assume E 11 ~E 15 The weights are 0.25, 0.20, 0.15, 0.15, and 0.25, respectively. After weighted calculation, the score of the indicator layer can be obtained.

[0124] Table 6. Factor Layer - Example of Indicator Scores

[0125]

[0126] Step 6-3-3: Using the Analytic Hierarchy Process (AHP), multiply the weights of the related elements layer by layer to obtain the index layer (I). 11 I 12 I 13 I 21 I 22 I 23 The overall hierarchical sorting of )

[0127] Then, based on the method in steps 6-3-1 to 6-3-2, I is obtained. 21 I 22 I 23 The indicator scores and overall hierarchical ranking;

[0128] By combining the scores of each indicator with their respective hierarchical ranking, a comprehensive score for the evaluation objective is obtained, leading to the final evaluation result.

[0129] The present invention starts from the experimental design method, constructs a heterogeneous multidimensional index system, quantifies the experimental conditions and integrates them into the performance evaluation numerical calculation method, without the need for external training data or domain knowledge preset, the evaluation results are transparent and traceable, and more objective and comprehensive.

[0130] This invention also provides an intelligent agent performance evaluation system based on the fusion of heterogeneous multidimensional indicator systems, comprising the following modules:

[0131] Heterogeneous multidimensional evaluation index system construction module: used to construct a heterogeneous multidimensional evaluation index system based on the determined evaluation objectives;

[0132] Experiment module: Used to design experimental schemes, set up experimental conditions, run experiments, and collect experimental data based on the constructed heterogeneous multidimensional evaluation index system;

[0133] The indicator calculation and scoring module is used to calculate and score indicators based on experimental data.

[0134] Evaluation Results Module: Used to calculate performance evaluation results based on the improved analytic hierarchy process.

[0135] The present invention also provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to perform the following steps:

[0136] Step 1: Define the evaluation objectives;

[0137] Step 2: Construct a heterogeneous, multidimensional evaluation index system based on the evaluation objectives;

[0138] Step 3: Design an experimental plan and set up experimental conditions based on the constructed heterogeneous multidimensional evaluation index system;

[0139] Step 4: Run the experiment and collect test data;

[0140] Step 5: Calculate and score the indicators based on the experimental data;

[0141] Step 6: Calculate the performance evaluation results based on the improved analytic hierarchy process.

[0142] The present invention also provides a computer-storable medium having a computer program stored thereon, wherein the computer program, when executed by a processor, performs the following steps:

[0143] Step 1: Define the evaluation objectives;

[0144] Step 2: Construct a heterogeneous, multidimensional evaluation index system based on the evaluation objectives;

[0145] Step 3: Design an experimental plan and set up experimental conditions based on the constructed heterogeneous multidimensional evaluation index system;

[0146] Step 4: Run the experiment and collect test data;

[0147] Step 5: Calculate and score the indicators based on the experimental data;

[0148] Step 6: Calculate the performance evaluation results based on the improved analytic hierarchy process.

[0149] The embodiments described above are merely one implementation method of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of the invention patent. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this patent application should be determined by the appended claims.

Claims

1. A method for evaluating the effectiveness of intelligent agents based on the fusion of heterogeneous multidimensional indicator systems, characterized in that, Includes the following steps: Step 1: Define the evaluation objectives; Step 2: Construct a heterogeneous, multidimensional evaluation index system based on the evaluation objectives; Step 3: Design an experimental plan and set up experimental conditions based on the constructed heterogeneous multidimensional evaluation index system; Step 4: Run the experiment and collect test data; Step 5: Calculate and score the indicators based on the experimental data; Step 6: Calculate the performance evaluation results based on the improved analytic hierarchy process.

2. The method for evaluating the effectiveness of intelligent agents based on the fusion of heterogeneous multidimensional index systems according to claim 1, characterized in that, The construction of a heterogeneous multidimensional evaluation index system based on the evaluation objectives in step 2 is specifically as follows: The heterogeneous multidimensional evaluation index system includes an objective layer, a criterion layer, an index layer, a factor layer, and a level layer; The target layer is the highest layer, which is used to characterize the determined evaluation target; The target layer is decomposed hierarchically downwards, based on the capabilities required to evaluate the target, to obtain several levels of evaluation criteria for the target, denoted as the criterion layer, where the first level is G1 to G... n The next level after G1 is G 11 ~G 1n G m The next level is Gm1~G mn ; The indicator layer is obtained by decomposing the last-level evaluation criteria of the criterion layer; Each evaluation criterion at the last level of the criterion layer has a corresponding condition structure, that is, influencing factors are set according to the evaluation criteria at the last level, forming a factor layer; Each factor in the factor layer is divided horizontally to form a horizontal layer.

3. The method for evaluating the effectiveness of intelligent agents based on the fusion of heterogeneous multidimensional index systems according to claim 2, characterized in that, The evaluation objective of the target layer is the detection effectiveness of the agent; The criteria layer includes target identification capability and target tracking capability; The metrics of the target recognition capability index layer include precision, recall, and F1 score; the metrics of the target tracking capability index layer include mean center position error, mean trajectory overlap rate, and multi-target tracking accuracy (MOTA). The factors of the target recognition capability factor layer include recognition distance, recognition category, target speed, visibility, and occlusion range; the factors of the target tracking capability factor layer include target speed, tracking number, and occlusion range.

4. The method for evaluating the effectiveness of intelligent agents based on the fusion of heterogeneous multidimensional index systems according to claim 2, characterized in that, The step 3, designing an experimental scheme based on the constructed heterogeneous multidimensional evaluation index system, specifically involves: Based on the factor levels defined by the last evaluation criteria of the criteria layer, different experimental schemes are designed and combined using orthogonal or uniform experimental methods to ensure that the level combinations of factors are uniformly comparable. Establish experimental conditions and set different experimental difficulties according to the experimental plan to meet the level combination of all factors in the designed experimental plan.

5. The method for evaluating the effectiveness of intelligent agents based on the fusion of heterogeneous multidimensional index systems according to claim 3, characterized in that, The step 5, which involves calculating and assigning scores based on experimental data, specifically involves: Step 5-1: Determine the values ​​of each indicator based on the collected experimental data: Accuracy: In this context, TP represents true positives and FP represents false positives. Recall rate: F1 score: Where Precision is the accuracy rate and Recall is the recall rate; Average center position error: Where t is the frame index, c p,t and c g,t Let be the predicted target center point coordinates and the actual target center point coordinates in each frame of data, respectively, and T be the total number of frames; Average trajectory overlap rate: Among them, B p,t and B g,t These represent the predicted bounding box and the ground truth bounding box regions in each frame of data; Multi-target tracking accuracy (MOTA): Among them, FP t 、FN t GT t IDSW t These represent the number of false positives, false negatives, true targets, and ID switching counts in each frame of data. Step 5-2: Design the scoring rules for the performance indicators, assign scores to each factor, and obtain different indicator results under each experimental scheme.

6. The method for evaluating the effectiveness of intelligent agents based on the fusion of heterogeneous multidimensional index systems according to claim 2, characterized in that, The calculation of the performance evaluation results based on the improved analytic hierarchy process in step 6 is specifically as follows: Step 6-1: Construct the evaluation matrix and perform a consistency evaluation; A rating matrix is ​​established based on the constructed heterogeneous multidimensional evaluation index system to characterize the relative importance of each element at each level of the heterogeneous multidimensional evaluation index system relative to a certain element or evaluation target at the next higher level. Step 6-2: Perform hierarchical single sorting, that is, determine the relative weight values ​​of elements in this level with respect to elements in the previous level: Among them, b ij,m q represents the value indicating the relative importance of elements at a certain level in the m-th rating matrix, and q represents the total number of rating matrices; Then, each column of the evaluation matrix is ​​normalized, the rows are summed, and normalized again to obtain the weights of the evaluation matrix at each level. n represents the number of levels in the constructed heterogeneous multidimensional evaluation index system; Step 6-3: Using the calculation results of each level of single ranking, calculate layer by layer in order from top to bottom to obtain the overall ranking of the final elements of the heterogeneous multidimensional index system relative to the evaluation target, and determine the final effectiveness evaluation result.

7. The method for evaluating the effectiveness of intelligent agents based on the fusion of heterogeneous multidimensional index systems according to claim 6, characterized in that, The hierarchical overall sorting in step 6-3 is specifically as follows: Step 6-3-1: Calculate the index scores I at each level in the experiment. Ln The data was processed and calculated to obtain the index scores I for each factor. E : Where m is the total number of levels under this factor, p is the number of times each level appears in the designed experiment, and I Ln For the index scores corresponding to each level, w n Weights for each level; By combining the index scores of the horizontal layer with the weights of each level, a weighted calculation is performed to obtain the index scores of the factor layer. Step 6-3-2: Combine the score results of the factor layer indicators with the weight of each factor, perform a weighted calculation to obtain the score results of the indicator layer; Step 6-3-3: Multiply the weights of the related elements layer by layer according to the analytic hierarchy process to obtain the overall hierarchical ranking of all indicators in the indicator layer. By combining the scores of each indicator with their respective hierarchical ranking, a comprehensive score for the evaluation objective is obtained, leading to the final evaluation result.

8. A system for evaluating the effectiveness of intelligent agents based on the fusion of heterogeneous multidimensional indicator systems, characterized in that, Includes the following modules: Heterogeneous multidimensional evaluation index system construction module: used to construct a heterogeneous multidimensional evaluation index system based on the determined evaluation objectives; Experiment module: Used to design experimental schemes, set up experimental conditions, run experiments, and collect experimental data based on the constructed heterogeneous multidimensional evaluation index system; The indicator calculation and scoring module is used to calculate and score indicators based on experimental data. Evaluation Results Module: Used to calculate performance evaluation results based on the improved analytic hierarchy process.

9. A computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1-7.

10. A computer-storable medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1-7.

Images