Intelligent decision algorithm evaluation method based on grading evaluation

Abstract

The invention provides an intelligent decision algorithm evaluation method based on grading evaluation, which comprises the following steps of: constructing a test scene of an intelligent decision algorithm, representing the test scene by adopting scene key representation elements, and extracting test factors under each scene key representation element; an evaluation track is formed based on the requirement for intelligent decision algorithm capability evaluation; according to the evaluation racing tracks of the intelligent decision algorithm, key influence factors of all the evaluation racing tracks are screened in the test factors, and a racing track factor pool of all the evaluation racing tracks is formed; correspondingly setting an initial level of a key influence factor in the track factor pool as a reference value according to the lowest requirement on the capability of the intelligent decision algorithm to be tested, and configuring a test scene according to the set key influence factor to form a reference test scene; and evaluating the grade of the intelligent decision algorithm in a test scene iterative optimization mode. The problem that the capability level of the intelligent decision algorithm cannot be comprehensively and accurately measured in the prior art is solved.

Description

Technical Field

[0001] The present invention belongs to the technical field of algorithm evaluation, and in particular relates to an intelligent decision-making algorithm evaluation method based on hierarchical evaluation. Background Art

[0002] Domestically, in terms of military intelligence testing and evaluation, with the winning rate as the basic metric, many domestic units have carried out a series of human-machine and machine-machine confrontation tests, and have successively held intelligent testing competitions. Abroad, in the military field, the U.S. military has relatively sufficient actual combat data, and there are already a large amount of data and thousands of combat rules to support the training of intelligent algorithms. In terms of intelligent testing, in typical projects such as AlphaAI, Alpha Dogfight, Loyal Wingman and ACE, a lot of testing work has been done on their intelligent algorithms through different testing platforms and methods, in order to promote the application of intelligent technology in the field of military operations through AI trust, AI assistance and AI autonomous route strategies. At the same time, in the ACE project to be promoted later, the U.S. military also hopes to evaluate the reliability of intelligent algorithms in actual applications by testing the pilots' trust in AI.

[0003] At present, the test scenarios for testing and evaluation of intelligent decision-making algorithms have not fully considered the generalization of intelligent decision-making algorithms. At the same time, the current evaluation of intelligent decision-making algorithms is mainly based on an evaluation index system. The construction of the evaluation index system is mainly based on expert experience, and the objectivity of the evaluation needs to be improved. Summary of the invention

[0004] In order to overcome the deficiencies in the prior art, the inventors have conducted intensive research and provided an intelligent decision-making algorithm evaluation method based on graded evaluation, and provided a sustainable evolutionary algorithm evaluation method for intelligent decision-making algorithms based on scenario classification and iterative confrontation, so as to solve the problem that the prior art cannot comprehensively and accurately measure the capability level of intelligent decision-making algorithms.

[0005] The technical solution provided by the present invention is as follows:

[0006] In the first aspect, an intelligent decision-making algorithm evaluation method based on hierarchical evaluation includes:

[0007] Construct test scenarios for intelligent decision-making algorithms, characterize test scenarios using scenario key characterization elements, and extract test factors under each scenario key characterization element;

[0008] Based on the demand for intelligent decision-making algorithm capability assessment, an assessment track is formed, and the assessment track is the assessment dimension of the intelligent decision-making algorithm;

[0009] According to the evaluation track of the intelligent decision-making algorithm, the key influencing factors of each evaluation track are screened from the test factors to form a track factor pool for each evaluation track;

[0010] According to the minimum requirements for the capabilities of the intelligent decision-making algorithm to be tested, the initial levels of the key influencing factors in the track factor pool are set as the benchmark values, and the test scenarios are configured according to the set key influencing factors to form benchmark test scenarios;

[0011] The level of intelligent decision-making algorithms is assessed through iterative optimization of test scenarios.

[0012] In a second aspect, an intelligent decision-making algorithm evaluation device based on hierarchical evaluation includes:

[0013] one or more processors;

[0014] a storage device for storing one or more programs,

[0015] When the one or more programs are executed by the one or more processors, the one or more processors implement the intelligent decision-making algorithm evaluation method based on hierarchical evaluation described in the first aspect.

[0016] In a third aspect, a readable storage medium stores a computer program, which, when executed by a processor, implements the intelligent decision-making algorithm evaluation method based on graded evaluation described in the first aspect.

[0017] In a fourth aspect, a computer program product is provided, comprising: a computer program (also referred to as code, or instruction), which, when executed, executes the intelligent decision-making algorithm evaluation method based on graded evaluation described in the first aspect.

[0018] According to the present invention, an intelligent decision-making algorithm evaluation method based on hierarchical evaluation has the following characteristics:

[0019] Beneficial effects:

[0020] (1) The present invention provides an intelligent decision-making algorithm evaluation method based on hierarchical evaluation. By extracting test factors from test scenarios and combining them with the test tracks of intelligent algorithms, the targeted evaluation of intelligent algorithms can be converted into a combination configuration of test factor levels. This can solve the problem of coupling of different aspects of capability characteristics in the evaluation of intelligent decision-making algorithms, and it is difficult to evaluate the specific capabilities of algorithms according to algorithm evaluation requirements.

[0021] (2) The present invention provides an intelligent decision-making algorithm evaluation method based on graded evaluation. Through the confrontation deduction of the intelligent decision-making algorithm in the test scenario, combined with the training evolution of the intelligent decision-making algorithm and the difficulty increase of the test scenario experimental factor combination, the intelligent decision-making algorithm and the test scenario can be alternately iterated, thereby giving a relative grade assessment in the iterative evolution of the intelligent decision-making algorithm, which can solve the problem that it is difficult to form a list of intelligent algorithm capabilities in the intelligent decision-making algorithm evaluation, and it is difficult to form a spectrum of intelligent decision-making algorithm capabilities later. Furthermore, this method can be used to drive the verification, improvement and application of intelligent decision-making algorithms in the fields of autonomous driving, game competition, etc. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Figure 1 This is a flow chart of an intelligent decision-making algorithm evaluation method based on hierarchical evaluation according to the present invention;

[0023] Figure 2 Schematic diagram of the key scene characterization elements in the path planning test scenario. DETAILED DESCRIPTION

[0024] The following detailed description of the present invention will make the features and advantages of the present invention more clear and explicit.

[0025] The word “exemplary” is used exclusively herein to mean “serving as an example, example, or illustration.” Any embodiment described herein as “exemplary” is not necessarily to be construed as preferred or advantageous over other embodiments.

[0026] The present invention provides an intelligent decision-making algorithm evaluation method based on hierarchical evaluation, such as Figure 1 As shown, the following steps are included:

[0027] Step 1: Construct a test scenario for the intelligent decision-making algorithm, characterize the test scenario using the key characterization elements of the scenario, and extract the experimental factors under the key characterization elements of each scenario. First, construct a test scenario for the intelligent decision-making algorithm to be evaluated, then conduct a study on the key characterization elements of the scenario based on the test scenario, and further extract experimental factors based on the key characterization elements of the scenario.

[0028] (1.1) Construct test scenarios based on the application scenarios of intelligent decision-making algorithms. Taking the target allocation algorithm as an example, construct a typical target allocation scenario based on expert experience and the application requirements of the intelligent decision-making algorithm. Taking the adversarial decision-making algorithm as an example, construct a red-blue adversarial scenario based on the application requirements of the intelligent decision-making algorithm.

[0029] (1.2) Establish the key characterization elements of the test scenario, analyze the attribute information under each key characterization element of the scenario, and extract the test factors under each key characterization element of the scenario based on the attribute information.

[0030] For the test scenarios of intelligent decision-making algorithms, five key scene representation elements are defined: tasks, roles, capabilities, agents, and environments. The tasks determine the task objectives and constraints of the test scenarios; the roles are the bridges that establish the relationship between tasks and agents; capabilities are the different ability attributes possessed by roles, and there is a many-to-many mapping relationship between capabilities and roles; and agents are the intelligent strategies adopted by roles.

[0031] Extract the attribute information contained in the key characterization elements of each scenario. The key characterization elements of the scenario are the top-level design of the test scenario, and the attributes are the bottom-level design of the test scenario. Each key characterization element of the scenario contains several attributes to describe the test scenario in detail. For example, in the path planning test scenario, the task element contains the attributes of the shortest total maneuvering distance and the shortest total maneuvering time; the role element contains the attribute of the number of vehicles to be planned; the ability element contains the attribute of the speed of the red team’s vehicle; the intelligent agent element contains the attributes of satellite detection strategy and detection interval; the environmental element contains the distribution range of the starting point of the red team’s vehicle, the distribution range of the target area, the straight-line distance between the starting point and the end point, and the terrain and landform attributes, see Figure 2 .

[0032] According to the attributes of the key characterization elements of the scene, test factors are extracted, and the attributes are quantitatively characterized through test factors to form a normalized characterization model of the test scene. For example, for the red-blue confrontation test scene, the attributes of the red and blue forces' composition and type ratio are extracted from the role elements, and four test factors are extracted: the number of red ammunition, the ratio of red ammunition, the number of blue ships, and the ratio of blue ship types. For each test factor, a reasonable parameter value range is given based on expert experience. For the exemplary path planning test scene, the key characterization elements and test factors of the scene are shown in Table 1 below.

[0033] Table 1

[0034]

[0035]

[0036] Step 2: Based on the demand for intelligent decision-making algorithm capability assessment, form an assessment track. The assessment track refers to the assessment dimension of the intelligent decision-making algorithm. Each assessment track examines and assesses one aspect of the capability characteristics of the intelligent decision-making algorithm. Typical assessment tracks for intelligent decision-making algorithms include autonomy, learning, collaboration, and task completion capabilities. For example, in the target allocation scenario, the assessment track can be selected as: the allocation effect of the target allocation intelligent decision-making algorithm (belonging to the task completion capability).

[0037] Step 3: According to the evaluation track of the intelligent decision-making algorithm, select the key influencing factors of each evaluation track from the experimental factors extracted in step 1 to form a track factor pool for each evaluation track.

[0038] Preferably, the experimental design factors related to each evaluation track are screened based on expert experience to form a preliminary pool of track factors, and then the key influencing factors in the preliminary pool of track factors are screened using a sensitivity analysis method to form a track factor pool. Among them, the key influencing factors are different from the experimental design factors in step 1 in that the key influencing factors can reflect the testing requirements for the intelligent decision-making algorithm and reflect the characteristics of the track. At the same time, the performance of the intelligent decision-making algorithm in the corresponding track is more sensitive to changes in the key influencing factors. If there are N evaluation tracks for the evaluation of the intelligent decision-making algorithm, there are N related track factor pools.

[0039] The implementation steps of the sensitivity analysis method include: for a certain test factor in the preliminary pool of track factors, keep other test factors in the track unchanged, continuously increase the level of the test factor, carry out simulation deduction, and record the changes in the evaluation results under the corresponding test factor level. Furthermore, by analyzing the mean and range of task scores at different levels, calculate the sensitivity of the evaluation results under the sample test to the change of the test factor level. Solve the sensitivity of the test factors in the preliminary pool of track factors one by one, set the sensitivity threshold, and include the factors whose sensitivity meets the threshold condition into the track factor combination.

[0040] Step 4: According to the minimum indicator requirements of the intelligent decision-making algorithm test scenario, set the initial level of the key influencing factors in the track factor pool as the benchmark value, and configure the test scenario according to the key influencing factors, such as the blue team elements, red team elements, environmental elements, and blue team strategies in the red-blue confrontation test scenario to form a benchmark test scenario (0-level test scenario). Conduct deductions based on the benchmark test scenario and the intelligent decision-making algorithm to obtain the algorithm confrontation results.

[0041] Step 5: Evaluate the level of the intelligent decision-making algorithm through iterative optimization of test scenarios.

[0042] Step 5.1, set the task completion threshold s (for example, when the red team and the blue team are in confrontation, the damage ratio of the red team to the blue team reaches s); use the benchmark test scenario as the initial test scenario, and conduct n tests under the constructed test scenario. If the average task completion level of the intelligent decision-making algorithm exceeds the threshold s in n tests, the intelligent decision-making algorithm to be evaluated is deemed to have passed the test of the corresponding test scenario level. Further adjust the level values ​​of the key influencing factors in the track factor pool to form a high-level level, increase the difficulty of the decision-making scenario, and continue to conduct tests.

[0043] Step 5.2: If the average task completion level of the intelligent decision-making algorithm does not exceed the threshold s in n tests, the algorithm level of the intelligent decision-making algorithm is determined to be the difficulty level of the corresponding test scenario.

[0044] If the ability of the intelligent decision-making algorithm is improved with the training evolution of the intelligent decision-making algorithm, repeat the above steps 5.1 to 5.2 until the test level of the optimized intelligent decision-making algorithm is evaluated. When the intelligent decision-making algorithm continues to evolve and forms different versions of the algorithm, this method can iteratively give the level assessment of the intelligent decision-making algorithm by continuously improving the test factor combination of the test scenario to form a gradient difficulty test scenario set with increasing difficulty.

[0045] The present invention also provides an intelligent decision-making algorithm evaluation device based on hierarchical evaluation, comprising:

[0046] one or more processors;

[0047] a storage device for storing one or more programs,

[0048] When the one or more programs are executed by the one or more processors, the one or more processors implement the intelligent decision-making algorithm evaluation method based on hierarchical evaluation described in the first aspect.

[0049] The present invention also provides a readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the intelligent decision-making algorithm evaluation method based on graded evaluation described in the first aspect.

[0050] The readable storage medium includes, but is not limited to, various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0051] The present invention also provides a computer program product, which includes: a computer program (also referred to as code, or instruction), which, when executed, executes the intelligent decision-making algorithm evaluation method based on graded evaluation described in the first aspect.

[0052] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium, for example, the computer instructions can be transmitted from a website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (digital subscriber line, DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, server or data center.

[0053] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0054] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the above-described device and product can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here.

[0055] The present invention has been described in detail above in conjunction with specific implementations and exemplary examples, but these descriptions cannot be understood as limiting the present invention. Those skilled in the art understand that, without departing from the spirit and scope of the present invention, a variety of equivalent substitutions, modifications or improvements may be made to the technical solution of the present invention and its implementation methods, all of which fall within the scope of the present invention. The scope of protection of the present invention shall be subject to the attached claims.

[0056] The contents not described in detail in the specification of the present invention belong to the common knowledge of those skilled in the art.

Claims

1. An intelligent decision-making algorithm evaluation method based on hierarchical evaluation, characterized in that: include: Construct test scenarios for intelligent decision-making algorithms, characterize test scenarios using scenario key characterization elements, and extract test factors under each scenario key characterization element; Based on the demand for intelligent decision-making algorithm capability assessment, an assessment track is formed, and the assessment track is the assessment dimension of the intelligent decision-making algorithm; According to the evaluation track of the intelligent decision-making algorithm, the key influencing factors of each evaluation track are screened from the test factors to form a track factor pool for each evaluation track; According to the minimum requirements for the capabilities of the intelligent decision-making algorithm to be tested, the initial levels of the key influencing factors in the track factor pool are set as the benchmark values, and the test scenarios are configured according to the set key influencing factors to form benchmark test scenarios; The level of intelligent decision-making algorithms is assessed through iterative optimization of test scenarios.

2. The intelligent decision-making algorithm evaluation method based on hierarchical evaluation according to claim 1 is characterized in that: The steps of constructing a test scenario for an intelligent decision-making algorithm, characterizing the test scenario using scenario key characterization elements, and extracting test factors under each scenario key characterization element include: Construct test scenarios based on the application scenarios of intelligent decision-making algorithms; Establish the scene key characterization elements of the test scene, analyze the attribute information under each scene key characterization element, and extract the experimental factors under each scene key characterization element according to the attribute information. The attribute information is used to describe the test scene in detail, and the experimental factors are used to quantitatively characterize the attributes.

3. The intelligent decision-making algorithm evaluation method based on hierarchical evaluation according to claim 1 is characterized in that: The key scene characterization elements of the test scenario are tasks, roles, capabilities, agents and environments.

4. The intelligent decision-making algorithm evaluation method based on hierarchical evaluation according to claim 1 is characterized in that: In the step of screening the key influencing factors of each evaluation track from the experimental factors according to the evaluation track of the intelligent decision-making algorithm to form a track factor pool for each evaluation track, the experimental design factors related to each evaluation track are screened based on expert experience to form a track factor preliminary pool, and then the sensitivity analysis method is used to screen the key influencing factors in the track factor preliminary pool to form a track factor pool.

5. The intelligent decision-making algorithm evaluation method based on hierarchical evaluation according to claim 1 is characterized in that: The step of evaluating the level of the intelligent decision-making algorithm by iterative optimization of the test scenario includes: Step 5.1, set the task completion threshold, use the benchmark test scenario as the initial test scenario, and conduct multiple tests under the constructed test scenario; if the average task completion level of the intelligent decision-making algorithm exceeds the threshold in multiple tests, the intelligent decision-making algorithm to be evaluated is deemed to have passed the test of the corresponding test scenario level; adjust the level values ​​of the key influencing factors in the track factor pool to form a high-level level, increase the difficulty of the decision-making scenario, and continue to conduct tests; Step 5.2: If the average task completion level of the intelligent decision-making algorithm does not exceed the threshold in multiple tests, the algorithm level of the intelligent decision-making algorithm is determined to be the difficulty level of the corresponding test scenario.

6. The intelligent decision-making algorithm evaluation method based on hierarchical evaluation according to claim 5 is characterized in that: If the intelligent decision-making algorithm obtains improved capabilities after training evolution, repeat the above steps 5.1 to 5.2 until the test level of the optimized intelligent decision-making algorithm is assessed.

7. An intelligent decision-making algorithm evaluation device based on hierarchical evaluation, characterized in that: include: one or more processors; a storage device for storing one or more programs, When the one or more programs are executed by the one or more processors, the one or more processors implement the intelligent decision-making algorithm evaluation method based on hierarchical evaluation as described in one of claims 1 to 6.

8. A readable storage medium, characterized in that: A computer program is stored thereon, and when the program is executed by a processor, the intelligent decision-making algorithm evaluation method based on hierarchical evaluation as described in one of claims 1 to 6 is implemented.

9. A computer program product, characterized in that The computer program product comprises: a computer program, which, when being executed, executes the intelligent decision-making algorithm evaluation method based on hierarchical evaluation as described in any one of claims 1 to 6.

Images