A scenario-level evaluation method and apparatus for complex intelligent decision-making systems based on agent walking.

Abstract

This invention relates to the field of electronic digital data processing, and in particular to a method and apparatus for scenario-level evaluation of complex intelligent decision-making systems based on agent-based roaming. The method includes the following steps: generating an evaluation scenario graph based on a knowledge graph, the scenario graph containing evaluation scenarios covering the evaluation objectives of the complex intelligent decision-making system; the evaluation agent actively roaming in the evaluation scenario graph according to a path planned by a path planning algorithm, evaluating the complex intelligent decision-making system using an evaluation dataset, and analyzing the evaluation results using an evaluation tool to output evaluation indicators; calculating the coverage rate of evaluation scenarios in the evaluation scenario graph after the current round of agent roaming, and if the coverage rate is less than a coverage threshold, removing the traversed evaluation scenarios in the evaluation scenario graph, and proceeding to the next round of agent-based roaming. This invention improves the comprehensiveness of evaluation scenarios and the efficiency of evaluation scenario execution.

Description

Technical Field

[0001] This invention relates to the field of electronic digital data processing, and in particular to a scenario-level evaluation method and apparatus for complex intelligent decision-making systems based on agent walking. Background Technology

[0002] With the rapid development of artificial intelligence technology, technologies such as intelligent decision-making, intelligent diagnosis, expert systems, and embodied intelligence have permeated various fields. An intelligent decision-making system, as a complex intelligent system, comprises multiple components, including intelligent perception modules, intelligent decision support modules, and intelligent execution modules.

[0003] Ordinary intelligent decision-making systems typically perform only single-function intelligent decisions. When evaluating such systems, batch software testing can achieve a relatively comprehensive assessment. However, complex intelligent decision-making systems contain multiple intelligent modules, such as situational analysis, information fusion, anomaly detection, and multi-criteria decision analysis. Each of these modules is essentially a separate, ordinary intelligent decision-making system. In such cases, batch software testing alone is insufficient to evaluate all intelligent modules. Therefore, a method of designing evaluation scenarios is employed to assess the complex intelligent decision-making system within these scenarios.

[0004] Traditional evaluation methods for intelligent decision-making systems typically rely on manually designed evaluation scenarios and test cases. However, this approach has several shortcomings. For example, manually designed scenarios may be subjective and limited, failing to comprehensively cover all key evaluation aspects of complex intelligent decision-making systems. Furthermore, manually designed evaluation scenarios are inefficient during execution and struggle to meet practical application needs. Summary of the Invention

[0005] To address the issues of subjectivity in evaluation scenario design and low efficiency during scenario execution, this invention provides a scenario-level evaluation method and apparatus for complex intelligent decision-making systems based on agent walking.

[0006] This invention provides a scenario-level evaluation method for complex intelligent decision-making systems based on agent walking, employing the following technical solution:

[0007] A scenario-level evaluation method for complex intelligent decision-making systems based on agent walking includes the following steps:

[0008] An evaluation scenario graph is generated based on a knowledge graph. The scenario graph contains evaluation scenarios that cover the evaluation objectives of the complex intelligent decision-making system. An evaluation agent carrying an evaluation dataset and an evaluation tool actively walks in the evaluation scenario graph. The evaluation agent evaluates the intelligent modules of the complex intelligent decision-making system through the evaluation dataset and obtains evaluation results. The evaluation tool analyzes the evaluation results and outputs evaluation indicators.

[0009] After the evaluation agent completes its walk in the current round, if the coverage rate of the evaluation scene in the evaluation scene graph is less than the coverage rate threshold, the evaluation scene that has been traversed in the evaluation scene graph is removed, and the evaluation agent's active walk is performed in the next round.

[0010] In one specific implementation, before the evaluation agent actively roams the evaluation scene graph, a path planning algorithm is used to plan the roaming path of the evaluation agent, so that the roaming path of the evaluation agent includes as many nodes as possible in the evaluation scene graph.

[0011] In one specific implementation, the path planning algorithm plans the walking path of the evaluation agent based on the evaluation task, with the goal of fully triggering the intelligent modules of the complex intelligent decision-making system.

[0012] In one specific implementation, before the path planning algorithm plans the walking path of the evaluation agent, it calculates the weight of each node in the evaluation scenario graph.

[0013] In one specific implementation, after the path planning algorithm plans the walking path of the evaluation agent, it clips the evaluation scene graph to generate an evaluation scene subgraph for the evaluation agent to walk on.

[0014] In one specific implementation, when the evaluation scene graph is generated, the completeness of the evaluation scene graph with respect to the knowledge graph is calculated, and the completeness is compared with a preset completeness threshold. If the completeness is less than the completeness threshold, it indicates that the evaluation scene needs to be generated again.

[0015] In one specific implementation, there are multiple evaluation agents in the evaluation scenario graph. Before actively roaming the evaluation scenario graph, each evaluation agent performs computational resource scheduling through a resource scheduling algorithm.

[0016] In one specific implementation scheme, the resource scheduling algorithm allocates the resources and working hours consumed by the evaluation agent during its walk as follows:

[0017] Each of the evaluated agents is treated as a particle, and an initial population is generated.

[0018] Calculate the fitness of each particle according to the evaluation task;

[0019] The particles were screened for crossover and mutation.

[0020] Update the position and velocity of the particles until the optimal evaluation agent walk scheme is obtained.

[0021] In one specific implementation scheme, when the evaluation agent walks through the evaluation scenario graph, it first evaluates each intelligent module of the complex intelligent decision-making system, and then performs a comprehensive evaluation of the complex intelligent decision-making system.

[0022] This invention also provides a scenario-level evaluation device for complex intelligent decision-making systems based on agent walking, employing the following technical solution:

[0023] A scenario-level evaluation device for complex intelligent decision-making systems based on agent walking, comprising:

[0024] One or more processors; memory;

[0025] And one or more computer programs, wherein the one or more computer programs are stored in the memory, and the one or more computer programs include instructions that, when executed by the processor, cause the scenario-level evaluation device for complex intelligent decision-making systems based on agent walking to perform the aforementioned scenario-level evaluation method for complex intelligent decision-making systems based on agent walking.

[0026] In summary, the present invention has the following beneficial effects:

[0027] 1. By automatically generating evaluation scenario diagrams through knowledge graphs, the key evaluation links of complex intelligent decision-making systems can be fully covered to make up for omissions that may occur in subjective human design.

[0028] 2. Based on the ability of the evaluation agent to actively roam, data acquisition of complex intelligent decision-making systems can be realized, thus solving the problem of low efficiency in evaluation task execution.

[0029] 3. A resource scheduling method based on genetic particle swarm optimization is proposed to realize resource scheduling and allocation for multiple evaluation agents in the evaluation process of complex intelligent decision-making systems, thereby solving the problem of insufficient resources in multi-agent evaluation tasks.

[0030] 4. After completing the current round of evaluation, calculate the coverage of the evaluation scenarios traversed by the evaluation agent. By comparing it with the preset coverage threshold, traverse the scenarios that have not been traversed, thereby conducting a comprehensive evaluation of the complex intelligent decision-making system. Attached Figure Description

[0031] Figure 1 This is an overall flowchart of a scenario-level evaluation method for complex intelligent decision-making systems based on agent movement.

[0032] Figure 2 This is an architecture diagram of a scenario-level evaluation method for complex intelligent decision-making systems based on agent roaming. Detailed Implementation

[0033] The following combination Figures 1-2 The present invention will be further described in detail below.

[0034] A scenario is a comprehensive representation of a series of interrelated events, activities, and situations within a specific time and space. It includes elements such as the physical environment, participants, interaction methods, and background information, and is used to describe a complete situation with internal logic.

[0035] Scene elements refer to the basic components that make up a scene. These elements work together to form a complete and logically consistent context, which typically includes the physical environment, participants, interaction methods, and background information.

[0036] Scenario relationships refer to the ways in which different scenarios connect and interact, describing how one scenario transitions to another, and the logical and causal relationships between them. These relationships can be chronological, causal, or conditionally triggered.

[0037] A scenario graph is a graph-like structure formed by organizing multiple interconnected scenarios according to certain relationships during the operation of a target object. Each node represents a specific scenario, and the edges between scenario nodes represent the relationships between scenarios. Scenario graphs are used to describe and analyze the state and behavior changes of a target object under different conditions.

[0038] The following describes the invention in further detail using an autonomous driving system as the complex intelligent decision-making system being evaluated. The autonomous driving system includes intelligent modules such as a temperature sensing module, a distance sensing module, a weather sensing module, a decision-making module, and an execution module. The temperature sensing module senses the temperature inside the vehicle; the distance module measures the distance between the autonomous vehicle and the lane edge, the distance to the vehicle in front, and the distance to obstacles in front and behind; the weather sensing module senses the current season and weather; the decision-making module acquires data from the temperature sensing module, distance sensing module, weather sensing module, etc., as well as current road conditions and the current distance to the destination, and plans the autonomous vehicle's driving path and driving status, generating driving plans and driving decisions for the execution module to execute.

[0039] The scenario-level evaluation method for complex intelligent decision-making systems based on agent walking includes the following steps:

[0040] S100 generates evaluation scenario diagrams based on knowledge graphs.

[0041] Knowledge graphs are structured semantic knowledge bases used to rapidly describe concepts and their relationships in the physical world.

[0042] Knowledge graphs effectively process, handle, and integrate complex document data, transforming it into simple and clear "entity-relationship-entity" triples. Finally, they aggregate knowledge from areas such as events and time-series information, enabling rapid knowledge response and reasoning. Since the evaluation object is an autonomous driving system, a knowledge graph of autonomous driving systems and related fields is selected to generate a graph-structured evaluation scenario graph, where each evaluation scenario is a node in the graph.

[0043] Table 1 below shows some knowledge in the knowledge graph. Based on the knowledge in Table 1, different driving scenarios that the autonomous driving system needs to deal with can be constructed. The driving scenario is the evaluation scenario.

[0044] Table 1

[0045]

[0046] Based on the knowledge in Table 1, the evaluation scenario diagram shows that different evaluation scenario designs and different routes can lead to various decisions. When the autonomous driving system generates driving plans and driving decisions, the intelligent modules called out are different. Furthermore, after the autonomous driving system generates driving plans and driving decisions, the driving plans and driving decisions executed by the execution module, as well as the motors and other components acted upon by the execution module, are also different.

[0047] By changing the knowledge used to construct evaluation scenarios based on the knowledge graph, new scenarios are generated, thereby constructing as many evaluation scenarios as possible. In other words, as many nodes as possible are generated in the evaluation scenario graph. The changes in knowledge form the scenario relationships between evaluation scenarios, that is, the connections between nodes in the evaluation scenario graph.

[0048] Using the data in Table 1 as an example, we first construct a driving scenario for roads in normal winter weather. In this evaluation scenario, the autonomous driving system automatically makes decisions and plans the vehicle's route based on the simulated environment, enabling the vehicle to reach its destination. Then, we construct a driving scenario for roads in severe winter weather. The only difference between this scenario and the normal winter weather scenario is that the weather changes from sunny to sleet to heavy snow, causing the autonomous driving system to change its strategy from K1 to K3, and consequently, the intelligent modules it calls. This process continues, gradually changing the knowledge in the evaluation scenarios by controlling variables to construct new evaluation scenarios. This results in as many evaluation scenarios as possible, ensuring that the constructed scenarios cover all possible combinations of knowledge in the knowledge graph. In different evaluation scenarios, as the knowledge changes, the intelligent modules called by the autonomous driving system differ, and the effectiveness of these modules also varies. Having as many evaluation scenarios as possible allows the autonomous driving system to call as many intelligent modules as possible and maximize their effectiveness, facilitating a comprehensive evaluation of the autonomous driving system's performance in subsequent evaluations.

[0049] The evaluation scenario in the evaluation scenario diagram represents the completeness of the effectiveness of the intelligent modules in the complex intelligent decision-making system. The coverage of the evaluation scenario can be assessed by comparing the completeness to a preset completeness threshold. For example, if the completeness is greater than or equal to the threshold, it indicates that the generated evaluation scenario allows the intelligent modules of the autonomous driving system to fully realize their potential, and the generation of evaluation scenarios can be stopped for further processing. Otherwise, it indicates that some intelligent modules are not performing effectively or are not performing fully, and the generation of evaluation scenarios continues to further enhance the effectiveness of the intelligent modules in the autonomous driving system. This significantly reduces the subjectivity and limitations in the design of evaluation scenarios.

[0050] S200 evaluates the active walk of the assessment agent to assess complex intelligent decision-making systems. A path planning algorithm is used to plan the agent's walk path.

[0051] The weight of each node in the evaluation scenario graph is calculated based on the complexity of the evaluation scenario. The higher the complexity of the evaluation scenario, the greater the weight of the node. The complexity of the evaluation scenario depends on the number of scenario elements and the position of the node in the evaluation scenario graph. The more scenario elements there are, the greater the out-degree and in-degree of the current node in the context, and the closer it is to the endpoint node, the higher the complexity of the evaluation scenario.

[0052] The evaluation agent is an agent with the ability to actively roam. In the evaluation scenario graph, the evaluation agent generates the optimal active roaming path in the evaluation scenario graph according to the preset evaluation task, thereby completing the evaluation task.

[0053] The evaluation scenario diagram sets up multiple evaluation agents, each carrying a corresponding evaluation dataset, evaluation tool, and evaluation standard set. The evaluation agents use the evaluation dataset to evaluate each intelligent module of the complex intelligent decision-making system, obtaining evaluation results. The evaluation tool analyzes the evaluation results in conjunction with the evaluation standard set, outputting evaluation indicators. After dimensionless transformation of the evaluation indicators, they are aggregated to obtain a comprehensive evaluation of the complex intelligent decision-making system.

[0054] The path planning algorithm, based on the evaluation scenario graph and node weights, aims to fully leverage the efficiency of the intelligent modules in the complex intelligent decision-making system. It plans the traversal path for each evaluation agent, calculating a path with a start and end point. This path, connected by lines in a graph structure, traverses multiple evaluation scenario nodes and represents the optimal or most feasible path for the evaluation agent to complete the evaluation task. The evaluation agent traverses the evaluation scenario nodes along the path, evaluating the complex intelligent decision-making system, and reaches the end point to complete the evaluation task.

[0055] Specifically, in order to obtain the evaluation scenario traversal path, i.e. the evaluation agent's traversal path, from the evaluation scenario roadmap, the evaluation scenario roadmap can be regarded as a Hidden Markov Model (HMM), and the Viterbi Algorithm can be used to obtain the optimal evaluation scenario path.

[0056] However, the typical Viterbi algorithm aims to find the shortest traversal path with the highest joint probability. Therefore, to encourage paths to contain as many nodes as possible, a penalty function is designed to penalize paths containing fewer nodes.

[0057] The path planning method for the evaluation scenario roadmap of complex intelligent systems is as follows:

[0058] (1) Construct an assessment scenario roadmap:

[0059] The evaluation scenario roadmap is represented as a directed graph. ,in It is a set of evaluation scenario nodes, representing the evaluation scenario; It is a set of edges, representing the relationships between different scenes; It is the node weight, which represents the importance of the scene node.

[0060] Each node It includes several evaluation elements (such as time, location, weather, etc.), and each evaluation element has a probability of occurrence.

[0061] (2) Define the Hidden Markov Model:

[0062] State set Corresponding to the set of evaluation scene nodes .

[0063] Initial state probability This represents the probability that the system starts from a certain scenario.

[0064] Set of state transition probability matrices This represents the set of probabilities of transitioning from one scene to another.

[0065] Set of observation probability matrices It represents the set of probabilities of observing certain evaluation elements in a certain scenario.

[0066] (3) Viterbi algorithm:

[0067] The Viterbi algorithm is used to find the most likely sequence of states (i.e., to evaluate the scene path), given a sequence of observations. Let... Indicates at time Status is The probability of finding the optimal path is given by the formula:

[0068]

[0069] Where N represents the total number of states, which is also the total number of nodes. , ; From state Transition to state The probability of that, that is, from the evaluation scenario diagram, the first... The node moves to the first... The probability of each node. ; It is in state Observed evaluation elements The probability of, i.e., the probability of, evaluating the first in the scenario graph Evaluation elements were observed in the evaluation scenario of each node. The probability, .

[0070] (4) Penalty function:

[0071] To traverse more nodes, a penalty function is introduced. This function penalizes paths containing fewer nodes. Let the path length be... The total number of nodes is The penalty function can then be defined as:

[0072]

[0073] In the Viterbi algorithm, a penalty function is added when updating the path probability, resulting in the following formula: .

[0074] It's easy to understand that after each path planning operation for an evaluation agent, the path planning algorithm prunes the evaluation scene graph, generating a subgraph of the evaluation scene graph representing the agent's path, which the agent then traverses. The pruned subgraph is removed from the original evaluation scene graph, and the next path planning operation begins, pruning a new subgraph from the remaining evaluation scene graph, and this process is repeated.

[0075] In the evaluation scenario diagram of the autonomous driving system, the evaluation agent's evaluation tasks include the intelligence of the autonomous driving system under weather changes, route changes, and road condition changes. Each evaluation agent corresponds to one or more evaluation tasks. Taking the evaluation agent's evaluation task as the intelligence of the autonomous driving system under weather changes, the evaluation scenario diagram is constructed using Table 1 as an example. The path planning algorithm constructs the following travel paths: winter normal weather road driving scenario - winter rainy weather road driving scenario - winter snowy weather road driving scenario. The evaluation agent travels through these three scenarios sequentially, evaluating the intelligent modules of the autonomous driving system. After completing the evaluation of each scenario, a comprehensive evaluation is output. When evaluating the autonomous driving system in the scenario, the evaluation agent evaluates each intelligent module called by the autonomous driving system and outputs the corresponding evaluation results. The evaluation results are analyzed using evaluation tools to output evaluation indicators. Multiple evaluation indicators are dimensionless and aggregated to output the comprehensive evaluation of the autonomous driving system in that evaluation scenario. After completing the travel through the three scenarios, the three comprehensive evaluations are aggregated to obtain the comprehensive evaluation result.

[0076] Because there are multiple evaluation agents in the evaluation scenario graph, a large number of evaluation paths are generated, which consumes a significant amount of computing resources. By scheduling the resources of the evaluation agents using resource scheduling algorithms, the waste of computing resources can be reduced, and the allocation of computing resources during the agent's movement and evaluation process can be made more rational.

[0077] For ease of understanding, the resource scheduling algorithm of this invention is specifically a genetic particle swarm optimization algorithm. In some other embodiments, it can be an ant colony optimization algorithm, a hybrid genetic algorithm, etc.

[0078] Specifically, each evaluation agent is treated as a particle, and an initial population is generated randomly or based on experience and initialized.

[0079] Based on the requirements of the evaluation task, the fitness of each particle is evaluated using fitness evaluation metrics. These metrics include time consumed and resources consumed.

[0080] Particles with high fitness are selected and subjected to crossover and mutation to generate a new population. Methods for selecting particles include roulette wheel selection and tournament selection, while crossover methods include single-point crossover and multi-point crossover.

[0081] Update the position and velocity of the particles.

[0082] Through the formula:

[0083]

[0084] To calculate the particle's current position, in the above formula, Indicates the first Individual particles Location at any given moment Indicates the first The particle in the first The speed of time Indicates the first The particle in the first The position at any given moment.

[0085] Through the formula:

[0086]

[0087] To calculate the particle's current velocity, in the above formula, Indicates inertia weight, and The acceleration constant, and It is a random constant. Indicates the first The optimal position of each individual particle. This indicates the globally optimal position.

[0088] By continuously updating the position and velocity of particles, the movement path and resource consumption of particles are optimized, thereby increasing the movement speed of particles and reducing particle resource consumption. This improves the walking efficiency of the evaluation agent and reduces the resource consumption of the evaluation agent during walking, thus obtaining the optimal resource scheduling scheme that maximizes the execution efficiency of the evaluation task and minimizes resource consumption.

[0089] Based on the optimal evaluation agent's traversal plan, resource scheduling and allocation are performed on the evaluation agent.

[0090] S300 calculates the coverage rate for the current round.

[0091] The coverage rate is the proportion of evaluation scenarios that the evaluation agent has actively traversed in the current round out of all evaluation scenarios in the evaluation scenario graph. The coverage rate is compared with a pre-input coverage rate threshold. If the coverage rate is greater than or equal to the coverage rate threshold, it indicates that the evaluation of the complex intelligent decision-making system is comprehensive enough. A multi-dimensional comprehensive evaluation of the complex intelligent decision-making system is then performed, and the final multi-dimensional comprehensive evaluation result is output. Otherwise, the evaluation scenarios that have been traversed in the evaluation scenario graph are removed, and step S200 is repeated to conduct a new round of active traversal by the evaluation agent.

[0092] The present invention also discloses a scenario-level evaluation device for a complex intelligent decision-making system based on agent walking, comprising one or more processors and a memory;

[0093] And one or more computer programs, wherein the one or more computer programs are stored in memory, and the one or more computer programs include instructions that, when executed by a processor, cause the scenario-level evaluation device for complex intelligent decision-making systems based on agent walking to perform the aforementioned scenario-level evaluation method for complex intelligent decision-making systems based on agent walking.

[0094] The above are all preferred embodiments of the present invention and are not intended to limit the scope of protection of the present invention. Therefore, all equivalent changes made in accordance with the structure, shape and principle of the present invention should be covered within the scope of protection of the present invention.

Claims

1. A method for evaluating a complex intelligent decision system scenario based on agent walking, characterized in that: The method comprises the following steps: generating an evaluation scene graph based on a knowledge graph, the scene graph containing evaluation scenes covering evaluation targets of the complex intelligent decision system; an evaluation agent carrying an evaluation dataset, an evaluation tool and an evaluation standard set actively walks in the evaluation scene graph, the evaluation agent evaluates intelligent modules of the complex intelligent decision system through the evaluation dataset, obtains an evaluation result, and the evaluation tool analyzes the evaluation result in combination with the evaluation standard set to output an evaluation index; after the evaluation agent walks in the current round, if the coverage rate of the evaluation scenes in the evaluation scene graph is less than a coverage rate threshold, the evaluation scenes that have been traversed in the evaluation scene graph are removed, and the next round of active walking of the evaluation agent is performed; when generating the evaluation scene graph based on the knowledge graph, new scenes are generated based on changes in knowledge when constructing the evaluation scenes, so as to construct as many evaluation scenes as possible; when the evaluation scene graph is generated, the completeness of the evaluation scene graph is calculated, and the completeness is compared with a preset completeness threshold; if the completeness is less than the completeness threshold, it is indicated that the evaluation scenes need to be continuously generated; before the evaluation agent actively walks in the evaluation scene graph, the walking path of the evaluation agent is planned through a path planning algorithm according to an evaluation task, so as to fully exert the efficiency of the intelligent modules of the complex intelligent decision system; before the path planning algorithm plans the walking path of the evaluation agent, the weight of each node in the evaluation scene graph is calculated; the path planning algorithm plans the walking path of the evaluation agent by introducing a penalty function to punish the path containing as few nodes as possible, so that the walking path of the evaluation agent contains as many nodes in the evaluation scene graph as possible.

2. The method of claim 1, wherein: After the path planning algorithm plans the walking path of the evaluation agent, the evaluation scene graph is cropped to generate an evaluation scene subgraph for the evaluation agent to walk.

3. The method of claim 1, wherein: There are multiple evaluation agents in the evaluation scene graph, and the evaluation agents actively walk in the evaluation scene graph by calculating resource scheduling through a resource scheduling algorithm.

4. The method of claim 3, wherein: The method for allocating resources consumed and working hours consumed by the evaluation agent walking by the resource scheduling algorithm is as follows: each evaluation agent is regarded as a particle to generate an initial population; the fitness of each particle is calculated according to an evaluation task; the particles are screened for crossover and mutation; the position and speed of the particle are updated until an optimal resource scheduling scheme is obtained.

5. The method of claim 1, wherein: When the evaluation agent walks in the evaluation scene graph, each intelligent module of the complex intelligent decision system is evaluated first, and then the complex intelligent decision system is comprehensively evaluated.

6. An agent-walking-based complex intelligent decision system scene-level evaluation device, characterized in that: comprise: one or more processors; a memory; And one or more computer programs, wherein one or more of the computer programs are stored in the memory, one or more of the computer programs include instructions that, when executed by the processor, cause the complex intelligent decision system scene level evaluation device based on agent walking to perform the complex intelligent decision system scene level evaluation method based on agent walking of any one of claims 1-5.

Images