Multi-agent behavior decision-making method and system based on interactive dynamic influence graph

Abstract

The application discloses a multi-agent behavior decision-making method and system based on an interactive dynamic influence graph, and particularly relates to the technical field of artificial intelligence, and the scheme comprises the following steps: acquiring observation environment information and action information of each agent in an agent cluster, and utilizing the observation environment information and / or the action information to form a behavior sequence of a corresponding agent; performing evolution calculation on the behavior sequence to obtain a strategy tree, and controlling the behavior of each agent according to the strategy tree. According to the scheme, the evolution calculation is utilized to iteratively output a more diverse behavior model according to the mutual influence relationship of each agent in the interactive dynamic influence graph, and the obtained behavior model is embedded into the interactive dynamic influence graph model, so that more effective and rich environment information is provided for the main agent to optimize its own decision-making, and the main agent and other agents can make more optimal decisions.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and in particular to a multi-agent behavior decision-making method and system based on interactive dynamic influence graphs. Background Technology

[0002] Intelligent decision-making is an important branch of artificial intelligence, involving how intelligent agents can make rational decisions based on perceived environmental information and existing knowledge and experience to achieve specific goals. Currently, typical research on multi-agent sequential decision-making under uncertain conditions is widely used in various scenarios, such as reconnaissance by multiple unmanned aerial vehicles, search missions of various rescue teams in natural disasters, and patrol schedules of multiple military units in security protection.

[0003] To address the aforementioned decision-making problem, many decision-making models have been developed in the prior art. However, most of these models focus on solving interactive dynamic influence graphs in a knowledge-driven manner. This approach cannot adequately understand the behavior of other agents in a multi-agent system, thus failing to accurately predict the behavioral models of each agent and effectively optimize its own decisions. Consequently, the knowledge-driven approach is inadequate in solving the diversity of agent behaviors. Summary of the Invention

[0004] In view of the shortcomings of the prior art, the purpose of this invention is to provide a multi-agent behavior decision-making method and system based on interactive dynamic influence graphs, aiming to solve the problem of the poor performance of the prior art in solving the diversity of agent behavior.

[0005] To achieve the above objectives, the first aspect of the present invention provides a multi-agent behavior decision-making method based on an interactive dynamic influence graph, which mainly includes the following steps:

[0006] Obtain the observation environment information and action information of each agent in the agent cluster, and use the observation environment information and / or the action information to form the behavior sequence of the corresponding agent;

[0007] Evolutionary calculations are performed on the behavior sequence to obtain a policy tree, and the evolutionary calculations are used to make decisions on the behavior of each agent based on the mutual influence relationships of each agent in the interactive dynamic influence graph;

[0008] Based on the policy tree, the behavior of each of the intelligent agents is controlled.

[0009] Optionally, the evolutionary calculation uses a particle swarm optimization algorithm, and the evolutionary calculation of the behavior sequence to obtain a policy tree includes:

[0010] Based on the actions and index positions in the action information of the behavior sequence, the current position of each agent is obtained;

[0011] Based on the action, the index position, and the probability of performing the action at the index position, the current speed of each agent is obtained;

[0012] Obtain the current best agent and the global best agent in the agent cluster. Based on the position of the current best agent, the position of the global best agent, and the current position and current speed of each agent, obtain the updated speed of the corresponding agent.

[0013] Based on the updated speed and current position of the agent, the updated position of the corresponding agent is obtained;

[0014] The optimal solution for the agent's position is obtained through iterative optimization, and the optimal solution is then transformed into a policy tree.

[0015] Optionally, obtaining the updated speed of the corresponding agent based on the position of the current optimal agent, the position of the globally optimal agent, and the current position and current speed of each agent includes:

[0016] Based on the current position of each agent and the position of the current optimal agent, the first position offset of the corresponding agent is obtained;

[0017] Based on the current position of each agent and the position of the globally optimal agent, the second position offset of the corresponding agent is obtained;

[0018] After assigning weights to the first position offset, the second position offset, and the current velocity of each agent, a vector summation operation is performed to obtain the updated velocity of the corresponding agent.

[0019] Optionally, obtaining the updated position of the agent based on the updated velocity and the current position includes:

[0020] The updated velocity and current position of each agent are vector summed to obtain the updated position of the corresponding agent.

[0021] Optionally, the evolutionary calculation uses an ant colony optimization algorithm, and the evolutionary calculation of the behavior sequence to obtain a policy tree includes:

[0022] The population is initialized based on the actions and locations in the action information of the behavioral sequence;

[0023] The pheromone table is initialized using the location, action, and environmental state of each agent in the population, as well as the environmental state in the observed environment information.

[0024] The updated pheromone table is obtained by using the total pheromone value obtained by each agent when performing the previous action and the total pheromone value obtained by each agent when performing the current action.

[0025] Based on the updated pheromone table, the updated population is obtained;

[0026] The optimal population is obtained through iterative optimization, and the optimal population is then transformed into a policy tree.

[0027] Optionally, initializing the pheromone table using the loci, actions, and environmental states from the observed environmental information of each agent in the population includes:

[0028] Based on the environmental state and actions at the location of each agent, the expected value of the corresponding agent performing the action is obtained;

[0029] Obtain the time slice corresponding to the behavior sequence, and obtain the confidence level of each agent based on the environmental state corresponding to the time slice;

[0030] The pheromone table is initialized based on the expected value and confidence level of each agent.

[0031] Optionally, obtaining the updated pheromone table using the total pheromone value obtained by each agent performing the previous action and the total pheromone value obtained by each agent performing the current action includes:

[0032] Calculate and sum the pheromone values ​​obtained by each agent in the population when performing the previous action at each site to obtain the total pheromone value when performing the previous action at each site;

[0033] Calculate and sum the pheromone values ​​obtained by each agent in the population when performing the current action at each location, and obtain the total pheromone value when performing the current action at each location;

[0034] Obtain the pheromone evaporation factor of the population, and based on the total pheromone value of the previous action, the total pheromone value of the current action, and the pheromone evaporation factor at each site, obtain the updated pheromone value at the corresponding site;

[0035] Based on the updated pheromone values, an updated pheromone table is obtained.

[0036] A second aspect of the present invention provides a multi-agent behavior decision-making system based on an interactive dynamic influence graph, the system comprising:

[0037] The behavior sequence acquisition module is used to acquire the observation environment information and action information of each agent in the agent cluster, and to use the observation environment information and / or the action information to form the behavior sequence of the corresponding agent;

[0038] The decision-making module is used to perform evolutionary calculations on the behavior sequence to obtain a policy tree. The evolutionary calculations are used to make decisions on the behavior of each agent based on the mutual influence relationships of each agent in the interactive dynamic influence graph.

[0039] The decision control module is used to control the behavior of each of the intelligent agents based on the policy tree.

[0040] A third aspect of the present invention provides an intelligent terminal, the intelligent terminal including a memory, a processor, and a multi-agent behavior decision-making program based on an interactive dynamic influence graph stored in the memory and executable on the processor, wherein when the multi-agent behavior decision-making program based on the interactive dynamic influence graph is executed by the processor, any one of the steps of the above-described multi-agent behavior decision-making method based on the interactive dynamic influence graph is implemented.

[0041] A fourth aspect of the present invention provides a computer-readable storage medium storing a multi-agent behavior decision-making program based on an interactive dynamic influence graph, wherein the multi-agent behavior decision-making program based on the interactive dynamic influence graph, when executed by a processor, implements any of the steps of the above-described multi-agent behavior decision-making method based on the interactive dynamic influence graph.

[0042] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0043] This invention acquires the observation environment information and action information of each agent in an agent cluster, and uses this information to form a behavior sequence for each agent. Evolutionary computation is then performed on the behavior sequence to obtain a policy tree, and the behavior of each agent is controlled according to the policy tree. This invention utilizes evolutionary computation to make decisions about the behavior of each agent based on the mutual influence relationships in an interactive dynamic influence graph, iterating to generate more diverse behavior models. The obtained behavior models of other agents are then embedded into the interactive dynamic influence graph model, thereby providing more effective and richer environmental information for the main agent to optimize its own decisions, and thus effectively promoting better decision-making by the main agent and other agents. Attached Figure Description

[0044] To more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0045] Figure 1 (a) is a schematic diagram of a multi-agent decision-making scenario with two agents;

[0046] Figure 1 (b) is a schematic diagram of the behavioral decision-making process of the agent in three consecutive time slices in I-DID;

[0047] Figure 1 (c) is a schematic diagram of the behavior model of a single agent in the environment and the corresponding probability distribution;

[0048] Figure 1 (d) is a schematic diagram of the strategy tree structure of the interactive dynamic influence graph;

[0049] Figure 2 This is a flowchart of the multi-agent behavior decision-making method based on interactive dynamic influence graphs of the present invention;

[0050] Figure 3 This is a schematic diagram of the predictive framework structure for an intelligent agent behavior model based on particle swarm optimization algorithm according to the present invention.

[0051] Figure 4 This is a schematic diagram of the principle of the first operator of the present invention;

[0052] Figure 5 This is a schematic diagram of the principle of the second operator of the present invention;

[0053] Figure 6 This is a schematic diagram of the principle of the third operator of the present invention;

[0054] Figure 7 This is a schematic diagram of the principle of the fourth operator of the present invention;

[0055] Figure 8 This is a schematic diagram of the predictive framework structure for an agent behavior model based on ant colony optimization algorithm according to the present invention.

[0056] Figure 9 This is a schematic diagram of the multi-agent tiger problem model structure of the present invention;

[0057] Figure 10 The tiger model of this invention is a line graph showing the diversity of agent j and the reward of agent i when agent j's model is within the prediction of agent i.

[0058] Figure 11The tiger model of this invention is a line graph showing the reward of agent i when agent j's model is not within the prediction of agent i.

[0059] Figure 12 This is a schematic diagram of the multi-agent UAV problem model structure of the present invention;

[0060] Figure 13 The line graph of the reward of agent i when agent j's model is not within the prediction of agent i in the multi-agent UAV problem model of the present invention;

[0061] Figure 14 This is a schematic diagram of the structure of the multi-agent behavior decision-making system based on interactive dynamic influence graphs according to the present invention;

[0062] Figure 15 This is a schematic diagram of the smart terminal framework structure of the present invention. Detailed Implementation

[0063] In the following description, specific details such as particular system architectures and techniques are set forth for illustrative purposes and not for limitation, in order to provide a thorough understanding of the embodiments of the invention. However, those skilled in the art will understand that the invention can be implemented in other embodiments without these specific details. In other instances, detailed descriptions of well-known systems, apparatuses, circuits, and methods are omitted so as not to obscure the description of the invention with unnecessary detail.

[0064] It should be understood that, when used in this specification and the appended claims, the term "comprising" indicates the presence of the described features, integrals, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or collections thereof.

[0065] It should also be understood that the terminology used in this specification is for the purpose of describing particular embodiments only and is not intended to limit the invention. As used in this specification and the appended claims, the singular forms “a,” “an,” and “the” are intended to include the plural forms unless the context clearly indicates otherwise.

[0066] It should also be further understood that the term "and / or" as used in this specification and the appended claims refers to any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.

[0067] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0068] Many specific details are set forth in the following description in order to provide a full understanding of the invention. However, the invention may also be practiced in other ways different from those described herein, and those skilled in the art can make similar extensions without departing from the spirit of the invention. Therefore, the invention is not limited to the specific embodiments disclosed below.

[0069] To address the shortcomings of existing knowledge-driven methods in solving the diversity of agent behaviors, this invention proposes a multi-agent behavior decision-making method based on interactive dynamic influence graphs.

[0070] Currently, multi-agent decision-making problems mainly focus on agents i and j existing in a partially observable environment, where each agent is independent of the others. Agent i and j will take actions upon receiving environmental observations and receive corresponding rewards based on the impact of their actions on the environmental state. For example, Figure 1 (a) shows a multi-agent decision-making scenario with two agents. Figure 1 As shown in (b), the behavior decision-making process of agent i in three consecutive time slices in I-DID is as follows: based on the current environmental state S, agent i observes other agents j O, and based on the observation results and the candidate behavior models M of other agents j, it needs to predict the behavior A of other agents j. This is because the reward obtained by agent i is determined by the behavior of both agent i and j. However, agent i does not know the behavior that other agents j will take before interacting with them. Therefore, agent i must solve a large number of possible behavior models of other agents j and provide the set of possible models of other agents j to agent i as a set of candidate behavior models. Figure 1 As shown in (c), this is a behavioral model of a single agent in the environment and the probability distribution of various possible states. Figure 1 (d) shows the policy tree as a solution to the interactive dynamic influence graph, which describes the best action of other agents j in each observation at each time step, and is usually called a behavior model.

[0071] To accurately predict the behavioral decisions of the primary intelligent agent, it is theoretically necessary to assume the existence of a large number of behavioral models of other intelligent agents. This increases the difficulty of solving the optimal decision for the primary intelligent agent based on the I-DID model. Therefore, this invention uses historical behavioral data of other intelligent agents and iteratively generates more diverse behavioral models through evolutionary computation methods to expand the behavioral models of other intelligent agents. These behavioral models are then embedded into an interactive dynamic influence graph model, thereby providing the primary intelligent agent with more effective and richer environmental information to optimize its own decisions, and ultimately effectively prompting the primary intelligent agent and other intelligent agents to make better decisions.

[0072] Exemplary methods

[0073] This invention provides a multi-agent behavior decision-making method based on interactive dynamic influence graphs. Deployed on electronic devices such as computers and servers, it is applied to scenarios involving agent behavior decision-making within interactive dynamic influence graphs, specifically addressing multi-agent behavior decision-making. Specifically, as... Figure 2 As shown, the main steps of the method in this embodiment include:

[0074] Step S100: Obtain the observation environment information and action information of each agent in the agent cluster, and use the observation environment information and / or action information to form the behavior sequence of the corresponding agent.

[0075] Specifically, in an agent cluster, each agent is independent of the others. When multiple agents run in a common and uncertain environment, interactions between agents usually occur. The behavior of an agent may affect the state of the environment, thereby affecting the behavior of other agents. Obviously, agents need to optimize their own behavioral decisions based on the environmental state information and the action information of other agents. Therefore, this invention predicts the behavioral decisions of other agents based on the behavioral sequences of other agents, thereby optimizing its own behavioral decisions.

[0076] Step S200: Perform evolutionary calculations on the behavior sequence to obtain a policy tree. The evolutionary calculations are used to make decisions on the behavior of each agent based on the mutual influence relationships among the agents in the interactive dynamic influence graph.

[0077] Specifically, by performing evolutionary calculations on the behavior sequence and iterating to generate more behavior models, the obtained behavior models are applied to the interactive dynamic influence graph model, which enables the agent to make better behavioral decisions based on the impact of other agents' behavior on the environmental state. This results in obtaining a policy tree composed of the policies of other agents at different time slices. Here, a time slice refers to the time period from when the agent completes the previous action to when it completes the current action.

[0078] Step S300: Control the behavior of each agent based on the policy tree.

[0079] Specifically, based on the observed environment information and action information on the policy tree of each agent, the behavior of the corresponding agent in each time slice is controlled.

[0080] In this embodiment, one of the agents in the agent cluster is regarded as the main agent. By utilizing the observation and action information of other agents, a more diverse behavioral model is generated through evolutionary computation. The behavioral models of other agents obtained by evolutionary computation are then embedded into an interactive dynamic influence graph model. This provides the main agent with more effective and richer environmental information to optimize its own decisions, thereby more accurately predicting the behavioral decisions of other agents and promoting the main agent to make better behavioral decisions. It is evident that the iterative optimization method of evolutionary computation adopted in this invention enables each agent to make better behavioral decisions, improving the efficiency and intelligence of the entire agent cluster.

[0081] Furthermore, in this embodiment, the evolutionary calculation uses the Particle Swarm Optimization (PSO) algorithm, the framework of which is shown in the figure below. Figure 3 As shown, evolutionary computation is performed on the behavior sequence to obtain the policy tree, including:

[0082] Step S210: Based on the action and index position in the action information of the action sequence, obtain the current position of each agent.

[0083] Specifically, in this embodiment, the solution of I-DID is called a Policy Tree. A breadth-first search is used to obtain the behavior sequence A of each agent, and the index position of the agent is initialized using the behavior sequence A, i.e., p = Ua. l a l ∈A, l represents the index position of the agent.

[0084] Step S220: Based on the action, index position, and the probability of performing the action at the index position, obtain the current speed of each agent.

[0085] Specifically, in this embodiment, the speed of the agent is represented by a triple, i.e., v = U[l, a] l ,Pr(a t )], l represents the index of the agent's position, a l Indicates the action at the current position, Pr(a l This indicates that the action to be performed at index position l is a. l The higher the probability, the more likely the agent will perform action a at that index position. lThe higher the probability, the more likely it is to occur, and this probability is initialized to 1 / |A|. It should be noted that the action here includes the agent's direction of movement and distance, which is a vector. This embodiment uses the probability of performing an action at the current index position to represent the agent's speed at that position, that is, it uses probability to predict unknown speeds, making the speed predictions more realistic and reasonable.

[0086] Step S230: Obtain the current best agent and the global best agent in the agent cluster. Based on the position of the current best agent, the position of the global best agent, and the current position and current speed of each agent, obtain the updated speed of the corresponding agent.

[0087] Specifically, since the speed of the agent is closely related to its index position, this embodiment uses the idea of ​​evolutionary computation to iteratively update the index positions of the locally optimal agent and the globally optimal agent, thereby gradually updating and optimizing the agent's speed. This makes the prediction of the agent's speed closer to reality and allows for more accurate observation of the real environmental state.

[0088] For ease of calculation, this embodiment defines four operators within the PSO framework, among which, Figure 4 The figure shows the operator for solving for position by adding position and velocity vectors, referred to as the first operator, and its expression is: Figure 5 The figure shows the operator for solving velocity by subtracting position and position vector, referred to as the second operator, and its expression is: Figure 6 The figure shows the operator for solving velocity by multiplying a constant and a velocity vector, referred to as the third operator, whose expression is: Figure 7 The diagram shows how to solve for velocity using the sum of velocity and velocity vectors, referred to as the fourth operator. Its expression is:

[0089] Step S240: Based on the agent's updated velocity and current position, obtain the updated position of the corresponding agent.

[0090] Specifically, the updated velocity and current position of each agent are vector-summed to obtain the updated position of the corresponding agent. This embodiment utilizes the updated velocity of the agents and updates the action at the corresponding index position according to set judgment conditions to achieve the update of the agent's position.

[0091] like Figure 4As shown, the agent's position is updated using the first operator. For example, given that an agent's action sequence at current position p1 includes 7 index positions [0, 6], and the action information at each index position is represented by the sequence {a2, a1, a2, a1, a1, a1, a2}, the updated velocity is {(0, a1, p1, a2}. 0 ), (4, a2, P 4 If the probability that the agent generates a random seed r0 in the next time slice is less than the probability P... 0 If the probability of the random seed being less than the updated velocity value, then the action at index 0 is adjusted to a1. Similarly, if the probability of the agent generating a random seed r4 in the next time slice is less than probability P... 4 If the action at index position 4 is adjusted to a2, then the action information at position p of the agent after the update is represented by the sequence {a1, a1, a2, a1, a2, a1, a2}. In other words, the agent's position changes with its own velocity. It's easy to understand that changes in velocity cause changes in the agent's direction and distance of movement, naturally leading to changes in the agent's position. This also verifies the rationality of the defined first operator.

[0092] Step S250: Obtain the optimal solution for the agent's position through iterative optimization, and transform the optimal solution into a policy tree.

[0093] In this embodiment, based on the historical behavior sequences of other agents obtained by the main agent, the particle swarm optimization algorithm is used to perform evolutionary calculations on the agent's behavior, thereby achieving iterative updates of the speed and position of other agents. This allows for real-time prediction of the behavior of other agents in the next time slice, and the predicted behavior and the environmental state observed at the corresponding position are converted into a policy tree. This not only increases the diversity of solutions but also provides a more vivid and intuitive display of the behavioral decisions of other agents.

[0094] Furthermore, based on the position of the current best agent, the position of the globally best agent, and the current position and velocity of each agent, the updated velocity of the corresponding agent is obtained, including:

[0095] Step S231: Based on the current position of each agent and the position of the current best agent, obtain the first position offset of the corresponding agent;

[0096] Specifically, through evolutionary computation, the agent in the agent cluster that is in the local optimal position in the current time slice can be selected in each iteration process, that is, the position of the current optimal agent. In this embodiment, the position offset between the current position of the agent and the agent in the current optimal position is called the first position offset of the agent, which is the speed of the agent at the index position where the action changes in the current time slice.

[0097] like Figure 5 As shown, the agent's velocity is updated using the second operator. For example, given that the action sequence of an agent at a locally optimal position p1 includes 7 index positions (0-6), and the action information at each index position is represented by the sequence {a2, a1, a2, a1, a1, a1, a2}, and an agent at its current position p2 also has 7 index positions (0-6), and the action information at each index position is represented by the sequence {a1, a1, a2, a1, a2, a1, a2}, then the second operator is used to compare the actions at the corresponding index positions. If the actions are inconsistent, then... The action at the corresponding index position at position p2 is replaced with the action at the same index position at position p1. In this embodiment, the actions at index positions 0 and 4 are different. Therefore, the action at index position 0 of the agent is adjusted to a2, and the probability of executing action a2 is 1 / |A|. The action at index position 4 is adjusted to a1, and the probability of executing action a2 is 1 / |A|. Thus, the position offset between position p1 and position p2 is obtained, which is represented as {(0, a2, 1 / |A|), (4, a1, 1 / |A|)}, that is, the speed of the agent at the index position where the action changes in the current time slice.

[0098] Step S232: Based on the current position of each agent and the position of the globally optimal agent, obtain the second position offset of the corresponding agent;

[0099] Specifically, by comparing the locally optimal agents obtained in each iteration, the globally optimal agent in the agent cluster at the current time slice is obtained and updated in real time. That is, the position of the globally optimal agent. In this embodiment, the position offset between the current position of the agent and the agent at the globally optimal position is called the second position offset of the agent, which is the speed of the agent at the index position where the action has changed up to the current time slice.

[0100] Step S233: After assigning weights to the first position offset, second position offset, and current velocity of each agent, perform vector summation to obtain the updated velocity of the corresponding agent.

[0101] Specifically, the agent's speed is updated according to the following formula:

[0102]

[0103] Among them, v n v represents the current speed of the agent. n+1 This represents the agent's velocity in the next iteration, where ω, c1, and c2 all represent weights, each ranging from (0, 1]. n Indicates the current position of the agent. p represents the position of the agent that is in a locally optimal position in this iteration. * This represents the position of the agent that is locally optimal up to the current iteration, and is solved using the second, third, and fourth operators.

[0104] Figure 6 The diagram shows the principle of the third operator. Given that the current velocity of an agent is v1 = {(0, a1, P)}, and assuming that the influence weight of the current velocity on the velocity after this iteration is ω, and the value range of ω is (0, 1], then under the action of this weight, the current velocity decays to {(0, a1, min(ωP, 1)}. Figure 7 The diagram shows the principle of the fourth operator. It is known that in one iteration, the first velocity change that causes the agent's velocity to change is v1 = {(0, a1, P1), (0, a2, P′1)}, and the second velocity change is v1 = {(0, a1, P2)}. The two velocity changes at the same index position are added according to the action type. Then the total velocity change of the agent in this iteration is {(0, a1, min(P1+P2, 1), (0, a2, P′1)}.

[0105] In this embodiment, the agent speed is updated based on the position of the agent with the local optimum, the position of the agent with the global optimum, and the current agent speed obtained in each iteration. Weights are set for these three influencing factors, and the weight values ​​can be reasonably adjusted according to the actual situation to improve the accuracy of the predicted speed.

[0106] Furthermore, the evolutionary computation employs the Ant Colony Optimization (ACO) algorithm, the framework of which is shown in the diagram below. Figure 8 As shown, evolutionary computation is performed on the behavior sequence to obtain the policy tree, including:

[0107] Step S260: Initialize the population based on the actions and locations in the action information of the behavior sequence;

[0108] Specifically, in this embodiment, the solution of I-DID is referred to as the policy tree. The ant colony ζ = (a1, ..., a2) is initialized by recording each site and the action at each site. l ), where a l Indicates action a l ∈A, l represents the site, and The number of actions in the policy tree is represented by |Ω|, where |Ω| represents the number of times the environmental state is observed, and T represents the time slice. It can be seen that the location is related to the number of times the environmental state is observed and the number of time slices.

[0109] Step S270: Initialize the pheromone table using the location and actions of each agent in the population, as well as the environmental state from the observed environmental information;

[0110] Specifically, in this embodiment, the pheromone Γ(l, a) = ER(l, a) at the ant colony and the action a at the ant colony are used to represent the pheromone Γ(l, a) at the ant colony, where ER(l, a) is the value of the pheromone. A pheromone table is initialized using the pheromone at each ant colony and the pheromone at the corresponding ant colony to represent the pheromone left by the ant colony at each ant colony.

[0111] Step S280: Using the total pheromone value obtained by each agent in performing the previous action and the total pheromone value obtained by each agent in performing the current action, obtain the updated pheromone table;

[0112] Specifically, by evaluating the pheromones generated by all agents performing the previous action at each location in the previous time slice, the total pheromone value at each location is obtained, and by evaluating the total pheromone value obtained by all agents performing the current action at each location in the current time slice, the updated pheromone value at each location is solved. Then, the updated pheromone value at each location is used to update the pheromone table. That is, the pheromone left over from the actions performed at each location in the previous iteration, as well as the pheromone generated by the actions performed at each location in the current iteration, are used to jointly update the pheromone at the corresponding location.

[0113] Step S290: Obtain the updated population based on the updated pheromone table;

[0114] Specifically, based on the pheromone table, a greedy strategy is used to update the agent, thereby obtaining the updated population.

[0115] Step S2100: Obtain the optimal population through iterative optimization, and transform the optimal population into a policy tree.

[0116] Specifically, by iterating each time, a locally optimal population is obtained, and the global optimal population is updated using the local optimal population. This allows for a simple and efficient way to obtain the optimal population, which is then transformed into a policy tree to guide the agent in making optimal behavioral decisions.

[0117] In this embodiment, based on the latest behavior sequences of other agents obtained by the main agent, the ant colony optimization algorithm is used to perform evolutionary calculations on the agent's behavior, thereby realizing iterative updates of the pheromone tables of other agents and the ant colony, thus predicting the behavior of other agents in real time, and using the predicted behavior and the environmental state observed at the corresponding location to convert into a policy tree, which not only increases the diversity of solutions, but also vividly and intuitively displays the behavioral decisions of other agents.

[0118] Furthermore, by utilizing the location and actions of each agent in the population, as well as the environmental state observed in the environmental information, the pheromone table is initialized, including:

[0119] Step S271: Based on the environmental state and actions at each location of each agent, obtain the expected value of the action to be performed by the corresponding agent;

[0120] Step S272: Obtain the time slices corresponding to the behavior sequences, and obtain the confidence of each agent based on the environmental state of each agent in the time slices;

[0121] Step S273: Initialize the pheromone table based on the expected value and confidence of each agent.

[0122] In one embodiment, the expression for initializing the pheromone table is based on the expected value and confidence level of each agent, specifically as follows:

[0123]

[0124] Where s represents the environment state in time slice t, S represents the set of environment states, a represents the action, R(s,a) represents the expected value of performing action a in environment state s, and b t (s) represents the agent's confidence level under environmental state s and time slice t.

[0125] In this embodiment, the expected value of an agent's action under a certain environmental state and the agent's confidence level are obtained using the environmental state of the agent at the time slice corresponding to the action. Based on the expected value and confidence level, pheromones at each site are initialized to obtain a pheromone table. As can be seen, the pheromone table includes the pheromone values ​​left by each agent at each site and time slice during the execution of actions. That is, it includes information for each site, each time slice, and each environmental state of the agent, comprehensively and accurately representing the pheromone information of each agent, thus laying the foundation for subsequent updates to the pheromone table and the search for the optimal population.

[0126] Furthermore, using the total pheromone value obtained by each agent when performing the previous action and the total pheromone value obtained by each agent when performing the current action, an updated pheromone table is obtained, including:

[0127] Step S281: Calculate and sum the pheromone values ​​obtained by each agent in the population when performing the previous action at each site to obtain the total pheromone value when performing the previous action at each site;

[0128] Step S282: Calculate and sum the pheromone values ​​obtained by each agent in the population when performing the current action at each site, and obtain the total pheromone value when performing the current action at each site;

[0129] Step S283: Obtain the pheromone evaporation factor of the population. Based on the total pheromone value of the previous action, the total pheromone value of the current action, and the pheromone evaporation factor at each site, obtain the updated pheromone value at the corresponding site.

[0130] Step S284: Obtain the updated pheromone table based on the updated pheromone values.

[0131] In one embodiment, the pheromone values ​​obtained by each agent in the population from performing the previous action at each location are calculated and summed to obtain the total pheromone value from performing the previous action at each location. Based on the pheromones generated by each agent performing actions at various locations in adjacent time slices, the expression for the updated pheromone table is obtained, specifically as follows:

[0132]

[0133] Γ n+1 (l, a)=(1-ρ)Γ n (l,a)+ΔΓ n+1 (l, a)

[0134] Where, γ ζ (l, a) represents the pheromone value generated by ant ζ performing the current action a at site l, pop represents the population consisting of all agents, and ΔΓ n+1 (l, a) represents the sum of pheromone values ​​generated by each agent in the entire population performing the current action a at point l. n (l, a) represents the sum of pheromone values ​​generated by each agent in the entire population performing the previous action a at site l in the previous time slice, ρ represents the pheromone evaporation factor, and (1-ρ)Γ n (l, a) represents the pheromone value left by each agent in the entire population when performing the previous action a at point l in the previous time slice. n+1(l, a) represents the updated pheromone. It can be seen that the pheromone table includes the sum of the pheromone values ​​obtained by each agent in the entire population when performing action a at site l in adjacent time slices. That is, the sum of the pheromone left after the pheromone evaporated after performing action a in the previous time slice and the pheromone generated when performing action a in the current time slice, so as to update the pheromone values ​​at each site in real time.

[0135] Furthermore, based on the updated pheromone table, an updated population is obtained according to the set update rules. For example, for each agent, given the actions at each position in the current iteration, the agent is updated using at least one of the following two methods: The first method is a greedy strategy, which involves counting the pheromone values ​​of each agent at the corresponding positions in the pheromone table, obtaining the maximum pheromone value at each position, and updating the agent's actions at each position according to the actions represented by the maximum pheromone values, until all agents are updated and the updated population is obtained; The second method is a random selection strategy, where if the random seed is less than the preset update threshold ∈, an action other than the current action is randomly selected from the action set to update the agent, until all agents are updated and the updated population is obtained. For example, if the current action is a1 and the action set is {a1, a2, a3}, if the random seed is less than the preset update threshold ∈, an action is randomly selected from a2 and a3 to update the agent's action at the current position. It should be noted that in a single iteration update, a greedy strategy or a random selection strategy can be used to update the agent, depending on the actual situation. Alternatively, some points can be selected to update actions according to the greedy strategy, while the remaining points can be updated according to the random selection strategy, thus achieving the update of a single agent.

[0136] This invention uses the multi-agent tiger problem and the multi-agent drone pursuit problem to verify the effectiveness of the interactive dynamic influence graph solution method.

[0137] Experiment 1: The Multi-Agent Tiger Problem

[0138] The multi-agent tiger problem is a classic problem in multi-agent planning systems. This experiment primarily considers two agents, such as... Figure 9As shown in the diagram, there are two agents, agent i and agent j, with agent i referred to as the primary agent. Both agents have three actions: open the left door, open the right door, or listen. Whether there's a tiger or money behind the door is uncertain. Agent i can determine the true situation by observing agent j's behavior. Agent i can determine which door agent j opened by hearing the sound of a door opening. If choosing to listen, agent i can hear a tiger's roar (GL if the roar comes from the left, GR if from the right) and a creaking sound of the door opening (CL if from the left, CR if from the right, or S if there's no sound). If the determination is correct, the primary agent receives a monetary reward; otherwise, it will be eaten by the tiger. Generally, if the decision is to open a door, whether left or right, the tiger's state is reset (i.e., it's still unknown which door the tiger is behind).

[0139] Figure 10 The figure shows the diversity of agent j's model and the reward line graph of agent i at time slices T=3, 4, and 5 in the Tiger model. It describes the possible models of agent j within the predictions of agent i, the diversity convergence graph of agent j, and the reward obtained from predicting the behavior of agent j from the perspective of agent i. Figure 10 (a) Figure 10 (b) and Figure 10 (c) The horizontal axis represents the number of iterations, and the vertical axis represents the diversity of the generative model. Figure 10 (d) Figure 10 (e) and Figure 10 (f) The horizontal axis represents the agent's policy tree, and the vertical axis represents the average reward corresponding to the policy tree. As can be seen from the figure, compared with the genetic algorithm (GA), the particle swarm algorithm (PM) and the ant colony algorithm (ACO) have better model diversity in solving the model of agent j, and are superior in predicting the behavior model of agent j.

[0140] Figure 11 The figure shows the reward line graphs for agent i in the Tiger model when the time slices are T=3 and T=5. It describes the reward obtained from agent i's perspective for predicting agent j's behavior when the possible model of agent j is not within the prediction of agent i. This is done by randomly generating the behavior model of agent j to simulate the situation. Figure 11 (a) and Figure 11 (b) The horizontal axis represents the agent's policy tree, and the vertical axis represents the average reward corresponding to the policy tree. As can be seen from the figure, the PM algorithm and the ACO algorithm have better performance in predicting the behavior of agent j.

[0141] Experiment 2: Multi-Agent UAV Problem

[0142] like Figure 12 As shown, the drone problem is planned in a 5x5 grid in the experiment. The problem is described as follows: In a partially observable environment, two drones do not know each other's exact location but can receive signals of their relative positions. The master agent i considers the other agent j as an enemy drone and will pursue it. Agent j tries to escape the pursuit of the master agent i and runs around, with its goal being to reach a safe house. This experiment uses IDID modeling to model the relative positions of the two agents. The agents have four actions: move up, move down, move left, move right, and listen. The environmental observations include enemy drones on the left, right, horizontal direction, and enemy drones detected. For the master agent i, a good assessment of the enemy drones is needed to achieve its goal. Particle swarm optimization and ant colony optimization algorithms are used to solve the drone problem. The correctness of the method is verified by comparing it with the genetic algorithm (GA) and the original exact algorithm for solving IDID.

[0143] Figure 13 The graphs shown are line graphs of rewards for agents i at time slices T=3 and T=4. They describe the rewards obtained by the subject agent for the predictions of other agents in a multi-agent drone pursuit problem, where the behavior of agent j is not within the prediction of the subject agent j. Figure 13 (a) and Figure 13 (b) The horizontal axis represents the agent's policy tree, and the vertical axis represents the average reward corresponding to the policy tree. As can be seen from the graph, at time slice T=3, the PM algorithm and ACO algorithm have higher prediction reliability, resulting in higher rewards. At time slice T=4, the PM algorithm still shows superiority, while the ACO algorithm's prediction performance is slightly lower than the GA algorithm. This is normal, and the possible reason is that the agent did not obtain useful information within a certain timeframe.

[0144] The time performance of the four methods shown in Table 1 is compared and analyzed below. Table 1 shows that for both the multi-agent tiger problem and the multi-agent UAV problem, performance improvement is greater when time slices are limited, and the PM algorithm has a certain advantage over the ACO algorithm. The ACO algorithm shows a higher speedup ratio compared to the PM algorithm in the multi-agent UAV problem.

[0145] Table 1:

[0146]

[0147] Exemplary System

[0148] like Figure 14As shown, corresponding to the above-described multi-agent behavior decision-making method based on interactive dynamic influence graphs, this embodiment of the invention also provides a multi-agent behavior decision-making system based on interactive dynamic influence graphs. The multi-agent behavior decision-making system based on interactive dynamic influence graphs includes:

[0149] The behavior sequence acquisition module is used to acquire the observation environment information and action information of each agent in the agent cluster, and to use the observation environment information and / or action information to form the behavior sequence of the corresponding agent;

[0150] The decision-making module is used to perform evolutionary calculations on the behavior sequence to obtain the policy tree. The evolutionary calculations are used to make decisions on the behavior of each agent based on the mutual influence relationships between the agents in the interactive dynamic influence graph.

[0151] The decision control module is used to control the behavior of each agent based on the policy tree.

[0152] Specifically, in this embodiment, the specific functions of the multi-agent behavior decision-making system based on interactive dynamic influence graphs can be referred to the corresponding description in the multi-agent behavior decision-making method based on interactive dynamic influence graphs, and will not be repeated here.

[0153] Based on the above embodiments, the present invention also provides a smart terminal, the principle block diagram of which can be as follows: Figure 15 As shown, the aforementioned intelligent terminal includes a processor, memory, network interface, and display screen connected via a system bus. The processor provides computing and control capabilities. The memory includes a non-volatile storage medium and internal memory. The non-volatile storage medium stores an operating system and a multi-agent behavior decision-making program based on interactive dynamic influence diagrams. The internal memory provides an environment for the operation of the operating system and the multi-agent behavior decision-making program based on interactive dynamic influence diagrams. The network interface of the intelligent terminal is used for communication with external terminals via a network connection. When executed by the processor, the multi-agent behavior decision-making program based on interactive dynamic influence diagrams implements the steps of any of the aforementioned multi-agent behavior decision-making methods based on interactive dynamic influence diagrams. The display screen of the intelligent terminal can be a liquid crystal display (LCD) or an e-ink display.

[0154] Those skilled in the art will understand that Figure 15 The block diagram shown is merely a partial structural diagram related to the present invention and does not constitute a limitation on the smart terminal to which the present invention is applied. A specific smart terminal may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0155] In one embodiment, a smart terminal is provided, the smart terminal including a memory, a processor, and a multi-agent behavior decision-making program based on an interactive dynamic influence graph stored in the memory and executable on the processor. When the multi-agent behavior decision-making program based on the interactive dynamic influence graph is executed by the processor, it implements the steps of any of the multi-agent behavior decision-making methods based on interactive dynamic influence graphs provided in the embodiments of the present invention.

[0156] This invention also provides a computer-readable storage medium storing a ride-hailing two-stage matching program. When the ride-hailing two-stage matching program is executed by a processor, it implements the steps of any of the ride-hailing two-stage matching methods provided in this invention.

[0157] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0158] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is merely an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the above device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit. Furthermore, the specific names of the functional units and modules are only for easy differentiation and are not intended to limit the scope of protection of this invention. The specific working process of the units and modules in the above system can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0159] In the above embodiments, the descriptions of each embodiment have different focuses. For parts that are not described in detail or recorded in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0160] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0161] In the embodiments provided by this invention, it should be understood that the disclosed apparatus / terminal devices and methods can be implemented in other ways. For example, the apparatus / terminal device embodiments described above are merely illustrative. For instance, the division of the above modules or units is merely a logical functional division, and in actual implementation, it can be divided in other ways. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed.

[0162] The above-described embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not mean that the essence of the corresponding technical solutions deviates from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included within the protection scope of the present invention.

Claims

1. A multi-agent behavior decision-making method based on interactive dynamic influence graphs, characterized in that, The main steps include: Obtain the observation environment information and action information of each agent in the agent cluster, and use the observation environment information and / or the action information to form the behavior sequence of the corresponding agent; Evolutionary calculations are performed on the behavior sequence to obtain a policy tree, and the evolutionary calculations are used to make decisions on the behavior of each agent based on the mutual influence relationships of each agent in the interactive dynamic influence graph; Based on the policy tree, the behavior of each of the intelligent agents is controlled; The evolutionary calculation uses a particle swarm optimization algorithm. The process of performing evolutionary calculations on the behavior sequence to obtain a policy tree includes: Based on the actions and index positions in the action information of the behavior sequence, the current position of each agent is obtained; Based on the action, the index position, and the probability of performing the action at the index position, the current speed of each agent is obtained; Obtain the current best agent and the global best agent in the agent cluster. Based on the position of the current best agent, the position of the global best agent, and the current position and current speed of each agent, obtain the updated speed of the corresponding agent. Based on the updated speed and current position of the agent, the updated position of the corresponding agent is obtained; The optimal solution for the agent's position is obtained through iterative optimization, and the optimal solution is transformed into a policy tree; The agent is a drone, the actions include the agent's direction of travel and distance, and the policy tree is the solution to an interactive dynamic influence graph.

2. The multi-agent behavior decision-making method based on interactive dynamic influence graphs according to claim 1, characterized in that, The step of obtaining the updated speed of the corresponding agent based on the position of the current best agent, the position of the globally best agent, and the current position and current speed of each agent includes: Based on the current position of each agent and the position of the current optimal agent, the first position offset of the corresponding agent is obtained; Based on the current position of each agent and the position of the globally optimal agent, the second position offset of the corresponding agent is obtained; After assigning weights to the first position offset, the second position offset, and the current velocity of each agent, a vector summation operation is performed to obtain the updated velocity of the corresponding agent.

3. The multi-agent behavior decision-making method based on interactive dynamic influence graphs according to claim 1, characterized in that, Obtaining the updated position of the corresponding agent based on the updated speed and the current position includes: The updated velocity and current position of each agent are vector summed to obtain the updated position of the corresponding agent.

4. The multi-agent behavior decision-making method based on interactive dynamic influence graphs according to claim 1, characterized in that, The evolutionary calculation uses an ant colony optimization algorithm. The process of performing evolutionary calculations on the behavior sequence to obtain a policy tree includes: The population is initialized based on the actions and locations in the action information of the behavioral sequence; The pheromone table is initialized using the location, action, and environmental state of each agent in the population, as well as the environmental state in the observed environment information. The updated pheromone table is obtained by using the total pheromone value obtained by each agent when performing the previous action and the total pheromone value obtained by each agent when performing the current action. Based on the updated pheromone table, the updated population is obtained; The optimal population is obtained through iterative optimization, and the optimal population is then transformed into a policy tree.

5. The multi-agent behavior decision-making method based on interactive dynamic influence graphs according to claim 4, characterized in that, The initialization of the pheromone table using the location, action, and environmental state from the observed environment information of each agent in the population includes: Based on the environmental state and actions at the location of each agent, the expected value of the corresponding agent performing the action is obtained; Obtain the time slice corresponding to the behavior sequence, and obtain the confidence level of each agent based on the environmental state corresponding to the time slice; The pheromone table is initialized based on the expected value and confidence level of each agent.

6. The multi-agent behavior decision-making method based on interactive dynamic influence graphs according to claim 4, characterized in that, The step of obtaining an updated pheromone table using the total pheromone value obtained by each agent performing the previous action and the total pheromone value obtained by each agent performing the current action includes: Calculate and sum the pheromone values ​​obtained by each agent in the population when performing the previous action at each site to obtain the total pheromone value when performing the previous action at each site; Calculate and sum the pheromone values ​​obtained by each agent in the population when performing the current action at each location, and obtain the total pheromone value when performing the current action at each location; Obtain the pheromone evaporation factor of the population, and based on the total pheromone value of the previous action, the total pheromone value of the current action, and the pheromone evaporation factor at each site, obtain the updated pheromone value at the corresponding site; Based on the updated pheromone values, an updated pheromone table is obtained.

7. A multi-agent behavior decision-making system based on interactive dynamic influence maps, characterized in that, The system includes: The behavior sequence acquisition module is used to acquire the observation environment information and action information of each agent in the agent cluster, and to use the observation environment information and / or the action information to form the behavior sequence of the corresponding agent, wherein the agent is a drone; The decision-making module is used to perform evolutionary calculations on the behavior sequence to obtain a policy tree. The evolutionary calculations are used to make decisions on the behavior of each agent based on the mutual influence relationships of each agent in the interactive dynamic influence graph. The evolutionary calculation uses a particle swarm optimization algorithm. The process of performing evolutionary calculations on the behavior sequence to obtain a policy tree includes: Based on the actions and index positions in the action information of the behavior sequence, the current position of each agent is obtained; Based on the action, the index position, and the probability of performing the action at the index position, the current speed of each agent is obtained; Obtain the current best agent and the global best agent in the agent cluster. Based on the position of the current best agent, the position of the global best agent, and the current position and current speed of each agent, obtain the updated speed of the corresponding agent. Based on the updated speed and current position of the agent, the updated position of the corresponding agent is obtained; The optimal solution for the agent's position is obtained through iterative optimization, and the optimal solution is transformed into a policy tree; The actions include the agent's direction of movement and distance, and the policy tree is the solution to the interactive dynamic influence graph; The decision control module is used to control the behavior of each of the intelligent agents based on the policy tree.

8. A smart terminal, characterized in that, The intelligent terminal includes a memory, a processor, and a multi-agent behavior decision-making program based on an interactive dynamic influence graph, which is stored in the memory and can run on the processor. When the multi-agent behavior decision-making program based on the interactive dynamic influence graph is executed by the processor, it implements the steps of the multi-agent behavior decision-making method based on the interactive dynamic influence graph as described in any one of claims 1-6.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a multi-agent behavior decision-making program based on an interactive dynamic influence graph, which, when executed by a processor, implements the steps of the multi-agent behavior decision-making method based on an interactive dynamic influence graph as described in any one of claims 1-6.

Images