Cooperative action method and device for multiple AI terminal agents
By calculating single-step regret value and cumulative regret value to adjust the action weights of AI terminal agents, and combining coarse correlation balancing and restart strategies, the adversarial problem of heterogeneous multi-agent collaborative actions in emergency rescue is solved, achieving efficient collaboration and optimal solutions in dynamic environments.
Patent Information
- Application Number
- CN202510992043.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-07-18
- Publication Date
- 2025-11-04
AI Technical Summary
In complex emergency rescue scenarios, the collaborative actions among heterogeneous multi-agent systems face challenges due to limited resources or differing task priorities, making it difficult to effectively coordinate and complete tasks.
By calculating single-step regret value and cumulative regret value, the action weights of AI terminal agents are adjusted. Combined with coarse correlation equilibrium constraints and restart strategy, the action strategy of multi-agent system is optimized to ensure that the reward of each AI terminal agent is maximized and the system reaches the optimal solution.
It achieves efficient collaboration among heterogeneous multi-agent systems in dynamic environments, ensuring that a single AI terminal agent obtains acceptable benefits while the agent system reaches the optimal solution, adapts to environmental changes, and avoids policy rigidity.
Smart Images
Figure CN120886243A_ABST
Abstract
Description
TECHNICAL FIELD
[0001] One or more embodiments of the present application relate to network communication technology, and in particular to a method and apparatus for coordinated action of multiple artificial intelligence (AI) terminal agents. BACKGROUND
[0002] In recent years, with the continuous development of artificial intelligence technology, AI terminal agents have emerged. Terminal devices, as a key link in the application of AI agents, have gradually integrated AI agents to form intelligent terminals, and have realized deep application in industry task scenarios. Multiple AI terminal agents work together to complete a task. For example, intelligent unmanned equipment for emergency rescue scenarios has received widespread attention from all walks of life. It can improve rescue efficiency and reduce casualties of rescue personnel. However, the deployment of unmanned equipment in complex social environments, three-dimensional urban spaces, and limited rescue conditions puts high demands on the reasonable planning and use of AI terminal agents integrated into the unmanned equipment. Such multi-task scenarios require heterogeneous multi-AI terminal agents that can perform different tasks to work together to complete emergency scene tasks such as personnel rescue, material distribution, and obstacle cleaning.
[0003] However, heterogeneous multi-agents need to perform different types of tasks, and the tasks are often mutually contradictory. For example, unmanned aerial vehicles and robots may need to cooperate in disaster rescue, but if resources are limited or task priorities are different, they may be in conflict. For example, one agent is responsible for medical rescue, and the other is responsible for material transportation. If both need the same channel or resource, a conflict will arise. Therefore, any better coordination of multiple AI terminal agents has become a problem to be solved. SUMMARY
[0004] One or more embodiments of the present application describe a method and apparatus for coordinated action of multiple AI terminal agents, which can better coordinate the action of multiple AI terminal agents.
[0005] According to a first aspect, a method for coordinated action of multiple AI terminal agents is provided. The multiple AI terminal agents form an agent system to collectively complete a task. The method includes:
[0006] For each AI terminal agent, the following processing is performed:
[0007] initializing the AI terminal agent;
[0008] determining a current action that the AI terminal agent needs to perform in completing the task;
[0009] determining a current time step;
[0010] In the current time step, after the AI terminal agent performs the current action, an observation reward of the AI terminal agent is obtained;
[0011] A baseline strategy expected reward of the AI terminal agent is determined;
[0012] A single-step regret value is calculated according to the observation reward and the baseline strategy expected reward;
[0013] The cumulative regret value of the AI terminal agent is determined according to the single-step regret value;
[0014] It is judged whether the AI terminal agent currently obtains an acceptable reward according to the cumulative regret value, and if not, the step of initializing the AI terminal agent is returned to be executed; and
[0015] If yes, the weight value of the current action performed by the AI terminal agent is updated, and the time step is incremented.
[0016] It is judged whether the agent system currently obtains an acceptable reward, and if not, the action strategy of each AI terminal agent in the agent system is restarted, and the processing performed for each AI terminal agent is returned to be executed.
[0017] The current action to be performed by the AI terminal agent when completing the task is determined, comprising:
[0018] The calculation is performed by using the following calculation formula:
[0019]
[0020] Wherein is the probability of the i-th AI terminal agent performing the k-th action; is the weight value of the k-th action of the i-th AI terminal agent, and the initial value is 1, K is the total number of actions that can be performed by the i-th AI terminal agent; and γ is a preset exploration parameter with a preset value;
[0021] The maximum probability value is selected from the K probabilities of the i-th AI terminal agent performing K actions;
[0022] The action used to obtain the maximum probability value is determined as the current action to be performed by the i-th AI terminal agent when completing the task.
[0023] The baseline strategy expected reward of the AI terminal agent is obtained, comprising:
[0024] The joint action probability distribution is initialized as q 0 , and a set of action combinations is sampled according to the distribution q 0 a joint action probability distribution q 0 is:
[0025]
[0026] is calculated by using the following calculation formula:
[0027]
[0028] wherein N is the number of AI terminal agents in the agent system; is the probability of the ith AI terminal agent performing the kth action in the tth time step; is the observed reward of the ith AI terminal agent performing the kth action in the tth time step, and K is the total number of actions that the ith AI terminal agent can perform; is the baseline strategy expected reward of the ith AI terminal agent in the tth time step.
[0029] The single-step regret value is calculated according to the observed reward and the baseline strategy expected reward, and the calculation includes:
[0030]
[0031] wherein, is the single-step regret value of the ith AI terminal agent in the tth time step; is the baseline strategy expected reward of the ith AI terminal agent in the tth time step, is the observed reward of the ith AI terminal agent in the tth time step.
[0032] The cumulative regret value of the AI terminal agent is determined according to the single-step regret value, and the determination includes:
[0033] is calculated by using the following calculation formula:
[0034]
[0035] wherein, is the cumulative regret value of the ith AI terminal agent after T cumulative time steps; T is the total number of updated time steps up to the current time.
[0036] The weight value of the current action performed by the AI terminal agent is updated, and the updating includes:
[0037] is calculated by using the following calculation formula:
[0038]
[0039] wherein γ is a preset exploration parameter with a preset value; ωt+1 i = ωt i + αt (Rt+1 i - ωt i ), (1) ωt+1 i = ωt i + αt (Rt+1 i - ωt i ), (1) is a reward estimate used to increase the weight of high-reward actions and reduce the weight of low-reward actions. The calculation formula of ωt+1 i is as follows:
[0040]
[0041] wherein is the probability of the i-th AI terminal agent executing the current action in the t-th time step.
[0042] The judgment agent system whether the current income is acceptable includes:
[0043] Define a coarse correlation equilibrium (CCE) constraint condition; the CCE constraint condition includes each policy combination executed by each AI terminal agent in the agent system in each time step;
[0044] Calculate the distance of the experience distribution to the CCE set; wherein the experience distribution includes each action combination executed by each AI terminal agent in the agent system in each time step observed in a preset time period in the past;
[0045] Determine the target CCE closest to the experience distribution;
[0046] Calculate the system expectation obtained by the action of the policy combination corresponding to the target CCE determined in the preset time period;
[0047] Determine whether the system expectation is greater than the preset lower bound of social welfare, if yes, determine that the agent system currently obtains an acceptable income, otherwise, determine that the agent system currently does not obtain an acceptable income.
[0048] According to the second aspect, a collaborative action device of a plurality of AI terminal agents is provided, the plurality of AI terminal agents form an agent system so as to complete a task together by the plurality of AI terminal agents; the device includes:
[0049] The single-agent reward processing module is configured to perform the following processing for each AI terminal agent: initializing the AI terminal agent; determining a current action to be performed by the AI terminal agent when completing the task; determining a current time step; obtaining an observed reward of the AI terminal agent after the AI terminal agent performs the current action in the current time step; determining a baseline strategy expected reward of the AI terminal agent; calculating a single-step regret value according to the observed reward and the baseline strategy expected reward; determining a cumulative regret value of the AI terminal agent according to the single-step regret value; judging whether the AI terminal agent currently obtains an acceptable reward according to the cumulative regret value, and if not, returning to the step of initializing the AI terminal agent; and if yes, updating a weight value of the current action performed by the AI terminal agent and incrementing the time step;
[0050] The system reward processing module is configured to judge whether the agent system currently obtains an acceptable reward, and if not, restart the action strategy of each AI terminal agent in the agent system and trigger the single-agent reward processing module to perform processing.
[0051] When determining the current action to be performed by the AI terminal agent when completing the task, the single-agent reward processing module is configured to perform:
[0052] The calculation is performed by using the following calculation formula:
[0053]
[0054] wherein is a probability of the i th AI terminal agent performing the k th action; is a weight value of the k th action of the i th AI terminal agent, and the initial value thereof is 1, K is a total number of actions that can be performed by the i th AI terminal agent, and γ is a preset exploration parameter having a preset value;
[0055] selecting a maximum probability value from K probabilities of the i th AI terminal agent performing K actions;
[0056] determining an action used to obtain the maximum probability value as the current action to be performed by the i th AI terminal agent when completing the task.
[0057] According to a third aspect, a computing device is provided, including a memory and a processor, the memory storing executable code, and the processor executing the executable code to implement the method of any of the embodiments of the application.
[0058] Therefore, each embodiment of the application has at least the following beneficial effects:
[0059] 1、The multiple AI terminal agents can better coordinate actions. The maximization of the self-reward of each heterogeneous AI terminal agent, i.e., the single AI terminal agent obtains an acceptable income, ensures that the agent system composed of the AI terminal agents reaches an optimal solution, i.e., the agent system also obtains an acceptable income.
[0060] 2、In the case of a constantly changing environment and incomplete information, the single AI terminal agent can explore the non-system specified action and realize the enhancement of the high-yield action weight and the reduction of the low-yield action weight by using the exploration parameter and updating the weight value of the action performed in the next time step.
[0061] 3、The AI terminal agent continuously using the experience strategy in the constantly changing rescue environment is not suitable. The strategy of the AI terminal agent in the dynamic and sudden environment is updated in real time by introducing the restart strategy mechanism.
[0062] 4、The embodiment of the application is suitable for the heterogeneous multi-agent system to process the multi-target task, and includes: 1) supporting the cooperative planning of the multi-agent system medical rescue, material transportation, obstacle cleaning and other antagonistic tasks; 2) adapting to the communication limited scene in the complex rescue environment which changes dynamically; 3) adaptive matching of the agent strategy update frequency and the environment change rate. BRIEF DESCRIPTION OF DRAWINGS
[0063] In order to more clearly illustrate the technical solutions of the embodiments of the application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiment or prior art description. Obviously, the drawings in the following description are some embodiments of the application, and for those skilled in the art, other drawings can also be obtained without creative labor on the basis of these drawings.
[0064] Figure 1 is a schematic diagram of the system architecture to which an embodiment of the application is applied.
[0065] Figure 2 is a schematic diagram of the system to which another embodiment of the application is applied.
[0066] Figure 3 is a flowchart of the cooperative action method of the multiple AI terminal agents in an embodiment of the application.
[0067] Figure 4 is a schematic diagram of the cooperative action method of the multiple AI terminal agents in an embodiment of the application.
[0068] Figure 5 is a structural schematic diagram of the cooperative action device of the multiple AI terminal agents in an embodiment of the application. DETAILED DESCRIPTION
[0069] The scheme provided by the present application is described below with reference to the drawings.
[0070] First of all, it should be noted that the terms used in the embodiments of the present application are merely for the purpose of describing specific embodiments and are not intended to limit the present application. The singular forms "a", "said" and "the" used in the embodiments of the present application and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise.
[0071] It should be understood that the term "and / or" used herein is merely to describe the association relationship of the associated objects, which means that there can be three relationships, for example, A and / or B can represent the three cases of A alone, A and B together, and B alone. In addition, the character " / " herein generally represents an "or" relationship between the front and rear associated objects.
[0072] In order to facilitate the understanding of the method provided by the present application, the system architecture related to and applicable to the present application is described first. As shown in Figure 1 、 Figure 2 , the system includes an agent system composed of a plurality of, i.e. at least two, AI terminal agents. It should be understood that Figure 1 、 2 The number of AI terminal agents in
[0073] In the embodiments of the present application, the cooperative action of a plurality of AI terminal agents can be applied to complete various types of tasks such as emergency rescue tasks. For example, earthquake rescue, forest fire extinguishing, etc. The AI terminal agent can be a drone, a robot, a rescue vehicle, etc.
[0074] Figure 3 is a flowchart of the cooperative action method of a plurality of AI terminal agents in an embodiment of the present application. Referring to Figure 3 , the plurality of AI terminal agents form an agent system so as to complete a task together by the plurality of AI terminal agents, and the method comprises:
[0075] In the agent system, the following processing is performed for each AI terminal agent, i.e. steps 301 to 319:
[0076] Step 301: initializing the AI terminal agent.
[0077] Step 303: determining the current action to be performed by the AI terminal agent in completing the task.
[0078] Step 305: determining the current time step.
[0079] Step 307: In the current time step, after the AI terminal agent performs the current action, the observation reward of the AI terminal agent is obtained.
[0080] Step 309: The baseline strategy expected reward of the AI terminal agent is determined.
[0081] Step 311: According to the observation reward and the baseline strategy expected reward, the single-step regret value is calculated.
[0082] Step 313: The cumulative regret value of the AI terminal agent is determined according to the single-step regret value.
[0083] Step 315: According to the cumulative regret value, it is judged whether the AI terminal agent currently obtains an acceptable reward, if not, return to step 301; if yes, execute step 317.
[0084] Step 317: The weight value of the current action performed by the AI terminal agent is updated.
[0085] Step 319: The time step is incremented.
[0086] Step 321: It is judged whether the agent system currently obtains an acceptable reward, if not, execute step 323, if yes, return to step 307.
[0087] Step 323: Restart the action strategy of each AI terminal agent in the agent system, and return to step 301.
[0088] As can be seen, by using the cooperative action method of the plurality of AI terminal agents shown in Figure 3 , the self-reward maximization of each heterogeneous AI terminal agent, i.e. the single AI terminal agent obtains an acceptable reward, while also ensuring that the agent system composed of each AI terminal agent reaches an optimal solution, i.e. the agent system also obtains an acceptable reward.
[0089] The following will explain each step shown in Figure 3 respectively.
[0090] When the agent system needs to perform a task, step 301 can be executed to initialize the AI terminal agent.
[0091] The strategy for performing the task can be set in advance, which includes: the actions respectively performed by each AI terminal agent in the agent system in a time period, such as a time step, set in advance, for example, within a time period, such as 5 minutes, the unmanned aerial vehicle performs the action of detecting the road condition on road B, the robot performs the action of carrying materials on road A, and the rescue vehicle performs the action of receiving materials.
[0092] Next, step 303: determine the current action that the AI terminal agent needs to perform in completing the task.
[0093] In an embodiment of the present application, the process of determining the current action that the AI terminal agent needs to perform in completing the task in step 303 includes:
[0094] The K probabilities of the i-th AI terminal agent performing K actions are calculated using the following calculation formula:
[0095]
[0096] Wherein is the probability of the i-th AI terminal agent performing the k-th action; is the weight value of the k-th action of the i-th AI terminal agent, and its initial value is 1, K is the total number of actions that the i-th AI terminal agent can perform; γ is a preset exploration parameter with a preset value;
[0097] Select the maximum probability value from the obtained K probabilities;
[0098] Determine the action used to obtain the maximum probability value as the current action that the i-th AI terminal agent needs to perform in completing the task.
[0099] Next, step 305 is performed: determine the current time step.
[0100] The time step is used for iteration, such as represented by the letter t.
[0101] Next, step 307 is performed: in the current time step, after the AI terminal agent performs the current action, the observation reward of the AI terminal agent is obtained.
[0102] Next, step 309 is performed: determine the baseline strategy expected reward of the AI terminal agent.
[0103] In an embodiment of the present application, in step 309, the baseline strategy expected reward of the AI terminal agent is determined, including:
[0104] Initialize the joint action probability distribution as q 0 , sample a set of action combinations 0 Then the joint action probability distribution q 0 is:
[0105]
[0106] The following calculation formula is used for calculation:
[0107]
[0108] wherein N is the number of AI terminal agents in the agent system; is the probability of the i-th AI terminal agent executing the k-th action in the t-th time step; is the observed reward of the i-th AI terminal agent executing the k-th action in the t-th time step, and K is the total number of actions executable by the i-th AI terminal agent; is the baseline strategy expected reward of the i-th AI terminal agent in the t-th time step.
[0109] Next, step 311 is performed: according to the observed reward and the baseline strategy expected reward, a single-step regret value is calculated.
[0110] This step 311 includes calculating by using the following calculation formula
[0111]
[0112] wherein, is the single-step regret value of the i-th AI terminal agent in the t-th time step; is the baseline strategy expected reward of the i-th AI terminal agent in the t-th time step, is the observed reward of the i-th AI terminal agent executing the current action in the t-th time step.
[0113] Next, step 313 is performed: according to the single-step regret value, a cumulative regret value of the AI terminal agent is determined.
[0114] This step 313 includes calculating by using the following calculation formula:
[0115]
[0116] wherein, is the cumulative regret value of the i-th AI terminal agent after T cumulative time steps; and T is the total number of updated time steps up to the current time.
[0117] Next, step 315 is performed: according to the cumulative regret value, it is determined whether the AI terminal agent has currently obtained an acceptable reward, if not, returning to step 301; if yes, step 317 is performed.
[0118] For the AI terminal agent, if the current acceptable reward is not obtained, it means that the action executed in the current time step may need to be replaced.
[0119] Next, step 317 is performed: the weight value of the current action executed by the AI terminal agent is updated.
[0120] In one embodiment of the present application, in step 317, the weight value of the current action performed by the AI terminal agent is updated, including:
[0121] The calculation is performed by using the following calculation formula:
[0122]
[0123] Wherein, γ is a preset exploration parameter with a preset value; is the weight value of the next time step after updating, is the weight value in the current time step; ω is a preset learning rate for controlling the updating range of the weight; K is the total number of actions that can be performed by the i-th AI terminal agent; is the reward estimate for increasing the weight of high-reward actions and reducing the weight of low-reward actions; The calculation formula of is as follows:
[0124]
[0125] Wherein is the probability of the i-th AI terminal agent performing the current action in the t-th time step.
[0126] For example, is usually set to for balancing the convergence speed and stability.
[0127] Next, step 319 is performed: incrementing the time step.
[0128] For example, t=t+1.
[0129] Next, step 321 is performed: judging whether the agent system currently obtains an acceptable income, if not, step 323 is performed, and if yes, step 307 is returned.
[0130] Each AI terminal agent respectively passes through the above steps 301 to 319, and then the self-reward maximization of the single AI terminal agent is realized, that is, the single AI terminal agent obtains an acceptable income.
[0131] In one embodiment of the present application, in step 321, judging whether the agent system currently obtains an acceptable income, including:
[0132] Defining a coarse correlation equalization CCE constraint condition; the CCE constraint condition includes each policy combination performed by each AI terminal agent in the agent system in each time step;
[0133] calculate the distance of the experience distribution to the CCE set; wherein, the experience distribution comprises: each action combination performed by each AI terminal agent in the agent system in each time step observed in a preset time period in the past;
[0134] determine the target CCE closest to the experience distribution;
[0135] calculate the system expectation obtained by the action under the target CCE corresponding strategy combination determined in the preset time period;
[0136] determine whether the system expectation is greater than the preset lower bound of social welfare, if yes, it is determined that the agent system currently obtains acceptable benefits, otherwise, it is determined that the agent system currently does not obtain acceptable benefits.
[0137] Step 323: restart the action strategy of each AI terminal agent in the agent system, and return to step 301.
[0138] Through the above steps 321 to 323, it is also ensured that the agent system composed of each AI terminal agent reaches the optimal solution, that is, the agent system also obtains acceptable benefits.
[0139] In an embodiment of the present application, the implementation process of step 321 comprises:
[0140] (1) define coarse correlation equilibrium CCE constraint condition:
[0141] If the joint probability distribution q is coarse correlation equilibrium, then and only if all single AI terminal agents i and any alternative action a' i ∈A i satisfy:
[0142] ∑ a-i ∈A -i q(a i ,a -i )[u i (a i ,a -i )-u i (a′ i ,a -i )]≥0, wherein: A i is the action space of single AI terminal agent i; q(a i ,a -i ) is the probability that single AI terminal agent i selects a i (that is, the action with the highest probability in the current time step) and other single AI terminal agent-i selects a -i , u i (·) is the utility function of single AI terminal agent i, and the equilibrium set is:
[0143]
[0144] (2) Calculate the distance of the experience distribution to the CCE set: the experience distribution embodies the combination of actions of each AI terminal agent, and the calculation process is to select an optimal action combination according to the global benchmark.
[0145] The goal is to find the CCE closest to the experience distribution, where the experience distribution is obtained by statistical data in the past time period, that is, the action frequency of each action combination observed in the past time period (it can be understood that the combination of actions with the highest probability in each iteration process), denoted as δ, called Dirac measure.
[0146] Where p is the norm distance, q t is the potential optimal strategy obtained by coarse correlation equalization, that is, the optimal action combination.
[0147] (3) Calculate whether the expected action of observing the strategy combination π in the T time period is greater than the specified social welfare lower bound:
[0148]
[0149] Where OPT-SW(g T ) is the optimal social welfare of centralized planning in the time range, λ and is a smooth parameter that measures the strength of game competition, β(C T ) is the number of action changes allowed by the dynamic benchmark C T Coverage of the optimal social welfare, g(T) is the sum of the regret value ε0 of the agent.
[0150] The various embodiments of the present application can achieve the following effects:
[0151] (1) In a complex emergency rescue scene, heterogeneous multi-agent needs to perform different task categories, and the tasks are often mutually antagonistic, the present application adjusts the probability of action of each agent in the system by adopting reinforcement high-yield action weight and coarse correlation equilibrium constraint. Realize the maximization of single AI terminal agent reward while ensuring that the multi-heterogeneous agent system reaches the optimal solution.
[0152] (2) In the case of dynamic changes in the environment and incomplete information, a single AI terminal agent can perform the next action according to the environment exploration (not necessarily performing the action in the action space specified by the system), avoiding system rigidity. The single AI terminal agent can obtain the maximum benefit in the dynamic environment, that is, if a larger benefit is obtained in the case of unawareness, the action is more beneficial, and the action benefit will be improved subsequently, achieving temporary response, and adjusting the strategy immediately after the information is recovered, avoiding conflict with other agents.
[0153] (3) The rescue environment is usually dynamic, and it may not be appropriate for the AI terminal agent to always follow the experience strategy. By detecting the mutation of the regret value of the single AI terminal agent and whether the agent system meets the expected lower bound of social welfare, it is judged whether the environment has changed sharply, and the mechanism of restarting the strategy is realized to realize the real-time updating of the strategy of the AI terminal agent in the dynamically changing environment.
[0154] (4) The rescue environment is harsh, and the AI terminal agent does not know the action information of other AI terminal agents. The embodiment of the present application realizes the action in the case of unknown strategy of other AI terminal agents. In addition, the rescue environment is usually dynamic, and it may not be appropriate for the AI terminal agent to always follow the same strategy. Therefore, the action strategy of the embodiment of the present application can be restarted.
[0155] Referring to Figure 4 In an embodiment of the present application, the following processing idea can be used: each AI terminal agent performs operations for its own task, calculates the benefit of the single AI terminal agent after this action, updates the weight using the benefit obtained by the single AI terminal agent, strengthens the reward of the high-benefit action of the single AI terminal agent, and weakens the reward of the low-benefit action of the single AI terminal agent. The rough correlation equilibrium is used to obtain the Nash equilibrium solution of the agent system closest to the past action combination experience distribution of the system, and the result is used as the reference strategy of the next step of the agent system, that is, the action combination of the agent system under the rough correlation equilibrium constraint is the best combination, and any change in the strategy of the AI terminal agent cannot obtain greater benefit. Finally, the cumulative regret value and the lower bound of the optimal social welfare are used to judge whether the single AI terminal agent and the agent system have obtained acceptable benefits. If not, it means that the system cannot adapt to the dynamically changing environment (that is, the frequency of updating the strategy of the agent is less than the frequency of changing the environment), and the whole agent system is restarted, and the specific implementation includes:
[0156] (1) Calculate the real-time return of a single AI terminal agent after performing an action, and dynamically update the action weight to strengthen high-yield strategies; wherein the strategies include: the respective actions of each AI terminal agent in the agent system pre-set in a time period, such as a time step, for example, within a time of 5 minutes, the UAV performs the action of detecting the road on B road, the robot performs the action of carrying materials on A road, and the rescue vehicle performs the action of receiving materials.
[0157] (2) Based on the historical action combination experience distribution, the joint probability distribution of minimizing the strategy distribution distance under the constraint of coarse correlation equilibrium (CCE) is solved to obtain the optimal Nash equilibrium solution of the agent system satisfying the condition as the benchmark strategy;
[0158] (3) Adopt cumulative regret value mechanism and social optimal welfare lower bound mechanism to judge the matching of agent strategy update frequency and environmental dynamic change, when the cumulative regret value of a single AI terminal agent exceeds the threshold or the return of the agent system is less than the optimal social welfare lower bound, trigger the system to restart the strategy, that is, to redistribute the above-mentioned task execution strategy.
[0159] For the above (1), calculating the real-time return of a single AI terminal agent after performing an action includes:
[0160] (1) According to the actual return of real-time action Calculate the reward estimate
[0161] (2) Adopt learning rate Control the weight update range;
[0162] The weight update formula is
[0163] The above coarse correlation equilibrium (CCE) constraint condition is defined as:
[0164] 1) For any agent i and its substitute action a' i ∈A i , satisfies
[0165] 2) Determine the optimal strategy distribution by minimizing the p-norm distance of the experience distribution δ and the CCE set
[0166] The above cumulative regret value mechanism specifically includes:
[0167] 1) Calculate the single-step regret value
[0168] 2) Accumulate the total regret value in T period
[0169] 3) When there is an agent i's , it is determined that the system policy update lags behind the environment change, triggering a global restart.
[0170] The above social welfare verification mechanism:
[0171] 1) Verify the effectiveness of system cooperation by comparing the actual social welfare expectation value with the lower bound of the optimal social welfare ;
[0172] 2) When verification fails, trigger system restart conditions.
[0173] Referring to Figure 5 , one embodiment of the present application proposes a multi-AI terminal agent cooperative action device, which comprises a plurality of AI terminal agents to form an agent system to complete a task together; the device comprises:
[0174] A single-agent income processing module 501 is configured to perform the following processing for each AI terminal agent: initializing the AI terminal agent; determining the current action that the AI terminal agent needs to perform when completing the task; determining the current time step; obtaining the observation income of the AI terminal agent after the AI terminal agent performs the current action in the current time step; determining the baseline strategy expected income of the AI terminal agent; calculating the single-step regret value according to the observation income and the baseline strategy expected income; determining the cumulative regret value of the AI terminal agent according to the single-step regret value; determining whether the AI terminal agent has obtained acceptable income according to the cumulative regret value, if not, returning to the step of initializing the AI terminal agent; and if yes, updating the weight value of the current action performed by the AI terminal agent, and incrementing the time step;
[0175] A system income processing module 502 is configured to determine whether the agent system has obtained acceptable income, if not, restart the action policy of each AI terminal agent in the agent system, and trigger the single-agent income processing module to perform processing.
[0176] In one embodiment of the device of the present application, the single-agent income processing module 501 is configured to perform the following when determining the current action that the AI terminal agent needs to perform when completing the task:
[0177] The following calculation formula is used for calculation:
[0178]
[0179] Wherein a probability of the i-th AI terminal agent performing the k-th action; a weight value of the k-th action of the i-th AI terminal agent, and the initial value is 1, K is the total number of actions that the i-th AI terminal agent can perform, and γ is a preset exploration parameter with a preset value;
[0180] selecting a maximum probability value from the K probabilities of the i-th AI terminal agent performing K actions;
[0181] determining the action used to obtain the maximum probability value as a current action that the i-th AI terminal agent needs to perform when completing the task.
[0182] An embodiment of the present application provides a computer readable storage medium, which stores a computer program, and the computer program causes a computer to execute the method in any one of the embodiments in the specification when the computer program is executed in the computer.
[0183] An embodiment of the present application provides a computing device, which comprises a memory and a processor, the memory stores executable code, and the processor executes the executable code to implement the method in any one of the embodiments in the specification.
[0184] It can be understood that the structure shown in the embodiments of the present application does not constitute a specific limitation on the device of the embodiments of the present application. In other embodiments of the specification, the device can include more or fewer components than the illustration, or combine certain components, or split certain components, or different component arrangements. The components shown can be implemented in hardware, software or a combination of software and hardware.
[0185] Each of the embodiments in the present application is described in a progressive manner, and the same and similar parts between the embodiments can be referred to each other. Each embodiment mainly describes the difference from other embodiments. Especially, for the device embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and the related parts can be referred to the part of the method embodiments.
[0186] Those skilled in the art should realize that, in one or more of the examples described above, the functions described in the present application can be implemented by hardware, software, a plug-in or any combination thereof. When implemented by software, the functions can be stored in a computer readable medium or transmitted as one or more instructions or codes on the computer readable medium.
[0187] The above detailed description of the specific embodiments of the present application has been given to illustrate the purpose, technical solutions and beneficial effects of the present application. It should be understood that the above description is only a specific embodiment of the present application and is not used to limit the protection scope of the present application. Any modification, equivalent replacement, improvement, etc. made on the basis of the technical solutions of the present application shall be included in the protection scope of the present application.
Claims
1. A method for the coordinated action of multiple artificial intelligence (AI) terminal agents, characterized in that, The multiple AI terminal agents form an intelligent agent system, enabling them to jointly complete a task; the method includes: For each AI terminal agent, the following processing is performed: Initialize the AI terminal agent; Determine the current action that the AI terminal agent needs to perform when completing the task; Determine the current time step; In the current time step, after the AI terminal agent performs the current action, the observation benefit of the AI terminal agent is obtained; Determine the expected return of the baseline strategy for this AI terminal agent; Calculate the single-step regret value based on the observed gains and the expected gains of the baseline strategy; The cumulative regret value of the AI terminal agent is determined based on the single-step regret value. Based on the cumulative regret value, determine whether the AI terminal agent has currently obtained an acceptable benefit. If not, return to the step of initializing the AI terminal agent; and, If so, update the weight value of the current action performed by the AI terminal agent and increment the time step; Determine whether the intelligent agent system has obtained an acceptable benefit. If not, restart the action strategy of each AI terminal agent in the intelligent agent system and return to execute the processing performed for each AI terminal agent.
2. The method according to claim 1, characterized in that, Determining the current action that the AI terminal agent needs to perform when completing the task includes: The calculation is performed using the following formula: in The probability that the i-th AI terminal agent will perform the k-th action; γ is the weight value of the k-th action of the i-th AI terminal agent, and its initial value is 1; K is the total number of actions that the i-th AI terminal agent can execute; γ is a preset exploration parameter with a preset value. Choose the maximum probability value from the K probabilities of the i-th AI terminal agent performing K actions; The action used when obtaining the maximum probability value is determined as the current action that the i-th AI terminal agent needs to perform when completing the task.
3. The method according to claim 1, characterized in that, The expected return of the baseline strategy obtained for the AI terminal agent includes: Initialize the joint action probability distribution as q 0 According to distribution q 0 Sample a set of action combinations Then the probability distribution of the joint action q 0 for: Calculate using the following formula: Where N is the number of AI terminal agents in the intelligent agent system; Let be the probability that the i-th AI terminal agent performs the k-th action at the t-th time step; The observed reward for the i-th AI terminal agent to perform the k-th action at the t-th time step, where K is the total number of actions that the i-th AI terminal agent can perform; Let be the expected return of the baseline strategy for the i-th AI terminal agent at the t-th time step.
4. The method according to claim 3, characterized in that, The calculation of the single-step regret value based on the observed returns and the expected returns of the benchmark strategy includes: in, Let be the single-step regret value of the i-th AI terminal agent at the t-th time step; Let i be the expected return of the baseline strategy of the i-th AI terminal agent at the t-th time step. Let be the observation gain of the i-th AI terminal agent at the t-th time step.
5. The method according to claim 4, characterized in that, The step of determining the cumulative regret value of the AI terminal agent based on the single-step regret value includes: The calculation is performed using the following formula: in, Let be the cumulative regret value of the i-th AI terminal agent after accumulating T time steps; T is the total number of time steps that have been updated up to the current execution point.
6. The method according to claim 1, characterized in that, The updating of the weight value of the current action performed by the AI terminal agent includes: The calculation is performed using the following formula: Wherein, γ is a preset exploration parameter with a preset value; The updated weight values will be used for the next time step. ω represents the weight value in the current time step; ω is the pre-set learning rate used to control the magnitude of weight updates; K is the total number of actions that the i-th AI terminal agent can execute. It is a reward estimate used to increase the weight of high-reward actions and decrease the weight of low-reward actions; The calculation formula is as follows: in Let be the probability that the i-th AI terminal agent performs the current action at the t-th time step.
7. The method according to claim 1, characterized in that, The determination of whether the intelligent agent system has obtained an acceptable benefit includes: Define the coarse correlation equilibrium (CCE) constraint condition; the CCE constraint condition includes the combination of strategies executed by each AI terminal agent in the agent system at each time step; Calculate the distance of the experience distribution to the CCE set; wherein, the experience distribution includes: the combination of actions performed by each AI terminal agent in the agent system at each time step observed in the past preset time period; Identify the target CCE that is closest to the empirical distribution; The system expectation obtained by calculating the actions of the strategy combination corresponding to the target CCE determined within a preset time period; Determine whether the expected social welfare of the system is greater than the preset lower bound. If it is, determine that the intelligent agent system has obtained an acceptable benefit; otherwise, determine that the intelligent agent system has not obtained an acceptable benefit.
8. A collaborative action device for multiple AI terminal intelligent agents, characterized in that, The multiple AI terminal agents form an intelligent agent system, enabling them to jointly complete a task; the device includes: The single-agent benefit processing module is configured to perform the following processing for each AI terminal agent: initialize the AI terminal agent; determine the current action that the AI terminal agent needs to perform when completing the task; determine the current time step; in the current time step, after the AI terminal agent performs the current action, obtain the observed benefit of the AI terminal agent; determine the baseline policy expected benefit of the AI terminal agent; calculate the single-step regret value based on the observed benefit and the baseline policy expected benefit; determine the cumulative regret value of the AI terminal agent based on the single-step regret value; determine whether the AI terminal agent has obtained an acceptable benefit based on the cumulative regret value; if not, return to the step of initializing the AI terminal agent; and if yes, update the weight value of the current action performed by the AI terminal agent and increment the time step. The system revenue processing module is configured to determine whether the intelligent agent system has obtained an acceptable revenue. If not, it restarts the action strategies of each AI terminal intelligent agent in the intelligent agent system and triggers the single intelligent agent revenue processing module to perform processing.
9. The apparatus according to claim 8, characterized in that, When the single-agent revenue processing module determines the current action that the AI terminal agent needs to perform to complete the task, it is configured to execute: The calculation is performed using the following formula: in The probability that the i-th AI terminal agent will perform the k-th action; γ is the weight value of the k-th action of the i-th AI terminal agent, and its initial value is 1; K is the total number of actions that the i-th AI terminal agent can execute; γ is a preset exploration parameter with a preset value. Choose the maximum probability value from the K probabilities of the i-th AI terminal agent performing K actions; The action used when obtaining the maximum probability value is determined as the current action that the i-th AI terminal agent needs to perform when completing the task.
10. A computing device comprising a memory and a processor, wherein the memory stores executable code, and the processor, when executing the executable code, implements the method of any one of claims 1-7.