Multi-agent dynamic task allocation method and system based on fuzzy reasoning, and medium

Abstract

The application discloses a multi-agent task allocation method and system based on fuzzy reasoning, and a medium, and relates to the field of multi-agent reinforcement learning.The method comprises the following steps: acquiring multi-dimensional features of all agents at historical time points and multi-dimensional features of all subtasks at historical time points, and determining a subtask selector network based on fuzzy reasoning according to the mean and covariance of all subtasks; training the subtask selector network and a subtask policy network by using a subtask evaluation network, with the minimum first TD loss function as the target; training an agent credit allocation network, with the minimum second TD loss function as the target; and inputting local observation information of each agent at the current time point, an execution action at the previous time point and a subtask at the previous time point into the trained agent policy network and the subtask selector network based on fuzzy reasoning in sequence, to perform subtask allocation.The application improves the efficiency and accuracy of multi-agent task allocation.

Description

Technical Field

[0001] This application relates to the field of multi-agent reinforcement learning technology, and in particular to a multi-agent task allocation method, system and medium based on fuzzy reasoning. Background Technology

[0002] With the introduction of the new generation of artificial intelligence development plans, autonomous intelligent systems have become a key focus of AI development. The tasks performed by intelligent agents have also shifted from simple and static to complex and dynamic, such as complex tasks like mountain target search. Compared to single agents, multi-agent systems have significant advantages in performing complex tasks, and have therefore seen widespread development in both military and civilian fields. In most collaborative tasks, existing multi-agent reinforcement learning methods have achieved good performance. However, facing more complex collaborative task scenarios and changing task requirements, due to the increased overall complexity of collaborative tasks, existing multi-agent reinforcement learning methods struggle to quickly and accurately assign functionally appropriate sub-tasks to the agents.

[0003] To address the aforementioned issues and better apply prior human knowledge to task allocation, several multi-agent cooperative policy learning methods based on predefined methods, offline learning, and imitation learning have been proposed. Among these, multi-agent reinforcement learning methods based on value function decomposition are the most widely studied. To address the difficulty of directly learning joint policies under dynamically changing environments and varying task requirements, multi-agent reinforcement learning based on task decomposition and allocation has been proposed. These methods decompose complex tasks into multiple meta-subtasks, combine distributed agents into a multi-agent group through task allocation, and assign subtasks to one or more agents for execution, achieving collaborative completion of the predetermined task objective.

[0004] However, most existing value decomposition-based task decomposition and allocation methods often rely on limited information and may not fully consider factors such as the capabilities, types, and responsibilities of each agent, thus failing to achieve fast and accurate task allocation. Furthermore, these methods are bottom-up task selection processes, lacking a top-down task allocation optimization and update process, which may lead to multiple agents competing to execute the same subtask, resulting in inefficient subtask allocation. Summary of the Invention

[0005] The purpose of this application is to provide a multi-agent task allocation method, system, and medium based on fuzzy reasoning, which improves the efficiency and accuracy of multi-agent task allocation.

[0006] To achieve the above objectives, this application provides the following solution:

[0007] Firstly, this application provides a multi-agent task allocation method based on fuzzy reasoning, the multi-agent task allocation method based on fuzzy reasoning includes:

[0008] Define the collaborative task scenario of the intelligent agent team; the intelligent agent team includes multiple intelligent agents; the collaborative task scenario includes multiple sub-tasks.

[0009] The collaborative task scenario is adaptively decomposed using a Gaussian fitting process to determine the mean and covariance of each sub-task, and the mean and covariance of each sub-task are updated online.

[0010] The system acquires multidimensional features of all agents' historical moments and multidimensional features of all subtasks' historical moments, and determines a subtask selector network based on fuzzy inference based on the mean and covariance of all subtasks; the multidimensional features include: action sequence, historical trajectory and contribution value.

[0011] A hierarchical collaborative architecture with dual time scales is constructed. The top layer of the hierarchical collaborative architecture includes: a subtask selector network based on fuzzy reasoning, a subtask evaluation network, and multiple subtask policy networks. The bottom layer of the hierarchical collaborative architecture includes: an agent credit allocation network and an agent policy network. The subtask evaluation network is used to evaluate the execution progress of each subtask and determine the total task value. The total task value represents the execution progress of the collaborative task scenario. The subtask policy network is used to determine the multidimensional features of the subtasks. The agent credit allocation network is used to evaluate the contribution of each agent to the agent team and determine the team value. The team value represents the overall efficiency of the agent team in the collaborative task scenario. The agent policy network is used to determine the multidimensional features and action values ​​of the agents.

[0012] Using the subtask evaluation network, the subtask selector network based on fuzzy reasoning and the subtask policy network are trained with the goal of minimizing the first TD loss function; the intrinsic reward of the subtask and the cumulative environmental reward are added to the first TD loss function.

[0013] The agent policy network is trained using the agent credit allocation network with the objective of minimizing the second TD loss function;

[0014] The local observation information of each agent at the current moment, the action executed at the previous moment, and the subtask at the previous moment are input into the trained agent policy network to update the multidimensional features of each agent; the local observation information includes at least: the position and velocity of the current agent, and the position and velocity of other agents within the observation range of the current agent;

[0015] The updated multidimensional features of each agent are input into the trained fuzzy inference-based subtask selector network for subtask allocation.

[0016] Optionally, the collaborative task scenario is adaptively decomposed using a Gaussian fitting process to determine the mean and covariance of each sub-task, and the mean and covariance of each sub-task are updated online, specifically including:

[0017] Obtain a set of supervised offline datasets; the offline datasets include: the historical trajectories of agents performing each subtask, and the environmental reward for each agent;

[0018] Using the historical trajectory of the agent performing each subtask as input and the environmental reward of each agent as output, the reward function for each subtask is obtained by Gaussian fitting.

[0019] The mean and covariance of each subtask are determined based on the reward function.

[0020] The mean and covariance of each subtask are updated online using the negative log-likelihood function.

[0021] Optionally, multidimensional features of all agents' historical moments and multidimensional features of all subtasks' historical moments are obtained, and a subtask selector network based on fuzzy inference is determined according to the mean and covariance of all subtasks, specifically including:

[0022] Define the basic rules for TSK format;

[0023] Under the aforementioned basic rules, each subtask is treated as a fuzzy set. Based on the multidimensional features of each agent at historical moments, the multidimensional features of each subtask at historical moments, and the mean and covariance of each subtask, a Gaussian membership function is constructed for each fuzzy set.

[0024] The basic rules are adjusted based on the Gaussian membership functions under all fuzzy sets to obtain fuzzy inference rules for the collaborative task scenario.

[0025] The fuzzy inference rules are applied to the subtask selector network to obtain a fuzzy inference-based subtask selector network; the subtask selector network includes a fully connected layer, a fuzzy inference layer and a normalization layer connected in sequence.

[0026] Optionally, the first TD loss function is:

[0027]

[0028] In the formula, For the first TD loss, θ φ ξ is a parameter of the subtask policy network. φ ε is used to evaluate the network parameters for sub-tasks. φ The parameters of the subtask selector network based on fuzzy reasoning are... R represents the expected value of the cumulative reward at the top level of the hierarchical collaborative architecture. T The cumulative reward for multiple steps in a sub-task, where γ is the discount factor. Evaluate the overall objective value of the network for each subtask. To evaluate the actual total task value of the network for subtasks, s T For the global environmental information at time T, φ T Let r be the subtask executed by all agents at time T. int As an intrinsic reward for sub-tasks, Let δ be the environmental reward at time t. c Choose a frequency for the subtask, where T > t.

[0029] Optionally, the second TD loss function is:

[0030]

[0031] In the formula, For the second TD loss, θ a Let ξ be the parameter of the agent policy network. a Parameters for the agent credit allocation network. This represents the expected value of the cumulative reward at the bottom layer of the hierarchical collaborative architecture. Let γ be the environmental reward at time t, and γ be the discount factor. The target team value for the credit allocation network for intelligent agents. The actual team value of the agent credit allocation network, s t Let u be the local observation information of all agents at time t. t Let t represent the actions performed by all agents at time t.

[0032] Optionally, the subtask policy network includes a fully connected layer, a recurrent layer, and a fully connected layer connected in sequence; the subtask evaluation network is a hybrid network in the QMIX algorithm.

[0033] Optionally, the agent policy network and the agent credit allocation network are the policy network and hybrid network in the VDN algorithm, respectively.

[0034] Optionally, the hierarchical collaborative architecture further includes: a top-level experience pool and a bottom-level experience pool;

[0035] The top-level experience pool is used to store the first data tuple at different times; the first data tuple includes at least: global environmental information, IDs of all subtasks executed by all agents, and multi-step cumulative rewards for all subtasks; the global environmental information includes at least: the type, position, and velocity of all agents, and the type and position of environmental entities; the multi-step cumulative rewards include: the intrinsic rewards of the subtasks and the cumulative environmental rewards;

[0036] The underlying experience pool is used to store second data tuples at different times; the second data tuple includes at least: local observation information, actions performed, and environmental rewards of all agents.

[0037] Secondly, this application also provides a computer system, including: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the multi-agent task allocation method based on fuzzy reasoning.

[0038] Thirdly, this application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the aforementioned multi-agent task allocation method based on fuzzy reasoning.

[0039] According to the specific embodiments provided in this application, the following technical effects are disclosed:

[0040] This application addresses collaborative task scenarios involving multiple agents. It employs a Gaussian fitting process to adaptively decompose the subtasks, obtaining the mean and variance of each subtask. Based on this, it combines the agent's action sequence, historical trajectory, and contribution value to form unique multi-dimensional agent features. These features are then used to design and construct a fuzzy inference-based subtask selector network, enabling dynamic and accurate subtask allocation. Furthermore, based on the subtask evaluation network and agent credit allocation network, the training method for other networks in the hierarchical collaborative architecture aims to minimize the Temporal Difference (TD) loss function. This allows the agent's subtask allocation process to more effectively adapt to dynamic changes in the environment and task requirements. Finally, the dual-timescale hierarchical collaborative architecture constructed in this application can generate a globally-considered subtask allocation result at the top level and input it into the bottom level for agent decision-making. Therefore, this application improves the efficiency and accuracy of multi-agent task allocation. Attached Figure Description

[0041] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0042] Figure 1 A flowchart of a multi-agent task allocation method based on fuzzy reasoning provided in an embodiment of this application;

[0043] Figure 2 An internal structure diagram of the dual-time-scale hierarchical collaborative architecture provided in the embodiments of this application;

[0044] Figure 3 This is a schematic diagram of the structure of a computer system provided in an embodiment of this application. Detailed Implementation

[0045] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0046] The purpose of this application is to provide a multi-agent task allocation method, system, and medium based on fuzzy reasoning, which improves the efficiency and accuracy of multi-agent task allocation.

[0047] To make the above-mentioned objectives, features and advantages of this application more apparent and understandable, the application will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0048] Example 1

[0049] This embodiment provides a multi-agent task allocation method based on fuzzy reasoning, such as... Figure 1 As shown, this multi-agent task allocation method based on fuzzy reasoning specifically includes:

[0050] Step S1: Determine the collaborative task scenario of the agent team; the agent team includes multiple agents; the collaborative task scenario involves multiple sub-tasks.

[0051] In one example, the intelligent agent team consists of a formation of N unmanned surface vessels (USVs). Their collaborative mission scenario involves guarding a circular area with a radius of 1 km within a 7.5 km × 7.5 km sea area. In this collaborative mission scenario, the USVs can perform δ... k Sub-tasks, such as: core area search, mission point exploration, and tracking of unfamiliar ships.

[0052] Step S2: Adaptively decompose the collaborative task scenario using Gaussian fitting to determine the mean and covariance of each sub-task, and update the mean and covariance of each sub-task online.

[0053] In this embodiment, step S2 specifically includes:

[0054] Step S21: Obtain a set of supervised offline datasets; the offline datasets include: the historical trajectories of the agents performing each subtask, and the environmental rewards for each agent.

[0055] Step S22: Using the historical trajectory of the agent performing each subtask as input x and the environmental reward of each agent as output y, Gaussian fitting is used to obtain the reward function f(x) for each subtask.

[0056] Step S23: Determine the mean σ and covariance l of each subtask based on the reward function f(x); where the expression for the reward function f(x) is:

[0057] f(x) = k T (K+μ 2 I) -1 y;

[0058] In the formula, k is the kernel vector, K is the kernel matrix, and μ 2 Let I be the variance of the noise, and I be the identity matrix; the kernel vector k satisfies:

[0059] k = [K SE (x1,x2),…,K SE (x i ,x j )];

[0060]

[0061] In the formula, K SE (·) is the Gaussian kernel function, x i ,x j For different input data (belonging to offline datasets).

[0062] Step S24: Use the negative log-likelihood function to update the mean and covariance of each subtask online.

[0063] Step S3: Obtain the multidimensional features of all agents' historical moments and the multidimensional features of all subtasks' historical moments, and determine the subtask selector network based on fuzzy inference according to the mean and covariance of all subtasks; the multidimensional features include: action sequence, historical trajectory and contribution value.

[0064] In the multidimensional features of an agent, the action sequence represents the actions performed by the agent at different times, the historical trajectory represents the trend of the agent's movement changes, and the contribution value represents the contribution of the agent's actions to the collaborative task scenario. Since there may be more than one agent performing a certain sub-task, the multidimensional features of the corresponding sub-task are obtained by averaging the multidimensional features of multiple agents.

[0065] In one example, the multidimensional features of agent a satisfy: Where, x a For the multidimensional features of agent a, For x a The i-th feature in, ea For an action sequence, τ a For historical trajectory, q a To contribute value.

[0066] In this embodiment, step S3 specifically includes:

[0067] Step S31: Define the basic rules in the form of TSK (Takagi-Sugeno-Kang).

[0068] In one example, if Belongs to fuzzy sets Belongs to fuzzy sets Belongs to fuzzy sets Then the corresponding rule output f l = [0,…,0,1,0,…,0].

[0069] Step S32: Under the basic rules, treat each subtask as a fuzzy set, and construct a Gaussian membership function for each fuzzy set based on the multidimensional features of each agent's historical moments, the multidimensional features of each subtask's historical moments, and the mean and covariance of each subtask.

[0070] The Gaussian membership function is:

[0071]

[0072] In the formula, x φ For the multidimensional features of subtask φ. Let φ be the fuzzy set corresponding to the subtask φ. For x a Gaussian membership degree.

[0073] Step S33: Adjust the basic rules according to the Gaussian membership functions under all fuzzy sets to obtain fuzzy inference rules for collaborative task scenarios.

[0074] Specifically, under the basic rules of the TSK form, the rule output f can be determined by comparing the magnitudes of the Gaussian membership degrees corresponding to each fuzzy set. l By analyzing the values ​​at different positions, we can obtain complete fuzzy inference rules, and the number of fuzzy inference rules obtained is consistent with the number of fuzzy sets.

[0075] Step S34: Apply the fuzzy inference rules to the subtask selector network to obtain a fuzzy inference-based subtask selector network; the subtask selector network includes a fully connected layer, a fuzzy inference layer and a normalization layer connected in sequence.

[0076] Among them, the subtask selector network based on fuzzy inference can realize the automatic allocation of subtasks for the agent. First, the trigger strength of each fuzzy inference rule is calculated using the agent's multidimensional features and their corresponding Gaussian membership degrees, specifically:

[0077]

[0078] In the formula, η l (x a ) represents the Gaussian membership degree of agent a belonging to subtask φ under fuzzy inference rule l;

[0079] Then, the fuzzy inference results are defuzzified using a weighted average calculation method to determine the subtask result selected by agent a, specifically:

[0080]

[0081] In the formula, φ a The subtask selected for agent a, and m is the number of fuzzy inference rules.

[0082] Step S4: Construct a hierarchical collaborative architecture with dual time scales.

[0083] like Figure 2 As shown, the dual-timescale hierarchical collaborative architecture consists of two layers: a top layer and a bottom layer. The dual-timescale refers to the top layer having a time step of T and the bottom layer having a time step of t, where the two time steps satisfy: T > t and one T corresponds to δ. c The hierarchical collaborative architecture consists of several layers. The top layer comprises a subtask selector network based on fuzzy reasoning, a subtask evaluation network, and multiple subtask policy networks. The bottom layer comprises an agent credit allocation network and an agent policy network.

[0084] Specifically, the subtask evaluation network is used to evaluate the progress of each subtask and determine the total task value. The overall task value represents the execution progress of the collaborative task scenario; the sub-task policy network is used to determine the multi-dimensional features of the sub-tasks; and the agent credit allocation network is used to evaluate each agent's contribution to the agent team and determine the team value. Team value represents the overall efficiency of an agent team in collaborative task scenarios; agent policy network is used to determine the multidimensional features and action value of agents (for example: if an agent has three actions, such as going straight, turning left, and turning right, then the action value refers to the value of performing these three actions).

[0085] In this embodiment, the subtask policy network includes a fully connected layer, a recurrent layer, and a fully connected layer connected in sequence; the subtask evaluation network is a hybrid network in the QMIX algorithm; and the agent policy network and agent credit allocation network are the policy network and hybrid network in the VDN algorithm, respectively.

[0086] As a preferred implementation, the dual-timescale hierarchical collaboration framework further includes: a top-level experience pool and a bottom-level experience pool;

[0087] The top-level experience pool stores the first data tuples at different times. These first data tuples include: global environmental information, local observation information of all agents, IDs of subtasks executed by all agents, and multi-step cumulative rewards for all subtasks (composed of intrinsic rewards and cumulative environmental rewards). Specifically, the global environmental information includes: the type, position, and velocity of all agents, and the type and position of environmental entities (referring to the collaborative task scenario). Local observation information includes: the current agent's position and velocity, the positions and velocities of other agents within the current agent's observation range, and basic information (velocity, position) of other entities (obstacles other than agents) within the current agent's observation range.

[0088] The underlying experience pool is used to store the second data tuple at different times; the second data tuple includes: local observation information, actions performed, and environmental rewards (given by the human) of all agents.

[0089] Step S5: Using the subtask evaluation network, train the fuzzy inference-based subtask selector network and subtask policy network with the objective of minimizing the first TD loss function; the intrinsic reward r of the subtask is added to the first TD loss function. int and cumulative environmental rewards

[0090] The first TD loss function is:

[0091]

[0092] In the formula, For the first TD loss, θ φ ξ is a parameter of the subtask policy network. φ ε is used to evaluate the network parameters for sub-tasks. φ The parameters of the subtask selector network based on fuzzy reasoning are... R represents the expected value of the cumulative reward at the top level of the hierarchical collaborative architecture. T The cumulative reward for multiple steps in a sub-task, where γ is the discount factor. To evaluate the total objective value of the network for sub-tasks, s T For the global environmental information at time T, φ T Let T be the subtask executed by all agents at time T. Let δ be the environmental reward at time t. c Choose a frequency for the subtask.

[0093] Step S6: Use the agent credit allocation network to train the agent policy network with the goal of minimizing the second TD loss function.

[0094] The second TD loss function is:

[0095]

[0096] In the formula, For the second TD loss, θ a Let ξ be the parameter of the agent policy network. a Parameters for the agent credit allocation network. This represents the expected value of the cumulative reward at the bottom layer of the hierarchical collaborative architecture. The target team value of the agent credit allocation network, s t Let s be the local observation information of all agents at time t. t satisfy Let u be the local observation information of agent N at time t. t Let u be the action executed by all agents at time t. t satisfy Let t be the action performed by agent N at time t.

[0097] Step S7: Input the local observation information of each agent at the current time step, the action executed at the previous time step, and the subtask at the previous time step into the trained agent policy network to update the multidimensional features of each agent.

[0098] Step S8: Input the updated multidimensional features of each agent into the trained fuzzy inference-based subtask selector network to perform subtask allocation.

[0099] In this embodiment, each agent performs actions under the trained agent policy network according to the assigned subtask. Because the agent policy network can determine the value of the agent's actions, the subtask assigned in step S8 is input into the trained agent policy network to evaluate the value of all actions under that subtask, and the action with the highest value is selected for execution.

[0100] In summary, under the hierarchical collaborative architecture with dual time scales, the subtask allocation process and policy learning process are trained separately, enabling collaborative policy learning among multiple agents. This improves the accuracy of subtask allocation for multiple agents in complex task scenarios and accelerates the completion efficiency of collaborative tasks. Furthermore, it should be noted that steps S1–S6 are required during the initial training; subsequent practical applications only require steps S7–S9.

[0101] Example 2

[0102] This embodiment provides a practical application scenario for a multi-agent task allocation method based on fuzzy reasoning, as detailed below:

[0103] The first step is to initialize a key area collaborative defense mission scenario involving multiple unmanned surface vessels.

[0104] (1) Confirm that the number of our unmanned surface vessels is N, and the sensing range of the unmanned surface vessel sensors is C. a ={(xx i ) 2 +(yy i ) 2 =d 2 |A(x i ,y i ),d};A(·) is the center of the unmanned surface vessel, (x i ,y i ) represents the coordinates of the unmanned surface vessel (USV), and d represents the sensing radius of the USV's sensors.

[0105] At the same time, initialize the location of the task point to be investigated. And the range of the region C = {(xa)} 2 +(yb) 2 =r 2 |O(a,b),r};m is the number of task points, O(·) is the center of the task point circle, (a,b) is the coordinate of the task point, and r is the radius of the scouting task point.

[0106] (2) Within the perception range of our unmanned surface vessels, the number of enemy unmanned surface vessels is N. e And the required number of enemy unmanned surface vessels to track, attribute e j ∈I={1,2}, this attribute indicates that the enemy unmanned surface vessel requires 1 or 2 friendly unmanned surface vessels to track it.

[0107] (3) Based on the type of the key area collaborative defense mission scenario, initialize three sub-tasks, namely key area search, mission point exploration and unfamiliar vessel tracking.

[0108] The second step is to clarify the objectives of the key area collaborative defense mission: Under the premise of collision avoidance, the red and blue sides are the defender and attacker, respectively. Our objective is to have 5 red unmanned surface vessels (USVs) guard the central circular key area C, maximize the search range of the vicinity, detect any USVs that appear as soon as possible, prevent the key area from being attacked by enemy USVs on the periphery, and conduct close-range reconnaissance of mission points appearing in the scene. Our USVs select the appropriate sub-task to execute based on the action sequence, historical trajectory, contribution value, and global environmental information through a fuzzy logic reasoning process. The sub-task selection frequency is predefined according to the complexity of the key area collaborative defense mission, and a two-layer experience pool is constructed and updated to store information such as the local observations, actions, environmental rewards, and sub-task selections of USVs during mission execution.

[0109] (1) A fully cooperative multi-unmanned surface vessel mission can be described as a locally observable Markov process, with a tuple describing the true state of the environment.<S,U,P,r,O> S is the state space, U is the action space, P is the state transition function, r is the same reward function shared by all unmanned surface vessels (USVs), and O is the local observation space of the USV.

[0110] (2) Based on the hierarchical collaborative architecture with dual time scales, construct and initialize the bottom-level experience pool and the top-level experience pool. In addition, it is also necessary to initialize the number of subtasks and the subtask selection frequency.

[0111] The third step is to design the reward function for the subtasks and determine the subtask types and characteristics.

[0112] (1) Based on predefined subtask types (key domain search, task point exploration and unfamiliar ship tracking), Gaussian fitting is performed on the subtasks using a set of supervised offline datasets.

[0113] (2) For any subtask, the historical trajectory of the unmanned surface vessel selected for that subtask is used as input and the environmental reward of the unmanned surface vessel is used as output. The reward function of each subtask is fitted and satisfies a Gaussian distribution.

[0114] (3) Use the negative log-likelihood function to update the two hyperparameters, mean and covariance, online.

[0115] The fourth step involves constructing a fuzzy inference rule base for subtask selection, determining the multidimensional features of the unmanned surface vessel (USV), and obtaining the corresponding multidimensional features for each subtask based on these USV features. Each subtask is treated as a fuzzy set, and based on the hyperparameters updated online during the Gaussian fitting process of the reward function, a Gaussian membership function is constructed for each fuzzy set (i.e., subtask), resulting in fuzzy inference rules for search, exploration, and tracking within that subtask scenario.

[0116] (1) Each subtask is a different fuzzy set of fuzzy reasoning. Based on the Gaussian fitting process of the subtask, the Gaussian membership function under each fuzzy set is obtained.

[0117] (2) Combining the multidimensional features of unmanned surface vessels and sub-tasks, construct fuzzy reasoning rules in the form of TSK.

[0118] The fifth step is to design a subtask selector network based on fuzzy reasoning. The subtask selector network is used to assign the most suitable subtask to each unmanned surface vessel (USV) for its current state. Through real-time reasoning of the fuzzy reasoning rule base mentioned above, the subtask with the highest matching degree of each USV's current duty is output.

[0119] (1) Calculate the triggering strength of each fuzzy rule by using the multidimensional features of the unmanned surface vessel and the membership function of the fuzzy set.

[0120] (2) The fuzzy reasoning results are defuzzified by using a weighted average calculation method.

[0121] The sixth step involves evaluating the value of subtasks under dynamic environmental conditions and changing task requirements, based on a subtask evaluation network. This network assesses the completion status of each subtask and achieves balanced subtask allocation based on global state information.

[0122] (1) Maintain a subtask policy network for each subtask, and define the subtask reward as the sum of intrinsic reward and environmental reward accumulated over multiple steps.

[0123] (2) For a subtask, the values ​​of the actions taken by the unmanned surface vessel executing the subtask at the current moment are accumulated to obtain the value of the subtask, and this value is input as a local value into the subtask evaluation network to obtain the total task value. The subtask evaluation network, subtask selector network, and multiple subtask policy networks are updated by minimizing the first TD loss function.

[0124] The seventh step involves the top layer dynamically allocating subtasks based on the dual-time-scale hierarchical collaborative architecture. The top layer then outputs the subtask allocation results to each unmanned surface vessel (USV) at the bottom layer. The USVs at the bottom layer then execute the task according to the policy network of the subtask they are currently assigned.

[0125] (1) At the top level, the subtask selector network based on fuzzy reasoning combines the multidimensional features of the unmanned surface vessel (USV) to infer and output the USV's subtasks through fuzzy rules, and evaluates the subtask value based on the hybrid network in the QMIX algorithm.

[0126] (2) At the bottom layer, the local observation information of the UAV at the current moment, the actions executed at the previous moment, and the selected sub-tasks are input into the UAV policy network to learn the policy, obtain the action value of the UAV, and input it as a local value into the hybrid network in the VDN algorithm to obtain the team value. The unmanned surface vessel (USV) credit allocation network and USV policy network are updated by minimizing the second TD loss function.

[0127] (3) Based on the first TD loss function and the second TD loss function, the networks in the top and bottom layers of the dual-timescale hierarchical collaborative architecture are optimized to determine the loss function for hierarchical learning:

[0128]

[0129] In the formula, The total loss represents the hierarchical collaborative architecture with two time scales.

[0130] Step 8: Experimental description and results.

[0131] The experimental environment in this embodiment consists of two parts: a "multi-UAV key area collaborative defense mission" scenario built based on Unity, and a "home resource collection in the benchmark multi-agent experimental environment MPE" scenario. In the multi-UAV key area collaborative defense mission, three sub-tasks are predefined: a key area collaborative search task, a task point exploration task, and an unfamiliar vessel tracking task. The UAVs collaboratively select these three sub-tasks without collision. The simulation environment has 5 UAVs, 3 task points, and 5 discrete "direction-velocity" actions. There are 4 unfamiliar vessels, and the attribute for tracking vessels is either 1 or 2. MPE is a collaborative multi-agent reinforcement learning environment designed under local observation conditions. The home resource collection scenario is selected as the benchmark experimental environment. The goal is to collect different colored resources near the home based on the different agents' collection capabilities for various types of resources under the sub-task of guarding intruders away from the home.

[0132] (1) The experimental results of the "multi-unmanned surface vessel key area collaborative defense mission" scenario are as follows:

[0133] This method was trained with 5 unmanned surface vessels (USVs), 3 mission points, and 4 unfamiliar vessels. The initial positions of the USVs and the mission points were randomized during the training process. The training rounds were 30,500,000, and the maximum number of steps for interaction with the environment in each round was 2,000.

[0134] In this study, the experimental results are represented by team rewards, with higher team rewards in a single round of experiments being better. In a task setting with 5 unmanned surface vessels (USVs), this method uses a sub-task selector network based on fuzzy inference to select appropriate sub-tasks for each USV based on its action sequence, historical trajectory, and contribution value. Considering the action sequence executed by USVs over a period of time, if a USV has continuously executed a certain type of sub-task, the current environment requires it to continue executing the same type of task. For example, if a USV has been continuously performing a key area collaborative search task recently, it needs to continue performing this task to maintain search coverage. Simultaneously, the historical trajectories of the current USV and other USVs are analyzed to understand past behavior patterns and task execution, using previous experience to make decisions. Finally, for the key area collaborative search task, the contribution value is evaluated based on the USV's search efficiency and coverage; for the task point exploration task, the contribution value is evaluated based on the USV's positioning accuracy and exploration efficiency; and for the unfamiliar vessel tracking task, the contribution value is evaluated based on the USV's target tracking capability. Thus, the sub-task selector network can optimally allocate USVs to execute different sub-tasks, maximizing overall task efficiency.

[0135] (2) The experimental results of the "Benchmark Multi-Agent Experimental Environment Home Resource Collection Task" scenario are as follows:

[0136] Table 1 shows the comparison results between the proposed method and various benchmark methods on resource collection tasks with different numbers of agents (the results are average team rewards). The proposed method and benchmark methods were trained in scenarios with n=4, n=5, and n being a random number (between 2 and 6), respectively. Training was performed 4 times in each scenario, with 50 inference rounds. The calculated average reward was used for comparison. Existing benchmark methods used for comparison included: Regularized softmax deep multi-agent q-learning (RES), Monotonic value function factorisation for deep multi-agent reinforcement learning (QMIX), and Learning to factorize with transformation for cooperative multi-agent reinforcement learning (QTRAN). Simultaneously, under the proposed method, hyperparameter comparison experiments were conducted with different numbers of sub-tasks (m=3, 4, 5).

[0137] Table 1 Comparison of different methods in resource collection tasks involving the number of agents.

[0138]

[0139] Example 3

[0140] This embodiment provides a computer system, which can be a server or a terminal, and its internal structure diagram can be as follows. Figure 3 As shown, the computer system includes a processor, memory, input / output (I / O) interfaces, and a communication interface. The processor, memory, and I / O interfaces are connected via a system bus, and the communication interface is also connected to the system bus via the I / O interfaces. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and a database. The internal memory provides the environment for the operating system and computer programs stored in the non-volatile storage media. The database stores video tag processing data. The I / O interfaces are used for exchanging information between the processor and external devices. The communication interface is used for communication with external terminals via a network connection. When executed by the processor, the computer program implements a multi-agent task allocation method based on fuzzy reasoning.

[0141] Those skilled in the art will understand that Figure 3 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer system to which the present application is applied. A specific computer system may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0142] Example 4

[0143] This embodiment provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps in the above-described method embodiments.

[0144] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM).

[0145] The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc., and are not limited to these.

[0146] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. The same or similar parts between the various embodiments can be referred to each other.

[0147] This document uses specific examples to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the methods and core ideas of this application. Furthermore, those skilled in the art will recognize that, based on the ideas of this application, there will be changes in the specific implementation methods and application scope. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A multi-agent task allocation method based on fuzzy reasoning, characterized in that, The multi-agent task allocation method based on fuzzy reasoning includes: Define the collaborative task scenario of the intelligent agent team; the intelligent agent team includes multiple intelligent agents; the collaborative task scenario includes multiple sub-tasks. The collaborative task scenario is adaptively decomposed using a Gaussian fitting process to determine the mean and covariance of each sub-task, and the mean and covariance of each sub-task are updated online. The system acquires multidimensional features of all agents' historical moments and multidimensional features of all subtasks' historical moments, and determines a subtask selector network based on fuzzy inference based on the mean and covariance of all subtasks; the multidimensional features include: action sequence, historical trajectory and contribution value. A hierarchical collaborative architecture with dual time scales is constructed. The top layer of the hierarchical collaborative architecture includes: a subtask selector network based on fuzzy reasoning, a subtask evaluation network, and multiple subtask policy networks. The bottom layer of the hierarchical collaborative architecture includes: an agent credit allocation network and an agent policy network. The subtask evaluation network is used to evaluate the execution progress of each subtask and determine the total task value. The total task value represents the execution progress of the collaborative task scenario. The subtask policy network is used to determine the multidimensional features of the subtasks. The agent credit allocation network is used to evaluate the contribution of each agent to the agent team and determine the team value. The team value represents the overall efficiency of the agent team in the collaborative task scenario. The agent policy network is used to determine the multidimensional features and action values ​​of the agents. Using the subtask evaluation network, the subtask selector network based on fuzzy reasoning and the subtask policy network are trained with the goal of minimizing the first TD loss function; the intrinsic reward of the subtask and the cumulative environmental reward are added to the first TD loss function. The agent policy network is trained using the agent credit allocation network with the objective of minimizing the second TD loss function; The local observation information of each agent at the current moment, the action executed at the previous moment, and the subtask at the previous moment are input into the trained agent policy network to update the multidimensional features of each agent; the local observation information includes at least: the position and velocity of the current agent, and the position and velocity of other agents within the observation range of the current agent; The updated multidimensional features of each agent are input into the trained fuzzy inference-based subtask selector network for subtask allocation.

2. The multi-agent task allocation method based on fuzzy reasoning according to claim 1, characterized in that, The collaborative task scenario is adaptively decomposed using a Gaussian fitting process to determine the mean and covariance of each sub-task, and the mean and covariance of each sub-task are updated online, specifically including: Obtain a set of supervised offline datasets; the offline datasets include: the historical trajectories of agents performing each subtask, and the environmental reward for each agent; Using the historical trajectory of the agent performing each subtask as input and the environmental reward of each agent as output, the reward function for each subtask is obtained by Gaussian fitting. The mean and covariance of each subtask are determined based on the reward function. The mean and covariance of each subtask are updated online using the negative log-likelihood function.

3. The multi-agent task allocation method based on fuzzy reasoning according to claim 1, characterized in that, The process involves acquiring multidimensional features of all agents' historical moments and multidimensional features of all subtasks' historical moments, and determining a subtask selector network based on fuzzy inference based on the mean and covariance of all subtasks. Specifically, this includes: Define the basic rules for TSK format; Under the aforementioned basic rules, each subtask is treated as a fuzzy set. Based on the multidimensional features of each agent at historical moments, the multidimensional features of each subtask at historical moments, and the mean and covariance of each subtask, a Gaussian membership function is constructed for each fuzzy set. The basic rules are adjusted based on the Gaussian membership functions under all fuzzy sets to obtain fuzzy inference rules for the collaborative task scenario. The fuzzy inference rules are applied to the subtask selector network to obtain a fuzzy inference-based subtask selector network; the subtask selector network includes a fully connected layer, a fuzzy inference layer and a normalization layer connected in sequence.

4. The multi-agent task allocation method based on fuzzy reasoning according to claim 1, characterized in that, The first TD loss function is: In the formula, For the first TD loss, θ φ ξ is a parameter of the subtask policy network. φ ε is used to evaluate the network parameters for sub-tasks. φ The parameters of the subtask selector network based on fuzzy reasoning are... R represents the expected value of the cumulative reward at the top level of the hierarchical collaborative architecture. T The cumulative reward for multiple steps in a sub-task, where γ is the discount factor. Evaluate the overall objective value of the network for each subtask. To evaluate the actual total task value of the network for subtasks, s T For the global environmental information at time T, φ T Let r be the subtask executed by all agents at time T. int As an intrinsic reward for sub-tasks, Let δ be the environmental reward at time t. c Choose a frequency for the subtask, where T > t.

5. The multi-agent task allocation method based on fuzzy reasoning according to claim 1, characterized in that, The second TD loss function is: In the formula, For the second TD loss, θ a Let ξ be the parameter of the agent policy network. a Parameters for the agent credit allocation network. This represents the expected value of the cumulative reward at the bottom layer of the hierarchical collaborative architecture. Let γ be the environmental reward at time t, and γ be the discount factor. The target team value for the credit allocation network for intelligent agents. The actual team value of the agent credit allocation network, s t Let u be the local observation information of all agents at time t. t Let t represent the actions performed by all agents at time t.

6. The multi-agent task allocation method based on fuzzy reasoning according to claim 1, characterized in that, The subtask policy network comprises a fully connected layer, a recurrent layer, and a fully connected layer connected in sequence; the subtask evaluation network is a hybrid network in the QMIX algorithm.

7. The multi-agent task allocation method based on fuzzy reasoning according to claim 1, characterized in that, The agent policy network and the agent credit allocation network are the policy network and hybrid network in the VDN algorithm, respectively.

8. The multi-agent task allocation method based on fuzzy reasoning according to claim 1, characterized in that, The hierarchical collaborative architecture also includes: a top-level experience pool and a bottom-level experience pool; The top-level experience pool is used to store the first data tuple at different times; the first data tuple includes at least: global environmental information, IDs of all subtasks executed by all agents, and multi-step cumulative rewards for all subtasks; the global environmental information includes at least: the type, position, and velocity of all agents, and the type and position of environmental entities; the multi-step cumulative rewards include: the intrinsic rewards of the subtasks and the cumulative environmental rewards; The underlying experience pool is used to store second data tuples at different times; the second data tuple includes at least: local observation information, actions performed, and environmental rewards of all agents.

9. A computer system, comprising: A memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that the processor executes the computer program to implement the multi-agent task allocation method based on fuzzy reasoning as described in any one of claims 1-8.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by a processor, the computer program implements the multi-agent task allocation method based on fuzzy reasoning as described in any one of claims 1-7.

Images