A port multi-agent task allocation method and system based on reinforcement learning

Abstract

The present invention discloses a port multi-agent task allocation method and system based on reinforcement learning, which belongs to the field of logistics management. The method includes: constructing a task set for various operation tasks in a port multi-task environment; constructing a dynamic model of heterogeneous multi-agents for different types of port intelligent equipment; constructing a global environment model based on the task set and the dynamic model; constructing a collaborative multi-agent task allocation mechanism, and using the task allocation mechanism to allocate and schedule tasks; constructing a collaborative reinforcement learning model based on the task set, the dynamic model, the global environment model and the task allocation mechanism, and using the collaborative reinforcement learning model to train the task execution strategy of the agent. The present invention can decompose complex tasks into multiple subtasks through the hierarchical design of high-level policy networks and low-level policy networks, which is easy to manage and optimize, can effectively handle multi-task environments, and enable agents to flexibly respond in different task sets.

Description

Technical Field

[0001] The present invention belongs to the technical field of logistics management, and in particular relates to a port multi-agent task allocation method and system based on reinforcement learning. Background Art

[0002] The multi-agent task allocation problem in ports is a complex combinatorial optimization problem that involves multiple agents (such as automated container handlers, drones, etc.) to efficiently coordinate and cooperate in the port environment to complete task allocation and resource scheduling. Traditional task allocation methods mainly rely on centralized control and fixed rules, which are often difficult to adapt to dynamically changing environments and tasks, resulting in low efficiency and waste of resources. In recent years, with the improvement of port automation and intelligence, task allocation optimization algorithms based on multi-agent systems have gradually become a research hotspot.

[0003] Existing task allocation algorithms include exact algorithms, heuristic algorithms, and intelligent algorithms. Exact algorithms, such as linear programming and integer programming, can provide global optimal solutions, but in practical applications, they are difficult to cope with the real-time allocation of large-scale tasks due to their high computational complexity. Heuristic algorithms, such as greedy algorithms and genetic algorithms, although they have improved the efficiency of solving problems, still have difficulty in dealing with complex constraints and dynamically changing environments.

[0004] In recent years, reinforcement learning, as an emerging intelligent algorithm, has attracted widespread attention due to its superior performance in dealing with complex decision-making problems. Reinforcement learning can effectively cope with dynamic changes and uncertainties by learning the optimal strategy through interaction with the environment, and is particularly suitable for task allocation problems in multi-agent systems. By designing a suitable reward mechanism, reinforcement learning can guide the agent to autonomously learn the optimal task allocation strategy, thereby improving the overall efficiency and flexibility of the system. In addition, with the development of deep learning, the application of deep reinforcement learning algorithms in multi-agent environments has gradually increased, showing good performance. By combining deep learning with reinforcement learning, these algorithms can handle high-dimensional state spaces and complex task allocation scenarios, providing a new solution for the optimization of multi-agent task allocation in ports. Therefore, the port multi-agent task allocation optimization algorithm based on reinforcement learning has become the frontier direction of current research due to its high efficiency, adaptability and versatility.

[0005] In modern port operation management, effective resource allocation is the key to improving operational efficiency and competitiveness. However, existing resource allocation methods usually rely on static plans and manpower scheduling, showing the following problems: low resource utilization: traditional methods often schedule according to preset plans, fail to respond to dynamic changes in real time, resulting in idle or overused resources; slow response speed: due to the lack of real-time data support, port managers find it difficult to respond quickly to unexpected situations, such as sudden large-scale arrival of goods or equipment failures; lack of flexibility: most existing systems lack flexible scheduling capabilities and cannot make dynamic resource adjustments based on real-time changes, reducing the agility of port operations. Existing technologies still have a lot of room for improvement in terms of flexibility and real-time nature of resource allocation.

[0006] At present, in order to reduce operating costs, the industry widely uses the principle of fixed scheduling to manage resources, but its flexibility and real-time performance are insufficient. In addition, although the machine learning methods developed in recent years have proposed a predictive analysis framework, their practical application in the complex and changeable port environment is still limited. In modern port operations, efficient allocation of resources is the key to improving operational efficiency and reducing operating costs. However, traditional resource allocation methods often rely on experience and static plans, which are difficult to cope with the dynamically changing port environment. This leads to problems such as resource waste, operational delays and inefficiency. Therefore, there is an urgent need for an algorithm that can dynamically optimize resource allocation to improve the overall operational efficiency of the port. Summary of the invention

[0007] The present invention aims to solve the deficiencies of the prior art and provides the following solutions:

[0008] A port multi-agent task allocation method based on reinforcement learning includes the following steps:

[0009] Construct task sets for various operational tasks in the port multi-task environment;

[0010] Construct a dynamic model of heterogeneous multi-agents for different types of port intelligent equipment;

[0011] Based on the task set and the dynamic model, construct a global environment model;

[0012] Constructing a collaborative multi-agent task allocation mechanism, and utilizing the task allocation mechanism to allocate and schedule tasks;

[0013] A collaborative reinforcement learning model is constructed based on the task set, the dynamics model, the global environment model and the task allocation mechanism, and the collaborative reinforcement learning model is used to train the task execution strategy of the intelligent agent.

[0014] Preferably, the method for constructing a task set includes:

[0015] A task graph is constructed using the dependencies and constraints of the tasks, wherein each node in the task graph represents a port operation to be performed, and each edge represents the dependencies and constraints of the tasks;

[0016] Calculate task priorities based on the task graph:

[0017] ,

[0018] Among them, P(T i ) represents task T i The priority of i Represents task T i Resource requirements, d i Represents task T i , α1 and α2 represent weight coefficients;

[0019] Construct the task set using the task priority , where T n represents the nth task, where i and n represent natural numbers.

[0020] Preferably, the method for constructing a heterogeneous multi-agent dynamics model includes:

[0021] Defining an Agent Set , where I m represents the mth agent, where m represents a natural number;

[0022] Define the state space set , where s m (t) represents agent I m The state at time t, t represents the time;

[0023] Define the action space set , where a m (t) represents agent I m The action taken at time t;

[0024] The dynamic model is constructed based on the state and the action , and use the dynamics model to update the agent state:

[0025] ,

[0026] Where f( ) represents the dynamic model function of the intelligent agent, S m (t+1) represents the state update of the agent.

[0027] Preferably, the method for constructing the global environment model includes:

[0028] Based on the geographical location, cargo distribution and operation requirements of the port, the state space of the port global environment is constructed:

[0029] ,

[0030] Among them, h( ) represents the function of the interaction between the agent and the global environment, Represents Agent I m In the global environment, Loc m (t) represents agent I m At the geographical location at time t, Cargo m (t) represents agent I m The distribution of goods at time t, Req m (t) represents agent I m The job requirement at time t, ε represents the global environment variable.

[0031] Preferably, the method for allocating and scheduling tasks using the task allocation mechanism includes:

[0032] Each agent independently calculates the cost function :

[0033] ,

[0034] Among them, β1, β2 and β3 represent weight coefficients, dist(T i , I m ) represents task T i With Agent I m The distance between i ) indicates the priority of the task, c delay (T i , I m ) represents the delay cost;

[0035] Select the initial task bundle , calculate the task bundle based on the cost function Total cost:

[0036] ;

[0037] Agents exchange bundle information and check whether there is a task conflict. If there is a task conflict, the conflicting task will be assigned to the agent that can perform the conflicting task at the lowest cost. After that, the agent will update its own task bundle and recalculate the execution order and total cost of the tasks.

[0038] Repeat the steps and iterate multiple times until the task allocation mechanism gradually converges and completes the task allocation and scheduling.

[0039] Preferably, the collaborative reinforcement learning model includes: a strategy network and a value network;

[0040] The policy network is composed of a high-level policy network and a low-level policy network:

[0041] ,

[0042] ,

[0043] in, represents a high-level policy network, which is used to select subtasks based on the task set and the global environment model, θ represents the parameters of the high-level policy network, h represents the high-level, represents the high-level value function, o represents the subtask, and s represents the current state; represents the low-level policy network, which is used to generate specific actions of the agent in each subtask content. Represents the parameters of the low-level policy network, l represents the low-level, represents the low-level value function, a represents a specific action, and argmax represents the maximum value function Q;

[0044] The value network is composed of a high-level value network and a low-level value network. The high-level value network is used to evaluate the effect of selecting subtasks, and the low-level value network is used to evaluate the action performance of the intelligent agent when performing specific tasks.

[0045] Preferably, the method of using the collaborative reinforcement learning model to train the task execution strategy of an intelligent agent includes: in a multi-agent environment, the intelligent agent interacts with the global environment model, through centralized training, multiple intelligent agents share an experience replay pool, and collaboratively optimize through the strategy network and the value network to achieve the best coordination state.

[0046] The present invention also provides a port multi-agent task allocation system based on reinforcement learning, wherein the allocation system applies any of the allocation methods described above, and comprises: a task set construction module, a dynamic model construction module, an environment model construction module, a task allocation module and a task execution optimization module;

[0047] The task set building module is used to build a task set for various operation tasks in the port multi-task environment;

[0048] The dynamic model building module is used to build a dynamic model of heterogeneous multi-agents for different types of port intelligent equipment;

[0049] The environment model construction module constructs a global environment model based on the task set and the dynamic model;

[0050] The task allocation module is used to construct a collaborative multi-agent task allocation mechanism, and utilize the task allocation mechanism to allocate and schedule tasks;

[0051] The task execution optimization module constructs a collaborative reinforcement learning model based on the task set, the dynamics model, the global environment model and the task allocation mechanism, and uses the collaborative reinforcement learning model to train the task execution strategy of the intelligent agent.

[0052] Compared with the prior art, the present invention has the following beneficial effects:

[0053] (1) Through the hierarchical design of high-level policy networks and low-level policy networks, the present invention can decompose complex tasks into multiple subtasks, which is easy to manage and optimize. It can effectively handle multi-task environments, allowing intelligent agents to respond flexibly to different task sets, avoiding the problem of poor performance of traditional models under complex tasks. The high-level policy network is responsible for global task selection, and the low-level policy network focuses on the specific execution of local subtasks, reducing the computational burden of traditional models in complex decision-making. Through clear task division, the high-level and low-level networks optimize global task planning and local action execution respectively, which can significantly improve decision-making efficiency and learning speed;

[0054] (2) The present invention can decompose multiple tasks into subtasks and share subtask strategies among similar tasks. Through meta-training and other methods, the model can learn common strategies in multiple task environments, enhance the adaptability of the intelligent agent to new tasks, and solve the problem that strategies in traditional models are difficult to transfer and generalize. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] In order to more clearly illustrate the technical solution of the present invention, the following briefly introduces the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0056] Figure 1 A schematic diagram of a method flow chart of an embodiment of the present invention;

[0057] Figure 2 Schematic diagram of the process of S4 and S5 of the embodiment of the present invention. DETAILED DESCRIPTION

[0058] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0059] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0060] Embodiment 1

[0061] In this embodiment, if Figure 1 As shown, a port multi-agent task allocation method based on reinforcement learning includes the following steps:

[0062] S1. Construct a task set for various operation tasks in the port multi-task environment.

[0063] In this embodiment, first, various operation tasks in the port multi-task environment are identified and generated to construct a task set. Typical tasks of a port include loading and unloading, transportation, scheduling, etc., and each type of task has different constraints and execution requirements. By analyzing factors such as task priority, execution time, and resource requirements, all tasks are represented as a task set. In the process of generating the task set, the port operation tasks are modeled using graph theory, and the task graph is constructed using the dependencies and constraints of the tasks. Each task node represents a port operation that needs to be performed, and each edge represents the sequence or dependency between tasks. A task set is generated based on the task graph to provide a basis for subsequent task allocation.

[0064] Specifically, the method for constructing a task set includes: constructing a task graph using the dependencies and constraints of the tasks, wherein each node in the task graph represents a port operation to be performed, and each edge represents the dependencies and constraints of the tasks; and calculating the task priority based on the task graph:

[0065] ,

[0066] Among them, P(T i ) represents task T i The priority of i Represents task T i Resource requirements, d i Represents task T i , α1 and α2 represent weight coefficients; task set is constructed using task priority , where T nrepresents the nth task, where i and n represent natural numbers.

[0067] S2. Construct a dynamic model of heterogeneous multi-agents for different types of port intelligent equipment.

[0068] Different agents (such as AGV, crane, trailer, etc.) have different physical properties and task capabilities, so it is necessary to build a dynamic model for each agent. The dynamic model is used to describe the state of the agent, the action space, and its motion characteristics and behavior patterns in the port task environment. In this embodiment, the dynamic model of heterogeneous multi-agents is defined by the state-action space. The state of the agent includes its physical properties such as position, speed, and load condition, while the action space consists of the executable tasks of the agent (such as transportation, lifting, etc.) and its movement trajectory. For each agent, its dynamic model is constructed using physics and optimization theory, and provides a decision basis for subsequent task allocation.

[0069] Specifically, the method of constructing a heterogeneous multi-agent dynamics model includes: defining an agent set , where I m represents the mth agent, where m represents a natural number; defines the state space set , where s m (t) represents agent I m The state at time t, t represents the time, the state space S contains dynamic information in the port, such as ship location, cargo type, available resources, including current resource status, task queue, environmental conditions (such as weather, traffic); define the action space set , where a m (t) represents agent I m The action taken at time t, the action space A covers the set of tasks that the agent can perform, such as loading and unloading goods, dispatching equipment, etc.; construct a dynamic model based on states and actions , and use the dynamics model to update the agent state:

[0070] ,

[0071] Where f( ) represents the dynamic model function of the intelligent agent, S m (t+1) represents the state update of the agent.

[0072] S3. Build a global environment model based on the task set and dynamic model.

[0073] The global environment model is mainly used to describe the operation scene of the port, including the task location, resource distribution, and constraints of task execution. The global environment model integrates the task set, the dynamic model of the agent, and the physical structure information of the port as the environmental input for multi-agent task allocation. In this embodiment, the state space of the port global environment is constructed based on environmental information such as the geographical location, cargo distribution, and operation requirements of the port. Each agent will interact with the environment when performing tasks, and the information in the global environment model can provide complete environmental state input for multi-agent task allocation.

[0074] Specifically, the method of constructing the global environment model includes: constructing the state space of the port global environment based on the geographical location, cargo distribution and operation requirements of the port:

[0075] ,

[0076] Among them, h( ) represents the function of the interaction between the agent and the global environment, Represents Agent I m In the global environment, Loc m (t) represents agent I m At the geographical location at time t, Cargo m (t) represents agent I m The distribution of goods at time t, Req m (t) represents agent I m The job requirement at time t, ε represents the global environment variable.

[0077] S4. Build a collaborative multi-agent task allocation mechanism and use the task allocation mechanism to allocate and schedule tasks.

[0078] Task allocation adopts a collaborative multi-agent task allocation mechanism, which includes a consensus-based bundle algorithm (CBBA). The CBBA mechanism allows agents to bid for each task. Through the task auction and allocation process between agents, it ensures that each task is executed by the most suitable agent, and each agent can reach a consensus on local and global task requirements. Furthermore, each agent first generates a local task allocation strategy based on its own state and environmental information. Then, through the CBBA mechanism, each agent conducts task auctions and selects the task that it can perform best. Through multiple rounds of bidding and feedback, each agent reaches a consensus and completes the allocation and scheduling of tasks. This consensus mechanism can optimize resource utilization in multi-agent task allocation, and reduce task conflicts and resource competition through real-time collaboration between agents. The cost function It is used to calculate the cost of the agent to perform tasks. In this context, the agent task allocation strategy is to minimize the total cost through the reinforcement learning algorithm to find the optimal strategy .

[0079] Specifically, the method of allocating and scheduling tasks using the task allocation mechanism includes: each agent independently calculates the cost function :

[0080] ,

[0081] Among them, β1, β2 and β3 represent weight coefficients, dist(T i , I m ) represents task T i With Agent I m The distance between i ) indicates the priority of the task, c delay (T i , I m ) represents the delay cost; select the initial task bundle , calculate task bundles based on cost function Total cost:

[0082] ;

[0083] The agents exchange bundle information and check whether there are task conflicts. If there are task conflicts, the conflicting tasks will be assigned to the agent that can execute the conflicting tasks at the lowest cost. The agents will then update their own task bundles and recalculate the execution order and total cost of the tasks. Repeat the steps and iterate multiple times until the task allocation mechanism gradually converges and completes the task allocation and scheduling.

[0084] S5. Construct a collaborative reinforcement learning model based on the task set, dynamics model, global environment model and task allocation mechanism, and use the collaborative reinforcement learning model to train the task execution strategy of the intelligent agent.

[0085] In the port multi-agent task allocation problem, the collaborative reinforcement learning (CMARL) model is used to optimize the multi-agent task allocation strategy. Based on the aforementioned task set, dynamic model, environment model and CBBA consensus mechanism, the present invention constructs the CMARL task space in this step and trains the agent's task execution strategy through reinforcement learning.

[0086] The collaborative reinforcement learning model includes: a policy network and a value network; the policy network consists of a high-level policy network and a low-level policy network:

[0087] ,

[0088] ,

[0089] in, represents the high-level policy network, which is used to select subtasks based on the task set and the global environment model, θ represents the parameters of the high-level policy network, h represents the high-level, represents the high-level value function, o represents the subtask, and s represents the current state of the environment, which is the environmental information based on which the agent makes decisions. The state s provides global or local information about the environment to help the high-level policy network select the appropriate subtask o; Represents the low-level policy network, which is used to generate specific actions of the agent (such as moving, carrying, etc.) in each subtask content. Represents the parameters of the low-level policy network, l represents the low-level, represents the low-level value function, a represents the specific action, and represents the value of the function based on the parameter θ or , so that its corresponding value function Q reaches the maximum value, thereby determining the optimal action a under a given state s and subtask o. In this embodiment, the high-level policy network is responsible for selecting appropriate subtasks or options (Options) based on the current state of the agent, the context of the task set, and the environmental information. Its input is the global state in the task set, and its output is the selected subtask or option. The high-level policy network contains two hidden layers and one output layer; the low-level policy network is used to generate specific action outputs based on the subtasks selected by the high-level policy. The input of the low-level policy network is the state under the current subtask and the local environmental information of the agent, and the output is the execution action of the agent in the current subtask. The low-level policy network consists of three hidden layers and one output layer, and all hidden layers use Leaky ReLU as the activation function.

[0090] The value network consists of a high-level value network and a low-level value network. The high-level value network is used to evaluate the effect of selecting subtasks, and the low-level value network is used to evaluate the performance of the agent when performing specific tasks:

[0091] ,

[0092] ,

[0093] Among them, μ represents the learning rate, r h represents the return value of the high-level lRepresents the low-level reward value, γ represents the discount factor, s' represents the next state, that is, the state reached by the agent after executing the action in the current state s, o' represents the subtask that may be selected in the next state s', and a' represents the specific action that may be selected in the next state s' and subtask o'; the method of using the collaborative reinforcement learning model to train the task execution strategy of the agent includes: in a multi-agent environment, the agent interacts with the global environment model, through centralized training, multiple agents share the experience replay pool, and collaboratively optimize through the policy network and the value network to achieve the best coordination state. Furthermore, the collaborative optimization of the policy and value network requires the simultaneous adjustment of the policy network (parameter θ) and the value network (parameter ), so that the total value is maximized. This collaborative optimization mechanism achieves the best coordination between strategy execution and value assessment in multiple dimensions of complex tasks:

[0094] ,

[0095] Among them, s t represents the state at time step t, o t represents the subtask selected at time step t, a t represents the specific action performed at time step t; through reinforcement learning training, the agent can gradually learn the optimal task allocation strategy and execution strategy to improve the operating efficiency in the port multi-task environment. The process of steps S4 and S5 is as follows Figure 2 shown.

[0096] Embodiment 2

[0097] In this embodiment, a port multi-agent task allocation system based on reinforcement learning includes: a task set construction module, a dynamic model construction module, an environment model construction module, a task allocation module and a task execution optimization module.

[0098] The task set construction module is used to construct task sets for various operation tasks in the port multi-task environment; the dynamic model construction module is used to construct dynamic models of heterogeneous multi-agents for different types of port intelligent equipment; the environment model construction module constructs a global environment model based on the task set and the dynamic model; the task allocation module is used to build a collaborative multi-agent task allocation mechanism, and use the task allocation mechanism to allocate and schedule tasks; the task execution optimization module constructs a collaborative reinforcement learning model based on the task set, dynamic model, global environment model and task allocation mechanism, and uses the collaborative reinforcement learning model to train the task execution strategy of the agent.

[0099] The embodiments described above are only descriptions of the preferred embodiments of the present invention and are not intended to limit the scope of the present invention. Without departing from the design spirit of the present invention, various modifications and improvements made to the technical solutions of the present invention by ordinary technicians in this field should all fall within the protection scope determined by the claims of the present invention.

Claims

1. A port multi-agent task allocation method based on reinforcement learning, characterized in that: The following steps are involved: Construct task sets for various operational tasks in the port multi-task environment; Construct a dynamic model of heterogeneous multi-agents for different types of port intelligent equipment; Based on the task set and the dynamic model, construct a global environment model; Constructing a collaborative multi-agent task allocation mechanism, and utilizing the task allocation mechanism to allocate and schedule tasks; Building a collaborative reinforcement learning model based on the task set, the dynamics model, the global environment model and the task allocation mechanism, and using the collaborative reinforcement learning model to train the task execution strategy of the intelligent agent; The methods for constructing a task collection include: A task graph is constructed using the dependencies and constraints of the tasks, wherein each node in the task graph represents a port operation to be performed, and each edge represents the dependencies and constraints of the tasks; Calculate task priorities based on the task graph: ， Among them, P(T i ) represents task T i The priority of i Represents task T i Resource requirements, d i Represents task T i , α1 and α2 represent weight coefficients; Construct the task set using the task priority , where T n represents the nth task, i and n represent natural numbers; Methods for constructing heterogeneous multi-agent dynamics models include: Defining an Agent Set , where I m represents the mth agent, where m represents a natural number; Define the state space set , where s m (t) represents agent I m The state at time t, t represents the time; Define the action space set , where a m (t) represents agent I m The action taken at time t; The dynamic model is constructed based on the state and the action , and use the dynamics model to update the agent state: ， Where f( ) represents the dynamic model function of the intelligent agent, S m (t+1) represents the state update of the agent; The method of allocating and scheduling tasks using the task allocation mechanism includes: Each agent independently calculates the cost function : ， Among them, β1, β2 and β3 represent weight coefficients, dist(T i , I m ) represents task T i With Agent I m The distance between i ) indicates the priority of the task, c delay (T i , I m ) represents the delay cost; Select the initial task bundle , calculate the task bundle based on the cost function Total cost: ； Agents exchange bundle information and check whether there is a task conflict. If there is a task conflict, the conflicting task will be assigned to the agent that can perform the conflicting task at the lowest cost. After that, the agent will update its own task bundle and recalculate the execution order and total cost of the tasks. Repeat the steps and iterate multiple times until the task allocation mechanism gradually converges and completes the task allocation and scheduling; The collaborative reinforcement learning model includes: a strategy network and a value network; The policy network is composed of a high-level policy network and a low-level policy network: ， ， in, represents a high-level policy network, which is used to select subtasks based on the task set and the global environment model, θ represents the parameters of the high-level policy network, h represents the high-level, represents the high-level value function, o represents the subtask, and s represents the current state; represents the low-level policy network, which is used to generate specific actions of the agent in each subtask content, ϕ represents the parameters of the low-level policy network, l represents the low-level, represents the low-level value function, a represents a specific action, and argmax represents the maximum value function Q; The value network is composed of a high-level value network and a low-level value network. The high-level value network is used to evaluate the effect of selecting subtasks, and the low-level value network is used to evaluate the action performance of the agent when performing specific tasks: ， ， Among them, μ represents the learning rate, r h Represents the high-level reward value, r l represents the low-level reward value, γ represents the discount factor, s' represents the next state, that is, the state reached by the agent after executing the action in the current state s, o' represents the subtask that may be selected in the next state s', and a' represents the specific action that may be selected in the next state s' and subtask o'; the method of using the collaborative reinforcement learning model to train the task execution strategy of the agent includes: in a multi-agent environment, the agent interacts with the global environment model, through centralized training, multiple agents share the experience replay pool, and collaboratively optimize through the strategy network and the value network to achieve the best coordination state; The collaborative optimization of strategy and value networks requires adjusting both the strategy network and the value network simultaneously to maximize the total value. This collaborative optimization mechanism enables the best coordination between strategy execution and value assessment in multiple dimensions of complex tasks: ， Among them, s t represents the state at time step t, o t represents the subtask selected at time step t, a t Represents the specific action performed at time step t; through reinforcement learning training, the intelligent agent can gradually learn the optimal task allocation strategy and execution strategy to improve the operating efficiency in the port multi-task environment.

2. According to claim 1, a port multi-agent task allocation method based on reinforcement learning is characterized in that: Methods for building a global environment model include: Based on the geographical location, cargo distribution and operation requirements of the port, the state space of the port global environment is constructed: ， Among them, h( ) represents the function of the interaction between the agent and the global environment, Represents Agent I m In the global environment, Loc m (t) represents agent I m At the geographical location at time t, Cargo m (t) represents agent I m The distribution of goods at time t, Req m (t) represents agent I m The job requirement at time t, ε represents the global environment variable.

3. According to the reinforcement learning-based port multi-agent task allocation method of claim 1, it is characterized in that: The method of using the collaborative reinforcement learning model to train the task execution strategy of an intelligent agent includes: in a multi-agent environment, the intelligent agent interacts with the global environment model, through centralized training, multiple intelligent agents share an experience replay pool, and collaboratively optimize through the strategy network and the value network to achieve the best coordination state.

4. A port multi-agent task allocation system based on reinforcement learning, wherein the allocation system applies the allocation method according to any one of claims 1 to 3, characterized in that: include: Task set building module, dynamic model building module, environment model building module, task allocation module and task execution optimization module; The task set building module is used to build a task set for various operation tasks in the port multi-task environment; The dynamic model building module is used to build a dynamic model of heterogeneous multi-agents for different types of port intelligent equipment; The environment model construction module constructs a global environment model based on the task set and the dynamic model; The task allocation module is used to construct a collaborative multi-agent task allocation mechanism, and utilize the task allocation mechanism to allocate and schedule tasks; The task execution optimization module constructs a collaborative reinforcement learning model based on the task set, the dynamics model, the global environment model and the task allocation mechanism, and uses the collaborative reinforcement learning model to train the task execution strategy of the intelligent agent.

Images