Multi-agent collaborative scheduling method based on reinforcement learning of spatiotemporal prediction model

Abstract

The application discloses a multi-agent collaborative scheduling method based on reinforcement learning of a space-time prediction model. The application first inputs a space-time state representation sequence into a space-time prediction model, updates the space-time prediction model according to historical space-time states and predicted space-time states, predicts future space-time state features, and then generates a space-time prediction intention. Finally, the space-time prediction intention is provided as explicit input information to the agent for auxiliary decision-making, thereby cultivating the collaboration ability among the agents. The application uses the space-time prediction model to predict the future action intention of each agent, provides the action intention of other agents for the agent, helps the agent to understand and predict the influence of other agents on the environment, reduces the mutual influence among the agents, improves the decision stability, improves the collaboration ability among the agents, and realizes efficient completion of collaborative tasks by multiple agents.

Description

Technical Field

[0001] This invention relates to the field of multi-agent reinforcement learning, specifically a multi-agent cooperative scheduling method based on spatiotemporal prediction models. Background Technology

[0002] Today, various types of robots, especially robot systems composed of multiple robots, play a very important role in various fields such as manufacturing, transportation, and daily life, showing great development potential in terms of efficiency, workload, and overall cost.

[0003] Multi-agent systems (MAS) are a key research area in AI, inspired by the collective behavior of agents in nature and learning through computer simulation and observation of cooperation among agents. Significant progress has been made in research on MAS based on deep reinforcement learning. Most MAS deep reinforcement learning methods that do not involve communication follow a centralized training and decentralized execution (CTDE) framework, in which each agent chooses actions based on its own observations. However, CTDE-based methods have a drawback: while an agent can share information with other agents during centralized training, during decentralized execution, it can only make decisions based on its own individual observations. In partially observable environments, due to the lack of shared signals and global state representation, agents struggle to determine the intentions of other agents outside their field of vision, resulting in a lack of willingness to cooperate and hindering task completion through collaboration.

[0004] To address the issue of low collaboration capabilities in multi-agent systems, researchers both domestically and internationally have employed communication-based multi-agent reinforcement learning techniques. The paper "Singh A, Jain T, Sukhbaatar S. Learning when to Communicate at Scale in Multiagent Cooperative and CompetitiveTasks. 2018 [2024-11-14]" proposes using a gating mechanism to determine whether to send messages to all agents and assigning individual rewards to each agent, encouraging them to exhibit more diverse behaviors in competitive / hybrid environments. The paper "Zhang K, Yang Z, Baar T. Decentralized multi-agent reinforcement learning with networked agents: recent advances [J]. 2021" constructs network communication. In decentralized execution, each agent makes individual decisions and receives local rewards without needing to infer the policies of other agents. In centralized learning, agents update consensus through the constructed network communication. The paper "Ding Z, Huang T, Lu Z. Learning Individually Inferred Communication for Multi-Agent Cooperation [J]. 2020" measures the stochastic impact of other agents' behaviors on its own policy learning and uses this to decide whether to communicate with other agents in a peer-to-peer manner. However, while the above method improves the cooperation ability between agents to some extent by utilizing communication, it also increases some requirements, such as the selection of communication partners, the construction of communication links, the type of communication messages, the selection of sending and integration methods, and the additional communication overhead caused by the above selections.

[0005] Multi-agent systems require mutual understanding of intent to coordinate and execute collaborative tasks. This is particularly important for multi-robot systems operating in complex environments with partial observability and limited communication bandwidth, where tasks such as object transport and search and rescue are performed by sharing the same physical space during interaction. The ability to coordinate through intent communication can reduce collisions between robots, thereby reducing maintenance costs caused by collision damage, and simultaneously improving task completion efficiency. Therefore, further in-depth research is urgently needed to develop methods for enabling agents to understand each other's intents and improve their collaborative capabilities. Summary of the Invention

[0006] To address the shortcomings of existing technologies, the technical problem this invention aims to solve is to provide a multi-agent cooperative scheduling method based on reinforcement learning using a spatiotemporal prediction model.

[0007] The technical solution of this invention to solve the aforementioned technical problem is to provide a multi-agent cooperative scheduling method based on reinforcement learning of a spatiotemporal prediction model, characterized in that the method includes the following steps:

[0008] Step 1: At the start of the round, initialize the environment E and the spatiotemporal state representation sequence TS(s), and obtain the initial state set of the multi-agent system.

[0009] Step 2: At each time step within this round, obtain the state set generated by the environment for each agent at that time step t.

[0010] Step 3: Convert the state set S obtained in Step 2 into a single state set. t The spatial information of each agent is encoded, and the encoded result is updated to the initialized global state feature to obtain the global state feature H(s) at time step t. t );

[0011] Step 4: Convert the global state features H(s) obtained in Step 3 into... t ) and historical global state characteristics H(s) 1 ),H(s 2 ),...,H(s t-1 The fusion process adds the global state features of each time step to the corresponding positions in the spatiotemporal state representation sequence TS(s), thereby updating the spatiotemporal state representation sequence TS(s) = {H(s)}. 1 ), H(s) 2 H(s) t )};

[0012] Step 5: Represent the spatiotemporal state sequence TS(s) = {H(s)} obtained in Step 4. 1 ), H(s) 2 H(s) t The input is fed into the spatiotemporal prediction model. Based on the historical spatiotemporal state and the predicted spatiotemporal state, the spatiotemporal prediction model is updated, and the future spatiotemporal state feature H(s) is predicted. t+1 );

[0013] Step 6: Generate Spatiotemporal Predictive Intent (SPM): First, construct a global intent graph M to store intent information. g Then iterate through all agents, based on the future spatiotemporal state features H(s) obtained in step 5. t+1 The actual pixel position And encoding parameters to update the global intention graph Mg Then, based on the position and angle information of each agent, the global intention graph M is processed. g By sequentially performing coordinate mapping and linear transformation operations, a local intention graph of a single agent is obtained. The obtained local intention maps of each agent are then stitched together to generate spatiotemporal prediction intentions.

[0014] Step 7: Apply the spatiotemporal prediction intent obtained in Step 6. Compared with the state set obtained in step 2

[0015] By concatenating the data, a new state representation is generated. Represent the new state Input into a multi-agent policy network middle;

[0016] Step 8: The multi-agent policy network uses the new state representation obtained in Step 7. Make decisions and select appropriate actions.

[0017] Step 9: Each agent performs its corresponding action. Interact with the environment to generate a set of exploration tracks And store it in the experience pool;

[0018] Step 10: Determine whether the multi-agent policy network needs to be updated at this time step. If so, proceed to step 11; otherwise, proceed to step 12.

[0019] Step 11: Randomly sample exploration trajectories from the experience pool, and then update the multi-agent policy network using the exploration trajectories.

[0020] Step 12: Determine if this time step is the end time step of this round. If so, proceed to step 13; otherwise, repeat steps 2 to 11.

[0021] Step 13: Determine if this round is the end round. If so, end the round and each agent learns the strategy for cooperating to complete the task, thus achieving cooperative scheduling. Otherwise, repeat steps 1 to 12.

[0022] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0023] (1) This invention uses a spatiotemporal prediction model to predict the future action intentions of each agent, providing the agent with the action intentions of other agents, which helps the agent understand and predict the impact of other agents on the environment, reduces the mutual influence between agents, improves decision stability, improves the collaboration ability between agents, and enables multiple agents to efficiently complete collaborative tasks.

[0024] (2) This invention constructs a spatiotemporal prediction model to generate a new intent representation suitable for multi-agent reinforcement learning: spatiotemporal prediction intent. When agents make decisions, the spatiotemporal prediction intents of other agents are used as additional state representation inputs to assist decision-making, enabling agents to choose efficient and conflict-free actions as much as possible, thereby improving the overall collaborative ability and efficiency of agents in completing tasks.

[0025] (3) In this invention, the state representation of the agent includes the spatiotemporal prediction intent generated by the spatiotemporal prediction model, which contains additional information that is beneficial to the agent in making efficient decisions.

[0026] (4) In this invention, the spatiotemporal prediction intent is encoded as an image aligned with the state map and action map on the pixel level, preserving the spatial structure. Therefore, compared with other encoding methods, this method enables fully convolutional networks to make more effective use of the information they contain, encouraging policy networks to learn emergency cooperative behaviors, from collision avoidance to spatial coordination to task specialization.

[0027] (5) Experiments were conducted in a variety of asynchronous heterogeneous (including robots with different functions such as pushing, lifting, or throwing) partially observable multi-agent environments. The results show that spatiotemporal prediction of intent can significantly enhance the collaborative ability among multi-agents and improve the task performance of heterogeneous robot teams. This invention aims to build a multi-agent reinforcement learning framework, which can be extended to most existing multi-agent reinforcement learning algorithms with minor modifications, achieving high scalability and transferability. Attached Figure Description

[0028] Figure 1 This is a schematic diagram of the overall framework of the present invention;

[0029] Figure 2 This is a flowchart of the spatial state feature encoding step 3 of the present invention;

[0030] Figure 3 This is a flowchart of the spatiotemporal prediction model for step 5 of the present invention;

[0031] Figure 4 This is a flowchart illustrating the spatiotemporal prediction intention of step 6 of the present invention. Detailed Implementation

[0032] Specific embodiments of the present invention are given below with reference to the accompanying drawings. These specific embodiments are only used to further illustrate the present invention in detail and do not limit the scope of protection of the present invention.

[0033] This invention provides a multi-agent cooperative scheduling method (hereinafter referred to as the method) based on reinforcement learning using a spatiotemporal prediction model, characterized by the following steps:

[0034] Step 1: At the start of the round, initialize the environment E and the spatiotemporal state representation sequence TS(s), and obtain the initial state set of the multi-agent system.

[0035] Preferably, in step 1, the multiple agents are robots. The overall method of this invention is applied to the collaborative scheduling of multiple robots to achieve collaborative scheduling and allocation of tasks.

[0036] Step 2: At each time step within this round, obtain the state set generated by the environment for each agent at that time step t. Each time step corresponds to a set of states;

[0037] Preferably, in step 2, the state set S t Each local state feature in All include four types of auxiliary graphs: (1) Local environment graph: encodes the state of agents and objects; (2) Agent graph: encodes the position of each agent in space and whether it carries an object; (3) Object shortest distance graph: the shortest distance from the agent to the object; (4) Target point shortest distance graph: the shortest distance from the agent to the target point.

[0038] Step 3: Convert the state set S obtained in Step 2 into a single state set. t The spatial information (including orientation and position information) of each agent is encoded, and the encoded results are updated to the initialized global state features to obtain the global state features H(s) at time step t. t This breaks the limitations of each intelligent agent's egocentric perspective, establishes global information that integrates the independent information of each intelligent agent, and provides an information foundation for spatiotemporal state representation and spatiotemporal prediction model modules;

[0039] Preferably, step 3 specifically includes:

[0040] Step 3.1: Initialize global state characteristics;

[0041] Step 3.2, from the state set Select the local state features of the i-th agent

[0042] Step 3.3: From local state features Extract the direction information ω of agent i, and then calculate the rotation angle θ of agent i based on the direction information ω and the preset direction representation in the global state features;

[0043] From local state features Extract the agent's position information, and then calculate the agent's local-global position vector based on the position information and the preset local observation range of the agent.

[0044] Preferably, in step 3.3, the rotation angle θ of the agent is as shown in equation (1);

[0045]

[0046] In equation (1), θ is the rotation angle. The direction angle is preset for the global state features, and ω is the local state features of the agent. The directional information in the data refers to the arc information of the agent.

[0047] Preferably, in step 3.3, the agent's local-global position vector As shown in equation (2);

[0048]

[0049] In equation (2), This is the agent's local-global position vector, i.e., the agent's position in the global state features; Local state features of the agent Location information in H; g W g These represent the width and height of the global map, respectively; O l This represents the local observation range of the intelligent agent.

[0050] Step 3.4: Analyze the rotation angle θ and local-global position vector obtained in Step 3.3. By concatenating the features, the local-global state features of the agent can be obtained.

[0051] Step 3.5: Combine local and global state features Update to global state characteristics;

[0052] Step 3.6: Repeat steps 3.2 to 3.5 (i.e., spatial state feature encoding, such as...) Figure 2 (As shown), until the local state features of all agents are updated to the global state features, the global state features at time step t are obtained.

[0053] Step 4: Convert the global state features H(s) obtained in Step 3 into... t ) and historical global state characteristics H(s) 1 ), H(s) 2 H(s) t-1 The fusion process adds the global state features of each time step to the corresponding positions in the spatiotemporal state representation sequence TS(s), thereby updating the spatiotemporal state representation sequence TS(s) = {H(s)}. 1),H(s 2 ),...,H(s t )};

[0054] Step 5: Represent the spatiotemporal state sequence TS(s) = {H(s)} obtained in Step 4. 1 ), H(s) 2 H(s) t The input is fed into the spatiotemporal prediction model. Based on the historical spatiotemporal state and the predicted spatiotemporal state, the spatiotemporal prediction model is updated, and the future spatiotemporal state feature H(s) is predicted. t+1 );

[0055] Preferably, in step 5, the process of the spatiotemporal prediction model is as follows (e.g.) Figure 3 As shown): First, initialize the future spatiotemporal state features H(s) t+1 Then, the local-global position vector matrix of the agent in the spatiotemporal state representation sequence TS(s) is extracted; then, the optimal prediction parameters for the current time step are obtained based on the local-global position vector of the current time step and the predicted position value of the previous time step; then, the optimal prediction parameters of the current time step and the local-global position vector are used to predict the predicted position value of the next time step; finally, the predicted position value is converted into the actual pixel position of the agent, and the future spatiotemporal state feature H(s) is updated. t+1 ).

[0056] Preferably, step 5 specifically includes:

[0057] Step 5.1: Initialize the future spatiotemporal state features H(s) t+1 );

[0058] Step 5.2: Select the spatiotemporal information of the i-th agent from the spatiotemporal state representation sequence TS(s). Extract the local-global position vector matrix.

[0059] Step 5.3, Update Phase: For the sake of formula simplicity, the local-global position vector is... Let it be denoted as position vector Based on the position vector at time step t With predicted location value Adjust the prediction parameters to obtain the optimal prediction parameters for time step t;

[0060] Preferably, the specific process of step 5.3 is shown in equation (3):

[0061]

[0062] In equation (3), It is the prediction error covariance matrix, Ft-1 Q is the Jacobian matrix of the state transition function at time step t-1. t-1 It is the process noise covariance matrix, H t R is the Jacobian matrix of the measurement function at the predicted location estimate. t It is the measurement noise covariance matrix, I is the identity matrix, (F t-1 ) T and (H) t ) T They are F t-1 and H t The transpose of .

[0063] Step 5.4, Prediction Stage: Use the optimal prediction parameters and location vector obtained in Step 5.3 for time step t. Obtain the predicted position value at time step t+1.

[0064] Preferably, the specific process of step 5.4 is shown in equation (4);

[0065]

[0066]

[0067] In equation (4), and These are the predicted position value and the prediction error covariance matrix at time t+1, respectively; the function f(·) represents the nonlinear state transition function, U t The control input is for time step t+1; matrix F t Q is the Jacobian matrix of the state transition function at time step t. t The process noise covariance matrix represents the uncertainty in the dynamics of the system.

[0068] Step 5.5: Obtain the predicted location value from Step 5.4. Transformed into the actual pixel position of the intelligent agent

[0069] Preferably, in step 5.5, the actual pixel position Represented as:

[0070]

[0071] In equation (5), the clip(·) operation means to... and Restricted to (0, H) g -1) and (0, W g Between -1), ensure that the pixel index is within the range of the image size.

[0072] Step 5.6: Set the actual pixel position of the agent. Store future spatiotemporal state characteristics;

[0073] Step 5.7: Repeat steps 5.2 to 5.6 until the actual pixel positions of all agents are stored in the future spatiotemporal state feature H(s). t+1 ).

[0074] Step 6: Generate Spatiotemporal Prediction Intent (SPM) (e.g.) Figure 4 As shown): First, a global intent graph M is constructed to store intent information. g Then iterate through all agents, based on the future spatiotemporal state features H(s) obtained in step 5. t+1 The actual pixel position The encoding parameter is used to update the global intention graph M. g Then, based on the position and angle information of each agent, the global intention graph M is processed. g By sequentially performing coordinate mapping (i.e., the transformation from physical location to pixel location) and linear transformation (i.e., cropping and rotation), a local intention graph of a single agent is obtained. The obtained local intention maps of each agent are then stitched together to generate spatiotemporal prediction intentions.

[0075] Preferably, the specific process of step 6 is as follows:

[0076] Step 6.1: Initialize the global intention graph M g Create a zero matrix of the same size as the simulation environment to store intent information;

[0077] Step 6.2: Traverse and fill all actual pixel positions of the agents. Skip agents that are in an idle state;

[0078] Step 6.3: Traverse the future spatiotemporal state features H(s) of all agents. t+1 Based on the encoding parameters provided by the environment, different encoding logic is executed to update the global intent graph;

[0079] Preferably, in step 6.3, when the value of the encoding parameter is circle, a value is placed at the target position of the agent, where 1 indicates the presence of an agent and 0 indicates the absence of an agent; when the value of the encoding parameter is binary, the future spatiotemporal state feature H(s) is... t+1 The pixels in the graph are set to the scaling value of the global intention map to generate a simple path representation; when the value of the encoding parameter is ramp, a linear interpolation method is used to generate an array of values ​​from 1 to 0, representing the future spatiotemporal state features H(s). t+1The intensity of the midpoint is calculated using a maximum value function to ensure that the value on the path is not lower than the existing value, and the length of the current path segment is added to the total path length for subsequent linear interpolation calculations, providing a path representation with gradually varying intensity.

[0080] Step 6.4: Use the dilation method to thicken the line segments on the map. After all agents have traversed the map, a global intention graph Mg is generated.

[0081] Step 6.5: Select the i-th agent and analyze the global state features H(s). t Obtain its position and angle at the current time step, i.e., the local-global position vector in step 3.3. and rotation angle θ, and use formula (6) to Convert to actual pixel position

[0082]

[0083] Step 6.6: Based on the rotation angle θ and actual pixel position obtained in Step 6.5... For the global intention graph M generated in step 6.4 g Perform pruning: at the current time step t, prune a local intention graph centered on the i-th agent that has the same local observation range as the agent.

[0084] Preferably, in step 6.6, the partial intention map Represented as:

[0085]

[0086] In equation (7), the function This means rounding x up to the nearest even number.

[0087] Step 6.7: Adjust the local intention graph based on the rotation angle θ of the agent at the current time step. Rotate the image to obtain a local intention map that matches the agent's angle.

[0088] Preferably, in step 6.7, for the local intention map For any point (x, y) in the matrix, the transformed point after rotation is represented as (x′, y′), as shown in equation (8):

[0089]

[0090] Step 6.8: Repeat steps 6.5 to 6.7 until all agents have generated local intention maps; then, concatenate the local intention maps of each agent to generate spatiotemporal prediction intentions.

[0091] Step 7: Apply the spatiotemporal prediction intent obtained in Step 6. Compared with the state set obtained in step 2

[0092] By concatenating the data, a new state representation is generated. Represent the new state Input into a multi-agent policy network middle;

[0093] Step 8: The multi-agent policy network uses the new state representation obtained in Step 7. Make decisions and select appropriate actions.

[0094] Step 9: Each agent performs its corresponding action. Interact with the environment to generate a set of exploration tracks

[0095] And store it in the experience pool;

[0096] Step 10: Determine whether the multi-agent policy network needs to be updated at this time step. If so, proceed to step 11; otherwise, proceed to step 12.

[0097] Step 11: Randomly sample exploration trajectories from the experience pool, and then update the multi-agent policy network using the exploration trajectories.

[0098] Step 12: Determine if this time step is the end time step of this round. If so, proceed to step 13; otherwise, repeat steps 2 to 11.

[0099] Step 13: Determine if this round is the end round. If so, end the round and each agent learns the strategy for cooperating to complete the task, thus achieving cooperative scheduling. Otherwise, repeat steps 1 to 12.

[0100] Any aspects not covered in this invention are applicable to existing technologies.

Claims

1. A multi-agent cooperative scheduling method based on spatiotemporal prediction model reinforcement learning, characterized in that, The method includes the following steps: Step 1: At the start of the round, initialize the environment E and the spatiotemporal state representation sequence TS(s), and obtain the initial state set of the multi-agent system. Step 2: At each time step within this round, obtain the state set generated by the environment for each agent at that time step t. Step 3: Convert the state set S obtained in Step 2 into a single state set. t The spatial information of each agent is encoded, and the encoded result is updated to the initialized global state feature to obtain the global state feature H(s) at time step t. t ); Step 4: Convert the global state features H(s) obtained in Step 3 into... t ) and historical global state characteristics H(s) 1 ),H(s 2 ),...,H(s t -1 The fusion process adds the global state features of each time step to the corresponding positions in the spatiotemporal state representation sequence TS(s), thereby updating the spatiotemporal state representation sequence TS(s) = {H(s)}. 1 ), H(s) 2 H(s) t )}; Step 5: Represent the spatiotemporal state sequence TS(s) = {H(s)} obtained in Step 4. 1 ), H(s) 2 H(s) t The input is fed into the spatiotemporal prediction model. Based on the historical spatiotemporal state and the predicted spatiotemporal state, the spatiotemporal prediction model is updated, and the future spatiotemporal state feature H(s) is predicted. t+1 ); Step 6: Generate Spatiotemporal Predictive Intent (SPM): First, construct a global intent graph M to store intent information. g Then iterate through all agents, based on the future spatiotemporal state features H(s) obtained in step 5. t+1 The actual pixel position And encoding parameters to update the global intention graph M g Then, based on the position and angle information of each agent, the global intention graph M is processed. g By sequentially performing coordinate mapping and linear transformation operations, a local intention graph of a single agent is obtained. The obtained local intention maps of each agent are then stitched together to generate spatiotemporal prediction intentions. Step 7: Apply the spatiotemporal prediction intent obtained in Step 6 Compared with the state set obtained in step 2 By concatenating the data, a new state representation is generated. Represent the new state Input into a multi-agent policy network middle; Step 8: The multi-agent policy network uses the new state representation obtained in Step 7. Make decisions and select appropriate actions. Step 9: Each agent performs its corresponding action. Interact with the environment to generate a set of exploration tracks And store it in the experience pool; Step 10: Determine whether the multi-agent policy network needs to be updated at this time step. If so, proceed to step 11; otherwise, proceed to step 12. Step 11: Randomly sample exploration trajectories from the experience pool, and then update the multi-agent policy network using the exploration trajectories. Step 12: Determine if this time step is the end time step of this round. If so, proceed to step 13; otherwise, repeat steps 2 to 11. Step 13: Determine if this round is the end round. If so, end the round and each agent learns the strategy for cooperating to complete the task, thus achieving cooperative scheduling. Otherwise, repeat steps 1 to 12.

2. The multi-agent cooperative scheduling method based on spatiotemporal prediction model reinforcement learning according to claim 1, characterized in that, In step 1, the multi-agent system uses robots.

3. The multi-agent cooperative scheduling method based on spatiotemporal prediction model reinforcement learning according to claim 1, characterized in that, In step 2, the state set S t Each local state feature in All include four types of auxiliary graphs: (1) Local environment graph: encodes the state of agents and objects; (2) Agent graph: encodes the position of each agent in space and whether it carries an object; (3) Object shortest distance graph: the shortest distance from the agent to the object; (4) Target point shortest distance graph: the shortest distance from the agent to the target point.

4. The multi-agent cooperative scheduling method based on spatiotemporal prediction model reinforcement learning according to claim 1, characterized in that, Step 3 specifically involves: Step 3.1: Initialize global state characteristics; Step 3.2, from the state set Select the local state features of the i-th agent Step 3.3: From local state features Extract the direction information ω of agent i, and then calculate the rotation angle θ of agent i based on the direction information ω and the preset direction representation in the global state features; From local state features Extract the agent's position information, and then calculate the agent's local-global position vector based on the position information and the preset local observation range of the agent. Step 3.4: Analyze the rotation angle θ and local-global position vector obtained in Step 3.

3. By concatenating the features, the local-global state features of the agent can be obtained. Step 3.5: Combine local and global state features Update to global state characteristics; Step 3.6: Repeat steps 3.2 to 3.5 until all agents' local state features are updated to global state features, thus obtaining the global state features at time step t.

5. The multi-agent cooperative scheduling method based on spatiotemporal prediction model reinforcement learning according to claim 4, characterized in that, In step 3.3, the rotation angle θ of the agent is as shown in equation (1); In equation (1), θ is the rotation angle. The direction angle is preset for the global state features, and ω is the local state features of the agent. The directional information in the data, i.e., the arc information of the agent; In step 3.3, the agent's local-global position vector As shown in equation (2); In equation (2), This is the agent's local-global position vector, i.e., the agent's position in the global state features; Local state features of the agent Location information in H; g W g These represent the width and height of the global map, respectively; O l This represents the local observation range of the intelligent agent.

6. The multi-agent cooperative scheduling method based on spatiotemporal prediction model reinforcement learning according to claim 1, characterized in that, In step 5, the spatiotemporal prediction model process is as follows: First, initialize the future spatiotemporal state features H(s). t+1 Then, the local-global position vector matrix of the agent in the spatiotemporal state representation sequence TS(s) is extracted; then, the optimal prediction parameters for the current time step are obtained based on the local-global position vector of the current time step and the predicted position value of the previous time step; then, the optimal prediction parameters of the current time step and the local-global position vector are used to predict the predicted position value of the next time step; finally, the predicted position value is converted into the actual pixel position of the agent, and the future spatiotemporal state feature H(s) is updated. t+1 ).

7. The multi-agent cooperative scheduling method based on spatiotemporal prediction model reinforcement learning according to claim 1 or 6, characterized in that, Step 5 specifically involves: Step 5.1: Initialize the future spatiotemporal state features H(s) t+1 ); Step 5.2: Select the spatiotemporal information of the i-th agent from the spatiotemporal state representation sequence TS(s). Extract the local-global position vector matrix. Step 5.3, Update Phase: Convert the local-to-global position vector Let it be denoted as position vector Based on the position vector at time step t With predicted location value Adjust the prediction parameters to obtain the optimal prediction parameters for time step t; Step 5.4, Prediction Stage: Use the optimal prediction parameters and location vector obtained in Step 5.3 for time step t. Obtain the predicted position value at time step t+1. Step 5.5: Obtain the predicted location value from Step 5.

4. Transformed into the actual pixel position of the intelligent agent Step 5.6: Set the actual pixel position of the agent. Store future spatiotemporal state characteristics; Step 5.7: Repeat steps 5.2 to 5.6 until the actual pixel positions of all agents are stored in the future spatiotemporal state feature H(s). t+1 ).

8. The multi-agent cooperative scheduling method based on spatiotemporal prediction model reinforcement learning according to claim 7, characterized in that, The specific process of step 5.3 is shown in equation (3): In equation (3), It is the prediction error covariance matrix, F t-1 Q is the Jacobian matrix of the state transition function at time step t-1. t-1 It is the process noise covariance matrix, H t R is the Jacobian matrix of the measurement function at the predicted location estimate. t It is the measurement noise covariance matrix, I is the identity matrix, (F t-1 ) T and (H) t ) T They are F t-1 and H t transpose; The specific process of step 5.4 is shown in equation (4); In equation (4), and These are the predicted position value and the prediction error covariance matrix at time t+1, respectively; the function f(·) represents the nonlinear state transition function, U t The control input is for time step t+1; matrix F t Q is the Jacobian matrix of the state transition function at time step t. t The process noise covariance matrix represents the uncertainty in the system dynamics. In step 5.5, the actual pixel position Represented as: In equation (5), the clip(·) operation means to... and Restricted to (0, H) g -1) and (0, W g Between -1), ensure that the pixel index is within the range of the image size.

9. The multi-agent cooperative scheduling method based on spatiotemporal prediction model reinforcement learning according to claim 1, characterized in that, The specific process for step 6 is as follows: Step 6.1: Initialize the global intention graph M g Create a zero matrix of the same size as the simulation environment to store intent information; Step 6.2: Traverse and fill all actual pixel positions of the agents. Skip agents that are in an idle state; Step 6.3: Traverse the future spatiotemporal state features H(s) of all agents. t+1 Based on the encoding parameters provided by the environment, different encoding logic is executed to update the global intent graph; Step 6.4: Use the dilation method to thicken the line segments on the map. After all agents have traversed the map, a global intention graph M is generated. g ; Step 6.5: Select the i-th agent and analyze the global state features H(s). t Obtain its position and angle at the current time step, i.e., the local-global position vector in step 3.

3. and rotation angle θ, and use formula (6) to Convert to actual pixel position Step 6.6: Based on the rotation angle θ and actual pixel position obtained in Step 6.5... For the global intention graph M generated in step 6.4 g Perform pruning: at the current time step t, prune a local intention graph centered on the i-th agent that has the same local observation range as the agent. Step 6.7: Adjust the local intention graph based on the rotation angle θ of the agent at the current time step. Rotate the image to obtain a local intention map that matches the agent's angle. Step 6.8: Repeat steps 6.5 to 6.7 until all agents have generated local intention maps; then, concatenate the local intention maps of each agent to generate spatiotemporal prediction intentions.

10. The multi-agent cooperative scheduling method based on spatiotemporal prediction model reinforcement learning according to claim 9, characterized in that, In step 6.3, when the value of the encoding parameter is circle, a value is placed at the target position of the agent, where 1 indicates the presence of an agent and 0 indicates the absence of an agent; when the value of the encoding parameter is binary, the future spatiotemporal state feature H(s) is... t+1 The pixels in the graph are set to the scaling value of the global intention map to generate a simple path representation; when the value of the encoding parameter is ramp, a linear interpolation method is used to generate an array of values ​​from 1 to 0, representing the future spatiotemporal state features H(s). t+1 The intensity of the midpoint is calculated using a maximum value function to ensure that the value on the path is not lower than the existing value, and the length of the current path segment is added to the total path length for subsequent linear interpolation calculations, providing a path representation with gradually changing intensity. In step 6.6, the partial intention map Represented as: In equation (7), the function This means rounding x up to the nearest even number; In step 6.7, for the local intention map For any point (x, y) in the matrix, the transformed point after rotation is represented as (x′, y′), as shown in equation (8):

Images