Multi-agent multi-task collaborative reinforcement learning method based on space-time fusion architecture

Abstract

The application belongs to the technical field of deep reinforcement learning, and discloses a multi-agent multi-task cooperative reinforcement learning method based on a space-time fusion architecture, which comprises the following steps: step 1, initializing a task sampling probability, forming an entity embedding vector sequence and a task embedding vector; step 2, inputting the entity embedding vector sequence and the task embedding vector into a noise-resistant feature extraction layer to obtain deep time sequence features; step 3, generating a dynamic weight matrix in real time to obtain local Q values output by an agent; step 4, a Transformer hybrid network receiving a global state vector of an environment, a task embedding vector and a local Q sequence to give a global action value; step 5, calculating a total loss; and step 6, based on the total loss, training an optimization unit to update all network parameters, and simultaneously, judging whether a preset evaluation round is reached. The application has stronger anti-interference ability and more stable control performance, and realizes adaptive control of multiple heterogeneous tasks.

Description

Technical Field

[0001] This application belongs to the field of deep reinforcement learning technology, specifically relating to a multi-agent, multi-task collaborative reinforcement learning method based on a spatiotemporal fusion architecture. Background Technology

[0002] With the development of artificial intelligence technology, multi-agent systems are increasingly being used in fields such as drone formation performances, automated warehousing and logistics, battlefield situational awareness, and complex video games. In practical applications, intelligent agent swarms often need to possess the ability to perform multiple tasks simultaneously through a single control model. For example, in the same scenario, they can perform both dispersed reconnaissance and target encirclement or formation maintenance.

[0003] However, existing multi-task, multi-agent reinforcement learning methods have significant limitations when facing high-fidelity, complex physical environments. First, at the perception level, mainstream algorithms typically employ the soft-attention mechanism Softmax, which tends to assign some attention to all objects within the field of view. When the environment contains numerous false targets, decoys, or irrelevant obstacles, the agent's computational resources are scattered by noise, leading to inefficient decision-making. Second, at the control level, real-world environments often possess physical inertia, such as gliding on ice or aircraft turning; relying solely on current position observations makes it difficult to infer velocity vectors, resulting in control oscillations. Moreover, at the multi-task level, the policy logic for different tasks may be diametrically opposed, such as approaching versus moving away. Traditional shared output layer architectures are prone to conflicts in updating neural network parameters, leading to catastrophic forgetting or negative transfer. While some techniques attempt to introduce large-scale pre-trained language models to assist task understanding, this incurs enormous computational overhead, making it unsuitable for edge deployment.

[0004] Therefore, designing a lightweight, noise-resistant decision-making method with temporal reasoning capabilities and the ability to deeply decouple multi-task strategies is a key problem that urgently needs to be solved in the field of multi-agent technology. Summary of the Invention

[0005] To address the problems existing in the prior art, this application provides a multi-agent, multi-task collaborative reinforcement learning method based on a spatiotemporal fusion architecture. This method constructs an end-to-end deep reinforcement learning control architecture specifically designed to solve complex collaborative control problems involving high-inertia physical environments, strong perceptual noise interference, and the coexistence of heterogeneous multi-tasks. This application also designs a complete process including environmental interaction, feature extraction, action generation, value evaluation, and training optimization to achieve adaptive control of multiple heterogeneous tasks.

[0006] To achieve the above objectives, this application employs the following technical solution:

[0007] This application discloses a multi-agent, multi-task collaborative reinforcement learning method based on a spatiotemporal fusion architecture. The method is implemented through a multi-agent, multi-task collaborative reinforcement learning system, which includes agent-side components and a central training module. Each agent at the agent-side component includes a spatiotemporal fusion architecture and a hypernetwork action generation module architecture. The hypernetwork action generation module architecture includes a hypernetwork, a statically learnable bias vector, and an action value calculation unit. The central training module includes a Transformer hybrid network, a dual experience replay pool, and a training optimization unit. Specifically, the multi-agent, multi-task collaborative reinforcement learning method includes the following steps:

[0008] Step 1: Initialize the agent terminal and the central training module, and initialize the task sampling probability. The environment generates the current task to be executed according to the task sampling probability, and synchronously provides the agent's original local multi-source heterogeneous observation data and global state vector to the multi-agent multi-task collaborative reinforcement learning system. Subsequently, the agent terminal performs vectorization encoding on the collected local multi-source heterogeneous observation data, and performs feature embedding through the entity encoder and task encoder of the spatiotemporal fusion architecture to form an entity embedding vector sequence and a task embedding vector.

[0009] Step 2: Construct a noise-resistant feature extraction layer based on sparse attention. Input the entity embedding vector sequence and task embedding vector obtained in Step 1 into the noise-resistant feature extraction layer to obtain deep temporal features after eliminating noise interference.

[0010] Step 3: The hypernetwork receives the task embedding vector generated in Step 1. As input, the dynamic weight matrix required by the action value calculation unit is generated in real time. The action value calculation unit receives the deep temporal features and dynamic weight matrix obtained in step 2. As input, combined with the static learnable bias vector calculation, the local output of the agent is obtained. Value, the agent utilizes the local Values ​​are selected and actions are executed, interactions with the environment generate trajectory sequences, and the trajectories are stored in the ordinary experience replay pool and the premium experience replay pool respectively according to the reward level.

[0011] Step 4: The Transformer hybrid network receives the global state vector of the environment and the task embedding vector generated in Step 1. and the local output of all agents The sequence of values ​​is evaluated using a Transformer hybrid network to provide the global action value. ;

[0012] Step 5: The training optimization unit samples from both the ordinary experience replay pool and the high-quality experience replay pool, based on the real environment reward and the global action value obtained in Step 4. The temporal difference loss and self-imitation loss are calculated separately, and then a dual optimization loss function is constructed and the total loss is calculated.

[0013] Step 6: Based on the total loss calculated in Step 5, the training optimization unit updates the full network parameters in the multi-agent multi-task collaborative reinforcement learning system. At the same time, it determines whether the preset evaluation round has been reached. If it has, it dynamically calculates the sampling probability of each task in the new round of training based on the average reward of each task in the testing phase. The multi-agent multi-task collaborative reinforcement learning system returns to Step 1 based on the sampling probability to start the next round of task sampling and data collection until the training termination condition is met.

[0014] A further improvement in this application is that step 1 specifically involves:

[0015] Step 1.1: Initialize the full network parameters of the multi-agent multi-task collaborative reinforcement learning system and initialize the sampling probability of each task. The environment samples from the preset task set according to the current task sampling probability to generate the instruction code of the task to be executed at the current moment, and resets the environment state according to the task to be executed.

[0016] Step 1.2: For any agent in the multi-agent, multi-task cooperative reinforcement learning system At the current time step The intelligent agent acquires local multi-source heterogeneous observation data, while the multi-agent multi-task collaborative reinforcement learning system acquires the global state vector of the environment. The local multi-source heterogeneous observation data includes a set of entity features and the code of the task instruction to be executed.

[0017] Step 1.3: Organize the entity features from all observed entity feature sets into a local observation sequence. Then, use an entity encoder to map the local observation sequence element-by-element, converting the physical vector of each entity feature into a high-dimensional entity embedding vector, forming an entity embedding vector sequence. For the task instruction encoding to be executed, the multi-agent, multi-task collaborative reinforcement learning system uses a spatiotemporal fusion architecture task encoder to map the task instruction encoding to be executed into continuous task embedding vectors. The mapping process is represented as follows:

[0018]

[0019]

[0020] in, Represents intelligent agents At time step Local observation sequence, This indicates the instruction code for the task to be executed. The nonlinear mapping function representing the entity encoder. This represents the mapping function of the task encoder. and These represent the output entity embedding vector sequence and task embedding vector, respectively.

[0021] A further improvement of this application is that: the spatiotemporal fusion architecture includes an entity encoder, a task encoder, and a noise-resistant feature extraction layer. The noise-resistant feature extraction layer includes an entity self-attention processing layer, a task cross-attention processing layer, and a cooperative self-attention processing layer. The internal structures of the entity self-attention processing layer, the task cross-attention processing layer, and the cooperative self-attention processing layer are the same, and each of the entity self-attention processing layer, the task cross-attention processing layer, and the cooperative self-attention processing layer includes a layer normalization unit, a sparse attention calculation unit, and a gated cyclic residual unit. The gated cyclic residual unit in the entity self-attention processing layer is a first GRU gated unit, the gated cyclic residual unit in the task cross-attention processing layer is a second GRU gated unit, and the gated cyclic residual unit in the cooperative self-attention processing layer is a third GRU gated unit.

[0022] The layer normalization unit is responsible for receiving the feature vectors of the entity self-attention processing layer, the task cross-attention processing layer, or the cooperative self-attention processing layer, which are input to the layer normalization unit. It also performs normalization processing on the input feature vectors of the entity self-attention processing layer, the task cross-attention processing layer, or the cooperative self-attention processing layer to unify the data distribution. The sparse attention computing unit receives the normalized features, calculates the interaction weights between entities through the Sparsemax function, extracts the clean spatial interaction features of the current time step, and passes them as new observation information to the next level.

[0023] The gated cyclic residual unit includes an update gate component, a reset gate component, and a state fusion component. The update gate component is responsible for calculating the update gate control signal to determine how much historical hidden state information to retain and how much new feature to write. The reset gate component is responsible for calculating the reset gate control signal to determine how much historical information to forget when calculating candidate states. The state fusion component is responsible for performing nonlinear transformations and the final weighted combination. The state fusion component first uses the reset gate to filter historical hidden states, calculates candidate hidden states by combining the current attention features, and then uses the update gate to perform weighted interpolation between the original historical hidden states and the newly calculated candidate hidden states, finally outputting a deep temporal feature vector that integrates long-term momentum dependence and current spatial awareness.

[0024] A further improvement of this application is that step 2 specifically includes the following steps:

[0025] Step 2.1: Input the entity embedding vector sequence obtained in Step 1 into the entity self-attention processing layer of the noise-resistant feature extraction layer, and then pass it through the first GRU gate unit to capture the spatial topological relationship and relative distance features between entities, and output the first intermediate feature. ;

[0026] Step 2.2: The first intermediate feature output in Step 2.1 The input noise-resistant feature extraction layer is processed by a task cross-attention layer. In the task cross-attention layer, the task embedding vector generated in step 1 is used as the key and value. The first intermediate feature is output in step 2.1. As a query, it then passes through the second GRU gating unit to obtain the second intermediate feature. Second intermediate feature It incorporates task context information;

[0027] Step 2.3, Second Intermediate Feature The collaborative self-attention processing layer, which enters the noise-resistant feature extraction layer, is used to enhance the communication features and collaborative representations within the multi-agent body. Finally, it passes through the third GRU gating unit to output deep temporal features.

[0028] A further improvement in this application is that, in step 2, the Sparsemax function is used to find the probability distribution with the closest Euclidean distance during the calculation of attention weights in the entity self-attention processing layer, the task cross-attention processing layer, and the collaborative self-attention processing layer.

[0029]

[0030] in, express Simplex For the original attention score, For attention weights.

[0031] A further improvement to this application is that step 3 specifically includes the following steps:

[0032] Step 3.1: Generate the dynamic weight matrix required by the action value calculation unit in real time. :

[0033]

[0034] in, The mapping function representing the hypernetic network;

[0035] Value assessment of each action by the agent The value is:

[0036]

[0037] in, For deep temporal features, Embed vectors for tasks. Represents a statically learnable bias vector;

[0038] Step 3.2: The agent's value assessment based on the result of step 3.1. The sequence of values ​​is used to determine the action to be executed using a preset action selection strategy. This determined action is then used to interact with the environment in real time, generating sequence data, defined as a trajectory. The multi-agent, multi-task collaborative reinforcement learning system establishes a dual experience storage mechanism: the collected trajectories are stored in a general experience replay pool, while simultaneously, the system monitors the cumulative rewards of each trajectory within the current batch in real time. The cumulative return is:

[0039]

[0040] in, The maximum time step for a single round. For time steps If the cumulative reward value of a certain trajectory is higher than the average reward level of the current batch, the trajectory is judged as a high-quality trajectory and stored in the high-quality experience replay pool.

[0041] A further improvement of this application is that: in step 4, the Transformer hybrid network includes a super network branch and a hybrid computation layer. The super network branch is used to generate hybrid parameters. The super network branch contains a backbone network and a parameter generation mapping layer. The backbone network has the same architecture as the noise-resistant feature extraction layer at the agent end, that is, it sequentially includes an entity self-attention processing layer, a task cross-attention processing layer and a cooperative self-attention processing layer.

[0042] A further improvement of this application is that step 4 specifically includes the following steps:

[0043] Step 4.1: Convert the global state vector of the environment The input is fed into a Transformer hybrid network to obtain deep global features. Specifically, this involves: converting the global state vector of the environment... As input, it is mapped into a high-dimensional state feature sequence through a state processing layer, and then fed into the backbone network of the hypernetwork branch for processing. Specifically, when passing through the task cross-attention processing layer, the task embedding vector generated in step 1 is introduced. The task information is fused; after passing through an entity self-attention processing layer, a task cross-attention processing layer, a cooperative self-attention processing layer, and gating processing, deep global features are obtained. ;

[0044] Step 4.2: The Transformer hybrid network includes a parameter generation mapping layer, which generates the deep global features obtained in Step 4.1. The input parameters are fed into a mapping layer, which, through linear transformation and dimensional reshaping, generates a mapping for blending local data. The value of the parameter, blending local The parameters of the value include the nonnegative mixture weight matrix. Non-negative mixed weight matrix and the first layer mixed bias vector and the second layer of mixed bias scalar ;

[0045] Step 4.3: Local outputs of all agents The sequence of values ​​is input into the hybrid computation layer, using the nonnegative hybrid weight matrix generated in step 4.2. Non-negative mixed weight matrix and the first layer mixed bias vector and the second layer of mixed bias scalar The global action value is obtained through nonlinear weighted combination calculation. .

[0046] A further improvement of this application is that step 5 specifically includes the following steps:

[0047] Step 5.1: Before calculating the temporal difference loss, the training optimization unit first uses the target network to calculate the expected target value of the current state-action pair. :

[0048]

[0049] Multi-agent, multi-task cooperative reinforcement learning systems update parameters by minimizing the mean squared error between the predicted global value and the target value.

[0050]

[0051] in, The size of the trajectory batch sampled from the ordinary experience playback pool. As an immediate reward signal for environmental feedback, This is a discount factor used to balance current rewards with future long-term rewards. This refers to the global state at the next moment after the action is performed. For the next joint action, The parameters representing the target network, Indicates the current state of the network. Joint actions and task instructions The value of predicting global actions.

[0052] Step 5.2, Total Loss Function for:

[0053]

[0054] in, The self-imitation loss weighting coefficient is used. For time-series difference loss based on Bellman equations, To address the self-imitation loss for high-return trajectories in a high-quality experience replay pool:

[0055]

[0056] in, This indicates the batch size of the trajectory sampled from the high-quality experience replay pool. This represents the set of samples drawn from the high-quality experience replay pool. For the first The cumulative reward of a trajectory For predictive value.

[0057] A further improvement in this application is that step 6 specifically comprises:

[0058] Step 6.1: Total loss calculated in Step 5 Update the full network parameters in the multi-agent multi-task collaborative reinforcement learning system;

[0059] Step 6.2: Determine if the preset evaluation rounds have been reached. If so, obtain the task. The current normalized test return is The maximum possible return is The level of mastery of this task is then indicated. Defined as:

[0060]

[0061] In the task sampling phase of the next training round, the multi-agent multi-task collaborative reinforcement learning system calculates the sampling probability of each task based on the degree of unfamiliarity with each task. :

[0062]

[0063] in, For the total number of tasks, The minimum sampling probability, Calculate the sum of attention metrics for all tasks in the task set. For traversal index, This is the temperature coefficient.

[0064] The beneficial effects of this application are:

[0065] This application constructs a hard attention filtering channel by introducing a Sparsemax activation mechanism into the feature extraction layer. This mechanism can automatically truncate the attention weights of false landmarks or distractors to absolute zero based on the criticality of environmental entities, thereby blocking the propagation of erroneous perception information at the source and ensuring that the agent accurately focuses on and processes the features of real targets in complex and distracting environments.

[0066] This application utilizes a mechanism in which hypernetworks dynamically generate action decision layer parameters based on task embedding vectors, thereby decoupling general perceptual features from specific task logic. This design enables the model to reconstruct the weight structure of the action network in real time based on the context information of the current task, thus flexibly adapting to and being compatible with various heterogeneous tasks with vastly different logics within a unified parameter framework, achieving seamless switching and collaborative execution in multi-task scenarios.

[0067] This application constructs a gating architecture that integrates Transformer global spatial awareness and GRU temporal memory, effectively balancing the extraction of instantaneous spatial features with the preservation of long-term temporal motion states. This mechanism enables the agent to fully utilize historical state information to smoothly predict motion trajectories in low-damping, high-inertia nonlinear physical environments, thereby generating stable and precise control commands and suppressing control oscillations caused by dynamic environmental changes.

[0068] This application adopts a training strategy that combines self-imitation learning with dynamic course sampling. On the one hand, it strengthens the memory and utilization of accidental success experiences, and on the other hand, it avoids overfitting on simple tasks and underfitting on difficult tasks by dynamically adjusting the sampling probability, thereby achieving high efficiency and convergence of model training in a multi-task environment. Attached Figure Description

[0069] Figure 1 This is a flowchart of the multi-agent, multi-task collaborative reinforcement learning method proposed in this application.

[0070] Figure 2 This is a schematic diagram of the Transformer hybrid network structure in this application.

[0071] Figure 3 This is a schematic diagram of the architecture of the hypernetic action generation module in this application.

[0072] Figure 4This is a schematic diagram of the training and optimization unit in this application.

[0073] Figure 5 This is a comparative diagram of this application and the current state-of-the-art multi-agent multi-task reinforcement learning benchmark algorithm TETQmix.

[0074] Figure 6 This is a schematic diagram of the spatiotemporal fusion architecture of this application. Detailed Implementation

[0075] The embodiments of this application will be disclosed below with reference to the drawings. For clarity, many practical details will be described in the following description. However, it should be understood that these practical details should not be used to limit this application. That is, in some embodiments of this application, these practical details are not essential.

[0076] This application presents a multi-agent, multi-task collaborative reinforcement learning method based on a spatiotemporal fusion architecture. The multi-agent, multi-task collaborative reinforcement learning method is implemented through a multi-agent, multi-task collaborative reinforcement learning system, which includes agent-side components and a central training module. Each agent on the agent-side component includes a spatiotemporal fusion architecture and a hypernetwork action generation module architecture. The central training module includes a Transformer hybrid network, a dual experience replay pool, and a training optimization unit.

[0077] like Figure 3 As shown, the hypernetic action generation module architecture includes a statically learnable bias vector. Static learnable bias vector It does not change with task embedding, but serves as a basic parameter shared by all tasks. The hypernetwork action generation module architecture includes a hypernetwork that receives the task embedding vector generated in step 1. As input, the dynamic weight matrix required by the decision-making layer is generated in real time. This module also includes a statically learnable bias vector. The statically learnable bias vector's value is optimized through gradients during training, but it does not change with task embedding; instead, it serves as a shared fundamental parameter across all tasks. Furthermore, this module includes an action value calculation unit that receives the deep temporal features and dynamic weight matrix obtained in step 2 as input, performs calculations, and generates local values. value.

[0078] like Figure 6As shown, the spatiotemporal fusion architecture includes an entity encoder, a task encoder, and a noise-resistant feature extraction layer. The noise-resistant feature extraction layer includes an entity self-attention processing layer, a task cross-attention processing layer, and a collaborative self-attention processing layer. The internal structures of the entity self-attention processing layer, the task cross-attention processing layer, and the collaborative self-attention processing layer are the same, and each of the entity self-attention processing layer, the task cross-attention processing layer, and the collaborative self-attention processing layer includes a layer normalization unit, a sparse attention calculation unit, and a core gated cyclic residual unit. The gated cyclic residual unit in the entity self-attention processing layer is a first GRU gated unit, the gated cyclic residual unit in the task cross-attention processing layer is a second GRU gated unit, and the gated cyclic residual unit in the collaborative self-attention processing layer is a third GRU gated unit.

[0079] The layer normalization unit is responsible for receiving the feature vectors of the entity self-attention processing layer, the task cross-attention processing layer, or the cooperative self-attention processing layer, which are input to the layer normalization unit. It then normalizes the input feature vectors of the entity self-attention processing layer, the task cross-attention processing layer, or the cooperative self-attention processing layer to unify the data distribution and ensure the stability of the gradient in subsequent calculations. Next, the sparse attention calculation unit receives the normalized features, calculates the interaction weights between entities using the Sparsemax function, extracts the clean spatial interaction features of the current time step, and passes them as new observation information to the next level.

[0080] The gated cyclic residual unit includes an update gate component, a reset gate component, and a state fusion component. The update gate component is responsible for calculating the update gate control signal, which determines how much historical hidden state information to retain and how much new feature to write. When processing physical motion information, when the agent is in a high-inertia, low-damping gliding state, the update gate generates a higher retention weight, forcing the network to retain the momentum information from the previous moment, thereby smoothing the motion trajectory and effectively suppressing control oscillations caused by environmental feedback lag. The reset gate component is responsible for calculating the reset gate control signal, which determines how much historical information to forget when calculating candidate states. During sudden changes in operating conditions such as emergency stops and sharp turns, the reset gate adjusts the signal to block past inertial flow, allowing the network to quickly ignore old momentum information and thus rapidly accept and respond to new observation features, ensuring control flexibility. The state fusion component is responsible for performing nonlinear transformations and the final weighted combination. The state fusion component first uses the reset gate to filter historical hidden states, calculates candidate hidden states by combining them with current attention features, and then uses the update gate to perform weighted interpolation between the original historical hidden states and the newly calculated candidate hidden states, finally outputting a deep temporal feature vector that integrates long-term momentum dependence and current spatial awareness.

[0081] like Figure 2As shown, the Transformer hybrid network includes a supernetwork branch and a hybrid computation layer. The supernetwork branch is used to generate hybrid parameters. The supernetwork branch contains a backbone network and a parameter generation mapping layer. The backbone network has the same architecture as the noise-resistant feature extraction layer at the agent end, namely, it includes an entity self-attention processing layer, a task cross-attention processing layer, and a cooperative self-attention processing layer in sequence. The entity self-attention processing layer uses the Sparsemax function and GRU gating to capture the spatial topological relationships between entities in the global state. The Sparsemax function maps the attention scores of non-critical entities to strict zero values, filtering out false landmark noise. The task cross-attention processing layer introduces task embedding vectors. As keys and values, task context information is injected; a collaborative self-attention processing layer enhances the feature representation, and the final output is a noise-resistant deep global feature containing task awareness and temporal information. .

[0082] In this embodiment, an improved multi-particle environment (MPE) is used as the experimental platform. To simulate the control challenges of high inertia and low friction in the real physical world, such as gliding on ice or zero-gravity movement in space, this application has deeply customized the environmental physics engine.

[0083] The specific environmental parameter settings are shown in Table 1.

[0084] Table 1 Environmental Parameter Settings

[0085]

[0086] like Figure 1 As shown, this application presents a multi-agent, multi-task cooperative reinforcement learning method, comprising the following steps:

[0087] Step 1: Initialize the agent and central training module, and initialize the task sampling probability. The environment generates the current task to be executed based on the task sampling probability, and synchronously provides the agent's original local multi-source heterogeneous observation data and global state vector to the multi-agent multi-task collaborative reinforcement learning system. Subsequently, the agent performs vectorization encoding on the collected local multi-source heterogeneous observation data, and performs feature embedding through the entity encoder and task encoder of the spatiotemporal fusion architecture to form an entity embedding vector sequence and a task embedding vector. Specifically, this includes the following steps:

[0088] Step 1.1: Initialize the full network parameters of the multi-agent multi-task collaborative reinforcement learning system and initialize the sampling probability of each task (average sampling in the initial stage). The environment samples from the preset task set according to the current task sampling probability to generate the instruction code of the task to be executed at the current moment, and resets the environment state according to the task to be executed.

[0089] Step 1.2: For any agent in the multi-agent, multi-task cooperative reinforcement learning system At the current time step The intelligent agent acquires local multi-source heterogeneous observation data through airborne sensors or communication modules. At the same time, the multi-agent multi-task collaborative reinforcement learning system acquires the global state vector of the environment. The local multi-source heterogeneous observation data includes a set of entity features and the code of the task instruction to be executed issued by the system at the current moment.

[0090] The entity feature set includes the agent's own motion state information, such as its current linear velocity, angular velocity, acceleration, and position coordinates. It also includes the relative state information of other entities within the detection range, including friendly agents, real landmarks, false interfering landmarks, and obstacles.

[0091] The system-issued task instruction code at the current moment indicates the type of task the agent should perform. The task set includes cooperative coverage, formation maintenance, cooperative push-box, active pursuit, and defensive avoidance. Specific definitions are as follows:

[0092] Cooperative coverage: requires agents to disperse and cover all real-world landmarks while avoiding collisions with each other.

[0093] Formation maintenance: requires agents to form regular polygons or specific geometric formations around a real landmark.

[0094] Collaborative Pushbox: This requires agents to cooperate in pushing an object to a target point and preventing hostile targets from getting close.

[0095] Active Pursuit: In this mission, the agent is slower than the enemy. The agent needs to cooperate to surround and capture the enemy. Collisions will earn positive rewards.

[0096] Defense and Evasion: In this mission, the agent is faster than the enemy, but the agent must survive the enemy's pursuit. If a collision occurs with the enemy, it will be penalized through negative rewards.

[0097] Step 1.3: Organize the entity features from all observed entity feature sets into a local observation sequence. Then, use an entity encoder to map the local observation sequence element-by-element, converting the physical vector of each entity feature into a high-dimensional entity embedding vector of dimension 32, forming an entity embedding vector sequence. For the task instruction encoding to be executed, the multi-agent multi-task collaborative reinforcement learning system uses a spatiotemporal fusion architecture task encoder to map the task instruction encoding to be executed into a continuous task embedding vector of dimension 32. The mapping process is represented as follows:

[0098]

[0099]

[0100] in, Represents intelligent agents At time step Local observation sequence, This indicates the instruction code for the task to be executed. The nonlinear mapping function representing the entity encoder. This represents the mapping function of the task encoder. and These represent the output entity embedding vector sequence and task embedding vector, respectively.

[0101] Step 2: Construct a noise-resistant feature extraction layer based on sparse attention. Traditional attention mechanisms often fail to completely eliminate the interference from the three false landmarks and dynamic noise caused by high inertial motion in the observation data. Therefore, this application designs a noise-resistant feature extraction layer based on the Sparsemax activation function. The entity embedding vector sequence and task embedding vector obtained in Step 1 are input into the noise-resistant feature extraction layer to remove noise interference and obtain deep temporal features. Specifically, the steps are as follows:

[0102] Step 2.1: Input the entity embedding vector sequence obtained in Step 1 into the entity self-attention processing layer of the noise-resistant feature extraction layer. Then, after passing through the first GRU gated unit, the spatial topological relationship and relative distance features between entities are captured by calculating the mutual attention weights between entities, and the first intermediate feature is output. The physical layer between entities here refers to the physical relationships between real objects in the environment, such as agent A and agent B, or agent A and landmark C. In neural networks, these entities exist as individual vector elements in a sequence of entity embedding vectors. The self-attention mechanism calculates the correlation, or weights, between vectors in this sequence.

[0103] Step 2.2: The first intermediate feature output in Step 2.1 The input noise-resistant feature extraction layer is processed by a task cross-attention layer. In the task cross-attention layer, the task embedding vector generated in step 1 is used as the key and value. The first intermediate feature is output in step 2.1. As a query, it then passes through the second GRU gating unit to obtain the second intermediate feature. Second intermediate feature It incorporates task context information; this design forces the abstract task intent to be injected into specific physical observation features. For example, in a cooperative box-pushing task, the model automatically focuses on the physical features related to pushing through cross-attention.

[0104] Step 2.3, Second Intermediate Feature The collaborative self-attention processing layer, which enters the noise-resistant feature extraction layer, is used to enhance the communication features and collaborative representations within the multi-agent body. Finally, it passes through the third GRU gating unit to output deep temporal features.

[0105] In the attention weight calculation process of the above layers, to achieve thorough noise filtering, this application abandons the traditional Softmax normalization function and innovatively adopts the Sparsemax function. Unlike Softmax, which always outputs non-zero probabilities, Sparsemax can directly truncate the probability weights of non-critical or interfering entities to strict zero, achieving hard attention filtering. Experiments show that this mechanism can reset the weights of three false landmarks to 0, thereby significantly improving the robustness of perception. Specifically, the Sparsemax function is used to find the probability distribution with the closest Euclidean distance in the process of calculating attention weights in the entity self-attention processing layer, task cross-attention processing layer, and cooperative self-attention processing layer.

[0106]

[0107] in, express Simplex For the original attention score, The attention weights are represented by this formula, which indicates the search for the probability distribution with the closest Euclidean distance. Due to the boundary properties of the simplex, this solution is often sparse. Through this mechanism, the agent can automatically ignore interference from false targets in the environment.

[0108] The weight calculation process for the entity self-attention processing layer, task cross-attention processing layer, and collaborative self-attention processing layer is as follows: the layer normalization unit standardizes the input feature vector; the sparse attention calculation unit transforms the processed features into query, key, and value through different linear transformations; then, the dot product of the query and key is calculated and scaled to obtain the original attention score. Then, the Sparsemax function is used to calculate the original attention score. Convert to attention weights Finally, the gated recurrent residual unit (GRU) receives the attention weights output by the sparse attention computation unit and the original uncomputed input feature vector, fuses the temporal information and performs a nonlinear transformation to output the final features of the attention layer.

[0109] Specifically, suppose the input of a certain attention sublayer is... The output after attention calculation is The final feature output after processing by the gating unit It is calculated using the following set of formulas:

[0110]

[0111]

[0112]

[0113]

[0114] in, Indicates an update to the door. This indicates that the door is being reset. This represents a candidate hidden state. and All are learnable weight matrices. For bias parameters, This represents the Sigmoid activation function. This represents element-wise multiplication. Through this step, the agent can smooth its current action based on historical movement trends, effectively suppressing overreactions.

[0115] Step 3: The hypernetwork receives the task embedding vector generated in Step 1. As input, the dynamic weight matrix required by the action value calculation unit is generated in real time. The action value calculation unit receives the deep temporal features and dynamic weight matrix obtained in step 2. As input, combined with the static learnable bias vector calculation, the local output of the agent is obtained. Value, the agent utilizes the local The system selects and executes actions, interacts with the environment to generate trajectory sequences, and stores these trajectories in a dual experience replay pool—a normal experience replay pool and a high-quality experience replay pool—based on the reward level. The weight matrix has a dimension of 32×5, corresponding to 5 discrete actions: up, down, left, right, and stop. Specifically, it includes the following steps:

[0116] Step 3.1: Generate the dynamic weight matrix required by the action value calculation unit in real time. :

[0117]

[0118] in, This represents the mapping function of the hypernet.

[0119] Value assessment of each action by the agent The value is:

[0120]

[0121] in, For deep temporal features, Embed vectors for tasks. Represents a statically learnable bias vector;

[0122] Step 3.2: The agent's value assessment based on the result of step 3.1. The sequence of values ​​is used to determine the action to be executed using a preset action selection strategy. This action is then used to interact with the environment in real time, generating a sequence of data defined as a trajectory. This trajectory is composed of observation data, decision-making actions, and reward signals from environmental feedback within consecutive time steps. The multi-agent, multi-task collaborative reinforcement learning system establishes a dual experience storage mechanism: the collected trajectories are stored in a general experience replay pool, while the system simultaneously monitors the cumulative rewards of each trajectory within the current batch. The cumulative return is:

[0123]

[0124] in, The maximum time step for a single round. For time steps If the cumulative reward value of a certain trajectory is higher than the average reward level of the current batch, the trajectory is judged as a high-quality trajectory and is additionally stored in the high-quality experience replay pool to provide data support for subsequent self-imitation learning.

[0125] Step 4: The Transformer hybrid network receives the global state vector of the environment and the task embedding vector generated in Step 1. and the local output of all agents The sequence of values, evaluated through a Transformer hybrid network, provides a global action value representing the overall performance. Step 4 specifically includes the following steps:

[0126] Step 4.1: Convert the global state vector of the environment The input is fed into the backbone of the Transformer hybrid network to obtain deep global features. Specifically, this involves: converting the global state vector of the environment... As input, it is mapped into a high-dimensional state feature sequence through a state processing layer, and then fed into the backbone network of the hypernetwork branch for processing. Specifically, when passing through the task cross-attention processing layer, the task embedding vector generated in step 1 is introduced. The task information is fused; after passing through an entity self-attention processing layer, a task cross-attention processing layer, a cooperative self-attention processing layer, and gating processing, deep global features are obtained. ;

[0127] Step 4.1 specifically includes the following steps:

[0128] Step 4.1.1, Entity Self-Attention Processing: Sparsemax and GRU gating are used to capture the spatial topological relationships between entities in the global state. The Sparsemax function maps the attention scores of non-critical entities to strict zero values, thereby filtering out false landmark noise.

[0129] Task cross-attention processing: Introducing task embedding vectors Inject task context information as both key and value;

[0130] Collaborative self-attention processing: Further enhances feature representation, and finally outputs noise-resistant deep global features containing task awareness and temporal information. ;

[0131] Step 4.2: The Transformer hybrid network includes a parameter generation mapping layer, which generates the deep global features obtained in Step 4.1. The input parameters are fed into a mapping layer, which, through linear transformation and dimensional reshaping, generates a mapping for blending local data. The value of the parameter, blending local The parameters of the value include the nonnegative mixture weight matrix. Non-negative mixed weight matrix and the first layer mixed bias vector and the second layer of mixed bias scalar Specifically:

[0132] Features are extracted from specific positions in the sequence, and a nonnegative mixed weight matrix is ​​generated through transformation and absolute value operations. Features are extracted from specific positions in the sequence and used as the first-layer mixing bias vector. Features are extracted from specific positions in the sequence, and a non-negative mixed weight matrix is ​​generated through absolute value operations. Features are extracted from specific positions in the sequence and input into the parameters to generate a mapping layer, which in turn generates a second layer of hybrid bias scalar. This yields the complete set of mixing parameters { , , , }

[0133] Step 4.3: Local outputs of all agents The sequence of values ​​is input into the hybrid computation layer, using the nonnegative hybrid weight matrix generated in step 4.2. Non-negative mixed weight matrix and the first layer mixed bias vector and the second layer of mixed bias scalar The global action value is obtained through nonlinear weighted combination calculation. Specifically:

[0134]

[0135] in, Represents the local output of all agents A row vector composed of values This represents the exponential linear unit activation function.

[0136] This process ensures that even when local observations are limited or there is interference, the system can still make an accurate value assessment of the overall strategy based on the cleaned global information, and satisfy the monotonicity constraint of individual rationality and collective rationality.

[0137] Step 5: The training optimization unit samples from both the ordinary experience replay pool and the high-quality experience replay pool, based on the real environment reward and the global action value obtained in Step 4. The temporal difference loss and self-imitation loss are calculated separately, and then a dual optimization loss function is constructed and the total loss is calculated.

[0138] like Figure 4 As shown, the training optimization unit samples from the ordinary empirical replay pool in step 3.2 to compute the underlying temporal difference loss. Simultaneously, sampling is performed from a high-quality experience replay pool to calculate the self-imitation loss. Through this dual sampling mechanism, the system can combine conventional reinforcement learning objectives with the imitation of historical successful experiences. In this embodiment, in addition to a regular experience replay pool with a capacity of 5000, the system maintains an additional high-quality experience replay pool with a capacity of 2000. This high-quality experience replay pool is specifically used to store high-quality trajectories whose historical cumulative returns are higher than the current policy's average return. Specifically, it includes the following steps:

[0139] Step 5.1: Before calculating the temporal difference loss, the training optimization unit first uses the target network to calculate the expected target value of the current state-action pair. Considering multi-tasking scenarios, this value is related to the task instructions. Related:

[0140]

[0141] Multi-agent, multi-task cooperative reinforcement learning systems update parameters by minimizing the mean squared error between the predicted global value and the target value.

[0142]

[0143] in, The size of the trajectory batch sampled from the ordinary experience playback pool. As an immediate reward signal for environmental feedback, This is a discount factor used to balance current rewards with future long-term rewards. This refers to the global state at the next moment after the action is performed. For the next joint action, This represents the periodic backup of the target network's parameters, which are the parameters of the entire network, consisting of the agent execution module and the central training module. Indicates the current state of the network. Joint actions and task instructions The value of predicting global actions.

[0144] Step 5.2, Total Loss Function for:

[0145]

[0146] in, The self-imitation loss weighting coefficients are used in this embodiment. , For the time-difference loss based on the Bellman equation, it is used for the value estimation of the learning basis. To address the self-imitation loss for high-reward trajectories in the high-quality experience replay pool, and to reinforce the memory of high-scoring experiences:

[0147]

[0148] in, This indicates the batch size of the trajectory sampled from the high-quality experience replay pool. This represents the set of samples drawn from the high-quality experience replay pool. For the first The cumulative reward of a trajectory For predictive value.

[0149] Step 6: Based on the total loss calculated in Step 5, the training optimization unit updates the full network parameters in the multi-agent multi-task collaborative reinforcement learning system using the gradient backpropagation algorithm. At the same time, it determines whether the preset evaluation round has been reached. If it has, it dynamically calculates the sampling probability of each task in the new round of training based on the average reward of each task in the testing phase. The multi-agent multi-task collaborative reinforcement learning system returns to Step 1 based on the sampling probability to start the next round of task sampling and data collection until the training termination condition is met.

[0150] During system training, the performance of each subtask is evaluated in the test environment every 10,000 steps, and the win rate or average reward for each task is calculated, specifically as follows:

[0151] Step 6.1: Total loss calculated in Step 5 The gradient backpropagation algorithm is used to update the full network parameters of the multi-agent multi-task collaborative reinforcement learning system.

[0152] Step 6.2: Determine if the preset evaluation rounds have been reached. If so, obtain the task. The current normalized test return is The maximum possible return is The level of mastery of this task is then indicated. Defined as:

[0153]

[0154] In the task sampling phase of the next training round, the multi-agent multi-task collaborative reinforcement learning system calculates the sampling probability of each task using the Softmax function based on the degree of unfamiliarity with each task. :

[0155]

[0156] in, The total number of tasks. To minimize the sampling probability, in this embodiment, the minimum sampling probability is set to 0.12. Calculate the sum of attention metrics for all tasks in the task set. For traversal index, For the temperature coefficient, in this embodiment, the temperature coefficient is... Scaling the metrics.

[0157] The simulation experiment parameter settings for this application are shown in Table 2.

[0158] Table 2 Simulation Experiment Parameter Settings

[0159]

[0160] By organically combining the above seven steps, this application achieves comprehensive optimization from perception and decision-making to training.

[0161] To verify the effectiveness of this application, a comparative experiment was conducted with the current state-of-the-art multi-agent multi-task reinforcement learning benchmark algorithm TETQmix. The experimental results are as follows: Figure 5 As shown, the horizontal axis represents the number of training steps, in units of: The vertical axis represents the test results. Figure 5 The solid line in the figure represents the average reward curve of the proposed method under multiple random seed experiments, the dashed line represents the average reward curve of the benchmark algorithm TETQmix, and the shaded area represents the standard deviation range of the corresponding algorithm in multiple experiments, reflecting the training stability of the algorithm. The experiments were conducted in a complex physical environment containing 5 controlled agents, 4 adversarial agents, 5 real landmarks and 3 moving fake landmarks, covering 5 heterogeneous tasks: cooperative coverage, formation maintenance, cooperative sowing, active pursuit and defense avoidance.

[0162] In terms of detailed performance on single tasks, the proposed method outperforms the comparative algorithms in most tasks, particularly demonstrating significant advantages in handling environmental complexity and task difficulty. In the cooperative coverage task, the proposed method exhibits a very rapid upward trend in the early stages of training, while TETQmix requires a longer exploration time and experiences greater initial fluctuations. Similarly, in the cooperative sowing task, the proposed method's reward curve not only rises faster but also shows significantly better stability after convergence compared to the comparative algorithms. This indicates that the self-imitation learning mechanism effectively alleviates the exploration challenge in high-inertia environments and significantly improves sample efficiency.

[0163] In challenging adversarial tasks, the advantages of this application are particularly prominent. In defense and avoidance tasks, the TETQmix algorithm experienced severe performance fluctuations and significant drops during the mid-training stage, demonstrating the instability of its strategy when facing dynamic threats. In contrast, the method in this application can quickly converge to a high-reward level and remain stable throughout, without significant performance collapse. This is mainly due to the effective shielding of interfering landmarks by the Sparsemax hard attention mechanism and the accurate prediction of high-speed motion states by gated temporal memory, enabling the agent to make more decisive and accurate decisions when facing enemy pursuit and environmental noise. In active pursuit tasks, although the comparative algorithm can occasionally reach high peak values ​​in the later stages of training, its curve fluctuates greatly, indicating that the strategy is not very reliable; while this application, although slightly conservative in extreme values, shows high training stability and robustness overall.

[0164] In the formation preservation task, although the final performance of both methods is comparable, the convergence curve of this application is smoother and the variance band is narrower. This demonstrates that the action generation mechanism based on hypernetworks can adapt well to tasks requiring fine-grained micro-management and avoids negative transfer during policy switching.

[0165] From the overall trend of average returns, the comprehensive advantages of the proposed method are further established. Throughout the entire 1 million-step training process, the average return curve of the proposed method consistently remained above the comparison algorithm, with a narrower standard deviation band. Particularly in the first half of training, the proposed method rapidly widened the gap with the benchmark algorithm thanks to the hypernetwork's rapid adaptability to heterogeneous tasks and its dynamic course learning strategy. As training progressed, although the benchmark algorithm gradually caught up, the proposed method maintained higher asymptotic performance and lower variance, demonstrating the algorithm's reliability in long-term operation.

[0166] In summary, the experimental results fully demonstrate that by combining hypernetic action generation, gating attention mechanism, and self-imitation learning, this application, compared with existing mainstream technologies, has faster convergence speed, stronger anti-interference ability, and more stable control performance when dealing with high inertia, strong noise, and multi-task heterogeneous scenarios, and can effectively solve the problem of multi-agent collaborative control in complex environments.

[0167] The above description is merely an embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principle of the present invention should be included within the scope of the claims of the present invention.

Claims

1. A multi-agent, multi-task collaborative reinforcement learning method based on a spatiotemporal fusion architecture, characterized in that: The multi-agent, multi-task collaborative reinforcement learning method is implemented through a multi-agent, multi-task collaborative reinforcement learning system. This system includes agent-side components and a central training module. Each agent at the agent-side component includes a spatiotemporal fusion architecture and a hypernetwork action generation module architecture. The hypernetwork action generation module architecture includes a hypernetwork, a statically learnable bias vector, and an action value calculation unit. The central training module includes a Transformer hybrid network, a dual experience replay pool, and a training optimization unit. Specifically, the multi-agent, multi-task collaborative reinforcement learning method includes the following steps: Step 1: Initialize the agent and central training module, and initialize the task sampling probability. Based on the task sampling probability, the environment generates the current task to be executed, and synchronously provides the agent's original local multi-source heterogeneous observation data and global state vector to the multi-agent multi-task collaborative reinforcement learning system. Subsequently, the agent performs vectorization encoding on the collected local multi-source heterogeneous observation data, and performs feature embedding through the entity encoder and task encoder of the spatiotemporal fusion architecture to form an entity embedding vector sequence and a task embedding vector. The local multi-source heterogeneous observation data includes an entity feature set and a task instruction code to be executed. The entity feature set contains the agent's own motion state information, i.e., the current linear velocity, angular velocity, acceleration, and position coordinates. At the same time, the entity feature set also contains the relative state information of other entities within the detection range. Other entities include friendly agents, real landmarks, false interference landmarks, and obstacles. The task instruction code to be executed issued by the system at the current moment is used to indicate what type of task the agent should currently perform. The task set includes cooperative coverage, formation maintenance, cooperative pushbox, active pursuit, and defensive avoidance, specifically defined as follows: Cooperative coverage: requires intelligent agents to disperse and cover all real-world landmarks while avoiding collisions with each other; Formation maintenance: requires agents to form a regular polygon or a specific geometric formation around a real-world landmark; Collaborative Sort Box: This requires agents to cooperate in pushing an object to a target point and preventing hostile targets from getting close. Active Pursuit: In this mission, the agent is slower than the enemy. The agent needs to cooperate to surround and capture the enemy. Collisions will earn positive rewards. Defense and Evasion: In this mission, the agent is faster than the enemy, but the agent must survive the enemy's pursuit. If a collision occurs with the enemy, it will be penalized through negative rewards. Step 2: Construct a noise-resistant feature extraction layer based on sparse attention. Input the entity embedding vector sequence and task embedding vector obtained in Step 1 into the noise-resistant feature extraction layer to obtain deep temporal features after eliminating noise interference. Step 3: The hypernetwork receives the task embedding vector generated in Step 1. As input, the dynamic weight matrix required by the action value calculation unit is generated in real time. The action value calculation unit receives the deep temporal features and dynamic weight matrix obtained in step 2. As input, combined with the static learnable bias vector calculation, the local output of the agent is obtained. Value, the agent utilizes the local Values ​​are selected and actions are executed, interactions with the environment generate trajectory sequences, and the trajectories are stored in the ordinary experience replay pool and the premium experience replay pool respectively according to the reward level. Step 4: The Transformer hybrid network receives the global state vector of the environment and the task embedding vector generated in Step 1. and the local output of all agents The sequence of values ​​is evaluated using a Transformer hybrid network to provide the global action value. ; Step 5: The training optimization unit samples from both the ordinary experience replay pool and the high-quality experience replay pool, based on the real environment reward and the global action value obtained in Step 4. The temporal difference loss and self-imitation loss are calculated separately, and then a dual optimization loss function is constructed and the total loss is calculated. Step 6: Based on the total loss calculated in Step 5, the training optimization unit updates the full network parameters in the multi-agent multi-task collaborative reinforcement learning system. Simultaneously, it determines whether a preset evaluation round has been reached. If so, it dynamically calculates the sampling probability of each task in the new training round based on the average reward of each task during the testing phase. The multi-agent multi-task collaborative reinforcement learning system then returns to Step 1 based on the sampling probability to begin the next round of task sampling and data acquisition, until the training termination condition is met; wherein: The spatiotemporal fusion architecture includes an entity encoder, a task encoder, and a noise-resistant feature extraction layer. The noise-resistant feature extraction layer includes an entity self-attention processing layer, a task cross-attention processing layer, and a cooperative self-attention processing layer. The entity self-attention processing layer, the task cross-attention processing layer, and the cooperative self-attention processing layer have the same internal structure, and each of the entity self-attention processing layer, the task cross-attention processing layer, and the cooperative self-attention processing layer includes a layer normalization unit, a sparse attention calculation unit, and a gated cyclic residual unit. The gated cyclic residual unit in the entity self-attention processing layer is a first GRU gated unit, the gated cyclic residual unit in the task cross-attention processing layer is a second GRU gated unit, and the gated cyclic residual unit in the cooperative self-attention processing layer is a third GRU gated unit. The layer normalization unit is responsible for receiving the feature vectors of the entity self-attention processing layer, the task cross-attention processing layer, or the cooperative self-attention processing layer, which are input to the layer normalization unit. It also performs normalization processing on the input feature vectors of the entity self-attention processing layer, the task cross-attention processing layer, or the cooperative self-attention processing layer to unify the data distribution. The sparse attention computing unit receives the normalized features, calculates the interaction weights between entities through the Sparsemax function, extracts the clean spatial interaction features of the current time step, and passes them as new observation information to the next level. The gated cyclic residual unit includes an update gate component, a reset gate component, and a state fusion component. The update gate component is responsible for calculating the update gate control signal, which determines how much historical hidden state information to retain and how much new feature to write. The reset gate component is responsible for calculating the reset gate control signal, which determines how much historical information to forget when calculating candidate states. The state fusion component is responsible for performing nonlinear transformations and the final weighted combination. The state fusion component first uses the reset gate to filter historical hidden states, calculates candidate hidden states by combining them with the current attention features, and then uses the update gate to perform weighted interpolation between the original historical hidden states and the newly calculated candidate hidden states, finally outputting a deep temporal feature vector that integrates long-term momentum dependence and current spatial awareness. In step 2, the entity self-attention processing layer, task cross-attention processing layer, and cooperative self-attention processing layer use the Sparsemax function to find the probability distribution with the closest Euclidean distance during the calculation of attention weights. in, express Simplex For the original attention score, For attention weights.

2. The multi-agent, multi-task collaborative reinforcement learning method based on a spatiotemporal fusion architecture as described in claim 1, characterized in that: Step 1 is as follows: Step 1.1: Initialize the full network parameters of the multi-agent multi-task collaborative reinforcement learning system and initialize the sampling probability of each task. The environment samples from the preset task set according to the current task sampling probability to generate the instruction code of the task to be executed at the current moment, and resets the environment state according to the task to be executed. Step 1.2: For any agent in a multi-agent, multi-task cooperative reinforcement learning system At the current time step The intelligent agent acquires local multi-source heterogeneous observation data, while the multi-agent multi-task collaborative reinforcement learning system acquires the global state vector of the environment. The local multi-source heterogeneous observation data includes a set of entity features and the code of the task instruction to be executed. Step 1.3: Organize the entity features from all observed entity feature sets into a local observation sequence. Then, use an entity encoder to map the local observation sequence element-by-element, converting the physical vector of each entity feature into a high-dimensional entity embedding vector, forming an entity embedding vector sequence. For the task instruction encoding to be executed, the multi-agent, multi-task collaborative reinforcement learning system uses a spatiotemporal fusion architecture task encoder to map the task instruction encoding to be executed into continuous task embedding vectors. The mapping process is represented as follows: in, Represents intelligent agents At time step Local observation sequence, This indicates the instruction code for the task to be executed. The nonlinear mapping function representing the entity encoder. This represents the mapping function of the task encoder. and These represent the output entity embedding vector sequence and task embedding vector, respectively.

3. The multi-agent, multi-task collaborative reinforcement learning method based on a spatiotemporal fusion architecture as described in claim 1, characterized in that: Step 2 specifically includes the following steps: Step 2.1: Input the entity embedding vector sequence obtained in Step 1 into the entity self-attention processing layer of the noise-resistant feature extraction layer, and then pass it through the first GRU gate unit to capture the spatial topological relationship and relative distance features between entities, and output the first intermediate feature. ; Step 2.2: The first intermediate feature output in Step 2.1 The input noise-resistant feature extraction layer is processed by a task cross-attention layer. In the task cross-attention layer, the task embedding vector generated in step 1 is used as the key and value. The first intermediate feature is output in step 2.

1. As a query, it then passes through the second GRU gating unit to obtain the second intermediate feature. Second intermediate feature It incorporates task context information; Step 2.3, Second Intermediate Feature The collaborative self-attention processing layer, which enters the noise-resistant feature extraction layer, is used to enhance the communication features and collaborative representations within the multi-agent body. Finally, it passes through the third GRU gating unit to output deep temporal features.

4. The multi-agent, multi-task collaborative reinforcement learning method based on a spatiotemporal fusion architecture as described in claim 1, characterized in that: Step 3 specifically includes the following steps: Step 3.1: Generate the dynamic weight matrix required by the action value calculation unit in real time. : in, The mapping function representing the hypernetic network; Value assessment of each action by the agent The value is: in, For deep temporal features, Embed vectors for tasks. Represents a statically learnable bias vector; Step 3.2: The agent's value assessment based on the result of step 3.

1. The sequence of values ​​is used to determine the action to be executed using a preset action selection strategy. This determined action is then used to interact with the environment in real time, generating sequence data, defined as a trajectory. The multi-agent, multi-task collaborative reinforcement learning system establishes a dual experience storage mechanism: the collected trajectories are stored in a general experience replay pool, while simultaneously, the system monitors the cumulative rewards of each trajectory within the current batch in real time. The cumulative return is: in, The maximum time step for a single round. For time step If the cumulative reward value of a certain trajectory is higher than the average reward level of the current batch, the trajectory is judged as a high-quality trajectory and stored in the high-quality experience replay pool.

5. The multi-agent, multi-task collaborative reinforcement learning method based on a spatiotemporal fusion architecture according to claim 1, characterized in that: In step 4, the Transformer hybrid network includes a super network branch and a hybrid computation layer. The super network branch is used to generate hybrid parameters. The super network branch contains a backbone network and a parameter generation mapping layer. The backbone network has the same architecture as the noise-resistant feature extraction layer at the agent end, that is, it includes an entity self-attention processing layer, a task cross-attention processing layer and a cooperative self-attention processing layer in sequence.

6. The multi-agent, multi-task collaborative reinforcement learning method based on a spatiotemporal fusion architecture according to claim 5, characterized in that: Step 4 specifically includes the following steps: Step 4.1: Convert the global state vector of the environment The input is fed into a Transformer hybrid network to obtain deep global features. Specifically, this involves: converting the global state vector of the environment... As input, it is mapped into a high-dimensional state feature sequence through a state processing layer, and then fed into the backbone network of the hypernetwork branch for processing. Specifically, when passing through the task cross-attention processing layer, the task embedding vector generated in step 1 is introduced. The task information is fused; after passing through an entity self-attention processing layer, a task cross-attention processing layer, a cooperative self-attention processing layer, and gating processing, deep global features are obtained. ; Step 4.2: The Transformer hybrid network includes a parameter generation mapping layer, which generates the deep global features obtained in Step 4.

1. The input parameters are fed into a mapping layer, which, through linear transformation and dimensional reshaping, generates a mapping for blending local data. The value of the parameter, blending local The parameters of the value include the nonnegative mixture weight matrix. Non-negative mixed weight matrix and the first layer mixed bias vector and the second layer of mixed bias scalar ; Step 4.3: Local outputs of all agents The sequence of values ​​is input into the hybrid computation layer, using the nonnegative hybrid weight matrix generated in step 4.

2. Non-negative mixed weight matrix and the first layer mixed bias vector and the second layer of mixed bias scalar The global action value is obtained through nonlinear weighted combination calculation. .

7. The multi-agent, multi-task collaborative reinforcement learning method based on a spatiotemporal fusion architecture according to claim 1, characterized in that: Step 5 specifically includes the following steps: Step 5.1: Before calculating the temporal difference loss, the training optimization unit first uses the target network to calculate the expected target value of the current state-action pair. : Multi-agent, multi-task cooperative reinforcement learning systems update parameters by minimizing the mean squared error between the predicted global value and the target value. in, The size of the trajectory batch sampled from the ordinary experience playback pool. As an immediate reward signal for environmental feedback, This is a discount factor used to balance current rewards with future long-term rewards. This refers to the global state at the next moment after the action is performed. For the next joint action, The parameters representing the target network, Indicates the current state of the network. Joint actions and task instructions The value of predicting global actions; Step 5.2, Total Loss Function for: in, For the self-imitation loss weighting coefficient, For time-series difference loss based on Bellman equations, To address the self-imitation loss for high-return trajectories in a high-quality experience replay pool: in, This indicates the batch size of the trajectory sampled from the high-quality experience replay pool. This represents the set of samples drawn from the high-quality experience replay pool. For the first The cumulative reward of a trajectory For predictive value.

8. The multi-agent, multi-task collaborative reinforcement learning method based on a spatiotemporal fusion architecture according to claim 6, characterized in that: Step 6 specifically involves: Step 6.1: Total loss calculated in Step 5 Update the full network parameters in the multi-agent multi-task collaborative reinforcement learning system; Step 6.2: Determine if the preset evaluation rounds have been reached. If so, obtain the task. The current normalized test return is The maximum possible return is The level of mastery of this task is then indicated. Defined as: In the task sampling phase of the next training round, the multi-agent multi-task collaborative reinforcement learning system calculates the sampling probability of each task based on the degree of unfamiliarity with each task. : in, The total number of tasks. The minimum sampling probability, Calculate the sum of attention metrics for all tasks in the task set. For traversal index, This is the temperature coefficient.

Images