A virtual-real migration method based on a general multi-agent parallel reinforcement learning framework

Abstract

The present application relates to a kind of real and virtual migration method based on general multi-agent parallel reinforcement learning framework, comprising the following steps: building parallel simulation environment adapted to task, loading the agent model required by task, randomly initializing environment parameter and agent state;Training stage, load multi-agent reinforcement learning algorithm, agent obtains local observation information from simulation environment, input local observation information and environment state into algorithm, policy network outputs action, agent executes action and interacts with environment to obtain reward, periodically randomly initialize environment parameter in training using domain randomization, training is carried out in combination with post-experience replay mechanism, save model parameters after training is completed;Execution stage, after model is lightened, it is deployed to actual agent, by layered instruction architecture, upper layer action instruction is sent by model, it is parsed into bottom drive signal by control unit, and action is executed by execution unit driven agent.

Description

Technical Field

[0001] This invention relates to the field of multi-agent cooperative control technology, and in particular to a virtual-real transfer method based on a general multi-agent parallel reinforcement learning framework. Background Technology

[0002] In recent years, artificial intelligence technology has made significant progress, especially in the field of reinforcement learning (RL). Reinforcement learning, by enabling agents to learn optimal policies through interaction with their environment, has become one of the core technologies for solving complex decision-making problems. In many application scenarios, such as robot control, autonomous driving, and smart logistics, multiple agents often need to work collaboratively to achieve common goals, thus spurring extensive research into multi-agent systems.

[0003] However, applying multi-agent reinforcement learning algorithms directly to the real physical world faces significant challenges. First, training in real-world environments is typically costly, time-consuming, and may involve unpredictable safety risks, especially in scenarios where equipment is expensive or operations are dangerous.

[0004] Virtual-to-real transfer technology has emerged to address this need, aiming to deploy models trained in simulators onto real hardware. Despite some success, research on virtual-to-real transfer for multi-agent systems is still in the exploratory stage, as the collaboration and communication strategies among multiple agents introduce more complex real-world problems during the transfer process.

[0005] Meanwhile, Multi-Agent Reinforcement Learning (MARL) provides an effective paradigm for solving complex cooperative tasks. However, how to effectively design such a framework to overcome the dual complexity brought about by multiple agents and multiple tasks is a key problem that urgently needs to be solved. Existing simulation environments are often only designed for specific types of tasks or agents, making it difficult to support diverse cooperative tasks and complex environmental terrains, which to some extent affects the innovation and validation of algorithms.

[0006] In summary, there is currently a lack of a virtual-to-real migration method that can deploy cooperative strategies trained in a simulation environment onto real-world agents, thereby improving the task execution and generalization capabilities of multi-agent systems in complex real-world environments. Summary of the Invention

[0007] The purpose of this invention is to overcome the shortcomings of the existing technology and provide a virtual-real transfer method based on a general multi-agent parallel reinforcement learning framework, so as to enable the policy network trained in the simulation environment to be directly deployed to real agents without additional training.

[0008] The objective of this invention can be achieved through the following technical solutions:

[0009] One aspect of the present invention provides a virtual-real transfer method based on a general multi-agent parallel reinforcement learning framework, comprising the following steps:

[0010] Build a parallel simulation environment adapted to the task, load the agent model required by the task, and randomly initialize the environment parameters and agent state;

[0011] During the training phase, a multi-agent reinforcement learning algorithm is loaded. The agent obtains local observation information from the simulation environment, inputs the local observation information and the environment state into the algorithm, the policy network outputs actions, the agent executes the actions and interacts with the environment to obtain rewards. During training, domain randomization is used to periodically and randomly initialize the environment parameters, and training is carried out in combination with a post-experience replay mechanism. After training is completed, the model parameters are saved.

[0012] During the execution phase, the model is lightweighted and then deployed to the actual intelligent agent. Through a hierarchical instruction architecture, the model sends upper-level action instructions, which are parsed into lower-level driving signals by the control unit, and then the execution unit drives the intelligent agent to perform actions.

[0013] As a preferred technical solution, the construction of a parallel simulation environment adapted to the task includes the following steps:

[0014] Based on the Issac Gym monolithic simulation engine, the simulation environment is extended to support the simulation and interaction of homogeneous and heterogeneous multi-agent systems, and the simulation process and data pipeline are optimized to adapt to multi-agent reinforcement learning algorithm standards.

[0015] As a preferred technical solution, the domain randomization method includes the following steps:

[0016] At the beginning of each training round, a set of dynamic parameters are randomly sampled and the terrain parameters are independently randomized and remain unchanged throughout the round;

[0017] The time step is randomized at preset intervals and overall randomization is performed. The dynamic parameters include agent mass, motor torque and sensor noise, and the terrain parameters include ground friction coefficient and terrain height map.

[0018] As a preferred technical solution, the training phase includes the following embedded system recognition process:

[0019] Based on dynamic parameters and the generated trajectory, the recognition process is embedded in the policy network using a recurrent neural network. By fusing the current input with the previous hidden state, the current hidden state is updated, thereby realizing the recognition of the agent's own dynamic parameters.

[0020] As a preferred technical solution, the post-event experience replay mechanism includes the following steps:

[0021] By reconstructing the dynamic goal, based on the failed trajectory under the original goal, a mapping function from state to goal is defined to represent the new goal satisfied by the final state, and a binary reward function is used to recalculate the reward for the entire trajectory.

[0022] As a preferred technical solution, the lightweighting process includes the following steps:

[0023] By removing redundant connections and neurons through model pruning, quantizing network weights and activation values, reducing data precision, and distilling complex policy networks into lightweight networks, the network can be adapted to embedded computing platforms.

[0024] As a preferred technical solution, the layered instruction architecture includes:

[0025] The strategy network outputs high-level, platform-independent upper-layer action commands. The intelligent agent control unit parses the upper-layer action commands into lower-layer drive signals. The execution unit executes actions based on the lower-layer drive signals, which include joint torque, speed, and position.

[0026] As a preferred technical solution, the parameters of the policy network are updated using the policy gradient method combined with collected experience, with the goal of updating ρ. μ Maximize the cumulative expected return under the distribution, where the cumulative expected return is:

[0027]

[0028] in, To accumulate expected reward, π represents the action policy, T represents the maximum time step, γ represents the discount factor, and N represents the number of agents. The reward for agent i at time step t. Let be the expected reward value under strategy π.

[0029] In another aspect, an electronic device is provided, comprising: one or more processors and a memory, wherein the memory stores one or more programs, the one or more programs including instructions for executing the virtual-to-real transfer method based on the aforementioned general multi-agent parallel reinforcement learning framework.

[0030] In another aspect, the present invention provides a computer-readable storage medium including one or more programs executable by one or more processors of an electronic device, said one or more programs including instructions for performing the aforementioned virtual-to-real transfer method based on a general multi-agent parallel reinforcement learning framework.

[0031] Compared with the prior art, the present invention has at least one of the following beneficial effects:

[0032] (1) Achieved efficient parallel simulation: In view of the problem that the existing multi-agent simulation environment is inefficient and cannot support ultra-large-scale multi-agent collaborative training, this invention is based on the Issac Gym single simulation engine extension, optimizes the simulation process and data pipeline, enables the framework to support the parallel operation of thousands of environment instances in a single thread, and supports multi-threaded extension, making full use of the parallel computing capabilities of the graphics computing unit, achieving efficient parallel simulation, and significantly improving the training efficiency of multi-agent reinforcement learning.

[0033] (2) Unified simulation and training of heterogeneous multi-agent systems is realized: In view of the problem that existing multi-agent simulation platforms are limited to homogeneous agent systems and cannot be compatible with heterogeneous multi-agent systems, the general multi-agent environment framework of this invention supports the simulation and interaction of homogeneous and heterogeneous multi-agent systems. Different types of agent models can be loaded and trained in the same environment, which enhances the framework's adaptability to diverse collaborative tasks and improves the universality of multi-agent reinforcement learning methods.

[0034] (3) Improved generalization ability and practicality of virtual-real transfer in multi-agent reinforcement learning: In view of the problem that when multi-agent reinforcement learning strategies are transferred from virtual simulation to real environment, due to the difference in dynamics between virtual and reality, a lot of debugging is required and the generalization ability is poor, the present invention adopts domain randomization in the training stage, periodically and randomly initializes environmental parameters, dynamic parameters, and terrain parameters, embeds system recognition, implicitly encodes environmental dynamic characteristics and post-experience playback mechanism through recurrent neural network, and optimizes reward calculation through dynamic target reconstruction, thereby enhancing the adaptability of the strategy to environmental changes. Attached Figure Description

[0035] Figure 1 This is a flowchart of the virtual-real transfer method based on a general multi-agent parallel reinforcement learning framework in the embodiment.

[0036] Figure 2 This is a schematic diagram illustrating the implementation of virtual-to-real migration in the embodiment;

[0037] Figure 3 This is a system relationship diagram of the virtual-to-real migration module and method in the embodiment;

[0038] Figure 4 This is a schematic diagram of the electronic device in the embodiment.

[0039] The system comprises: 1. Virtual-to-real migration system; 2. Simulation environment unit; 3. Multi-agent reinforcement learning unit; 4. Policy network unit; 5. Agent; 6. Control unit; and 7. Execution unit. Detailed Implementation

[0040] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0041] Example 1

[0042] To address the challenge of transferring multi-agent reinforcement learning from simulation to reality, this embodiment provides a virtual-to-real transfer method based on a general multi-agent parallel reinforcement learning framework. This method aims to directly deploy policy networks trained in a simulation environment to real agents without additional training. (See also...) Figure 1 The method includes the following steps:

[0043] Step S101: Build a virtual simulation platform and a device platform respectively, and deploy sensors and collaborative task scenarios based on a general multi-agent environment framework.

[0044] Specifically, this step involves building a parallel simulation environment based on the task content, loading the agent model required for the task, and randomly initializing the environment parameters and agent state.

[0045] Step S102: Load the heterogeneous multi-agent system model and initialize the environment parameters.

[0046] Specifically, this step loads a multi-agent reinforcement learning algorithm. Each agent obtains local observation information from the simulation environment, inputs local observations, environmental states, and other data into the algorithm, and the policy network outputs executable actions. The agents execute these actions and interact with the environment to obtain rewards.

[0047] Step S103: Input the task instructions and the observation data captured by the sensors into the multi-agent reinforcement learning algorithm, and train the policy model and perform simulation.

[0048] Specifically, this step involves periodically and randomly initializing environmental parameters during training, and using a post-experience replay mechanism to enhance learning efficiency.

[0049] The domain randomization method randomly samples a set of dynamic parameters and independently randomizes terrain parameters at the beginning of each round, keeping these parameters constant throughout the round. The system randomizes the time step and performs overall randomization at regular intervals to train robust policies adaptable to different conditions. Dynamic parameters may include agent mass, motor torque, joint damping coefficient, gravity magnitude, time intervals between adjacent actions, and sensor noise from state observations. Independently randomized terrain parameters may include ground friction coefficient, damping coefficient, and terrain height map. Based on the sampled dynamic parameters and the generated trajectory, a recurrent neural network is used to embed the system's recognition process within the policy for identification.

[0050] The post-event experience replay mechanism employs dynamic goal reconstruction. Based on the failed trajectory under the original goal, a mapping function from state to goal is defined to represent the goal satisfied by the final state, and the reward for the trajectory is recalculated under the new goal. For reward setting, a binary reward function is used as the reward function based on the new goal after dynamic goal reconstruction, and the reward is recalculated for the entire trajectory.

[0051] Step S104: After compressing the policy model, deploy it to the actual intelligent agent. The policy model transmits upper-level action instructions to the heterogeneous multi-agent system through the communication interface.

[0052] Specifically, this step involves lightweighting the model and deploying it to the actual intelligent agent. Through a hierarchical instruction architecture, the model sends upper-level action instructions, which are then parsed into lower-level driving signals by the control unit and driven by the execution unit to execute the actions of the intelligent agent.

[0053] Lightweight processing distills complex policy networks into lightweight networks through methods such as model pruning and reducing data precision, thereby reducing the computational requirements for deploying the model and enabling it to run on embedded computing platforms.

[0054] Step S105: After receiving the action policy, the agent control system evaluates and analyzes it. If the current action meets the task instruction requirements, the agent is allowed to execute the current action by default, and the agent drives the underlying execution unit to execute the action. If the current action does not meet the task instruction requirements, the agent control system will send an instruction to stop the current action and require the multi-agent reinforcement learning algorithm to collect all data again and repeat steps S103 to S105 until the action meets the task instruction requirements issued by the agent control system, and then stop the process.

[0055] Thus, this method enables the deployment of policies trained in simulation environments to real agents without additional training. It also provides comprehensive benchmark tests for currently popular reinforcement learning algorithms under different settings, supporting the virtual-to-real transfer of multi-agent reinforcement learning. Compared to existing technologies, this method supports native GPU parallel training during the training phase, allowing thousands of environments to run in parallel within a single thread, significantly improving sampling efficiency, shortening training time, and requiring less hardware performance and storage space.

[0056] Example 2

[0057] Building upon Example 1, this example provides a virtual-to-real migration system based on the Universal Multi-Agent Environment Framework (UniMAEF) to implement the virtual-to-real migration method of Example 1. The system extends the Issac Gym monolithic simulation engine, enabling it to support the simulation and interaction of homogeneous and heterogeneous multi-agent systems; it optimizes the simulation process and data pipeline, ensuring the simulation environment framework conforms to multi-agent reinforcement learning algorithm standards.

[0058] See Figure 2 The virtual-real migration system 1 based on UniMAEF includes a parallel simulation environment unit 2, a multi-agent reinforcement learning unit 3, a policy network unit 4, agents 5, a control unit 6, and an execution unit 7.

[0059] Parallel simulation environment unit 2 provides a high-fidelity parallel physical environment. Its input consists of the actions performed by agent 5, and its output includes the environment state and reward value. Compared to the frequency of agent 5's actions, parallel simulation environment unit 2 performs physical simulation calculations at a higher frequency to provide near-real-world physical feedback. The simulator supports running thousands of environment instances in parallel within a single thread, fully utilizing the parallel computing capabilities of the graphics computing unit for physical simulation, thus providing a foundation for large-scale multi-agent training. Furthermore, parallel simulation environment unit 2 integrates various types of sensors by default, providing agents with a rich observation space.

[0060] Multi-agent reinforcement learning unit 3 is used to train policies in the environment. It integrates various popular multi-agent reinforcement learning algorithms, covering online multi-agent reinforcement learning, offline multi-agent reinforcement learning, single-agent reinforcement learning, and heterogeneous multi-agent reinforcement learning. The input to multi-agent reinforcement learning unit 3 includes local observation information of the agents and environmental states, and the output is the action policy. After the input information is processed by the algorithm, the policy network provides the action policy based on the state at the current time step.

[0061] Model network unit 4 is used to learn a robust policy that can stably perform the task and infer the dynamics of the current environment. The input to policy π includes the current state s. t Task objective g and historical trajectory code h t The output is the action that agent 5 can execute, i.e., π(a t |s t ,h t (g). Model network unit 4 uses a recurrent neural network to implicitly embed historical states and action trajectories into the policy, enabling the policy to infer the dynamic characteristics of the current environment without explicitly specifying the parameters to be identified.

[0062] Specifically, the steps for training a robust policy using the parameter domain randomization method are as follows: In each training round, from the parameter distribution ρ μ The sampling dynamics parameter μ; at time step t within a round, the i-th agent obtains the observation value. Hidden state h t Given the task objective g, the policy network π generates actions. action Execute in a simulation environment parameterized as μ to produce the next state s. t+1 and rewards All agents communicate via the state transition function P(s) t+1 |s t ,a t ) migrate to the next state; empirical data (s t+1 ,s t ,a t ,r t (g, μ) are stored; the policy network parameters are updated using the policy gradient method combined with collected experience, with the goal of updating ρ. μ Maximize the cumulative expected return under the distribution, as shown in formula (1):

[0063]

[0064] Where π is the action policy, T is the maximum time step, γ is the discount factor, and N is the number of agents. The reward for agent i at time step t.

[0065] Specifically, the steps for embedded system identification using a policy network are as follows: At time step t, the policy network receives the current input; the recurrent neural network (RNN) units inside the network compare the current input with the hidden state h from the previous time step. t-1 Combine and update to the current hidden state h t As shown in formula (2):

[0066]

[0067] Wherein, the hidden state h t It implicitly encodes historical trajectory information, characterizing the dynamics of the current environment; Let g be the local observation information of agent i at time t, g be the task objective, and a be the local observation information of agent i at time t. t Let t be the action output by the policy network at time t.

[0068] Specifically, the steps to achieve efficient deployment through model lightweighting are as follows: apply model pruning to remove redundant connections and neurons; secondly, quantize network weights and activation values; distill complex policy networks into lightweight networks; and finally, adjust data precision to reduce storage pressure.

[0069] Specifically, the steps for executing actions through a hierarchical strategy are as follows: the strategy network outputs high-level, platform-independent action commands; the agent control unit parses the high-level commands into specific low-level drive signals such as joint torque, speed, or position commands; and the agent execution unit receives the low-level drive signals and executes the actions.

[0070] Example 3

[0071] Based on Example 1, see Figure 3 This embodiment provides a virtual-to-real migration method for quadruped robots. The quadruped robot is a highly flexible and obstacle-crossing industrial mobile robot with a carbon fiber and titanium alloy composite body; the wheeled robot is a high-speed and highly maneuverable six-wheeled industrial mobile robot with an all-aluminum alloy body; and the tracked robot is a tracked industrial robot with high off-road capability and high load-bearing capacity using high-strength steel track structures. Together, they form a heterogeneous multi-agent system. This embodiment uses this heterogeneous multi-agent system as the agent executing action commands and is implemented based on the technical solution of this invention.

[0072] The specific steps when a heterogeneous multi-agent system performs relevant tasks through UniMAEF include:

[0073] Step S101: Build the virtual simulation platform and the equipment platform respectively, and deploy sensors and collaborative task scenarios based on the general multi-agent environment framework;

[0074] Step S102: Load the heterogeneous multi-agent system model and initialize the environment parameters;

[0075] Step S103: Input the task instructions and the observation data captured by the sensors into the multi-agent reinforcement learning algorithm, and train the policy model and perform simulation.

[0076] Step S104: After compressing the policy model, deploy it to the actual intelligent agent. The policy model transmits upper-layer action instructions to the heterogeneous multi-agent system through the communication interface.

[0077] Step S105: After receiving the action policy, the agent control system evaluates and analyzes it. If the current action meets the task instruction requirements, the agent is allowed to execute the current action by default, and the agent drives the underlying execution unit to execute the action. If the current action does not meet the task instruction requirements, the agent control system will send an instruction to stop the current action and require the multi-agent reinforcement learning algorithm to collect all data again and repeat steps S103 to S105 until the action meets the task instruction requirements issued by the agent control system, and then stop the process.

[0078] Compared with the prior art, the present invention has the following beneficial effects:

[0079] (1) Relying on a high-performance simulation engine, this invention supports running thousands of environments simultaneously in a single thread, and also supports multiple threads, achieving more efficient parallel simulation and providing support for low-cost training.

[0080] (2) This invention is compatible with truly heterogeneous multi-agent systems, supports loading and simulating different types of agents in the same environment, overcomes the technical bottleneck of existing platforms being limited to homogeneous agents, and greatly enhances universality.

[0081] (3) This invention enhances the adaptability of the strategy under diverse dynamic conditions by a wide range of parameter perturbations and global randomization, so that the trained strategy does not need much debugging when deployed to the real system, significantly reducing the dependence on high-precision simulation models and improving the practicality and efficiency of virtual-real transfer.

[0082] (4) This invention introduces a variety of preset tasks, supports custom modifications, and adds a large number of target objects from open source datasets and multimodal sensors, providing benchmark testing functions for various algorithms, so that multi-agent reinforcement learning algorithms can be fully tested.

[0083] Example 4

[0084] Based on Embodiment 1, this embodiment provides an electronic device, including: one or more processors and a memory, wherein the memory stores one or more programs, and the one or more programs include instructions for executing the virtual-real transfer method based on the general multi-agent parallel reinforcement learning framework as described in Embodiment 1.

[0085] like Figure 4At the hardware level, the electronic device includes a processor, internal bus, network interface, memory, and non-volatile memory, and may also include other hardware required for the business operations. The processor reads the corresponding computer program from the non-volatile memory into memory and then runs it to achieve the above-mentioned functions. Figure 1 The method described herein. Of course, in addition to software implementation, this invention does not exclude other implementation methods, such as logic devices or a combination of hardware and software, etc. That is to say, the execution subject of the following processing flow is not limited to each logic unit, but can also be hardware or logic devices.

[0086] Memory may include non-persistent storage in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0087] Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0088] This invention relates to the field of multi-agent reinforcement learning simulation technology, and proposes a virtual-to-real transfer method based on the Universal Multi-Agent Environment Framework (UniMAEF). This method completes policy network model training in a high-fidelity simulation environment, supports the direct transfer and deployment of the policy network model to real agents, and remains stable and efficient under unknown scenarios and dynamic perturbations, thereby significantly improving the applicability and generalization ability of multi-agent reinforcement learning methods. UniMAEF mainly consists of the following three parts: a high-fidelity simulation environment, a multi-agent reinforcement learning algorithm framework, and a policy network model. The specific process is as follows: A parallel simulation environment is built according to the task content, the required agent model is loaded, and environment parameters are randomly initialized. During the training phase, a multi-agent reinforcement learning algorithm is loaded, and data such as the agent's local observations and environmental state are input into the algorithm. The policy network outputs executable action commands, the agent executes the action commands, interacts with the environment, obtains rewards, and updates the environment state. The network model parameters generated during training are saved. During the execution phase, the model is lightweighted and deployed to the agent. The model sends upper-level action commands to the agent through a communication interface. The agent receives the upper-level action commands, the control unit parses the commands into lower-level driving signals, and the execution unit executes the actions according to the lower-level driving signals. This invention achieves efficient parallel simulation, provides benchmark tests for various algorithms, and is compatible with heterogeneous multi-agent systems. The training process uses a randomization method, effectively bridging the gap between simulation and reality, realizing the transfer from virtual environments to real-world scenarios, and improving the generalization ability of multi-agent reinforcement learning.

[0089] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in the present invention, and these modifications or substitutions should all be covered within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A virtual-real transfer learning method based on a general multi-agent parallel reinforcement learning framework, characterized in that, Includes the following steps: Build a parallel simulation environment adapted to the task, load the agent model required by the task, and randomly initialize the environment parameters and agent state; During the training phase, a multi-agent reinforcement learning algorithm is loaded. The agent obtains local observation information from the simulation environment, inputs the local observation information and the environment state into the algorithm, the policy network outputs actions, the agent executes the actions and interacts with the environment to obtain rewards. During training, domain randomization is used to periodically and randomly initialize the environment parameters, and training is carried out in combination with a post-experience replay mechanism. After training is completed, the model parameters are saved. During the execution phase, the model is lightweighted and then deployed to the actual intelligent agent. Through a hierarchical instruction architecture, the model sends upper-level action instructions, which are parsed into lower-level driving signals by the control unit and then driven by the execution unit to execute the actions of the intelligent agent. The process of randomizing the domain includes the following steps: At the beginning of each training round, a set of dynamic parameters are randomly sampled and the terrain parameters are independently randomized and remain unchanged throughout the round; The time step is randomized at preset intervals and overall randomization is performed. The dynamic parameters include agent mass, motor torque and sensor noise, and the terrain parameters include ground friction coefficient and terrain height map.

2. The virtual-real transfer method based on a general multi-agent parallel reinforcement learning framework according to claim 1, characterized in that, The steps for building a parallel simulation environment adapted to the task are as follows: Based on the Issac Gym monolithic simulation engine, the simulation environment is extended to support the simulation and interaction of homogeneous and heterogeneous multi-agent systems, and the simulation process and data pipeline are optimized to adapt to multi-agent reinforcement learning algorithm standards.

3. The virtual-real transfer method based on a general multi-agent parallel reinforcement learning framework according to claim 1, characterized in that, The training phase includes the following embedded system recognition process: Based on dynamic parameters and the generated trajectory, the recognition process is embedded in the policy network using a recurrent neural network. By fusing the current input with the previous hidden state, the current hidden state is updated, thereby realizing the recognition of the agent's own dynamic parameters.

4. The virtual-real transfer method based on a general multi-agent parallel reinforcement learning framework according to claim 1, characterized in that, The post-event experience replay mechanism includes the following steps: By reconstructing the dynamic goal, based on the failed trajectory under the original goal, a mapping function from state to goal is defined to represent the new goal satisfied by the final state, and a binary reward function is used to recalculate the reward for the entire trajectory.

5. The virtual-real transfer method based on a general multi-agent parallel reinforcement learning framework according to claim 1, characterized in that, The lightweighting process includes the following steps: By removing redundant connections and neurons through model pruning, quantizing network weights and activation values, reducing data precision, and distilling complex policy networks into lightweight networks, the network can be adapted to embedded computing platforms.

6. The virtual-real transfer method based on a general multi-agent parallel reinforcement learning framework according to claim 1, characterized in that, The hierarchical instruction architecture includes: The strategy network outputs high-level, platform-independent upper-layer action commands. The intelligent agent control unit parses the upper-layer action commands into lower-layer drive signals. The execution unit executes actions based on the lower-layer drive signals, which include joint torque, speed, and position.

7. The virtual-real transfer method based on a general multi-agent parallel reinforcement learning framework according to claim 1, characterized in that, The parameters of the policy network are updated using the policy gradient method combined with collected experience, with the goal of... Maximize the cumulative expected return under the distribution, where the cumulative expected return is: in, To accumulate expected returns, For action strategy, For the maximum time step, As a discount factor, For the number of agents, For intelligent agents At time step The reward In strategy The expected reward value.

8. An electronic device, characterized in that, include: One or more processors and a memory, wherein the memory stores one or more programs, the one or more programs including instructions for executing the virtual-real transfer method based on the general multi-agent parallel reinforcement learning framework as described in any one of claims 1-7.

9. A computer-readable storage medium, characterized in that, Includes one or more programs executable by one or more processors of an electronic device, said one or more programs including instructions for performing the virtual-to-real transfer method based on the general multi-agent parallel reinforcement learning framework as described in any one of claims 1-7.

Images