A method, system, medium, and device for controlling a dual quadruped tandem assembly

Abstract

The present application relates to the technical field of robot control, and provides a double quadruped series combination control method, system, medium and equipment, comprising: for the front and rear subsystems, respectively acquiring observation data, using respective subsystem actor networks to generate actions suitable for respective subsystems; integrating the actions of the two subsystems into a whole action that can be executed on the combination, and deploying the whole action to the combination for environmental interaction; after environmental interaction, for the front and rear subsystems, respectively updating the observation data, using respective subsystem critic networks to generate state value estimates of respective subsystems, calculating respective overall losses based on the state value estimates, and after updating the respective subsystem actor networks and critic networks according to the overall losses, entering the next round of action generation and environmental interaction. While ensuring overall coordination, the training efficiency and the flexibility of the two subsystems are improved.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of robot control, and particularly relates to a dual-quadruped series combination control method, system, medium and equipment. BACKGROUND

[0002] The statements in this section merely provide background information related to the present application and do not necessarily constitute prior art.

[0003] With the increase in the number of robot legs and feet, especially in multi-legged systems such as dual-quadruped series robots, the complexity of control also increases, bringing great challenges to the design of robot control systems. In these multi-legged robots, especially in large-scale robot systems, traditional centralized control strategies gradually show many limitations, especially in real-time performance, stability and scalability.

[0004] Traditional multi-legged robot control methods usually include two main ways: motion planning methods and model-based control methods.

[0005] Motion planning methods achieve robot motion through predefined gait trajectories, but this method usually has poor adaptability when facing complex environments, lacking sufficient flexibility and robustness.

[0006] Model-based control methods rely on accurate modeling of robot dynamics, although this method can theoretically provide better control performance, but it has high requirements for the accuracy of the robot model and computing resources, especially in real-time control processes, the computational complexity is extremely high, and it is difficult to meet the real-time performance requirements of large-scale robot systems; moreover, in multi-legged robots like dual-quadruped spliced robots, the front and rear legs have independent motion characteristics and dynamic behaviors, and traditional models often have difficulty in capturing these complex motion characteristics at the same time. Therefore, the single model control method has certain limitations in improving the performance of the robot system.

[0007] Reinforcement Learning (RL) as a data-driven learning method has shown great application potential in multi-legged robot control in recent years. Reinforcement Learning can learn complex motion strategies through experience without an accurate model through continuous interaction with the environment. In particular, the Proximal Policy Optimization (PPO) algorithm based on policy gradient has become the mainstream choice in the field of multi-legged robot control due to its convergence, stability and robustness.

[0008] However, the existing reinforcement learning methods still mostly adopt centralized control strategy, that is, all foot ends are controlled by a unified policy network, which simplifies the policy structure, but fails to fully utilize the dynamic independence of the front and rear leg parts when dealing with the complex multi-legged system of the double quadruped tandem robot; since the movement characteristics of the front and rear leg parts are physically different, this control method limits the flexibility and scalability of the overall policy; in addition, the centralized control method makes the dimensions of the observation space and the action space increase dramatically when the robot faces a large-scale robot, which not only makes the policy network become large and complex, but also leads to a decrease in convergence speed during the training process, and even may lead to the failure of training. SUMMARY

[0009] To solve the technical problems in the background art, the present application provides a double quadruped tandem combination control method, system, medium and equipment, two sets of actor-critic multi-layer perception machines are set, the observation data of the subsystems are processed respectively in the training process, the actions of the corresponding subsystems are generated, the actions of the two subsystems are integrated into an overall action, which is deployed to the combination, and then the environment is interacted, and the policy is updated independently, so as to achieve the purpose of distributed control, while ensuring the overall coordination, the training efficiency and the flexibility of the two subsystems are improved.

[0010] To achieve the above purpose, the present application adopts the following technical solutions:

[0011] The first aspect of the present application provides a double quadruped tandem combination control method, which comprises:

[0012] For the front and rear subsystems, respectively acquire observation data, and use the respective subsystem actor network to generate actions suitable for the respective subsystems;

[0013] The actions of the two subsystems are integrated into an overall action that can be executed on the combination, and the overall action is deployed to the combination for environment interaction;

[0014] After the environment interaction, for the front and rear subsystems, respectively update the observation data, use the respective subsystem critic network to generate the state value estimation of the respective subsystem, calculate the overall loss of the respective subsystem based on the state value estimation, and after updating the respective subsystem actor network and critic network according to the overall loss, enter the next round of action generation and environment interaction.

[0015] Further, the overall loss includes policy loss, value loss and entropy loss.

[0016] Further, if the difference between the immediate reward of the rear subsystem and the immediate reward of the front subsystem is greater than a threshold value in the iteration process, the learning rate of the rear subsystem is limited until the difference between the immediate rewards of the two subsystems is within the threshold value.

[0017] Further, for the front subsystem, the observation data comprises the local state of the front subsystem and the global state of the combination.

[0018] Further, for the rear subsystem, the observation data comprises the local state of the rear subsystem and the global state of the combination.

[0019] Further, the global state comprises the speed, direction and position of the combination.

[0020] Further, the local state comprises the joint angle, joint position, instruction and action of the subsystem.

[0021] The second aspect of the present application provides a dual-quadruped series combination control system, comprising:

[0022] An action generation module configured to: for the front and rear subsystems, respectively, obtain observation data, use the respective subsystem actor network to generate an action suitable for the respective subsystem;

[0023] An action integration module configured to: integrate the actions of the two subsystems into an overall action that can be executed on the combination, and deploy the overall action to the combination for environmental interaction;

[0024] A network update module configured to: after environmental interaction, for the front and rear subsystems, respectively, update the observation data, use the respective subsystem critic network to generate a state value estimate of the respective subsystem, based on the state value estimate, calculate the overall loss of the respective subsystem, and after updating the respective subsystem actor network and critic network according to the overall loss, enter the next round of action generation and environmental interaction.

[0025] The third aspect of the present application provides a computer readable storage medium having a computer program stored thereon, the program being executed by a processor to implement the steps of a dual-quadruped series combination control method as described above.

[0026] The fourth aspect of the present application provides a computer device comprising a computer readable storage medium, a processor and a computer program stored on the computer readable storage medium and executable on the processor, the processor executing the program to implement the steps of a dual-quadruped series combination control method as described above.

[0027] Compared with the prior art, the present application has the following advantages:

[0028] The application sets two sets of actor-critic multi-layer perceptron, respectively processes observation data of the subsystem in the training process, generates actions of the corresponding subsystem, integrates the actions of the two subsystems into a whole action, and deploys the whole action to the combination body to interact with the environment, and independently updates the strategy to achieve the purpose of distributed control, while ensuring the overall coordination, the training efficiency and the flexibility of the two subsystems are improved.

[0029] In the application, if the difference between the real-time reward of the latter subsystem and the real-time reward of the former subsystem in the iteration process is greater than a threshold, the learning rate of the latter subsystem is limited, so that the reward values between the subsystems will not differ too much, and the stability of the optimized strategy update is ensured.

[0030] The application distinguishes the reward function, so that the former and latter subsystems can respectively perform local optimization while still maintaining collaborative work consistent with the global goal.

[0031] The application distinguishes the observation space, which ensures that the data of the former and latter subsystems will not interfere with each other, while ensuring the coordination of global information. BRIEF DESCRIPTION OF DRAWINGS

[0032] The drawings accompanying the specification of the application form part of the application and serve to further understand the application, the illustrative embodiments of the application and the description thereof serve to explain the application and do not constitute an improper limitation of the application.

[0033] Figure 1 is a flowchart of a double-quadruped series combination body control method according to an embodiment of the application;

[0034] Figure 2 is a strategy independent optimization flowchart of a reinforcement learning algorithm according to an embodiment of the application;

[0035] Figure 3 is a network structure diagram of an actor-critic multi-layer perceptron of a former subsystem according to an embodiment of the application;

[0036] Figure 4 is a schematic diagram of a double-quadruped series combination body according to an embodiment of the application;

[0037] Figure 5 is a structural schematic diagram of a computer device according to an embodiment of the application. DETAILED DESCRIPTION

[0038] In order to make the purpose, technical scheme and advantages of the embodiments of the application clearer, the technical scheme of the embodiments of the application will be described clearly and completely below with reference to the drawings in the embodiments of the application.

[0039] It should be noted that the following detailed description is illustrative and intended to provide further explanation of the invention. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains.

[0040] Example 1

[0041] This embodiment provides a control method for a dual quadruped tandem assembly.

[0042] For the control requirements of bipedal quadrupedal assembly robots, traditional centralized control methods are facing increasing challenges. Against this backdrop, how to effectively utilize the dynamic independence of the forelegs and hindlegs of multi-legged robots and improve the flexibility and scalability of strategies has become a key issue that urgently needs to be addressed in the field of multi-legged robots.

[0043] This embodiment provides a control method for a bipedal quadrupedal serial robot, which aims to achieve distributed control of the bipedal quadrupedal serial robot through modular strategy design and optimize the robot's motion capabilities in complex environments through reinforcement learning.

[0044] During training, the robot continuously generates new observation data through interaction with the environment. To avoid data interference between different subsystems, the observation data is divided into front subsystem observation data and back subsystem observation data. The front and back subsystem observation data consists of global state and local state: the global state contains shared information, such as velocity, direction, and position; while the local state involves the specific state of each subsystem, such as joint angle, joint position, commands, and actions.

[0045] Corresponding to the classification of the observation space, the reward function is also differentiated based on the preceding and following subsystems, with separate reward functions designed for the preceding and following subsystems. The results of the reward functions are used in the calculation of policy loss and value loss, which in turn contribute to the overall loss. The design of the reward functions allows each subsystem to be optimized independently during training, while simultaneously achieving overall optimization through a shared global objective.

[0046] To manage the data generated during training, this invention defines two independent experience buffers, which are used to store observation data, action data, reward data, and other information generated by the pre-subsystem and post-subsystem during training, respectively. This data will be used to calculate the reward and advantage function, and then used in subsequent policy updates (i.e., using the reward and advantage function to construct the overall loss for updating policy parameters). The existence of the experience buffers ensures that the training data of the pre-subsystem and post-subsystem do not interfere with each other, while also enabling more efficient processing of data during training.

[0047] The focus of this invention is to achieve independent policy optimization. The PPO algorithm is selected for policy optimization, and two optimization modules are designed: a pre-subsystem PPO module and a post-subsystem PPO module. Two actor-commentator multilayer perceptrons are also designed for policy generation. Both modules are used for calculating value loss and pruning the objective function, evaluating the differences between the old and new policies, and dynamically adjusting the learning rate of each subsystem as needed. In this process, the PPO module not only optimizes the network of each subsystem independently but also ensures the stability and convergence of the overall training process. The parameters of the actor-commentator multilayer perceptron in each subsystem are continuously updated during training, thereby improving their respective control performance.

[0048] The policy network generates new actions based on the current fixed policy parameters; after the new actions are used to interact with the environment, a new batch of data is obtained; at this time, the policy parameters are updated by optimizing the policy loss function, and the updated parameters are used for the next round of data collection; this process is continuously looped and iterated, so that the control strategy of the entire bipedal serial robot system is optimized, and finally distributed control is achieved.

[0049] Through a distributed control framework, the front and rear subsystems can independently optimize their respective motion strategies based on sharing global information, thereby significantly improving the robot's performance and control efficiency in complex tasks.

[0050] like Figure 4 As shown, the bipedal quadrupedal tandem assembly comprises two connected subsystems, each of which is a quadrupedal robot.

[0051] This embodiment provides a control method for a dual quadruped tandem assembly, comprising the following steps:

[0052] Step 1: Design the observation space and reward function of the twin system, such as... Figure 1 As shown.

[0053] (1) Classification of observation data.

[0054] During the interaction between a robot and its environment, a large amount of observation data (also known as state) is generated. This data first needs to be classified, and can be divided into two categories: pre-subsystem observation data and post-subsystem observation data. Both pre-subsystem and post-subsystem observation data consist of two parts: the overall global state and the local states of each subsystem. Specifically, pre-subsystem observation data includes the local states of the pre-subsystem and the global state of the overall system (assembly), while post-subsystem observation data includes the local states of the post-subsystem and the global state of the overall system. The global state includes shared robot motion information such as velocity, direction, and position; the local state contains specific information about each subsystem, such as joint angles, joint positions, commands, and actions. This classification method allows the robot to effectively avoid data interference during training and leverages the shared global state to coordinate the behavior of each subsystem, thereby improving training efficiency and the robot's adaptability in multi-task environments.

[0055] Before classifying and processing the observation data, it is necessary to classify and store it. Two experience buffer pools (previous subsystem experience buffer pool and subsequent subsystem experience buffer pool) are designed to classify and store the observation data. The previous subsystem experience buffer pool stores the observation data of the previous subsystem, and the subsequent subsystem experience buffer pool stores the observation data of the subsequent subsystem.

[0056] (2) Classification of reward functions.

[0057] The strategy updates independently based on different rewards; therefore, the reward functions also need to be differentiated. These reward functions are categorized according to subsystems, designed as pre-subsystem reward functions and post-subsystem reward functions. Both consist of two parts: global state rewards and local state rewards. Global state rewards focus on factors related to the coordination and stability of the entire robot system, such as gravity direction and base height. Local state rewards focus on the motion and control performance of each subsystem, such as step frequency, step height, and swing period.

[0058] In the reward function of subsystem i, part is a reward function related to the global state, and part is a reward function related to the local state. The global state reward is the sum of the reward values ​​of the reward functions related to the global state, and the local reward function is the sum of the reward values ​​of the reward functions related to the local state. The global state reward and the local reward function together constitute the immediate reward of the subsystem. .

[0059] The design of the reward function enables each subsystem to optimize independently based on local tasks, while ensuring the overall coordination and stability of the system through global state rewards, thereby achieving efficient robot movement and enabling overall and independent optimization of the subsystems.

[0060] Step 2: Design and independent updating of the actor-critic neural network in the twin system.

[0061] During training, the observation, action, reward, and value data generated by the subsystems, as well as the relevant parameters of the PPO algorithm, are temporarily stored in their respective experience buffer pools for the PPO algorithm to use for network parameter updates.

[0062] In this embodiment, two independent actor-critic neural network multilayer perceptrons are defined: the actor-critic multilayer perceptron for the front subsystem and the actor-critic multilayer perceptron for the back subsystem, for independent updating of the strategies of the two subsystems.

[0063] Two actor-commenter multilayer perceptrons, with identical architectures, process data from two separate experience buffers: the one processing observations from the preceding subsystem is called the preceding subsystem actor-commenter multilayer perceptron, and the one processing observations from the following subsystem is called the following subsystem actor-commenter multilayer perceptron. The preceding subsystem actor-commenter multilayer perceptron is responsible for evaluating the preceding subsystem's state data and generating the next action; the following subsystem actor-commenter multilayer perceptron evaluates the following subsystem's state data and generates the corresponding action.

[0064] Both the pre-subsystem actor-critic multilayer perceptron and the post-subsystem actor-critic multilayer perceptron include a subsystem actor multilayer perceptron and a subsystem critic multilayer perceptron. The inputs to both subsystem actor and subsystem critic multilayer perceptrons are observation data from a single quadruped; the former outputs 12 actions for a single quadruped, while the latter outputs a state value estimate for a single quadruped.

[0065] Taking the actor-critic multilayer perceptron of the former subsystem as an example, such as Figure 3 As shown, this includes a pre-subsystem actor multilayer perceptron and a pre-subsystem critic multilayer perceptron. The pre-subsystem actor multilayer perceptron has three layers: an input layer, a hidden layer, and an output layer. After processing the observation data, the pre-subsystem actor multilayer perceptron outputs the action that the pre-subsystem will perform. The pre-subsystem critic multilayer perceptron also has three layers: an input layer, a hidden layer, and an output layer. After processing the observation data, the pre-subsystem critic multilayer perceptron outputs an estimate of the pre-subsystem's state value.

[0066] The actor-critic multilayer perceptron of the front and rear subsystems processes the observation data of their respective systems and can independently update the parameters of their respective neural networks.

[0067] Step 3: Design an optimization strategy based on reinforcement learning algorithms.

[0068] This invention employs the PPO algorithm to optimize the policy function. By designing a pre-subsystem policy optimization module (pre-subsystem PPO module) and a post-subsystem policy optimization module (post-subsystem PPO module), it achieves independent optimization of both policies. Figure 2 As shown.

[0069] (1) Calculation of action probability: Using the current subsystem actor multilayer perceptron, the action probability is recalculated for each mini-batch of data and compared with the action probability calculated by the old strategy (the subsystem actor multilayer perceptron before the update) to obtain the ratio of the new and old strategies. It outputs the entropy of the action probability; at the same time, it uses the subsystem critic multilayer perceptron to calculate the current state value, obtain the state value estimate, and compare it with the actual reward (cumulative reward) to calculate the value loss (abbreviated as value loss).

[0070] (2) Dynamically adjust the learning rate: In addition to using KL (Kullback-Leible) divergence to adjust the learning rate, the difference in rewards between subsystems is also taken into account. Specifically, if the difference between the immediate reward of the later subsystem and the immediate reward of the earlier subsystem during the iteration process is greater than a threshold, the learning rate of the later subsystem is limited until the reward value between the two systems is lower. The difference is kept within the threshold, so that the reward values ​​between subsystems will not differ too much. By dynamically adjusting the learning rate, the stability of policy updates can be optimized.

[0071] (3) Overall loss calculation: The overall loss includes strategy loss, value loss and entropy loss. Strategy optimization is achieved by optimizing the overall loss. In the following formula, i equals 1 and 2, representing the pre-subsystem and post-subsystem, respectively.

[0072] ① Policy Loss: The policy loss is used to optimize the policy network (subsystem actor network), and the calculation formula is as follows: ;in, Comparison of old and new strategies for the subsystem; It is a hyperparameter of the clipping range; It is the estimation of the advantage function; It represents the immediate reward of the subsystem; clip is a mechanism to limit the magnitude of policy updates and prevent policy updates from being too large.

[0073] The calculation formula is: ;in, Let be the TD (Time Difference) error of the i-th subsystem at time step t. Let be the TD error of the i-th subsystem at time step t+1. Let be the TD error of the i-th subsystem at time step t+2, and so on; For the subsystem discount factor; This is the smoothing factor for the subsystem.

[0074] The calculation formula is: ;in, The instantaneous reward obtained by the i-th subsystem through its interaction with the environment at time step t. Let be the instantaneous reward obtained by the i-th subsystem through interaction with the environment at time step t+1. The instantaneous reward obtained by the i-th subsystem through interaction with the environment at time step t+2; For subsystem state State value estimation under the following conditions; This represents the state of the i-th subsystem at time step t; This represents the state of the i-th subsystem at time step t+1. This represents the state of the i-th subsystem at time step t+2, and so on.

[0075] ② Value Loss: Value loss is used to optimize the value network (subsystem critic network) to more accurately estimate the true value of a state. The calculation formula is as follows: ;in, The cumulative return for the subsystem is calculated using the following formula: .

[0076] By measuring the difference between the value estimate and the actual return, gradient descent is used to minimize... Update the value network parameters.

[0077] ③ Entropy Loss: Entropy loss is used to encourage exploration and prevent the strategy from prematurely converging to a local optimum. The calculation formula is: ;in, ; It is the entropy of the strategy, representing the degree of randomness of the strategy; The current policy for subsystem i, with parameters as follows: ; For the subsystem state; For the subsystem in state The actions taken. It should be noted that a policy is a mapping from states to action distributions, and a policy network is a neural network that implements this mapping. The parameters of the network determine the specific form of the policy.

[0078] The policy optimization process based on reinforcement learning algorithms can be briefly described as follows: Calculate the advantage function estimate. ; using strategy loss Update parameter strategy parameters Gradient descent is used to optimize and limit the range of changes between the old and new policies; by minimizing Update the value network; through entropy loss Adjust the entropy of the strategy; use this batch of data to optimize the strategy network and value network multiple times.

[0079] Step 4: Action generation and deployment based on independent strategies.

[0080] This invention uses a single-strategy runner. By combining the modular methods designed in steps 1, 2 and 3, it can generate the actions of the preceding and following subsystems respectively, and integrate these actions into a control command applicable to the entire quadruped robot, thereby realizing simulation execution. After execution, the obtained reward value and other feedback information are recorded, and a log is output to facilitate subsequent training and parameter tuning.

[0081] The same-policy runner generates pre-subsystem and post-subsystem actions based on continuously updated actor-critic network parameters, and integrates them into a single action that is executed on the robot, generating new observation data, including rewards, end flags, and other feedback information.

[0082] (1) Initialize the PPO module and actor-critic multilayer perceptron module of the front and rear subsystems: In this stage, the PPO module and actor-critic multilayer perceptron module of the front and rear subsystems are initialized and loaded to ensure that the robot can be controlled according to the current strategy during training.

[0083] (2) Generate and integrate subsystem actions: Using the actor multilayer perceptron of the front and rear subsystems (i.e., the actor network of the front subsystem and the actor network of the rear subsystem), the action probability is calculated based on the observation data of each subsystem, and compared with the action probability calculated by the old strategy (the actor network before the update) to obtain the ratio of the new and old strategies. It outputs the entropy of the action probability (entropy is a scalar value derived from the action probability distribution, used to quantify the "randomness" of the strategy); based on the action probability distribution, it selects the action to be executed by the next subsystem (i.e., the action of the previous subsystem and the action of the next subsystem). The same-policy runner integrates the actions of these two subsystems into a whole action that can be executed on the entire eight-legged robot (specifically, it prunes and splices the action tensors of the two subsystems) and deploys it simultaneously to the robot (the combined robot) for environmental interaction.

[0084] (3) Environmental interaction and observation update: After the overall action is deployed to the robot for simulation execution, new observation data is generated and an immediate reward is obtained. This observation data and immediate reward will be used for network update, optimization of strategy and network parameters. At the same time, after obtaining the immediate reward of the subsystem, if the difference between the immediate reward of the subsequent subsystem and the immediate reward of the previous subsystem is greater than a threshold during the iteration process, the learning rate of the subsequent subsystem is limited until the reward value between the two subsystems is lower. The difference is within the threshold.

[0085] Specifically, the pre-subsystem observation data includes the local state of the pre-subsystem and the global state of the overall system; the post-subsystem observation data includes the local state of the post-subsystem and the global state of the overall system; the pre-subsystem experience buffer stores the pre-subsystem observation data and immediate rewards; the post-subsystem experience buffer stores the post-subsystem observation data and immediate rewards; the pre-subsystem critic network processes the pre-subsystem observation data and outputs an estimate of the pre-subsystem state value; the post-subsystem critic network processes the post-subsystem observation data and outputs an estimate of the post-subsystem state value; the pre-subsystem actor network processes the pre-subsystem observation data and outputs the pre-subsystem actions; the post-subsystem... After processing the observation data of the subsequent subsystem, the actor network outputs the subsequent subsystem action. The actions, immediate rewards, and estimated state values ​​of the preceding subsystem are input into the preceding subsystem PPO module to obtain the overall loss of the preceding subsystem (including policy loss, value loss, and entropy loss). The critic network and actor network of the preceding subsystem are updated based on the overall loss of the preceding subsystem. The actions, immediate rewards, and estimated state values ​​of the subsequent subsystem are input into the subsequent subsystem PPO module to obtain the overall loss of the subsequent subsystem (including policy loss, value loss, and entropy loss). The critic network and actor network of the subsequent subsystem are updated based on the overall loss of the subsequent subsystem.

[0086] This embodiment provides a control method for a bipedal quadrupedal tandem assembly. By decomposing the robot into two subsystems (front and rear) for independent training and optimization, the method improves training efficiency and the flexibility of the two subsystems while ensuring overall coordination.

[0087] This embodiment provides a control method for a dual quadruped series assembly that distinguishes the observation space. This distinction ensures that the data of the preceding and following subsystems do not interfere with each other, while also guaranteeing the coordination of global information.

[0088] This embodiment provides a control method for a dual quadruped series combined system that distinguishes reward functions, enabling the front and rear subsystems to perform local optimizations while maintaining coordinated operation consistent with the global objective.

[0089] This embodiment provides a control method for a dual quadruped tandem assembly, in which the data of the front and rear subsystems are stored separately, ensuring complete data independence and avoiding mutual interference, thereby enabling more efficient data processing during the training process.

[0090] This embodiment provides a control method for a dual quadruped series combined system, which focuses on modularizing two sets of PPO algorithm modules. Correspondingly, there are two sets of actor-critic multilayer perceptrons. During the training process, the observation data of the subsystems are processed to generate the actions of the corresponding subsystems, and the policies are updated independently to achieve the purpose of distributed control.

[0091] Example 2

[0092] This embodiment provides a dual quadruped tandem assembly control system, which specifically includes:

[0093] The motion generation module is configured to: acquire observation data for each of the preceding and following subsystems, use the actor network of each subsystem, and generate motions suitable for each subsystem.

[0094] The action integration module is configured to integrate the actions of two subsystems into a single action that can be executed on the assembly, and deploy the single action to the assembly for environmental interaction.

[0095] The network update module is configured to: after environmental interaction, update the observation data for the preceding and following subsystems respectively, use the respective subsystem critic network to generate state value estimates for each subsystem, calculate the overall loss based on the state value estimates, update the respective subsystem actor network and critic network based on the overall loss, and then proceed to the next round of action generation and environmental interaction.

[0096] It should be noted that each module in this embodiment corresponds one-to-one with each step in Embodiment 1, and their specific implementation processes are the same, so they will not be repeated here.

[0097] Example 3

[0098] This embodiment provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the control method for a biquadruple tandem assembly as described in Embodiment 1 above.

[0099] Example 4

[0100] This embodiment provides a computer device, such as... Figure 5 As shown, the system includes a computer-readable storage medium 1003, a processor 1001, a communication interface 1002, and a computer program stored on the computer-readable storage medium 1003 and executable on the processor 1001. The processor 1001, communication interface 1002, and computer-readable storage medium 1003 can be connected via a bus or other means. The communication interface 1002 is used to receive and transmit data. When the processor 1001 executes the program, it implements the steps in the bipedal tandem assembly control method described in Embodiment 1 above.

[0101] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A control method for a dual quadruped tandem assembly, characterized in that, include: Design the observation space and reward function for the twin system; The observation data is classified into two categories: pre-subsystem observation data and post-subsystem observation data. Both pre-subsystem and post-subsystem observation data consist of two parts: the overall global state and the local state of each subsystem. The reward functions are classified according to the subsystem, and designed as pre-subsystem reward functions and post-subsystem reward functions respectively. Design and independent updating of the actor-critic neural network in the twin system; Two independent actor-critic neural network multilayer perceptrons are defined: the actor-critic multilayer perceptron for the first subsystem and the actor-critic multilayer perceptron for the second subsystem, which are used for independent updates of the policies of the two subsystems. The actor-critic multilayer perceptrons for the first and second subsystems process the observation data of their respective systems. Design optimization strategies based on reinforcement learning algorithms; The PPO algorithm is used to optimize the policy function. By designing a pre-subsystem policy optimization module and a post-subsystem policy optimization module, independent optimization of the two policies is achieved. This includes: action probability calculation, dynamic adjustment of the learning rate, and overall loss calculation. The total loss includes strategy loss, value loss, and entropy loss; The policy loss is used to optimize the policy network, and its calculation formula is as follows: ;in, Comparison of old and new strategies for the subsystem; It is a hyperparameter of the clipping range; It is the estimation of the advantage function; It represents the immediate reward of a subsystem; clip is a mechanism to limit the magnitude of policy updates. The value loss is used to optimize the value network, and the calculation formula is as follows: ;in, Accumulated returns for the subsystem; For subsystem state State value estimation under the following conditions; This represents the state of the i-th subsystem at time step t; The formula for calculating the entropy loss is as follows: ; in, ; It is the entropy of the strategy; The current policy for subsystem i, with parameters as follows: ; For the subsystem state; For the subsystem in state The actions taken; Action generation and deployment based on independent strategies; The same-policy runner generates pre-subsystem and post-subsystem actions based on continuously updated actor-critic network parameters, and integrates them into a single action that is executed on the robot, generating new observation data.

2. The control method for a dual quadruped tandem assembly as described in claim 1, characterized in that, If the difference between the immediate reward of the subsequent subsystem and the immediate reward of the preceding subsystem during the iteration process is greater than a threshold, the learning rate of the subsequent subsystem is limited until the difference between the immediate rewards of the two subsystems is within the threshold.

3. The control method for a dual quadruped tandem assembly as described in claim 1, characterized in that, The global state includes the velocity, direction, and position of the composite.

4. The control method for a dual quadruped tandem assembly as described in claim 1, characterized in that, The local state includes the joint angles, joint positions, commands, and actions of the subsystem.

5. A control system for a dual quadruped series assembly, employing a dual quadruped series assembly control method as described in any one of claims 1-4, characterized in that, include: The motion generation module is configured to: acquire observation data for each of the preceding and following subsystems, use the actor network of each subsystem, and generate motions suitable for each subsystem. The action integration module is configured to integrate the actions of two subsystems into a single action that can be executed on the assembly, and deploy the single action to the assembly for environmental interaction. The network update module is configured to: after environmental interaction, update the observation data for the preceding and following subsystems respectively, use the respective subsystem critic network to generate state value estimates for each subsystem, calculate the overall loss based on the state value estimates, update the respective subsystem actor network and critic network based on the overall loss, and then proceed to the next round of action generation and environmental interaction.

6. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the steps in the control method for a biquadruple tandem assembly as described in any one of claims 1-4.

7. A computer device comprising a computer-readable storage medium, a processor, and a computer program stored on the computer-readable storage medium and executable on the processor, characterized in that, When the processor executes the program, it implements the steps in the control method for a biquadruple tandem assembly as described in any one of claims 1-4.

Images