Multi-degree-of-freedom robot collaborative control reinforcement learning method, device, equipment and storage medium

Abstract

The application discloses a multi-freedom-degree mechanical arm and four-legged robot cooperative control reinforcement learning method and device, equipment and storage medium, relates to the field of robot control technology, and comprises the following steps: through the centralized training dispersed execution framework, the multi-agent cooperative strategy training is carried out on the mechanical arm state information, four-legged robot state information and environment state information, and the cooperative control strategy is obtained; based on the cooperative control strategy, the mechanical arm action instruction and the four-legged robot action instruction are generated; the mechanical arm action instruction and the four-legged robot action instruction are sent to the corresponding actuator, so that the mechanical arm and the four-legged robot cooperatively execute the task; the multi-dimensional reward value is determined according to the mechanical arm state information, the four-legged robot state information and the environment state information; the cooperative control strategy is updated according to the multi-dimensional reward value, and the real-time coordination and adaptive ability of the robot system are improved, so as to support stable cooperative work of complex tasks.

Description

Technical Field

[0001] This invention relates to the field of robot control technology, and in particular to a reinforcement learning method, apparatus, device, and storage medium for the cooperative control of a multi-degree-of-freedom robotic arm and a quadruped robot. Background Technology

[0002] With the increasing demands for complex task execution capabilities in fields such as industrial automation, smart warehousing, disaster relief, and home services, single-type robots are no longer sufficient to meet the requirements of diverse operational scenarios. For example, in industrial handling and assembly scenarios, both high-precision operation capabilities and wide-range mobility are required; in disaster relief environments, systems are required to adapt to rugged terrain and perform precise detection and grasping operations; and in the field of service robots, the goal is to achieve autonomous retrieval and delivery functions in dynamic environments.

[0003] However, existing multi-robot collaboration technologies mainly rely on separate teleoperation or control methods based on fixed rules, which have the following drawbacks: The lack of a real-time task coordination mechanism between the robotic arm and the quadruped robot makes it difficult to cope with dynamic environmental changes, resulting in low collaboration efficiency; control strategies based on preset rules cannot adapt to unknown or complex environments, leading to poor system adaptability; traditional reinforcement learning methods are usually designed for single robots and lack cross-robot platform collaboration strategies, making it difficult to achieve joint optimization of multi-degree-of-freedom motion and multi-operation tasks, resulting in high training costs. These problems cause heterogeneous robot systems to easily experience motion conflicts, balance instability, and task failures during collaborative operations, severely restricting their application in real-world scenarios.

[0004] The above content is only used to help understand the technical solution of the present invention and does not represent an admission that the above content is prior art. Summary of the Invention

[0005] The main objective of this invention is to provide a reinforcement learning method, device, equipment, and storage medium for the collaborative control of a multi-degree-of-freedom robotic arm and a quadruped robot, aiming to solve the technical problems of low collaborative efficiency, poor environmental adaptability, and high training costs in dynamic tasks caused by the lack of a unified training framework and real-time coordination mechanism in collaborative systems of multi-degree-of-freedom robotic arms and quadruped robots.

[0006] To achieve the above objectives, this invention provides a reinforcement learning method for cooperative control of a multi-degree-of-freedom robotic arm and a quadruped robot, the method comprising the following steps: By using a centralized training and decentralized execution framework, multi-agent cooperative strategies are trained on the state information of the robotic arm, the quadruped robot, and the environment to obtain a cooperative control strategy. Based on the aforementioned collaborative control strategy, action commands for the robotic arm and quadruped robot are generated. The robotic arm motion commands and the quadruped robot motion commands are sent to the corresponding actuators so that the robotic arm and the quadruped robot can work together to perform the task. During task execution, a multi-dimensional reward value is determined based on the robotic arm status information, the quadruped robot status information, and the environmental status information. The collaborative control strategy is updated based on the multi-dimensional reward value.

[0007] Furthermore, to achieve the above objectives, the present invention also proposes a storage medium storing a reinforcement learning program for the cooperative control of a multi-degree-of-freedom robotic arm and a quadruped robot. When the multi-degree-of-freedom robotic arm and quadruped robot cooperative control reinforcement learning program is executed by a processor, it implements the steps of the multi-degree-of-freedom robotic arm and quadruped robot cooperative control reinforcement learning method described above.

[0008] In addition, to achieve the above objectives, this application also provides a computer program product, which includes a computer program that, when executed by a processor, implements the steps of the reinforcement learning method for cooperative control of a multi-degree-of-freedom robotic arm and a quadruped robot as described above.

[0009] One or more technical solutions proposed in this application have at least the following technical effects: By combining a centralized training and distributed execution framework with a multi-dimensional reward function, end-to-end real-time coordination and adaptive learning of heterogeneous robots in dynamic environments is achieved. This effectively solves the technical problems of low collaborative efficiency, poor environmental adaptability, and high training costs caused by the lack of a unified training mechanism. This scheme enables the robotic arm and quadruped robot to perform joint optimization based on shared reward signals, significantly improving the system's collaborative stability and operational accuracy in complex tasks. Simultaneously, task orientation is enhanced through environmental state information fusion, and domain randomization and online fine-tuning ensure the robustness of the strategy's transfer from simulation to real-world deployment. Attached Figure Description

[0010] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0011] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0012] Figure 1This is a flowchart illustrating an embodiment of the reinforcement learning method for collaborative control of a multi-degree-of-freedom robotic arm and a quadruped robot in this application. Figure 2 This is a structural block diagram provided for Embodiment 2 of the reinforcement learning method for cooperative control of a multi-degree-of-freedom robotic arm and a quadruped robot in this application; Figure 3 This is a schematic diagram of the module structure of the reinforcement learning device for collaborative control of a multi-degree-of-freedom robotic arm and a quadruped robot according to an embodiment of this application. Figure 4 This is a schematic diagram of the equipment structure of the hardware operating environment involved in the reinforcement learning method for collaborative control of a multi-degree-of-freedom robotic arm and a quadruped robot in the embodiments of this application.

[0013] The purpose, features, and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0014] It should be understood that the specific embodiments described herein are merely illustrative of the technical solutions of this application and are not intended to limit this application.

[0015] To better understand the technical solution of this application, a detailed description will be provided below in conjunction with the accompanying drawings and specific implementation methods.

[0016] It should be noted that the executing entity in this embodiment can be a computing service device with data processing, network communication, and program execution functions, such as a tablet computer, personal computer, or mobile phone, or an electronic device capable of performing the above functions, a multi-degree-of-freedom robotic arm and quadruped robot collaborative control reinforcement learning device, etc. The following description uses a multi-degree-of-freedom robotic arm and quadruped robot collaborative control reinforcement learning device as an example to illustrate this embodiment and the following embodiments.

[0017] Based on this, embodiments of this application provide a reinforcement learning method for collaborative control of a multi-degree-of-freedom robotic arm and a quadruped robot, referring to... Figure 1 , Figure 1 This is a flowchart illustrating the first embodiment of the reinforcement learning method for collaborative control of a multi-degree-of-freedom robotic arm and a quadruped robot according to this application.

[0018] In this embodiment, the reinforcement learning method for collaborative control of the multi-degree-of-freedom robotic arm and quadruped robot includes steps S10 to S50: Step S10: Train the multi-agent cooperative strategy by using a centralized training distributed execution framework to train the state information of the robotic arm, the state information of the quadruped robot, and the state information of the environment, and obtain the cooperative control strategy. It should be noted that the purpose of this step is to build a unified training architecture for heterogeneous robots and achieve joint strategy optimization between the robotic arm and the quadruped platform under shared task objectives.

[0019] The centralized training and distributed execution framework refers to a hybrid paradigm in which a central controller gathers the states of each agent and evaluates the value of joint actions during the training phase, while each agent independently makes decisions using local observations during the execution phase. Multi-agent cooperative policy training refers to the synchronous updating of multiple policy networks through reinforcement learning algorithms to maximize the sharing of reward functions. Cooperative control policy refers to two neural network models generated after training, which can be deployed on the robotic arm controller and the quadruped robot controller, respectively.

[0020] Understandably, by inputting three types of state information into a centralized evaluation network to calculate the joint action advantage value, and using a decentralized policy network to generate action sampling distributions for the robotic arm and the quadruped robot respectively, after multiple rounds of iterative optimization, a set of policy network parameters that can coordinate the behavior of the two is obtained, which serves as the cooperative control strategy for subsequent deployment.

[0021] It should be understood that the robotic arm and the quadruped robot are treated as different intelligent agents, using a centralized training and distributed execution (CTDE) strategy. Each agent generates action strategies based on local observations, while sharing task reward signals to achieve collaborative learning.

[0022] In this specific implementation, the embodiment targets the "handling and placement" task in an industrial environment. The system includes a quadrupedal mobile platform and a six-DOF robotic arm, unitreeZ1. The robotic arm is fixed to the back of the quadrupedal robot and achieves state sharing via high-speed Ethernet.

[0023] Reinforcement learning systems employ multimodal state inputs: The states of the four legs include: base angular velocity, gravity projection, joint position, joint velocity, and the previous movement; The states of the robotic arm include: joint position, joint speed, and gripper force; Environmental conditions include: the location of the target object and the obstacle map.

[0024] The action space is a joint control vector:

[0025] in, For the 12-dimensional expected joint angles of the quadruped robot, The desired joint angles in 6 dimensions for the robotic arm.

[0026] The training process includes: The system initializes from the simulation environment (Isaac Gym / Mujoco); the quadruped robot moves to the designated area; the robotic arm stably controls the end effector posture on the motion platform to achieve dynamic grasping; the reinforcement learning module performs a reward function... The algorithm updates the policy; enhances generalization ability through domain randomization and perturbation; and transfers the trained policy to a real robot. In the handling task, the robotic arm needs to grasp objects while in a quadrupedal walking state. This algorithm utilizes a cooperative reward system. By ensuring that the acceleration of the robotic arm and legs is consistent, the overall system's center of gravity is balanced, reducing platform sway and improving the success rate of grasping.

[0027] Experimental results show that, under the strategy of collaborative training, the walking stability of the quadruped robot is improved by about 35%, the grasping success rate is improved by about 27%, and the average energy consumption of the task is reduced by 18%.

[0028] In one feasible implementation, step S10 includes steps A11 to A13: A11: Multi-agent cooperative strategy training is performed based on the state information of the robotic arm, the quadruped robot and the environment, and various disturbance factors are introduced during the training process to obtain an enhanced cooperative control strategy. It should be noted that the purpose of this step is to improve the robustness of the collaborative control strategy to real-world uncertainties and avoid performance crashes when migrating from simulation to reality.

[0029] The simulation environment refers to a virtual training platform built in a computer that includes a physics engine, sensor models, and terrain scenes; multiple perturbation factors include four types of randomized parameters: sensor noise and delay, changes in ground friction coefficient, joint friction and drive error, and camera attitude perturbation; the enhanced cooperative control strategy refers to a policy network that is tolerant of parameter deviations after domain randomization training.

[0030] Understandably, by applying random perturbations to the physical parameters and observation data of the simulation environment during training, the agent can explore and learn general policies under diverse conditions. After millions of training steps, a policy model with perturbation resistance is derived, achieving a smooth transfer from simulation to reality.

[0031] A12: Deploy the enhanced collaborative control strategy to real robot systems; It should be noted that the purpose of this step is to transform the training results in the virtual environment into decision logic that can be executed by real hardware.

[0032] A real robot system refers to a hardware platform consisting of a physical robotic arm, a quadruped robot body, actuators, and sensors; strategy deployment refers to loading the neural network weight file obtained from simulation training into the computing unit of the robot controller.

[0033] Understandably, through format conversion and interface adaptation, the enhanced collaborative control strategy is converted into a model format supported by the embedded system and burned into the memory of the robotic arm controller and the quadruped robot motion control board, thus completing the hardware installation of the control algorithm.

[0034] A13: During the operation of a real robot system, the collaborative control strategy can be further optimized through online fine-tuning.

[0035] It should be noted that the purpose of this step is to compensate for the residual deviation between simulation modeling errors and actual dynamic characteristics.

[0036] Online fine-tuning refers to using actual collected data to perform lightweight gradient updates on the policy network during real system operation; residual differences refer to physical characteristics that cannot be fully reproduced through simulation, such as unmodeled joint flexibility, transmission clearance, and ground deformation.

[0037] It is understandable that by periodically collecting state transition samples during real task execution, the top-level parameters of the policy network can be incrementally adjusted with a low learning rate, so that the control policy can gradually adapt to the unique dynamic characteristics of the hardware and maintain long-term operational stability.

[0038] It should be understood that this strategy employs an Actor-Critic architecture neural network, supporting PPO or SAC. It supports Sim2Real transfer learning and can be deployed to real-world robotic systems after simulation training.

[0039] Step S20: Based on the cooperative control strategy, generate motion instructions for the robotic arm and the quadruped robot; It should be noted that the purpose of this step is to translate high-level strategy decisions into motion instructions that can be recognized by the low-level actuators.

[0040] The motion commands for a robotic arm refer to a vector sequence containing the target angle, angular velocity, or torque of each joint; the motion commands for a quadruped robot refer to a set of control signals containing the leg swing phase sequence, the target position of the foot end, or the joint driving force.

[0041] Understandably, by inputting the current state of the robotic arm, the quadruped robot, and the environment into the deployed cooperative control strategy, and after calculation via neural network forward propagation, the action vectors within discrete time steps are output respectively, and the final command is generated after amplitude limiting and safety checks.

[0042] In this specific implementation, the example is used for target search tasks in complex terrain. A quadruped robot is used for motion and environmental perception, while the robotic arm carries a depth camera for high-precision local detection.

[0043] The system is implemented using a multi-task reinforcement learning framework:

[0044] in, As a reward for walking stability; For path tracking accuracy; For environmental coverage; Rewards are used to guide the robotic arm.

[0045] When the quadruped robot moves on uneven ground, the robotic arm automatically adjusts the camera's orientation to maintain a stable field of view. and The joint optimization minimizes the impact of the robotic arm's movement on chassis disturbance. In a simulated ruin scenario, the system can complete a scan of a 20-square-meter area within 10 minutes, with a positioning error of less than 5 cm, far superior to independent control baselines.

[0046] In home or hotel environments, quadruped robots equipped with robotic arms perform "retrieval-delivery" tasks. The quadruped platform handles path planning and obstacle avoidance, while the robotic arm performs grasping and placement operations. Dynamic task allocation is achieved through a joint policy network.

[0047] The joint strategy structure is as follows:

[0048] The system is trained in 4096 parallel environments in end-to-end simulation and the PPO algorithm is used to update the strategy.

[0049] Introducing a hierarchical reward function structure:

[0050] in, To ensure stable movement on all fours; To improve the accuracy of data capture; To reduce the impact of robotic arm movements on chassis posture.

[0051] Ultimately, the system can pick up and deliver a 0.5m high desktop object within 2 seconds, with smooth and jitter-free operation.

[0052] On the assembly line, quadruped robots transport parts, while robotic arms perform assembly actions. Both share the global task objective state and achieve joint optimization of "delivery + assembly" through collaborative reinforcement learning.

[0053] Key innovations include the introduction of a collaborative stability term. Reduce chassis disturbance during robotic arm operation; through dynamic consistency terms Limit energy consumption; use a parallel training architecture to accelerate multi-condition learning.

[0054] On average, assembly time was reduced by 23% and system power fluctuation was reduced by 30% across 20 different component tasks.

[0055] To address the simulation-to-real discrepancy, this invention proposes a migration mechanism based on domain randomization and adaptive control.

[0056] The simulation incorporates disturbances such as sensor noise and delay, changes in ground friction coefficient, joint friction and drive error, and camera attitude perturbation. Rapid adaptation is achieved through periodic online fine-tuning. Experimental results show that performance degradation after migration is less than 10%, demonstrating robust end-to-end collaborative control.

[0057] Step S30: Send the robotic arm motion command and the quadruped robot motion command to the corresponding actuators so that the robotic arm and the quadruped robot can work together to perform the task. It should be noted that the purpose of this step is to establish an information link between the decision-making and execution levels, so as to enable the physical implementation of control commands.

[0058] The corresponding actuators refer to the servo drives of the robotic arm joints and the motors or hydraulic drive units of the quadruped robot's legs; the cooperative execution task refers to the quadruped robot adjusting its posture to maintain platform stability when the robotic arm performs grasping / placing operations.

[0059] Understandably, the robotic arm's motion commands are sent to the arm controller via a real-time communication bus, and the quadruped robot's motion commands are sent to the motion controller. After receiving the commands, the driver adjusts the output torque according to the control cycle, so that the two robot bodies move synchronously to achieve a common goal.

[0060] Step S40: During task execution, determine multi-dimensional reward values ​​based on the robotic arm status information, quadruped robot status information, and environmental status information; It should be noted that the purpose of this step is to quantitatively evaluate the merits of the current actions and provide monitoring signals for strategy optimization.

[0061] The comprehensive score, quantified by multi-dimensional reward values, is composed of seven weighted sub-rewards: basic stability reward measures the balance of the quadrupedal base, dynamic constraint reward penalizes joint overload and abrupt movement, contact and collision reward constrains safe interaction, trajectory tracking reward evaluates the accuracy of speed command following, gait reward optimizes foot-ground contact pattern, robotic arm operation reward evaluates the accuracy of grasping posture, and coordination reward measures the coordination of arm-foot movements.

[0062] Understandably, by analyzing the three types of state information in real time, substituting them into a preset reward function to calculate the score of each sub-item, and then outputting a scalar reward value after weighted summation, the higher the value, the more the current behavior conforms to the task objective and safety constraints.

[0063] Step S50: Update the collaborative control strategy based on the multi-dimensional reward values.

[0064] It should be noted that the purpose of this step is to use reward feedback to drive the strategy to continuously evolve and form a closed-loop optimization mechanism.

[0065] Policy update refers to applying gradient adjustments to the neural network parameters of the collaborative control policy; closed loop refers to the iterative process of "execution-observation-reward-update".

[0066] Understandably, when the multi-dimensional reward values ​​are fed back to the training module, the probability ratio between the new and old policies is calculated using the proximal policy optimization algorithm. The loss function is constructed by combining the reward advantage value, and the policy network weights of the robotic arm and the quadruped robot are updated through backpropagation, so that subsequent actions tend to be more inclined to high-reward behaviors. After multiple iterations, the policy performance is gradually improved.

[0067] It should be understood that this solution is applicable to multi-robot collaborative scenarios such as industrial handling, warehouse sorting, handling in complex outdoor environments, and disaster relief.

[0068] This embodiment provides a reinforcement learning method for cooperative control of a multi-degree-of-freedom robotic arm and a quadruped robot. By using a centralized training and distributed execution framework to uniformly model the states of heterogeneous robots, the policy networks of the robotic arm and quadruped robot complete joint optimization under the drive of shared reward signals. This eliminates the bottleneck in cooperative efficiency caused by communication delays and manual rule design in traditional separate control. The cooperative reward term in the multi-dimensional reward function incorporates the deviation between the arm's end-effector acceleration and leg motion acceleration, as well as the relative position of the system's centroid and the supporting polygon, into the evaluation system. This forces the policy to actively balance the disturbances introduced by the robotic arm's dynamic operations with the stability requirements of the quadruped posture during the learning process, thereby suppressing the base sway amplitude during rapid grasping and improving overall stability. The multi-source disturbances introduced during the training phase, such as sensor noise, friction coefficient, and joint errors, cause the policy parameters to traverse a sufficiently large parameter space in the simulation environment. Combined with an online fine-tuning mechanism, residual model errors in the real system are continuously compensated, forming a closed loop of "simulation pre-training - reality adaptation".

[0069] Based on the first embodiment of this application, in the second embodiment of this application, the content that is the same as or similar to that in Embodiment 1 above can be referred to the above description, and will not be repeated hereafter. Based on this, please refer to... Figure 2 Step S40 includes steps S401 to S403: Step S401: Determine the basic stability reward value, dynamic constraint reward value, contact and collision reward value, trajectory tracking reward value, and gait reward value based on the quadruped robot's state information; It should be noted that the purpose of this step is to construct a comprehensive evaluation system for the motion performance of quadruped robots. Five independent reward values ​​are generated through decomposition calculation, which respectively constrain stability, safety, command tracking and gait quality, laying the underlying control foundation for heterogeneous collaboration.

[0070] A quadruped robot refers to a platform robot with four legs that can move autonomously; motion performance refers to the robot's comprehensive performance in terms of speed, stability, and energy consumption; the evaluation system refers to a quantitative evaluation framework composed of multiple reward functions; the basic stability reward value is used to measure the degree of body balance; the dynamic constraint reward value is used to limit joint torque and acceleration; the contact and collision reward value is used to avoid dangerous physical contact; the trajectory tracking reward value is used to evaluate the accuracy of speed command following; the gait reward value is used to optimize the foot-ground interaction mode; the multi-dimensional reward value refers to the comprehensive score obtained by subsequent weighted summation.

[0071] This step periodically reads the quadruped robot's state information during task execution, and outputs five scalar reward values ​​after three sets of parallel processing flows, which serve as an important component of the multi-dimensional reward value.

[0072] In the specific implementation, the reinforcement learning reward function (quadruped robot + robotic arm collaboration) uses the following formula to calculate the total reward:

[0073] in, For each sub-reward item, For the corresponding weights.

[0074] The formula for the basic stability bonus is as follows:

[0075] in, The vertical linear velocity of the robot base; The angular velocity components of the base in the x,y plane; The projection of the gravity vector onto the x,y plane is used to evaluate attitude stability. This represents the current average height of the robot's base. The desired target height; The vertical coordinate of the body; The average height of the robot dog's supporting legs; The formula for the dynamic constraint term is as follows:

[0076] in, That is, joint torque; That is, the joint angle; That is, joint angular velocity; That is, the control period dt; That is, the output action at the current moment.

[0077] The formulas for rewards related to contact and collision are as follows:

[0078] in, That is, the contact force of the j-th leg; That is, the contact force threshold; That is, the upper and lower limits of the joint angle; That is, the maximum permissible torque of the joint; That is, the soft constraint coefficient.

[0079] The formula for trajectory tracking and behavioral reward is as follows:

[0080] in, That is, the desired planar linear velocity command; That is, the robot's actual planar linear velocity; That is, the target angular velocity; That is, the actual angular velocity; The formula for the speed-matching weight parameters gait and ground interaction reward is as follows:

[0081] in, That is, the time the j-th leg is in the air; That is, the planar and vertical components of the ground reaction force; That is, the default attitude angle when stationary; The formula for the maximum permissible contact force of the robotic arm is as follows:

[0082] in, That is, the position and rotation matrix of the end effector; That is, the desired grasping position and posture; That is, the current force of the gripper. That is, whether the capture was successful; That is, the capture error weighting coefficient.

[0083] The rewards for collaboration between the robotic arm and the quadruped robot are as follows:

[0084] in, That is, the end effector acceleration of the robotic arm and leg. That is, the synergy strength weight That is, the location of the system's center of mass. That is, supporting the geometric center of the polygon That is, the torque of the joints of the robotic arm and the legs.

[0085] In one feasible implementation, step S401 includes steps A21 to A26: A21: Based on the quadruped robot's state information, obtain the vertical linear velocity of the quadruped robot's base, the horizontal angular velocity of the base, the projection of the gravity vector onto the horizontal plane, the current average height of the base, and the target height of the base; It should be noted that the purpose of this sub-step is to extract the kinematic features of the base from the raw sensor data, in order to prepare input data for subsequent stability calculations.

[0086] The vertical linear velocity of the base refers to the rate of motion of the robot's torso center along the direction of gravity, obtained by integrating accelerometer data in the inertial measurement unit (IMU); the IMU is a sensor module integrating accelerometers and gyroscopes; the horizontal angular velocity of the base refers to the rotational angular rate of the torso around the front-back and left-right axes, directly measured by the gyroscopes; the gravity vector refers to the standard gravitational acceleration vector; the horizontal projection refers to the component of the spatial vector mapped onto the horizontal plane; the current average base height refers to the vertical distance from the geometric center of the robot body to the ground, calculated by subtracting the average foot height of the supporting legs from the height of the robot body root; the target base height is the preset desired height above the ground, set by the task requirements; coordinate transformation refers to converting vector representations between different coordinate systems.

[0087] Understandably, by accessing the IMU data, joint encoder data, and foot force sensor data in the quadruped robot's state information, and after coordinate transformation and numerical calculation, the above five physical quantities are extracted as inputs for sub-steps A22 to A25.

[0088] A22: Determine the vertical velocity penalty term based on the vertical linear velocity of the base; It should be noted that the purpose of this step is to punish any unintended movement of the base in the vertical direction and maintain the stability of the fuselage.

[0089] The vertical velocity penalty term is a negative reward component used to suppress vertical motion; unintended motion refers to passive displacement not required by instructions; vertical jitter refers to high-frequency up-and-down bouncing; high-frequency oscillation refers to high-frequency reciprocating motion; a monotonically decreasing negative function is a function whose output decreases as the input increases, and the output value is always non-positive. This penalty term is usually designed as a negative correlation function of the vertical linear velocity magnitude of the base, mapping small speeds to small penalties. During violent bounces, the penalty amplitude increases rapidly, forcing the strategy to suppress vertical jitter. This penalty term is sensitive to high-frequency oscillations and can effectively filter out bumps during walking.

[0090] By substituting the vertical linear velocity of the base extracted by A21 into a monotonically decreasing negative function, a non-positive scalar value is output. The more negative the value, the more intense the vertical motion and the stronger the negative feedback to the strategy.

[0091] A23: Determine the angular velocity penalty term based on the angular velocity of the horizontal surface of the base; It should be noted that the purpose of this step is to punish the rotational disturbance of the base in the horizontal plane and maintain the horizontal attitude of the fuselage.

[0092] The angular velocity penalty term is used to suppress the negative reward component of rotational motion; rotational disturbance refers to non-commanded body rotation; roll refers to rotation about the front-to-back axis; pitch refers to rotation about the left-to-right axis; attitude change refers to a sudden change in attitude angle. This penalty term is typically designed as a negative correlation function of the angular velocity modulus of the base's horizontal plane, suppressing roll and pitch angular velocities to prevent the robot from swaying due to arm movement or uneven terrain. This penalty term is sensitive to attitude changes, ensuring the base remains stable during dynamic operation.

[0093] By substituting the angular velocity of the base horizontal plane extracted by A21 into the negative function, a non-positive scalar value is output. The more negative the value, the more violent the rotation, and the more the strategy tends to generate smooth motion.

[0094] A24: Determine the attitude stability penalty term based on the projection of the gravity vector onto the horizontal plane; It should be noted that the purpose of this step is to punish the tilting attitude of the fuselage and ensure that the base remains vertical in the direction of gravity.

[0095] The attitude stability penalty term is a negative reward component used to correct tilt; fuselage tilt refers to the angle at which the torso deviates from the direction of gravity. This penalty term is typically designed as a negative correlation function of the magnitude of the gravity vector projected onto the horizontal plane. When the projection is zero, the fuselage is perfectly horizontal, and the penalty is minimal; when the fuselage tilts, the projection magnitude increases, and the penalty magnitude increases accordingly, driving the policy to restore a horizontal attitude.

[0096] This penalty term provides sensitive feedback on the fuselage angle, especially applying a strong negative excitation at large tilt angles. By substituting the gravity projection modulus extracted from the A21 into the negative function for calculation, a non-positive scalar value is output, enabling a quantitative assessment of attitude tilt.

[0097] A25: Determine the height deviation penalty based on the current average height of the base and the target height of the base; It should be noted that the purpose of this step is to maintain the robot body at a preset height during operation, preventing it from being too high or too low from affecting the operating range of the robotic arm or the stability of its four legs.

[0098] The height deviation penalty term is a negative reward component used to constrain the height above the ground; the target base height is the desired reference height; the desired height above the ground refers to the designed working height; and the workspace refers to the three-dimensional area accessible to the robotic arm. This penalty term is designed as a negative correlation function between the deviation between the current average base height and the target base height. When the actual height deviates from the target height, a penalty is incurred regardless of whether it is positive or negative, forcing the strategy to stabilize the base at the desired height.

[0099] This penalty ensures the robotic arm has a stable workspace while preventing overextension or retraction of the quadruped legs. The deviation function is calculated by substituting the current average base height extracted by A21 with the target base height, outputting a non-positive scalar value that is optimal when the height deviation is zero.

[0100] A26: The basic stability bonus value is obtained by summing the vertical velocity penalty, angular velocity penalty, attitude stability penalty, and height deviation penalty.

[0101] It should be noted that the purpose of this sub-step is to integrate the four types of stability penalties into a single basic stability reward value.

[0102] Summation refers to the algebraic addition of multiple values; weighted summation refers to addition with coefficients; scalar signal refers to a single numerical signal; algebraic addition refers to numerical addition without considering units; weight coefficients are used to adjust the importance of different terms. This integration process unifies physical constraints of different dimensions into scalar signals, making them easier for reinforcement learning algorithms to process.

[0103] By performing a weighted summation calculation, a scalar is output as a basic stability reward value. The closer this value is to zero, the more stable the quadrupedal base is, providing a clear stability optimization objective for the strategy.

[0104] In one feasible implementation, step S401 further includes steps A31 to A36: A31: Based on the quadruped robot's state information, obtain the quadruped robot's joint torque, joint angular velocity, joint angular acceleration, current output action, previous output action, contact force of each leg, joint angle, and preset joint angle range; It should be noted that the purpose of this sub-step is to extract the joint and contact force data of the quadruped robot, in order to prepare input for the calculation of dynamic constraints and safety boundaries.

[0105] Joint torque refers to the rotational torque output by each joint actuator, estimated by a motor current sensor; the motor current sensor is a device that measures motor current and calculates torque; joint angular velocity refers to the angular rate of joint rotation, calculated differentially by an encoder; the encoder is a sensor that measures joint angles; joint angular acceleration is the time-domain derivative of angular velocity, reflecting motion smoothness; the time-domain derivative refers to differentiating with respect to time; output action refers to the control command generated by the strategy network; action change rate refers to the difference in action between adjacent moments; a six-dimensional force sensor is a sensor that measures three-dimensional force and three-dimensional torque; driving torque refers to the torque output by the actuator.

[0106] By accessing the joint actuator data, encoder data, and force sensor data in the quadruped robot's state information, the above eight physical quantities are extracted and used as inputs for sub-steps A32 to A35.

[0107] A32: Determine the dynamic constraint reward value based on joint torque, joint angular velocity, and joint angular acceleration; It should be noted that the purpose of this sub-step is to punish high-energy-consuming and high-impact dynamic behaviors, extend hardware lifespan, and improve motion smoothness.

[0108] The dynamic constraint reward value is a negative reward that comprehensively evaluates the dynamic costs; high energy consumption refers to excessive motor power consumption; high impact refers to mechanical shock caused by sudden acceleration changes; hardware lifespan refers to the duration for which the equipment can operate safely; motion smoothness refers to the smoothness of the motion trajectory; and the negative correlation combination function refers to the combination of multiple negative components. This penalty term is typically designed as a negative correlation combination function of the joint torque magnitude, joint angular velocity magnitude, joint angular acceleration magnitude, and motion change rate magnitude, constraining motion intensity from four aspects to prevent motor overload, motion overshoot, and sudden command changes. This penalty term is sensitive to high-frequency oscillations and violent motions, forcing the strategy to generate smooth and energy-efficient trajectories.

[0109] By substituting the torque, angular velocity, angular acceleration, and motion change rate extracted from A31 into the combination function, a non-positive scalar value is output. The more negative the value, the higher the dynamic cost. The strategy tends to generate smooth motions with low energy consumption and low impact.

[0110] A33: Generate a velocity over-limit penalty term when the joint angular velocity exceeds the angular velocity constraint threshold; It should be noted that the purpose of this step is to prevent the joint speed from exceeding the physical limit, thereby avoiding damage to the motor or reducer.

[0111] The angular velocity constraint threshold is the upper limit of the maximum permissible angular velocity; the physical limit refers to the physical load-bearing capacity of the hardware; the reducer is a transmission device that reduces motor speed and increases torque; the speed limit violation penalty is a negative reward when the angular velocity exceeds the limit. The angular velocity constraint threshold is the maximum permissible angular velocity of the joint, calculated from the motor's rated speed and the reduction ratio. The speed limit violation penalty is generated when the angular velocity exceeds this threshold, and is usually designed as a negative correlation function of the violation amplitude, with stronger penalties for more violations.

[0112] By comparing the joint angular velocity extracted by A31 with a preset threshold, a penalty value is calculated and output when an excess is detected; otherwise, zero is output. This penalty term sets a speed red line for the strategy, ensuring that the learning process is carried out within the physically feasible range.

[0113] A34: Generate a torque over-limit penalty term when the joint torque exceeds the torque constraint threshold; It should be noted that the purpose of this step is to prevent the joint output torque from exceeding the driving capacity, thereby avoiding motor overheating or stalling.

[0114] The torque constraint threshold is the maximum allowable torque limit; drive capability refers to the motor's ability to output torque; motor overheating refers to the motor temperature exceeding the rated value; stalled rotor refers to the motor's inability to rotate; the torque over-limit penalty is a negative reward when the torque exceeds the limit. The torque constraint threshold is the maximum allowable torque of the joint, determined by the motor's peak torque. The torque over-limit penalty is generated when the torque exceeds this threshold, using a negative correlation function; the greater the over-limit, the stronger the penalty.

[0115] By comparing the joint torque extracted by A31 with a preset threshold, a corresponding penalty value is generated when the limit is exceeded. This penalty item ensures that the policy learning process does not exceed the hardware driving capability, providing a security guarantee for real deployment.

[0116] A35: A collision penalty is generated when the magnitude of the contact force exceeds the preset contact force threshold, and an angle over-limit penalty is generated when the joint angle exceeds the preset joint angle range. It should be noted that the purpose of this procedure is to prevent the feet from coming into dangerous contact with the environment or for joint movement to exceed mechanical limits.

[0117] The preset contact force threshold is the safe upper limit of contact force; mechanical limit refers to the physical boundary of joint movement; collision penalty is the negative reward when the contact force exceeds the limit; angle over-limit penalty is the negative reward when the joint angle exceeds the limit. The preset contact force threshold is the safe upper limit of foot impact force; the preset joint angle range is the physical boundary of joint movement.

[0118] A collision penalty term is generated when the contact force magnitude exceeds a threshold, and an angle over-limit penalty term is generated when the joint angle exceeds the range. Both are negative values, and their magnitudes are positively correlated with the degree of over-limit. By comparing the contact force and joint angle extracted by A31 with the corresponding safety thresholds, and calculating the penalty value upon triggering, this step ensures motion safety from both the external environment and internal structure perspectives, preventing training exploration from damaging the hardware.

[0119] A36: Integrate the speed limit violation penalty, torque limit violation penalty, collision penalty, and angle limit violation penalty to obtain the contact and collision reward value.

[0120] It should be noted that the purpose of this sub-step is to integrate dynamics and safety constraints to form a unified contact and collision reward value.

[0121] Contact and collision reward values ​​are scalars that comprehensively assess sports safety; integrated operation refers to aggregating multiple penalty items.

[0122] This integration aggregates disparate safety constraints into a single signal, facilitating unified processing by the policy network. By performing a weighted summation, it outputs contact and collision reward values. The closer this value is to zero, the safer and more compliant the action, providing a supervisory signal for policy safety optimization.

[0123] In one feasible implementation, step S401 further includes steps A41 to A49: A41: Based on the quadruped robot's state information, obtain the desired planar linear velocity command, actual planar linear velocity, target angular velocity, actual angular velocity, time in the air for each leg, planar component of ground reaction force, and stationary state flag; It should be noted that the purpose of this sub-step is to extract the quadruped robot's motion commands and foot-ground interaction data, in order to prepare input for trajectory tracking and gait quality evaluation.

[0124] The desired planar linear velocity command is generated by the task planning layer, including forward and lateral velocities; the task planning layer is the module that generates higher-level motion commands; the actual planar linear velocity is calculated by the body state estimation module; the body state estimation module is an algorithm that fuses sensor data to estimate the robot's state; the target angular velocity command and its actual value are the desired and measured spin rates, respectively; the time each leg is in the air refers to the duration of a single leg being off the ground, determined by the contact force threshold detection; the planar component of the ground reaction force is the projection of the contact force onto the horizontal plane; the stationary state flag is a Boolean value, set to true when the velocity command is below the threshold; the maximum allowable contact force is the upper limit of gait impact.

[0125] By accessing the instruction cache, odometer data, contact force sensor data, and state machine flags in the quadruped robot's state information, the above eight physical quantities are extracted as inputs for sub-steps A42 to A48.

[0126] A42: Determine the linear velocity deviation based on the desired planar linear velocity command and the actual planar linear velocity; It should be noted that the purpose of this sub-step is to quantify the tracking error of the quadruped robot to the linear velocity command, as the basis for calculating the trajectory tracking reward.

[0127] Linear velocity deviation is the Euclidean distance between the desired planar linear velocity command and the actual planar linear velocity; Euclidean distance refers to the straight-line distance between two points in space. This deviation reflects the robot's speed following accuracy in the forward and backward, left and right directions; the smaller the deviation, the more accurately the robot can execute high-level movement commands.

[0128] By subtracting the command velocity vector extracted by A41 from the actual velocity vector and calculating the magnitude, a non-negative scalar value is output, providing a quantitative basis for reward calculation.

[0129] A43: Determine the angular velocity deviation based on the target angular velocity and the actual angular velocity; It should be noted that the purpose of this sub-step is to quantify the tracking error of the quadruped robot to angular velocity commands.

[0130] Angular velocity deviation is the absolute value of the difference between the target angular velocity command and the actual angular velocity; steering speed refers to the rate of rotation about the vertical axis.

[0131] This deviation reflects the control precision of the robot's steering speed, ensuring that the robot accurately positions itself during rotational motion. By subtracting the commanded angular velocity scalar extracted by A41 from the actual angular velocity scalar and taking the absolute value, a non-negative scalar value is output to evaluate steering following performance.

[0132] A44: Determine the trajectory tracking reward value based on linear velocity deviation and angular velocity deviation; It should be noted that the purpose of this sub-step is to convert speed deviation into a reward signal to incentivize precise instruction tracking.

[0133] The trajectory tracking reward is a positive reward used to evaluate the accuracy of velocity following; a monotonically decreasing function is one that decreases in output as the input increases. This function maps linear velocity deviation and angular velocity deviation to the interval (0,1], with the reward being maximum when the deviation is zero and decreasing rapidly as the deviation increases. This function provides high gradient feedback for small deviations, making it suitable for reinforcement learning optimization.

[0134] By substituting the linear velocity deviation of A42 and the angular velocity deviation of A43 into the decreasing function, a positive reward value is output. The larger the value, the better the tracking performance, driving the robot to execute path planning accurately.

[0135] A45: Determine the leg time in the air bonus based on the time each leg spends in the air; It should be noted that the purpose of this step is to encourage standardized gait periodicity and improve walking efficiency and stability.

[0136] The leg airtime reward is a positive incentive that encourages the periodic lifting of the legs off the ground; gait periodicity refers to the characteristic of the legs swinging and supporting the ground in a regular pattern; ground contact moment refers to the moment when the foot first touches the ground; the reference value is the desired airtime; trotting gait refers to the gait pattern of diagonal leg movements in sync. This design encourages the legs to stay in the air for an appropriate amount of time, forming a trotting, walking, or running gait, avoiding dragging or high-frequency stepping.

[0137] By comparing the time each leg spends in the air extracted by A41 with reference values, the reward is calculated and accumulated at the moment of ground contact, and a scalar that can be positive or negative is output to guide the strategy to generate regular and efficient foot trajectories.

[0138] A46: Determine the static posture penalty item based on the static state flag; It should be noted that the purpose of this sub-step is to prevent the robot from experiencing attitude drift or low-frequency oscillations when it is stationary and waiting.

[0139] The static posture penalty is a negative reward that suppresses posture changes during standby; low-frequency oscillation refers to reciprocating motion with a low frequency; standby refers to the robot being in a ready state; and unintentional twitching refers to slight, non-commanded movements.

[0140] This penalty only takes effect when the speed command is close to zero, suppressing unintentional joint twitching during idle. By determining that the stationary flag of A41 is true, the joint angle deviation is calculated and substituted into the negative value function, outputting a non-positive scalar to ensure stable posture and energy saving during idle.

[0141] A47: A tripping penalty is generated when the planar component of the ground reaction force is greater than a preset multiple of the vertical component; It should be noted that the purpose of these steps is to prevent the feet from tripping or slipping due to excessive lateral force, thus ensuring walking safety.

[0142] Tripping penalty is a negative reward to avoid foot jamming; lateral force refers to the component force in the horizontal plane; vertical component refers to the component force perpendicular to the ground; multiplier is a preset proportional coefficient; drag force refers to the horizontal resistance that hinders movement; slipping refers to the foot losing grip.

[0143] This condition detects abnormal horizontal drag forces, which typically occur when the foot gets stuck on an obstacle or the ground is too slippery. By comparing the planar force components extracted by A41 with the vertical force components, a negative penalty is generated when a multiple condition is met, suppressing dangerous gaits and preventing the robot from becoming unstable and tipping over.

[0144] A48: A contact force over-limit penalty is generated when the magnitude of the contact force exceeds the maximum permissible contact force; It should be noted that the purpose of this step is to limit the peak impact of the ground and protect the mechanical structure from instantaneous impact damage.

[0145] The contact force exceeding the limit penalty is a negative reward that limits the peak impact force; instantaneous impact refers to a strong impact in a short period of time; structural strength refers to the material's ability to withstand external forces; hardware fatigue refers to the performance degradation caused by long-term impact.

[0146] This penalty takes effect when the robot walks on rough terrain or lands quickly, preventing impact forces from exceeding structural strength. By comparing the contact forces of each leg extracted by A41 with a preset upper limit, a penalty value is calculated and accumulated when the limit is exceeded, outputting a non-positive scalar value to avoid hardware fatigue or damage caused by high impacts.

[0147] A49: Gait reward value is obtained by integrating the leg time in the air reward, static posture penalty, tripping penalty, and contact force exceeding limit penalty.

[0148] It should be noted that the purpose of this sub-step is to integrate gait-related rewards and penalties to form a unified gait reward value.

[0149] Gait reward value is a scalar that comprehensively evaluates the quality of foot-ground interaction; weighted summation is weighted addition; foot-ground interaction refers to the contact mechanics process between the foot and the ground.

[0150] This integration aggregates scattered gait constraints into a single signal, facilitating unified optimization of the policy network. By performing a weighted summation, a gait reward value is output, which comprehensively reflects gait periodicity, static stability, anti-slip properties, and impact control level, providing a supervisory signal for policy optimization of foot-ground interaction.

[0151] Step S402: Determine the robotic arm operation reward value and the cooperative reward value based on the robotic arm status information, the quadruped robot status information and the environmental status information; It should be noted that the purpose of this step is to quantitatively evaluate the robotic arm's operational accuracy and arm-foot coupling coordination. By generating robotic arm operation reward values ​​and collaborative reward values ​​through two parallel calculations, the joint optimization of grasping tasks and platform stability can be achieved.

[0152] This step reads the status information of the robotic arm, the quadruped robot, and the environment during task execution. After two sets of independent processing flows, it outputs two scalar reward values, which serve as key components of the multi-dimensional reward value.

[0153] In one feasible implementation, step S402 includes steps A51 to A56: A51: Based on the robotic arm status information and environmental status information, obtain the end effector position, target position, end effector rotation matrix, target rotation matrix, gripper force, target force, and grasping success flag; It should be noted that the purpose of this sub-step is to extract the end-effector state and environmental target information to prepare input for evaluating the grasping accuracy.

[0154] The end effector position is calculated using the robotic arm's forward kinematics, representing the gripper's three-dimensional coordinates in Cartesian space. Forward kinematics is a mathematical model that calculates the end effector pose using joint angles. The target position is derived from the object detection module in the environmental state information, representing the geometric center or feature point of the object to be grasped. The object detection module is a perception algorithm that identifies the target object's pose. The end effector rotation matrix describes the gripper's posture and is a 3×3 orthogonal matrix. An orthogonal matrix refers to a rotation matrix with a determinant of 1. The target rotation matrix is ​​generated by the grasping task planning and represents the desired alignment posture. The gripper force is measured by fingertip force sensors and is a scalar or vector representation of the fingertip contact force. The fingertip force sensor is a force measurement device installed at the gripper's fingertips. The target force is the theoretical gripping force or torque required to achieve stable grasping. The grasping success flag is jointly determined by the force closed loop and the position closed loop, set to true when the object's pose relative to the gripper is stable and the applied force meets the target. The force closed loop is a feedback loop that controls the magnitude of the force, and the position closed loop is a feedback loop that controls the position.

[0155] By accessing the joint encoder data and force sensor data in the robotic arm's status information, and combining them with the target object's pose in the environmental status information, the above seven physical quantities are extracted as inputs for sub-steps A52 and A53.

[0156] A52: Calculate the positional deviation between the end effector position and the target position, calculate the rotational deviation between the end effector rotation matrix and the target rotation matrix, and calculate the force deviation between the gripper force and the target force. It should be noted that the purpose of this sub-step is to calculate the multidimensional deviation between the robotic arm's end effector and the target, which serves as the basis for calculating the operational reward.

[0157] Positional deviation is the Euclidean distance between the end effector position and the target position; Euclidean distance refers to the straight-line distance between two points in space; rotational deviation is the norm distance between the end effector rotation matrix and the target rotation matrix; norm distance is a measure of the difference between matrix elements; force deviation is the absolute difference between the gripper force and the target force; force control accuracy is the degree of accuracy of force control. These deviations quantify the alignment accuracy before gripping and the force control accuracy during gripping; the smaller the deviation, the more precise the operation.

[0158] By subtracting the pose and force data extracted from A51 pairwise and calculating the distance, three non-negative scalar values ​​are output, providing a quantitative basis for reward calculation.

[0159] A53: Determine the robotic arm operation reward value based on positional deviation, rotational deviation, force deviation, and successful grasping flag; It should be noted that the purpose of this step is to transform operational deviations into reward signals, guiding precise grasping and stable holding.

[0160] The robotic arm operation reward value is a comprehensive positive reward for evaluating grasping accuracy; the force control reward is the reward component for force control accuracy; and the gripping finger represents the stable grasping state of the object. This value typically consists of three parts: position reward, which is a monotonically decreasing function of positional deviation, with the maximum reward when the deviation is zero; attitude reward, which is a monotonically decreasing function of rotational deviation; and force control reward, which combines the grasping success indicator with the force deviation, providing a positive reward and penalizing force deviation after successful grasping.

[0161] Position and orientation rewards employ exponential or quadratic decay functions to provide high-gradient feedback for minor deviations, while force control rewards ensure a firm grasp without damaging the object. By substituting the three types of deviations calculated by A52 into a combination function, a comprehensive reward value is output. A larger value indicates a more precise grasping operation, providing a clear operational optimization objective for the strategy.

[0162] A54: Based on the state information of the robotic arm and the quadruped robot, obtain the end-effector acceleration of the robotic arm, the end-effector acceleration of the legs, the position of the system's center of mass, the geometric center of the supporting polygon, the joint torque of the robotic arm, and the joint torque of the legs; It should be noted that the purpose of this sub-step is to extract arm-foot kinematic and dynamic coupling data to prepare input for the evaluation of coordination.

[0163] The end effector acceleration of the robotic arm is obtained by mapping the joint acceleration of the arm to the Jacobian matrix, describing the intensity of the arm's motion; the Jacobian matrix is ​​the mapping matrix between joint velocities and end effector velocities; the end effector acceleration of the legs is obtained by averaging the accelerations of the four legs, reflecting the dynamic response of the platform; the position of the system's center of mass is calculated by weighting the mass distribution of the robotic arm and the quadruped robot, representing the overall equilibrium state; the mass distribution is the spatial distribution of the mass of each component; the geometric center of the supporting polygon is calculated from the coordinates of all current support foot contact points, representing the geometric representation of the stable region; the convex hull center is the geometric center of the supporting region; the joint torques of the robotic arm and the joint torques of the legs are the driving torques of the two structural components, reflecting the energy consumption distribution.

[0164] By accessing the joint acceleration in the robotic arm's state information, the foot state and joint torque in the quadruped robot's state information, and combining them with the mass model, the above six physical quantities are extracted as inputs for sub-steps A55 and A56.

[0165] A55: Obtain the acceleration deviation between the end-effector acceleration of the robotic arm and the end-effector acceleration of the leg; calculate the positional deviation between the system's center of mass and the geometric center of the supporting polygon; and calculate the sum of the squares of the modulus of the joint torques of the robotic arm and the leg. It should be noted that the purpose of this sub-step is to quantify the synchronization and overall balance of arm-foot movements, serving as the basis for calculating synergistic rewards.

[0166] Acceleration deviation is the difference in magnitude between the end-effector acceleration of the robotic arm and the end-effector acceleration of the leg, reflecting the consistency of the movement rhythm; rhythm consistency refers to the degree of synchronization between the arm and leg movements; position deviation is the Euclidean distance between the system's center of mass and the geometric center of the supporting polygon, measuring the overall balance maintenance capability; balance maintenance capability refers to the ability to maintain balance without tipping over; the sum of squared magnitudes is the sum of the squared moments of the robotic arm joints and the squared moments of the leg joints, characterizing the total energy consumption level. A smaller acceleration deviation indicates more synchronized arm and leg movements; a smaller position deviation indicates the center of mass is closer to the center of the stable region; and a smaller sum of squared moments indicates lower energy consumption.

[0167] By calculating the pairwise deviation and energy consumption index of the arm-leg state data extracted from A54, three scalar values ​​are output to provide a quantitative basis for collaborative rewards.

[0168] A56: Determine the collaborative reward value based on the acceleration deviation, position deviation, and the sum of squares of the modulus.

[0169] It should be noted that the purpose of this sub-step is to convert the arm-foot coupling index into a reward signal, thereby forcing the policy to learn spatiotemporally synchronized coordinated actions.

[0170] The collaborative reward value is a combination of negative and positive rewards evaluating arm-leg coordination; cross-platform coupling optimization refers to collaborative optimization across the robotic arm and quadruped platform. This value typically consists of three parts: acceleration coordination reward, a monotonically decreasing function of acceleration deviation, prompting the arm and leg to align their movement rhythms; center of mass stability reward, a negative correlation function of positional deviation, penalizing deviation of the center of mass from the support center; and energy consumption reward, a negative correlation function of the sum of squared torques, achieving optimal energy consumption distribution between the robotic arm and quadruped. The acceleration reward ensures that the quadrupeds are in a stable support phase during rapid robotic arm movements, the center of mass reward maintains overall balance to prevent tipping, and the energy consumption reward avoids local motor overload.

[0171] By substituting the three types of indicators calculated by A55 into the combination function, a collaborative reward value is output. The closer this value is to zero, the better the arm-foot coordination, providing a supervisory signal for cross-platform coupling optimization of the strategy.

[0172] Step S403: The multi-dimensional reward value is obtained by weighted summation of the basic stability reward value, dynamic constraint reward value, contact and collision reward value, trajectory tracking reward value, gait reward value, robotic arm operation reward value and cooperation reward value.

[0173] It should be noted that the purpose of this step is to integrate the seven categories of sub-rewards calculated separately into a single scalar signal, which facilitates gradient optimization in reinforcement learning algorithms. Weighted summation refers to addition with weights; scalarization refers to converting a multi-dimensional vector into a single numerical value; gradient optimization refers to updating parameters along the gradient direction. The basic stability reward value, dynamic constraint reward value, contact and collision reward value, trajectory tracking reward value, and gait reward value are derived from the calculation results of S401; the robotic arm operation reward value and cooperative reward value are derived from the calculation results of S402.

[0174] Weighted summation refers to the linear superposition of each reward value after assigning an adjustable weight coefficient. The weight coefficients can be dynamically adjusted according to the task type; for example, increasing the weight of trajectory tracking in a movement task and increasing the weight of robotic arm operation in a manipulation task. The multi-dimensional reward value is the scalar evaluation of the final output, and its magnitude directly determines the direction and magnitude of the policy update.

[0175] By multiplying the seven reward values ​​by their corresponding weights and summing them, a scalarized multi-dimensional reward value is output. This value contains both local performance and global coordination information. The policy network updates its parameters with the goal of maximizing this value, ultimately enabling efficient collaborative operation of heterogeneous robots in complex dynamic environments.

[0176] This embodiment provides a reinforcement learning method for the cooperative control of a multi-degree-of-freedom robotic arm and a quadruped robot. It achieves end-to-end cooperative optimization of heterogeneous robots through a multi-dimensional reward function: basic stability rewards and cooperative centroid deviation form a dual-loop balance control, significantly reducing base sway caused by robotic arm movement; dynamic constraints and contact collision rewards form soft and hard safety barriers, ensuring hardware safety and transfer robustness; trajectory tracking and gait rewards work together to improve movement efficiency; robotic arm operation rewards alleviate the sparse reward problem through continuous evaluation, accelerating grasping convergence; cooperative rewards force the arm-leg movement timing to align and maintain the centroid within the support area, achieving load balancing. A weighted summation mechanism enables the strategy to automatically weigh multiple objectives. Finally, with the support of domain randomization and online fine-tuning, the system can autonomously generate spatiotemporally synchronized action sequences without manual coordination logic, completing the movement-grasping-placement task, significantly improving autonomous adaptation and task success rate in dynamic scenarios.

[0177] It should be noted that the above examples are only for understanding this application and do not constitute a limitation on the reinforcement learning method for collaborative control of multi-degree-of-freedom robotic arms and quadruped robots in this application. Any simple modifications based on this technical concept are within the protection scope of this application.

[0178] This application also provides a reinforcement learning device for collaborative control of a multi-degree-of-freedom robotic arm and a quadruped robot. Please refer to [link / reference]. Figure 3 The reinforcement learning device for collaborative control of a multi-degree-of-freedom robotic arm and a quadruped robot includes: The strategy training module 10 is used to train the multi-agent cooperative strategy by training the state information of the robotic arm, the state information of the quadruped robot, and the state information of the environment through a centralized training distributed execution framework, so as to obtain the cooperative control strategy. The instruction generation module 20 is used to generate robotic arm motion instructions and quadruped robot motion instructions based on the cooperative control strategy. Control module 30 is used to send the robotic arm motion commands and the quadruped robot motion commands to the corresponding actuators so that the robotic arm and the quadruped robot can work together to perform tasks. The strategy update unit 40 is used to determine multi-dimensional reward values ​​based on the state information of the robotic arm, the quadruped robot, and the environment during task execution; and to update the cooperative control strategy based on the multi-dimensional reward values.

[0179] The multi-degree-of-freedom robotic arm and quadruped robot cooperative control reinforcement learning device provided in this application employs the multi-degree-of-freedom robotic arm and quadruped robot cooperative control reinforcement learning method described in the above embodiments. This method addresses the technical problems of low cooperative efficiency, poor environmental adaptability, and high training costs in dynamic tasks caused by the lack of a unified training framework and real-time coordination mechanism in multi-degree-of-freedom robotic arm and quadruped robot cooperative systems. Compared with the prior art, the beneficial effects of the multi-degree-of-freedom robotic arm and quadruped robot cooperative control reinforcement learning device provided in this application are the same as those of the multi-degree-of-freedom robotic arm and quadruped robot cooperative control reinforcement learning method described in the above embodiments. Furthermore, other technical features in the multi-degree-of-freedom robotic arm and quadruped robot cooperative control reinforcement learning device are the same as those disclosed in the methods of the above embodiments, and will not be elaborated upon here.

[0180] In one embodiment, the strategy update unit 40 is further configured to determine a basic stability reward value, a dynamic constraint reward value, a contact and collision reward value, a trajectory tracking reward value, and a gait reward value based on the quadruped robot's state information; The robot arm operation reward value and the cooperative reward value are determined based on the robot arm status information, the quadruped robot status information and the environmental status information. The multi-dimensional reward value is obtained by weighting and summing the basic stability reward value, dynamic constraint reward value, contact and collision reward value, trajectory tracking reward value, gait reward value, robotic arm operation reward value, and cooperation reward value.

[0181] In one embodiment, the strategy update unit 40 is further configured to obtain the vertical linear velocity of the base, the horizontal angular velocity of the base, the projection of the gravity vector onto the horizontal plane, the current average height of the base, and the target height of the base based on the quadruped robot's state information. The vertical velocity penalty term is determined based on the vertical linear velocity of the base; The angular velocity penalty term is determined based on the angular velocity of the horizontal surface of the base. The attitude stability penalty term is determined based on the projection of the gravity vector onto the horizontal plane. The height deviation penalty is determined based on the current average height of the base and the target height of the base. The basic stability bonus value is obtained by summing the vertical velocity penalty, angular velocity penalty, attitude stability penalty, and height deviation penalty.

[0182] In one embodiment, the strategy update unit 40 is further configured to obtain the joint torque, joint angular velocity, joint angular acceleration, current output action, previous output action, contact force of each leg, joint angle and preset joint angle range of the quadruped robot based on the quadruped robot state information. The dynamic constraint reward value is determined based on joint torque, joint angular velocity, and joint angular acceleration. When the joint angular velocity exceeds the angular velocity constraint threshold, a velocity over-limit penalty term is generated. A torque over-limit penalty term is generated when the joint torque exceeds the torque constraint threshold. A collision penalty is generated when the magnitude of the contact force exceeds a preset contact force threshold, and an angle over-limit penalty is generated when the joint angle exceeds a preset joint angle range. By integrating the speed over-limit penalty, torque over-limit penalty, collision penalty, and angle over-limit penalty, we obtain the contact and collision reward value.

[0183] In one embodiment, the strategy update unit 40 is further configured to obtain the desired planar linear velocity command, actual planar linear velocity, target angular velocity, actual angular velocity, time in the air for each leg, planar component of ground reaction force, and stationary state flag based on the quadruped robot's state information. Determine the linear velocity deviation based on the desired planar linear velocity command and the actual planar linear velocity; Determine the angular velocity deviation based on the target angular velocity and the actual angular velocity; The trajectory tracking reward value is determined based on the linear velocity deviation and angular velocity deviation. Leg time in the air bonus is determined based on the time each leg spends in the air. Determine the static posture penalty item based on the static state flag; A tripping penalty is generated when the planar component of the ground reaction force is greater than a preset multiple of the vertical component. A contact force over-limit penalty is generated when the magnitude of the contact force exceeds the maximum permissible contact force. The gait reward value is obtained by integrating the leg time in the air reward, static posture penalty, tripping penalty, and contact force exceeding the limit penalty.

[0184] In one embodiment, the strategy update unit 40 is further configured to obtain the end effector position, target position, end effector rotation matrix, target rotation matrix, gripper force, target force, and grasping success flag based on the robotic arm state information and environmental state information. Calculate the positional deviation between the end effector position and the target position, calculate the rotational deviation between the end effector rotation matrix and the target rotation matrix, and calculate the force deviation between the gripper force and the target force; The robotic arm operation reward value is determined based on positional deviation, rotational deviation, force deviation, and successful grasping flag. Based on the state information of the robotic arm and the quadruped robot, the end-effector acceleration of the robotic arm, the end-effector acceleration of the legs, the position of the system's center of mass, the geometric center of the supporting polygon, the joint torque of the robotic arm, and the joint torque of the legs are obtained. Obtain the acceleration deviation between the end-effector acceleration of the robotic arm and the end-effector acceleration of the leg; calculate the positional deviation between the system's center of mass and the geometric center of the supporting polygon; and calculate the sum of the squares of the modulus of the joint torques of the robotic arm and the leg. The collaborative reward value is determined based on the acceleration deviation, position deviation, and the sum of squares of the modulus.

[0185] In one embodiment, the strategy training module 10 is also used to train a multi-agent cooperative strategy based on the state information of the robotic arm, the state information of the quadruped robot and the state information of the environment, and to introduce a variety of disturbance factors during the training process to obtain an enhanced cooperative control strategy. Deploy the enhanced collaborative control strategy to real robotic systems; During the operation of a real robot system, the collaborative control strategy is further optimized through online fine-tuning.

[0186] This application provides a reinforcement learning device for cooperative control of a multi-degree-of-freedom robotic arm and a quadruped robot. The multi-degree-of-freedom robotic arm and quadruped robot cooperative control reinforcement learning device includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute the multi-degree-of-freedom robotic arm and quadruped robot cooperative control reinforcement learning method in the above embodiment 1.

[0187] The following is for reference. Figure 4This document illustrates a structural schematic diagram of a reinforcement learning device suitable for implementing the cooperative control of a multi-degree-of-freedom robotic arm and a quadruped robot in the embodiments of this application. The multi-degree-of-freedom robotic arm and quadruped robot cooperative control reinforcement learning device in the embodiments of this application may include, but is not limited to, mobile terminals such as mobile phones, laptops, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Descriptions), PMPs (Portable Media Players), and in-vehicle terminals (e.g., in-vehicle navigation terminals), as well as fixed terminals such as digital TVs and desktop computers. Figure 4 The multi-degree-of-freedom robotic arm and quadruped robot cooperative control reinforcement learning device shown is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of this application.

[0188] like Figure 4 As shown, the multi-degree-of-freedom robotic arm and quadruped robot cooperative control reinforcement learning device may include a processing unit 1001 (e.g., a central processing unit, a graphics processing unit, etc.), which can perform various appropriate actions and processes according to programs stored in ROM (Read Only Memory) 1002 or programs loaded from storage device 1003 into RAM (Random Access Memory) 1004. RAM 1004 also stores various programs and data required for the operation of the multi-degree-of-freedom robotic arm and quadruped robot cooperative control reinforcement learning device. The processing unit 1001, ROM 1002, and RAM 1004 are interconnected via bus 1005. Input / output (I / O) interface 1006 is also connected to the bus. Typically, the following systems can be connected to I / O interface 1006: input devices 1007 including, for example, touchscreens, touchpads, keyboards, mice, image sensors, microphones, accelerometers, gyroscopes, etc.; output devices 1008 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 1003 including, for example, magnetic tapes, hard disks, etc.; and communication devices 1009. Communication device 1009 allows the multi-degree-of-freedom robotic arm and quadruped robot cooperative control reinforcement learning device to communicate wirelessly or wiredly with other devices to exchange data. Although the figure shows a multi-degree-of-freedom robotic arm and quadruped robot cooperative control reinforcement learning device with various systems, it should be understood that it is not required to implement or possess all the systems shown. More or fewer systems can be implemented alternatively.

[0189] Specifically, according to the embodiments disclosed in this application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments disclosed in this application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device, or installed from storage device 1003, or installed from ROM 1002. When the computer program is executed by processing device 1001, it performs the functions defined in the methods of the embodiments disclosed in this application.

[0190] The multi-degree-of-freedom robotic arm and quadruped robot cooperative control reinforcement learning device provided in this application employs the multi-degree-of-freedom robotic arm and quadruped robot cooperative control reinforcement learning method described in the above embodiments. This method addresses the technical problems of low cooperative efficiency, poor environmental adaptability, and high training costs in dynamic tasks caused by the lack of a unified training framework and real-time coordination mechanism in multi-degree-of-freedom robotic arm and quadruped robot cooperative systems. Compared with the prior art, the beneficial effects of the multi-degree-of-freedom robotic arm and quadruped robot cooperative control reinforcement learning device provided in this application are the same as those of the multi-degree-of-freedom robotic arm and quadruped robot cooperative control reinforcement learning method described in the above embodiments. Furthermore, other technical features of this multi-degree-of-freedom robotic arm and quadruped robot cooperative control reinforcement learning device are the same as those disclosed in the previous embodiment method, and will not be elaborated upon here.

[0191] It should be understood that the various parts disclosed in this application can be implemented using hardware, software, firmware, or a combination thereof. In the description of the above embodiments, specific features, structures, materials, or characteristics can be combined in any suitable manner in one or more embodiments or examples.

[0192] The above are merely specific embodiments of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

[0193] This application provides a computer-readable storage medium having computer-readable program instructions (i.e., a computer program) stored thereon, which are used to execute the multi-degree-of-freedom robotic arm and quadruped robot cooperative control reinforcement learning method described in the above embodiments.

[0194] The computer-readable storage medium provided in this application may be, for example, a USB flash drive, but is not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to: electrical connections having one or more wires, portable computer disks, hard disks, RAM (Random Access Memory), ROM (Read Only Memory), EPROM (Erasable Programmable Read Only Memory or Flash Memory), optical fibers, CD-ROM (CD-Read Only Memory), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, system, or device. The program code contained on the computer-readable storage medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (Radio Frequency), etc., or any suitable combination thereof.

[0195] The aforementioned computer-readable storage medium may be included in the multi-degree-of-freedom robotic arm and quadruped robot cooperative control reinforcement learning device; or it may exist independently and not be assembled into the multi-degree-of-freedom robotic arm and quadruped robot cooperative control reinforcement learning device.

[0196] The aforementioned computer-readable storage medium carries one or more programs. When these programs are executed by a multi-degree-of-freedom robotic arm and quadruped robot cooperative control reinforcement learning device, the device performs the following: training a multi-agent cooperative strategy on the robotic arm's state information, the quadruped robot's state information, and the environment's state information using a centralized training distributed execution framework, thereby obtaining a cooperative control strategy; generating robotic arm motion commands and quadruped robot motion commands based on the cooperative control strategy; sending the robotic arm's motion commands and quadruped robot motion commands to the corresponding actuators to enable the robotic arm and quadruped robot to collaboratively perform tasks; determining multi-dimensional reward values ​​based on the robotic arm's state information, the quadruped robot's state information, and the environment's state information; and updating the cooperative control strategy based on the multi-dimensional reward values.

[0197] Computer program code for performing the operations of this application can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, as well as conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including LAN (Local Area Network) or WAN (Wide Area Network)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0198] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0199] The modules described in the embodiments of this application can be implemented in software or hardware. The names of the modules do not necessarily limit the functionality of the unit itself.

[0200] The readable storage medium provided in this application is a computer-readable storage medium that stores computer-readable program instructions (i.e., computer programs) for executing the aforementioned multi-degree-of-freedom robotic arm and quadruped robot cooperative control reinforcement learning method. This solves the technical problems of low cooperation efficiency, poor environmental adaptability, and high training costs in multi-degree-of-freedom robotic arm and quadruped robot cooperative systems during dynamic tasks due to the lack of a unified training framework and real-time coordination mechanism. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided in this application are the same as those of the multi-degree-of-freedom robotic arm and quadruped robot cooperative control reinforcement learning method provided in the above embodiments, and will not be elaborated upon here.

[0201] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the reinforcement learning method for cooperative control of a multi-degree-of-freedom robotic arm and a quadruped robot as described above.

[0202] The computer program product provided in this application can solve the technical problems of low collaboration efficiency, poor environmental adaptability, and high training costs in multi-degree-of-freedom robotic arm and quadruped robot collaborative systems in dynamic tasks due to the lack of a unified training framework and real-time coordination mechanism. Compared with the prior art, the beneficial effects of the computer program product provided in this application are the same as those of the multi-degree-of-freedom robotic arm and quadruped robot collaborative control reinforcement learning method provided in the above embodiments, and will not be repeated here.

[0203] The above description is only a part of the embodiments of this application and does not limit the patent scope of this application. All equivalent structural transformations made under the technical concept of this application and using the contents of the specification and drawings of this application, or direct / indirect applications in other related technical fields, are included in the patent protection scope of this application.

Claims

1. A reinforcement learning method for collaborative control of a multi-degree-of-freedom robotic arm and a quadruped robot, characterized in that, The method includes: By using a centralized training and decentralized execution framework, multi-agent cooperative strategies are trained on the state information of the robotic arm, the quadruped robot, and the environment to obtain a cooperative control strategy. Based on the aforementioned collaborative control strategy, action commands for the robotic arm and quadruped robot are generated. The robotic arm motion commands and the quadruped robot motion commands are sent to the corresponding actuators so that the robotic arm and the quadruped robot can work together to perform the task. During task execution, a multi-dimensional reward value is determined based on the robotic arm status information, the quadruped robot status information, and the environmental status information. The collaborative control strategy is updated based on the multi-dimensional reward value.

2. The method as described in claim 1, characterized in that, The step of determining the multi-dimensional reward value based on the robotic arm state information, the quadruped robot state information, and the environmental state information includes: The basic stability reward value, dynamic constraint reward value, contact and collision reward value, trajectory tracking reward value, and gait reward value are determined based on the state information of the quadruped robot. The robotic arm operation reward value and the cooperative reward value are determined based on the robotic arm status information, the quadruped robot status information, and the environmental status information. A multi-dimensional reward value is obtained by weighted summation of the basic stability reward value, the dynamic constraint reward value, the contact and collision reward value, the trajectory tracking reward value, the gait reward value, the robotic arm operation reward value, and the collaboration reward value.

3. The method as described in claim 2, characterized in that, The step of determining the basic stability reward value based on the quadruped robot's state information includes: Based on the quadruped robot's state information, obtain the vertical linear velocity of the quadruped robot's base, the horizontal angular velocity of the base, the projection of the gravity vector onto the horizontal plane, the current average height of the base, and the target height of the base; The vertical velocity penalty term is determined based on the vertical linear velocity of the base. The angular velocity penalty term is determined based on the angular velocity of the horizontal surface of the base. The attitude stability penalty term is determined based on the projection of the gravity vector onto the horizontal plane. A height deviation penalty is determined based on the current average height of the base and the target height of the base. The basic stability reward value is obtained by summing the vertical velocity penalty, the angular velocity penalty, the attitude stability penalty, and the height deviation penalty.

4. The method as described in claim 2, characterized in that, The step of determining the dynamic constraint reward value and the contact and collision reward value based on the quadruped robot's state information includes: Based on the quadruped robot's state information, obtain the quadruped robot's joint torque, joint angular velocity, joint angular acceleration, current output action, previous output action, contact force of each leg, joint angle, and preset joint angle range; The dynamic constraint reward value is determined based on the joint torque, the joint angular velocity, and the joint angular acceleration. When the joint angular velocity exceeds the angular velocity constraint threshold, a velocity over-limit penalty term is generated. When the joint torque exceeds the torque constraint threshold, a torque over-limit penalty term is generated. A collision penalty term is generated when the magnitude of the contact force exceeds a preset contact force threshold, and an angle over-limit penalty term is generated when the joint angle exceeds the preset joint angle range. The speed over-limit penalty, torque over-limit penalty, collision penalty, and angle over-limit penalty are integrated to obtain the contact and collision reward value.

5. The method as described in claim 2, characterized in that, The step of determining the trajectory tracking reward value and gait reward value based on the quadruped robot's state information includes: Based on the quadruped robot's state information, the desired planar linear velocity command, actual planar linear velocity, target angular velocity, actual angular velocity, time in the air for each leg, planar component of ground reaction force, and stationary state flag are obtained. The linear velocity deviation is determined based on the desired planar linear velocity command and the actual planar linear velocity. The angular velocity deviation is determined based on the target angular velocity and the actual angular velocity; The trajectory tracking reward value is determined based on the linear velocity deviation and the angular velocity deviation; The leg time in the air reward items are determined based on the time each leg spends in the air. The static posture penalty term is determined based on the static state flag; A tripping penalty is generated when the planar component of the ground reaction force is greater than a preset multiple of the vertical component. A contact force over-limit penalty term is generated when the magnitude of the contact force exceeds the maximum permissible contact force. The gait reward value is obtained by integrating the leg time in the air reward, the static posture penalty, the tripping penalty, and the contact force exceeding the limit penalty.

6. The method as described in claim 2, characterized in that, The step of determining the robotic arm operation reward value and the cooperative reward value based on the robotic arm state information, the quadruped robot state information, and the environmental state information includes: Based on the robotic arm status information and the environmental status information, obtain the end effector position, target position, end effector rotation matrix, target rotation matrix, gripper force, target force, and grasping success flag; Calculate the positional deviation between the end effector position and the target position, calculate the rotational deviation between the end effector rotation matrix and the target rotation matrix, and calculate the force deviation between the gripper force and the target force; The robotic arm operation reward value is determined based on the position deviation, the rotation deviation, the force deviation, and the successful grasping flag. Based on the state information of the robotic arm and the state information of the quadruped robot, the end-effector acceleration of the robotic arm, the end-effector acceleration of the legs, the position of the system's center of mass, the geometric center of the supporting polygon, the joint torque of the robotic arm, and the joint torque of the legs are obtained. Obtain the acceleration deviation between the end-effector acceleration of the robotic arm and the end-effector acceleration of the leg; calculate the positional deviation between the system's center of mass and the geometric center of the supporting polygon; and calculate the sum of the squares of the modulus of the joint torques of the robotic arm and the joint torques of the leg. The cooperative reward value is determined based on the acceleration deviation, the position deviation, and the sum of squares of the modulus.

7. The method as described in claim 1, characterized in that, The step of training a multi-agent cooperative strategy by using a centralized training distributed execution framework to train the state information of the robotic arm, the quadruped robot, and the environment to obtain a cooperative control strategy includes: Based on the state information of the robotic arm, the state information of the quadruped robot, and the state information of the environment, a multi-agent cooperative strategy is trained, and various disturbance factors are introduced during the training process to obtain an enhanced cooperative control strategy. The enhanced collaborative control strategy is deployed to a real robot system; During the operation of a real robot system, the collaborative control strategy is further optimized through online fine-tuning.

8. A multi-robot collaborative control device, characterized in that, The device includes: The strategy training module is used to train multi-agent cooperative strategies by using a centralized training distributed execution framework to train the state information of the robotic arm, the state information of the quadruped robot, and the state information of the environment, thereby obtaining a cooperative control strategy. The instruction generation module is used to generate robotic arm motion instructions and quadruped robot motion instructions based on the cooperative control strategy. The control module is used to send the robotic arm motion commands and the quadruped robot motion commands to the corresponding actuators, so that the robotic arm and the quadruped robot can work together to perform tasks. The strategy update unit is used to determine a multi-dimensional reward value based on the state information of the robotic arm, the state information of the quadruped robot, and the state information of the environment during task execution; and to update the cooperative control strategy based on the multi-dimensional reward value.

9. A reinforcement learning device for collaborative control of a multi-degree-of-freedom robotic arm and a quadruped robot, characterized in that, The device includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the reinforcement learning method for cooperative control of a multi-degree-of-freedom robotic arm and a quadruped robot as described in any one of claims 1 to 7.

10. A storage medium, characterized in that, The storage medium is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, it implements the steps of the reinforcement learning method for cooperative control of a multi-degree-of-freedom robotic arm and a quadruped robot as described in any one of claims 1 to 7.

Images