Robot dynamic operation system and method and electronic equipment

Abstract

The invention relates to the technical field of robot dynamic operation and intelligent control, in particular to a robot dynamic operation method and system and electronic equipment. Comprising the following steps: implicitly representing and predicting a moving object track based on an encoder-decoder architecture so as to understand a high dynamic environment; environment characterization and a robot body state are fused, and an operation strategy is autonomously learned through large-scale distributed reinforcement learning; a curriculum learning and antagonistic disturbance framework is introduced, and a task space strategy is converted into a whole-body cooperative control instruction of the mobile mechanical arm. According to the method, the defects that a traditional method is high in dependence on an environment model, poor in strategy self-adaption and weak in control cooperation are overcome, integrated autonomous intelligence of sensing, decision making and control of the robot in the dynamic environment is achieved, and the execution efficiency and robustness of a dynamic operation task are remarkably improved.

Description

Technical Field

[0001] This invention belongs to the field of robotics technology, specifically relating to a robot dynamic operating system, method, and electronic device. Background Technology

[0002] Dynamic manipulation, an important branch of robotics, aims to enable robots to precisely capture, throw, and strike moving objects, much like humans do. This technology has broad application prospects in scenarios such as space debris cleanup, automated painting of large components, smart home services, and human-computer interactive training.

[0003] Currently, research in this field mainly faces the following challenges and shortcomings: First, at the level of environmental perception and prediction, traditional methods largely rely on establishing accurate dynamic models of moving objects to predict their trajectories. However, in complex, unstructured environments, the dynamic parameters of moving objects (such as mass, air resistance coefficient, and rotation) are difficult to accurately obtain and identify, and once the model is determined, it lacks adaptability. For example, in a scenario of hitting a ping-pong ball, factors such as the ball's rotation and airflow velocity significantly affect its flight trajectory, making prediction methods based on fixed physical models have poor generalization ability and large prediction errors in practical applications.

[0004] Secondly, at the operational decision-making and planning level, most existing robot operation strategies rely on precise dynamic models or extensive human teaching and programming. These methods appear rigid and lack adaptability when facing highly dynamic and uncertain environments, failing to generate optimal operation strategies in real time. While imitation learning-based methods can learn from human demonstrations, they are often limited by the quality of the teaching data and the differences between the robot's structure and that of humans, making it difficult to fully utilize the robot's inherent mechanical potential. Although reinforcement learning offers a possibility for autonomous learning, it generally suffers from sparse rewards and low sample efficiency in dynamic operation tasks, and the performance of the trained strategies degrades significantly when transferred from simulation to the real world.

[0005] Finally, at the motion execution and control level, the composite robot system consisting of a mobile chassis and a robotic arm is a typical redundant system, with a control complexity far exceeding that of a fixed-base robotic arm. Existing research typically handles the path planning of the mobile platform and the motion planning of the robotic arm separately, or employs simplified control strategies, making it difficult to achieve true coordination between the two at high speeds. This separate planning and control approach often fails to guarantee the overall motion smoothness and energy efficiency of the end effector when tracking dynamic trajectories, and easily overlooks collision constraints between the robotic arm and the mobile platform, as well as between robot links, leading to execution failures or safety issues.

[0006] Therefore, existing technologies lack a solution for robot dynamic manipulation that integrates high-speed dynamic perception, autonomous learning decision-making, and whole-body coordinated control. There is an urgent need for an environment understanding method that does not rely on precise physical models, a decision-making mechanism capable of efficient autonomous learning and strong generalization, and a real-time optimized control framework that can fully coordinate the mobile platform and robotic arm while satisfying various physical constraints, in order to address the challenges of dynamic manipulation tasks. Summary of the Invention

[0007] To address the three major bottlenecks in existing technologies for robot dynamic operation—strong dependence on environmental perception models, poor adaptability of decision-making strategies, and weak coordination of whole-body control—this invention aims to provide a robot dynamic operation scheme that integrates implicit representation and constraint optimization.

[0008] A method for dynamic robot operation, the core of which lies in adopting a three-layer architecture of representation layer - decision layer - control layer, specifically includes the following steps: S1. Environmental Characterization and Prediction: Constructing an implicit trajectory characterization and prediction model based on a distributed decoder and a variational autoencoder based on gated recurrent units, specifically including: Variational encoder: The input is the historical trajectory sequence of a ping-pong ball from time 1 to time m. Encoding is performed through a GRU network, and the output is a Gaussian distribution defining the mean μ and variance σ of the implicit encoding vector Z. The implicit encoding vector Z is calculated using the following reparameterization technique: Where θ represents the encoder network parameters, ⊙ denotes element-wise multiplication, and the trajectory reconstruction decoder: This decoder reconstructs the trajectory from time 1 to m using another GRU network based on the implicit encoded vector Z. Its output is defined by the following Gaussian distribution: This branch is only used for model training and does not participate in the final inference prediction; Distributed decoder: Based on the same implicit encoding vector Z, it predicts the future trajectory from time m+1 to n through a GRU network and fully connected layers. Its output is also defined as: It is used to provide predictions of future trajectories; The training in step S1 includes: first, training the variational encoder to learn accurate trajectory encoding and reconstruction; then, fixing the encoder parameters and training the distributed decoder separately; finally, during deployment, the system inputs the historical trajector observed in real time into the encoder to obtain Z, and uses the predictive decoder to output future trajectory predictions, providing key information for subsequent decision-making.

[0009] S2. Operation Strategy Generation: Based on large-scale distributed reinforcement training, the robot autonomously learns and generates its action strategy. A reinforcement learning model is constructed, which integrates the implicit trajectory representation and prediction model obtained in step S1 with the robot's own state information and task target points into state observations. Through large-scale distributed training, the robot autonomously learns and generates the target actions of its end effector in the task space.

[0010] Specifically, this step involves constructing a reinforcement learning model whose state observations integrate the implicit trajectory representation vector obtained in step S1, as well as the historical pose and velocity sequences of the robot's end effector (body perception information), and the coordinates of the task target point in the robot's base coordinate system. This reinforcement learning model is trained in a large-scale parallel environment built on the NVIDIA Isaac Sim simulation platform, employing a learning process that progresses from easy to difficult and an adversarial perturbation framework. By dynamically adjusting the task difficulty and introducing adversarial environmental changes, the robustness and generalization ability of the policy are continuously enhanced, ultimately outputting the robot's end effector target action in the task space.

[0011] Two mechanisms, curriculum learning and adversarial perturbation framework, are introduced to train reinforcement learning strategies.

[0012] Course Learning: Training does not begin in the most difficult environment. Instead, a curriculum is designed to progress from easy to difficult. Beginner: Learn basic interception skills in an ideal environment (no noise, no wind, objects moving at a constant speed).

[0013] Intermediate: Gradually increase the perturbation range of the dynamic parameters and introduce simple trajectory mutations.

[0014] Advanced: Introducing "adversarial perturbations," where an adversarial network learns how to slightly alter the trajectory of an object (such as simulating an unpredictable gust of wind in a simulation), so that the robot's strategy remains robust even in the "worst-case" scenario.

[0015] Adversarial perturbations: The goal of this adversarial network is to cause the robot to fail its task, while the robot's policy network aims to overcome these perturbations and successfully complete the task. The two networks engage in a game of mutual competition during training, evolving together.

[0016] By breaking down task difficulty, the course effectively addresses the problem of low exploration efficiency caused by sparse rewards in the early stages of reinforcement learning. The agent first masters the basics before tackling more challenging tasks, resulting in a more stable and faster learning process.

[0017] Strategies "baptized" by adversarial perturbations exhibit a strong adaptability to various unknown and adversarial dynamic changes in the real world. This far exceeds the potential of ordinary domain randomization, minimizing the performance degradation from simulation to reality.

[0018] It can automatically generate increasingly challenging training environments, reducing reliance on manually designed simulation scenarios.

[0019] S3. Full-body coordinated control: Based on the target motion of the end effector generated in step S2, a smooth motion trajectory of the end effector is generated in the task space using polynomial interpolation. For each path point on the trajectory, a full-body coordinated controller based on nonlinear model predictive control (NMPC) is used to calculate the control commands of the robot's mobile chassis and each joint of the robotic arm in real time.

[0020] Specifically, this step involves: First, based on the target motion generated by S2, a smooth motion trajectory with continuous acceleration of the end effector is generated within the task space using fifth-order polynomial interpolation. Then, for the generated smooth end effector trajectory, a whole-body cooperative controller based on nonlinear model predictive control (NMPC) is employed. Within a look-ahead time window, this controller performs rolling optimization based on the overall nonlinear dynamics model of the mobile arm, solving in real-time an optimization problem that simultaneously satisfies multiple constraints such as trajectory tracking in the task space, generalized velocity and acceleration limits, and self-collision avoidance. This allows for the calculation of the optimal cooperative control commands for the robot's mobile chassis and each joint of the robotic arm.

[0021] Instead of considering only the current path point, it predicts the dynamic behavior of the robot system for several steps in the future within a finite time domain. It obtains the optimal sequence of control commands for the current and future time period by solving a constrained nonlinear optimization problem, but only executes the first step, and then repeats the process in the next control cycle.

[0022] The optimization problem directly incorporates a nonlinear dynamics model of the mobile robotic arm, which can more accurately describe the system's behavior under high-speed motion.

[0023] NMPC plans the movement of the mobile chassis and the posture adjustment of the robotic arm in advance, making the overall movement smoother and more energy-efficient. Because it explicitly considers system dynamics, NMPC can better handle effects such as inertia and Coriolis force in high-speed dynamic operation, significantly improving control accuracy and stability. Its ability to handle nonlinear constraints allows complex self-collision avoidance and dynamic constraints to be met more precisely, resulting in higher safety.

[0024] A robot dynamic operating system for implementing a robot dynamic operation method includes: a visual perception module for acquiring and processing image data and obtaining the historical trajectory sequence of a moving object; the visual perception module includes a high-speed binocular camera capable of reconstructing the three-dimensional position of the moving object at a frequency of not less than 200Hz.

[0025] An electronic device includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, it implements steps such as those of a robot dynamic manipulation method. The electronic device is an industrial computer or embedded computing unit that integrates the trajectory prediction module, strategy decision module, and motion control module of the robot dynamic manipulation method. The electronic device communicates and exchanges data with the vision perception module and the robot actuator through the ROS robot operating system.

[0026] A computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of a robot dynamic manipulation method. The storage medium is a non-transient storage medium, including a computer hard disk, solid-state drive, USB flash drive, or optical disk; the program includes neural network model code for performing the environment characterization and prediction steps, reinforcement learning policy network code for performing the operation strategy generation steps, and trajectory generation and optimization solution code for performing the whole-body coordinated control steps.

[0027] Compared with the prior art, the advantages of the present invention include: (1) The present invention provides a robot dynamic operating system, method, and electronic device, which provides a method for representing and predicting the trajectory of high-speed moving objects without relying on a precise physical model, so as to improve the robot's perception and understanding capabilities in highly dynamic environments. It also provides a method that can integrate environmental information and the robot's own state and can efficiently and autonomously learn operating strategies through large-scale simulation, so as to enhance the robot's decision-making intelligence and adaptability.

[0028] (2) The robot dynamic operating system, method and electronic device provided by the present invention can provide a real-time calculation method that can realize the full-body coordination of the mobile platform and the robotic arm and optimize control under multiple constraints, so as to ensure the accuracy, smoothness and safety of dynamic operation tasks. Attached Figure Description

[0029] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0030] Figure 1 This is a schematic diagram of the overall kinematic model of a robot dynamic operating system, method, and electronic device according to the present invention. Figure 2 This is a schematic diagram illustrating the trajectory tracking control effect of a robot dynamic operating system, method, and electronic device in this invention. Figure 3 This is a comparison diagram of the experimental effects of a robot dynamic operating system, method and electronic equipment in a dynamic ball-hitting task according to the present invention; Figure 4 This is a table of performance indicators for a robot dynamic operating system, method, and electronic device in this invention. Detailed Implementation

[0031] In view of the shortcomings of the prior art, the inventors of this invention, through long-term research and extensive practice, have proposed the technical solution of this invention. The technical solution, its implementation process, and principles will be further explained below with reference to the accompanying drawings and specific implementation examples in the embodiments of this application.

[0032] It should be noted that the embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and should not be construed as limiting the present invention. The described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, the present invention covers any substitutions, modifications, equivalent methods and solutions made on the spirit, principles and scope of the present invention as defined by the claims. All other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0033] In the description of this application, the terms "first," "second," "third," and similar words do not indicate any order, quantity, or importance, but are merely used to distinguish different components. Similarly, the terms "a" or "one," and similar words, do not indicate a quantity limitation, but rather indicate the presence of at least one. The terms "comprising" or "including," and similar words, mean that the elements or objects preceding "comprising" or "including" encompass the elements or objects listed following "comprising" or "including," and their equivalents, but do not exclude other elements or objects. The terms "connected" or "linked," and similar words, are not limited to physical or mechanical connections, but can include electrical connections, whether direct or indirect.

[0034] In the description of this application, the terms "center," "upper," "lower," "front," "rear," "left," "right," "vertical," "horizontal," "top," "bottom," "inner," and "outer," etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are used solely for the convenience of describing this application and for simplification, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on this application. Furthermore, when using positional terms such as "both sides," "outer side," and "upper and lower," it should be understood that they are used only for ease of understanding and description, taking into account that the structure may be oriented to other positions.

[0035] In the description of this application, unless otherwise expressly specified and limited, the technical or scientific terms used shall have the ordinary meaning understood by a person with ordinary skills in the art to which this application pertains. Terms such as “installation,” “connection,” and “joining” shall be interpreted broadly, for example, as fixed connection, detachable connection, mating connection, or integral connection. For a person skilled in the art, the specific meaning of the above terms in this application can be understood according to the specific circumstances.

[0036] The present invention aims to introduce and explain the structural composition of a robot dynamic operating system, method, and electronic device, as well as the cooperation relationship between the various components. Unless otherwise specified, the dimensions, materials, and manufacturing processes of the various components in the robot dynamic operating system, method, and electronic device in the present invention can be selected according to specific circumstances, and no special limitations or explanations are made here.

[0037] Furthermore, to provide the public with a better understanding of the present invention, certain specific details are described in detail in the following description of the invention. However, those skilled in the art will fully understand the invention even without these detailed descriptions.

[0038] Example 1 Please see Figures 1-4 A method for dynamic robot operation, the core of which lies in adopting a three-layer architecture of representation layer - decision layer - control layer, specifically including the following steps: S1. Environmental Representation and Prediction: Based on a neural network architecture of variational autoencoder-distributed decoder, the historical trajectory sequence of a moving object is encoded to obtain a low-dimensional implicit trajectory representation vector. The distributed decoder then predicts the future trajectory of the moving object based on this implicit trajectory representation vector. Specifically, it includes: Variational encoder: Input is a sequence of historical trajectories from time 1 to time m. Encoding is performed through a GRU network, and the output is a Gaussian distribution defining the mean μ and variance σ of the implicit encoding vector Z. The implicit encoding vector Z is calculated using the following reparameterization technique: Where θ represents the encoder network parameters, ⊙ denotes element-wise multiplication, and the trajectory reconstruction decoder: This decoder reconstructs the trajectory from time 1 to m using another GRU network based on the implicit encoded vector Z. Its output is defined by the following Gaussian distribution: This branch is only used for model training and does not participate in the final inference prediction; Distributed decoder: Based on the same implicit encoding vector Z, it predicts the future trajectory from time m+1 to n through a GRU network and fully connected layers. Its output is also defined as: It is used to provide predictions of future trajectories; The training in step S1 includes: first, training the variational encoder to learn accurate trajectory encoding and reconstruction; then, fixing the encoder parameters and training the distributed decoder separately; finally, during deployment, the system inputs the historical trajectory observed in real time into the encoder to obtain Z, and uses the predictive decoder to output future trajectory predictions, providing key information for subsequent decision-making.

[0039] S2. Operation Strategy Generation: Based on large-scale distributed reinforcement training, the robot autonomously learns and generates its action strategy. A reinforcement learning model is constructed, which integrates the implicit trajectory representation and prediction model obtained in step S1 with the robot's own state information and task target points into state observations. Through large-scale distributed training, the robot autonomously learns and generates the target actions of its end effector in the task space.

[0040] Specifically, this step involves constructing a reinforcement learning model whose state observations integrate the implicit trajectory representation vector obtained in step S1, as well as the historical pose and velocity sequences of the robot's end effector (body perception information), and the coordinates of the task target point in the robot's base coordinate system. This reinforcement learning model is trained in a large-scale parallel environment built on the NVIDIA Isaac Sim simulation platform, employing a learning process that progresses from easy to difficult and an adversarial perturbation framework. By dynamically adjusting the task difficulty and introducing adversarial environmental changes, the robustness and generalization ability of the policy are continuously enhanced, ultimately outputting the robot's end effector target action in the task space.

[0041] Two mechanisms, curriculum learning and adversarial perturbation framework, are introduced to train reinforcement learning strategies.

[0042] Course Learning: Training does not begin in the most difficult environment. Instead, a curriculum is designed to progress from easy to difficult. Beginner: Learn basic interception skills in an ideal environment (no noise, no wind, objects moving at a constant speed).

[0043] Intermediate: Gradually increase the perturbation range of the dynamic parameters and introduce simple trajectory mutations.

[0044] Advanced: Introducing "adversarial perturbations," where an adversarial network learns how to slightly alter the trajectory of an object (such as simulating an unpredictable gust of wind in a simulation), so that the robot's strategy remains robust even in the "worst-case" scenario.

[0045] Adversarial perturbations: The goal of this adversarial network is to cause the robot to fail its task, while the robot's policy network aims to overcome these perturbations and successfully complete the task. The two networks compete against each other during training and evolve together.

[0046] By breaking down task difficulty, the course effectively addresses the problem of low exploration efficiency caused by sparse rewards in the early stages of reinforcement learning. The agent first masters the basics before tackling more challenging tasks, resulting in a more stable and faster learning process.

[0047] Strategies "baptized" by adversarial perturbations exhibit a strong adaptability to various unknown and adversarial dynamic changes in the real world. This far exceeds the potential of ordinary domain randomization, minimizing the performance degradation from simulation to reality.

[0048] It can automatically generate increasingly challenging training environments, reducing reliance on manually designed simulation scenarios.

[0049] S3. Full-body coordinated control: Based on the target motion of the end effector generated in step S2, a smooth motion trajectory of the end effector is generated in the task space using polynomial interpolation. For each path point on the trajectory, a full-body coordinated controller based on nonlinear model predictive control (NMPC) is used to calculate the control commands of the robot's mobile chassis and each joint of the robotic arm in real time.

[0050] Specifically, this step involves the following steps: First, based on the target motion generated by S2, a smooth motion trajectory with continuous acceleration of the end effector is generated within the task space using fifth-order polynomial interpolation. Then, for the generated smooth end effector trajectory, a whole-body cooperative controller based on nonlinear model predictive control (NMPC) is employed. Within a look-ahead time window, this controller performs rolling optimization based on the overall nonlinear dynamics model of the mobile robotic arm, solving in real-time an optimization problem that simultaneously satisfies multiple constraints, including trajectory tracking in the task space, generalized velocity and acceleration limits, and collision avoidance. This allows for the calculation of the optimal cooperative control commands for the robot's mobile chassis and each joint of the robotic arm.

[0051] Instead of considering only the current path point, it predicts the dynamic behavior of the robot system for several steps in the future within a finite time domain. It obtains the optimal sequence of control commands for the current and future time period by solving a constrained nonlinear optimization problem, but only executes the first step, and then repeats the process in the next control cycle.

[0052] The optimization problem directly incorporates a nonlinear dynamics model of the mobile robotic arm, which can more accurately describe the system's behavior under high-speed motion.

[0053] NMPC plans the movement of the mobile chassis and the posture adjustment of the robotic arm in advance, making the overall movement smoother and more energy-efficient. Because it explicitly considers system dynamics, NMPC can better handle effects such as inertia and Coriolis force in high-speed dynamic operation, significantly improving control accuracy and stability. Its ability to handle nonlinear constraints allows complex self-collision avoidance and dynamic constraints to be met more precisely, resulting in higher safety.

[0054] A robot dynamic operating system for implementing a robot dynamic operation method includes: a visual perception module for acquiring and processing image data and obtaining the historical trajectory sequence of a moving object; the visual perception module includes a high-speed binocular camera capable of reconstructing the three-dimensional position of the moving object at a frequency of not less than 200Hz.

[0055] An electronic device includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, it implements steps such as those of a robot dynamic manipulation method. The electronic device is an industrial computer or embedded computing unit that integrates the trajectory prediction module, strategy decision module, and motion control module of the robot dynamic manipulation method. The electronic device communicates and exchanges data with the vision perception module and the robot actuator through the ROS robot operating system.

[0056] A computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of a robot dynamic manipulation method. The storage medium is a non-transient storage medium, including a computer hard disk, solid-state drive, USB flash drive, or optical disk; the program includes neural network model code for performing the environment characterization and prediction steps, reinforcement learning policy network code for performing the operation strategy generation steps, and trajectory generation and optimization solution code for performing the whole-body coordinated control steps.

[0057] Experimental verification To verify the effectiveness of the method of this invention, table tennis ball return experiments were conducted on simulation and preliminary physical platforms. (See attached...) Figure 3 and attached Figure 4As shown, the mobile robotic arm using the method of this invention has a significantly higher success rate and landing accuracy than traditional pre-programmed trajectory methods and single reinforcement learning strategies that do not incorporate implicit representations when dealing with balls of different speeds and rotations, fully demonstrating the superiority of this invention in dynamic operation tasks.

[0058] Example 2 Taking the typical high-dynamic task of a mobile robotic arm hitting a ping-pong ball as an example, the specific implementation process of the present invention is explained.

[0059] Environmental Characterization and Prediction—Implicit Representation and Prediction Algorithm for Moving Object Trajectories Based on Distributed Decoder This step aims to achieve accurate perception and prediction of moving objects in highly dynamic environments.

[0060] Step 1: Data Acquisition and Preparation. A high-speed binocular vision system was used to acquire flight trajectory data of the ping-pong ball at a frequency of 200Hz. A triangulation algorithm was then used to obtain its three-dimensional spatial position sequence in real time, forming historical trajectory observation data. This embodiment collected over 1000 complete motion trajectories acquired in real and simulated environments for model training and testing.

[0061] 1.1 Model Construction A trajectory implicit representation and prediction model based on distributed decoders is constructed. This model is based on a variational autoencoder constructed from gated recurrent units, mainly consisting of one encoder and two distributed decoders.

[0062] Encoder: Input is the historical trajectory sequence of a ping-pong ball from time 1 to time m. Encoding is performed using a GRU network, outputting the mean μ and variance σ of a Gaussian distribution defining the implicit encoding vector Z. The implicit encoding vector Z is calculated using the following reparameterization technique: Where θ represents the encoder network parameters, and ⊙ denotes element-wise multiplication. Trajectory Reconstruction Decoder: This decoder reconstructs the trajectory from time 1 to m using another GRU network based on the implicitly encoded vector Z. Its output is defined by the following Gaussian distribution: This branch is only used for model training and does not participate in the final inference prediction.

[0063] Distributed Decoder: This decoder predicts future trajectories from time m+1 to n using the same implicit encoded vector Z, through a GRU network and fully connected layers. Its output is also defined as: This branch is the core of the invention, used to provide future trajectory predictions.

[0064] 1.2 Model Training and Usage: Training is divided into two stages: First, the variational encoder is trained to learn accurate trajectory encoding and reconstruction; then, the encoder parameters are fixed, and the trajectory prediction decoder is trained separately. Upon final deployment, the system inputs the historical trajectories observed in real time into the encoder to obtain Z, and uses the prediction decoder to output future trajectory predictions, providing crucial information for subsequent decision-making.

[0065] Step 2, Operation Strategy Generation – Research on Dynamic Operation Strategies for Robots Based on Large-Scale Distributed Reinforcement Learning

[0066] 2.1 Simulation Environment Setup Based on the NVIDIA Isaac Sim platform, a simulation environment was built including a mobile robotic arm, a ping-pong table, and ping-pong balls. This environment was replicated 1000 times to create a large-scale parallel training scenario. Domain randomization was employed to randomly perturb the dynamic parameters such as the elastic coefficient of the paddle, the coefficient of friction, and the mass of the ball in groups of 10 environments, in order to improve the robustness of the strategy and its transferability from simulation to reality.

[0067] 2.2 Setting up a reinforcement learning framework State space: State observations are formed by fusing multiple types of information through a multilayer perceptron. : Where Z is the implicit trajectory representation vector obtained in step 1. Features of the historical pose and velocity sequence of the robot's end effector The coordinates of the target landing point area are given by [coordinates], and || represents vector concatenation. Action space: The target pose and velocity of the robot's end effector (racket) in the task space. Reward function: Designed based on the distance between the return ball's landing point and the target point, the return ball's velocity, and whether a collision or robot movement exceeds the limit.

[0068] Algorithm and Training: Implement the multi-agent TD3 algorithm based on value set evaluation. The optimization objective of the algorithm is to maximize the following expected reward: Where N is the number of agents in the parallel environment. The cumulative reward obtained by the i-th agent. For the corresponding weights, The policy network is to be learned. A centralized value function is used to evaluate the network and efficiently optimize it using data generated in a large-scale parallel environment.

[0069] 3. Whole-body coordinated control – a constraint optimization-based overall control algorithm for a mobile robotic arm 3.1 Trajectory Generation After the policy network outputs the target hitting action, a continuous and smooth motion trajectory with acceleration is generated between the current position of the robot's end effector and the target position using a fifth-order polynomial interpolation method. This continuous trajectory is then discretized and sampled to obtain a series of path points.

[0070] 3.2 Overall Kinematic Modeling A global kinematic model of the mobile robotic arm is established. By combining the Jacobian matrix of the mobile chassis and the geometric Jacobian matrix of the robotic arm, the global Jacobian matrix J of the system is obtained. This matrix establishes the mapping relationship from the robot's generalized velocity vector ξ to the end effector's task space velocity v, i.e. .

[0071] 3.3 Problem Formulation and Solution: For each path point on the trajectory, the expected task space velocity is... Given that the inverse kinematics solution is transformed into the following linear quadratic programming problem for real-time solution: Subject to constraints: Link pairs i,j used for all collision avoidance purposes in: It is a positive definite weight matrix used to adjust the weights of the generalized velocity. η is the operability index based on the robot's current configuration q. η is a coefficient used to balance the velocity norm and operability weights. These represent the lower and upper limits of the components of the generalized velocity. The minimum distance between links i and j must be greater than the safety threshold. .

[0072] By solving this optimization problem in real time, the coordinated control commands of the mobile chassis and the joints of the robotic arm that satisfy multiple constraints can be obtained.

[0073] It should be understood that the above embodiments are only for illustrating the technical concept and features of the present invention, and are intended to enable those skilled in the art to understand the content of the present invention and implement it accordingly. It should not be considered that the specific implementation of the present invention is limited to these descriptions. For those skilled in the art, several simple deductions or substitutions can be made without departing from the concept of the present invention. All equivalent changes or modifications made in accordance with the spirit and essence of the present invention should be covered within the protection scope of the present invention.

Claims

1. A method of dynamic operation of a robot, characterized by, Comprise the following steps: S1, environment characterization and prediction: build trajectory implicit representation and prediction model based on distributed decoder and gated recurrent unit based variational autoencoder, specifically comprising: Variational encoder: input is the history trajectory sequence of ping-pong from time 1 to m Encode by GRU network, output the mean μ and variance σ of Gaussian distribution defining the implicit encoding vector Z, which is calculated by the following reparameterization trick: where θ is the encoder network parameter, and denotes element-wise multiplication, Trajectory Reconstruction Decoder: This decoder reconstructs the trajectory from time 1 to m according to the hidden encoding vector Z through another set of GRU networks with its output defined by the following Gaussian distribution: This branch is only used for model training and does not participate in the final inference prediction. Distributed decoder: according to the same implicit encoding vector Z, through the GRU network and the full connection layer, the future trajectory of m+1 to n moment is predicted , the output is also defined as: , for providing future trajectory prediction; S2, operation strategy generation: constructing a reinforcement learning model, fusing the trajectory implicit representation vector obtained in step S1, the robot body state information and the task target point into the state observation value, and through large-scale distributed training, autonomously learning to generate the end effector target action of the robot in the task space; S3, whole body cooperative control: according to the end effector target action generated in step S2, a smooth motion trajectory of the end effector is generated in the task space by using the polynomial interpolation method; for each path point on the trajectory, a whole body cooperative controller based on nonlinear model predictive control is used to calculate the control instructions of the robot mobile chassis and each joint of the mechanical arm in real time.

2. The method of claim 1, wherein: The type training in the step S1 includes: first, training the variational encoder part to learn accurate trajectory encoding and reconstruction; then, fixing the parameters of the encoder, training the distributed decoder alone, and finally deploying the system, inputting the observed historical trajectory into the encoder to obtain Z, and outputting the future trajectory prediction using the prediction decoder to provide key information for subsequent decision-making.

3. The method of claim 1, wherein: The reinforcement learning model in the step S2 is trained by easy-to-difficult curriculum learning and adversarial disturbance framework, the robustness and generalization ability of the strategy are continuously strengthened by dynamically adjusting the task difficulty and introducing adversarial environmental changes, and finally the end effector target action of the robot in the task space is output.

4. The method of claim 3, wherein: The state observation value in the step S2 specifically includes: introducing curriculum learning and adversarial disturbance framework to train the reinforcement learning strategy, the learning strategy includes: training a curriculum from easy to difficult Primary: learn basic interception skills in an ideal environment, Intermediate: gradually increase the disturbance range of the dynamics parameters and introduce simple trajectory mutations; Advanced: introduce adversarial disturbance to learn how to slightly change the motion trajectory of the object so that the strategy of the robot remains robust even in the worst case.

5. The method of claim 1, wherein: The polynomial interpolation method in the step S3 is a quintic polynomial interpolation method, which is used to generate smooth trajectories with continuous acceleration.

6. The method of claim 5, wherein: For the generated end smooth motion trajectory, a whole-body collaborative controller based on nonlinear model predictive control is used in the step S3. The controller performs rolling optimization within a forward-looking time window based on the overall nonlinear dynamics model of the mobile manipulator, and solves the optimization problem of simultaneously satisfying the task space trajectory tracking, generalized velocity and acceleration limits, and self-collision avoidance multiple constraints in real time, so as to calculate the optimal collaborative control instructions of the robot mobile chassis and each joint of the manipulator.

7. The method of claim 6, wherein: The nonlinear model predictive control obtains the optimal control instruction sequence for the current and future period of time by solving a constrained nonlinear optimization problem, but only executes the first step, and then repeats the process in the next control cycle.

8. A robotic dynamic operating system for implementing the method of any one of claims 1-7, characterized by: Comprise: A visual perception module for collecting and processing image data to obtain a historical trajectory sequence of a moving object, the visual perception module comprising a high-speed binocular camera; A trajectory prediction module built-in variational autoencoder and distributed decoder based on gated recurrent unit, for executing the environment characterization and prediction step, outputting the future trajectory prediction of the moving object; A strategy decision module built-in multi-agent reinforcement learning model based on value centralized evaluation, running in a GPU accelerated large-scale parallel simulation environment, for executing the operation strategy generation step, outputting the target action of the end effector; A motion control module is configured to perform the whole-body coordinated control step, generate a smooth trajectory by using a quintic polynomial interpolation according to the target motion, and introduce a curriculum learning and adversarial perturbation framework to solve control commands of each joint of the mobile chassis and the robot arm in real time. A robot execution mechanism including an omnidirectional mobile chassis and a multi-degree-of-freedom collaborative robot arm is configured to receive the control commands and perform coordinated motion to complete a dynamic operation task.

9. An electronic device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that: The processor executes the program to implement the steps of the method according to any one of claims 1-7; the electronic device is an industrial computer or an embedded computing unit integrated with the trajectory prediction module, the policy decision module and the motion control module according to claim 8; and the electronic device communicates with and exchanges data with the visual perception module and the robot execution mechanism through a ROS robot operating system.

10. A computer readable storage medium having stored thereon a computer program, characterized in that: The program is executed by the processor to implement the steps of the method according to any one of claims 1-7.

Images