Robot adaptive trajectory generation method and system

Abstract

The invention discloses a robot adaptive trajectory generation method and system, and belongs to the technical field of intelligent robot control. The method comprises the following steps: firstly, extracting input-output joint distribution from multiple groups of expert demonstration trajectories, and constructing a probability reference trajectory containing a mean value and a covariance through GMM and GMR; and based on an information theory principle, a non-parameterized KMP prediction model is deduced, and a smooth trajectory and uncertainty thereof can be directly output for any high-dimensional input. On the basis, key adjustable parameters in the KMP model are defined as learnable variables, and the learnable variables are embedded into a reinforcement learning strategy framework. By setting a reward function and utilizing a reinforcement learning algorithm for online optimization, the generated trajectory can keep an expert demonstration style and can continuously improve a specific task performance index, adaptive trajectory generation of the robot in a complex dynamic scene is realized, and thus sample efficiency, task performance and operation safety are considered.

Description

Technical Field

[0001] This invention relates to the field of intelligent robot control technology, and in particular to a method and system for generating adaptive trajectories for robots. Background Technology

[0002] In the field of intelligent robots, the core challenge in achieving truly intelligent operations lies in enabling robots to quickly learn human skills and autonomously optimize themselves according to actual task requirements. Currently, the mainstream technical approaches fall into two main categories: imitation learning and reinforcement learning.

[0003] Imitation learning methods, such as Dynamic Motion Primitives (DMP), Probabilistic Motion Primitives (ProMP), and Kernelized Motion Primitives (KMP), can efficiently reproduce complex motor skills from a small number of expert demonstrations. Among them, KMP, due to its non-parametric nature, natural support for high-dimensional inputs, and flexible online constraint injection mechanism, has become one of the most advanced imitation learning tools. However, the fundamental limitation of imitation learning lies in its "passive reproduction" attribute—it can only learn "how to do it," but cannot understand "why to do it that way" or "how to do it better." When the demonstration itself is suboptimal, or when the task environment / goal changes, simple imitation often cannot guarantee optimal performance.

[0004] Reinforcement learning (RL) offers a paradigm of "active optimization." Through trial-and-error interaction with the environment and reward feedback, RL can learn policies that maximize long-term returns. Deep reinforcement learning has achieved success in many complex control tasks. However, its application in robotics faces serious challenges: the high-dimensional continuous action space leads to extremely low sample efficiency; random exploration may cause safety risks; and the training process is unstable, making it difficult to converge to a reliable policy.

[0005] To combine the advantages of both approaches, researchers have proposed various fusion schemes, such as initializing RL policies using behavioral cloning (BC) or adding imitation regularization terms to the RL loss function. However, these methods typically treat the imitation module as a fixed, black-box prior, failing to fully utilize the rich, interpretable, structured parameters within advanced imitation models like KMP. Existing techniques lack a mechanism that allows the "style" or "adaptability" of the KMP model itself to be directly optimized in RL, thereby achieving precise performance improvements while preserving the demonstrative manifold. In summary, existing imitation learning and reinforcement learning methods still have significant shortcomings in the following aspects:

[0006] Optimization and prior knowledge are disconnected: imitation learning provides prior knowledge, and reinforcement learning is responsible for optimization, but there is a lack of deeply coupled parameters between the two.

[0007] Sample efficiency bottleneck: Pure reinforcement learning methods require a lot of trial and error interactions to converge on real robots. Each interaction is time-consuming and risky, resulting in extremely low sample efficiency, making it difficult to apply in practical tasks.

[0008] Task performance blind spot: Pure imitation learning methods cannot be optimized for specific engineering indicators (such as energy consumption, accuracy, compliance);

[0009] Safety and stability: Random exploration in RL is high-risk in physical systems and lacks safety constraints from the model.

[0010] Therefore, there is an urgent need for a robot adaptive trajectory generation method and system that can use the KMP structured model as a framework and fine-tune its intrinsic parameters through reinforcement learning to achieve high efficiency, high robustness, and task-oriented characteristics. Summary of the Invention

[0011] This invention provides a robot adaptive trajectory generation method and system, which uses the KMP structured model as the skeleton and fine-tunes its intrinsic parameters through reinforcement learning to achieve adaptive trajectory generation of the robot in complex dynamic scenarios, thereby balancing sample efficiency, task performance and operational safety.

[0012] To achieve the above objectives, the technical solution adopted by the present invention is as follows:

[0013] This invention provides a method for generating adaptive trajectories for robots, comprising:

[0014] S1: Obtain multiple sets of expert demonstration trajectories, wherein the expert demonstration trajectory consists of an input variable sequence and a corresponding robot output variable sequence;

[0015] S2: Perform Gaussian mixture modeling on the joint distribution of the expert demonstration trajectories, and extract the probability reference trajectory through Gaussian mixture regression. The probability reference trajectory includes the output mean vector and covariance matrix corresponding to each input point.

[0016] S3: Construct a kernel function, and based on the probability reference trajectory and the kernel function, derive a non-parametric kernelized motion primitive prediction model by minimizing the KL divergence, which is used to predict the mean and covariance of the output for any query input;

[0017] S4: Identify and extract a set of adjustable parameters sensitive to trajectory performance from the kernelized motion primitive model as learnable variables, wherein the learnable variables include at least one of kernel function bandwidth, covariance scaling factor, and local coordinate system offset;

[0018] S5: Encapsulate the nucleated motion primitive model into a policy function that is differentiable with respect to the learnable variables, and define a reward function according to the target task;

[0019] S6: Optimize the learnable variables by interacting with the environment through reinforcement learning algorithms and utilizing the obtained reward signals;

[0020] S7: Apply the optimized learnable variables to the kernelized motion primitive model to generate robot motion trajectories that meet the task performance indicators in real time.

[0021] Furthermore, in S1, the sequence of input variables includes at least one of time, robot joint position, robot end effector speed, acceleration, pose, external sensor signals, or user input.

[0022] Furthermore, in S3, the kernel function includes a linear kernel, a polynomial kernel, or a radial basis function kernel.

[0023] Furthermore, in S6, the reinforcement learning algorithm includes proximal policy optimization, soft actor-commentator, or deep deterministic policy gradient.

[0024] Furthermore, when a new task constraint is received, the new task constraint is encoded into a Gaussian distribution and added to the reference trajectory database as a new data point; wherein, the confidence weight of the covariance matrix of the Gaussian distribution is defined as an additional learnable variable and optimized together in step S7.

[0025] The present invention also provides a robot adaptive trajectory generation system, comprising:

[0026] Acquisition module: used to acquire multiple sets of expert demonstration trajectories, wherein the expert demonstration trajectory consists of an input variable sequence and a corresponding robot output variable sequence;

[0027] Gaussian Mixture Modeling Module: Used to perform Gaussian mixture modeling on the joint distribution of the expert demonstration trajectories, and extract probabilistic reference trajectories through Gaussian mixture regression. The probabilistic reference trajectories include the output mean vector and covariance matrix corresponding to each input point.

[0028] Kernel function construction module: used to construct kernel functions, and based on the probability reference trajectory and the kernel functions, to derive a non-parametric kernelized motion primitive prediction model by minimizing KL divergence, used to predict the mean and covariance of the output for any query input;

[0029] Identification and extraction module: used to identify and extract a set of adjustable parameters sensitive to trajectory performance from the kernelized motion primitive model as learnable variables, wherein the learnable variables include at least one of kernel function bandwidth, covariance scaling factor, and local coordinate system offset;

[0030] Definition module: used to encapsulate the nucleated motion primitive model into a policy function that is differentiable with respect to the learnable variables, and to define a reward function according to the target task;

[0031] Optimization module: used to interact with the environment through reinforcement learning algorithms and optimize the learnable variables using the obtained reward signals;

[0032] Trajectory generation module: used to apply the optimized learnable variables to the kernelized motion primitive model, and to generate robot motion trajectories that meet the task performance indicators in real time.

[0033] Compared with the prior art, the technical solution disclosed in this invention has the following beneficial effects:

[0034] Compared with existing technologies, this invention constructs a five-layer closed-loop learning architecture that includes expert trajectory teaching, probabilistic reference trajectory construction, KMP non-parametric modeling, reinforcement learning policy construction, reinforcement learning optimization, and online execution. The key adjustable parameters of the KMP model are defined as the output variables of the RL policy, and a differentiable, structured, task-driven closed-loop optimization architecture is constructed, enabling the robot to efficiently inherit expert skills and autonomously improve its task performance. Attached Figure Description

[0035] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0036] Figure 1 This is a schematic diagram of the robot adaptive trajectory generation method provided in an embodiment of the present invention. Detailed Implementation

[0037] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0038] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0039] This invention provides a robot adaptive trajectory generation method and system. By constructing a five-layer closed-loop learning architecture that includes expert trajectory teaching, probabilistic reference trajectory construction, KMP nonparametric modeling, reinforcement learning policy construction, reinforcement learning optimization, and online execution, the key adjustable parameters of the KMP model are defined as the output variables of the RL policy. A differentiable, structured, task-driven closed-loop optimization architecture is constructed, enabling the robot to efficiently inherit expert skills and autonomously improve its task performance.

[0040] like Figure 1 As shown, this embodiment of the invention provides a robot adaptive trajectory generation method, including:

[0041] S1: Acquire multiple sets of expert demonstration trajectories. The expert demonstration trajectory consists of an input variable sequence and a corresponding robot output variable sequence; that is, expert teaching trajectory data acquisition and preprocessing.

[0042] In this embodiment of the invention, raw motion trajectory data of the robot during the execution of a specific task is acquired through manual expert guidance or sensor recording, such as end-effector pose, velocity, acceleration, and corresponding timestamps. Key features are extracted and standardized from the collected data to form a standardized dataset for subsequent modeling.

[0043] Expert instructors guide the robot's end effector to complete preset tasks (such as obstacle avoidance grinding, curved surface welding, etc.) through teaching drag, teleoperation, or virtual reality guidance. The system collects the robot's motion data in the task space in real time, including:

[0044] 1. Timestamp ;

[0045] 2. Actuator end effector position (ZYX);

[0046] 3. Joint angle ;

[0047] 4. External sensor signals (such as force / torque, visual information);

[0048] The raw trajectory data is represented as a sequence. Input variables Can be time Environmental information or user commands, output variables These are robot control variables (such as actuator end-effector pose or joint angle).

[0049] S2: Gaussian mixture modeling is performed on the joint distribution of the expert demonstration trajectory, and a probabilistic reference trajectory is extracted by Gaussian mixture regression. The probabilistic reference trajectory contains the output mean vector and covariance matrix corresponding to each input point, that is, the probabilistic reference trajectory is constructed based on GMM / GMR.

[0050] In this embodiment of the invention, GMM is used for the joint input-output dataset. Probabilistic modeling is performed to obtain the joint probability distribution of maturity. Then, through GMR, at each input point... Calculate the conditional output distribution at the location The analytical solution is obtained, thus yielding a vector containing the mean. Covariance Matrix The probability reference trajectory.

[0051] Specifically, GMM is used on the joint dataset. Modeling to obtain its probability density function Next, the conditional distribution was calculated analytically using GMR. To obtain each input point The corresponding conditional mean vector and conditional covariance matrix This allows us to obtain a probability reference trajectory containing complete statistical information. .

[0052] S3: Construct a kernel function, and based on the probabilistic reference trajectory and the kernel function, derive a non-parametric kernelized motion primitive prediction model by minimizing the KL divergence, which is used to predict the mean and covariance of the output for any query input; that is, a non-parametric KMP prediction model.

[0053] In this embodiment of the invention, trajectory generation is viewed as finding a mapping function. The goal is to make the output distribution as close as possible to the aforementioned probability reference trajectory. Therefore, the optimization objective is defined as minimizing the KL divergence between the two. Through mathematical derivation, this problem can be decomposed into two subproblems: mean prediction and covariance prediction. Introducing the kernel trick yields closed-form solutions for both mean and covariance prediction. This model does not require explicit definition of basis functions; therefore, it naturally supports inputs of arbitrary dimensions and can simultaneously output the trajectory mean and the complete covariance.

[0054] Specifically, this invention models trajectory generation as minimizing the prediction distribution. Compared with the reference distribution The optimization problem of KL divergence between KL divergences. This is addressed by introducing the RBF kernel function. The closed-form KMP prediction mean can be derived. and predicted covariance This model does not require explicit basis functions, supports inputs of any dimension, and its output contains complete uncertainty information.

[0055] S4: Identify and extract a set of adjustable parameters sensitive to trajectory performance from the kernelized motion primitive model as learnable variables. The learnable variables include at least one of kernel function bandwidth, covariance scaling factor, and local coordinate system offset.

[0056] S5: Encapsulate the nucleated motion primitive model into a differentiable policy function with respect to learnable variables, and define the reward function according to the target task; that is, define learnable variables and construct a differentiable policy based on KMP.

[0057] The key innovation in this invention lies in identifying a subset of parameters in the KMP model that have a significant impact on task performance and defining them as learnable variables. These variables include, but are not limited to:

[0058] Kernel function bandwidth ( The sensitivity of the control model to changes in input affects the smoothness and flexibility of the trajectory.

[0059] Covariance scaling factor ( ): Adjust the output covariance to control the smoothness or exploration level of trajectory execution.

[0060] Local coordinate system offset ( ): Fine-tune the origin of the local-KMP coordinate system to enhance its adaptability to external disturbances.

[0061] The current robot state (such as joint angles, object pose, force sensor readings, etc.) is mapped to the input variables of the KMP model. The KMP model then outputs a probability distribution of an action (the expected position at the next moment) based on the current learnable variable values. This process constitutes a structured strategy that is differentiable about the learnable variables.

[0062] Specifically, in this embodiment, to achieve fine-grained control over the trajectory generation process, this invention identifies a subset of parameters from the KMP model that have a significant impact on task performance and possess physical interpretability, and defines them as learnable variables. Specifically, the following two types of core parameters are selected:

[0063] Kernel function bandwidth This parameter controls the RBF core. Its scope of application. Larger. Smaller values ​​make the model output smoother but result in a sluggish response. Values ​​make the trajectory more flexible but may introduce high-frequency jitter. Through optimization... The value can adaptively balance the smoothness of the trajectory with its sensitivity to changes in input.

[0064] Covariance scaling factor The predicted covariance matrix output by the KMP model This directly reflects the uncertainty or compliance of trajectory execution. This invention introduces a scalar scaling factor. Let the covariance actually used for policy sampling be set as . The size determines the "exploration level" of the robot's movements: When the size is larger, the strategy is more exploratory, which is conducive to achieving compliant control in contact tasks; When the size is smaller, the strategy is more conservative, which is beneficial for high-precision positioning tasks.

[0065] Therefore, a learnable variable vector is defined as Based on this, reinforcement learning strategies are constructed. At every decision-making moment The system first obtains the current environmental status. This state includes: the current Cartesian pose, velocity, and acceleration of the robot's end effector, as well as readings from the six-dimensional force / torque sensors. Subsequently, the state... Mapping to the input variables of the KMP model The current and learnable variables Inputting the data into the KMP model yields a Gaussian distribution:

[0066]

[0067] in:

[0068] : Represents the expected end position at the next moment;

[0069] Since both the mean and covariance predictions of KMP can be expressed as analytic functions of the kernel matrix, and the elements of the kernel matrix are... The smooth function, therefore the strategy is about It is completely differentiable and meets the basic requirements of policy gradient algorithms.

[0070] S6: Through the interaction with the environment via reinforcement learning algorithms, learnable variables are optimized using the obtained reward signals; that is, parameter optimization driven by reinforcement learning.

[0071] In this embodiment of the invention, a reward function is designed based on specific task objectives (such as grasping success rate, assembly accuracy, and energy consumption minimization). Advanced reinforcement learning algorithms (such as PPO and SAC) are employed to allow the robot to perform tasks in simulated or real-world environments. The algorithm collects state, action, and reward sequences, calculates the policy gradient, and iteratively updates learnable variables to maximize cumulative rewards. This process achieves autonomous fine-tuning of the KMP's "intrinsic style."

[0072] Specifically, the Proximal Policy Optimization (PPO) algorithm is used to optimize learnable variables, specifically applied to robotic casting deburring and surface polishing scenarios. This task requires the robot's end effector to move along a complex curved surface trajectory while maintaining stable contact force to ensure polishing quality and avoid workpiece damage or tool overload. The process includes: reward function design, PPO parameter initialization, data collection, advantage estimation, policy updating, and online deployment and practical application.

[0073] S7: Apply the optimized learnable variables to the kernelized motion primitive model to generate robot motion trajectories that meet the task performance indicators in real time; that is, real-time control system integration.

[0074] In this embodiment of the invention, optimized learnable variables are embedded into the KMP model for real-time trajectory generation. The system can respond to external interactions on a millisecond timescale.

[0075] In S1, the input variable sequence includes at least one of time, robot joint position, robot end effector velocity, acceleration, pose, external sensor signals, or user input. The key features are not limited to time series, robot end effector Cartesian space pose, velocity information, acceleration, etc., but may also include environmental perception signals (such as force sensors), human-machine interaction input, and multi-sensor fusion data to provide a richer and more comprehensive input dimension.

[0076] Gaussian Mixture Model (GMM) is a probabilistic model used for cluster analysis. It assumes that all data points are composed of several Gaussian distributions. The model parameters are iteratively optimized using the Expectation-Maximization (EM) algorithm. In the E-step, the posterior probability of each data point with respect to each Gaussian component is calculated. In the M-step, the model parameters (mean vector, covariance matrix, and mixing weights) are re-estimated using the posterior probabilities calculated in the E-step, until the model converges or reaches the preset number of iterations.

[0077] In S3, kernel functions include linear kernels, polynomial kernels, and radial basis function (RBF) kernels. The choice of kernel function is flexible, allowing for different forms to be selected based on specific application scenarios, such as linear kernels, polynomial kernels, and RBF kernels. The choice of kernel function directly affects the performance of the KMP model; a suitable kernel function can enhance the model's ability to support extrapolation tasks and improve its learning performance on complex input patterns.

[0078] In S6, reinforcement learning algorithms include proximal policy optimization, soft actor-critic, or deep deterministic policy gradient. The reinforcement learning optimization process is first performed in a high-fidelity simulation environment, and then transferred to a real robot system after the policy converges to ensure training safety and efficiency.

[0079] The method also includes encoding new task constraints into a Gaussian distribution upon receipt and adding them as new data points to the reference trajectory database. The confidence weights of the covariance matrix of the Gaussian distribution are defined as additional learnable variables and optimized in S7. That is, this method simultaneously supports online constraint injection: new constraints are encoded into a Gaussian distribution, and the weights of their covariance (i.e., confidence) can also be used as learnable variables for optimization within the RL framework, achieving more intelligent constraint fusion. In the online constraint modulation mechanism, the confidence of new constraints (i.e., the size of the covariance matrix) is no longer manually preset but is automatically adjusted by reinforcement learning based on the task context (such as obstacle hazard level and pathpoint importance) as part of the learnable variables.

[0080] This invention also provides a robot adaptive trajectory generation system, comprising:

[0081] Acquisition module: Used to acquire multiple sets of expert demonstration trajectories. The expert demonstration trajectory consists of an input variable sequence and a corresponding robot output variable sequence.

[0082] The embodiments in this specification specifically include the following steps:

[0083] S1.1 Establishing the Teaching Dataset: The raw motion data of the robot's end effector in the task space is obtained through teaching methods (such as drag-and-drop teaching and teleoperation teaching). This dataset records the pose characteristics of the end effector over time, and its general vector form is:

[0084]

[0085] in, It is a multidimensional vector. Represents a timestamp or query index. This represents the n-dimensional position and orientation information of the robot's end effector in the task space.

[0086] S1.2 Multi-dimensional sensors synchronously acquire complete status information:

[0087] Joint information acquisition: The angles of each joint are measured in real time using an optical encoder or rotary transformer installed at each joint. and angular velocity .

[0088] End-effector pose calculation: Joint information obtained using joint information and By combining known robot kinematics models, the pose of the end effector in the task space is obtained in real time through forward kinematics calculations. and speed The details are as follows:

[0089]

[0090] in, This represents the Jacobian matrix.

[0091] Gaussian Mixture Modeling Module: Used to perform Gaussian mixture modeling on the joint distribution of expert demonstration trajectories, and extract probabilistic reference trajectories through Gaussian mixture regression. The probabilistic reference trajectory contains the output mean vector and covariance matrix corresponding to each input point.

[0092] The embodiments in this specification specifically include the following steps:

[0093] S2.1 uses GMM to analyze the joint dataset. The model is defined by the following probability density function:

[0094]

[0095] in:

[0096] : Represents a joint dataset The probability density distribution;

[0097] : Represents the first The prior probabilities of a Gaussian distribution, also known as the mixture weights, satisfy the following: .

[0098] : Represents a Gaussian distribution;

[0099] S2.2 Analyzes the conditional distribution using GMR. To obtain each input point The corresponding conditional mean and covariance:

[0100]

[0101]

[0102] in: Represents activation weight

[0103]

[0104] A probability reference trajectory containing complete statistical information can be obtained from equations (4) and (5). .

[0105] Kernel function construction module: used to construct kernel functions, and based on the probabilistic reference trajectory and kernel functions, to derive a non-parametric kernelized motion primitive prediction model by minimizing KL divergence, used to predict the mean and covariance of the output for any query input;

[0106] The embodiments in this specification specifically include the following steps:

[0107] S3.1 This module models trajectory generation as minimizing the prediction distribution. Compared with the reference distribution The problem of optimizing the KL divergence between KL divergences. This is addressed by introducing a kernel function:

[0108]

[0109] From equation (7), the closed-form KMP prediction mean and covariance can be derived:

[0110]

[0111]

[0112] in:

[0113] : Represents the kernel matrix between training input points, ;

[0114] : Represents the query point Kernel vectors of training points;

[0115] ;

[0116] ;

[0117] Regularization factor to prevent the occurrence of singular matrices;

[0118] The KMP prediction model does not require explicit basis functions, supports inputs of arbitrary dimensions, avoids the curse of dimensionality under high-dimensional inputs, and outputs complete uncertainty information.

[0119] The identification and extraction module is used to identify and extract a set of adjustable parameters sensitive to trajectory performance from the kernelized motion primitive model as learnable variables. The learnable variables include at least one of kernel function bandwidth, covariance scaling factor, and local coordinate system offset. The definition module is used to encapsulate the kernelized motion primitive model into a policy function that is differentiable with respect to the learnable variables, and to define the reward function according to the target task.

[0120] These two modules are the key bridge connecting Structured Imitation Prior (KMP) and Goal-Oriented Optimization (RL). Their function is to transform the KMP model into a parameterized, differentiable, task-aware reinforcement learning policy. This module can be further divided into four collaborative functional units:

[0121] S4.1 Learnable variable extraction:

[0122] This unit is responsible for identifying, filtering, and configuring a set of adjustable parameters that are highly sensitive to task performance and have physical meaning from the constructed KMP model, formally defining them as a vector of learnable variables. Taking the casting grinding scenario as an example, this unit has the following definable variables:

[0123] Kernel function bandwidth :RBF core The core hyperparameters. The value directly determines the corresponding scale of KMP for the input space (casting surface curvature, normal variation). This unit initializes it with the root mean square value of the teaching data input dimension and sets the search range. This helps prevent underfitting or overfitting.

[0124] Covariance scaling factor Used to dynamically adjust the size of the KMP output covariance matrix; the actual policy covariance is... . The size determines the "exploration level" of the robot's movements: When the size is larger, the strategy is more exploratory, which is conducive to achieving compliant control in contact tasks; When the size is smaller, the strategy is more conservative, which is beneficial for high-precision positioning tasks.

[0125] Local coordinate system offset If the local-KMP extrapolation mechanism is used, it can be... Included To compensate for casting clamping errors or visual positioning deviations.

[0126] S4.2 State-Input Mapping Unit:

[0127] This unit is responsible for processing the environment state within the reinforcement learning framework. The input variables required to convert to the KMP model .state Provided in real time by various sensors on the robot, including the real-time pose of the robot's end effector. The angles of each joint of the robot angular velocity angular acceleration Six-dimensional mechanical sensor readings .

[0128] S4.3KMP Strategy Encapsulation Unit:

[0129] This unit is the core computational engine of this module. It receives the current parameters from the learnable variable extraction unit. The kernel originates from the query input of the state-input mapping unit. It calls the KMP model already built in the probability trajectory modeling module, performs forward inference, and generates an action instruction in the form of a Gaussian distribution:

[0130]

[0131] This unit can implement gradient tracking of the entire computation graph using the automatic differentiation framework TensorFlow, ensuring that the policy output is related to the gradient of the entire computation graph. gradient It can be efficiently computed, providing a foundation for subsequent PPO policy gradient updates.

[0132] S4.4 Policy Interface and Caching Unit:

[0133] This unit provides a standardized API for the Policy Optimization Unit (PPO trainer) to use. Its main functions include:

[0134] Policy sampling interface: During the training phase, based on the current... and state ,from Mid-sampling action And record Used for subsequent calculation of the dominance function.

[0135] Deterministic inference interface: During the deployment phase, it directly returns the mean. As a control instruction, randomness is disabled to ensure execution stability.

[0136] Parameter synchronization cache: maintenance The latest copy ensures consistency between training and inference.

[0137] Through the close collaboration of the four functional units mentioned above, the reinforcement learning policy construction module successfully transforms the originally static KMP model into a dynamic, learnable, task-driven intelligent policy, laying a solid foundation for subsequent performance optimization based on PPO. This design not only retains the trajectory smoothness and uncertainty modeling advantages of KMP, but also endows it with the endogenous ability to continuously evolve through task feedback, which is one of the core innovations that distinguishes this invention from existing technologies.

[0138] Optimization module: Used to interact with the environment through reinforcement learning algorithms and optimize learnable variables using the reward signals obtained.

[0139] The embodiments in this specification specifically include the following steps:

[0140] This module is the core of this invention for achieving autonomous task performance. Its goal is to use reinforcement learning algorithms to drive learnable variables using task reward signals. This module, which involves iterative updates, consists of the following functional units:

[0141] S5.1 Reward Function Configuration Unit:

[0142] Contact force control reward:

[0143]

[0144] in:

[0145] In this embodiment, the normal grinding force is used. , , , .

[0146] Tracking accuracy bonus:

[0147]

[0148] in:

[0149] To track the weighting coefficients, take ;

[0150] This represents the actual position and pose of the robot's end effector in the world coordinate system.

[0151] This is the theoretically desirable pose;

[0152] The Euclidean distance between the actual pose and the theoretical pose of the robot's end effector;

[0153] Task completion reward:

[0154]

[0155] Security penalty:

[0156]

[0157] This unit encodes equations (12) to (15) into executable functions and outputs a scalar reward. To the strategy optimization unit.

[0158] S5.2 Simulation Interactive Environment Unit

[0159] This unit constructs a simulation platform to reproduce the dynamic characteristics of a real polishing system. This unit receives action commands from the reinforcement learning policy construction module. The virtual robot is driven to perform the polishing action and then returns to the next state. With rewards to form a complete Update the tuple for use in strategy optimization.

[0160] S5.3 Strategy Optimization Unit:

[0161] This unit implements the complete training process of the Proximal Policy Optimization (PPO) algorithm, specifically including:

[0162] Data buffer management: Collect trajectory segments from multiple parallel simulations and build an experience playback buffer.

[0163] Dominance function estimation: Generalized dominance estimation (GAE) is used. , Calculate the dominance value at each time step. .

[0164] Strategy loss calculation: Constructing the PPO pruning objective function;

[0165]

[0166] in:

[0167] represent Expectations under the passage of time;

[0168] Represents the probability ratio between the old and new strategies;

[0169] Represents the dominant function;

[0170] This represents the probability of cropping. .

[0171] Parameter update: Using the Adam optimizer (learning rate) )right Multiple rounds of iterative training are conducted, with multiple mini-batch samples sampled in each round for updating.

[0172] Convergence criterion: When the average cumulative reward fluctuation over 10 consecutive rounds is less than 2% and the polishing quality index is met, the strategy is considered converged, and the parameters are output. .

[0173] Trajectory generation module: Used to apply optimized learnable variables to the kernelized motion primitive model to generate robot motion trajectories that meet task performance indicators in real time.

[0174] S6.1 This module is responsible for receiving the optimized parameters, and KMP can then generate a better trajectory based on the updated parameters.

[0175] Furthermore, in S1.1, the teaching method is not limited to physical dragging or teleoperation, but can be extended to virtual reality guidance, offline programming simulation playback, or digital twin teaching based on human motion capture. The collected raw data vectors timestamps in It can be replaced with any task progress variable, such as path parameters. This decouples trajectory generation from its dependence on absolute time, thereby improving its adaptability to different execution speeds.

[0176] Furthermore, the forward kinematics calculation in S1.2 not only outputs the end effector pose. With speed It can also calculate acceleration simultaneously. Incorporating acceleration information into the teaching dataset can significantly improve the ability of subsequent GMM modeling to represent dynamic characteristics, especially suitable for high-speed or high-precision tasks (such as polishing, grinding and assembly).

[0177] Furthermore, in S2.2, activation weights Not only reflects the first A Gaussian component in the input The confidence level at each point also implicitly encodes the temporal continuity of the trajectory. Because the Gaussian kernel has local support properties, the activation weights change smoothly between adjacent time points, naturally suppressing high-frequency oscillations in the GMR results, allowing the generation of physically feasible trajectories without additional post-processing.

[0178] Furthermore, S3.1 uses the RBF kernel function, but other kernel functions such as the multinomial kernel and Mahalanobis distance kernel can also be used. The choice of kernel function directly affects the generalization ability of the KMP model. In particular, when the input contains multi-scale features (such as time + position + force), multi-scale kernel functions can be used depending on the input features.

[0179] Furthermore, solving equations (8) and (9) of the KMP prediction model in S3.1 involves matrices. Inverse calculation. To improve computational efficiency and ensure numerical stability, Cholesky decomposition is often used in practical deployments instead of direct inversion. This method can reduce computational complexity. Down to This effectively avoids numerical overflow caused by ill-conditioned matrices.

[0180] Furthermore, the trajectory generated in S6.1 is not limited to position information; it can also include information such as velocity, acceleration, and contact force. If force-position hybrid control is required, simply adjust the covariance matrix... The mapping is a compliance stiffness matrix, which enables the invention to be applied in fields such as grinding and polishing.

[0181] The basic principles of the present invention have been described above with reference to specific embodiments. However, it should be noted that the advantages, benefits, and effects mentioned in the present invention are merely examples and not limitations, and should not be considered as essential features of each embodiment of the present invention. Furthermore, the specific details disclosed above are for illustrative and facilitative purposes only, and are not limitations. These details do not limit the present invention to the necessity of employing the aforementioned specific details.

[0182] The block diagrams of devices, apparatuses, devices, and systems involved in this invention are merely illustrative examples and are not intended to require or imply that they must be connected, arranged, or configured in the manner shown in the block diagrams. As those skilled in the art will recognize, these devices, apparatuses, devices, and systems can be connected, arranged, and configured in any manner. Words such as “comprising,” “including,” “having,” etc., are open-ended terms meaning “including but not limited to,” and are used interchangeably with them. The terms “or” and “and” as used herein refer to the terms “and / or,” and are used interchangeably with them unless the context clearly indicates otherwise. The term “such as” as used herein refers to the phrase “such as but not limited to,” and is used interchangeably with it.

[0183] It should also be noted that in the apparatus, device, and method of the present invention, the components or steps can be disassembled and / or recombined. These disassemblies and / or recombinations should be considered as equivalent solutions of the present invention.

[0184] The above description of the disclosed aspects is provided to enable any person skilled in the art to make or use the invention. Various modifications to these aspects will be readily apparent to those skilled in the art, and the general principles defined herein can be applied to other aspects without departing from the scope of the invention. Therefore, the invention is not intended to be limited to the aspects shown herein, but rather to be carried out within the widest scope consistent with the principles and novel features disclosed herein.

[0185] It should be understood that the qualifying terms "first", "second", "third", "fourth", "fifth" and "sixth" used in the description of the embodiments of the present invention are only used to more clearly illustrate the technical solutions and are not intended to limit the scope of protection of the present invention.

[0186] The above description has been given for purposes of illustration and description. Furthermore, this description is not intended to limit the embodiments of the invention to the forms disclosed herein. Although numerous exemplary aspects and embodiments have been discussed above, those skilled in the art will recognize certain variations, modifications, alterations, additions, and sub-combinations therein.

Claims

1. A method for generating adaptive trajectories for robots, characterized in that, include: S1: Obtain multiple sets of expert demonstration trajectories, wherein the expert demonstration trajectory consists of an input variable sequence and a corresponding robot output variable sequence; S2: Perform Gaussian mixture modeling on the joint distribution of the expert demonstration trajectories, and extract the probability reference trajectory through Gaussian mixture regression. The probability reference trajectory includes the output mean vector and covariance matrix corresponding to each input point. S3: Construct a kernel function, and based on the probability reference trajectory and the kernel function, derive a non-parametric kernelized motion primitive prediction model by minimizing the KL divergence, which is used to predict the mean and covariance of the output for any query input; S4: Identify and extract a set of adjustable parameters sensitive to trajectory performance from the kernelized motion primitive model as learnable variables, wherein the learnable variables include at least one of kernel function bandwidth, covariance scaling factor, and local coordinate system offset; S5: Encapsulate the nucleated motion primitive model into a policy function that is differentiable with respect to the learnable variables, and define a reward function according to the target task; S6: Optimize the learnable variables by interacting with the environment through reinforcement learning algorithms and utilizing the obtained reward signals; S7: Apply the optimized learnable variables to the kernelized motion primitive model to generate robot motion trajectories that meet the task performance indicators in real time.

2. The robot adaptive trajectory generation method according to claim 1, characterized in that, In step S1, the sequence of input variables includes at least one of time, robot joint position, robot end effector speed, acceleration, pose, external sensor signals, or user input.

3. The robot adaptive trajectory generation method according to claim 1, characterized in that, In S3, the kernel function includes a linear kernel, a polynomial kernel, or a radial basis function kernel.

4. The robot adaptive trajectory generation method according to claim 1, characterized in that, In S6, the reinforcement learning algorithm includes proximal policy optimization, soft actor-commentator, or deep deterministic policy gradient.

5. The robot adaptive trajectory generation method according to claim 1, characterized in that, It also includes encoding the new task constraint into a Gaussian distribution form when a new task constraint is received, and adding it as a new data point to the reference trajectory database; wherein the confidence weight of the covariance matrix of the Gaussian distribution is defined as an additional learnable variable and optimized together in step S7.

6. A robot adaptive trajectory generation system, characterized in that, include: Acquisition module: used to acquire multiple sets of expert demonstration trajectories, wherein the expert demonstration trajectory consists of an input variable sequence and a corresponding robot output variable sequence; Gaussian Mixture Modeling Module: Used to perform Gaussian mixture modeling on the joint distribution of the expert demonstration trajectories, and extract probabilistic reference trajectories through Gaussian mixture regression. The probabilistic reference trajectories include the output mean vector and covariance matrix corresponding to each input point. Kernel function construction module: used to construct kernel functions, and based on the probability reference trajectory and the kernel functions, to derive a non-parametric kernelized motion primitive prediction model by minimizing KL divergence, used to predict the mean and covariance of the output for any query input; Identification and extraction module: used to identify and extract a set of adjustable parameters sensitive to trajectory performance from the kernelized motion primitive model as learnable variables, wherein the learnable variables include at least one of kernel function bandwidth, covariance scaling factor, and local coordinate system offset; Definition module: used to encapsulate the nucleated motion primitive model into a policy function that is differentiable with respect to the learnable variables, and to define a reward function according to the target task; Optimization module: used to interact with the environment through reinforcement learning algorithms and optimize the learnable variables using the obtained reward signals; Trajectory generation module: used to apply the optimized learnable variables to the kernelized motion primitive model, and to generate robot motion trajectories that meet the task performance indicators in real time.

Images