Robot learning control method and device based on dynamic confidence domain and motion primitive intervention, medium and equipment

Abstract

This application provides a robot learning control method, device, medium, and equipment based on dynamic trust region and motion primitive intervention, including: inputting the robot's current state into a policy network to generate preliminary action commands; dynamically calculating a non-uniform trust region threshold based on the robot's current end-effector pose, comparing the pose deviation of the current end-effector pose relative to the target point with the non-uniform trust region threshold to determine the intervention trigger; generating a smooth motion trajectory in real time based on the dynamic motion primitive model for the current end-effector pose and the target point pose, and outputting the final action command; sending the final action command to the robot for execution, storing the new state and reward signal after the action execution in an experience playback buffer, and training and updating the policy network based on the data in the experience playback buffer. This significantly improves the safety, reproducibility, and autonomous learning capability of robot reinforcement learning training, forming a positive feedback adaptive learning loop.

Description

Technical Field

[0001] This application relates to the field of robot learning and control technology, and in particular to robot learning and control methods, devices, media and equipment based on dynamic trust regions and motion primitive intervention. Background Technology

[0002] In reinforcement learning (RL) training for autonomous robot tasks (such as precision assembly, flexible grasping, and multi-step collaboration), purely autonomous exploration faces fundamental bottlenecks such as low sample efficiency, high safety risks, and huge trial-and-error costs on real machines. The existing process is as follows: First, the operator provides a small number of high-quality task demonstrations using a spatial mouse or similar device to initialize the policy network. Then, during the autonomous execution of the policy, the operator monitors the robot's status in real time. When a collision, deviation from the target, or getting stuck in a local invalid loop is predicted, the operator immediately takes over control and manually corrects the actions. All executed actions (including intervention actions) are stored in an experience replay buffer for subsequent iterative optimization of the policy network. However, this approach suffers from the following technical problems: 1. High subjectivity and lack of quantifiability in intervention decisions: Whether to intervene depends entirely on the operator's instantaneous experience, lacking objective judgment criteria related to the robot's state (such as end-effector pose and force feedback). This leads to significant differences in intervention timing among different operators, making the training process difficult to reproduce and resulting in large fluctuations in strategy performance. 2. Inconsistent quality of intervention execution: Manual intervention actions are affected by operator fatigue, proficiency, and reaction delay, easily producing jitter, overshoot, or non-smooth trajectories. This not only reduces the safety of single interventions but also injects low-quality training data into the strategy network, slowing down the convergence speed. 3. Static and fixed intervention strategies that cannot evolve with strategy capabilities: Intervention frequency thresholds and safety tolerances are usually preset fixed values, unable to respond to the gradually improving capabilities of the strategy network during training. Therefore, a technical solution is urgently needed to construct a quantifiable, reproducible, and adaptive closed-loop system that integrates intervention decision-making, execution, and strategy evolution to improve the accuracy of robot learning and control. Summary of the Invention

[0003] In view of this, the purpose of this application is to provide a robot learning control method, device, medium and equipment based on dynamic trust region and motion primitive intervention. By constructing "dynamic trust region determination + dynamic motion primitive model + policy network update", the safety, reproducibility and autonomous evolution capability of robot reinforcement learning training are significantly improved, the labor cost is significantly saved, and a positive feedback adaptive learning loop is formed.

[0004] This application provides a robot learning control method based on dynamic trust region and motion primitive intervention, the robot learning control method including: The robot's current state is input into a preset policy network to generate preliminary action instructions for the robot; wherein, the current state includes at least six-dimensional force information, multi-view image information, and tactile information of the robot's end effector; The non-uniform trust region threshold is dynamically calculated based on the robot's current end-effector pose. The pose deviation of the robot's current end-effector pose relative to the target point is compared with the non-uniform trust region threshold. When the non-uniform trust region threshold is exceeded for multiple consecutive control cycles, it is determined as an intervention trigger. In response to the intervention trigger, the current end pose and the target point pose are generated in real time based on the pre-trained dynamic motion primitive model to generate a smooth motion trajectory and output the final action command. The final action command is sent to the robot for execution. The new state and reward signal after the action are executed are stored in the experience replay buffer. Data is sampled from the experience replay buffer to train and update the policy network.

[0005] In one possible implementation, the dynamic calculation of the non-uniform trust region threshold based on the robot's current end-effector pose includes: Based on the different intervals of the robot's position deviation in the z-direction, a piecewise linear function is used to calculate the non-uniform trust region threshold. Specifically, when the position deviation in the z-direction is lower than the first height threshold, a preset tolerance is used as the non-uniform trust region threshold; when the position deviation in the z-direction is in the region between the second and third height thresholds, the tolerance is linearly increased with height; when the position deviation in the z-direction is higher than the third height threshold, the tolerance is non-linearly enhanced based on a scaling factor.

[0006] In one possible implementation, the pre-trained dynamic motion primitive model generates a smooth motion trajectory in real time based on the current end-effector pose and the target point pose, and outputs a final action command, including: Taking the current end pose as the starting point and the pose of the target point as the ending point, the DMP differential equation is solved by numerical integration to generate a Cartesian space trajectory sequence that smoothly converges from the starting point to the ending point. The trajectory sequence is converted into a joint space command sequence through inverse kinematics solution, and the joint space command sequence is used as the final motion command.

[0007] In one possible implementation, the dynamic motion primitive model is determined through the following steps: Time normalization and alignment are performed on multiple demonstration trajectories of the same task by experts to obtain the processed demonstration trajectory. The DMP differential equation is used to invert multiple processed demonstration trajectories to determine the forced terms of each processed demonstration trajectory. The model weight parameters are determined by fitting the forced term of each normalized demonstration trajectory based on the radial basis function. Save the model weight parameters, basis function parameters, and system gain as a model file.

[0008] In one possible implementation, after outputting the final action command, the robot learning control method further includes: The historical intervention trigger frequency is statistically analyzed using a fixed time window, and the historical intervention trigger frequency is compared with the stage threshold in the preset attenuation strategy table; When the frequency of historical interventions is lower than the current stage threshold, the intervention probability is automatically reduced and the scaling factor of the non-uniform trust region threshold is increased. When the frequency of historical interventions is higher than the current stage threshold, the intervention probability is automatically increased and the scaling factor of the non-uniform trust region threshold is reduced.

[0009] In one possible implementation, after comparing the pose deviation of the robot's current end-effector relative to the target point with the non-uniform trust region threshold, the robot learning control method further includes: If the non-uniform trust region threshold is not exceeded during multiple consecutive control cycles, the initial action command is sent to the robot for execution.

[0010] This application embodiment also provides a robot learning control device based on dynamic trust region and motion primitive intervention, the robot learning control device comprising: The strategy network module is used to input the robot's current state into a preset strategy network to generate preliminary action instructions for the robot; wherein, the current state includes at least six-dimensional force information, multi-view image information, and tactile information of the robot's end effector; The dynamic trust region determination module is used to dynamically calculate the non-uniform trust region threshold based on the robot's current end-effector pose. It compares the pose deviation of the robot's current end-effector pose relative to the target point with the non-uniform trust region threshold. When the non-uniform trust region threshold is exceeded for multiple consecutive control cycles, it is determined to be an intervention trigger. The dynamic motion primitive trajectory generation module is used to respond to the intervention trigger by generating a smooth motion trajectory in real time based on the current end pose and the pose of the target point using a pre-trained dynamic motion primitive model, and outputting the final action command. The policy update module is used to send the final action instruction to the robot for execution, store the new state and reward signal after the action is executed in the experience playback buffer, and sample data from the experience playback buffer to train and update the policy network.

[0011] In one possible implementation, the dynamic trust region determination module is used to dynamically calculate the non-uniform trust region threshold based on the robot's current end-effector pose: Based on the different intervals of the robot's position deviation in the z-direction, a piecewise linear function is used to calculate the non-uniform trust region threshold. Specifically, when the position deviation in the z-direction is lower than the first height threshold, a preset tolerance is used as the non-uniform trust region threshold; when the position deviation in the z-direction is in the region between the second and third height thresholds, the tolerance is linearly increased with height; when the position deviation in the z-direction is higher than the third height threshold, the tolerance is non-linearly enhanced based on a scaling factor.

[0012] This application embodiment also provides an electronic device, including: a processor, a memory, and a bus. The memory stores machine-readable instructions executable by the processor. When the electronic device is running, the processor communicates with the memory via the bus. When the machine-readable instructions are executed by the processor, the steps of the robot learning control method based on dynamic trust region and motion primitive intervention as described above are performed.

[0013] This application also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, performs the steps of the robot learning control method based on dynamic trust regions and motion primitive intervention as described above.

[0014] This application provides a robot learning control method, apparatus, medium, and device based on dynamic trust region and motion primitive intervention. The robot learning control method includes: inputting the robot's current state into a preset policy network to generate preliminary action commands for the robot; wherein the current state includes at least six-dimensional force information, multi-view image information, and tactile information of the robot's end effector; dynamically calculating a non-uniform trust region threshold based on the robot's current end effector pose, comparing the pose deviation of the robot's current end effector pose relative to the target point with the non-uniform trust region threshold, and determining an intervention trigger when the non-uniform trust region threshold is exceeded for multiple consecutive control cycles; responding to the intervention trigger, performing real-time smooth motion trajectory generation processing on the current end effector pose and the target point pose based on a pre-trained dynamic motion primitive model, and outputting a final action command; sending the final action command to the robot for execution, storing the new state and reward signal after the action execution in an experience playback buffer, and sampling data from the experience playback buffer to train and update the policy network. By constructing a "dynamic trust region determination + dynamic motion primitive model + policy network update" approach, the safety, reproducibility, and autonomous learning capabilities of robot reinforcement learning training are significantly improved, while significantly saving human resources costs and forming a positive feedback adaptive learning loop.

[0015] To make the above-mentioned objectives, features and advantages of this application more apparent and understandable, preferred embodiments are described below in detail with reference to the accompanying drawings. Attached Figure Description

[0016] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of this application and should not be regarded as a limitation of the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0017] Figure 1 A flowchart illustrating a robot learning control method based on dynamic trust region and motion primitive intervention, provided in an embodiment of this application; Figure 2 One of the structural schematic diagrams of a robot learning control device based on dynamic trust region and motion primitive intervention provided in the embodiments of this application; Figure 3 This is a second schematic diagram of a robot learning control device based on dynamic trust region and motion primitive intervention, provided as an embodiment of this application. Figure 4 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0018] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. The components of the embodiments of this application described and shown in the accompanying drawings can generally be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of this application provided in the accompanying drawings is not intended to limit the scope of the claimed application, but merely represents selected embodiments of this application. Based on the embodiments of this application, every other embodiment obtained by those skilled in the art without inventive effort falls within the scope of protection of this application.

[0019] First, the applicable scenarios for this application will be introduced. This application can be applied to the field of robot learning and control technology.

[0020] Research has revealed the following technical problems in existing technologies: 1. High subjectivity and lack of quantifiability in intervention decisions: Whether to intervene depends entirely on the operator's instantaneous experience judgment, lacking objective judgment criteria related to the robot's state (such as end-effector pose and force feedback). This leads to significant differences in intervention timing among different operators, making the training process difficult to reproduce and resulting in large fluctuations in strategy performance. 2. Inconsistent quality of intervention execution: Manual intervention actions are affected by operator fatigue, proficiency, and reaction delay, easily producing jitter, overshoot, or non-smooth trajectories. This not only reduces the safety of single interventions but also injects low-quality training data into the strategy network, slowing down the convergence speed. 3. Static and fixed intervention strategies that cannot evolve with strategy capabilities: Intervention frequency thresholds and safety tolerances are usually preset fixed values, unable to respond to the gradually improving capabilities of the strategy network during training. Therefore, a technical solution is urgently needed to construct a quantifiable, reproducible, and adaptive closed-loop system that integrates intervention decision-making, execution, and strategy evolution to improve the accuracy of robot learning and control.

[0021] Based on this, the embodiments of this application provide a robot learning control method based on dynamic trust region and motion primitive intervention. By constructing "dynamic trust region determination + dynamic motion primitive model + policy network update", the safety, reproducibility and autonomous evolution capability of robot reinforcement learning training are significantly improved, the labor cost is significantly saved, and a positive feedback adaptive learning loop is formed.

[0022] Please see Figure 1 , Figure 1 This is a flowchart illustrating a robot learning control method based on dynamic trust regions and motion primitive intervention, provided as an embodiment of this application. Figure 1 As shown in the embodiments of this application, the robot learning control method includes: S101: Input the robot's current state into a preset policy network to generate the robot's initial action instructions; wherein, the current state includes at least the robot's end effector's six-dimensional force information, multi-view image information, and tactile information.

[0023] In this step, the current state of the robot is processed using a policy network to generate preliminary action instructions for the robot.

[0024] The policy network, based on its current weight parameters, calculates and outputs a continuous motion vector of dimension n through forward propagation, where n is the number of controllable degrees of freedom of the robot (e.g., n=6 for a 6-DOF robotic arm). This motion vector represents the expected displacement increment of the robot's end effector in Cartesian space or the target angle change of each joint, which is the preliminary motion command generated in this step.

[0025] It should be noted that the policy network is a pre-trained deep learning network, and this part does not specify the type of network.

[0026] S102: Dynamically calculate the non-uniform trust region threshold based on the robot's current end-effector pose, compare the pose deviation of the robot's current end-effector pose relative to the target point with the non-uniform trust region threshold, and determine that intervention is triggered when the non-uniform trust region threshold is exceeded for multiple consecutive control cycles.

[0027] In this step, the non-uniform trust region threshold is dynamically calculated based on the robot's current end-effector pose. The pose deviation of the robot's current end-effector pose relative to the target point is compared with the non-uniform trust region threshold. When the non-uniform trust region threshold is exceeded for N consecutive control cycles, it is determined to be an intervention trigger.

[0028] In one possible implementation, the dynamic calculation of the non-uniform trust region threshold based on the robot's current end-effector pose includes: calculating the non-uniform trust region threshold using a piecewise linear function according to the different intervals where the robot's position deviation in the z-direction is located; wherein, when the position deviation in the z-direction is lower than a first height threshold, a preset tolerance is used as the non-uniform trust region threshold; when the position deviation in the z-direction is in the region between a second height threshold and a third height threshold, the tolerance is linearly increased with height; when the position deviation in the z-direction is higher than the third height threshold, the tolerance is non-linearly enhanced based on a scaling factor.

[0029] Here, when the robot's current end-effector pose is received, especially the pose deviation [Δx, Δy, Δz, Δroll, Δpitch, Δyaw] relative to the target point, a dynamic, non-uniform trust region model is pre-defined. The key to this model is that its decision threshold is not a fixed value, but rather a function of the robot's end-effector height, specifically implemented through piecewise linear calculation.

[0030] Where Δx, Δy, and Δz are positional deviations. Z 0, Z 1, Z 2 corresponds to the first height threshold, the second height threshold, and the third height threshold, respectively. d 0 represents the preset tolerance. is the scaling factor. Here, when the position deviation in the z-direction is at the first height threshold, a strict tolerance is applied (e.g., position tolerance 0.001m, rotation tolerance 0.02rad). When the position deviation in the z-direction is at the second or third height threshold, the tolerance increases linearly with height. When the position deviation in the z-direction is higher than the third height threshold, the tolerance increases more rapidly under the influence of the scaling factor.

[0031] It should be noted that the first altitude threshold, the second altitude threshold, and the third altitude threshold increase sequentially.

[0032] In one possible implementation, after comparing the pose deviation of the robot's current end-effector pose relative to the target point with the non-uniform trust region threshold, the robot learning control method further includes: if the non-uniform trust region threshold is not exceeded for multiple consecutive control cycles, then the preliminary action command is sent to the robot for execution.

[0033] S103: In response to the intervention trigger, based on the pre-trained dynamic motion primitive model, perform real-time smooth motion trajectory generation processing on the current end pose and the target point pose, and output the final action command.

[0034] In this step, when the intervention is triggered, the preloaded dynamic motion primitive model is invoked, starting from the current end pose and ending at the target point pose, to generate a smooth and safe motion trajectory in real time. This trajectory is then converted into the final action command, completely replacing the initial action command.

[0035] In one possible implementation, the pre-trained dynamic motion primitive model generates a smooth motion trajectory in real time based on the current end-effector pose and the target point pose, and outputs a final action command, including: A: Taking the current end pose as the starting point and the target point pose as the ending point, the DMP differential equation is solved by numerical integration to generate a Cartesian space trajectory sequence that smoothly converges from the starting point to the ending point.

[0036] Here, the current end-effector pose is set as the initial state of the DMP system, and the target point pose is set as the attractor state. Starting from phase s=1, forward numerical integration is performed on the DMP standard equations. The forcing term f(s) is calculated in real-time based on the current phase s and the pre-learned weights w_i. The integrator (e.g., using the Euler method) outputs the desired state x_next for the next time step at each step. Integration continues until phase s approaches 0, and a sequence of trajectory points smoothly converging from the current end-effector pose to the target point pose is output in real-time streaming.

[0037] B: The trajectory sequence is converted into a joint space command sequence through inverse kinematics solution, and the joint space command sequence is used as the final action command.

[0038] Here, the generated Cartesian space trajectory points are converted into a joint space command sequence through the robot's inverse kinematics (IK) solver, and the joint space command sequence is used as the final motion command.

[0039] In one possible implementation, the dynamic motion primitive model is determined through the following steps: a: Time normalization and alignment are performed on multiple demonstration trajectories of the same task by experts to obtain the processed demonstration trajectory.

[0040] Here, the operator manipulates the robot using a teaching device (such as a spatial mouse) to complete 1 to 5 successful task operations. The system synchronously records the time series of each demonstration trajectory in the task space at a high frequency (e.g., 100Hz): {t, x, , }, where x is the end-effector pose deviation. and The first and second derivatives (i.e., velocity and acceleration) are given. Since the velocity and absolute time in the demonstration may differ, time normalization is required for the N original trajectories collected. Typically, a monotonically decreasing phase variable s (from 1 to 0) is introduced to replace the actual time t, achieving spatiotemporal decoupling of the trajectories. Subsequently, all trajectories are aligned on phase s.

[0041] b: Based on the DMP differential equation, perform inversion processing on multiple processed demonstration trajectories to determine the forced term of each processed demonstration trajectory; based on the radial basis function, fit the forced term of each normalized demonstration trajectory to determine the model weight parameters; save the model weight parameters, basis function parameters, and system gain as a model file.

[0042] Here, for each aligned demonstration trajectory, the goal is to learn the nonlinear forcing term f(s) in its dynamic system. This is achieved by substituting the demonstration data into the core of the DMP's differential equation: f(s) = τ² _demo - α(β(x_goal - x_demo) - τ _demo), where τ is the time scaling factor. `_demo` represents the acceleration of the demonstration trajectory at a certain moment, `α` is the damping coefficient, `β` is the stiffness coefficient, `x_goal` is the target state (attractor) of the demonstration trajectory, and `x_demo` is the position of the demonstration trajectory at a certain moment. _demo represents the velocity of the demonstration trajectory at a certain moment.

[0043] The required forcing term can then be derived. Subsequently, using methods such as local weighted regression, a set of radial basis functions (RBF) is used to fit the forcing term derived from all demonstration trajectories, and a set of optimal weight parameters w_i is obtained. This process is called "DMP model training". The final generated and saved model file is essentially this set of weight parameters {w_i}, the center and width of the basis functions, and the system gain α.

[0044] In one possible implementation, after outputting the final action command, the robot learning control method further includes: (1): The historical intervention trigger frequency is statistically analyzed using a fixed time window, and the historical intervention trigger frequency is compared with the stage threshold in the preset attenuation strategy table.

[0045] It should be noted that the preset decay strategy table is stored in non-volatile memory in the form of an array. Each record contains three fields: the current stage intervention probability, the current stage trigger frequency threshold, and the next stage intervention probability.

[0046] (2) When the frequency of the historical intervention is lower than the current stage threshold, the intervention probability is automatically reduced and the scaling factor of the non-uniform trust region threshold is increased. When the frequency of the historical intervention is higher than the current stage threshold, the intervention probability is automatically increased and the scaling factor of the non-uniform trust region threshold is decreased.

[0047] Here, the comparison and state transition are performed: The preset attenuation strategy table is traversed. When the historical intervention trigger frequency is lower than the current stage trigger frequency threshold, the current intervention probability is automatically reduced, and the scaling factor of the non-uniform trust region threshold is increased. When the historical intervention trigger frequency is higher than the current stage threshold, the current intervention probability is automatically increased, and the scaling factor of the non-uniform trust region threshold is decreased. This scaling factor update result is fed back to the dynamic trust region determination module in real time to modulate its high-zone tolerance threshold calculation, forming a strictly periodic closed-loop control rhythm.

[0048] S104: Send the final action instruction to the robot for execution, store the new state and reward signal after the action is executed into the experience playback buffer, and sample data from the experience playback buffer to train and update the policy network.

[0049] In this step, the final action command is sent to the robot for execution, the new state and reward signal after the action are executed are stored in the experience replay buffer, and the policy network is trained and updated by sampling data from the experience replay buffer.

[0050] Here, the new state system observed by the system after the action is executed is calculated based on the preset task reward function to obtain an instant reward signal. This reward signal is used to measure the quality of the action execution (for example, a positive reward is given for a decrease in the distance to the target point, and a negative reward is given for a collision or exceeding the safe area).

[0051] The system samples data from the experience replay buffer to train and update the policy network, completing one learning cycle. Specifically, the training process of the policy network is as follows: Sampling Policy: In each training cycle, the system randomly samples a batch of experience samples from the experience replay buffer. Network Structure and Optimization Objective: The policy network adopts an Actor-Critic architecture. The Actor network (i.e., the policy network) outputs the action distribution or deterministic actions, while the Critic network evaluates the value of state-action pairs. The policy network is updated using a reinforcement learning algorithm, preferably Proximal Policy Optimization (PPO). Network Parameter Update: Gradient descent is used to update the parameters of the policy network and the Critic network. Multiple gradient updates can be performed in each training cycle (e.g., K updates after collecting N steps of experience) to improve data utilization efficiency. Association with the Policy Network: The updated policy network will be used for the next round of policy action generation, thus realizing a closed-loop learning mechanism of "data acquisition → policy optimization → policy deployment". Through continuous iteration, the policy network gradually learns to autonomously generate high-quality action instructions in complex environments, while the dynamic trust domain and DMP intervention mechanism provide security for the learning process.

[0052] In this application, a three-in-one automated intervention method is constructed, consisting of a "standardized intervention execution layer based on dynamic motion primitives (DMP)," a "segmented dynamic trust region determination layer highly coupled with the end-effector space," and an "adaptive decay control layer with actual intervention rate as feedback signal." Compared with existing technologies such as HIL-SERL that rely on full-process human monitoring and subjective intervention, this method significantly improves the safety, reproducibility, and autonomous evolution capability of robot reinforcement learning training. Its dynamic trust region model actively relaxes tolerance in low-risk areas (such as the first height threshold of the end-effector) to encourage exploration, while in high-risk areas (such as the end-effector approaching the third height threshold), it activates millisecond-level strict thresholds to ensure safety. The DMP trajectory generator offline models human expert experience into reusable, smooth, and dynamically feasible motion templates, ensuring consistent quality of each intervention action. The adaptive decay mechanism automatically expands the trust region and reduces intervention tendency in a stepwise manner based on historical intervention rates, forming a positive feedback loop of "the stronger the strategy, the less intervention, and the deeper the exploration." This fundamentally solves the core defects of traditional human-loop methods, such as subjective intervention, arbitrary execution, and rigid strategies.

[0053] This application provides a robot learning control method, apparatus, medium, and device based on dynamic trust region and motion primitive intervention. The robot learning control method includes: inputting the robot's current state into a preset policy network to generate preliminary action commands for the robot; wherein the current state includes at least six-dimensional force information, multi-view image information, and tactile information of the robot's end effector; dynamically calculating a non-uniform trust region threshold based on the robot's current end effector pose, comparing the pose deviation of the robot's current end effector pose relative to the target point with the non-uniform trust region threshold, and determining an intervention trigger when the non-uniform trust region threshold is exceeded for multiple consecutive control cycles; responding to the intervention trigger, performing real-time smooth motion trajectory generation processing on the current end effector pose and the target point pose based on a pre-trained dynamic motion primitive model, and outputting a final action command; sending the final action command to the robot for execution, storing the new state and reward signal after the action execution in an experience playback buffer, and sampling data from the experience playback buffer to train and update the policy network. By constructing a "dynamic trust region determination + dynamic motion primitive model + policy network update" approach, the safety, reproducibility, and autonomous learning capabilities of robot reinforcement learning training are significantly improved, while significantly saving human resources costs and forming a positive feedback adaptive learning loop.

[0054] Please see Figure 2 , Figure 3 , Figure 2 One of the structural schematic diagrams of a robot learning control device based on dynamic trust region and motion primitive intervention provided in the embodiments of this application; Figure 3This is a second schematic diagram of a robot learning control device based on dynamic trust region and motion primitive intervention, provided as an embodiment of this application. Figure 2 As shown, the robot learning control device 200 includes: The strategy network module 210 is used to input the robot's current state into a preset strategy network to generate preliminary action instructions for the robot; wherein, the current state includes at least six-dimensional force information of the robot's end effector, multi-view image information, and tactile information; The dynamic trust region determination module 220 is used to dynamically calculate the non-uniform trust region threshold based on the robot's current end pose, compare the pose deviation of the robot's current end pose relative to the target point with the non-uniform trust region threshold, and determine that intervention is triggered when the non-uniform trust region threshold is exceeded for multiple consecutive control cycles. The dynamic motion primitive trajectory generation module 230 is used to respond to the intervention trigger by generating a smooth motion trajectory in real time based on the current end pose and the pose of the target point according to the pre-trained dynamic motion primitive model, and outputting the final action command. The policy update module 240 is used to send the final action instruction to the robot for execution, store the new state and reward signal after the action is executed into the experience playback buffer, and sample data from the experience playback buffer to train and update the policy network.

[0055] Furthermore, the dynamic trust region determination module 220 is used to dynamically calculate the non-uniform trust region threshold based on the robot's current end-effector pose: Based on the different intervals of the robot's position deviation in the z-direction, a piecewise linear function is used to calculate the non-uniform trust region threshold. Specifically, when the position deviation in the z-direction is lower than the first height threshold, a preset tolerance is used as the non-uniform trust region threshold; when the position deviation in the z-direction is in the region between the second and third height thresholds, the tolerance is linearly increased with height; when the position deviation in the z-direction is higher than the third height threshold, the tolerance is non-linearly enhanced based on a scaling factor.

[0056] Furthermore, the dynamic motion primitive trajectory generation module 230 is used by the pre-trained dynamic motion primitive model to generate a smooth motion trajectory in real time based on the current end pose and the target point pose, and outputs the final action command: Taking the current end pose as the starting point and the pose of the target point as the ending point, the DMP differential equation is solved by numerical integration to generate a Cartesian space trajectory sequence that smoothly converges from the starting point to the ending point. The trajectory sequence is converted into a joint space command sequence through inverse kinematics solution, and the joint space command sequence is used as the final motion command.

[0057] Furthermore, such as Figure 3 As shown, the robot learning control device 200 also includes an intervention probability adaptive adjustment module 250, which is used for: The historical intervention trigger frequency is statistically analyzed using a fixed time window, and the historical intervention trigger frequency is compared with the stage threshold in the preset attenuation strategy table; When the frequency of historical interventions is lower than the current stage threshold, the intervention probability is automatically reduced and the scaling factor of the non-uniform trust region threshold is increased. When the frequency of historical interventions is higher than the current stage threshold, the intervention probability is automatically increased and the scaling factor of the non-uniform trust region threshold is reduced.

[0058] Furthermore, the policy update module 240 is also used for: If the non-uniform trust region threshold is not exceeded during multiple consecutive control cycles, the initial action command is sent to the robot for execution.

[0059] This application provides a robot learning control device based on dynamic trust region and motion primitive intervention. The robot learning control device includes: a policy network module, used to input the robot's current state into a preset policy network to generate preliminary action commands for the robot; wherein the current state includes at least six-dimensional force information, multi-view image information, and tactile information of the robot's end effector; a dynamic trust region determination module, used to dynamically calculate a non-uniform trust region threshold based on the robot's current end effector pose, compare the pose deviation of the robot's current end effector pose relative to the target point with the non-uniform trust region threshold, and determine an intervention trigger when the non-uniform trust region threshold is exceeded for multiple consecutive control cycles; a dynamic motion primitive trajectory generation module, used to respond to the intervention trigger by generating a smooth motion trajectory in real time based on a pre-trained dynamic motion primitive model of the current end effector pose and the pose of the target point, and outputting a final action command; and a policy update module, used to send the final action command to the robot for execution, store the new state and reward signal after the action in an experience playback buffer, and sample data from the experience playback buffer to train and update the policy network. By constructing a "dynamic trust region determination + dynamic motion primitive model + policy network update" approach, the safety, reproducibility, and autonomous learning capabilities of robot reinforcement learning training are significantly improved, while significantly saving human resources costs and forming a positive feedback adaptive learning loop.

[0060] Please see Figure 4 , Figure 4 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Figure 4 As shown, the electronic device 400 includes a processor 410, a memory 420, and a bus 430.

[0061] The memory 420 stores machine-readable instructions executable by the processor 410. When the electronic device 400 is running, the processor 410 communicates with the memory 420 via the bus 430. When the machine-readable instructions are executed by the processor 410, they can perform the operations described above. Figure 1 The steps of the robot learning control method based on dynamic trust region and motion primitive intervention in the method embodiment shown are specifically implemented in the method embodiment and will not be repeated here.

[0062] This application also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, can perform the above-described actions. Figure 1 The steps of the robot learning control method based on dynamic trust region and motion primitive intervention in the method embodiment shown are specifically implemented in the method embodiment and will not be repeated here.

[0063] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0064] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. The apparatus embodiments described above are merely illustrative. For example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. Furthermore, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Additionally, the shown or discussed mutual couplings, direct couplings, or communication connections may be through some communication interfaces; indirect couplings or communication connections between devices or units may be electrical, mechanical, or other forms.

[0065] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0066] In addition, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0067] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a processor-executable, non-volatile, computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0068] Finally, it should be noted that the above-described embodiments are merely specific implementations of this application, used to illustrate the technical solutions of this application, and not to limit them. The scope of protection of this application is not limited thereto. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that any person skilled in the art can still modify or easily conceive of changes to the technical solutions described in the foregoing embodiments, or make equivalent substitutions for some of the technical features, within the scope of the technology disclosed in this application. Such modifications, changes, or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be covered within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A robot learning control method based on dynamic trust region and motion primitive intervention, characterized in that, The robot learning control method includes: The robot's current state is input into a preset policy network to generate preliminary action instructions for the robot; wherein, the current state includes at least six-dimensional force information, multi-view image information, and tactile information of the robot's end effector; The non-uniform trust region threshold is dynamically calculated based on the robot's current end-effector pose. The pose deviation of the robot's current end-effector pose relative to the target point is compared with the non-uniform trust region threshold. When the non-uniform trust region threshold is exceeded for multiple consecutive control cycles, it is determined as an intervention trigger. In response to the intervention trigger, the current end pose and the target point pose are generated in real time based on the pre-trained dynamic motion primitive model to generate a smooth motion trajectory and output the final action command. The final action command is sent to the robot for execution. The new state and reward signal after the action are executed are stored in the experience replay buffer. Data is sampled from the experience replay buffer to train and update the policy network.

2. The robot learning control method according to claim 1, characterized in that, The dynamic calculation of the non-uniform trust region threshold based on the robot's current end-effector pose includes: Based on the different intervals of the robot's position deviation in the z-direction, a piecewise linear function is used to calculate the non-uniform trust region threshold. Specifically, when the position deviation in the z-direction is lower than the first height threshold, a preset tolerance is used as the non-uniform trust region threshold; when the position deviation in the z-direction is in the region between the second and third height thresholds, the tolerance is linearly increased with height; when the position deviation in the z-direction is higher than the third height threshold, the tolerance is non-linearly enhanced based on a scaling factor.

3. The robot learning control method according to claim 1, characterized in that, The pre-trained dynamic motion primitive model generates a smooth motion trajectory in real time based on the current end-effector pose and the target point pose, and outputs the final action command, including: Taking the current end pose as the starting point and the pose of the target point as the ending point, the DMP differential equation is solved by numerical integration to generate a Cartesian space trajectory sequence that smoothly converges from the starting point to the ending point. The trajectory sequence is converted into a joint space command sequence through inverse kinematics solution, and the joint space command sequence is used as the final motion command.

4. The robot learning control method according to claim 1, characterized in that, The dynamic motion primitive model is determined through the following steps: Time normalization and alignment are performed on multiple demonstration trajectories of the same task by experts to obtain the processed demonstration trajectory. The DMP differential equation is used to invert multiple processed demonstration trajectories to determine the forced terms of each processed demonstration trajectory. The model weight parameters are determined by fitting the forced term of each normalized demonstration trajectory based on the radial basis function. Save the model weight parameters, basis function parameters, and system gain as a model file.

5. The robot learning control method according to claim 1, characterized in that, After outputting the final action command, the robot learning control method further includes: The historical intervention trigger frequency is statistically analyzed using a fixed time window, and the historical intervention trigger frequency is compared with the stage threshold in the preset attenuation strategy table; When the frequency of historical interventions is lower than the current stage threshold, the intervention probability is automatically reduced and the scaling factor of the non-uniform trust region threshold is increased. When the frequency of historical interventions is higher than the current stage threshold, the intervention probability is automatically increased and the scaling factor of the non-uniform trust region threshold is reduced.

6. The robot learning control method according to claim 1, characterized in that, After comparing the pose deviation of the robot's current end-effector relative to the target point with the non-uniform trust region threshold, the robot learning control method further includes: If the non-uniform trust region threshold is not exceeded during multiple consecutive control cycles, the initial action command is sent to the robot for execution.

7. A robot learning control device based on dynamic trust region and motion primitive intervention, characterized in that, The robot learning control device includes: The strategy network module is used to input the robot's current state into a preset strategy network to generate preliminary action instructions for the robot; wherein, the current state includes at least six-dimensional force information, multi-view image information, and tactile information of the robot's end effector; The dynamic trust region determination module is used to dynamically calculate the non-uniform trust region threshold based on the robot's current end-effector pose. It compares the pose deviation of the robot's current end-effector pose relative to the target point with the non-uniform trust region threshold. When the non-uniform trust region threshold is exceeded for multiple consecutive control cycles, it is determined to be an intervention trigger. The dynamic motion primitive trajectory generation module is used to respond to the intervention trigger by generating a smooth motion trajectory in real time based on the current end pose and the pose of the target point using a pre-trained dynamic motion primitive model, and outputting the final action command. The policy update module is used to send the final action instruction to the robot for execution, store the new state and reward signal after the action is executed in the experience playback buffer, and sample data from the experience playback buffer to train and update the policy network.

8. The robot learning control device according to claim 7, characterized in that, The dynamic trust region determination module is used to dynamically calculate the non-uniform trust region threshold based on the robot's current end-effector pose. Based on the different intervals of the robot's position deviation in the z-direction, a piecewise linear function is used to calculate the non-uniform trust region threshold. Specifically, when the position deviation in the z-direction is lower than the first height threshold, a preset tolerance is used as the non-uniform trust region threshold; when the position deviation in the z-direction is in the region between the second and third height thresholds, the tolerance is linearly increased with height; when the position deviation in the z-direction is higher than the third height threshold, the tolerance is non-linearly enhanced based on a scaling factor.

9. An electronic device, characterized in that, include: The device includes a processor, a memory, and a bus. The memory stores machine-readable instructions executable by the processor. When the electronic device is running, the processor communicates with the memory via the bus. The machine-readable instructions are executed by the processor to perform the steps of the robot learning control method based on dynamic trust regions and motion primitive intervention as described in any one of claims 1 to 6.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, performs the steps of the robot learning control method based on dynamic trust regions and motion primitive intervention as described in any one of claims 1 to 6.

Images