A method, system, device, program product, and medium for efficient skill learning of robots.

Abstract

This invention relates to the field of robot skill learning technology, specifically to an efficient robot skill learning method, system, device, program product, and medium. The method includes: obtaining the robot's task policy equation in Cartesian space; obtaining the expected trajectory at each time step of the robot's task policy; constructing the cost function of the robot's task policy; and obtaining the robot's final expected trajectory. This invention utilizes interactive reinforcement learning to integrate human and machine intelligence, improving algorithm efficiency and overcoming the limitations of existing reinforcement learning methods in robot operation applications. It enables real-world robot skill learning, allowing robots to be applied in a wider range of scenarios.

Description

Technical Field

[0001] This invention relates to the field of robot skill learning technology, specifically to a method, system, device, program product, and medium for efficient robot skill learning. Background Technology

[0002] Robots have broad application prospects in the future. Applying reinforcement learning algorithms to robotics problems enables robots to learn skills through interaction with their environment, allowing them to perform better in unstructured environments. However, robotic learning is constrained by physical limitations, making it impractical to learn successful strategies for complex motor skills in real-world systems. Solving robotic problems using reinforcement learning algorithms requires handling continuous spaces, leading to relatively long training times and high training costs. Furthermore, it limits their application in the real world. Considering these issues, on the one hand, employing policy search-based reinforcement learning methods reduces the agent's exploration space by parameterizing policies, thus improving its scalability and enabling the application of reinforcement learning methods in robotics problems; on the other hand, leveraging human knowledge and experience to accelerate the learning process is a feasible solution for realizing the application of reinforcement learning algorithms in robotics.

[0003] Interactive machine learning, or learning from demonstrations, allows for the rapid transfer of human knowledge to agents; however, most existing methods require expert demonstrations. This work proposes using human correction suggestions in the motion domain to learn motion trajectories by providing examples of task execution to the agent or robot through demonstrations. These demonstrations are recorded and used to compute models of motion primitives for task execution. With learning from demonstrations, the robot can quickly learn policies to satisfy simple tasks, or, in complex situations, learn policies that approach their satisfaction.

[0004] However, learning from demonstrations has many limitations: firstly, it requires expert demonstrations; secondly, the skills learned by the robot depend on the quality of the demonstrations provided; and thirdly, some demonstrations may be suboptimal or ambiguous, thus limiting the performance of the robot's learning strategies.

[0005] Therefore, there is a need to provide an efficient skill learning method, system, device, program product, and medium for robots to solve the above problems. Summary of the Invention

[0006] This invention provides a method, system, device, program product, and medium for efficient robot skill learning, addressing the problems that existing robot skill learning methods require expert demonstrations; the skills learned by the robot depend on the quality of the provided demonstrations; and some demonstrations may be suboptimal or ambiguous, thus limiting the performance of robot learning strategies.

[0007] The present invention provides a method for efficient skill learning in robots, which employs the following technical solution: including:

[0008] The task strategy equations of the robot in Cartesian space are obtained using impedance control methods.

[0009] Obtain the dynamic motion primitive model corresponding to the task strategy equation, and use the dynamic motion primitive model to represent the desired trajectory in the robot's task strategy.

[0010] Based on the cost function of the robot's task strategy, the weight parameters of the dynamic motion primitive model are updated using the path integral strategy improvement method. After each update of the weight parameters, the weight parameters are corrected using a pre-set interactive learning framework model with an adjustable learning rate until the cost function converges, thus obtaining the updated target dynamic motion primitive model.

[0011] The robot's final desired trajectory is obtained based on the target dynamic motion primitive model.

[0012] Preferably, the robot's task strategy equation in Cartesian space is:

[0013]

[0014] In the formula, x d This represents the robot's desired trajectory;

[0015] The first derivative represents the robot's desired trajectory;

[0016] The second derivative of the robot's desired trajectory;

[0017] M d Indicates the mass parameters of the robot system;

[0018] C d This represents the damping parameters of the robot system;

[0019] K d Represents the stiffness parameters of the robot system;

[0020] x represents the robot's actual trajectory;

[0021] The first derivative represents the robot's actual trajectory;

[0022] The second derivative represents the robot's actual trajectory;

[0023] F e This represents the contact force between the robot and the object it is in contact with.

[0024] Preferably, the interactive learning framework model with an adjustable learning rate is used by the user to provide action suggestions to the robot based on the state; the robot determines the human feedback model based on the action suggestions; the human feedback model passes the human suggestions to the dynamic motion primitive model and corrects the weight parameters of the dynamic motion primitive model.

[0025] Preferably, the expression for the dynamic motion primitive model is:

[0026]

[0027] In the formula, The second derivative of the robot's desired trajectory;

[0028] The first derivative represents the robot's desired trajectory;

[0029] x d This represents the robot's desired trajectory;

[0030] α y and β y It is a constant;

[0031] g represents the target point of the desired trajectory;

[0032] f(t) represents the trajectory shape learner model.

[0033] Preferably, the expression for the trajectory shape learner model is:

[0034]

[0035] In the formula, ψ i (t) represents the Gaussian function;

[0036] θ i This represents the weights of the dynamic motion primitive model.

[0037] Preferably, the expression for the cost function of the robot's task strategy is:

[0038]

[0039] In the formula, J(τ) i ) is the trajectory τ i In t i to t N Total cost over time;

[0040] It represents the trajectory τ i At the t N The ultimate cost at that moment;

[0041] r tThis represents the immediate cost of the robot executing its task strategy at time t.

[0042] This represents the instantaneous control cost of the robot executing its task strategy at time t. This represents the transpose of the policy parameter matrix;

[0043] θ t Represents the strategy parameter matrix;

[0044] R represents a positive semi-definite weight matrix;

[0045] t i Indicates the initial time;

[0046] t N Indicates the final moment;

[0047] τ i It is in t i to t N Expected trajectory x within time interval d A set of.

[0048] A highly efficient skill learning system for robots based on human-machine hybrid reinforcement learning includes:

[0049] The task policy equation construction module is used to obtain the robot's task policy equation in Cartesian space using impedance control methods.

[0050] The desired trajectory representation module is used to obtain the dynamic motion primitive model corresponding to the task policy equation, and to represent the desired trajectory in the robot's task policy using the dynamic motion primitive model.

[0051] The dynamic motion primitive model update module is used to update the weight parameters of the dynamic motion primitive model based on the cost function of the robot's task strategy. It uses the path integral strategy improvement method to update the weight parameters. After each update of the weight parameters, the weight parameters are corrected using a pre-set interactive learning framework model with an adjustable learning rate until the cost function converges, and the updated target dynamic motion primitive model is obtained.

[0052] The final expected trajectory module is used to obtain the robot's final expected trajectory based on the target dynamic motion primitive model.

[0053] An electronic device includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the computer program is executed by the processor, it implements the steps of the efficient robot skill learning method based on human-machine hybrid reinforcement learning of the present invention.

[0054] A program product includes a computer program that, when run, performs the steps of the robot efficient skills learning method based on human-machine hybrid reinforcement learning according to the present invention.

[0055] A storage medium having a computer program stored thereon, which, when run, performs the steps of the present invention’s efficient robot skill learning method based on human-machine hybrid reinforcement learning.

[0056] The beneficial effects of this invention are:

[0057] This invention models the robot's strategy as motion primitives to achieve dimensionality reduction in the state and action spaces. Then, it combines an interactive learning framework model with an adjustable learning rate with a path integral-based policy improvement method to achieve an efficient human-machine hybrid reinforcement learning algorithm. This algorithm is applied to robotic problems to enable efficient skill learning for robots. Specifically, the reinforcement learning algorithm, combining the interactive learning framework model with an adjustable learning rate and the path integral-based policy improvement method, updates the weight parameters of the dynamic motion primitive model until the cost function converges, obtaining the target dynamic motion primitive model. The optimal task strategy is then output using this target dynamic motion primitive model. This invention ensures the convergence of the reinforcement learning algorithm while accelerating its convergence speed by utilizing human knowledge, making it feasible for application in high-dimensional problems such as robot operation. Furthermore, by using interactive reinforcement learning to integrate human and machine intelligence, it improves algorithm efficiency, overcomes the limitations of existing reinforcement learning in robot operation applications, and enables real-world robot skill learning, allowing robots to be applied in a wider range of scenarios. Attached Figure Description

[0058] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0059] Figure 1 This is a flowchart of an efficient skill learning method for robots according to the present invention;

[0060] Figure 2 This is a schematic diagram of the structure of the interactive learning framework model with adjustable learning rate in an embodiment of the present invention;

[0061] Figure 3 This is a schematic diagram of the human-machine hybrid reinforcement learning algorithm in an embodiment of the present invention;

[0062] Figure 4 This is a schematic diagram of the human-computer interaction interface in an embodiment of the present invention;

[0063] Figure 5 The image shows the trajectory curve obtained by using only the path integral-based strategy improvement method in Comparative Example 1 of the present invention.

[0064] Figure 6 This is a cost curve obtained by using only the path integral-based strategy improvement method in Comparative Example 1 of the present invention.

[0065] Figure 7 The image shows the trajectory curves obtained in Comparative Example 2 of this invention using the improved strategy based on path integral and the traditional interactive learning framework model.

[0066] Figure 8 The cost curves are obtained in Comparative Example 2 of this invention using the path integral-based strategy improvement method and the traditional interactive learning framework model.

[0067] Figure 9 The image shows a trajectory curve obtained using a path integral-based strategy improvement method and an interactive learning framework model with an adjustable learning rate in an embodiment of the present invention.

[0068] Figure 10 This is a cost curve obtained using a path integral-based strategy improvement method and an interactive learning framework model with an adjustable learning rate in an embodiment of the present invention. Detailed Implementation

[0069] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0070] This invention provides an embodiment of an efficient skill learning method for robots. This embodiment improves upon the traditional interactive learning framework (COACH) by constructing a COACH with an adjustable learning rate to enhance the efficiency of transferring human knowledge to the agent. Specifically, it integrates the adjustable learning rate COACH with a path integral-based policy improvement (PI) method. 2This paper proposes a method for robot skill learning that combines human-machine hybrid reinforcement learning algorithms to achieve efficient skill learning in robots. First, the policy is modeled as motion primitives to reduce the dimensionality of the state and action spaces. Then, by utilizing the optimal parameters of the task policy in a reinforcement learning algorithm with an adjustable learning rate (COACH), the proposed method combines human knowledge and reinforcement algorithms through interactive reinforcement learning. While ensuring the convergence of the reinforcement learning algorithm, the use of human knowledge accelerates its convergence speed, making it feasible for application in high-dimensional problems such as robot manipulation. In summary, the goal of this embodiment is to improve algorithm efficiency by integrating human and machine intelligence through interactive reinforcement learning, overcoming the limitations of existing reinforcement learning in robot manipulation applications, and enabling real-world robot skill learning, allowing robots to be applied in a wider range of scenarios.

[0071] Specifically, this implementation example Figure 1 As shown, it includes:

[0072] S1. Obtain the robot's task strategy equation in Cartesian space;

[0073] Specifically, the task policy equations of the robot in Cartesian space are obtained using impedance control methods; the expression of the task policy equations of the robot in Cartesian space is:

[0074]

[0075] In the formula, x d This represents the robot's desired trajectory;

[0076] The first derivative represents the robot's desired trajectory;

[0077] The second derivative of the robot's desired trajectory;

[0078] M d Indicates the mass parameters of the robot system;

[0079] C d This represents the damping parameters of the robot system;

[0080] K d Represents the stiffness parameters of the robot system;

[0081] x represents the robot's actual trajectory;

[0082] The first derivative represents the robot's actual trajectory;

[0083] The second derivative represents the robot's actual trajectory;

[0084] F e This represents the contact force between the robot and the object it is in contact with.

[0085] S2. Obtain the dynamic motion primitive model corresponding to the task strategy equation;

[0086] Specifically, the dynamic motion primitive model is a nonlinear dynamic system. It includes a second-order system, which changes the final state of the system to modify the target position of the desired trajectory; and a trajectory shape learner model, which fits various trajectory shapes through the normalized linear superposition of multiple nonlinear basis functions. Specifically, a self-stabilizing second-order dynamic system is used to construct an "attraction point" model. This "attraction point" changes the final state of the system, thereby modifying the target position of the trajectory. The trajectory shape learner fits various trajectory shapes through the normalized linear superposition of multiple nonlinear basis functions. The dynamic motion primitive model is used to parameterize the desired trajectory at each moment in the robot's task strategy; the expression for the dynamic motion primitive model is:

[0087]

[0088] In the formula, The second derivative of the robot's desired trajectory;

[0089] The first derivative represents the robot's desired trajectory;

[0090] x d This represents the robot's desired trajectory;

[0091] α y and β y It is a constant;

[0092] g represents the target point of the desired trajectory;

[0093] f(t) represents the trajectory shape learner model.

[0094] The expression for the trajectory shape learner model is as follows:

[0095]

[0096] In the formula, ψ i (t) represents the Gaussian function;

[0097] θ i This represents the weights of the dynamic motion primitive model.

[0098] S3. Obtain the updated target dynamic motion primitive model;

[0099] Specifically, based on the cost function of the robot's task strategy, the weight parameters of the dynamic motion primitive model are updated using the path integral strategy improvement method. After each update of the weight parameters, the weight parameters are corrected using a pre-set interactive learning framework model with an adjustable learning rate until the cost function converges, thus obtaining the updated target dynamic motion primitive model.

[0100] Step 31: The expression for the cost function of the robot's task strategy is:

[0101]

[0102] In the formula, J(τ) i ) is the trajectory τ i In t i to t N Total cost over time;

[0103] It represents the trajectory τ i At the t N The ultimate cost at that moment;

[0104] r t This represents the immediate cost of the robot executing its task strategy at time t.

[0105] This represents the instantaneous control cost of the robot executing its task strategy at time t.

[0106] This represents the transpose of the policy parameter matrix;

[0107] θ t Represents the strategy parameter matrix;

[0108] R represents a positive semi-definite weight matrix;

[0109] t i Indicates the initial time;

[0110] t N Indicates the final moment;

[0111] τ i It is in t i to t N Expected trajectory x within time interval d A set of.

[0112] Step 32: Update the weight parameters of the dynamic motion primitive model using the path integral strategy improvement method. After each update of the weight parameters, use a pre-set interactive learning framework model with an adjustable learning rate to correct the weight parameters until the cost function converges, and obtain the updated target dynamic motion primitive model.

[0113] In this embodiment, the Path Integrals Policy Improvement (PIP) method updates the weight parameters of the dynamic motion primitive model. In each iteration, noise is added to the weight parameters for M explorations, and these weight parameters are executed to collect different trajectory execution effects and costs. Based on the weight parameters and cost function obtained from the M explorations, the dynamic motion primitive model is updated to improve it and obtain a less costly model. After each update, the weight parameters are corrected using a pre-set interactive learning framework model with an adjustable learning rate, i.e., as shown below. Figure 2 As shown, the interactive learning framework model with adjustable learning rate of the present invention includes: The user provides action suggestions to the robot based on the state, i.e., the robot's actual trajectory position at time t; the action suggestions are represented by binary signals +1, -1, and 0, respectively, indicating increasing the action amplitude, decreasing the action amplitude, and keeping the action amplitude constant. The robot constructs a human feedback model based on supervised learning methods using the action suggestions collected from the user according to different states. After learning, this model replaces the human user in providing action suggestions to the robot. The acquired action suggestions are passed to a dynamic motion primitive model, and the weight parameters of the dynamic motion primitive model are corrected. Based on the corrected dynamic motion primitive model, a new desired trajectory is determined, and the robot tracks the new desired trajectory in the environment; as shown... Figure 3 This describes the flowchart of a human-machine hybrid reinforcement learning algorithm, combining a path integral-based policy improvement method with an interactive learning framework model with an adjustable learning rate. The process begins with an initial policy. During the implementation of this initial policy, a human user provides feedback (i.e., human suggestions) based on the robot's performance, grounded in the difference between the actual and desired trajectories. Based on this human feedback, the policy is perturbed, introducing changes to its parameters to alter the robot's behavior. The update step compares the perturbed policy with the original policy and selects the better-performing policy as the implementation policy. The COACH framework (an interactive learning framework model with an adjustable learning rate) allows for adjustment of the learning rate to accommodate rapid policy optimization, based on the robot's performance in the environment and the effectiveness of human feedback. The entire process is repeated in multiple iterations, adjusting the policy each time based on the previous learning outcome, until the policy completes the predetermined learning task.

[0114] The weight parameter update rules for the dynamic motion primitive model are as follows:

[0115] Equation (5) is the cost expression for each test trajectory:

[0116]

[0117] In the formula, S({τ i} k ) represents the cost of the i-th trajectory in the k-th exploration trial;

[0118] R represents the positive semi-weighted matrix of the cost of secondary control;

[0119] N represents the total number of moments;

[0120] The trajectory in the k-th exploration experiment is represented by the trajectory at point t. N The cost at moment t, N The cost at each moment is the cost at the end of the trajectory;

[0121] Indicates the t-th j The control cost of implementing a strategy at any given time Indicates the t-th j The strategy parameters corresponding to each time point; express transpose;

[0122] Indicates the t-th j The instantaneous cost at time t, i.e., the cost at time t j The immediate cost of a strategy executed at a given moment;

[0123] This indicates that in the k-th exploration experiment, at the t-th time... j The sum of the total costs at each moment.

[0124] In equation (5), the t-th j The policy parameters corresponding to each time point are obtained from equation (6):

[0125]

[0126] In the formula, Indicates the t-th j The policy parameters corresponding to each time point; θ represents the policy parameters; Indicates the t-th j Noise parameters corresponding to each time point;

[0127]

[0128] In the formula, Indicates the t-th j The gradient of the policy parameters at each time step; R -1 Represents the inverse matrix of matrix R; This indicates that the Kth exploratory experiment will be conducted on the tth... j Noise at any given moment The projection matrix is ​​projected onto the parameter space, i.e., projected onto the gradient direction of the policy parameters.

[0129]

[0130] In the formula, P({τ i} k () represents the probability of trajectory τ at time i in the k-th exploration experiment;

[0131] {τ i} k Let τ represent the trajectory τ at the i-th moment in the k-th exploration experiment;

[0132] S({τ i} k ) represents the cumulative cost of the trajectory starting from time i in the k-th trial. In this embodiment, the cumulative cost includes the sum of the instantaneous cost of the path and the control cost.

[0133] λ represents the projection scale of noise onto the parameter space. When λ is large, the probability difference between different trajectories will decrease; when λ is small, the trajectory with lower cost will obtain a higher probability.

[0134] K represents the total number of exploratory trials;

[0135] l represents the l-th exploratory experiment.

[0136] Equation (9) is obtained from equations (7) and (8):

[0137]

[0138] In the formula, Indicates the t-th j The updated value of the strategy parameter at each time point indicates the adjustment that needs to be made to the strategy parameter at that time.

[0139] This indicates that the Kth exploratory experiment will be conducted on the tth... j Noise at any given moment The projection matrix projected onto the parameter space;

[0140] P({τ i} k () represents the probability of trajectory τ at time i in the k-th exploration experiment;

[0141] Equation (10) is:

[0142]

[0143] In the formula, [δθ] j This represents the average value of the updated policy parameters at each time step.

[0144] N represents the number of time points;

[0145] Indicates the t-th i The average of the updated values ​​of the policy parameters at each time point;

[0146] l represents the lth exploratory experiment;

[0147] The Gaussian function at time t i The output at each time step is used to weight the parameter updates calculated at that time step.

[0148] The Gaussian function at time t l The output at each time step is used to weight the parameter updates calculated at that time step.

[0149] Equation (11) is:

[0150] θ 更新后 ←θ+ (11)

[0151] In the formula, δθ represents the updated value of the policy parameter; θ represents the policy parameter, θ 更新后 This indicates the updated strategy parameters.

[0152] S4. Obtain the robot's final desired trajectory;

[0153] Specifically, the final desired trajectory of the robot is obtained based on the target dynamic motion primitive model. This embodiment is verified in a one-dimensional via-point task. Simulation experiments are conducted on a Windows 10 platform using Python 3.9.7. With via-point as the target task, the trajectory is designed to pass through 0.1 seconds in 0.3 seconds. For the implementation using the COACH framework, the exit condition is set to an error threshold of 1e-3. For the PI-based implementation... 2 In the implementation, the difference in y-values ​​at t=0.3s between two iterations is used as the criterion for whether the strategy converges, with a threshold set to 1e-4. A human-computer interaction interface was built based on PyQt, as shown below. Figure 4 As shown.

[0154] Comparative Example 1

[0155] Only using PI 2In the algorithm (a policy improvement method based on path integral), each iteration requires 10 explorations, the noise parameter is a random value, and the policy converges after 134 iterations. 2 The trajectory curve corresponding to the algorithm is as follows Figure 5 As shown, the cost curve is as follows Figure 6 As shown.

[0156] Comparative Example 2

[0157] Using PI 2 In the +COACH iteration, each iteration requires 10 explorations. The exploration part consists of random exploration by the agent and exploration using human correction feedback. In the autonomous exploration part, the noise parameters are random values. The COACH part is configured with the following parameters: learning rate β = 0.05, bias = 0.05, error e = 0.5, and 63 iterations. The resulting PI... 2 +COACH's trajectory curve is as follows Figure 7 As shown, the cost curve is as follows Figure 8 As shown.

[0158] Embodiments of the present invention

[0159] The adjustable learning rate COACH+PI disclosed in the embodiments of the present invention 2 At that time, using a human-computer interaction interface, the noise parameter was given by a human user after observing the results of each iteration, and the policy converged after 18 iterations; the learning rate of the COACH+PI algorithm was adjustable. 2 The corresponding trajectory curve is as follows Figure 9 As shown, the cost curve is as follows Figure 10 As shown.

[0160] A comparison of the experimental results of Comparative Example 1, Comparative Example 2, and the embodiments of the present invention demonstrates that, among COACH-based algorithms, the adjustable-learning-rate COACH has a faster learning speed and can complete the task metrics in fewer rounds; in PI-based algorithms... 2 In the algorithm, with the basic PI 2 Compared to other algorithms, using the COACH framework with adjustable learning rates and human feedback accelerates the learning speed. The results show that combining COACH with adjustable learning rates and PI... 2 The method has been significantly improved.

[0161] according to Figure 5 , Figure 7 and Figure 9 As shown, when the trajectories obtained by the three methods are the same, from Figure 6 , Figure 8 , Figure 10 It can be concluded that the method of the present invention requires the fewest iterations, that is, the learning effect is better.

[0162] An efficient robot skill learning system based on human-machine hybrid reinforcement learning includes: a task policy equation construction module, a desired trajectory representation module, a cost function construction module, a dynamic motion primitive model update module, and a final desired trajectory module. The task policy equation construction module uses impedance control to obtain the robot's task policy equation in Cartesian space. The desired trajectory representation module uses a dynamic motion primitive model to represent the desired trajectory of the robot at each time step in its task policy. The cost function construction module constructs the cost function of the robot's task policy based on the desired trajectory at each time step. The dynamic motion primitive model update module updates the weight parameters of the dynamic motion primitive model based on the cost function of the robot's task policy using a path integral policy improvement method. After each update of the weight parameters, a pre-set interactive learning framework model with an adjustable learning rate is used to correct the weight parameters until the cost function converges, resulting in the updated target dynamic motion primitive model. The final desired trajectory module obtains the robot's final desired trajectory based on the target dynamic motion primitive model.

[0163] An electronic device includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the computer program is executed by the processor, it implements the steps of the efficient robot skill learning method based on human-machine hybrid reinforcement learning of the present invention.

[0164] A program product includes a computer program that, when run, performs the steps of the robot efficient skills learning method based on human-machine hybrid reinforcement learning according to the present invention.

[0165] A storage medium having a computer program stored thereon, which, when run, performs the steps of the present invention’s efficient robot skill learning method based on human-machine hybrid reinforcement learning.

[0166] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for efficient skill learning in robots based on human-machine hybrid reinforcement learning, characterized in that, include: The task strategy equations of the robot in Cartesian space are obtained using impedance control methods. Obtain the dynamic motion primitive model corresponding to the task strategy equation, and use the dynamic motion primitive model to represent the desired trajectory in the robot's task strategy. Based on the cost function of the robot's task strategy, the weight parameters of the dynamic motion primitive model are updated using the path integral strategy improvement method. After each update of the weight parameters, the weight parameters are corrected using a pre-set interactive learning framework model with an adjustable learning rate until the cost function converges, thus obtaining the updated target dynamic motion primitive model. The robot's final desired trajectory is obtained based on the target dynamic motion primitive model.

2. The efficient skill learning method for robots based on human-machine hybrid reinforcement learning according to claim 1, characterized in that, The robot's task strategy equation in Cartesian space is: In the formula, x d This represents the robot's desired trajectory; The first derivative represents the robot's desired trajectory; The second derivative of the robot's desired trajectory; M d Indicates the mass parameters of the robot system; C d This represents the damping parameters of the robot system; K d Represents the stiffness parameters of the robot system; x represents the robot's actual trajectory; The first derivative representing the robot's actual trajectory; The second derivative represents the robot's actual trajectory; F e This represents the contact force between the robot and the object it is in contact with.

3. The efficient skill learning method for robots based on human-machine hybrid reinforcement learning according to claim 1, characterized in that, An interactive learning framework model with an adjustable learning rate is used by users to provide action suggestions to the robot based on the state; the robot determines the human feedback model based on the action suggestions; the human feedback model passes the human suggestions to the dynamic motion primitive model and corrects the weight parameters of the dynamic motion primitive model.

4. The efficient skill learning method for robots based on human-machine hybrid reinforcement learning according to claim 1, characterized in that, The expression for the dynamic motion primitive model is: In the formula, The second derivative of the robot's desired trajectory; The first derivative represents the robot's desired trajectory; x d This represents the robot's desired trajectory; α y and β y It is a constant; g represents the target point of the desired trajectory; f(t) represents the trajectory shape learner model.

5. The efficient skill learning method for robots based on human-machine hybrid reinforcement learning according to claim 4, characterized in that, The expression for the trajectory shape learner model is: In the formula, ψ i (t) represents the Gaussian function; θ i This represents the weights of the dynamic motion primitive model.

6. The efficient skill learning method for robots based on human-machine hybrid reinforcement learning according to claim 1, characterized in that, The expression for the cost function of the robot's task strategy is: In the formula, J(τ) i ) is the trajectory τ i In t i to t N Total cost over time; It represents the trajectory τ i At the t N The ultimate cost at that moment; r t This represents the immediate cost of the robot executing its task strategy at time t. This represents the instantaneous control cost of the robot executing its task strategy at time t. This represents the transpose of the policy parameter matrix; θ t Represents the strategy parameter matrix; R represents a positive semi-definite weight matrix; t i Indicates the initial time; t N Indicates the final moment; τ i It is in t i to t N Expected trajectory x within time interval d A set of.

7. A highly efficient skill learning system for robots based on human-machine hybrid reinforcement learning, characterized in that, include: The task policy equation construction module is used to obtain the robot's task policy equation in Cartesian space using impedance control methods. The desired trajectory representation module is used to obtain the dynamic motion primitive model corresponding to the task policy equation, and to represent the desired trajectory in the robot's task policy using the dynamic motion primitive model. The dynamic motion primitive model update module is used to update the weight parameters of the dynamic motion primitive model based on the cost function of the robot's task strategy. It uses the path integral strategy improvement method to update the weight parameters. After each update of the weight parameters, the weight parameters are corrected using a pre-set interactive learning framework model with an adjustable learning rate until the cost function converges, and the updated target dynamic motion primitive model is obtained. The final expected trajectory module is used to obtain the robot's final expected trajectory based on the target dynamic motion primitive model.

8. An electronic device, characterized in that, It includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program, when executed by the processor, implements the steps of the method as described in any one of claims 1-6.

9. A program product, characterized in that, Includes a computer program, which, when run, performs the steps of the method according to any one of claims 1-6.

10. A storage medium, characterized in that, It stores a computer program that, when run, performs the steps of the method according to any one of claims 1-6.

Images