Method for personnel simulation reality hybrid training for robot operation skill learning

Abstract

The application discloses a personnel simulation reality hybrid training method for robot operation skill learning, first, a complex operation task stage division and motion planning generation based on rule knowledge guidance are carried out, prior knowledge of a person is used to analyze the operation skill learning process, and the operation process is divided into several stages; secondly, an operation skill learning optimization method fusing expert preference evaluation and fixed index evaluation is constructed; then, an operation skill transfer and evaluation framework from a simulation environment to an actual scene is constructed, exploration parameters and action instructions in the actual environment are generated based on a similarity function, and the operation skill transfer is completed; finally, a robot skill learning and skill transfer logical framework is constructed, and the completion degree of robot operation skill learning and the completeness of skill transfer are gradually improved through an iterative method. The application effectively improves the efficiency of robot skill learning.

Description

Technical Field

[0001] This invention belongs to the field of robotics technology, specifically relating to a method for hybrid training of personnel in a simulated reality environment. Background Technology

[0002] Learning robot manipulation skills is a hot topic, and reinforcement learning and its derivatives are a primary means of achieving this. However, reinforcement learning also has inherent drawbacks, which are particularly evident in the process of learning robot manipulation skills. These include low learning efficiency, difficulty in applying it to real-world scenarios, and a lack of transparency in the learning process. With technological advancements, human-robot hybrid training offers hope for addressing these issues. Summary of the Invention

[0003] To overcome the shortcomings of existing technologies, this invention provides a human-simulated reality hybrid training method for robot operation skill learning. First, it divides complex operation tasks into stages and generates motion plans based on rule-based knowledge guidance, using prior human knowledge to analyze the process of the operation skill to be learned, dividing the operation process into several stages. Second, it constructs an operation skill learning optimization method that integrates expert preference evaluation and fixed index evaluation. Then, it constructs an operation skill transfer and evaluation framework from the simulation environment to the real-world scenario, generating exploration parameters and action commands in the real-world environment based on a similarity function to complete the transfer of operation skills. Finally, it constructs a logical framework for robot skill learning and skill transfer, gradually improving the completion rate of robot operation skill learning and the completeness of skill transfer through an iterative method. This invention effectively improves the efficiency of robot skill learning.

[0004] The technical solution adopted by this invention to solve its technical problem includes the following steps:

[0005] Step 1: Pre-processing of the operation;

[0006] Step 1-1: Divide the operation process into stages;

[0007] The set of all kinematic and dynamic parameters of the environment and robot involved in the operation process is called the state space, denoted by the set S;

[0008] Extract all intermediate states required to complete the operation task. Let there be n intermediate states, then all intermediate states are represented as follows: <S0,S1,S2,…,S n Based on these n intermediate states, the entire operation process is divided into n stages, using... <G1,G2,…,G n >Indicates; Stage G i Indicates from state S i-1 to state S iOne state transformation, expressed as follows:

[0009] G i = trigger(S i-1 , S i ) (I)

[0010] Step 1-2: Define the constraint conditions for phase transition;

[0011] If there are n operation process phases, n-1 transition condition judgment functions will be generated accordingly, expressed as <C1, C2, …, C n > ; Transition condition C i represents a judgment of whether the state of phase G i-1 satisfies the conditions of phase G i ;

[0012] Transition condition C i is expressed as follows:

[0013]

[0014] Where conditions(S i ) ∈ S i , represents the state that phase G i must satisfy;

[0015] Step 1-3: Generate the action plan of the robot;

[0016] Phase G i is considered as a point-to-point motion trajectory, and the robot obtains the constrained motion trajectory according to the environmental constraint conditions and task requirements, expressed as follows:

[0017]

[0018] Where parameter W i = <w i1 , w i2 , …, w im > represents the parameters required for motion trajectory generation, and b i represents the constraint conditions; The set of all action plans is expressed as: A = <A1, A2, …, A n >;

[0019] Step 1-4: Complete the extraction of key feature parameters;

[0020] The key feature parameters are the parameters to be optimized by the optimization algorithm, represented by the symbol K. The selection range of the parameters includes: conditions(S), W, and b;

[0021] Step 2: Optimization of parameters in the simulation environment;

[0022] Step 2-1: update the motion planning interacting with the simulation environment;

[0023] Take N vector values of the feature parameter K using Gaussian distribution, denoted as: Where μ and σ are the mean and standard deviation of the Gaussian distribution of the feature parameter, respectively. Then update the motion planning in step 1-3:

[0024] Step 2-2: generate the execution action policy;

[0025] The trajectory tracking is realized by using tracking control method:

[0026]

[0027] Where is the tracking error at time t;

[0028] Step 2-3: calculate the reward function of the action policy;

[0029] Collect N trajectories ρ:<ρ1,ρ2,…,ρ N > from step 2-2, and construct the reward value Calculate the reward value of each execution task

[0030] Step 2-4: optimization of key feature parameters;

[0031] According to the reward function generated in step 2-3, the gradient of the reward value to the feature parameter variance is calculated by using gradient ascent algorithm, and then the learning rate α of parameter update is adjusted by the following formula:

[0032]

[0033]

[0034] α=α0+ε μ (7)

[0035] Where parameter ε μ represents the judgment of the parameter iteration step length by people, which is made based on the corresponding parameter value of the reward value;

[0036] The formula of parameter update is as follows:

[0037] μ←μ+αΔμ (8)

[0038] σ←σ-Δσ-ε σ (9)

[0039]

[0040] Where the parameter Δσ is a constant, and the parameter ε σ Adjust according to the actual situation of the task; the closer to the maximum reward value, the higher the parameter ε should be. σ The smaller;

[0041] After an update is completed, return to step 2-1 until the reward value stabilizes;

[0042] Step 3: Skill transfer from simulation environment to actual operating environment;

[0043] Step 3-1: Constructing the similarity function;

[0044] To compare the similarity between the dynamic parameters of the simulated environment and the actual environment, if the mathematical properties of this parameter are known, the KL divergence of the parameter's characteristic distribution in the simulated and actual environments can be directly calculated; otherwise, the dynamic properties of this parameter need to be estimated through comparative experiments. The final KL divergence matrix is ​​as follows, serving as a measure of similarity:

[0045] R=[D(x1,x′1),D(x2,x′2),…,D(x p ,x′ p ),...] T

[0046] Where D(x) p ,x′ p ) represents the dynamic parameter x p KL divergence of characteristics in simulation and reality;

[0047] Step 3-2: Feature parameter selection and transfer;

[0048] We measure the environmental dynamic parameters involved in each feature parameter K, and assign an importance weight w to each dynamic parameter based on its importance to the feature parameter. The importance weight matrix W is then constructed as follows:

[0049] W = [w1, w2, ..., w p ,…] T

[0050] Multiplying this value by the corresponding term in the similarity function matrix yields the relative error between the simulation and real environments for this feature parameter:

[0051] E(K) = W T *R (11)

[0052] If the relative error value corresponding to feature parameter K is greater than or equal to the threshold, then feature parameter K is used as a parameter that needs to be further explored in the real environment; if the relative error value is less than the threshold, then the value of feature parameter K is not changed, and it is directly transferred to the real environment; thus, the feature parameter K explored in the real environment is obtained.physical ;

[0053] Step 3-3: transfer of motion planning;

[0054] Compare the spatial features of the simulation and the actual environment, and compare the standard design of the simulation environment with the control parameters of the actual robot;

[0055] Step 4: skill optimization part in actual environment;

[0056] Step 4-1: generate action strategy for interaction with real environment;

[0057] Use Gaussian distribution to update feature parameters K physical Take N' vector values, denoted as: Update motion planning: Realize trajectory tracking by the method of formula (4);

[0058] Step 4-2: optimization of feature parameters;

[0059] Construct reward function Update feature parameters according to the methods of formula (5)-(10);

[0060] After one update is completed, return to step 4-1 until the algorithm converges, and the algorithm ends;

[0061] The final robot system obtains feature parameters in the actual operation environment, and thus masters the operation skill.

[0062] The beneficial effects of the present application are as follows:

[0063] (1) The rule-based operation stage task division used in the present application can effectively improve the interpretability of robot operation skill learning, and effectively reduce the state space of robot parameter exploration, with higher control precision.

[0064] (2) The operation skill learning framework proposed in the present application can realize the learning of multi-step complex skill strategy with fewer mechanical arm and environment interaction times, effectively improving the efficiency of robot skill learning.

[0065] (3) The skill transfer framework from simulation environment to actual operation environment proposed in the present application can greatly improve the safety in actual operation process, maximize the use of data obtained from simulation environment, and reduce the training time in actual operation environment. BRIEF DESCRIPTION OF DRAWINGS

[0066] Figure 1 The method framework diagram of the present application.

[0067] Figure 2Flow chart of the method of the present application.

[0068] Figure 3 Schematic diagram of a seven-degree-of-freedom mechanical arm jack task simulation scene in an embodiment of the present application.

[0069] Figure 4 Schematic diagram of a seven-degree-of-freedom mechanical arm jack task actual scene in an embodiment of the present application.

[0070] Figure 5 Schematic diagram of a jack operation stage division in an embodiment of the present application.

[0071] Figure 6 Convergence diagram of the t parameter obtained by simulation optimization in an embodiment of the present application. down Convergence diagram of the t parameter obtained by simulation optimization in an embodiment of the present application.

[0072] Figure 7 Convergence diagram of the x parameter obtained by simulation optimization in an embodiment of the present application. down Convergence diagram of the x parameter obtained by simulation optimization in an embodiment of the present application.

[0073] Figure 8 Convergence diagram of the t parameter obtained by simulation optimization in an embodiment of the present application. para Convergence diagram of the t parameter obtained by simulation optimization in an embodiment of the present application.

[0074] Figure 9 Convergence diagram of the x parameter obtained by simulation optimization in an embodiment of the present application. para Convergence diagram of the x parameter obtained by simulation optimization in an embodiment of the present application.

[0075] Figure 10 Convergence diagram of the reward function obtained by simulation optimization in an embodiment of the present application.

[0076] Figure 11 Convergence diagram of the reward function obtained by simulation optimization in an embodiment of the present application.

[0077] Figure 12 Convergence diagram of the reward function obtained by simulation optimization in an embodiment of the present application. DETAILED DESCRIPTION

[0078] The present application will be further described below in conjunction with the accompanying drawings and embodiments.

[0079] The present application aims to solve the problems of poor operation process explainability, long optimization time, and difficulty in application in actual scenes caused by the use of traditional learning algorithms in the process of complex operation skill learning of robots, and proposes a robot operation skill efficient learning method that can realize skill transfer from a simulation environment to an actual scene and is guided by human knowledge.

[0080] To achieve the above-mentioned purpose, the invention content involved in the present application is as follows:

[0081] (1) Based on the rule knowledge guidance of the complex operation task stage division and the generation of motion planning. The application uses the prior knowledge of human to analyze the flow of the operation skill to be learned, divides the operation process into several stages, each stage can be regarded as a point-to-point motion trajectory, and then determines the conversion condition between stages, the conversion condition is defined in the form of "if… then…", which represents a constraint of the environment on the agent.

[0082] (2) An operation skill learning optimization method combining expert preference evaluation and fixed index evaluation is constructed. The application uses the parameter exploration based policy gradient (PGPE) method as the basic method of operation skill learning optimization, uses the reward function as the feedback to the operator, adjusts the exploration variance, learning rate and other parameters of the exploration parameter by using the expert preference evaluation method, so as to speed up the convergence speed of the algorithm.

[0083] (3) An operation skill transfer and evaluation framework from a simulation environment to an actual scene is constructed. The application measures the difference between the simulation environment and the actual operation environment by constructing a similarity function, generates the exploration parameters and action instructions in the actual environment according to the similarity function, and completes the transfer of the operation skill.

[0084] (4) A logical framework for robot skill learning and skill transfer is constructed, as shown in Figure 1 , the completion degree of robot operation skill learning and the integrity degree of skill transfer can be gradually improved by the iterative method.

[0085] The application provides a personnel-simulation-real mixed training method for efficient operation skill learning of robots, as shown in Figure 2 , the operation skill process is analyzed, the operation stage division, key feature extraction, reinforcement learning strategy optimization, skill transfer and skill optimization in the actual operation scene are researched, and the specific implementation steps of the application are as follows:

[0086] Firstly, the operation process preprocessing part mainly includes four steps:

[0087] First step: dividing the operation process stage. The collection of all kinematics and dynamics parameters of the environment and robot involved in the operation process is called state space, which is represented by set S; for most operation tasks, it cannot be completed by a simple motion trajectory, so it is necessary to divide the operation process stage to efficiently complete the operation task. According to the prior knowledge obtained by human observation, all intermediate states required to complete the operation task can be extracted, and there are n intermediate states, so that all intermediate states can be represented as: <S0,S1,S2,…,Sn> nBased on these n intermediate states, the entire operation process can be divided into n stages, using... <G1,G2,…,G n > indicates. Stage G i Indicates from state S i-1 to state S i A state transition is represented by the following formula:

[0088] G i =trigger(S i-1 ,S i (1)

[0089] The description of the operation phase is in natural language, and its main purpose is to enable the observer to understand the impact of the robot's actions during this phase.

[0090] Step 2: Building upon Step 1, define the constraints for stage transitions. If there are n operation stages, n-1 transition condition judgment functions will be generated accordingly. <C1,C2,…,C n > indicates. Transformation condition C i Indicates the phase G i-1 Does the state satisfy stage G? i A conditional judgment.

[0091] The transition condition is described using a rule-based language that integrates quantitative expressions. The syntax of this rule-based language takes the form of a logical language like "If...then...", and its purpose is to define a non-linear mapping from a state to a decision variable; hence, the transition condition C... i It can be represented in the following form:

[0092]

[0093] Where conditions(S) i )∈S i , indicating stage G i The state that must be satisfied.

[0094] Step 3: Based on steps 1 and 2, generate the robot's motion plan. (Stage G) i This can be viewed as a point-to-point motion trajectory. Based on environmental constraints and task requirements, the robot can obtain the constrained motion trajectory, expressed by the following formula:

[0095]

[0096] Where parameter W i = <w i1 ,w i2 ,…,w im > represents the parameters required for generating the motion trajectory, bi represents the constraint condition. The set of all action plans is represented as: A = <A1, A2, …, A n >.

[0097] Fourth step: On the basis of the first, second and third steps, the extraction of key feature parameters is completed. The key feature parameters are the parameters to be optimized by the optimization algorithm, denoted by K. The selection range of the parameters includes: conditions(S), W, and b.

[0098] Secondly, the optimization of parameters in the simulation environment, mainly including the following four steps:

[0099] Fifth step: update the motion planning interacting with the simulation environment. On the basis of the previous four steps, this method uses Gaussian distribution to take N vector values of the feature parameters K, denoted as: where μ and σ are the mean and standard deviation of the Gaussian distribution of the feature parameters. Then update the motion planning in step three:

[0100] Sixth step: generate the execution action strategy. According to the motion planning A θ generated in the fifth step, the method of tracking control is used to realize trajectory tracking:

[0101]

[0102] where is the tracking error at time t.

[0103] Seventh step: calculate the reward function of the action strategy. Collect N trajectories ρ: <ρ1, ρ2, …, ρ N from the sixth step, construct the reward value Calculate the reward value of each execution task

[0104] Eighth step: optimization of key feature parameters. According to the reward function generated in the seventh step, calculate the gradient of the reward value to the variance of the feature parameters according to the gradient ascent algorithm, and then adjust the learning rate α of the parameter update through the following formula:

[0105]

[0106] where parameter ε μ represents a person's judgment of the iteration step of the parameter, which is made based on the situation of the parameter value corresponding to the reward value.

[0107] The formula for updating the parameters is as follows:

[0108] μ←μ+αΔμ (8)

[0109] σ←σ-Δσ-εσ (9)

[0110]

[0111] where parameter Δσ is a constant value, and parameter ε σ is adjusted according to the actual situation of the operation task. Generally speaking, the closer to the maximum reward value point, the smaller the parameter ε σ ;

[0112] After a complete update, return to the fifth step until the reward value is relatively stable.

[0113] Then is the skill transfer part from the simulation environment to the actual operation environment, mainly including the following three steps:

[0114] Step 9: Construction of similarity function. Compare the similarity of the dynamic parameters of the simulation environment and the actual environment. If the accurate mathematical characteristics of the parameter are known, the KL divergence of the parameter characteristic distribution in the simulation environment and the real environment can be directly calculated. Otherwise, the dynamics of the parameter needs to be estimated, which can be obtained by means of comparative test. The finally obtained KL divergence matrix is as follows, which is used as a measure of similarity.

[0115] R = [D(x1, x'1), D(x2, x'2), …, D(x p ,x′ p ), …] T

[0116] where D(x p ,x′ p ) is the KL divergence of the dynamics parameter x p in simulation and reality.

[0117] Step 10: Screening and transfer of characteristic parameters. On the basis of Step 9, measure the environment dynamics parameters involved in each characteristic parameter K, and give importance weight w to each dynamics parameter according to its importance to the characteristic parameter. Write the importance weight matrix W:

[0118] W = [w1, w2, …, w p ,…] T

[0119] Multiply the corresponding items of the similarity function matrix to obtain the relative error value of the simulation environment and the real environment of this characteristic parameter:

[0120] E(K) = W T *R (11)

[0121] If the relative error value corresponding to this feature parameter is large, it indicates that the results obtained from the simulation environment are unreliable, and this feature parameter will be used as a parameter that needs further exploration in the real environment; if the relative error value is less than the threshold, the original feature parameter value is not changed, and it is directly transferred to the real environment. The feature parameter to be explored in the real environment is: K. physical .

[0122] Step 11: Motion Planning Transfer. Building on Step 10, compare the spatial characteristics of the simulation and the actual environment, and design the control parameters of the actual robot according to the standards of the simulation environment, so as to make the performance indicators of the robot system in reality and simulation as similar as possible, such as motion time, tracking accuracy, and control convergence time.

[0123] Finally, there is the skill optimization part in the real-world environment, which mainly includes the following two steps:

[0124] Step 12: Generate action strategies for interacting with the real environment. Use a Gaussian distribution on the feature parameters K. physical Take N′ vector values, represented as: Update your exercise plan: Trajectory tracking is achieved using the method of equation (4).

[0125] Step 13: Feature parameter optimization. Constructing the reward function. The characteristic parameters are updated according to the methods in equations (5) to (9).

[0126] After each update, return to step 12 until the algorithm converges and terminates. Ultimately, the robot system will obtain relatively ideal feature parameters for the actual operating environment, thus mastering this operational skill. Specific implementation examples:

[0128] To demonstrate the effectiveness of the proposed human-simulation-reality hybrid training system for efficient robot operation skills learning, a seven-DOF robotic arm was used to learn pin insertion skills. The selected seven-DOF robotic arm was a KUKA iiwa; the cross-sectional shapes of the hole and pin were circular, with radii of 29.8 mm for the pin and 30.3 mm for the hole. The simulation operation scenario is as follows: Figure 3 As shown, the simulation environment was built using Issac Sim; the actual operation scenario is as follows. Figure 4 As shown. The specific implementation process is as follows:

[0129] 1) Divide the operation process into stages. Based on prior human knowledge, the insertion process is divided into the following 5 stages according to the contact state: initial alignment, descent contact, horizontal movement, posture adjustment, and insertion, such as... Figure 5 As shown.

[0130] 2) Building upon the first step, define the constraints for stage transitions. Five stages will generate four transition conditions, described in rule language as follows:

[0131] C1: If the pin end position satisfies:<X0,φ0> Then, we enter the "descent contact" phase.

[0132] C2: The force at the end of the pin satisfies: F z >ε z Then, we enter the "horizontal movement" phase.

[0133] C3: If the pin force and speed satisfy: F y >ε y and v y <∈ y Then we enter the "posture adjustment" stage.

[0134] C4: If the pin end posture satisfies: Then we proceed to the "insertion" stage.

[0135] make<X0,φ0> Let ε be a fixed position above the hole plane. z =3.8N, ε y =5N,∈ y =0.01m / s.

[0136] 3) Based on steps one and two, generate the robot's motion plan. Use the "minimum jitter trajectory" as the basic form of motion planning; it minimizes speed jitter during movement. The expression is: Select initial position d i Termination position d f Running time t f This allows us to determine a trajectory.

[0137] 4) Building upon steps one, two, and three, extract the key feature parameters. Considering that the purpose of learning operational skills is to complete this task quickly and safely, the following four upper-level control parameters were selected: descent time t down descent distance x down Horizontal movement distance x para Horizontal movement time t para .

[0138] 5) Update the motion planning for interaction with the simulation environment. Use a Gaussian distribution to assign N = 20 vector values ​​to the characteristic parameter K, represented as: The initial values ​​are respectively: Generate based on the stage division and the motion trajectory of each stage.

[0139] 6) Generate and execute motion strategies. Inverse kinematics is used to track the motion trajectory at the robotic arm's end effector. The current position error is calculated. Adopted To perform control, the control parameters for the vertical and horizontal directions are: K p =[10000,5000], K I =[50,50], K D =[1000,1000].

[0140] 7) Calculate the reward function of the action policy. Collect N trajectory segments ρ: <ρ1,ρ2,…,ρ from step 6. N The reward value is constructed as follows:

[0141] The force-bearing parts of the robotic arm:

[0142] Time section:

[0143] Success and failure:

[0144] Calculate the reward value for each task execution.

[0145] 8) Optimization of key feature parameters. The gradient ascent algorithm is used to calculate the gradient of the reward value with respect to the variance and standard deviation of the feature parameters. The learning rate for each iteration is estimated according to equations (5) to (7) so that the change in parameters for each iteration does not exceed 10% of the maximum change.

[0146] The variance of the exploration parameters is calculated according to equation (9), and the exploration variance is reduced to half of its original value every 20 iterations.

[0147] After 80 iterations in the simulation environment, the parameters basically stabilized. The graphs of the four exploratory parameter iterations are as follows: Figures 6-9 As shown; the curve of the reward function is as follows. Figure 10 As shown.

[0148] 9) Construction of the similarity function. (This is related to the descent time t.) down and descent distance x down The relevant dynamic parameters are the vertical elastic modulus of the hole plane and the horizontal displacement x. para and horizontal movement time t paraThe relevant kinetic parameters are the friction coefficient and the horizontal elastic modulus. Since the exact differential equations of these parameters cannot be obtained, their distributions can only be estimated. This patent uses the stable contact force in the vertical direction without feedback as an estimate of the numerical elastic modulus; uses the average horizontal moving speed as an estimate of the friction coefficient; and uses the stable contact force in the horizontal direction without feedback as an estimate of the horizontal elastic modulus.

[0149] Actual demonstration verification: the vertical direction contact force in the simulation environment reaches 140N±5N; the real environment is 41N±10N; the average horizontal moving speed in the simulation environment is 30mm / s±0.5mm / s; the average horizontal moving speed in the real environment is 27.5mm / s±1.2mm / s; the horizontal direction contact force in the simulation environment reaches 14.8N±1.3N; the horizontal direction contact force in the real environment reaches 15.3N±4.5N.

[0150] The KL divergence matrix of the calculated parameters is:

[0151] R=[49.3231,2.6324,0.7896] T

[0152] 10) Selection and transfer of characteristic parameters: the exploration parameters are given the importance weight of the kinetic parameters: t down =x down =[1,0,0] T ; x para =t para =[0,0.5,0.5] T , the calculation results of formula (10) are as follows:

[0153] E(t down ) = E(x down ) = 49.3231

[0154] E(t para ) = E(x para ) = 1.7110

[0155] It can be seen that the error parameter calculation result of the falling time t down and the falling distance x down is much larger than that of the other two parameters, so it needs to be the parameter K physical further explored in the actual environment; the error parameters of parameters x para and t para are small, and the parameters obtained by simulation optimization can be used.

[0156] 11) Transfer of motion planning. Compare the spatial features of the simulation and the real environment, and adjust the control parameters of the real robot according to the standard design of the simulation environment, so that the performance indicators of the robot system in real and simulation, such as motion time, tracking accuracy, control convergence time, etc. are as similar as possible. The actual robot uses impedance control motion mode. In order to realize the precise motion tracking of the robot, through debugging, the impedance control parameters in the vertical and horizontal directions are respectively: 3000 N / m, 1000 N / m.

[0157] 12) Generate action strategy for interaction with real environment. Use Gaussian distribution to update the feature parameters K physical Take N' = 10 vector values, denoted as: Update the motion planning:

[0158] 13) Optimization of feature parameters. Construct the reward function Update the feature parameters according to the method of equations (5) to (10).

[0159] After 5 iterations, the reward value is close to the optimal value. The results of parameter iteration in the real environment are shown in Figure 11 , and the reward function curve is shown in Figure 12 .

Claims

1. A human simulation-reality hybrid training method for learning robot operation skills, characterized in that, Includes the following steps: Step 1: Pre-processing of the operation; Step 1-1: Divide the operation process into stages; The set of all kinematic and dynamic parameters of the environment and robot involved in the operation process is called the state space, denoted by the set S; Extract all intermediate states required to complete the operation task. Let there be n intermediate states, then all intermediate states are represented as follows: <S0,S1,S2,…,S n Based on these n intermediate states, the entire operation process is divided into n stages, using... <G1,G2,…,G n >Indicates; Stage G i Indicates from state S i-1 to state S i A state transition is represented by the following formula: G i =trigger(S i-1 ,S i ) (1) Step 1-2: Define the constraints for the phase transition; If there are n operation stages, then n-1 transformation condition judgment functions will be generated accordingly. <C1,C2,…,C n > indicates; conversion condition C i Indicates the phase G i-1 Does the state satisfy stage G? i A conditional judgment; Transformation condition C i It can be represented in the following form: Where conditions(S) i )∈S i , indicating stage G i The state that must be satisfied; Steps 1-3: Generate the robot's motion plan; Phase G i Viewed as a point-to-point motion trajectory, the robot obtains the constrained motion trajectory based on environmental constraints and task requirements, expressed by the following formula: Where parameter W i = <w i1 ,w i2 ,…,w im > represents the parameters required for generating the motion trajectory, b i The constraints are represented; the set of all action plans is represented as: A = <A1,A2,…,A n >; Steps 1-4: Complete the extraction of key feature parameters; Key feature parameters are the parameters that the optimization algorithm wants to optimize, denoted by the symbol K. The selection range of parameters includes: conditions(S), W, and b. Step 2: Parameter optimization in the simulation environment; Step 2-1: Update the motion plan that interacts with the simulation environment; Using a Gaussian distribution to take N vector values ​​for the feature parameter K, it can be represented as: Where μ and σ are the mean and standard deviation of the Gaussian distribution of the characteristic parameters, respectively; then update the motion planning in steps 1-3: Step 2-2: Generate the action strategy; Track tracking is achieved using a tracking control method: in It is the tracking error at time t; Steps 2-3: Calculate the reward function of the action policy; From step 2-2, collect N trajectory segments ρ:<ρ1,ρ2,…,ρ N >, construct reward value Calculate the reward value for each task execution. Steps 2-4: Optimization of key feature parameters; Based on the reward function generated in steps 2-3, the gradient ascent algorithm is used to calculate the gradient of the reward value with respect to the variance of the feature parameters. Then, the learning rate α for parameter updates is adjusted using the following formula: α=α0+ε μ (7) Where the parameter ε μ The representative's judgment on the parameter iteration step size is based on the parameter values ​​corresponding to the reward value; The formula for parameter update is as follows: μ←μ+αΔμ (8) s←s-Ds-e σ (9) Where the parameter Δσ is a constant, and the parameter ε σ Adjust according to the actual situation of the task; the closer to the maximum reward value, the higher the parameter ε should be. σ The smaller; After an update is completed, return to step 2-1 until the reward value stabilizes; Step 3: Skill transfer from simulation environment to actual operating environment; Step 3-1: Constructing the similarity function; To compare the similarity between the dynamic parameters of the simulated environment and the actual environment, if the mathematical properties of this parameter are known, the KL divergence of the parameter's characteristic distribution in the simulated and actual environments can be directly calculated; otherwise, the dynamic properties of this parameter need to be estimated through comparative experiments. The final KL divergence matrix is ​​as follows, serving as a measure of similarity: R=[D(x1,x′1),D(x2,x′2),…,D(x p ,x′ p ),…] T Where D(x) p ,x′ p ) represents the dynamic parameter x p KL divergence of characteristics in simulation and reality; Step 3-2: Feature parameter selection and transfer; We measure the environmental dynamic parameters involved in each feature parameter K, and assign an importance weight w to each dynamic parameter based on its importance to the feature parameter. The importance weight matrix W is then constructed as follows: In=[in1,in2,…,in p ,…] T Multiplying this value by the corresponding term in the similarity function matrix yields the relative error between the simulation and real environments for this feature parameter: E(K)=W T *R (11) If the relative error value corresponding to feature parameter K is greater than or equal to the threshold, then feature parameter K is used as a parameter that needs to be further explored in the real environment; if the relative error value is less than the threshold, then the value of feature parameter K is not changed, and it is directly transferred to the real environment; thus, the feature parameter K explored in the real environment is obtained. physical ; Step 3-3: Shift in motion planning; Compare the spatial characteristics of the simulation and the actual environment, and design the control parameters of the actual robot according to the standards of the simulation environment; Step 4: Skill optimization in a real-world environment; Step 4-1: Generate action strategies for interacting with the real environment; Using a Gaussian distribution on the characteristic parameter K physical Take N′ vector values, represented as: Update your exercise plan: Trajectory tracking is achieved using the method in equation (4); Step 4-2: Optimization of feature parameters; Constructing a reward function The feature parameters are updated according to the methods in equations (5) to (10); After each update is completed, return to step 4-1 until the algorithm converges and then ends. Ultimately, the robot system obtains the characteristic parameters in the actual operating environment, and thus masters this operating skill.

Images