Human-machine cooperation humanoid motion autonomous planning method and system based on motion primitives and double DQNs

Abstract

The invention relates to a human-machine cooperation humanoid motion autonomous planning method and system based on motion primitives and double DQNs, and belongs to the technical field of robots. According to the method, through a human-human cooperative motion experiment, human arm action primitive types are extracted, and selection factors of the human arm action primitive types are analyzed; a double-DQN primitive decision-making method based on a long-short-term memory network is adopted, a reward function is designed according to the action primitive frequency, a loss function related to the reward function is constructed, and selection of primitive types is achieved; establishing an autonomous planning method; and through a human-machine cooperation experiment, collaborator feedback data is obtained, double-DQN model parameters are optimized, and finally, the robot humanoid motion autonomous planning ability and the acceptability of the collaborator to robot motion are verified. According to the method, the problem of robot humanoid motion autonomous planning in contact type man-machine cooperation is solved, and the humanoid degree of robot motion and the acceptability of a collaborator to robot motion are remarkably improved.

Description

TECHNICAL FIELD

[0001] The application relates to a human-robot collaborative humanoid motion autonomous planning method and system based on action primitives and double DQN, and belongs to the technical field of robots. BACKGROUND

[0002] Under the background of intelligent rapid development, robot collaboration is facing new technical requirements. Modern collaboration scenarios not only require robots to have environmental autonomous perception and human-like decision-making ability, but also put forward higher standards for robot motion humanization to ensure that human operators can intuitively understand and accurately predict robot behavior. Current humanoid motion planning methods mainly face non-contact human-robot collaboration scenarios, such as the material transfer task between conveyors A and B as shown in the figure. In such scenarios, the robot usually completes the complete trajectory planning before motion execution and strictly follows the preset parameters for execution during the motion process without real-time adjustment according to the environment dynamics. Figure 1

[0003] However, when it comes to contact human-robot collaboration scenarios (such as collaborative assembly work on conveyor B), the limitations of traditional methods become apparent. In such scenarios, physical contact between humans and robots can cause dynamic changes in robot motion goals, forcing the planned motion trajectory to be adjusted in real time. More critically, the unpredictability or non-humanization of robot motion can cause human collaborators to feel anxious, uneasy, or even resistant, thereby significantly reducing collaboration willingness and work efficiency.

[0004] Therefore, in a contact human-robot collaboration environment, the robot not only needs to generate a highly humanized motion trajectory, but also must have adaptive motion planning capability based on real-time feedback from the collaborator. The realization of such humanoid motion autonomous planning capability is of decisive significance for improving the collaborator's trust in the robot and establishing a natural and smooth human-robot collaboration relationship. SUMMARY

[0005] In view of the deficiencies of the prior art, the present application provides a human-robot collaborative humanoid motion autonomous planning method based on action primitives and double DQN, which enhances the humanization of robot motion from the kinematics level by introducing human arm action primitives. Secondly, the double deep Q network (double DQN) algorithm is used to enhance the autonomous decision-making ability of the robot in human-robot collaboration, so that it can autonomously adjust the motion strategy according to the real-time changes of the human-robot collaboration state.

[0006] The technical solution of the present application is as follows: A human-robot collaborative humanoid motion autonomous planning method based on action primitives and double DQN, the steps are as follows: Step 1, obtain human-human collaboration experimental data and preprocess them; ​Step 2, extract the human arm motion primitive types in human-human collaboration and analyze the main influencing factors affecting the selection of primitive types; Step 3, use the double DQN primitive decision method based on long short-term memory network (LSTM) to select the primitive types in human-human collaboration; Based on the action primitive frequency, design the reward function of the double DQN model, and construct the loss function of the double DQN as a function related to the reward function; Step 4, integrate the action primitive type selection method of step 3 into the human motion planning, and establish an autonomous human motion planning method including the admittance control module, the action primitive decision module, the primitive parameter estimation module, the human-machine mapping module and the joint design module; Step 5, carry out human-machine collaboration experiment based on the autonomous human motion planning method, and obtain human-machine collaboration experimental data containing real-time feedback of the collaborator; Step 6, design the reward function of the double DQN model based on the collaborator feedback data, and correct the parameters of the double DQN primitive type decision model based on LSTM; Step 7, based on the primitive decision correction model in step 6, carry out human-machine collaboration experiment to verify the degree of robot human motion and the acceptance of robot motion in human-machine collaboration.

[0007] Preferably, in step 1, the human-human collaboration experiment refers to designing a standardized experimental scene, requiring two human collaborators to start from the same starting point, cooperate by holding a force sensor together, and finally reach the same target point. One of the two is the leader and the other is the follower. The core purpose of this experiment is to collect natural motion characteristics of humans in the process of contact collaboration, including but not limited to: changes in joint angles of elbows, shoulders and wrists, end motion position, speed, acceleration and contact force (measured by force sensor).

[0008] Preferably, in step 1, the obtained experimental data is preprocessed, and the preprocessing includes removing noise and eliminating invalid motion data; The method for removing noise is to replace the current value with the mean value in the sliding window for time series data to suppress high-frequency noise, and the formula used is: (1) k is the window size, further, k=5, x t+i is the original data, y t is the data after denoising; The method for eliminating invalid motion data is to remove the data segment whose hand end motion speed is less than 0.001 m / s except at the beginning and end of the motion; The preprocessed collaboration data is used as the basis data for analyzing the influence factors of primitive types.

[0009] Preferably, in step 2, a human arm action primitive is defined, which is a way of expressing human arm movement. By analyzing the structure and movement characteristics of the human arm, it is proposed to express the movement of the human arm by the position (P) and posture (O) of the arm end, the self-rotation angle (Φ) and the interaction force (F), and the four basic movement elements are defined as basic action primitives; wherein, P O F P [ x , y , z ] and O [ r x , r y , r z ] respectively represent three Euler positions and corresponding direction angles, the interaction force between the human hand or robot and the external environment, and the self-rotation angle Φ is defined as the rotation angle of the elbow of the arm rotating from position E to position E1 around the dashed line axis SW, which is the line connecting the hand and the shoulder. Similarly, the movement of a robot arm with a humanoid arm can also be expressed by basic action primitives; Human arm action primitive type extraction: the human arm action primitive type extraction process in human-human collaboration is: 1) extract the original movement data of the follower's right arm from the BVH format data derived from the human body motion capture system; 2) obtain the angle change sequence of each joint in the collaboration task; 3) preprocess the obtained joint angle data, which includes removing noise and eliminating invalid movement data; 4) obtain the basic action primitives, including: the self-rotation angle Φ, the position P of the arm end, and the change sequence of the posture O; 5) determine the action primitive type of each frame of movement according to the change of the basic action primitive value, including ΦPOF, ΦPF, POF, Φ, F, PF, ΦPO, PO.

[0010] ​​​The main influencing factor analysis process of primitive type selection is: (1) According to the change rule of human-human collaborative interaction force, the interaction force is divided into three force intervals, which are low interaction force interval [0, 35%), medium interaction force interval [35%, 65%) and high interaction force interval [65%, 100%], wherein the percentage represents the ratio of the interaction force to the peak value; based on the ergonomic characteristics of human arm, the motion space of human-human collaborative motion is divided into three position regions, which are human-human collaborative regions A, B and C, wherein region A contains starting points S1 and S2 and target points E1, E2, E3 and E4, region B contains starting point S3 and target points E5 and E6, and region C contains starting points S4 and S5 and target points E7, E8, E9 and E10; according to the speed and acceleration law of human-human collaborative motion, the collaborative motion process is divided into four continuous motion stages: starting stage, characterized by starting low speed and increasing acceleration; intermediate stage 1, characterized by continuously increasing speed to peak value; intermediate stage 2, characterized by stable maintenance of peak value; and ending stage, characterized by speed reduction and motion termination; (2) Based on the human-human collaboration experimental data obtained in step 1, the primitive types (such as ΦPOF, ΦPF, POF, Φ, PF, ΦPO and PO) and frequencies of the three force intervals, three position regions and four motion stages are counted respectively, and the frequency is the ratio of the number of occurrences of a certain type of primitive to the total number of primitives; (3) Variance analysis is used to verify the significance of the influence of interaction force, spatial position and motion stage on the selection of primitive type; the variance calculation formula is: (2) and are the between-group and within-group mean squares, and are the between-group and within-group degrees of freedom, and are the between-group and within-group sum of squares; each force interval or position region or motion stage is a group, and there are 10 groups in total; (4) According to the F value obtained from the variance calculation formula and the corresponding degrees of freedom, refer to the F distribution table to find the critical F value in the table, if the calculated F value is greater than the critical F value in the table, then P<0.05, when P<0.05, it indicates that the influence of interaction force, spatial position and motion stage on the selection of primitive type is significant, otherwise, it is not significant; the parameters that are significantly affected are determined as the main influencing factors, which refer to the interaction force, spatial position and motion stage; the result determined here is "interaction force is the main influencing factor" / "spatial position is the main influencing factor" / "motion stage is the main influencing factor".

[0011] Preferably, in step 3, the double DQN primitive decision-making method based on LSTM includes: (3-1) a double network structure, including an estimated network Q updated in real time and a target network Q' synchronized periodically, wherein the Q and Q' networks are designed based on LSTM; (3-2) a loss function based on a reward function, used to guide the optimization process of network parameters; and (3-3) an experience replay pool, used to store the state of an agent at a certain time, an action performed in the current state, and a feedback reward given after the action is performed.

[0012] Further preferably, in (3-1), the learnable weights of the LSTM network are input weights W [ W i , W f , W g , W o ] T , recurrent weights R w [ R i , R f , R g , R o ] T and biases b [ b i , b f , b g , b o ] T , wherein f , g , i and o represent a forget gate, a candidate cell, an input gate, and an output gate, respectively; the forget gate f is used to forget information that has little influence on the selection of the primitive type, so as to reduce the influence of excessive memory information on the LSTM network. The motion state information that has a great influence on the selection of the primitive type is added to the cell state σ g by using an activation function c (sigmoid), that is:

[0013] f t = σ g ( W fS t + R f a t-1 + b f ) (3) f t for t Always forget the door, S t is the motion status information, a t-1 for t -1 The action primitive type selected at the moment, W f 、 R f and b f is the weight coefficient of the forget gate; σ g The output is a value between [0, 1]. The closer the output value is to 1, the more influential the corresponding information is on the selection of primitive type and will be retained. Otherwise, it will be forgotten. The model will automatically adjust the weight and bias of the forget gate according to the characteristics of the data to determine which information needs to be forgotten. The motion state information with greater influence selected here determines the specific data information: the size of the interaction force at the current moment, position data, etc.

[0014] Candidate Unit g and input gate i The effective information of the current motion state information and the primitive type information of the previous moment are extracted together, and the activation function is used σ c (tanh) and σ g (sigmoid) added to the cell state c Middle; the motion state information S that has a higher degree of influence on the selection of primitive type t The easier it is to be memorized into the unit state, that is: g t = σ c ( W g S t + R g a t-1 + b g ) (4) i t = σg ( W i S t + R i a t-1 + b i ) (5) g t for t Time to choose the door, i t for t Enter the door at all times, W g 、 R g and b g is the candidate gate weight coefficient, W i 、 R i and b i is the input gate weight coefficient, σ g ( x )=(1+ e -x ) -1 , ; Output Gate o Used to integrate the current motion state and the primitive type of the previous moment, using the activation function σ g (sigmoid) extracts the information, namely: o t = σ g ( W o S t + R o a t-1 + b o ) (6) o t for t Output gate at all times, W o 、 R o and b o is the output gate weight coefficient, whereσ g ( x )=(1+ e -x ) -1 ; The cell state c encompasses the "summary memory" of all input motion state information of the LSTM network at previous time steps, which integrates t the cell state information at time -1, t the forget information at time -1, t the input information at time -1, and t the information that needs to be remembered at time -1 as the cell state information at time t, that is: t (7) wherein, represents the Hadamard product, the corresponding elements of the vectors are multiplied; t The cell state information at time t is processed by the activation function, and then combined with the output gate information at time t to obtain the output information at time t t t a t , that is, the basic type; ; (8) To optimize the double DQN network parameters, a special training data set is constructed, and the input feature vector of the training data set contains the position information of the human arm / robot end motion X [ x , y , z ], motion speed information , acceleration information , interaction force information , self-rotation angle Φ value and collaborative heart rate change value r mc , the training data input set S t The mathematical expression can be expressed as: (9) wherein, N i is the total number of samples, i is the sample index; N t is the total number of frames of each sample, t is the frame number index of each sample, r mc = r / r m , r ​​​For the cooperator heart rate, r m For the human-human collaboration / human-robot collaboration heart rate maximum value; The output layer of the double DQN network will be mapped to one of the eight primitive action types, and the output set a t Can be expressed as: (10) For the convenience of model training, 8 types of action primitives are mapped to discrete integer labels: 1-ΦPOF, 2-ΦPF, 3-POF, 4-Φ, 5-F, 6-PF, 7-ΦPO, 8-PO.

[0015] Further preferably, in (3-2), the loss function L Is defined as follows: (11) Where, S t And S t+1 Respectively, the state of the robot t At time t and t +1 time; a t And a t+1 Respectively, the action primitive type selected at time t At time t and t +1 time; Q ( S t , a t ) and Q ´( S t+1 , a t+1 ) are the estimated network t Value at time t and Q +1 time target network t Value; Q Indicates that all possible a t+1 , select the one that makes Q ´( S t+1 , a t+1 ) the maximum action primitive type; Is the discount rate / hyperparameter, which converts many steps of future rewards to the current point; Is the learning rate; R Is the reward function; The reward function is defined as: ​ (12) wherein, p is the frequency of each primitive type; a 1 and b 1 is a set coefficient, further preferably, according to the statistical frequency distribution of each primitive type, a1=10, b1=0.4.

[0016] Preferably, in step 4, the human-like motion autonomous planning method comprises an admittance control module, an action primitive decision module, a primitive parameter estimation module, a human-robot mapping module and a joint design module. The admittance control module is used to drive the robot to conform to the motion of the human leader, and the adopted admittance control model is: (13) M ∈ R 6×6 , C ∈ R 6×6 and K ∈ R 6×6 are respectively a virtual mass matrix, a virtual damping matrix and a virtual stiffness matrix; is a Cartesian space interaction force vector, f x , f y , f z respectively represent the xyz axis interaction force values, respectively represent the xyz axis torque values; X d =[ x d , y d , z d , r xd , r yd , r zd ] T is a desired pose vector, x d , y d , z d respectively are the xyz axis desired positions, r xd , r yd , r zd respectively are the xyz axis desired attitude angles;X r [ x r , y r , z r , r xr , r yr , r zr ] T are actual position vectors of the robot, x r , y r , z r are actual position vectors of the robot, r xr , r yr , r zr are actual position vectors of the robot, is a desired velocity, is an actual velocity; is a desired acceleration, is an actual acceleration.

[0017] For the convenience of application to robot control, the admittance control model can be converted into a discrete form so as to solve the desired trajectory of the robot: (14) (15) (16) wherein, X d ( t )、 and are the desired position, the desired velocity and the desired acceleration at time t ; X r ( t )、 and are the actual position, the actual velocity and the actual acceleration at time t ; for the convenience of formula, let , ; T is a sampling period.

[0018] The action primitive decision module determines the type of action primitive that the robot should perform at the next time point in real time according to the current motion state of the robot; the action primitive decision module receives real-time motion state data of the robot in human-robot collaboration S t , the information is filtered and updated through the forget gate, input gate and output gate of the LSTM to determine the optimal action primitive type that the robot should perform at the next time point; the robot motion is planned according to the selected action primitive type, and the new reward is fed back to the LSTM after the motion is executed, and the motion state of the robot is taken as the input of the next cycle to form a closed-loop control system to optimize the fluency and efficiency of human-robot collaboration; The primitive parameter estimation module is used to convert the expected motion pose of the robot from the admittance control module X d =[ x d , y d , z d , r xd , r yd , r zd ] T and the action primitive type of the action primitive decision module into the expected value of the basic action primitive Φ at the next time point, and the conversion formula is as follows: (17) is the expected value of the basic action primitive Φ at the next time point, when the basic action primitive Φ is contained in the primitive type (such as the primitive type ΦPOF), the expected value of the basic action primitive Φ at the next time point is calculated according to the above formula, otherwise, the existing value is maintained; The human-robot mapping module is used to map the data of each basic action primitive of the human arm to the data of each basic action primitive of the robot, and to make corresponding adjustments to the action according to the configuration difference (the adjustment is to modify the model parameters W, R w and b) after designing a new reward function according to the heart rate of the collaborator in step 6. The joint design module is used to determine the values of the joint angles of the robot from the determined values of each basic action primitive.

[0019] Preferably, in step 5, a human-like motion autonomous planning method is used to plan the motion of the robot during the human-robot collaboration motion experiment; in order to obtain the feedback data of the collaborator participating in human-robot collaboration, i.e. the heart rate of the collaborator, the human collaborator always wears a heart rate monitoring bracelet during the experiment, so that the motion of the robot can be adjusted through the heart rate state subsequently.

[0020] Preferably, in step 6, the reward function of the double DQN model based on the feedback of the collaborator is: (18) Wherein, r is the heart rate of the collaborator, a 2 and b 2 is a set coefficient, and further preferably, a 2 and b 2 According to the characteristics of the change of the heart rate of the collaborator during human-robot collaboration, a2=0.15 and b2=130 are taken.

[0021] The human-robot collaboration motion data collected in step 5 are used, and the reward function newly designed is combined to correct the parameters (W, R w and b) of the double DQN primitive type decision model based on LSTM in step 3, so that the robot can move like a human while the collaborator can accept the behavior of the robot.

[0022] Preferably, in step 7, the human-like motion autonomous planning method based on the corrected model of the primitive decision is used to plan the motion of the robot; the degree of human-likeness of the motion of the robot is evaluated from subjective and objective aspects; the method of measuring the heart rate of the collaborator is used to objectively evaluate the acceptance of the collaborator to the motion of the robot; the method of questionnaire survey is used to subjectively evaluate the acceptance of the collaborator to the motion of the robot; the questionnaire survey includes the emotional state questionnaire and the robot motion acceptance questionnaire.

[0023] A computer readable storage medium having a program stored thereon, the program being executed by a processor to implement the steps in the human-robot collaboration human-like motion autonomous planning method based on action primitives and double DQN as described above.

[0024] An electronic device comprising a memory, a processor, and a program stored on the memory and executable on the processor, wherein the processor implements the steps in the human-robot collaboration human-like motion autonomous planning method based on action primitives and double DQN as described above when executing the program.

[0025] The present application has the following advantages: 1. For the real-time adjustment problem of motion caused by real-time changes of targets in contact human-robot collaboration, a human-like motion autonomous planning framework comprising four modules is proposed. The framework simulates the human collaboration behavior mode, gives the robot efficient online motion re-planning ability, enables the robot to effectively collaborate with humans like human collaboration, and thus fully utilizes the advantages of human-robot collaboration.

[0026] 2. For the problem of autonomous selection of action primitives according to the feedback of the collaborator, a double DQN model based on LSTM is proposed. According to the reaction (change of emotional state) of the collaborator, the robot can autonomously adjust its movement in order to improve the acceptance of the robot movement. BRIEF DESCRIPTION OF DRAWINGS

[0027] Figure 1 Schematic diagram of human-robot collaborative assembly scene; Figure 2 Schematic diagram of human-human collaborative scene; Wherein (a) is an experimental scene, (b) is a position diagram containing a starting point and a target point; Figure 3 Schematic diagram of human arm / robot arm basic action primitive; Wherein (a) is a schematic diagram of the basic action primitive of the human arm, (b) is a schematic diagram of the basic action primitive of the humanoid arm robot; Figure 4 Schematic diagram of double DQN-based primitive decision framework; Figure 5 Schematic diagram of LSTM network structure; Figure 6 Schematic diagram of the framework of the autonomous planning method of the humanoid motion; Figure 7 Schematic diagram of the action primitive decision module frame; Figure 8 Schematic diagram of human-robot collaborative experiment; Wherein (a) is an experimental scene, (b) is a schematic diagram of setting a starting point and a target point; Figure 9 Schematic diagram of robot acceptance questionnaire. DETAILED DESCRIPTION

[0028] The technical solutions in the embodiments of the application will be described clearly and completely below with reference to the drawings in the embodiments of the application. Obviously, the described embodiments are only part of the embodiments of the application, rather than all the embodiments of the application. Based on the embodiments in the application, all other embodiments obtained by those skilled in the art without creative labor fall within the protection scope of the application.

[0029] In the following description, many specific details are set forth in order to provide a thorough understanding of the application, but the application can also be implemented in other ways different from those described herein, and those skilled in the art can make similar generalizations without departing from the concept of the application, therefore the application is not limited to the specific embodiments disclosed below.

[0030] Embodiment 1 An action primitive and double DQN-based human-robot collaborative simulation human motion autonomous planning method, the human arm action primitive and the double deep Q network (DQN) based on long short-term memory network (LSTM) are combined to establish a robot simulation human motion autonomous planning method, which is used for robot motion planning in contact human-robot collaboration; the method comprises the following steps: Step 1, obtaining human-human collaboration experimental data and preprocessing.

[0031] The human-human collaboration experiment refers to designing a standardized experimental scene, requiring two human collaborators to start from the same starting point, hold a force sensor together, and finally reach the same target point, and the two human collaborators respectively act as the leader and follower of the collaborative motion; the leader guides the follower from the starting point to the target point by holding the handle of the force measuring device. The leader tries to keep the guiding path straight. The follower follows the leader by holding the other handle of the force measuring device. The follower does not need to know the exact motion path and target, and only actively follows the motion of the leader by relying on its own displacement, speed and force, giving the leader an impression of submission. The experimental scene and the starting point and target point are set as shown in Figure 2

[0032] The core purpose of the experiment is to collect the natural motion characteristics of humans in the contact collaboration process, including but not limited to: changes in joint angles of the elbow, shoulder and wrist, end motion position, speed, acceleration and contact force (measured by force sensor).

[0033] The obtained experimental data is preprocessed, and the preprocessing includes removing noise and eliminating invalid motion data.

[0034] The method for removing noise is to replace the current value with the mean value in the sliding window for time series data to suppress high-frequency noise, and the formula used is: (1) k is the window size, further, k=5, x t+i is the original data, y t is the data after denoising.

[0035] The method for eliminating invalid motion data is to remove the data segment whose end motion speed is less than 0.001 m / s except for the start and end of the motion.

[0036] The preprocessed collaboration data is used as the basic data for analyzing the main influencing factors of the primitive type.

[0037] Step 2, extracting human arm action primitive types in human-human collaboration and analyzing the main influencing factors of primitive type selection.

[0038] ​Define the human arm action primitive, which is a way to express the human arm movement. Through the analysis of the structure and movement characteristics of the human arm, it is proposed to use the arm end position ( P ) and posture ( O ), rotation angle (Φ) and interaction force ( F ) to express the movement of the human arm, and define these four basic movement elements as basic action primitives; among them, P =[ x , y , z ]and O =[ r x , r y , r z ] represent the three Euler positions and the corresponding direction angles, It represents the interaction force between the human hand or robot and the external environment. The rotation angle Φ is defined as the rotation angle of the arm elbow from position E to position E1 around the dotted axis SW. The dotted axis SW is the line connecting the hand and the shoulder, as shown in Figure 3 Similarly, the motion of a robot arm with a humanoid arm can also be expressed using basic action primitives, such as Figure 3 As shown in (b).

[0039] Extraction of human arm motion primitive types: The process of extracting human arm motion primitive types in human-human collaboration is as follows: 1) Extract the original motion data of the follower's right arm from the BVH format data exported by the human motion capture system; 2) Obtain the angle change sequence of each joint in the collaborative task; 3) Preprocess the obtained joint angle data, which includes removing noise and eliminating invalid motion data; 4) Obtain basic motion primitives, including: the change sequence of the rotation angle Φ, the arm end position P, and the posture O; 5) Determine the motion primitive type of each frame of motion according to the change of the basic motion primitive values, including ΦPOF, ΦPF, POF, Φ, F, PF, ΦPO, PO.

[0040] The analysis process of the main influencing factors for the selection of primitive types is as follows: (1) According to the variation law of human-human collaborative interaction force, the interaction force is divided into three force intervals, namely, low interaction force interval [0, 35%), medium interaction force interval [35%, 65%) and high interaction force interval [65%, 100%), where the percentage represents the ratio of interaction force to peak value; based on the ergonomic characteristics of human arms, the motion space of human-human collaborative movement is divided into three position areas, namely, human-human collaborative areas A, B and C, where area A includes starting points S1 and S2 and target points E1, E2, E3 and E4, area B includes starting point S3 and target points E5 and E6, and area C includes starting points S4 and S5 and target points E7, E8, E9 and E10, as shown in Figure 2.Figure 2 The human-human collaboration motion process is divided into four continuous motion stages according to the human-human collaboration motion speed and acceleration law: a starting stage, characterized by starting at a low speed and increasing acceleration; a middle stage 1, characterized by continuously increasing speed to a peak value; a middle stage 2, characterized by stably maintaining the peak value; and an ending stage, characterized by decreasing speed and motion termination.

[0041] (2) Based on the human-human collaboration experimental data obtained in step 1, the element types (such as ΦPOF, ΦPF, POF, Φ, PF, ΦPO, and PO) and frequencies in the three force intervals, three position regions, and four motion stages are counted respectively. The frequency is the ratio of the number of occurrences of a certain element type to the total number of elements.

[0042] (3) Variance analysis is used to verify the significance of the influence of interaction force, spatial position, and motion stage on the selection of element types. The variance calculation formula is: (2) and are the between-group and within-group means, respectively, and are the between-group and within-group degrees of freedom, respectively, and are the between-group and within-group sum of squares, respectively; each force interval or position region or motion stage is a group, and there are a total of 10 groups.

[0043] (4) According to the F value obtained from the variance calculation formula and the corresponding degrees of freedom, consult the F distribution table to find the critical F value in the table. If the calculated F value is greater than the critical F value in the table, then P<0.05. When P<0.05, it indicates that the interaction force, spatial position, and motion stage have a significant influence on the selection of element types. Otherwise, it is not significant. The parameters that have a significant influence are determined as the main influencing factors, which refer to the interaction force, spatial position, and motion stage. The result determined here is that the interaction force is the main influencing factor / the spatial position is the main influencing factor / the motion stage is the main influencing factor.

[0044] Step 3, a double DQN element decision method based on long short-term memory network (LSTM) is used to select the element types of human-human collaboration.

[0045] Based on the action element frequency, the reward function of the double DQN model is designed, and the loss function of the double DQN is constructed as a function related to the reward function.

[0046] The double DQN element decision method framework based on long short-term memory network LSTM is as shown in Figure 4As shown, it includes three parts, which are: (3-1) double network structure, containing real-time updated estimation network Q and periodically synchronized target network Q', wherein the Q and Q' networks are designed based on LSTM; (3-2) loss function based on reward function, used to guide the optimization process of network parameters; (3-3) experience replay pool, used to store the state of the agent at a certain time, the action performed under the current state, and the feedback reward given after the action is performed.

[0047] In (3-1), the LSTM network structure is as shown in Figure 5 The learnable weights of the LSTM network are input weights W [ W i , W f , W g , W o ] T , recurrent weights R w [ R i , R f , R g , R o ] T and bias b [ b i , b f , b g , b o ] T , wherein f , g , i and o represent the forget gate, the candidate cell, the input gate and the output gate, respectively; the forget gate f is used to forget the information that has little influence on the selection of the primitive type, so as to reduce the influence of excessive memory information on the LSTM network. The motion state information that has great influence on the selection of the primitive type is added to the cell state σ g (sigmoid) by using the activation function c , that is:

[0048] f t = σ g ( W f St + R f a t-1 + b f ) (3) f t for t Always forget the door, S t is the motion status information, a t-1 for t -1 The action primitive type selected at the moment, W f 、 R f and b f is the weight coefficient of the forget gate; σ g The output is a value between [0, 1]. The closer the output value is to 1, the more influential the corresponding information is on the selection of primitive type and will be retained. Otherwise, it will be forgotten. The model will automatically adjust the weight and bias of the forget gate according to the characteristics of the data to determine which information needs to be forgotten. The motion state information with greater influence selected here determines the specific data information: the size of the interaction force at the current moment, position data, etc.

[0049] Candidate Unit g and input gate i The effective information of the current motion state information and the primitive type information of the previous moment are extracted together, and the activation function is used σ c (tanh) and σ g (sigmoid) added to the cell state c Middle; the motion state information S that has a higher degree of influence on the selection of primitive type t The easier it is to be memorized into the unit state, that is: g t = σ c ( W g S t + R g a t-1 + b g ) (4) i t = σ g( W i S t + R i a t-1 + b i ) (5) g t for t the candidate gate at time t, i t for t the input gate at time t, W g , R g and b g are the candidate gate weight coefficients, W i , R i and b i are the input gate weight coefficients, σ g ( x )=1+ e -x ) -1 , .

[0050] the output gate o is used to integrate the motion state at the current time and the primitive type at the last time, and the information therein is extracted by using an activation function σ g (sigmoid), i.e.: o t = σ g ( W o S t + R o a t-1 + b o ) (6) o t for t the output gate at time t, W o , R o and b o are the output gate weight coefficients.σ g ( x )=(1+ e -x ) -1 。

[0051] The cell state c encompasses the "summary memory" of all input motion state information of the LSTM network at previous time steps, which integrates t the cell state information at time -1, t the forget information at time -1, t the input information at time -1, and t the information that needs to be remembered at time -1 as the cell state information at time t, that is: t c t = f t * c t-1 + i t * o t * x t (7) (7) wherein, denotes the Hadamard product, the corresponding elements of the vectors are multiplied.

[0052] t The cell state information at time t is processed by the activation function, and then combined with the output gate information at time t to obtain the output information at time t (i.e., the basic element type) t o t = σ (W o * x t + U o * h t-1 + b o ) (8) t a t ; 。

[0053] (8) To optimize the double DQN network parameters, a special training data set is constructed, and the input feature vector of the training data set contains the position information of the human arm / robot end motion X =[ x , y , z ], motion speed information , acceleration information , interaction force information , self-rotation angle Φ value and collaborator heart rate change value r mc , the training data input set S t The mathematical expression can be expressed as: (9) wherein, N i is the total number of samples, i is the sample index; N t is the total number of frames of each sample, t is the frame number index of each sample, r mc = r / ​r m , r For the collaborator's heart rate, r m Maximum heart rate for human-human collaboration / human-machine collaboration.

[0054] The output layer of the dual DQN network will be mapped to one of the eight primitive action types, and its output set a t It can be expressed as: (10) To facilitate model training, the eight action primitives are mapped to discrete integer labels: 1-ΦPOF, 2-ΦPF, 3-POF, 4-Φ, 5-F, 6-PF, 7-ΦPO, and 8-PO. That is, the action primitive types are represented by numbers 1 to 8 for model training.

[0055] In (3-2), the loss function L The definition is as follows: (11) in, S t and S t+1 Robots t Moment and t +1 moment status; a t and a t+1 They are t Moment and t +1 The action primitive type selected at the moment; Q ( S t , a t )and Q ´( S t+1 , a t+1 ) are respectively t Time Estimation Network Q Value and t +1 moment target network Q value; Iterates through all possible a t+1 , choose so that Q ´( S t+1 , a t+1 )The largest action primitive type; is the discount rate / hyperparameter, which converts the rewards of many future steps to the current point; is the learning rate; R is the reward function.

[0056] The relationship between the reward function R and the frequency of each primitive is defined by analyzing the frequency of each primitive in human-human collaboration as follows: (12) where, p is the frequency of each primitive type; a 1 and b 1 are set coefficients, and a1=10 and b1=0.4 are taken according to the statistical distribution of the frequency of each primitive type.

[0057] Step 4: The action primitive type selection method in step 3 is integrated into the human-robot motion planning to establish a human-robot motion autonomous planning method including a compliance control module, an action primitive decision module, a primitive parameter estimation module, a human-robot mapping module, and a joint design module, as shown in Figure 6 .

[0058] The compliance control module is used to drive the robot to conform to the motion of the human leader, and the adopted compliance control model is: (13) M ∈ R 6×6 , C ∈ R 6×6 and K ∈ R 6×6 are the virtual mass matrix, the virtual damping matrix, and the virtual stiffness matrix, respectively; is the interaction force vector in Cartesian space, f x , f y , f z represent the interaction force values in the x, y, and z axes, respectively, represent the torque values in the x, y, and z axes, respectively; X d =[ x d , y d , z d , r xd , r yd , r zd ] T is the desired pose vector, x d ,y d , z d are the expected positions of the xyz axes, r xd , r yd , r zd are the desired attitude angles of the xyz axes respectively; X r =[ x r , y r , z r , r xr , r yr , r zr ] T is the actual pose vector, x r , y r , z r are the actual positions of the xyz axes, r xr , r yr , r zr are the actual attitude angles of xyz axes respectively; is the expected speed, is the actual speed; is the expected acceleration, is the actual acceleration.

[0059] To facilitate application in robot control, the admittance control model can be converted into a discrete form to solve the desired trajectory of the robot: (14) (15) (16) Where, X d ( t ), and They are t The expected position, expected velocity, and expected acceleration at the moment; X r ( t ), and They are tThe actual position, actual speed and actual acceleration at the moment; for the convenience of formula, let , ; T be the sampling period.

[0060] The action primitive decision module determines the type of action primitive that the robot should perform at the next moment in real time according to the current motion state of the robot; the action primitive decision module receives the real-time motion state data of the robot in human-robot collaboration S t (such as position, speed, interaction force), filters and updates the information through the forget gate, input gate and output gate of the long short-term memory network (LSTM) to determine the optimal action primitive type that the robot should perform at the next moment; according to the selected action primitive type, the motion of the robot is planned, and after the motion is executed, the new reward is fed back to the LSTM, and the motion state of the robot is taken as the input of the next cycle to form a closed-loop control system to optimize the fluency and efficiency of human-robot collaboration, as shown in Figure 7 .

[0061] The primitive parameter estimation module is used to convert the expected motion pose of the robot from the admittance control module X d [ x d , y d , z d , r xd , r yd , r zd ] T and the action primitive type of the action primitive decision module into the expected value of the basic action primitive Φ at the next moment, and the conversion formula is as follows: (17) is the expected value of the basic action primitive Φ at the next moment, when the primitive type contains the basic action primitive Φ (such as the primitive type ΦPOF), the expected value of the basic action primitive Φ at the next moment is calculated according to the above formula, otherwise, the existing value is maintained.

[0062] The human-robot mapping module is used to map the data of each basic action primitive of the human arm to the data of each basic action primitive of the robot, and to make corresponding adjustments to the action according to the configuration difference (the adjustment is to modify the model parameters W, R w and b after designing a new reward function according to the heart rate of the collaborator in step 6).

[0063] The joint design module is used to determine the values of the joint angles of the robot from the determined values of each basic action primitive.

[0064] Step 5, human-robot collaboration experiment based on human-like motion autonomous planning method is carried out, and the human-robot collaboration motion experiment setting and experiment process are similar to the human-human collaboration experiment, as shown in Figure 8 . The human-robot collaboration experiment data containing real-time feedback of the collaborator are obtained.

[0065] In the human-robot collaboration motion experiment process, the human-like motion autonomous planning method is used to plan the robot motion; in order to obtain the feedback data of the collaborator participating in the human-robot collaboration, i.e. the heart rate of the human collaborator, the human collaborator always wears a heart rate monitoring bracelet during the experiment, so as to adjust the robot motion through the heart rate state subsequently.

[0066] In this embodiment, 3 starting points (blue squares) and 6 target points (red circles) are set, as shown in Figure 8 (b) of the figure. During the experiment, the participant (leader) guides the robot (follower) from the starting point position to the vicinity of the target point position along the expected path through the interactive force. The right hand of the participant is connected with the robot through the handle of the force sensor. The force sensor collects the interactive force between the human and the robot, and transmits the collected force data to the robot. The robot plans its own motion in real time according to the direction and size of the interactive force, and adopts the human-like motion autonomous planning method in the motion planning process. The left wrist of the participant wears a heart rate monitoring bracelet, and the bracelet sends the heart rate data of the collaborator through Bluetooth. These heart rate data are used as an important basis for evaluating and optimizing the behavior of the robot.

[0067] Step 6, the reward function of the double DQN model is designed based on the feedback data of the collaborator, and the parameters of the double DQN primitive type decision model based on LSTM are corrected.

[0068] The reward function of the double DQN model designed based on the heart rate data of the collaborator is: (18) Wherein, r is the heart rate of the collaborator, a 2 and b 2 are set coefficients, a 2 and b 2 are set coefficients,

[0069] The human-robot collaboration motion data collected in step 5 are used, and the reward function is designed in combination with the new reward function. The parameters (W, R w and b) of the double DQN primitive type decision model based on LSTM in step 3 are corrected, so that the robot motion is human-like and the collaborator can accept the behavior of the robot.

[0070] Step 7, based on the primitive decision correction model in step 6, carry out human-robot collaboration experimental verification to verify the degree of robot human-like motion and the acceptance of robot motion in human-robot collaboration.

[0071] Use the human-like motion autonomous planning method based on the primitive decision correction model to plan the robot motion, and the experimental scene and setting are as shown in Figure 8 From the subjective and objective aspects, evaluate the degree of robot human-like motion; use the method of measuring the heart rate of the collaborator to objectively evaluate the acceptance of the robot motion by the collaborator; select the heart rate of the participant when participating in the human-human collaboration experiment as the benchmark, the greater the difference between the heart rate of the participant when participating in the human-robot collaboration experiment and the benchmark, the more the motion of the robot in human-robot collaboration can cause emotional fluctuations of the participant. This means that there is a big difference in the participant's state of mind when facing human-robot collaboration and when participating in human-human collaboration, and the acceptance of the robot motion by the collaborator is smaller.

[0072] Use the method of questionnaire survey to subjectively evaluate the acceptance of the robot motion by the collaborator; the questionnaire survey includes the emotional state questionnaire and the robot motion acceptance questionnaire. The emotional state questionnaire mainly evaluates the emotional state of the participant when participating in the human-robot collaboration motion. The better the emotional state of the participant, the higher the acceptance of the robot motion. Select the state-trait anxiety inventory (STAI) in the emotional detection aspect which is more recognized and simple and easy to implement. Select the questionnaire score of the participant after participating in the human-human collaboration experiment as the benchmark, the greater the difference between the questionnaire score of the participant after participating in the human-robot collaboration experiment and the benchmark, the more the motion of the robot in human-robot collaboration can cause emotional fluctuations of the participant, and the acceptance of the robot motion by the collaborator is smaller.

[0073] The two methods of measuring the heart rate of the participant and the emotional state questionnaire survey are indirect evaluation of the acceptance of the robot motion based on the two motion planning methods by the participant, while the subjective acceptance measurement questionnaire is direct evaluation of the acceptance of the robot motion by the participant. In human-robot collaboration, the higher the degree of human-like motion of the robot, the easier the collaborator understands the motion intention of the robot, so as to trust the robot more and be more willing to collaborate with it. The enhancement of such trust and collaboration willingness promotes the fluency of human-robot collaboration, makes the collaborator feel that the robot has a positive collaboration attitude, and finally accepts the motion of the robot. Therefore, from the fluency of collaboration motion, the trust of the robot and the positivity of the robot, design the questionnaire survey to evaluate the acceptance of the robot motion by the collaborator. The questionnaire survey design is as shown in Figure 9 .

[0074] Performance evaluation: the experimental results are evaluated in terms of performance, including the amount of heart rate change, the amount of change in the state anxiety questionnaire score of the state-trait anxiety inventory (STAI), and the robot movement acceptance questionnaire. The experimental results show that the method can effectively realize autonomous planning of human-like motion, and significantly improve the acceptance of the robot motion by the collaborator.

[0075] Embodiment 2 A computer readable storage medium having stored thereon a program, the program being executed by a processor to implement the steps in the human-robot collaborative autonomous planning method of human-like motion based on action primitives and double DQN as described in Embodiment 1.

[0076] Embodiment 3 An electronic device comprising a memory, a processor, and a program stored on the memory and executable on the processor, the processor implementing the steps in the human-robot collaborative autonomous planning method of human-like motion based on action primitives and double DQN as described in Embodiment 1 when executing the program.

Claims

1. A method for autonomous humanoid motion planning in human-machine collaboration based on action primitives and dual DQN, characterized by: Here are the steps: Step 1: Obtain human-human collaborative experiment data and preprocess it; Step 2: extract the primitive types of human-arm movements in human-human collaboration and analyze the main influencing factors affecting the selection of primitive types; Step 3: Use the dual DQN primitive decision method based on the long short-term memory network (LSTM) to select the primitive type for human-human collaboration; Based on the frequency of action primitives, the reward function of the dual DQN model is designed, and the loss function of the dual DQN is constructed as a function related to the reward function; Step 4: Integrate the action primitive type selection method in step 3 into humanoid motion planning, and establish a humanoid motion autonomous planning method including an admittance control module, an action primitive decision module, a primitive parameter estimation module, a human-machine mapping module, and a joint design module; Step 5: Conduct a human-machine collaboration experiment based on the humanoid motion autonomous planning method to obtain human-machine collaboration experiment data including real-time feedback from collaborators; Step 6: Design the reward function of the dual DQN model based on the collaborator feedback data and modify the parameters of the LSTM-based dual DQN primitive type decision model; Step 7: Conduct a human-machine collaboration experiment based on the primitive decision correction model in step 6 to verify the degree of human-like movement of the robot and the acceptance of the robot's movement in human-machine collaboration.

2. The method for autonomous human-machine collaborative humanoid motion planning based on action primitives and dual DQN according to claim 1 is characterized in that: In step 1, the human-human collaboration experiment refers to designing a standardized experimental scenario, requiring two human collaborators to start from the same starting position, collaborate by holding a force sensor together, and eventually reach the same target point together, with one of the two being the leader and the other being the follower; Collect the natural motion characteristics of humans during contact collaboration, including: changes in joint angles of the elbow, shoulder, and wrist, end-of-motion position, velocity, acceleration, and contact force; Preferably, in step 1, the acquired experimental data is preprocessed, and the preprocessing includes removing noise and eliminating invalid motion data; The noise removal method is to replace the current value of the time series data with the mean value in the sliding window to suppress high-frequency noise. The formula used is: （1） k is the window size, further, k=5, x t+i is the original data, y t is the data after noise removal; The method of eliminating invalid motion data is to remove the data segments where the speed of the arm end is less than 0.001m / s except at the beginning and end of the motion; The preprocessed collaboration data is used as the basic data for the analysis of the influencing factors of primitive types.

3. The method for autonomous human-machine collaborative humanoid motion planning based on action primitives and dual DQN according to claim 1, characterized in that: In step 2, the human arm action primitive is defined. The human arm action primitive is a way to express the human arm movement. By analyzing the structure and movement characteristics of the human arm, it is proposed to use the arm end position P and posture O , rotation angle Φ and interaction force F To express the movement of the human arm, and define these four basic movement elements as basic action primitives; among them, P =[ x , y , z ]and O =[ r x , r y , r z ] represent the three Euler positions and the corresponding direction angles, It represents the interaction force between the human hand or robot and the external environment. The rotation angle Φ is defined as the rotation angle of the arm elbow from position E to position E1 around the dotted axis SW. The dotted axis SW is the line connecting the hand and the shoulder. Similarly, the movement of the robot arm with a humanoid arm is also expressed using basic action primitives. Extraction of human arm action primitive types: The process of extracting human arm action primitive types in human-human collaboration is as follows: 1) Extract the original motion data of the follower's right arm from the BVH format data exported by the human motion capture system; 2) Obtain the angle change sequence of each joint in the collaborative task; 3) Preprocess the obtained joint angle data, including removing noise and eliminating invalid motion data; 4) Obtain basic action primitives, including: the change sequence of the rotation angle Φ, the arm end position P, and the posture O; 5) Determine the action primitive type of each frame of motion based on the change of the basic action primitive values, including ΦPOF, ΦPF, POF, Φ,F,PF, ΦPO, PO; The analysis process of the main influencing factors for the selection of primitive types is as follows: (1) the interaction force is divided into three force intervals, namely the low interaction force interval [0, 35%), the medium interaction force interval [35%, 65%) and the high interaction force interval [65%, 100%), where the percentage represents the ratio of the interaction force to the peak value; the movement space of human-human collaborative movement is divided into three position areas, namely human-human collaborative areas A, B and C, where area A includes the starting points S1 and S2 and the target points E1, E2, E3 and E4, area B includes the starting point S3 and the target points E5 and E6, and area C includes the starting points S4 and S5 and the target points E7, E8, E9 and E10; the collaborative movement process is decomposed into four consecutive movement stages: the starting stage, characterized by the start and acceleration increase; the intermediate stage 1, characterized by the continuous increase in speed to the peak value; the intermediate stage 2, characterized by the stable maintenance of speed at the peak level; the ending stage, marked by the decrease in speed and the termination of movement; (2) Based on the human-human collaboration experimental data obtained in step 1, the primitive types and frequencies of the three force intervals, three position areas, and four motion stages are counted respectively. The frequency is the ratio of the number of occurrences of a certain primitive type to the total number of primitives. (3) ANOVA was used to verify the significance of the effects of interaction force, spatial position, and movement phase on the selection of primitive types. The variance calculation formula is: （2） and are the between-group and within-group mean squares, and are the between-group and within-group degrees of freedom, respectively. and The sums of squares between and within groups are respectively; each force interval or position area or movement stage is a group, with a total of 10 groups; (4) According to the F value and the corresponding degrees of freedom obtained by the variance calculation formula, consult the F distribution table and find the corresponding critical F value in the table. If the calculated F value is greater than the critical F value in the table, then P < 0.

05. When P < 0.05, it indicates that the interaction force, spatial position and movement stage have a significant impact on the selection of primitive type. Otherwise, it is not significant. The parameters with significant impact are determined as the main influencing factors. The parameters refer to the interaction force, spatial position and movement stage.

4. The method for autonomous human-machine collaborative humanoid motion planning based on action primitives and dual DQN according to claim 1, characterized in that: In step 3, the dual DQN primitive decision method based on the long short-term memory network (LSTM) includes: (3-1) a dual network structure, including a real-time updated estimation network Q and a periodically synchronized target network Q', wherein the Q and Q' networks are designed based on LSTM; (3-2) a loss function based on the reward function, which is used to guide the optimization process of the network parameters; (3-3) an experience replay pool, which is used to store the state of the agent at a certain moment, the actions performed in the current state, and the feedback rewards given after the actions are performed; Further preferably, in (3-1), the learnable weight of the LSTM network is the input weight W =[ W i , W f , W g , W o ] T , cycle weight R w =[ R i , R f , R g , R o ] T and bias b =[ b i , b f , b g , b o ] T ,in, f 、 g 、 i and o Respectively represent the forget gate, candidate unit, input gate and output gate; forget gate f Used to forget the information that has little influence on the selection of primitive type, using the activation function σ g (sigmoid) Add motion state information that has a greater impact on primitive type selection to the cell state c In Chinese, that is: f t = σ g ( W f S t + R f a t-1 + b f ) （3） f t for t Always forget the door, S t is the motion status information, a t-1 for t -1 The action primitive type selected at the moment, W f 、 R f and b f is the weight coefficient of the forget gate; σ g The output is a value between [0, 1]. The closer the output value is to 1, the more the corresponding information has an impact on the selection of primitive type and will be retained. Otherwise, it will be forgotten. The model will automatically adjust the weight and bias of the forget gate based on the characteristics of the data to determine which information needs to be forgotten. Candidate Unit g and input gate i The current motion state information and the primitive type information of the previous moment are extracted together, and the activation function is used σ c and σ g Add to cell state c Middle; the motion state information S that has a higher degree of influence on the selection of primitive type t The easier it is to be memorized into the unit state, that is: g t = σ c ( W g S t + R g a t-1 + b g ) （4） i t = σ g ( W i S t + R i a t-1 + b i ) （5） g t for t Time to choose the door, i t for t Enter the door at all times, W g 、 R g and b g is the candidate gate weight coefficient, W i 、 R i and b i is the input gate weight coefficient, σ g ( x )=(1+ e -x ) -1 , ; Output Gate o Used to integrate the current motion state and the primitive type of the previous moment, using the activation function σ g (sigmoid) extracts the information, namely: o t = σ g ( W o S t + R o a t-1 + b o ) （6） o t for t Output gate at all times, W o 、 R o and b o is the output gate weight coefficient, where σ g ( x )=(1+ e -x ) -1 ; The cell state c covers the summary memory of all input motion state information at the historical moment of the LSTM network. t -1 moment unit status information, t Always forget information, t Always input information and t The information that needs to be remembered at all times is integrated as t The unit status information at the moment, namely: （7） in, represents the Hadamard product, which multiplies corresponding elements of the vectors; t The unit state information is processed by the activation function and then combined with t The output gate information at each moment is combined to obtain t Output information at all times a t , that is, primitive types; ; （8） Construct a training dataset whose input feature vector contains the position information of the human arm / robot end motion X =[ x , y , z ], motion speed information , acceleration information , interactive information , rotation angle Φ value and collaborator's heart rate change measurement value r mc , training data input set S t The mathematical expression can be expressed as: （9） in, N i is the total number of samples, i is the sample index; N t The total number of frames for each sample, t For each sample frame index, r mc = r / r m , r For the collaborator's heart rate, r m Maximum heart rate for human-human collaboration / human-machine collaboration; The output layer of the dual DQN network will be mapped to one of the eight primitive action types, and its output set a t Expressed as: （10） Map 8 types of action primitives to discrete integer labels: 1-ΦPOF, 2-ΦPF, 3-POF, 4-Φ, 5-F, 6-PF, 7-ΦPO, 8-PO; Further preferably, in (3-2), the loss function L The definition is as follows: （11） in, S t and S t+1 Robots t Moment and t +1 moment status; a t and a t+1 They are t Moment and t +1 The action primitive type selected at the moment; Q ( S t , a t )and Q ´( S t+1 , a t+1 ) are respectively t Time Estimation Network Q Value and t +1 moment target network Q value; Iterates through all possible a t+1 , choose so that Q ´( S t+1 , a t+1 )The largest action primitive type; is the discount rate / hyperparameter, which converts the rewards of many future steps to the current point; is the learning rate; R is the reward function; The reward function is defined as: （12） in, p is the frequency of each primitive type; a 1 and b 1 is the setting coefficient, and more preferably, a1=10, b1=0.

4.

5. The method for autonomous human-machine collaborative humanoid motion planning based on action primitives and dual DQN according to claim 1, characterized in that: In step 4, the humanoid motion autonomous planning method includes an admittance control module, an action primitive decision module, a primitive parameter estimation module, a human-machine mapping module and a joint design module; The admittance control module is used to drive the robot to follow the movement of the human leader. The adopted admittance control model is: （13） M ∈ R 6×6 、 C ∈ R 6×6 and K ∈ R 6×6 are the virtual mass matrix, virtual damping matrix and virtual stiffness matrix respectively; is the Cartesian space interaction force vector, f x 、 f y 、 f z Respectively represent the xyz axial interaction force values, Respectively represent the xyz axial torque values; X d =[ x d , y d , z d , r xd , r yd , r zd ] T is the desired pose vector, x d , y d , z d are the expected positions of the xyz axes, r xd , r yd , r zd are the desired attitude angles of the xyz axes respectively; X r =[ x r , y r , z r , r xr , r yr , r zr ] T is the actual pose vector, x r , y r , z r are the actual positions of the xyz axes, r xr , r yr , r zr are the actual attitude angles of xyz axes respectively; is the expected speed, is the actual speed; is the expected acceleration, is the actual acceleration; The admittance control model is converted into discrete form: （14） （15） （16） Where, X d ( t ), and They are t The expected position, expected velocity, and expected acceleration at the moment; X r ( t ), and They are t Actual position, actual velocity and actual acceleration at the moment; make , ; T is the sampling period; The action primitive decision module determines the action primitive type to be executed at the next moment in real time according to the current motion state of the robot; the action primitive decision module receives the real-time motion state data of the robot in human-machine collaboration. S t , the information is filtered and updated through the forget gate, input gate, and output gate of the LSTM to determine the optimal action primitive type that the robot should execute at the next moment; the robot's movement is planned based on the selected action primitive type, and after the movement is executed, the new reward is fed back to the LSTM. The robot's movement state is used as the input for the next cycle, forming a closed-loop control system; The primitive parameter estimation module is used to convert the robot's desired motion posture from the admittance control module into X d =[ x d , y d , z d , r xd , r yd , r zd ] T The action primitive type of the action primitive decision module is converted into the expected value of the basic action primitive Φ at the next moment. The conversion formula is as follows: （17） is the expected value of the basic action primitive Φ at the next moment. When the primitive type contains the basic action primitive Φ, the expected value of the basic action primitive Φ at the next moment is calculated according to the above formula; otherwise, the current value is maintained. The human-machine mapping module is used to map the basic action primitive data of the human arm to the basic action primitive data of the robot; The joint design module is used to determine the values ​​of the robot's joint angles based on the determined values ​​of the basic action primitives.

6. The method for autonomous humanoid motion planning based on action primitives and dual DQN for human-machine collaboration according to claim 1, characterized in that: In step 5, during the human-robot collaborative motion experiment, the robot motion is planned using a humanoid motion autonomous planning method; and feedback data of the collaborator participating in the human-robot collaboration, namely the collaborator's heart rate, is obtained.

7. The method for autonomous humanoid motion planning based on action primitives and dual DQN for human-machine collaboration according to claim 1, characterized in that: In step 6, the reward function of the dual DQN model designed based on the heart rate data fed back by the collaborator is: （18） in, r For the collaborator's heart rate, a 2 and b 2 is the setting coefficient, and further preferably, a 2 and b 2 According to the characteristics of the collaborator's heart rate changes during human-machine collaboration, a2=0.15, b2=130; Using the human-machine collaborative motion data collected in step 5 and combining it with the newly designed reward function, the parameters W, R of the LSTM-based dual DQN primitive type decision model in step 3 are adjusted. w and b are corrected.

8. The method for autonomous humanoid motion planning based on action primitives and dual DQN for human-machine collaboration according to claim 1, characterized in that: In step 7, the robot motion is planned using the humanoid motion autonomous planning method based on the modified primitive decision model; and the degree of humanoid motion of the robot is evaluated.

9. A computer-readable storage medium, characterized in that A program is stored thereon, which, when executed by a processor, implements the steps in the human-machine collaborative humanoid motion autonomous planning method based on action primitives and dual DQN as described in any one of claims 1 to 8.

10. An electronic device, characterized in that: The invention comprises a memory, a processor, and a program stored in the memory and executable on the processor, wherein when the processor executes the program, the steps in the method for autonomous planning of human-machine collaborative humanoid motion based on action primitives and dual DQN are implemented as described in any one of claims 1 to 8.

Images