Multi-axis hole assembly method and system based on hierarchical reinforcement learning and distributed learning

Abstract

The application provides a multi-axis hole assembly method and system based on hierarchical reinforcement learning and distributed learning, and comprises the following steps: establishing a main control assembly strategy model based on deep reinforcement learning by using mechanical arm state data and action data; constructing a plurality of sub-process networks based on different assembly interaction environments; updating and training the main control assembly strategy model by using the interaction data obtained by the plurality of constructed sub-process networks, to obtain a trained main control assembly strategy model; and executing a multi-axis hole assembly task of the mechanical arm by using the trained main control assembly strategy model. Compared with the ordinary reinforcement learning algorithm, the method of updating the overall network by using a plurality of different environment sub-process networks can improve the final effect of robot learning, improve the efficiency of robot learning, and save learning time.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of robot assembly, and particularly relates to a multi-axis hole assembly method and system based on hierarchical reinforcement learning and distributed learning. BACKGROUND

[0002] The statements in this section merely provide background information related to the present application and do not necessarily constitute the prior art.

[0003] The learning efficiency of assembly tasks and how to deal with complex assembly objects are one of the problems to be solved for robots to improve their complex assembly skills. In the process of multi-axis hole assembly and complex electrical connector assembly, due to the complexity of the assembly object and the difficulty of obtaining interactive data, the robot often needs to spend a long time in the learning process, and the reward function in the interactive process is difficult to shape, which brings difficulties to the learning process of the robot. Therefore, how to make the robot learn the assembly skills of complex multi-axis holes more efficiently, reduce the learning time of the robot, and be able to deal with the assembly of complex multi-axis holes and other objects is a problem to be solved at present. SUMMARY

[0004] In order to overcome the shortcomings of the prior art, the present application provides a multi-axis hole assembly method and system based on hierarchical reinforcement learning and distributed learning, which uses the method of updating the overall network by constructing multiple sub-process networks in different environments. Compared with ordinary reinforcement learning algorithm, it can improve the final effect of robot learning and improve the efficiency of robot learning and save learning time.

[0005] To achieve the above purpose, the first aspect of the present application provides a multi-axis hole assembly method based on hierarchical reinforcement learning and distributed learning, comprising:

[0006] establishing a main control assembly strategy model based on deep reinforcement learning, the model input is the state of the robot arm, and the output is the action of the robot arm;

[0007] constructing multiple sub-process networks based on different assembly interactive environments, updating and training the main control assembly strategy model using the robot arm interactive data obtained from the multiple sub-process networks, and obtaining the trained main control assembly strategy model;

[0008] wherein the sub-process network comprises a high-level policy network and a low-level policy network, the high-level policy network obtains a high-level policy value according to the robot arm state data at the current time, and the low-level policy network obtains the action of the robot arm at the next time according to the high-level policy value and the robot arm state data at the current time;

[0009] using the trained main control assembly strategy model to perform the multi-axis hole assembly task of the robot arm.

[0010] The second aspect of the present application provides a multi-axis hole assembly system based on hierarchical reinforcement learning and distributed learning, comprising:

[0011] A total strategy model module is established: a master control assembly strategy model based on deep reinforcement learning is established using robot state data and action data;

[0012] A total strategy model training module: a plurality of sub-process networks based on different assembly interaction environments are constructed, and the robot interaction data obtained by the plurality of sub-process networks are used to update and train the master control assembly strategy model to obtain a trained master control assembly strategy model;

[0013] The sub-process network comprises a high-level strategy network and a bottom-level strategy network, the high-level strategy network obtains a high-level strategy value according to the robot state data at the current time, and the bottom-level strategy network obtains the action of the robot at the next time according to the high-level strategy value and the robot state data at the current time;

[0014] An execution control module: the trained master control assembly strategy model is used to execute the multi-axis hole assembly task of the robot.

[0015] The third aspect of the present application provides a computer device, comprising a processor, a memory and a bus, the memory stores machine readable instructions executable by the processor, when the computer device is running, the processor and the memory communicate through the bus, and the machine readable instructions are executed by the processor to execute the multi-axis hole assembly method based on hierarchical reinforcement learning and distributed learning.

[0016] The fourth aspect of the present application provides a computer readable storage medium, the computer readable storage medium stores a computer program, and the computer program is executed by the processor to execute the multi-axis hole assembly method based on hierarchical reinforcement learning and distributed learning.

[0017] The above one or more technical solutions have the following beneficial effects:

[0018] The present application uses the method of updating the overall network by constructing a plurality of sub-process networks in different environments, which can improve the final effect of robot learning and improve the efficiency of robot learning compared with ordinary reinforcement learning algorithm, and save learning time.

[0019] The sub-process network of the present application comprises a high-level strategy network and a bottom-level strategy network, the high-level strategy network and the bottom-level strategy network in each sub-process are trained to speed up the learning of the network, the master control assembly strategy network is updated by the sub-process network, which can reduce the learning time of the robot and can cope with the assembly of complex multi-axis holes and other objects.

[0020] Advantages of the present application's additional aspects will become apparent in the description that follows, in part, from the description, or become apparent from practice of the application. BRIEF DESCRIPTION OF DRAWINGS

[0021] The accompanying drawings, which form a part of this specification, are included to provide a further understanding of the application, and are incorporated by reference herein. The embodiments depicted herein are provided by way of example only, and together with the specification serve to explain the application.

[0022] Figure 1 A model learning flowchart based on hierarchical reinforcement learning and distributed learning in embodiment one of the application;

[0023] Figure 2 A hierarchical reinforcement learning flowchart in embodiment one of the application;

[0024] Figure 3 A reward construction flowchart in embodiment one of the application;

[0025] Figure 4 A bottom strategy network update flowchart in embodiment one of the application;

[0026] Figure 5 A high-level strategy network update flowchart in embodiment one of the application. DETAILED DESCRIPTION

[0027] It should be noted that the following detailed description is merely exemplary and is intended to provide further description of the application. Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs.

[0028] It should be noted that the terms used herein are merely for the purpose of describing specific embodiments and are not intended to limit the exemplary embodiments according to the present application.

[0029] In the case of no conflict, the embodiments in the application and the features in the embodiments can be combined with each other.

[0030] Embodiment one

[0031] As shown in the following, the present embodiment discloses a multi-axis hole assembly method based on hierarchical reinforcement learning and distributed learning, comprising: Figures 1-2 Step 1: Establish a main control assembly strategy model based on deep reinforcement learning using mechanical arm state data and action data;

[0032]

[0033] ​Step 2: Construct a plurality of sub-process networks based on different assembly interaction environments, update and train the main control assembly strategy model using the mechanical arm interaction data obtained from the plurality of constructed sub-process networks, and obtain a trained main control assembly strategy model;

[0034] The sub-process network includes a high-level strategy network and a bottom-level strategy network. The high-level strategy network obtains a high-level strategy value according to the mechanical arm state data at the current moment. The bottom-level strategy network obtains the action of the mechanical arm at the next moment according to the high-level strategy value and the mechanical arm state data at the current moment.

[0035] Step 3: Use the trained main control assembly strategy model to perform the multi-axis hole assembly task of the mechanical arm.

[0036] In this embodiment, a system including a mechanical arm, an end six-dimensional force sensor, two industrial cameras, and an assembly object is built. The system forms a state space of the network through the position information, force information, and image information of the mechanical arm end in multiple environments. A shared feature space is constructed through feature extraction of the state, and an experience database is established. The network learning is accelerated by training the high-level strategy and the bottom-level strategy in each process. The reward function of the bottom-level strategy is shaped by a human-in-the-loop. Then, the experience of each process is transmitted to the main process and the main network is updated. The main network assigns the updated network weights to each sub-network. The network output is the action of the mechanical arm at the next moment.

[0037] Specifically, in step 1 of this embodiment, the network input state is defined as where s p =[x, y, z, a, b, g] represents the pose of the mechanical arm end component; s τ =[F x ,F y ,F z ,M x ,M y ,M z ] represents the contact force / torque of the mechanical arm end, represents the image data obtained by the camera, and a t =[Dx, Dy, Dz, Da, Db, Dg] represents the next assembly action of the mechanical arm.

[0038] The network structure of the main control assembly strategy model is consistent with that of the sub-process network. The main control assembly strategy model does not participate in environment interaction, but only uses the data transmitted by the sub-process network to update its own network structure.

[0039] In step 2 of this embodiment, n sub-process networks based on different assembly interaction environments (i.e., different assembly objects) are constructed. Each sub-process network includes a high-level strategy network and a bottom-level strategy network.

[0040] Specifically, the high-level policy network adopts DQN, i.e., deep Q network, and contains an Option-value network, i.e., an option-value network, and the input of the high-level policy network is the state S of the robot arm t , and the output is the high-level policy value o t .

[0041] As shown in Figure 4 , the bottom-level policy network adopts a SAC model and contains two pairs of Actor networks and Critic networks. In the evaluation network, the input of the Actor network is the state, which includes the state of the end of the robot arm and the output of the high-level policy, and the output is the corresponding action. The input of the Critic network is the state-action pair, and the output is the loss value of the Actor network, which is used to update the Actor network. In the target network, the inputs of the Actor network and the Critic network are both the next state, the output of the Actor network is the next action, and the output of the Critic network is the loss value of the Critic, which is used to update the Critic in the evaluation network.

[0042] The bottom-level policy network selects the action under the state s based on the high-level policy network by the following formula:

[0043] a t =μ o (s)+ε,ε~N(0,σ)

[0044] wherein μ o (s) represents the bottom-level policy under the selection of the high-level policy o, and ε is used to generate random noise. The robot arm performs the action a t , and obtains the reward r t , and enters the next state s t+1 , and stores (s t , a t , r t , s t+1 ) in the bottom-level experience pool.

[0045] As shown in Figure 3 , the data obtained by interacting with the initial policy and the environment and the state pair (s t , a t ) are sorted manually according to experience, and the data with labels are input into the reward function learning model for training.

[0046] Specifically, the manual sorting is based on the assembly force of the assembly process, the assembly depth of each step, the assembly speed, etc. The label is the sequence number of the sorting, i.e., the priority.

[0047] The reward function learning model is composed of a first convolutional layer, a pooling layer, a second convolutional layer and a full connection layer connected in sequence. The input of the reward function learning model is a state-action pair (s t ,a t ) with a label, and the output is a reward value of the current state-action pair.

[0048] The output of the reward function learning model is used as a reward value to participate in the continuous update of the initial policy. The initial policy interacts with the environment to generate state-action pairs, and a human participates in sorting and learning a model that can obtain a reward function. The model outputs a reward value to update the initial policy, and the cycle is repeated.

[0049] The initial policy is the policy that has been learned so far. The learning of the initial policy and the learning of the reward function are alternately performed. In the process of learning the reward function, the current policy can be referred to as the initial policy.

[0050] In the embodiment, the data (s t ,a t ,s t+1 ,R t+1 ) in the bottom experience pool is used to update the bottom policy network. The bottom policy network is updated and trained by using the SAC network. Specifically:

[0051] 1) Update the policy network Actor:

[0052] Calculate the Q value of the state-action pair under the current policy network:

[0053] Q(s t ,a t )=Q Critic (s t ,a t )

[0054] Wherein Q Critic represents the Q value of the Critic network.

[0055] Calculate the entropy of the action generated by the policy network:

[0056] H(π(a t |s t ))=-∫π(a t |s t )logπ(a t |s t )da

[0057] Wherein, π represents the policy, and H represents the entropy.

[0058] Calculate the target entropy of the policy network:

[0059] H target= target_entropy * H (π (a t | s t ))

[0060] where H target represents the target entropy of the policy network.

[0061] The parameters of the policy network are updated using gradient descent method:

[0062]

[0063] where J (θ Actor ) is the objective function of the policy network, θ Actor is the parameter of the policy network, and a is a hyperparameter used to ensure that the actions generated by the policy network have a certain exploratory nature.

[0064] 2) Update the Critic network:

[0065] The target of the Q value is calculated using the collected experience data:

[0066] y = r t + γ (1 - d) Q TargetCritic (s t+1 , π TargetActor (s t+1 ))

[0067] where r t is the reward value, γ is the discount factor, d is a flag indicating whether the terminal state is reached, s t+1 is the next state, Q TargetCritic is the target Q network, and π TargetActor is the action generated by the target policy network.

[0068] The parameters of the evaluation Critic network are updated using gradient descent method:

[0069]

[0070] where J (θ Critic ) is the objective function of the Critic network.

[0071] 3) The parameters of the target Critic network are updated using the sliding average method:

[0072] θ TargetCritic ← τ θ Critic + (1 - τ) θ TargetCritic

[0073] where θ TargetCritic represents the parameters of the target Critic network, and θ Criticdenotes the Critic network parameters. τ < 1 is used to control the speed of the moving average.

[0074] Repeat steps 1)~3) above until the network update is finished.

[0075] As shown in Figure 5 , in this embodiment, the update training for the high-level policy network is:

[0076] 1) The Q value and V value of the high-level network are calculated using the following formula:

[0077]

[0078]

[0079] where s t denotes the state of the high-level network, o t denotes the action of the high-level network, i.e. the high-level policy, denotes the reward function, denotes the mean value.

[0080] 2) The advantage function of the high-level policy is calculated as follows, which represents the importance of the selected state-action pair.

[0081]

[0082] 3) The DQN network outputs the final high-level policy o, and the probability of selecting o is 1-ε.

[0083]

[0084] 4) According to the state s t+1 , the estimation of the target Q value function is updated as follows:

[0085] target = r + γ * max(Q(s t+1 , o ))

[0086] where γ is the discount factor, used to weigh the importance of the current reward and the future reward.

[0087] 5) Finally, using the current state s t , the action o t performed, the observed new state s t+1 , and the reward value r t+1 , the Q value function estimation of the current state is updated as follows:

[0088] Q(s t , o t ) = Q(s t , o t ) + α * (target - Q(s t , ot ))

[0089] In this embodiment, the interaction data obtained in each sub-process is passed to the main process to update the main network model, and the updated main network model assigns network weights to each sub-network:

[0090] φ1←φ

[0091] φ2←φ ...

[0093] φ n ←φ

[0094] wherein φ represents the weight of the main network, φ1, φ2,..., φ n represent the weights of each sub-network.

[0095] Using the trained main network offline model, complex multi-axis hole assembly tasks can be performed.

[0096] Embodiment Two

[0097] The purpose of this embodiment is to provide a multi-axis hole assembly system based on hierarchical reinforcement learning and distributed learning, comprising:

[0098] Establish a total strategy model module: use robot state data and action data to establish a main control assembly strategy model based on deep reinforcement learning;

[0099] Total strategy model training module: construct multiple sub-process networks based on different assembly interaction environments, use the robot interaction data obtained from the multiple sub-process networks to update and train the main control assembly strategy model, and obtain a trained main control assembly strategy model;

[0100] Wherein, the sub-process network includes a high-level policy network and a low-level policy network, the high-level policy network obtains a high-level policy value according to the robot state data at the current time, and the low-level policy network obtains the action of the robot at the next time according to the high-level policy value and the robot state data at the current time;

[0101] Execution control module: use the trained main control assembly strategy model to perform multi-axis hole assembly tasks of the robot.

[0102] Embodiment Three

[0103] The purpose of this embodiment is to provide a computing device, comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, wherein the processor executes the program to implement the steps of the above method.

[0104] Embodiment Four

[0105] The purpose of the present embodiment is to provide a computer-readable storage medium.

[0106] A computer-readable storage medium having stored thereon a computer program which, when executed by a processor, performs the steps of the above method.

[0107] The steps and methods involved in the devices of the above embodiments two, three and four correspond to the method of embodiment one, and the specific embodiments can be seen from the relevant description of embodiment one. The term "computer-readable storage medium" should be understood to include a single medium or multiple media of one or more instruction sets; it should also be understood to include any medium capable of storing, encoding or carrying a set of instructions for execution by a processor and causing the processor to perform any of the methods in the present application.

[0108] Those skilled in the art should understand that the above-mentioned modules or steps of the present application can be realized by a general computer device, alternatively, they can be realized by program codes executable by a computing device, so that they can be stored in a storage device for execution by a computing device, or they can be respectively made into individual integrated circuit modules, or a plurality of modules or steps among them can be made into a single integrated circuit module to realize. The present application is not limited to any specific combination of hardware and software.

[0109] Although the specific embodiments of the present application are described above in combination with the accompanying drawings, it is not a limitation on the scope of protection of the present application, and those skilled in the art should understand that various modifications or changes made by those skilled in the art on the basis of the technical solutions of the present application without creative labor are still within the scope of protection of the present application.

Claims

1. A multi-axis hole assembly method based on hierarchical reinforcement learning and distributed learning, characterized by comprising: establishing a master control assembly strategy model based on deep reinforcement learning using mechanical arm state data and action data; constructing multiple sub-process networks based on different assembly interaction environments, updating and training the master control assembly strategy model using the mechanical arm interaction data obtained from the multiple sub-process networks, and obtaining a trained master control assembly strategy model; wherein the sub-process network includes a high-level strategy network and a low-level strategy network, the high-level strategy network obtains a high-level strategy value according to the current time mechanical arm state data, and the low-level strategy network obtains the next time mechanical arm action according to the high-level strategy value and the current time mechanical arm state data; using the trained master control assembly strategy model to perform the multi-axis hole assembly task of the mechanical arm; the low-level strategy network includes an evaluation network and a target network, the evaluation network and the target network each include an action network and an evaluation network, the state data of the mechanical arm and the output of the high-level strategy network are input into the action network of the evaluation network to obtain the action under the current state of the mechanical arm; the state data of the mechanical arm and the action data are input into the evaluation network of the evaluation network to obtain the first loss value of the action network, and the action network in the evaluation network is updated according to the loss value; the next time mechanical arm state data is input into the action network and the evaluation network in the target network, the output of the action network in the target network is the corresponding action of the next time, the output of the evaluation network in the target network is the second loss value of the evaluation network, and the evaluation network in the evaluation network is updated according to the second loss value. The mechanical arm state data includes the pose of the mechanical arm end part, the contact force / torque of the mechanical arm end, and the assembly image data obtained by the camera. The current state data of the mechanical arm is input into the high-level strategy network to obtain the corresponding high-level strategy value. The current time mechanical arm state, the current time mechanical arm state corresponding action, the reward obtained by executing the action corresponding to the current state of the mechanical arm, and the next time mechanical arm action are stored in the low-level experience pool, and the low-level strategy network is updated using the low-level experience pool. The current time mechanical arm state and the current time mechanical arm state corresponding action are used as data and state pairs, the data and state pairs are sorted manually according to experience, the serial number after sorting is used as the label of the corresponding data and state pair, the data-state pair and the pair are trained according to the corresponding label reward function model, and the reward value of the input mechanical arm state and the corresponding action is obtained based on the trained reward model.

2. The multi-axis hole assembly method based on hierarchical reinforcement learning and distributed learning of claim 1, wherein, The training of the action network of the low-level strategy network is: calculating the Q value of the state-action under the current strategy network and the entropy of the action; obtaining the target entropy of the strategy network according to the entropy of the action; updating the parameters of the action network of the strategy network using the gradient descent method combined with the Q value of the state-action and the target entropy; 3. The multi-axis hole assembly method based on hierarchical reinforcement learning and distributed learning of claim 1, wherein, the training of the evaluation network of the low-level strategy network is: calculating the state-action based on experience data 4. The multi-axis hole assembly method based on hierarchical reinforcement learning and distributed learning of claim 1, wherein, ​ 5. The multi-axis hole assembly method based on hierarchical reinforcement learning and distributed learning of claim 4, wherein, ​ 6. The multi-axis hole assembly method based on hierarchical reinforcement learning and distributed learning of claim 1, wherein, ​ ​ The evaluation network in the target network is updated by using the gradient descent method combined with the state-action Q value target to update the parameters of the evaluation network in the evaluation network, and the parameters of the evaluation network in the target network are updated by using the sliding average method and the parameters of the evaluation network in the evaluation network.

7. A multi-axis hole assembly system based on hierarchical reinforcement learning and distributed learning, adopting the multi-axis hole assembly method based on hierarchical reinforcement learning and distributed learning as claimed in claims 1-6, characterized in that, Comprise: Establish a total strategy model module: use the state data and action data of the robot arm to establish a main control assembly strategy model based on deep reinforcement learning; A total strategy model training module: construct a plurality of sub-process networks based on different assembly interaction environments, update and train the main control assembly strategy model using the robot arm interaction data obtained by the plurality of constructed sub-process networks, and obtain a trained main control assembly strategy model; Wherein, the sub-process network includes a high-level policy network and a low-level policy network, the high-level policy network obtains a high-level policy value according to the state data of the robot arm at the current time, and the low-level policy network obtains the action of the robot arm at the next time according to the high-level policy value and the state data of the robot arm at the current time. An execution control module: use the trained main control assembly strategy model to execute the multi-axis hole assembly task of the robot arm. Comprise: A processor, a memory and a bus, the memory stores machine readable instructions executable by the processor, when the computer equipment runs, The processor and the memory communicate through the bus, the machine readable instructions are executed by the processor to execute the multi-axis hole assembly method based on hierarchical reinforcement learning and distributed learning according to any one of claims 1 to 6.

8. A computer device, comprising: The computer readable storage medium stores a computer program, which is executed by the processor to execute the multi-axis hole assembly method based on hierarchical reinforcement learning and distributed learning according to any one of claims 1 to 6. ​ ​ ​ ​ 9. A computer-readable storage medium, characterized in that, ​

Images