Component assembly method

Abstract

The application provides a component assembly method, comprising: acquiring state information of a component to be installed gripped by a mechanical arm; the state information comprises three-dimensional force, three-dimensional torque and a pose; inputting the state information into an assembly model to acquire a predicted action output by the assembly model; generating an action instruction based on the predicted action; the action instruction is used for controlling the mechanical arm to move; sending the action instruction to the mechanical arm; the mechanical arm is used for assembling the component to be installed; wherein the assembly model is obtained by training a double-delay deep deterministic policy gradient learning network based on a state information sample set. The component assembly method provided in the application embodiment realizes component installation automation by predicting the action of the mechanical arm based on the double-delay deep deterministic policy gradient learning network and controlling the mechanical arm to grip the component to be installed for installation.

Description

Technical Field

[0001] This invention relates to the field of manufacturing technology, and in particular to a component assembly method. Background Technology

[0002] In recent years, my country's robotics technology has developed rapidly, and robots have been widely used in component assembly tasks. However, the assembly environment has become increasingly complex, which has limited the application of robots. For example, in the field of assembling shafts and holes of large and heavy components, there are problems such as the uncertainty of the shaft component's state and excessive contact force between the shaft and hole, making it difficult to automate the assembly of large and heavy components.

[0003] Currently, component assembly tasks still rely on manual labor, but manual operations suffer from low efficiency and high costs. Therefore, the automation of component assembly has become an urgent problem to be solved. Summary of the Invention

[0004] This invention provides a component assembly method to solve the problems of high efficiency and low cost in manual component assembly in the prior art, and to realize the automation of component assembly.

[0005] This invention provides a component assembly method, comprising:

[0006] Acquire the status information of the component to be installed, which is held by the robotic arm; the status information includes three-dimensional force, three-dimensional torque, and pose;

[0007] The state information is input into the assembly model to obtain the predicted action output by the assembly model;

[0008] Based on the predicted motion, motion commands are generated; the motion commands are used to control the movement of the robotic arm.

[0009] The robotic arm sends the action command; the robotic arm is used to assemble the component to be installed.

[0010] The assembly model is obtained by training a dual-delay deep deterministic policy gradient learning network based on a set of state information samples; the pose is obtained based on distance information collected by a distance sensor; and the distance information is the distance between the component to be installed and the target position.

[0011] Optionally, the assembly model includes Actor1 network, Actor2 network, Critic1 network, Critic2 network, Critic3 network and Critic4 network;

[0012] The Actor1 network is used to input the current state information S at the current moment. t Then, output the S. t Predict action A at the corresponding current time.t And obtain the S based on the preset reward formula t The corresponding reward value r t The Actor1 network is also used based on the A t Obtain and output the state information S at the next time step. t+1 ;

[0013] The Actor2 network is used to input the S t+1 Then, output the S. t+1 Corresponding prediction action A t+1 ;

[0014] The Critic1 network is used to input the S t+1 and the A t+1 Then, output the S t+1 and the A t+1 The evaluation value q1;

[0015] The Critic2 network is used to input the S t+1 and the A t+1 Then, output the S t+1 and the A t+1 The evaluation value q2;

[0016] The Critic3 network is used to input the S t and the A t Then, output the S t and the A t The evaluation value q3;

[0017] The Critic4 network is used to input the S t and the A t Then, output the S t and the A t The evaluation value is q4.

[0018] Optionally, the preset reward formula is:

[0019]

[0020] Where r represents the reward value; F R Represents radial force. F x F represents the force acting along the x-axis. y F represents the force acting along the y-axis. T This represents the preset radial force threshold; β represents the hyperparameter used for scaling adjustment.

[0021] Optionally, the hyperparameter β can take values ​​in the range of [0,1].

[0022] Optionally, the hyperparameter β is set to 0.5.

[0023] Optionally, the assembly model is trained using the following method:

[0024] The state information sample S t The input is fed into the assembly model network to obtain the predicted action A output by the assembly model. t Reward value r t Next-moment state information sample S t+1 Predicted action A at the next moment t+1 Evaluation values ​​q1, q2, q3, and q4;

[0025] The assembly model is updated based on the evaluation value q1, the evaluation value q2, the evaluation value q3, and the evaluation value q4;

[0026] Repeat the above steps until the reward value r is reached. t The algebraic sum meets the preset threshold.

[0027] Optionally, updating the assembly model based on the evaluation value q1, the evaluation value q2, the evaluation value q3, and the evaluation value q4 includes:

[0028] Obtain the minimum value of q1 and q2, min(q1, q2);

[0029] The parameters of the Critic3 network and the parameters of the Critic4 network are updated based on the minimum value min(q1, q2), q3, and q4.

[0030] S t and the A t The updated evaluation value q3′ is obtained by inputting it into the updated Critic3 network.

[0031] The updated evaluation value q3′ is used as the loss function to update the parameters of the Actor1 network;

[0032] Based on the updated parameters of the Actor1 network, update the parameters of the Actor2 network;

[0033] Based on the updated parameters of the Critic3 network, update the parameters of the Critic1 network.

[0034] Based on the updated parameters of the Critic4 network, the parameters of the Critic2 network are updated.

[0035] Optionally, both the Actor1 network and the Actor2 network include 2K+1 layers of action subnetworks activated by the tanh function; each action subnetwork layer includes three fully connected layers and one Dropout layer;

[0036] The Critic1 network, the Critic2 network, the Critic3 network, and the Critic4 network each include 2K+1 layers of evaluation subnetworks activated by the tanh function; each evaluation subnetwork layer includes three fully connected layers.

[0037] Where K is a positive integer.

[0038] Optionally, K can be 2.

[0039] Optionally, the pose is obtained based on distance information collected by a distance sensor; the distance information is the distance between the component to be installed and the target position.

[0040] The component assembly method provided in this invention uses a dual-delay deep deterministic policy gradient learning network to predict the movements of a robotic arm and control the robotic arm to grip and install the component to be installed, thereby automating the component installation process. Attached Figure Description

[0041] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0042] Figure 1 This is a flowchart illustrating the component assembly method provided in an embodiment of the present invention;

[0043] Figure 2 This is a schematic diagram of the assembly structure of a large and heavy component provided in an embodiment of the present invention;

[0044] Figure 3 This is a schematic diagram of the assembly model provided in an embodiment of the present invention;

[0045] Figure 4 This is one of the schematic diagrams of the assembly model training process provided in the embodiments of the present invention;

[0046] Figure 5 This is the second schematic diagram of the assembly model training process provided in the embodiment of the present invention.

[0047] Figure label:

[0048] 1: Robotic arm; 2: Force sensor; 3: First major heavy component;

[0049] 4: Distance sensor; 5: The second heaviest component. Detailed Implementation

[0050] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0051] The following is combined with Figures 1-5 The component assembly method provided by the present invention is described.

[0052] Figure 1 This is a flowchart illustrating the component assembly method provided in an embodiment of the present invention, as shown below. Figure 1 As shown, the component assembly method provided in this embodiment of the invention includes:

[0053] Step 110: Obtain the status information of the component to be installed held by the robotic arm; the status information includes three-dimensional force, three-dimensional torque and pose.

[0054] The component assembly method provided in this invention can be applied to various component assembly scenarios, such as precision component assembly or heavy component assembly scenarios; the assembly object can be various types of parts, such as fork-type parts, disc-type parts, box-type parts, or shaft-hole-type parts.

[0055] Figure 2 This is a schematic diagram of the assembly structure of a large and heavy component provided in an embodiment of the present invention, such as... Figure 2 As shown, when the component assembly method is applied to the hole-finding installation of large and heavy components, the component to be installed can be the first large and heavy component 3, and the robotic arm 1 can be a multi-axis robotic arm with motion control function. The second large and heavy component 5 remains stationary, and the robotic arm 1 clamps the first large and heavy component 3 and moves it towards the second large and heavy component 5 for shaft hole assembly. A force sensor 2 is installed at the end of the robotic arm 1. The force sensor 2 can be a six-dimensional force / torque sensor, which is used to measure three-dimensional force and three-dimensional torque. The pose can be detected based on pose sensors or computer vision methods. Optionally, the pose is obtained based on distance information collected by a distance sensor; the distance information is the distance between the component to be installed and the target position. A distance sensor 4 is set at the upper end of the second large and heavy component 5. The distance sensor 4 can include multiple distance sensors, which are used to calculate the change in distance and orientation angle of the pose at each moment relative to the pose at the previous moment, thereby realizing real-time pose estimation. It is understood that... Figure 2 These are examples provided for ease of understanding in the embodiments of the present invention. The embodiments of the present invention do not specifically limit the assembly structure of large and heavy components.

[0056] The state information includes three-dimensional force, three-dimensional torque, and pose. Three-dimensional force refers to the force exerted on the component to be installed in three-dimensional space, including the force in the x-direction, y-direction, and z-direction in the three-dimensional coordinate system O-xyz; three-dimensional torque includes the torque in three-dimensional space, including the torque in the x-direction, y-direction, and z-direction; pose refers to the position and orientation of the component to be installed.

[0057] Step 120: Input the state information into the assembly model and obtain the predicted action output by the assembly model;

[0058] In this step, the assembly model is obtained by training a dual-delay deep deterministic policy gradient learning network based on a set of state information samples. Value-based learning networks tend to overestimate state values, and accumulated errors can cause the network to fall into suboptimal policies, resulting in low accuracy of the predicted actions. The dual-delay deep deterministic policy gradient learning network provided in this embodiment of the invention can solve the overestimation and high variance phenomena that occur in learning network predictions, thereby improving the accuracy of predicted actions.

[0059] It is understandable that the predicted action here can be the action of the first major heavy component 3 moving from its current position to its final position, as predicted by the assembly model based on the state information. For example, if the first major heavy component 3 is located at spatial coordinates (112,55,67), the predicted action can be a single action of moving from (112,55,67) to the target hole. The predicted action can also be a combination of multiple actions, such as the first action of moving from (112,55,67) to (0,55,67) based on the state information of the first major heavy component 3 when it is located at (112,55,67), and the second action of moving from (0,55,67) to the target hole based on the state information of the first major heavy component 3 when it is located at (0,55,67).

[0060] Step 130: Generate motion instructions based on the predicted motion; the motion instructions are used to control the movement of the robotic arm.

[0061] In one embodiment, a PC host computer is connected to the robotic arm 1. After the assembly model outputs predicted actions, the PC host computer generates action commands based on the predicted actions. It can be understood that since the action commands here are generated based on the predicted actions, when there are multiple predicted actions, multiple action commands are generated sequentially according to the time order of the predicted actions.

[0062] Step 140: Send the action command to the robotic arm; the robotic arm is used to assemble the component to be installed.

[0063] In one embodiment, a PC sends motion commands to the robotic arm 1, and the robotic arm 1 clamps the first heavy component 3 to perform hole searching based on the motion commands.

[0064] The component assembly method provided in this invention predicts the actions of a robotic arm based on a dual-delay deep deterministic policy gradient learning network, controls the robotic arm to grip and install the component to be installed, thereby automating component installation. Furthermore, since the dual-delay deep deterministic policy gradient learning network can overcome the defects of overestimation and high variance in reinforcement learning networks, it improves the robustness and compliance of component assembly.

[0065] Optionally, Figure 3 This is a schematic diagram of the assembly model structure provided in the embodiment of the present invention, such as... Figure 3 As shown, the assembly model includes Actor1 network, Actor2 network, Critic1 network, Critic2 network, Critic3 network and Critic4 network;

[0066] The Actor1 network is used to input the current state information S at the current moment. t Then, output the S. t Predict action A at the corresponding current time. t And obtain the S based on the preset reward formula t The corresponding reward value r t The Actor1 network is also used based on the A t Obtain and output the state information S at the next time step. t+1 ;

[0067] The Actor2 network is used to input the S t+1 Then, output the S. t+1 Corresponding prediction action A t+1 ;

[0068] The Critic1 network is used to input the S t+1 and the A t+1 Then, output the S t+1 and the A t+1 The evaluation value q1;

[0069] The Critic2 network is used to input the S t+1 and the A t+1 Then, output the S t+1 and the A t+1 The evaluation value q2;

[0070] The Critic3 network is used to input the S t and the A t Then, output the S t and the A t The evaluation value q3;

[0071] The Critic4 network is used to input the S t and the A t Then, output the S t and the A t The evaluation value is q4.

[0072] In this embodiment of the invention, the assembly model employs a dual-delay deep deterministic gradient learning network, comprising six sub-networks: two Actor networks and four Critic networks. For the Actor networks, the input is state information, and the output is a predicted action; for the Critic networks, the input is the state and its corresponding predicted action, and the output is an evaluation value. A larger evaluation value corresponding to the predicted action output by the Actor network indicates better network training performance; a more accurate evaluation value output by the Critic network indicates better network training performance. The evaluation values ​​are used to update the network parameters.

[0073] For the Actor1 network, in addition to the predicted action output, it also uses the reward value r corresponding to the input state information obtained based on a preset reward formula. t After the Actor1 network outputs the predicted action, it generates an action command based on the predicted action, collects and outputs the state information of the component to be installed after the action command is completed, i.e., the state information S at the next moment. t+1 It is understandable that the predicted action output by the assembly model in step 120 is output by the trained Actor1 network.

[0074] Optionally, the preset reward formula is:

[0075]

[0076] Where r represents the reward value; F R Represents radial force. F x F represents the force acting along the x-axis. y F represents the force acting along the y-axis. T This represents the preset radial force threshold; β represents the hyperparameter used for scaling adjustment.

[0077] In one embodiment, the state information S1 of the component to be installed at a first moment is obtained, and S1 includes the force F acting on the component in the x-axis direction in the three-dimensional coordinate system O-xyz. x1 and torque M x1The force F in the y-axis direction y1 and torque M y1 The force F in the z-axis direction z1 and torque M z1 And pose P1; input the state information S1 of the component to be installed at the first moment into the Actor1 network, and the Actor1 network outputs the predicted action A1 and the radial force threshold F. T The settings are related to action A1, and the reward value for the current state can be calculated using known parameters and preset reward formulas.

[0078] The preset reward formula provided in this embodiment of the invention has a reward function that takes values ​​between [-1, 1]. When F R The value is less than or equal to F T When the model assembles, it receives a positive reward; otherwise, it receives a negative reward. Taking the logarithm can scale the reward value to a reasonable range, which is beneficial for network convergence.

[0079] Optionally, the hyperparameter β can take values ​​in the range of [0,1].

[0080] Optionally, the hyperparameter β is set to 0.5.

[0081] Optionally, both the Actor1 network and the Actor2 network include 2K+1 layers of action subnetworks activated by the tanh function; each action subnetwork layer includes three fully connected layers and one Dropout layer;

[0082] The Critic1 network, the Critic2 network, the Critic3 network, and the Critic4 network each include 2K+1 layers of evaluation subnetworks activated by the tanh function; each evaluation subnetwork layer includes three fully connected layers.

[0083] Where K is a positive integer.

[0084] For Actor1 and Actor2 networks, each action subnetwork consists of three fully connected layers and one Dropout layer connected in series. Optionally, a Dropout layer is connected after every two fully connected layers.

[0085] Optionally, K can be 2.

[0086] Optionally, Figure 4 This is one of the schematic diagrams of the assembly model training process provided in the embodiments of the present invention, such as... Figure 4 As shown, the assembly model is trained using the following method:

[0087] Step 410, transfer the state information sample S tThe input is fed into the assembly model network to obtain the predicted action A output by the assembly model. t Reward value r t Next-moment state information sample S t+1 Predicted action A at the next moment t+1 Evaluation values ​​q1, q2, q3, and q4;

[0088] Specifically, the state information sample S t It was obtained through random sampling.

[0089] It is understandable that for the state information sample S at the next moment... t+1 When the distance sensor 4 detects that the first heavy component 3 and the second heavy component 5 have been assembled, it will no longer output the status information sample S for the next moment. t+1 .

[0090] In one embodiment, an initial position is randomly generated, and the robotic arm is controlled by an action thread to move the first heavy component 3 to the initial position. The current state of the first heavy component 3 is collected to obtain state information sample S0(F). x0 F y0 F z0 M x0 M y0 M z0 The state information sample S0 is input into the Actor1 network in the assembly model network to obtain the predicted action A0, reward value r0, and the state information sample S1 (F) output by the Actor1 network. x1 F y1 F z1 M x1 M y1 M z1 (P1). Store (S0, A0, r0, S1) as a tuple in set V. Optionally, during training, since the network parameters have not yet been updated, the predicted actions output by the Actor1 network are random. S0 can be predicted multiple times to obtain multiple different predicted actions A0′, A0″, A0″′, etc., and the corresponding tuples are stored in set V, thereby increasing the richness of the training samples.

[0091] The learning thread randomly samples state information samples S1 from set V, and inputs S1 into the Actor2 network to obtain the predicted action A1 output by the Actor2 network. Optionally, noise N can be added to the state information samples S1, and the state information samples with added noise N can be input into the Actor2 network to obtain the corresponding predicted action output by the Actor2 network. Adding noise can simulate the disturbances encountered during component assembly in practical applications, thereby optimizing the model.

[0092] Input S1 and A1 into the Critic1 network to obtain the Critic1 network output evaluation value q1 for S1 and A1;

[0093] Input S1 and A1 into the Critic2 network to obtain the Critic2 network output evaluation value q2 for S1 and A1;

[0094] Input S0 and A0 into the Critic3 network to obtain the Critic3 network output evaluation value q3 for S0 and A0;

[0095] Input S0 and A0 into the Critic4 network to obtain the Critic4 network output evaluation value q4 for S0 and A0.

[0096] Because the Critic1 and Critic2 networks have different initial parameters, their output evaluation values ​​differ even with the same input. The same applies to the Critic3 and Critic4 networks. This approach is beneficial for optimizing network parameters.

[0097] Step 420: Update the assembly model based on the evaluation value q1, the evaluation value q2, the evaluation value q3, and the evaluation value q4;

[0098] Optionally, updating the assembly model based on the evaluation value q1, the evaluation value q2, the evaluation value q3, and the evaluation value q4 includes:

[0099] Step 421: Obtain the minimum value min(q1, q2) of q1 and q2;

[0100] As mentioned above, evaluation-based learning networks suffer from overestimation. By using two Critic networks to output evaluation values ​​and then selecting the smaller one, the overestimation problem can be avoided as much as possible, thereby improving the accuracy of the assembly model.

[0101] Step 422: Update the parameters of the Critic3 network and the parameters of the Critic4 network based on the minimum value min(q1, q2), q3, and q4;

[0102] Specifically, the mean squared loss cl1 for the minimum values ​​min(q1, q2) and q3 is calculated, and the mean squared loss cl2 for the minimum values ​​min(q1, q2) and q4 is calculated. cl1 and cl2 are added together to obtain the total mean squared loss cl. Backpropagation based on the total mean squared loss cl is used to update the Critic3 and Critic4 networks. The Critic3 and Critic4 networks are used to make the training of the Critic network more stable. The Critic3 network is also used to constrain the loss function in the Action1 network.

[0103] Step 423, the S t and the A t The updated evaluation value q3′ is obtained by inputting it into the updated Critic3 network.

[0104] Input S0 and A0 into the updated Critic3 network to obtain the updated evaluation value q3′ of S0 and A0 output by the updated Critic3 network.

[0105] Step 424: Update the parameters of the Actor1 network using the updated evaluation value q3′ as the loss function;

[0106] Optionally, Actor1 can be updated after the Critic3 network has been updated multiple times, such as after the Critic3 network has been updated four times. The Actor1 network updates lag behind the Critic3 network updates, ensuring the stability of Actor1 updates.

[0107] Step 425: Update the parameters of the Actor2 network based on the updated parameters of the Actor1 network;

[0108] Specifically, a certain proportion of the weights from Actor1 are added to the weights of Actor2 network to create new weights for Actor2, effectively performing a soft update of Actor2 based on the parameters of Actor1 network. This method of updating network parameters ensures stable and fast network training.

[0109] Step 426: Update the parameters of the Critic1 network based on the updated parameters of the Critic3 network;

[0110] Specifically, a certain proportion of the weights of Critic3 are added to the weights of Critic1 network to form the new weights of Critic1, that is, the Critic1 network is softly updated based on the parameters of the Critic3 network.

[0111] Step 427: Update the parameters of the Critic2 network based on the updated parameters of the Critic4 network.

[0112] Specifically, a certain proportion of the weights of Critic4 are added to the weights of the Critic2 network to form the new weights of Critic2, that is, the Critic2 network is softly updated based on the parameters of the Critic4 network.

[0113] The weight ratios in steps 425, 426, and 427 can be set according to the training effect. This embodiment of the invention does not limit the value of the weight ratio.

[0114] Step 430: Repeat the above steps until the reward value r is reached. t The algebraic sum meets the preset threshold.

[0115] In one embodiment, S0 is input into the Actor1 network in the assembly model to obtain tuple v1(S0, A0, r0, S1); S1 is input into the Actor1 network in the assembly model to obtain tuple v(S1, A1, r1, S2); ...; S n-1 The input to the Actor1 network in the assembly model yields the tuple v. n-1 (S n-1 A n-1 r n-1 S n When the algebraic sum of the reward values ​​R t = r0 + r1 + ... + r n-1 Training ends when the preset threshold is met. It's understandable that the steps involved in this process, such as updating network parameters, random sampling, and adding noise, are as described above and will not be repeated here.

[0116] Optionally, before step 410, the method further includes:

[0117] Step 400: Collect teaching data;

[0118] The operator controls the robotic arm 1 via computer to insert the first heavy component 3 into the second heavy component 5. Force sensor 2 collects three-dimensional force and torque information of the first heavy component 3 during assembly, and distance sensor 4 collects its pose information. The computer records the state information of the heavy component 1 at different times, forming a teaching sample set (S1, S2, ..., S...). n It is understandable that controlling the robotic arm 1 to assemble the first major heavy component 3 from different initial positions can generate multiple sets of different state information teaching samples.

[0119] Optionally, at least 200 teaching data points are collected, meaning the total number N of state information teaching samples is at least 200.

[0120] Step 401: Pre-train the assembly model based on the teaching data.

[0121] The state information teaching samples from step 400 are stored in set S. Since actions are usually strongly correlated in reinforcement learning, the state information teaching samples are scattered and placed in set S to eliminate the correlation between them. The pre-trained assembly model then predicts actions based on the known state information in set S, improving the usability of the predicted actions. Furthermore, because the state information teaching samples in set S are uncorrelated, the predicted actions output by the assembly model have a certain degree of randomness, ensuring that the assembly model has some exploration during training, thus leading to better training.

[0122] Figure 5 This is the second schematic diagram of the assembly model training process provided in the embodiment of the present invention, as shown below. Figure 5 As shown, the assembly model training method is as follows:

[0123] S1, Teaching data acquisition. The operator controls the robotic arm 1 via computer to load the first heavy component 3 into the second heavy component 5, and uses the status information collected during the assembly process as teaching data.

[0124] S2, Construction of the Dual-Delay Deep Deterministic Policy Gradient Learning Network. The dual-delay deep deterministic policy gradient learning network designed in this embodiment consists of 6 sub-networks, including 2 Actor networks and 4 Critic networks.

[0125] S3 is a hole-finding algorithm for assembling large and heavy components.

[0126] S301, the teaching data collected in step S1 is grouped into a set T = {t1, t2, t3, ..., tn}, where n = 1, 2, 3, ..., and the set T is stored in the memory bank.

[0127] S302, based on the data in set T, pre-train the dual-delay deep deterministic policy gradient learning network to obtain the pre-trained assembly model.

[0128] S303 applies an action thread strategy and a learning thread strategy to train a dual-delay deep deterministic policy gradient learning network online.

[0129] (1) Actor1 network: The state S at time t t Input into Actor1 generates action A t and reward value r t Action A tInput is given to the robot to make it move, and then the state after the movement is collected as S. t+1 And put (S) t A t ,r t ,S t+1 The set is stored as a tuple in set V, and set V is stored in memory.

[0130] (2) Actor2 network: Randomly sample a state S from set V. t+1 Random noise is added and input into the Actor2 network to generate action A. t+1 .

[0131] (3) Critic1 network: random sampling state S t+1 Action A output by the Actor2 network t+1 The values ​​q1 are then input into the Critic1 network.

[0132] (4) Critic2 network: random sampling state S t+1 Action A output by the Actor2 network t+1 The values ​​q2 are then input into the Critic2 network.

[0133] (5) Critic3 network: The state S at time t t Action A output by the Actor1 network t The values ​​q3 are then input into the Critic3 network to obtain the valuation q3.

[0134] (6) Critic4 network: The state S at time t t Action A output by the Actor1 network t t, together with the input into the Critic4 network, yields the estimated value q4.

[0135] (7) Critic3 and Critic4 network update: Take the minimum value of q1 and q2 min{q1, q2}, then calculate the mean square loss cl1 of q3 and min{q1, q2} and the mean square loss cl2 of q4 and min{q1, q2}, and finally update the weights of the two network models of Critic3 and Critic4 by cl1+cl2.

[0136] (8) Actor1 network update: After the Critic3 network weights are updated four times, the state S at time t is updated. t And the state S at time t t The action A obtained by inputting t into actor1 tThe output values ​​obtained by inputting these values ​​into Critic3 are used as the output values ​​of the loss function of the Actor1 network, and then the weights of the Actor1 network are updated based on the loss function.

[0137] (9) Actor2 network, Critic1 network and Critic2 network update: After each update of Actor1 network, Actor2 network, Critic1 network and Critic2 network are updated in sequence.

[0138] Actor2 network update: Add a certain proportion of the weights of Actor1 to the weights of Actor2 network, and use them as the new weights of Actor2.

[0139] Critic1 network update: Add a certain proportion of the weights of Critic3 to the weights of Critic1 network to create new weights for Critic1.

[0140] Critic2 network update: Add a certain proportion of the weights of Critic4 to the weights of Critic2 network to create the new weights for Critic2.

[0141] S304: Training ends when the algebraic sum of the reward value during training is increased and converged through continuous actions, and training stops when the preset algebraic sum threshold or the preset number of iterations is reached.

[0142] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A component assembly method, characterized in that, include: Obtain the status information of the component to be installed held by the robotic arm; The state information includes three-dimensional force, three-dimensional torque, and pose; The state information is input into the assembly model to obtain the predicted action output by the assembly model; Based on the predicted motion, motion commands are generated; the motion commands are used to control the movement of the robotic arm. The robotic arm sends the action command; the robotic arm is used to assemble the component to be installed. The assembly model is obtained by training a dual-delay deep deterministic policy gradient learning network based on a set of state information samples. The assembly model includes Actor1 network, Actor2 network, Critic1 network, Critic2 network, Critic3 network and Critic4 network. The Actor1 network is used to input the current state information. Then, output the above. Predict the action at the corresponding current moment. and obtain the reward based on the preset reward formula Corresponding reward value The Actor1 network is also used based on the Obtain and output the state information at the next time step. ; The Actor2 network is used to input the... Then, output the above. Corresponding predictive action ; The Critic1 network is used to input the... and stated Then, output the... and stated Evaluation value ; The Critic2 network is used to input the... and stated Then, output the... and stated Evaluation value ; The Critic3 network is used to input the... and stated Then, output the... and stated Evaluation value ; The Critic4 network is used to process the input... and stated Then, output the... and stated Evaluation value ; The preset reward formula is as follows: ； in, Represents the reward value; Represents radial force. , This represents the force acting along the x-axis. This represents the force acting along the y-axis. This represents the preset radial force threshold; β represents the hyperparameter used for scaling adjustment.

2. The component assembly method according to claim 1, characterized in that, The hyperparameter β has a value range of [0,1].

3. The component assembly method according to claim 1, characterized in that, The hyperparameter β is set to 0.

5.

4. The component assembly method according to claim 1, characterized in that, The assembly model is trained using the following method: Sample of status information The input is fed into the assembly model network to obtain the predicted action output by the assembly model. Reward Value Next-moment state information sample Predicting actions at the next moment Evaluation value Evaluation value Evaluation value And the evaluation value q4; Based on the evaluation value The evaluation value The evaluation value and the evaluation value Update the assembly model; Repeat the above steps until the reward value is reached. The algebraic sum meets the preset threshold.

5. The component assembly method according to claim 4, characterized in that, Based on the evaluation value The evaluation value The evaluation value and the evaluation value Updating the assembly model includes: Obtain the and stated minimum value ; Based on the minimum value The and stated Update the parameters of the Critic3 network and the Critic4 network; The and stated The input is fed into the updated Critic3 network to obtain the updated evaluation value output by the updated Critic3 network. ; The updated evaluation value The parameters of the Actor1 network are updated as a loss function; Based on the updated parameters of the Actor1 network, update the parameters of the Actor2 network; Based on the updated parameters of the Critic3 network, update the parameters of the Critic1 network. Based on the updated parameters of the Critic4 network, the parameters of the Critic2 network are updated.

6. The component assembly method according to claim 1, characterized in that, Both the Actor1 network and the Actor2 network include Each layer employs an action subnetwork activated by the tanh function; each action subnetwork comprises three fully connected layers and one Dropout layer. The Critic1 network, the Critic2 network, the Critic3 network, and the Critic4 network all include Each layer employs an evaluation subnetwork activated by the tanh function; each evaluation subnetwork comprises three fully connected layers. Where K is a positive integer.

7. The component assembly method according to claim 6, characterized in that, The value of K is 2.

8. The component assembly method according to claim 1, characterized in that, The pose is obtained based on distance information collected by a distance sensor; the distance information is the distance between the component to be installed and the target position.

Images