An industrial robot shaft hole assembly method, device and equipment based on deep reinforcement learning

Abstract

The application relates to an industrial robot shaft hole assembly method, device and equipment based on deep reinforcement learning. The method comprises the following steps: constructing a state set and an action set of an industrial robot; inputting the state set and the action set into a trained shaft hole assembly algorithm model based on deep reinforcement learning to output selected assembly actions; executing the assembly actions to complete shaft hole assembly and recording the state when the shaft hole assembly is completed, taking the state as the input of the next state to select the next assembly action. Compared with the prior art, the application has the advantages of not needing to accurately measure the shaft hole position in advance, strong adaptability, application, improved production efficiency and precision and the like.

Description

Technical Field

[0001] This invention relates to the field of shaft and hole assembly technology for industrial robots, and in particular to a method, apparatus and equipment for shaft and hole assembly of industrial robots based on deep reinforcement learning. Background Technology

[0002] Shaft and hole assembly is a classic task for industrial robots. Based on whether the shaft and hole are in contact during the assembly process, this task can be divided into non-contact and contact methods. Contact methods often employ tactile sensing control, but they suffer from poor safety, complex processes, and low efficiency. While this method can achieve an optimal control strategy, its complexity in acquiring teaching samples and system design leads to poor versatility, hindering its application in robot shaft and hole assembly. Non-contact methods generally use vision alignment to complete shaft and hole assembly. This method offers better safety, but its assembly efficiency and accuracy need further improvement. Traditional shaft and hole assembly methods in industrial robots often rely on manual programming or modeling based on known shaft and hole models. These methods require reprogramming for changing environments or new shaft and hole models, thus having significant limitations.

[0003] To address this issue, vision-based methods primarily utilize cameras at the robot's end effector to acquire positional information of the shaft hole and workpiece, thereby achieving shaft hole assembly. However, accuracy is limited by factors such as camera precision and ambient lighting. Another common method is force feedback control. This method uses force sensors to acquire force and torque information at the robot's end effector and uses feedback control to control the robot's movement, thus achieving shaft hole assembly. Force-based control methods offer higher accuracy, but require manual design of features and control strategies, making them difficult to adapt to complex scenarios and changing environments. Summary of the Invention

[0004] The purpose of this invention is to overcome the shortcomings of the existing technology and provide an industrial robot shaft hole assembly method, device and equipment based on deep reinforcement learning. The deep reinforcement learning method of this invention has stronger adaptability and generalization ability, does not require manual design of features and control strategies, can quickly adapt to different scenarios and environments, and can autonomously learn effective control strategies.

[0005] The objective of this invention can be achieved through the following technical solutions:

[0006] In a first aspect, the present invention discloses an industrial robot shaft hole assembly method based on deep reinforcement learning, the method comprising the following steps:

[0007] S1: Construct the state set and motion set of the industrial robot:

[0008] S2: Input the aforementioned state set and action set into the trained deep reinforcement learning-based shaft hole assembly algorithm model, and output the selected assembly action;

[0009] S3: Perform the assembly action to complete the shaft hole assembly, record the state when the shaft hole is completed, use this state as the input for the next state, and select the next assembly action.

[0010] Furthermore, the state set in S1 is specifically a set of states at the end of the robotic arm, including force, torque, speed, acceleration, and position information of the shaft hole at the end of the robotic arm; the action set includes the displacement magnitude and direction information of the assembly insertion.

[0011] Furthermore, a state in the state set specifically includes force and torque, velocity, acceleration, and position information of the robotic arm's end effector. A state can be represented as s. t :

[0012] s t =[F x F y F z T x T y T z v x v y v z a x a y a z p x p y p z ]

[0013] The forces in the three directions are F. x F y F z The torques in the three directions are T respectively. x T y T z The terminal velocity is v = (v x v y v z ), acceleration is a = (a x a y a z The position of the shaft hole is p = (p x p y p z );

[0014] One action in the action set can be represented as a.t :

[0015] a t =[d,u x u y u z ]

[0016] Where the displacement magnitude is d, and the direction vector is...

[0017] Optionally, the state set and action set of the industrial robot constructed in S1 are preprocessed, including data filtering, removal of static forces, and normalization.

[0018] Specifically, the training process of the shaft-hole assembly algorithm model based on deep reinforcement learning in S2 includes the following steps:

[0019] S21. Input the constructed state into the shaft hole assembly algorithm model to construct a reward function for the deep reinforcement learning algorithm. For the input state, the reward function provides an immediate reward r. t feedback;

[0020] S22. Under the input state, the DQN algorithm is used to learn to select the corresponding action, including at each time step, taking the current state as input and using a deep neural network to output the Q value of each action;

[0021] S23. Use an experience replay cache to store previous states, actions, rewards, and the next state (s t a t r t s t+1 During the training process, a greedy strategy is used to select the action that maximizes the Q-value.

[0022] S24. Repeat steps S21 to S23, try different reward functions, deep neural network structures and parameter iterative optimization of the shaft hole assembly algorithm model, and output the action with the maximum Q value selected based on the input state.

[0023] Furthermore, the reward function R in S21 is:

[0024] R = R pos +R force +R action

[0025] Among them, R pos R represents the proximity to the target location. force R represents the continuity of force feedback change. action Indicates the smoothness of the movement;

[0026] The proximity to the target location can be determined by calculating the Euclidean distance between the current location and the target location, that is:

[0027]

[0028] Among them, w pos It is the weighting factor for positional rewards, p t This is the current position, p g d is the target position, and d is a scaling factor used to control the size of the reward;

[0029] The continuity of force feedback changes can be characterized by calculating the difference between the current force feedback and the force feedback at the previous moment, i.e.:

[0030]

[0031] Among them, w force It is the weighting factor for force feedback rewards, f t It is the force feedback at the current moment, f t-1 It is the force feedback from the previous moment;

[0032] The smoothness of a motion can be characterized by calculating the difference between the current motion and the previous motion, i.e.:

[0033]

[0034] Among them, w action It is the weighting factor for action reward, a t It refers to the action at the current moment, a. t-1 It represents the action at the previous moment, and σ is the standard deviation coefficient, used to control the size of the reward.

[0035] Furthermore, in S23, the DQN algorithm is used to learn and select the corresponding action, which is achieved through a combination of two methods: experience replay and the main network and the target network.

[0036] Furthermore, the action of selecting the maximum Q value according to the greedy strategy in S23 specifically includes: randomly selecting an action with a probability of ∈, gradually decreasing the value of ∈, and selecting the action of the maximum Q value with a probability of 1-∈.

[0037] Secondly, the present invention discloses an industrial robot shaft hole assembly device based on deep reinforcement learning. The device includes the following components: a CRRC6 industrial six-axis robot, a six-dimensional force sensor, and a six-axis robot host computer. The device is used to implement any of the methods described above.

[0038] Thirdly, the present invention discloses an industrial robot shaft and hole assembly device based on deep reinforcement learning. The device includes the following main components: a base, a shaft, a hole, a force sensor connector, a shaft and hole connector, and a motion control console. The device is used to implement any of the methods described above.

[0039] Compared with the prior art, the present invention has the following beneficial effects:

[0040] (1) Traditional shaft hole assembly usually requires precise measurement of shaft hole position. However, the shaft hole assembly based on deep reinforcement learning of this invention can autonomously perceive and adjust the position and posture of the robotic arm through the learning process, thereby achieving the purpose of correctly inserting the shaft hole without the need to accurately measure the shaft hole position in advance.

[0041] (2) The shaft hole assembly based on deep reinforcement learning of this invention can adapt to different shaft hole sizes and shapes. Through the learning process, it can autonomously perceive and adjust the movements of the robotic arm, thereby achieving efficient assembly of different types of shaft holes and has strong adaptability and applicability.

[0042] (3) The shaft hole assembly based on deep reinforcement learning of this invention can adapt to complex environmental changes through the learning process, such as changes in the posture and position of the robotic arm, external interference, etc., thereby improving the reliability and robustness of the assembly and enabling it to quickly adapt to complex environments.

[0043] (4) Compared with the traditional method of manually adjusting parameters and rules, the shaft hole assembly based on deep reinforcement learning of this invention can learn autonomously without the need for manual feature extraction and rule design, thereby greatly reducing the workload of manual intervention and adjustment, and has the intelligence of autonomous learning, which improves production efficiency and accuracy. Attached Figure Description

[0044] Figure 1 This is a flowchart of the method of the present invention;

[0045] Figure 2 This is a flowchart of the training process for the shaft-hole assembly model based on deep reinforcement learning according to the present invention.

[0046] Figure 3 This is a flowchart illustrating the specific implementation of the present invention: industrial robot shaft and hole assembly based on deep reinforcement learning.

[0047] Figure 4 This is a flowchart of the DQN algorithm based on deep reinforcement learning and Q-Learning of the present invention;

[0048] Figure 5 and Figure 6 This is a physical image of the CRRC industrial six-axis robot shaft hole assembly equipment using the method of this invention. Detailed Implementation

[0049] The present invention will now be described in detail with reference to the accompanying drawings and specific embodiments. These embodiments are based on the technical solution of the present invention and provide detailed implementation methods and specific operating procedures. However, the scope of protection of the present invention is not limited to the following embodiments.

[0050] Example

[0051] like Figure 1 and Figure 3 As shown in the figure, this embodiment provides a method for assembling shaft holes in industrial robots based on deep reinforcement learning. The specific steps of this method are as follows:

[0052] Step 1: Construct the state set and motion set of the industrial robot. Specifically, the state can include sensors, such as force sensors and position sensors, measuring force, velocity, acceleration, and position information. The motion can include the magnitude and direction of displacement during assembly and insertion. The state can include the force and torque in six directions measured by sensors, and the velocity, acceleration, and position information of the robotic arm's end effector. Let the forces in the three directions be F... x F y F z The torques in the three directions are T respectively. x T y T z The terminal velocity is v = (v x v y v z ), acceleration is a = (a x a y a z The position of the shaft hole is p = (p x p y p z If the state is s, then the state is s. t It can be represented as:

[0053] s t =[F x F y F z T x T y T z v x v y v z a x a y a z p x p y p z ]

[0054] The action can include the magnitude and direction information of the displacement during assembly insertion. Let the magnitude of the displacement be d, and the direction vector be... Then action a t It can be represented as:

[0055] a t =[d,u x u y u z ]

[0056] In actual shaft and hole assembly, the acquired status information may be affected by various factors, such as noise and inconsistent sampling frequencies. Therefore, some data preprocessing operations are required to improve the accuracy and reliability of the status information.

[0057] Specifically, data preprocessing operations include:

[0058] Data filtering: A low-pass filter is used to filter the force sensor measurement data to remove high-frequency noise and unstable interference signals, thereby reducing data fluctuations and errors.

[0059] Static force removal: Since force sensors are affected by static forces such as gravity, these static forces need to be removed from the data, retaining only the dynamic force signal. This can be achieved using gravity compensation or zero-point calibration; in this case, zero-point calibration is used.

[0060] Normalization: The Z-Score-based normalization method is used to scale the measurements to a fixed range, which is [-1, 1] to eliminate the differences between different units of measurement for acceleration, velocity, displacement and force, and improve the comparability of state information.

[0061] Step 2: Input the constructed state set and action set into the trained deep reinforcement learning-based shaft hole assembly algorithm model, and output the selected assembly actions.

[0062] like Figure 2 As shown, specifically, it includes the following steps:

[0063] S21. Input the state into the shaft hole assembly algorithm model based on deep reinforcement learning, and construct a suitable reward function for the deep reinforcement learning algorithm, including the proximity of the target position, the continuity of force feedback changes, and the smoothness of the action.

[0064] For the shaft-hole assembly task based on force feedback, the following reward function can be designed:

[0065] R = R pos +R force +R action

[0066] Among them, R pos R represents the proximity to the target location.force R represents the continuity of force feedback change. action It indicates the smoothness of the movement.

[0067] The proximity to the target location can be determined by calculating the Euclidean distance between the current location and the target location, that is:

[0068]

[0069] Among them, w pos It is the weighting factor for positional rewards, p t This is the current position, p g Here, is the target position, and d is a scaling factor used to control the size of the reward. This reward function allows the robot to receive a reward as it gets closer to the target position, with the reward gradually increasing as it gets closer.

[0070] The continuity of force feedback changes can be characterized by calculating the difference between the current force feedback and the force feedback at the previous moment, i.e.:

[0071]

[0072] Among them, w force It is the weighting factor for force feedback rewards, f t It is the force feedback at the current moment, f t-1 It is the force feedback from the previous moment.

[0073] The smoothness of a motion can be characterized by calculating the difference between the current motion and the previous motion, i.e.:

[0074]

[0075] Among them, w action It is the weighting factor for action reward, a t It refers to the action at the current moment, a. t-1 σ represents the action at the previous moment, and σ is the standard deviation coefficient used to control the magnitude of the reward. This reward function encourages the robot to make its movements as smooth as possible and its force feedback to tend to be stable, thus making the assembly process smoother.

[0076] In summary, the final reward function can be obtained:

[0077]

[0078] Where, p t p g f t f t-1 a t and a t-1These are the state variables and action variables at the current moment and the previous moment.

[0079] The above formula is used to construct a reward function, which can obtain immediate reward feedback based on a given state and action.

[0080] S22. The DQN algorithm is used to learn to select the appropriate action in a given state. At each time step, the robotic arm's end effector takes the current state as input, and the deep neural network outputs the Q-value for each action. Specifically, during the assembly process, for a given state s and action a, the immediate reward r is... t Feedback is used to predict the Q-value Q(s, a; θ) using a deep neural network, where θ are the parameters of the neural network. In DQN, Q cannot be directly computed... * Therefore, a deep neural network is used to approximate Q. * From this, the optimal Q value can be obtained.

[0081] function Q * for:

[0082] Q * (s, a) = max E[r] t +γQ * (s t+1 a t+1 )|s t =s,a t =a]

[0083] Where r t The instant reward obtained at time step t, s t+1 and a t+1 Let be the state and action at time step t+1, and γ be the discount factor. This formula represents the maximum expected cumulative reward obtainable by taking action a in state s.

[0084] To improve the efficiency and stability of learning, two methods were used: experience playback and a combination of the main network and the target network.

[0085] refer to Figure 4 During the learning process, the neural network is not only updated using the current state and actions, but also stored in an experience replay cache. Each time the neural network is updated, some experience samples are randomly sampled from the experience replay cache to train the neural network, thus avoiding data correlation problems that may occur during training.

[0086] Experience replay:

[0087] Experience replay is performed according to the greedy strategy, and the network parameters are updated by using the data in the experience replay buffer to train the network. The network parameters are updated by minimizing the error between the current Q value and the target Q value.

[0088] The implementation steps include the following:

[0089] 1. Initialize ∈ value: Set an initial ∈ value, such as 0.9 or 0.8.

[0090] 2. Generate random numbers: Generate a random number r between 0 and 1.

[0091] 3. Determine between greedy or exploratory: If the generated random number r is less than ∈, then execute the exploratory strategy and randomly select an action; otherwise, execute the greedy strategy and select the action with the maximum Q value in the current state.

[0092] 4. Update ∈ value: Gradually reduce the ∈ value based on factors such as the number of training sessions or training rounds, so as to explore more in the early stage of training and make more use of it in the later stage.

[0093] Combination of main network and target network:

[0094] The following are the parameter update steps for the main network and the target network:

[0095] 1. First, copy the parameters of the main network to the target network.

[0096] 2. During training, at regular intervals (e.g., 1000 steps), the parameters of the main network are copied to the target network. This copying uses a soft update method.

[0097] 3. When calculating the target Q value, the target network is used. That is, the next state S′ is passed in, the Q value of all actions is calculated, and then the largest Q value is selected as the target Q value to update the Q value of the current state.

[0098] 4. After calculating the target Q value, calculate the loss function and update the parameters of the main network using the backpropagation algorithm.

[0099] 5. Repeat steps 2-4.

[0100] Using a target network in this way makes the calculation of the target Q-value more stable, avoiding the instability problem of the Q-network target. At the same time, the parameters of the target network are not updated as frequently as the main network, reducing the instability caused by network parameter changes and making training more stable.

[0101] S23. To train the neural network, an experience replay cache is used to store the previous state, action, reward, and next state (s t a t r t s t+1 During training, a batch (s) is randomly selected from the experience replay cache. t at r t s t+1 ), and use them to train neural networks.

[0102] Specifically, the following loss function is used:

[0103] L(θ)=E (s，a，r，s′) [(yQ(s,a;θ)) 2 ]

[0104] Where y = r + γmax a′ Q(s′,a′;θ-) is the target Q value, where r is the immediate reward, γ is the discount factor, s′ is the next state, a′ is the next action, and θ- is the target network parameter used to calculate the target value.

[0105] During training, an ∈-greedy policy is used to select actions. Specifically, an action is randomly selected with probability ∈, and the action with the largest Q value is selected with probability 1-∈. As training progresses, the value of ∈ is gradually decreased to explore the optimal policy.

[0106] S24. Repeat steps S21 to S23 multiple times until the shaft hole reaches the target depth, or the number of iterations exceeds the limit. Different reward functions, neural network structures, and hyperparameters can also be tested to improve model performance.

[0107] Step 3: Perform the assembly actions described above to complete the shaft hole assembly, and record the state when the shaft hole assembly is completed. This state will be used as the input for the next state, selecting the next assembly action. Specifically, execute the selected action and observe the changes in the environmental state, including information such as position, velocity, acceleration, force feedback magnitude and direction, and torque. After executing the selected action, the environmental state will change, including changes in the position, velocity, and acceleration of the robotic arm's end effector, as well as information such as the magnitude and direction of force and torque measured by the force sensor. This information can be used as the input for the next state, in order to select the next action.

[0108] In this shaft-hole assembly method based on deep reinforcement learning, deep networks are used to handle high-dimensional, nonlinear state and action spaces, enabling better simulation of complex assembly tasks and improving the robot's assembly accuracy and efficiency. An experience replay buffer reduces correlation between samples, increases the diversity and efficiency of training samples, and allows for the reuse of historical experience, improving training stability. A target network reduces instability and volatility during deep neural network training, while also increasing learning efficiency and speed. In summary, this shaft-hole assembly method can reduce the difficulty and cost of manual assembly, improve production efficiency and accuracy, and has practical and commercial value.

[0109] The preferred embodiments of the present invention have been described in detail above. It should be understood that those skilled in the art can make numerous modifications and variations based on the concept of the present invention without creative effort. Therefore, all technical solutions that can be obtained by those skilled in the art based on the concept of the present invention through logical analysis, reasoning, or limited experimentation on the basis of existing technology should be within the scope of protection defined by the claims.

Claims

1. A method for assembling shaft holes in an industrial robot based on deep reinforcement learning, characterized in that, The method includes the following steps: S1: Construct the state set and motion set of the industrial robot: The state set is specifically a set of states at the end of the robotic arm, including force, torque, velocity, acceleration, and position information of the shaft hole at the end of the robotic arm; the action set includes the magnitude and direction information of the displacement during assembly and insertion. A specific state in the state set includes force and torque, velocity and acceleration of the robotic arm's end effector, and position information of the shaft hole. A state is represented as... : The forces in the three directions are respectively The torques in the three directions are respectively The terminal velocity is acceleration is The position of the shaft hole is ; One action in the action set is represented as: : Wherein, the magnitude of the displacement is The direction vector is ; S2: Input the aforementioned state set and action set into the trained deep reinforcement learning-based shaft hole assembly algorithm model, and output the selected action; The training process of the shaft-hole assembly algorithm model based on deep reinforcement learning specifically includes the following steps: S21. Input the constructed state into the shaft-hole assembly algorithm model, and construct a reward function for the shaft-hole assembly algorithm model. For the input state, the reward function provides an immediate reward. feedback; The reward function R for: in, Indicates the proximity to the target location. This indicates the continuity of force feedback changes. Indicates the smoothness of the movement; The proximity to the target location is determined by calculating the Euclidean distance between the current location and the target location, that is: in, It is a weighting factor for position rewards. This is the current location. It is the target location. It is a scaling factor used to control the size of the location reward; The continuity of force feedback change is characterized by calculating the difference between the current force feedback and the force feedback at the previous moment, that is: in, It is the weighting factor for force feedback rewards. It is the force feedback at the current moment. It is the force feedback from the previous moment; The smoothness of a motion is characterized by calculating the difference between the current motion and the previous motion, i.e.: in, It is a weighting factor for action rewards. It refers to the action at the current moment. It refers to the action taken at the previous moment. It is the standard deviation coefficient, used to control the magnitude of the action reward; S22. Under the input state, the DQN algorithm is used to learn to select the corresponding action, including at each time step, taking the current state as input and using a deep neural network to output the Q value of each action; S23. Use an experience replay cache to store previous states, actions, rewards, and the next state. Training a deep neural network, during the training process, according to A greedy strategy selects the action that maximizes the Q value. S24. Repeat steps S21 to S23 until the shaft hole reaches the target depth, or the number of iterations exceeds the limit. Based on the input state, output the action with the maximum Q value selected. S3: Perform the aforementioned actions to complete the shaft hole assembly, record the state when the shaft hole assembly is completed, use this state as the input for the next state, and select the next action.

2. The industrial robot shaft hole assembly method based on deep reinforcement learning according to claim 1, characterized in that, The state set and action set of the industrial robot in S1 are preprocessed, including data filtering, static force removal and normalization.

3. The industrial robot shaft hole assembly method based on deep reinforcement learning according to claim 1, characterized in that, In S23, the DQN algorithm is used to learn and select the corresponding action, which is achieved by combining two methods: experience replay and a combination of the main network and the target network.

4. The industrial robot shaft hole assembly method based on deep reinforcement learning according to claim 1, characterized in that, According to the S23 The actions that select the maximum Q value using a greedy strategy specifically include: The probability of randomly selecting an action gradually decreases. The value, in The probability of choosing the action with the maximum Q value.

5. An industrial robot shaft hole assembly device based on deep reinforcement learning, characterized in that, The device includes the following components: a CRRC6 industrial six-axis robot, a six-dimensional force sensor, and a six-axis robot host computer, and is used to implement the method described in any one of claims 1-4.

6. An industrial robot shaft hole assembly device based on deep reinforcement learning, characterized in that, The device includes the following components: a base, a shaft, a hole, a force sensor connector, a shaft-hole connector, and a motion control console. The device is used to implement the method described in any one of claims 1-4.

Images