Robot tracking control learning method based on postmortem screening experience playback

Abstract

The invention relates to the field of mechanical arm trajectory tracking, in particular to a robot tracking control learning method based on postmortem screening experience playback, which comprises the following steps: initializing target network parameters and an experience pool, and storing a state transition tuple in the experience pool; when the number of the state transition tuples in the experience pool is larger than the preset number of the state transition tuples, the action complex state is determined according to the track smoothness and the node covariance degree of the robot mechanical arm; determining an experience pool optimization mode according to the complex state of the action; selecting experience of which the estimated position deviation is greater than or equal to the standard estimated position deviation as screening experience, and determining an experience selection mode according to the number of the screening experience; updating the cr-it ic network and the actor network, and carrying out soft updating on the target network of the cr-it ic network and the target network of the actor network respectively; and according to the reward evaluation value, soft update weights of the target network of the cr-it-ic network and the target network of the actor network are adjusted. According to the method, the tracking effect and the tracking stability of the model are improved.

Description

Technical Field

[0001] The present invention relates to the field of robot arm trajectory tracking, and in particular to a robot tracking control learning method based on post-screening experience playback. Background Art

[0002] In the trajectory tracking of the robotic arm, when the intelligent agent is trained through the deep reinforcement learning control algorithm, the learning efficiency of the intelligent agent is low due to defects such as long training time and low sample utilization. Therefore, experience replay is often used to improve the sample utilization rate, but inappropriate reward methods will cause deviations in the learning of the intelligent agent, resulting in poor tracking effect and tracking stability of the trained model. Therefore, how to improve the tracking effect and tracking stability of the trained model is a technical problem that needs to be urgently solved by technical personnel in this field.

[0003] Patent publication number CN118493388A discloses a deep reinforcement learning robotic arm grasping method for sparse rewards, including: first, analyzing the characteristics of the robotic arm grasping task, modeling it as a Markov decision problem, designing binary sparse rewards, reducing the complexity of reward function design, and reducing design costs; secondly, using the DDPG algorithm as the main deep reinforcement learning training algorithm framework, building an Actor-Critic structure network, and processing the continuous state action space; then, designing a post-experience playback mechanism, using the G-HGG algorithm for auxiliary target generation, using the pre-trained action network for action screening and adding exploration noise and energy functions to process the accumulated experience pool, enhancing experience utilization, and improving training efficiency and grasping success rate; finally, building a robotic arm model and scene information, using interactive data for optimized training, and realizing robotic arm target grasping. It can be seen that the above technical solution has the following problems: the reward method is single when replaying experience, and the reward method cannot be adaptively adjusted according to the actual state of the robotic arm, and only the next state is set as the tracking target for reward, resulting in the reward function being unable to accurately describe the action when the tracking task is highly complex, resulting in poor tracking effect and tracking stability of the trained model. Summary of the invention

[0004] To this end, the present invention provides a robot tracking control learning method based on post-screening experience replay, so as to overcome the problems in the prior art that the reward method is single when performing experience replay, the reward method cannot be adaptively adjusted according to the actual state of the robot arm, and only the next state is set as the tracking target for reward, resulting in the reward function being unable to accurately describe the action when the tracking task is highly complex, resulting in poor tracking effect and tracking stability of the trained model.

[0005] To achieve the above object, the present invention provides a robot tracking control learning method based on post-screening experience playback, comprising:

[0006] Initialize the target network parameters and experience pool, and store the state transition tuple in the experience pool;

[0007] When the number of state transition tuples in the experience pool is greater than the preset number of state transition tuples, the complex state of the action is determined according to the trajectory smoothness and node covariance of the robot manipulator;

[0008] The optimization method of determining the experience pool according to the complex state of the action is to determine the reward value of the pre-updated experience in the experience pool according to the position error and the estimated position deviation, or to determine the reward value of the pre-updated experience in the experience pool according to the estimated position deviation;

[0009] Select the experience whose estimated position deviation is greater than or equal to the standard estimated position deviation as the screening experience, and determine the experience selection method to select the screening experience for the experience combination or reference sequence according to the number of screening experiences, and record the screening experience selected by the experience selection method as the pre-update experience;

[0010] Update the critical network and actor network, and perform soft updates on the target networks of the critical network and actor network respectively;

[0011] The soft update weights of the target network of the critical network and the target network of the actor network are adjusted according to the reward evaluation value.

[0012] Furthermore, the complex state of the action includes:

[0013] A first action complexity state in which the trajectory smoothness is within a first preset trajectory smoothness range or the node covariance is within a first preset node covariance range;

[0014] A second action complexity state in which the trajectory smoothness is within a second preset trajectory smoothness range and the node covariance is within a second preset node covariance range;

[0015] Among them, all values ​​within the first preset trajectory smoothness range are greater than all values ​​within the second preset trajectory smoothness range, and all values ​​within the first preset node covariance range are greater than all values ​​within the second preset node covariance range.

[0016] Furthermore, the experience pool optimization method is determined according to the complexity of the action, wherein:

[0017] In the first complex action state, the experience pool optimization method is to determine the reward value of the pre-updated experience in the experience pool according to the position error and the estimated position deviation;

[0018] In the second complex action state, the experience pool optimization method is to determine the reward value of the pre-updated experience in the experience pool according to the estimated position deviation.

[0019] Further, the state transition tuples whose estimated position deviation is greater than or equal to the standard estimated position deviation are selected as screening experience;

[0020] The estimated position deviation at time t is determined based on the difference between the position error at time t and the position error at time t+1.

[0021] Furthermore, the experience selection method is determined according to the number of screening experiences, wherein:

[0022] If the number of screening experiences is greater than or equal to the preset number of screening experiences, screening experiences are selected for the experience combination;

[0023] If the number of screening experiences is less than the preset number of screening experiences, screening experience selection is performed for the reference sequence.

[0024] Furthermore, experience selection is performed based on experience combinations, including:

[0025] Perform combination analysis on each screening experience according to the reference order. When performing combination analysis on a single screening experience, record the screening experience as the target screening experience, and record other screening experiences other than the target screening experience as reference screening experiences. Detect the reward evaluation deviation between each reference screening experience after the reference order of the target screening experience and the target screening experience in turn until the reward evaluation deviation between the reference screening experience and the target screening experience is greater than the preset reward evaluation deviation, then stop the detection, and record the set of each screening experience before the reference order of the reference screening experience whose reward evaluation deviation with the target screening experience is greater than the preset reward evaluation deviation as an experience combination, and continue to perform combination analysis on the screening experiences that are not recorded as experience combinations until each screening experience is recorded as an experience combination, then stop the combination analysis;

[0026] Screening experience selection is performed for each experience combination. When screening experience selection is performed for a single experience combination, a preset proportion of screening experiences in the experience combination is randomly selected, and each selected screening experience is used as a pre-update experience.

[0027] Furthermore, the reference sequence is screened and selected empirically, including:

[0028] The sequence after sorting the screening experiences according to the reference order is recorded as the reference sequence, the reference sequence is divided into M equal parts, and the screening experiences at each equal part are selected as pre-update experiences.

[0029] Furthermore, the critic network is updated;

[0030] Use update formula (1) to update the critic network loss value L Q (θ) is updated, and formula (1) is:

[0031]

[0032] Where N is the number of experiences randomly selected for updating after determining the reward value of the pre-updated experience in the experience pool, θ is the initialization parameter of the critic network, and i is 1, 2, 3, ..., N, s i is the current state of the i-th experience used for updating, which includes the actual position, actual speed, target position and target speed of the robot arm, a i is the current action of the i-th experience used for updating, the action is the torque output by the robot to the joint angle, Q θ (s i , a i ) is the s of the critic network with initialization parameter θ for the i-th experience used for updating i and a i The prediction score, r i is the reward value of the i-th experience used for updating, s′ i is the next state of the i-th experience used for updating, y(r i , s′ i ) is determined by formula (2), which is:

[0033] y(r i , s′ i )=r+γQ θ′ (s′ i , π Φ′ (s′ i )) (2),

[0034] Use stochastic gradient descent to update the critic network. The update formula is formula (3):

[0035]

[0036] Among them, r is the reward value, γ is the discount factor, θ′ is the target network parameter of the critic network, π is the actor network to be trained, Ф′ is the target network parameter of the actor network, π Ф' (s′ i ) is the target network parameter Φ′ of the actor network for the i-th experience used for updating s′ i The predicted action, Q θ' (s′ i , π Ф′ (s′ i )) is the target network parameter θ′ of the critic network for the i-th experience s′ used for updating i and πФ' (s′ i )’s prediction score.

[0037] Furthermore, the actor network is updated using stochastic gradient descent, and the update formula is formula (4):

[0038]

[0039] Where a is equal to π Ф (s i ), Ф is the initialization parameter of the actor network, π Ф (s i ) is the s of the actor network with initialization parameter Ф for the i-th experience used for updating i Predicted actions.

[0040] Furthermore, the target network of the critic network is soft updated using formula (5), and the target network of the actor network is soft updated using formula (6).

[0041] θ′=αθ+(1-α)θ′ (5),

[0042] Ф′=αФ+(1-α)Ф′ (6),

[0043] Among them, α is the soft update weight, α is greater than 0 and less than 1, and the soft update weights of the target network of the critic network and the target network of the actor network are adjusted according to the reward evaluation value;

[0044] If the reward evaluation value is within the first preset reward evaluation value range, the soft update weight is adjusted to decrease;

[0045] If the reward evaluation value is within the second preset reward evaluation value range, the soft update weight is increased and adjusted;

[0046] Wherein, each value within the first preset reward evaluation value range is greater than each value within the second preset reward evaluation value range;

[0047] The reward evaluation value is determined according to a reward fluctuation coefficient and a reward mean coefficient. The reward fluctuation coefficient is positively correlated with the reward evaluation value, and the reward mean coefficient is positively correlated with the reward evaluation value.

[0048] Compared with the prior art, the beneficial effect of the present invention lies in that, in the technical scheme of the present invention, the complexity of the action is determined according to the trajectory smoothness and the node covariance, and the complexity of the robot arm action is effectively reflected by the trajectory smoothness and the node covariance, and then different experience pool optimization methods are adaptively selected, so that the selection of the experience pool optimization method is more in line with the actual application scenario, avoiding the problem in the prior art that the reward method cannot be adaptively adjusted according to the actual state of the robot arm, and only the next state is set as the tracking target for reward, reducing the intelligent agent's exploration of meaningless actions, and then being able to accurately describe the action when the complexity of the tracking task is high, thereby improving the tracking effect and tracking stability of the trained model.

[0049] Furthermore, the present invention determines the reward value of the pre-updated experience in the experience pool based on the position error and the estimated position deviation, which is conducive to positive rewards for correct actions, thereby enhancing the logic of reward feedback and output actions, reducing the intelligent agent's exploration of meaningless actions, and enabling the intelligent agent to automatically adapt to changes in the environment during the learning process, thereby being able to accurately describe actions when the complexity of the tracking task is high; determining the reward value based on the estimated position deviation is beneficial when the complexity of the tracking task is low, which is conducive to reducing the computational burden while ensuring tracking efficiency, thereby optimizing the model's accuracy and stability in tracking the robot arm's trajectory.

[0050] Furthermore, the present invention selects the experience in which the estimated position deviation is greater than or equal to the standard estimated position deviation as the screening experience, and effectively reflects through the estimated position deviation that the state transition tuple is beneficial to the intelligent body to reach the end point, and then screens out the state transition tuple that is beneficial to completing the goal for reward, avoiding the problem of inaccurate state description due to rewards, and thus making the state transition tuple with guiding significance have a higher reward return, avoiding the problem that in case the selected state transition tuple cannot guide the intelligent body to reach the tracking target, causing the deep reinforcement learning control algorithm to fail to obtain effective experience for a long time and fail to converge, or even have no positive convergence trend, thereby improving the accuracy of the model in tracking the trajectory of the robotic arm.

[0051] Furthermore, in the present invention, the experience selection method is determined according to the number of screening experiences to select screening experiences for experience combinations or reference sequences. The number of screening experiences effectively reflects the richness of the screening experience for post-experience playback, and then different experience selection methods are adaptively selected, so that the choice of experience selection method is more in line with the actual application scenario, and can comprehensively consider the correlation and difference between the screening experiences, thereby ensuring the diversity of the screening experience, ensuring that the selected screening experience is more in line with actual needs, and thereby improving training efficiency and training accuracy.

[0052] Furthermore, in the present invention, the soft update weights of the target network of the critical network and the target network of the actor network are adjusted according to the reward evaluation value, and the beneficial degree of model learning is effectively reflected through the reward evaluation value, so that the model can adaptively adjust the soft update weights of the target network of the critical network and the target network of the actor network according to the actual situation, guide the model to adjust its learning strategy to adapt to the new environment, and then guide the model to pay more attention to behaviors that are conducive to obtaining high rewards when updating parameters, which helps the model gradually approach the optimal strategy during the training process, thereby improving the tracking effect and tracking stability of the trained model. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] Figure 1 It is a schematic diagram of a robot tracking control learning method based on post-screening experience playback of the present invention;

[0054] Figure 2 This is a flow chart of the present invention for determining the complex state of an action according to trajectory smoothness and node covariance;

[0055] Figure 3 This is a flow chart of the present invention for determining an optimization method for the experience pool according to a complex state of an action;

[0056] Figure 4 The present invention is a flow chart of determining the experience selection method according to the number of screening experiences. DETAILED DESCRIPTION

[0057] In order to make the objects and advantages of the present invention more clearly understood, the present invention is further described below in conjunction with embodiments; it should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0058] The preferred embodiments of the present invention are described below with reference to the accompanying drawings. It should be understood by those skilled in the art that these embodiments are only used to explain the technical principles of the present invention and are not intended to limit the protection scope of the present invention.

[0059] It should be noted that, in the description of the present invention, terms such as "up", "down", "left", "right", "inside" and "outside" indicating directions or positional relationships are based on the directions or positional relationships shown in the drawings. This is merely for the convenience of description and does not indicate or imply that the device or element must have a specific orientation, be constructed and operated in a specific orientation. Therefore, it cannot be understood as a limitation on the present invention.

[0060] In addition, it should be noted that in the description of the present invention, unless otherwise clearly specified and limited, the terms "installed", "connected", and "connected" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or it can be indirectly connected through an intermediate medium, or it can be the internal communication of two components. For those skilled in the art, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0061] See also Figures 1 to 4 As shown, the present invention provides a robot tracking control learning method based on post-screening experience playback, comprising:

[0062] Initialize the target network parameters and experience pool, and store the state transition tuple in the experience pool;

[0063] When the number of state transition tuples in the experience pool is greater than the preset number of state transition tuples, the complex state of the action is determined according to the trajectory smoothness and node covariance of the robot manipulator;

[0064] The optimization method of determining the experience pool according to the complex state of the action is to determine the reward value of the pre-updated experience in the experience pool according to the position error and the estimated position deviation, or to determine the reward value of the pre-updated experience in the experience pool according to the estimated position deviation;

[0065] Select the experience whose estimated position deviation is greater than or equal to the standard estimated position deviation as the screening experience, and determine the experience selection method to select the screening experience for the experience combination or reference sequence according to the number of screening experiences, and record the screening experience selected by the experience selection method as the pre-update experience;

[0066] Update the critic network and actor network, and perform soft updates on the target network of the critic network and actor network respectively;

[0067] The soft update weights of the target network of the critic network and the target network of the actor network are adjusted according to the reward evaluation value.

[0068] The application scenario of the present invention is robot arm trajectory tracking. The present invention adopts DDPG deep reinforcement learning algorithm for learning. The DDPG deep reinforcement learning algorithm includes an actor network, a target network of the actor network, a critic network, a target network of the critic network and a reward function. The actor network is used to output the action of the robot arm, and the critic network is used to score the action output by the actor network and predict the impact of the action on the environment. The target networks of the actor network and the critic network are both updated using soft updates, which is easy for technicians in this field to understand and will not be elaborated on in detail.

[0069] Initialize the target network parameters and experience pool, and store the state transfer tuple in the experience pool, including: using random network parameters θ and Ф to initialize the critic network Q θ and actor network π Ф , initialize the target network parameters θ′=θ, Ф′=Ф, the initialization parameters Ф of the actor network and the initialization parameters θ of the critic network, and sample the weight values ​​from the normal distribution or the uniform distribution, and initialize the bias to zero or a small constant. These weights and biases will constitute the initial parameters of the comment network and are initialized for the experience pool. Use the actor network output action to input the environment to get the next state, and store the current state transfer tuple (s, a, r, s′) as experience in the experience pool, where s is the current state of the experience, a is the current action of the experience, r is the reward value, and s′ is the next state of the experience. If s′ is the end state, the environment state is reset. The end state is that the actual position of the robot arm is the same as the target position and the actual speed is the same as the target speed. This is easy for technicians in this field to understand and will not be described in detail.

[0070] The number of state transition tuples is the total number of state transition tuples currently stored in the experience pool. The value of the preset number of state transition tuples can be determined by the user according to the actual application scenario. The greater the user's demand for improving the accuracy of the model, the larger the value of the preset number of state transition tuples. A value of the preset number of state transition tuples is provided, and the average value of the number of state transition tuples corresponding to the historical records with qualified marks whose accuracy can meet the user's needs when screening experience selection is performed is recorded as the preset number of state transition tuples.

[0071] The historical records in the present invention include but are not limited to the number of state transition tuples, trajectory smoothness, node covariance, number of screening experiences and reward evaluation value during the historical model training process, and each historical record corresponds to a qualified mark, which records that the tracking effect and tracking stability of the trained model can meet user needs.

[0072] Specifically, the complex states of actions include:

[0073] A first action complexity state in which the trajectory smoothness is within a first preset trajectory smoothness range or the node covariance is within a first preset node covariance range;

[0074] A second action complexity state in which the trajectory smoothness is within a second preset trajectory smoothness range and the node covariance is within a second preset node covariance range;

[0075] Among them, all values ​​within the first preset trajectory smoothness range are greater than all values ​​within the second preset trajectory smoothness range, and all values ​​within the first preset node covariance range are greater than all values ​​within the second preset node covariance range.

[0076] Among them, the present invention is provided with a continuous cycle monitoring cycle, and the complex state of the action is determined once at the end of each monitoring cycle. The length of the monitoring cycle can be set according to the needs of the user. The greater the user's demand for the accuracy of complex state monitoring, the shorter the monitoring cycle length is. A value of the monitoring cycle length is provided, and the monitoring cycle length is 30 minutes.

[0077] The method for confirming the trajectory smoothness is to detect the moving trajectory of any point at the end of the robot's mechanical arm that is currently performing tracking control learning within the current monitoring cycle, record the point as the target end point, and establish a three-dimensional rectangular coordinate system with the position of the target end point as the origin at the end of the monitoring cycle. The moving trajectory is divided into B equal parts, and the initial position and terminal position of each equal point and the moving trajectory are recorded as reference points. The initial position is the position of the end effector of the mechanical part at the beginning of the current monitoring cycle, and the terminal position is the position of the end effector of the mechanical part at the end of the current monitoring cycle. A vector tangent to the moving trajectory is drawn at each reference point. The direction of the vector is the same as the moving direction. The vector corresponding to the jth reference point is j=1, 2, 3, ..., B+2, calculate the angle corresponding to each trajectory segment, the trajectory segment is the moving trajectory between the jth reference point and the j+1th reference point, the calculation formula of the angle ε between the vector corresponding to the jth reference point and the vector corresponding to the j+1th reference point is formula (7),

[0078]

[0079] According to the moving direction of the moving trajectory, smooth detection is performed on each angle. When smooth detection is performed on a single angle, the angle is recorded as the target angle. The angle change value of the angle adjacent to the target angle after the moving direction of the moving trajectory is compared with the target angle. If the angle change value is less than the preset angle change value, the target angle is recorded as the smooth angle, and smooth detection is continued for the angles that have not been smoothed until smooth detection is completed for each angle, then the smooth detection is stopped; the number of smooth angles is recorded as the trajectory smoothness;

[0080] The value of the preset angle change value can be determined by the user according to the actual application scenario. It can be understood that the larger the value of the preset angle change value, the greater the fluctuation of the movement trajectory of the robot arm. A preset angle change value is provided, and the preset angle change value is 30°.

[0081] The end of the robotic arm is the end that is in direct contact with the object. The end of the robotic arm includes but is not limited to a gripper, a suction cup, a welding gun, and a drill. The value of B can be determined by the user according to the actual application scenario. It can be understood that the larger the value of B, the greater the measurement accuracy of the trajectory smoothness. A value of B is provided, and B is 50.

[0082] The node covariance is the number of nodes that change simultaneously in a single monitoring cycle; the node is the point connecting various components or joints in the robotic arm, which is easy for technicians in this field to understand and will not be described in detail.

[0083] All values ​​within the first preset trajectory smoothness range are greater than or equal to the preset trajectory smoothness, all values ​​within the second preset trajectory smoothness range are less than the preset trajectory smoothness, all values ​​within the first preset node covariance range are greater than or equal to the preset node covariance, and all values ​​within the second preset node covariance range are less than the preset node covariance.

[0084] The values ​​of the preset trajectory smoothness and the preset node covariance can be determined by the user according to the actual application scenario. It can be understood that the smaller the value of the preset trajectory smoothness and the larger the value of the preset node covariance, the more complex the robot arm movement is. A value of the preset trajectory smoothness and the preset node covariance is provided, and the experience pool optimization method is to determine the historical records of the reward value according to the estimated position deviation, and the average value of the trajectory smoothness corresponding to the historical records that can meet the user's needs is recorded as the preset trajectory smoothness, and the average value of the node covariance corresponding to the historical records that can meet the user's needs is recorded as the preset node covariance.

[0085] Specifically, the experience pool optimization method is determined according to the complexity of the action, where:

[0086] In the first complex action state, the experience pool optimization method is to determine the reward value of the pre-updated experience in the experience pool according to the position error and the estimated position deviation;

[0087] In the second complex action state, the experience pool optimization method is to determine the reward value of the pre-updated experience in the experience pool according to the estimated position deviation.

[0088] Wherein, time t is the time corresponding to any state transition tuple. At time t, the position error at time t = the actual position at time t - the target position at time t;

[0089] In the first complex action state, the reward value at time t = estimated coefficient × estimated position deviation at time t + position coefficient × position error at time t; the values ​​of the estimated coefficient and the position coefficient can be obtained by the user through deep learning convolutional neural network learning based on historical records. It can be understood that the present invention reflects the feedback obtained by the intelligent communication from the environment after taking a certain action through the reward value to judge the degree of guidance of the current action on the tracking effect. The user can use historical records for deep learning to determine the influence of the estimated position deviation and position error on the degree of guidance of the trajectory tracking effect, and then select the values ​​of the estimated coefficient and the position coefficient accordingly, and provide a value of the estimated coefficient and the position coefficient, with the estimated coefficient being 0.5 and the position coefficient being 0.5.

[0090] In the second complex action state, the reward value at time t is positively correlated with the estimated position deviation at time t.

[0091] Specifically, the state transition tuples whose estimated position deviation is greater than or equal to the standard estimated position deviation are selected as screening experience;

[0092] The estimated position deviation at time t is determined based on the difference between the position error at time t and the position error at time t+1.

[0093] Among them, the standard estimated position deviation is 0, time t is the time corresponding to any state transition tuple, and the estimated position deviation at time t = position error at time t - (position error at time t+1).

[0094] Specifically, the experience selection method is determined according to the number of screening experiences, wherein:

[0095] If the number of screening experiences is greater than or equal to the preset number of screening experiences, screening experiences are selected for the experience combination;

[0096] If the number of screening experiences is less than the preset number of screening experiences, screening experience selection is performed for the reference sequence.

[0097] Among them, the number of screening experiences is the total amount of screening experiences. The value of the preset number of screening experiences can be determined according to the actual application scenario. The greater the user's demand for improving the accuracy of the model, the larger the value of the preset number of screening experiences. A value of the preset number of screening experiences is provided, and the average value of the number of screening experiences corresponding to the historical records whose accuracy can meet the user's needs and are marked as qualified when selecting screening experiences is recorded as the preset number of screening experiences.

[0098] Specifically, experience selection is performed based on experience combinations, including:

[0099] Perform combination analysis on each screening experience according to the reference order. When performing combination analysis on a single screening experience, record the screening experience as the target screening experience, and record other screening experiences other than the target screening experience as reference screening experiences. Detect the reward evaluation deviation between each reference screening experience after the reference order of the target screening experience and the target screening experience in turn until the reward evaluation deviation between the reference screening experience and the target screening experience is greater than the preset reward evaluation deviation, then stop the detection, and record the set of each screening experience before the reference order of the reference screening experience whose reward evaluation deviation with the target screening experience is greater than the preset reward evaluation deviation as an experience combination, and continue to perform combination analysis on the screening experiences that are not recorded as experience combinations until each screening experience is recorded as an experience combination, then stop the combination analysis;

[0100] Screening experience selection is performed for each experience combination. When screening experience selection is performed for a single experience combination, a preset proportion of screening experiences in the experience combination is randomly selected, and each selected screening experience is used as a pre-update experience.

[0101] The method for confirming the reference order is as follows: it can be understood that the screening experience is a state transition tuple corresponding to a certain time, each screening experience corresponds to a different time, and the time corresponding to the screening experience corresponds to a position in the moving trajectory. The reference order is the time sequence of reaching each position in the moving trajectory;

[0102] The moving trajectory is the moving trajectory of any point at the end of the robot arm currently undergoing tracking control learning within the current monitoring cycle;

[0103] The value of the preset ratio can be determined by the user according to the actual application scenario. The higher the accuracy requirement of the training model, the larger the value of the preset ratio. A preset ratio value is provided, and the preset ratio is 20%;

[0104] The confirmation method after the reference sequence and before the reference sequence is that, for a screening experience, the screening experience is recorded as the target experience, the position of the time corresponding to the target experience in the moving trajectory is detected, and the position is recorded as the target position. The screening experience corresponding to the time of each position after the moving trajectory reaches the target position is after the reference sequence, and the screening experience corresponding to the time of each position before the moving trajectory reaches the target position is before the reference sequence.

[0105] The reward evaluation deviation is the absolute value of the difference between the reward values ​​of two screening experiences. The value of the preset reward evaluation deviation can be determined by the user according to the actual application scenario. It can be understood that the larger the value of the preset reward evaluation deviation, the greater the difference between the two screening experiences. A value of the preset reward evaluation deviation is provided. The detection experience extraction method is to perform historical records of screening experience extraction on experience combinations, and the average value of the reference deviations corresponding to each experience combination corresponding to the historical records that can meet user needs is recorded as the preset reward evaluation deviation. The reference deviation is the reward evaluation deviation corresponding to any two screening experiences in an experience combination.

[0106] Specifically, the reference sequence is screened for empirical selection, including:

[0107] The sequence after sorting the screening experiences according to the reference order is recorded as the reference sequence, the reference sequence is divided into M equal parts, and the screening experiences at each equal part are selected as pre-update experiences.

[0108] The value of M can be determined by the user according to actual needs. Here, a value of M is provided, which is 50.

[0109] Specifically, the critic network is updated;

[0110] Use update formula (1) to update the critic network loss value L Q (θ) is updated, and formula (1) is:

[0111]

[0112] Where N is the number of experiences randomly selected for updating after determining the reward value of the pre-updated experience in the experience pool, θ is the initialization parameter of the critic network, and i is 1, 2, 3, ..., N, s i is the current state of the i-th experience used for updating, which includes the actual position, actual speed, target position and target speed of the robot arm, a i is the current action of the i-th experience used for updating, the action is the torque output by the robot to the joint angle, Q θ (s i , a i) is the s of the critic network with initialization parameter θ for the i-th experience used for updating i and a i The prediction score, r i is the reward value of the i-th experience used for updating, s′ i is the next state of the i-th experience used for updating, y(r i , s′ i ) is determined by formula (2), which is:

[0113] y(r i , s′ i )=r+γQ θ′ (s′ i ,π Φ′ (s′ i )) (2), use stochastic gradient descent to update the critic network, and the update formula is formula (3),

[0114]

[0115] Among them, r is the reward value, γ is the discount factor, θ′ is the target network parameter of the critic network, π is the actor network to be trained, Ф′ is the target network parameter of the actor network, π Ф' (s′ i ) is the target network parameter Φ′ of the actor network for the i-th experience used for updating s′ i The predicted action, Q θ' (s′ i , π Ф' (s′ i )) is the target network parameter θ′ of the critic network for the i-th experience s′ used for updating i and π Ф' (s′ i )’s prediction score.

[0116] Among them, N is the number of experiences for updating that are randomly selected after determining the reward value of the pre-update experience in the experience pool. It can be understood that after optimizing the reward value of the pre-update experience according to different experience pool optimization methods, each state transition tuple of the experience pool is recorded as optimized experience, and N optimized experiences are randomly selected from the experience pool. These N optimized experiences are the experiences for updating. The value of N can be determined by the user according to the actual application scenario. The greater the accuracy of the model training required by the user, the larger the value of N. A value of N is provided, which is 40% of the number of experiences used for updating. Q is the critic network that needs to be trained; the loss value of the critic network is expressed by L Q (θ) is used to represent LQ (θ) is used to measure the Q predicted by the critic network θ (s i , a i ) and the target Q value of the i-th experience used for updating, y(r i , s′ i ) is the target Q value of the i-th experience used for updating, is the gradient of the loss function L(θ) with respect to the parameter θ, and the loss function L(θ) is Q θ (s i , a i ) and y(r i , s′ i ), where the calculation formula of L(θ) is the same as L Q (θ), which is easily understood by those skilled in the art and will not be described in detail.

[0117] Specifically, updates are made to the actor network;

[0118] Among them, the actor network is updated using stochastic gradient descent, and the update formula is formula (4):

[0119]

[0120] Where a is equal to π Ф (s i ), Ф is the initialization parameter of the actor network, π Ф (s i ) is the s of the actor network with initialization parameter Ф for the i-th experience used for updating i Predicted actions.

[0121] in, is the policy gradient of the actor network during parameter update, is π Ф (s i ) with respect to the parameters, Q θ (s i , a) Gradient with respect to parameter a.

[0122] Specifically, the target network of the critic network is soft updated using formula (5), and the target network of the actor network is soft updated using formula (6).

[0123] θ′=αθ+(1-α)θ′ (5),

[0124] Φ′=αΦ+(1-α)Φ′ (6),

[0125] Among them, α is the soft update weight, α is greater than 0 and less than 1, and the soft update weights of the target network of the critic network and the target network of the actor network are adjusted according to the reward evaluation value;

[0126] If the reward evaluation value is within the first preset reward evaluation value range, the soft update weight is adjusted to decrease;

[0127] If the reward evaluation value is within the second preset reward evaluation value range, the soft update weight is increased and adjusted;

[0128] Wherein, each value within the first preset reward evaluation value range is greater than each value within the second preset reward evaluation value range;

[0129] The reward evaluation value is determined according to a reward fluctuation coefficient and a reward mean coefficient. The reward fluctuation coefficient is positively correlated with the reward evaluation value, and the reward mean coefficient is positively correlated with the reward evaluation value.

[0130] Among them, the soft update weight is the proportion of updating the target network of the critic network or the target network of the actor network. The value of α directly affects the stability and performance of the reinforcement learning algorithm. Users can make preliminary settings based on experience and adjust the soft update weights of the target network of the critic network and the target network of the actor network according to the reward evaluation value in the subsequent training process to ensure the stability of the reinforcement learning algorithm. α can be initially set to 0.05;

[0131] Reward evaluation value = reward fluctuation coefficient + reward mean coefficient. The reward fluctuation coefficient is positively correlated with the reward fluctuation value, and the reward mean coefficient is positively correlated with the reward mean. The reward fluctuation value is the difference between the maximum reward value corresponding to each pre-update experience and the minimum reward value corresponding to each pre-update experience. The reward mean is the average of the reward values ​​corresponding to each pre-update experience.

[0132] All values ​​within the first preset reward evaluation value range are greater than the first preset reward evaluation value, and all values ​​within the second preset reward evaluation value range are less than the second preset reward evaluation value, wherein the first preset reward evaluation value is greater than the second preset reward evaluation value;

[0133] The values ​​of the first preset reward evaluation value and the second preset reward evaluation value can be determined according to the actual application scenario. It can be understood that the reward evaluation value reflects the adaptability of the model's learning strategy to the environment. The smaller the value of the first preset reward evaluation value and the larger the value of the second preset reward evaluation value, the greater the adjustment intensity of the reinforcement learning algorithm, and thus the greater the adaptability of the model's learning strategy to the environment. By improving the values ​​of the first preset reward evaluation value and the second preset reward evaluation value, the maximum value of the reward evaluation value corresponding to the historical record that can meet the user's needs is recorded as the first preset reward evaluation value, and the minimum value of the reward evaluation value corresponding to the historical record that can meet the user's needs is recorded as the second preset reward evaluation value.

[0134] So far, the technical solutions of the present invention have been described in conjunction with the preferred embodiments shown in the accompanying drawings. However, it is easy for those skilled in the art to understand that the protection scope of the present invention is obviously not limited to these specific embodiments. Without departing from the principle of the present invention, those skilled in the art can make equivalent changes or substitutions to the relevant technical features, and the technical solutions after these changes or substitutions will fall within the protection scope of the present invention.

[0135] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A robot tracking control learning method based on post-screening experience playback, characterized in that: include: Initialize the target network parameters and experience pool, and store the state transition tuple in the experience pool; When the number of state transition tuples in the experience pool is greater than the preset number of state transition tuples, the complex state of the action is determined according to the trajectory smoothness and node covariance of the robot manipulator; The optimization method of determining the experience pool according to the complex state of the action is to determine the reward value of the pre-updated experience in the experience pool according to the position error and the estimated position deviation, or to determine the reward value of the pre-updated experience in the experience pool according to the estimated position deviation; Select the experience whose estimated position deviation is greater than or equal to the standard estimated position deviation as the screening experience, and determine the experience selection method to select the screening experience for the experience combination or reference sequence according to the number of screening experiences, and record the screening experience selected by the experience selection method as the pre-update experience; Update the critic network and actor network, and perform soft updates on the target network of the critic network and actor network respectively; The soft update weights of the target network of the critic network and the target network of the actor network are adjusted according to the reward evaluation value.

2. The robot tracking control learning method based on post-screening experience playback according to claim 1 is characterized in that: Complex action states include: A first action complexity state in which the trajectory smoothness is within a first preset trajectory smoothness range or the node covariance is within a first preset node covariance range; A second action complexity state in which the trajectory smoothness is within a second preset trajectory smoothness range and the node covariance is within a second preset node covariance range; Among them, all values ​​within the first preset trajectory smoothness range are greater than all values ​​within the second preset trajectory smoothness range, and all values ​​within the first preset node covariance range are greater than all values ​​within the second preset node covariance range.

3. The robot tracking control learning method based on post-screening experience playback according to claim 2 is characterized in that: The experience pool optimization method is determined according to the complexity of the action, where: In the first complex action state, the experience pool optimization method is to determine the reward value of the pre-updated experience in the experience pool according to the position error and the estimated position deviation; In the second complex action state, the experience pool optimization method is to determine the reward value of the pre-updated experience in the experience pool according to the estimated position deviation.

4. The robot tracking control learning method based on post-screening experience playback according to claim 3 is characterized in that: Select the state transition tuple whose estimated position deviation is greater than or equal to the standard estimated position deviation as the screening experience; The estimated position deviation at time t is determined based on the difference between the position error at time t and the position error at time t+1.

5. The robot tracking control learning method based on post-screening experience playback according to claim 4 is characterized in that: The experience selection method is determined according to the number of screening experiences, wherein: If the number of screening experiences is greater than or equal to the preset number of screening experiences, screening experiences are selected for the experience combination; If the number of screening experiences is less than the preset number of screening experiences, screening experience selection is performed for the reference sequence.

6. The robot tracking control learning method based on post-screening experience playback according to claim 5 is characterized in that: Screening experience selection based on experience combination, including: Perform combination analysis on each screening experience according to the reference order. When performing combination analysis on a single screening experience, record the screening experience as the target screening experience, and record other screening experiences other than the target screening experience as reference screening experiences. Detect the reward evaluation deviation between each reference screening experience after the reference order of the target screening experience and the target screening experience in turn until the reward evaluation deviation between the reference screening experience and the target screening experience is greater than the preset reward evaluation deviation, then stop the detection, and record the set of each screening experience before the reference order of the reference screening experience whose reward evaluation deviation with the target screening experience is greater than the preset reward evaluation deviation as an experience combination, and continue to perform combination analysis on the screening experiences that are not recorded as experience combinations until each screening experience is recorded as an experience combination, then stop the combination analysis; Screening experience selection is performed for each experience combination. When screening experience selection is performed for a single experience combination, a preset proportion of screening experiences in the experience combination is randomly selected, and each selected screening experience is used as a pre-update experience.

7. The robot tracking control learning method based on post-screening experience playback according to claim 5 is characterized in that: Empirical selection for screening against reference sequences, including: The sequence after sorting the screening experiences according to the reference order is recorded as the reference sequence, the reference sequence is divided into M equal parts, and the screening experiences at each equal part are selected as pre-update experiences.

8. The robot tracking control learning method based on post-screening experience playback according to claim 7 is characterized in that: Update the critic network; Use update formula (1) to update the critic network loss value L Q (θ) is updated, and formula (1) is: Where N is the number of experiences randomly selected for updating after determining the reward value of the pre-updated experience in the experience pool, θ is the initialization parameter of the critic network, and i is 1, 2, 3, ..., N, s i is the current state of the i-th experience used for updating, which includes the actual position, actual speed, target position and target speed of the robot arm, a i is the current action of the i-th experience used for updating, the action is the torque output by the robot arm on the joint angle, Q θ (s i , a i ) is the s of the critic network with initialization parameter θ for the i-th experience used for updating i and a i The prediction score, r i is the reward value of the i-th experience used for updating, s' i is the next state of the i-th experience used for updating, y(r i , s' i ) is determined by formula (2), which is: y(r i ,s' i )=r+γQ θ' (s' i ,π Φ' (s' i )) (2), Use stochastic gradient descent to update the critic network. The update formula is formula (3): Among them, r is the reward value, γ is the discount factor, θ' is the target network parameter of the critic network, π is the actor network to be trained, Ф' is the target network parameter of the actor network, π Ф' (s' i ) is the target network parameter Φ' of the actor network for the i-th experience used for updating s' i The predicted action, Q θ' (s' i , π Ф' (s' i )) is the target network parameter θ' of the critic network for the i-th experience used for updating s' i and π Ф' (s' i )’s prediction score.

9. The robot tracking control learning method based on post-screening experience playback according to claim 8 is characterized in that: Use stochastic gradient descent to update the actor network. The update formula is formula (4): Where a is equal to π Ф (s i ), Ф is the initialization parameter of the actor network, π Ф (s i ) is the s of the actor network with initialization parameter Ф for the i-th experience used for updating i Predicted actions.

10. The robot tracking control learning method based on post-screening experience playback according to claim 9 is characterized in that: Formula (5) is used to perform soft update on the target network of the critic network, and formula (6) is used to perform soft update on the target network of the actor network. θ'=αθ+(1-α)θ' (5), Φ'=αΦ+(1-α)Φ' (6), Among them, α is the soft update weight, α is greater than 0 and less than 1, and the soft update weights of the target network of the critic network and the target network of the actor network are adjusted according to the reward evaluation value; If the reward evaluation value is within the first preset reward evaluation value range, the soft update weight is adjusted to decrease; If the reward evaluation value is within the second preset reward evaluation value range, the soft update weight is increased and adjusted; Wherein, each value within the first preset reward evaluation value range is greater than each value within the second preset reward evaluation value range; The reward evaluation value is determined according to a reward fluctuation coefficient and a reward mean coefficient. The reward fluctuation coefficient is positively correlated with the reward evaluation value, and the reward mean coefficient is positively correlated with the reward evaluation value.

Images