A method for accelerating the training of underactuated robotic arms using single-timescale behavior perception temporal difference.

Abstract

This invention discloses a method for accelerating the training of underactuated robotic arms using single-timescale behavior-aware temporal difference, belonging to the field of underactuated robotic arm control. The method first establishes a reinforcement learning environment model, using attitude sensors to collect state information such as the robotic arm's angle and angular velocity, and preprocessing it into six-dimensional data. Then, it uses a neural network combined with other methods to complete action selection and execution, obtaining immediate rewards. Finally, it synchronously updates the target parameter θ and auxiliary parameter ω based on a single-timescale behavior-aware temporal difference algorithm, iteratively training until the policy converges. This invention solves the problems of slow convergence and low accuracy of traditional methods, improving training efficiency and control accuracy. It can be extended to complex underactuated robot systems and has high practical value.

Description

Technical Field

[0001] This invention relates to the field of underactuated robotic arm control technology, and in particular to a method for accelerating the training of underactuated robotic arms using single-timescale behavior perception temporal difference. Background Technology

[0002] Underactuated dual-joint robotic arms are typical underactuated systems, consisting of two robotic arms driven solely by a drive mechanism at their joint. While this structure offers advantages in terms of lightweight design and energy efficiency, it also results in highly nonlinear and strongly coupled dynamics, requiring the robotic arms to effectively utilize their degrees of freedom to complete pre-defined movements. Traditional control methods, relying on precise mathematical models, struggle to handle these complex characteristics and fail to achieve stable and accurate control.

[0003] Reinforcement learning offers a novel solution for the control of underactuated systems. By training an agent to learn control strategies, the free end of a robotic arm can reach a target height. However, existing reinforcement learning training methods suffer from slow training convergence speed and insufficient policy optimization accuracy, making it difficult to meet the control efficiency and accuracy requirements of practical operations. Therefore, there is an urgent need for a single-timescale behavior-aware temporal difference method for accelerating the training of underactuated robotic arms, which can improve training effectiveness and quickly converge to the optimal policy. Summary of the Invention

[0004] The purpose of this invention is to provide a method for accelerating the training of underactuated robotic arms using single-timescale behavior perception temporal difference, which solves the problems of slow convergence and low accuracy in existing training methods, and achieves efficient and precise control of underactuated dual-joint robotic arms.

[0005] To achieve the above objectives, the technical solution of the present invention is implemented as follows:

[0006] A single-timescale behavior-aware temporal difference-based method for accelerating the training of underactuated robotic arms is provided to expedite the training process of automated control of underactuated dual-joint robotic arms. The method includes the following steps:

[0007] S1. Establish a reinforcement learning environment model for the working environment of the underactuated dual-joint robotic arm and instantiate the trained neural network model.

[0008] S2. Obtain the state information of the underactuated dual-joint robotic arm using an attitude sensor. ;

[0009] S3. Obtain the status information. The input is fed into the neural network to obtain the corresponding features. Combining linear methods, i.e., the formula: The value corresponding to each executable action can be calculated. Value, ultimately used Method selection action And execute, record the current features ;in The search rate is a weight used to balance the random exploration of the strategy with the utilization of existing strategies. It represents an action, where 'i' is the action index. These are all available actions in the current state;

[0010] S4, Execute Action Enter the next state Receive rewards Calculate again Each action Seeking The action with the highest value Then use Method selection action and record ;

[0011] S5. For data obtained from a single sampling, the target parameters of the policy are calculated using a single-timescale behavior-aware temporal difference algorithm. and auxiliary parameters Update the strategy to optimize the current approach.

[0012] S6. Repeat steps S2-S5, updating after each training round. The parameters are adjusted until the policy converges or the maximum number of training rounds is reached.

[0013] Furthermore, in step S1, reinforcement learning environment modeling is performed based on the operation requirements of the underactuated dual-joint robotic arm. This environment is a discrete motion environment, where the drive device can only perform three discrete motions: clockwise torque, no torque, and counterclockwise torque. A sparse reward mechanism is adopted, where the reward is set to 0 when the robotic arm reaches the designated position and -1 when it does not reach the designated position.

[0014] Furthermore, the maximum number of training rounds is set within the range of 300-1000 times based on actual task requirements.

[0015] Furthermore, the maximum number of training rounds is set to 500 by default.

[0016] Furthermore, in step S2, the state information of the underactuated dual-joint robotic arm Specifically, this includes the angle of the first robotic arm relative to the vertical direction. The rotation direction of the first robotic arm The angle of the first robotic arm relative to the second robotic arm The rotation direction of the second robotic arm Angular velocity of the upper end of the first robotic arm Angular velocity at the connection between the first and second robotic arms .

[0017] Furthermore, the attitude sensor includes an angle sensor and an angular velocity sensor.

[0018] Furthermore, in step S4, an instant reward is obtained through the location detection module. The position detection module is used to determine in real time whether the free end of the underactuated dual-joint robotic arm has reached the specified height. It works synchronously with the attitude sensor. When the specified position is reached, the reward is 0, and when the specified position is not reached, the reward is -1.

[0019] Furthermore, in step S5, the objective function for algorithm optimization is constructed as follows:

[0020]

[0021] in, Indicates policy-based The expectation operator, The error is expressed by the formula: , This is the discount factor;

[0022] For target parameters Perform gradient descent to obtain Update formula:

[0023]

[0024] Auxiliary parameters in the objective function gradient:

[0025] Setting the gradient to 0, we can obtain:

[0026] For auxiliary parameters Perform gradient descent to obtain Update formula:

[0027]

[0028] The final update formula is:

[0029]

[0030]

[0031] in, This indicates that the next action is obtained through behavioral policy sampling. This indicates that the next action is obtained through sampling of the target policy. The learning rate is used to update both parameters. These are the parameters for the correction term.

[0032] Furthermore, in step S6, after each round of training, update The value is updated by using the following method: Multiply by an attenuation coefficient less than 1.

[0033] Beneficial effects: (1) The present invention adopts a single time scale behavior perception temporal difference algorithm to synchronously update the target parameter θ and auxiliary parameter ω, effectively reducing the mean square projection Bellman error and improving the training convergence speed; compared with the traditional Q-learning algorithm, this method can obtain the optimal control strategy faster and the robotic arm takes fewer steps to reach the target position.

[0034] (2) This invention clarifies the discrete characteristics of the reinforcement learning environment, the reward mechanism and the range of key parameter values, and limits the type of attitude sensor and the data processing method, making the method highly feasible.

[0035] (3) This invention utilizes ε The dynamic adjustment of ε in the greedy method achieves a smooth transition between exploration and exploitation; in the early stage of training, the strategy mainly focuses on random exploration to ensure strategy diversity; in the later stage of training, the strategy mainly focuses on exploitation to quickly converge to the optimal solution.

[0036] (4) This invention is not only applicable to underactuated dual-joint robotic arms, but can also be extended to more complex underactuated robot systems, providing strong support for high-level automated control. Attached Figure Description

[0037] The accompanying drawings, which form part of this invention, are used to provide a further understanding of the invention. The illustrative embodiments of the invention and their descriptions are used to explain the invention and do not constitute an undue limitation of the invention. In the drawings:

[0038] Figure 1 This is an algorithm flowchart of the underactuated robotic arm training acceleration method based on single-timescale behavior perception temporal difference, as described in an embodiment of the present invention.

[0039] Figure 2 This is a schematic diagram illustrating an example of the underactuated robotic arm training acceleration method based on single-timescale behavior perception temporal difference as described in an embodiment of the present invention.

[0040] Figure 3 The figure shows the experimental results of the underactuated robotic arm training acceleration method based on single-timescale behavior perception temporal difference as described in an embodiment of the present invention. Detailed Implementation

[0041] It should be noted that, unless otherwise specified, the embodiments and features described in the present invention can be combined with each other.

[0042] The present invention will now be described in detail with reference to the accompanying drawings and embodiments.

[0043] The single-timescale behavior-aware temporal difference algorithm of this invention refers to synchronously updating the target parameters at the same time scale. and auxiliary parameters A time-series difference algorithm to minimize mean square projection Bellman error (MSPBE).

[0044] Example 1

[0045] See Figures 1-3 A single-timescale behavior-aware temporal difference-based method for accelerating the training of underactuated robotic arms is proposed. This method is used to speed up the training process of automated control for underactuated dual-joint robotic arms and includes the following steps:

[0046] S1. Establish a reinforcement learning environment model for the working environment of the underactuated dual-joint robotic arm and instantiate the trained neural network model.

[0047] In step S1, reinforcement learning environment modeling is performed according to the operation requirements of the underactuated dual-joint manipulator. This environment is a discrete motion environment, where the drive device can only perform three discrete motions: clockwise torque, no torque, and counterclockwise torque. A sparse reward mechanism is adopted, where the reward is set to 0 when the manipulator reaches the specified position and -1 when it does not reach the specified position.

[0048] S2. Obtain the state information of the underactuated dual-joint robotic arm using an attitude sensor. ;

[0049] In step S2, the current state of the underactuated dual-joint robotic arm The actual input to the neural network specifically includes the angle of the first robotic arm relative to the vertical direction. The rotation direction of the first robotic arm The angle of the first robotic arm relative to the second robotic arm The rotation direction of the second robotic arm Angular velocity of the upper end of the first robotic arm Angular velocity at the connection between the first and second robotic arms In this embodiment, the underactuated dual-joint robotic arm obtains information through an attitude sensor. , , , The data processing method is as follows: , Calculate the cosine and sine values ​​separately. , The original measurement values ​​are directly retained and finally integrated into a six-dimensional state data input into the neural network;

[0050] S3. Obtain the status information. The input is fed into the neural network to obtain the corresponding features. Combining linear methods, i.e., the formula: The value corresponding to each executable action can be calculated. Value, ultimately used Method selection action And execute, record the current features ;in The search rate is a weight used to balance the random exploration of the strategy with the utilization of existing strategies. It represents an action, where 'i' is the action index. These are all available actions in the current state;

[0051] The The specific methods to choose from are as follows: This is a number between 0 and 1, set to 0.1 in the experiment, and updated after each training round. A random number between 0 and 1 is randomly generated; if it is less than 1... Then, randomly select one action from all executable actions as... Conversely, select from all executable actions. The biggest action as ;

[0052] S4, Execute Action Enter the next state Receive rewards Calculate again Each action Seeking The action with the highest value Then use Method selection action and record ;

[0053] In step S4, an instant reward is obtained through the location detection module. The position detection module is used to determine in real time whether the free end of the underactuated dual-joint robotic arm has reached the specified height. It works synchronously with the attitude sensor. When the specified position is reached, the reward is 0, and when the specified position is not reached, the reward is -1.

[0054] S5. For data obtained from a single sampling, the target parameters of the policy are calculated using a single-timescale behavior-aware temporal difference algorithm. and auxiliary parameters Update the strategy to optimize the current approach.

[0055] In step S5, the objective function for algorithm optimization is constructed as follows:

[0056]

[0057] in, Indicates policy-based The expectation operator, The error is expressed by the formula: , This is the discount factor;

[0058] For target parameters Perform gradient descent to obtain Update formula:

[0059]

[0060] Auxiliary parameters in the objective function gradient:

[0061] Setting the gradient to 0, we can obtain:

[0062] For auxiliary parameters Perform gradient descent to obtain Update formula:

[0063]

[0064] The final update formula is:

[0065]

[0066]

[0067] in, This indicates that the next action is obtained through behavioral policy sampling. This indicates that the next action is obtained through sampling of the target policy. The learning rate is used to update both parameters. These are the parameters for the correction term.

[0068] In the specific implementation, In the experiment, it was set to 0.01. This is a correction parameter, set to 1 in the experiment.

[0069] S6. Repeat steps S2-S5, updating after each training round. The parameters are adjusted until the policy converges or the maximum number of training rounds is reached.

[0070] In step S6, update after each round of training. The value is updated by using the following method: Multiply by an attenuation factor less than 1, with the attenuation factor ranging from 0.9991 to 0.9993.

[0071] The maximum number of training rounds in this embodiment is set to 500 by default, and can be adjusted within the range of 300-1000 rounds according to actual job requirements.

[0072] It should be noted that as the number of training rounds increases, The value will gradually decrease, so that in the early stage of the algorithm, the strategy will conduct more random exploration, and in the later stage of training, the algorithm will make decisions based more on the existing strategy, thus achieving a smooth transition from random exploration to strategy utilization.

[0073] The method proposed in this embodiment is compared with traditional training methods, and the statistical results are as follows: Figure 3 As shown.

[0074] exist Figure 3 In this paper, the proposed single-timescale behavior-aware temporal difference-based method for accelerating underactuated robotic arm training outperforms traditional policy optimization algorithms in terms of convergence results. Figure 3 (Using Q-learning as a representation), the trained strategy can eventually reach a better strategy, which in a specific instance means that the robotic arm ultimately reaches the target height in fewer steps.

[0075] In summary, the method of this invention can improve the training effect of automated control of underactuated dual-joint robotic arms by using a single-timescale behavior-aware temporal difference algorithm, learn the optimal control strategy, improve the performance of underactuated systems, and assist underactuated robotic arms in achieving their objectives.

[0076] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for accelerating the training of an underactuated robotic arm using single-timescale behavior-aware temporal difference, characterized in that, To accelerate the training process for automated control of an underactuated dual-joint robotic arm, the following steps are included: S1. Establish a reinforcement learning environment model for the working environment of the underactuated dual-joint robotic arm and instantiate the trained neural network model. S2. Obtain the state information of the underactuated dual-joint robotic arm using an attitude sensor. ; S3. Obtain the status information. The input is fed into the neural network to obtain the corresponding features. Combining linear methods, i.e., the formula: The value corresponding to each executable action can be calculated. Value, ultimately used Method selection action And execute, record the current features ;in The search rate is a weight used to balance the random exploration of the strategy with the utilization of existing strategies. It represents an action, where 'i' is the action index. These are all available actions in the current state; S4, Execute Action Enter the next state Receive rewards Calculate again Each action Seeking The action with the highest value Then use Method selection action and record ; S5. For data obtained from a single sampling, the target parameters of the policy are calculated using a single-timescale behavior-aware temporal difference algorithm. and auxiliary parameters Update the strategy to optimize the current approach. S6. Repeat steps S2-S5, updating after each training round. The parameters are adjusted until the policy converges or the maximum number of training rounds is reached.

2. The method for accelerating the training of underactuated robotic arms using single-timescale behavior perception temporal difference as described in claim 1, characterized in that, In step S1, reinforcement learning environment modeling is performed based on the operation requirements of the underactuated dual-joint robotic arm. This environment is a discrete motion environment, where the drive device can only perform three discrete motions: clockwise torque, no torque, and counterclockwise torque. A sparse reward mechanism is adopted, where the reward is set to 0 when the robotic arm reaches the specified position and -1 when it does not reach the specified position.

3. The method for accelerating the training of underactuated robotic arms using single-timescale behavior perception temporal difference as described in claim 1, characterized in that, The maximum number of training rounds is set within the range of 300-1000 times based on actual operational needs.

4. The method for accelerating the training of underactuated robotic arms using single-timescale behavior perception temporal difference as described in claim 3, characterized in that, The maximum number of training rounds is set to 500 by default.

5. The method for accelerating the training of underactuated robotic arms using single-timescale behavior perception temporal difference as described in claim 1, characterized in that, In step S2, the state information of the underactuated dual-joint robotic arm Specifically, this includes the angle of the first robotic arm relative to the vertical direction. The rotation direction of the first robotic arm The angle of the first robotic arm relative to the second robotic arm The rotation direction of the second robotic arm Angular velocity of the upper end of the first robotic arm Angular velocity at the connection between the first and second robotic arms .

6. The method for accelerating the training of underactuated robotic arms using single-timescale behavior perception temporal difference as described in claim 1, characterized in that, The attitude sensor includes an angle sensor and an angular velocity sensor.

7. The method for accelerating the training of underactuated robotic arms using single-timescale behavior perception temporal difference as described in claim 1, characterized in that, In step S4, an instant reward is obtained through the location detection module. The position detection module is used to determine in real time whether the free end of the underactuated dual-joint robotic arm has reached the specified height. It works synchronously with the attitude sensor. When the specified position is reached, the reward is 0, and when the specified position is not reached, the reward is -1.

8. The method for accelerating the training of an underactuated robotic arm using single-timescale behavior perception temporal difference as described in claim 1, characterized in that, In step S5, the objective function for algorithm optimization is constructed as follows: in, Indicates policy-based The expectation operator, The error is expressed by the formula: , This is the discount factor; For target parameters Perform gradient descent to obtain Update formula: Auxiliary parameters in the objective function gradient: Setting the gradient to 0, we can obtain: For auxiliary parameters Perform gradient descent to obtain Update formula: The final update formula is: in, This indicates that the next action is obtained through behavioral policy sampling. This indicates that the next action is obtained through sampling of the target policy. The learning rate is used to update both parameters. These are the parameters for the correction term.

9. The method for accelerating the training of an underactuated robotic arm using single-timescale behavior perception temporal difference as described in claim 1, characterized in that, In step S6, update after each round of training. The value is updated by using the following method: Multiply by an attenuation coefficient less than 1.

Images