A mechanical arm grabbing method based on FO-DDPG algorithm
By proposing a robotic arm grasping method based on the fractional DDPG algorithm, and by optimizing the grasping strategy using fractional calculus and deep learning, the adaptability and robustness issues of traditional methods in complex environments are solved, achieving more efficient grasping performance and stability.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- CHANGCHUN UNIV OF SCI & TECH
- Filing Date
- 2023-12-08
- Publication Date
- 2026-05-19
AI Technical Summary
Traditional robotic arm grasping methods are difficult to adapt to uncertain environments and changes in target objects, lack robustness and adaptability, and cannot effectively handle friction and shape changes in complex environments.
We employ a fractional-order Deep Deterministic Differential Policy Gradient (DDPG) algorithm, combined with fractional-order calculus theory, to construct Actor and Critic networks. By using an empirical replay buffer and gradient descent, we optimize the grasping strategy, capture long-term dependencies in the state space, and improve grasping performance and robustness.
It improves the accuracy and adaptability of robotic arm grasping, enhances grasping performance and robustness in complex environments, and improves the stability and generalization ability of grasping strategies.
Smart Images
Figure CN117484507B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of robot control technology, specifically to a robotic arm grasping method based on the FO-DDPG algorithm. Background Technology
[0002] The technical background of robotic arm grasping technology based on the fractional DDPG algorithm includes the challenges of robotic arm grasping tasks, the basic principles of the DDPG algorithm, and the application of fractional calculus.
[0003] Challenges of Robotic Arm Grasping Tasks: Robotic arm grasping is a complex and challenging task. It involves accurately sensing and controlling the position, force, and posture of the robotic arm in three-dimensional space to achieve stable grasping of a target object. Robotic arms must overcome uncertainties, friction, changes in object shape, and environmental interference during grasping tasks. Traditional robotic arm grasping methods typically rely on predefined rules or manually designed control strategies, which struggle to adapt to uncertain environments and changes in the target object. Therefore, to improve the performance and robustness of robotic arm grasping tasks, new methods and algorithms are needed.
[0004] Background of the DDPG algorithm: The DDPG algorithm is a reinforcement learning algorithm suitable for solving decision-making problems in a continuous action space. It combines deep neural networks and policy gradient methods to learn the optimal policy through interaction with the environment. The DDPG algorithm guides the machine agent's decision-making by establishing value functions and policy functions, and improves the policy performance through iterative optimization.
[0005] The introduction of fractional calculus: Fractional calculus is an extension of traditional integer calculus, introducing derivatives and integration operations of non-integer order. Fractional calculus can more accurately model and handle complex nonlinear, nonlocal, and long-time dependent dynamic systems. In robotic arm grasping tasks, the introduction of fractional calculus concepts and techniques can improve the modeling capabilities of value functions and policy functions, thereby improving the grasping performance and robustness of the robotic arm.
[0006] The fractional-order DDPG algorithm is a method that applies the ideas and techniques of fractional calculus to robotic arm grasping tasks. This algorithm approximates the gradient information of the value function using fractional derivatives, enabling it to better capture long-term dependencies in the state space. Furthermore, introducing fractional integral operations to optimize the policy function allows for weighted application of past experience to guide current grasping decisions, improving the quality and robustness of the grasping strategy. Summary of the Invention
[0007] (a) Technical problems to be solved
[0008] To address the shortcomings of existing technologies, this invention provides a robotic arm grasping method based on the FO-DDPG algorithm, which solves the problems mentioned in the background section.
[0009] (II) Technical Solution
[0010] To achieve the above objectives, the present invention specifically adopts the following technical solution:
[0011] A robotic arm grasping method based on FO-DDPG includes the following steps:
[0012] S1. Define the target and environment: Define the specific target of the robotic arm's grasping task, such as grasping an object of a specific shape or placing an object in a specific position, and define the physical environment of the robotic arm, including the structure of the robotic arm, sensor configuration and the grasping actions that can be performed.
[0013] S2. Construct a fractional-order DDPG network structure: Design and implement the structures of the Actor network and the Critic network, where the Actor network is used to learn the grasping policy and the Critic network is used to estimate the action value. Use fractional-order calculus to optimize the value function network and the policy network to capture long-term dependencies in the state space.
[0014] S3. Data Acquisition and Preprocessing: Randomly execute grasping actions in the environment and record the observations, grasping actions, and reward signals of the robotic arm. Perform preprocessing on the observations and reward signals, such as normalization and noise reduction, to improve training effectiveness and stability.
[0015] S4. Experience Replay Buffer: Construct an experience replay buffer to store the grasping actions and observation information performed by the robotic arm. In each training step, a batch of samples is randomly selected from the experience replay buffer to increase sample diversity and reduce sample correlation during training.
[0016] S5. Fractional DDPG training process: According to the update rules of the fractional DDPG algorithm, training iterations are performed repeatedly. In each iteration, actions are selected from the Actor network based on the observations, and the value of the actions is estimated using the Critic network. The target value function and loss function are calculated, and the parameters of the Actor network and Critic network are updated using the gradient descent method.
[0017] S6. Strategy Evaluation and Optimization: During training, the robotic arm's grasping strategy is periodically evaluated to assess grasping performance and robustness. Based on the evaluation results, the network parameters are adjusted and optimized to improve the quality and stability of the grasping strategy.
[0018] S7. Grasping Task Execution: Using the trained policy network, the robotic arm is deployed to the actual grasping task. By observing the state of the environment and the target object in real time, the grasping task is executed according to the grasping action output by the policy network.
[0019] Furthermore, in S1, determining the target and environment includes defining the specific target of the robotic arm's grasping task, such as grasping an object of a specific shape or placing an object in a specific position, and determining the physical environment of the robotic arm, including the structure of the robotic arm, sensor configuration, and executable grasping actions.
[0020] Furthermore, in S2, a fractional-order DDPG network structure is constructed, including the design and implementation of the structures of the Actor network and the Critic network. The Actor network is used to learn the grasping policy, and the Critic network is used to estimate the action value. Fractional-order calculus is used to optimize the value function network and the policy network to capture long-term dependencies in the state space.
[0021] Furthermore, the data acquisition and preprocessing in S3 includes randomly executing grasping actions in the environment and recording the observations, grasping actions, and reward signals of the robotic arm. The observations and reward signals are then preprocessed, such as normalized and denoised, to improve the training effect and stability.
[0022] Furthermore, the experience replay buffer in S4 includes constructing an experience replay buffer for storing the grasping actions and observation information performed by the robotic arm. In each training step, a batch of samples is randomly selected from the experience replay buffer to increase the diversity of samples and reduce the correlation between samples during the training process.
[0023] Furthermore, the fractional-order DDPG training process in S5 includes repeatedly performing training iterations according to the update rules of the fractional-order DDPG algorithm. In each iteration, a grabbing action is selected from the Actor network based on the observation value, and the value of the action is estimated using the Critic network. The target value function and loss function are calculated, and the parameters of the Actor network and the Critic network are updated using the gradient descent method.
[0024] Furthermore, the strategy evaluation and optimization in S6 includes periodically evaluating the robotic arm's grasping strategy during the training process to assess grasping performance and robustness, and adjusting and optimizing the network parameters based on the evaluation results to improve the quality and stability of the grasping strategy.
[0025] Furthermore, the grasping task execution in S7 includes using a trained policy network to deploy the robotic arm into the actual grasping task, and executing the grasping task according to the grasping action output by the policy network by observing the state of the environment and the target object in real time.
[0026] (III) Beneficial Effects
[0027] Compared with existing technologies, this invention provides a fractional-order DDPG algorithm, which has the following advantages:
[0028] Improving gripping performance: Fractional-order control theory can capture the nonlinear characteristics and temporal correlations of robotic arm motion, thereby providing more accurate modeling and control capabilities. By combining fractional-order control theory and the DDPG algorithm, the gripping strategy of the robotic arm can better adapt to different environments and object characteristics, thus improving gripping performance.
[0029] Improving robustness: Traditional control methods often fail to fully account for the uncertainties and variability of robotic arms in complex environments. The fractional-order DDPG algorithm, which uses techniques such as experience replay and target networks, can reduce the variance of the action strategy and improve the gripping robustness of the robotic arm in unstable and noisy environments.
[0030] Adaptability and generalization ability: The deep learning-based DDPG algorithm can be trained to adapt to different robotic arms and tasks and has strong generalization ability. Fractional-order control theory provides more accurate system modeling, which can better adapt to the characteristics and shapes of different objects, thereby improving the adaptability and generalization ability of robotic arm grasping technology.
[0031] Long-range dependency and memory effect: Fractional derivatives can capture the long-range dependency and memory effect of system dynamics. In robotic arm grasping tasks, the robotic arm's movements are often affected by previous states and actions. The fractional derivative-based DDPG algorithm can better handle this long-range dependency and memory effect, improving the accuracy and efficiency of robotic arm grasping. Attached Figure Description
[0032] Figure 1 This is a diagram of the overall network structure of the present invention;
[0033] Figure 2 This is a diagram of the improved playback experience sampling network structure of the present invention;
[0034] Figure 3 This is a diagram of the Actor network structure of the present invention;
[0035] Figure 4 This is a diagram of the Critic network structure of the present invention. Detailed Implementation
[0036] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.
[0037] Example
[0038] like Figure 1-4 As shown in the figure, an embodiment of the present invention proposes a robotic arm grasping method based on the FO-DDPG algorithm, which includes the following steps:
[0039] S1. Define the target and environment: Define the specific target of the robotic arm's grasping task, such as grasping an object of a specific shape or placing an object in a specific location. Define the physical environment of the robotic arm, including its structure, sensor configuration, and the grasping actions that can be performed.
[0040] S2. Constructing a Fractional-Order DDPG Network Structure: Design and implement the structures of the Actor network and Critic network, where the Actor network is used to learn the grasping policy, and the Critic network is used to estimate the action value. Optimize the value function network and policy network using fractional-order calculus to capture long-term dependencies in the state space. Initialize network parameters: Initialize the weights and biases of the Actor and Critic networks. Initialize the parameters of the target network to calculate the target value function. Train for the grasping task: In each training cycle, reset the state of the robotic arm and set the initial time step t = 0. Based on the current state s... t Select the grabbing action a from the Actor network t And execute the action. A new state s is observed. t+1 and reward signal r t . will (s t ,a t ,s t+1 ,r t Stored in the experience playback buffer.
[0041] S3. Data Acquisition and Preprocessing: Randomly execute grasping actions in the environment and record the robotic arm's observations, grasping actions, and reward signals. Preprocess the observations and reward signals, such as normalization and noise reduction, to improve training effectiveness and stability.
[0042] S4. Experience Replay Buffer: An experience replay buffer is constructed to store the grasping actions and observation information performed by the robotic arm. In each training step, a batch of samples is randomly sampled from the experience replay buffer for network updates. The observation value s for each sample is obtained. iGrab action a i Reward signal r i , and the next state s i+1 This is to increase sample diversity and reduce sample correlation during training.
[0043] S5. Fractional DDPG Training Process: Training iterations are performed repeatedly according to the update rules of the fractional DDPG algorithm. In each iteration, actions are selected from the Actor network based on observations, and the value of the actions is estimated using the Critic network. The target value function and loss function are calculated, and the parameters of the Actor and Critic networks are updated using gradient descent. The target value function y is calculated using the target Critic network. i ;
[0044] y i =r i +γQ′(s i+1 ,μ′(s i+1 )) (1)
[0045] Where Q′ is the target Critic network, μ′ is the target Actor network, and γ is the discount factor. Update the parameters of the Critic network and minimize the loss function:
[0046]
[0047] Where N is the number of samples in the batch.
[0048] Using the action value estimated by the Critic network as the optimization objective, the gradient of the Actor network is calculated:
[0049]
[0050] Update the parameters of the Actor network using gradient descent:
[0051]
[0052] Where α is the learning rate.
[0053] S6. Strategy Evaluation and Optimization: During training, the robotic arm's grasping strategy is periodically evaluated to assess its grasping performance and robustness. Based on the evaluation results, the network parameters are adjusted and optimized to improve the quality and stability of the grasping strategy. A soft update method is used to progressively update the parameters of the Actor and Critic networks to the target network.
[0054] θ Q′ ←τθ Q +(1-τ)θ Q′ (5)
[0055] θ μ′ ←τθ μ +(1-τ)θ μ′ (6) Where τ is the soft update coefficient.
[0056] S7. Grasping Task Execution: Using the trained policy network, deploy the robotic arm to perform the actual grasping task. By observing the state of the environment and the target object in real time, execute the grasping task according to the grasping actions output by the policy network. Repeat steps two through six until the preset number of training rounds is reached or the grasping performance meets the requirements.
[0057] Through the above steps, the fractional-order DDPG algorithm can progressively optimize the robotic arm's grasping strategy, enabling it to successfully perform grasping tasks under different environments and object characteristics. The symbols in the formulas represent the variables and operations used in the corresponding steps.
[0058] Finally, it should be noted that the above descriptions are merely preferred embodiments of the present invention and are not intended to limit the present invention. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing embodiments or make equivalent substitutions for some of the technical features. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.
Claims
1. A robotic arm grasping method based on FO-DDPG, characterized in that: Includes the following steps: S1. Define the target and environment: Define the specific target of the robotic arm's grasping task, such as grasping an object of a specific shape or placing an object in a specific position, and define the physical environment of the robotic arm, including the structure of the robotic arm, sensor configuration, and the grasping actions that can be performed. S2. Construct a fractional-order DDPG network structure: Design and implement the structures of the Actor network and the Critic network, where the Actor network is used to learn the grasping policy and the Critic network is used to estimate the action value. Use fractional-order calculus to optimize the value function network and the policy network to capture long-term dependencies in the state space. S3. Data Acquisition and Preprocessing: Randomly execute grasping actions in the environment and record the observations, grasping actions, and reward signals of the robotic arm. Preprocess the observations and reward signals by normalizing and reducing noise to improve training effectiveness and stability. S4. Experience Replay Buffer: Construct an experience replay buffer to store the grasping actions and observation information performed by the robotic arm. In each training step, a batch of samples is randomly selected from the experience replay buffer to increase sample diversity and reduce sample correlation during training. S5. Fractional DDPG training process: According to the update rules of the fractional DDPG algorithm, training iterations are performed repeatedly. In each iteration, actions are selected from the Actor network based on the observations, and the value of the actions is estimated using the Critic network. The target value function and loss function are calculated, and the parameters of the Actor network and Critic network are updated using the gradient descent method. S6. Strategy Evaluation and Optimization: During training, the robotic arm's grasping strategy is periodically evaluated to assess grasping performance and robustness. Based on the evaluation results, the network parameters are adjusted and optimized to improve the quality and stability of the grasping strategy. S7. Grasping Task Execution: Using the trained policy network, the robotic arm is deployed to the actual grasping task. By observing the state of the environment and the target object in real time, the grasping task is executed according to the grasping action output by the policy network.