Arm-hand robot grabbing method based on deep reinforcement learning

Abstract

The invention discloses an arm-hand robot grabbing method based on deep reinforcement learning. The arm-hand robot grabbing method comprises the following core steps: 1) providing a strategy network structure based on a sparse causal time self-attention mechanism; (2) a post experience recombination method is provided, so that the utilization efficiency of successful samples by the algorithm is improved; 3) designing a self-adaptive conservative Q learning value network updating method, and dynamically adjusting the intensity of a regularization item through a self-adaptive adjustment mechanism based on a time sequence difference error and an average reward; the method can effectively improve the convergence speed of the algorithm, balance exploration and stability requirements in the training process, and improve the stability of the algorithm. In addition, according to the strategy network structure of the method, through local window sparse connection and an LSTM series structure, the modeling capacity of single-step, local and overall action characteristics is effectively enhanced, and finally the success rate of the grabbing task can be effectively improved.

Description

Technical Field

[0001] The present invention is applied to a high-degree-of-freedom arm-hand robot system, and particularly relates to a grasping method for an arm-hand robot based on deep reinforcement learning. Background Art

[0002] Traditional manipulator grasping methods rely on precise modeling and optimization. Although suitable for structured environments, they have poor adaptability in complex dynamic scenarios. Grasping methods based on deep learning, although improving the intelligent level of visual perception and grasping pose prediction, have a strong dependence on large-scale labeled data. Multimodal grasping methods integrate visual and tactile information, improving the perception accuracy, but still lack real-time performance and generalization ability. The grasping method based on deep reinforcement learning combines the advantages of perception and decision-making, and has the ability of autonomous learning and policy optimization. This method can optimize the grasping policy through unsupervised interaction and show strong adaptability and generalization ability in unstructured environments. Although it faces challenges such as low sample efficiency, high training cost, and difficulty in virtual-real migration, its advantages in intelligence and flexibility make it one of the most promising directions in grasping method research.

[0003] Both the field of deep reinforcement learning and the field of robot grasping have received great attention and produced many results. However, due to the high degree of freedom of arm-hand robots, deep reinforcement learning faces problems such as low sample efficiency, slow convergence speed, easy to fall into local optimum, unstable algorithms, and limitations in network structure in such applications. These problems lead to slow convergence speed of task learning and low task success rate, and the learned trajectories may be unsafe or sub-optimal.

[0004] Currently, the mainstream deep reinforcement learning algorithms include DDPG, TD3, PPO, and SAC, etc., which are continuously optimized in terms of sample efficiency, policy stability, and generalization ability. Among them, DDPG is suitable for continuous action spaces but is sensitive to hyperparameters; TD3 improves stability through delayed updates and policy perturbations; PPO has a good policy update mechanism but is limited in high-dimensional action spaces. In contrast, SAC is based on the maximum entropy policy optimization framework, which can improve the sample utilization efficiency while ensuring policy stability, and is particularly suitable for tasks with high-dimensional and complex action spaces, such as arm-hand robot grasping. Summary of the Invention

[0005] At present, due to the high degree of freedom of the arm-hand robot, reinforcement learning faces problems such as low sample efficiency, slow convergence speed, unstable algorithm, and limitations in the network structure in such applications. These problems lead to a slow convergence speed of task learning and poor task effects. Therefore, in view of the grasping scenario of the arm-hand robot, the present invention solves the problems of slow convergence speed, unstable algorithm, and limitations in the network structure, and provides a grasping method for the arm-hand robot based on deep reinforcement learning.

[0006] A grasping method for an arm-hand robot based on deep reinforcement learning, the method comprising the following steps:

[0007] 1) Define the state space S of the arm-hand robot (a 50-dimensional vector, including the angles s qpos and angular velocities s qvel of 18 joints, the 3D vector position s grip-pos of the grasping center of the five-finger hand, and the 4D vector orientation s grip-quat , the 3D vector position s object-pos and 4D vector orientation s object-quat ) of the object to be grasped), the action space A (a 12-dimensional vector, including the incremental form control of the 6D vector pose a robot of the end-effector of the manipulator, and the 6D vector position a hand ) of the dexterous hand, as well as the reward function R(s t , a t ), where s t is the state at time t, and a t is the action to be executed by the agent at time t (where s t ∈S, a t ∈A);

[0008] 2) Initialize the value network Q θ , the policy network and related parameters (such as the regularization coefficient range λ min and λ max , the reward threshold R threshold , etc.);

[0009] 3) In each Episode, generate the action a t through the main policy network π(s t ), and obtain the state transition data (s t , a t , r t , s t+1 ) after interacting with the environment;

[0010] 4) Use the off-policy experience replay technique to assign a new target g to the experience data and modify the corresponding reward r, and the experience becomes (s t ||g, a t , r′t ,s t+1 || g), and store it in the hierarchical experience pool (buffer_1, buffer_2, buffer_3). The recombination module recombines the data in the experience pool to generate new experience samples for training;

[0011] 5) Calculate the average reward of the last 4 Episodes. Enter the update phase, randomly sample data from the experience pool, calculate the target TD value and TD error. After normalizing the TD error, dynamically adjust the regularization weight η according to the average reward value and TD error. Minimize the loss function of the regularization term to optimize Q θ and update the policy network by maximizing the value network value Q(s, π(s))

[0012] Compared with the prior art, the present invention has the following technical effects:

[0013] Compared with the native experience pool of the SAC algorithm and the method of experience replay after the event, this method can effectively improve the convergence speed of the algorithm; compared with conservative Q-learning, this method is based on the adaptive adjustment mechanism of temporal difference error and average reward, dynamically adjusts the strength of the regularization term, balances the exploration and stability requirements during the training process, and improves the stability of the algorithm; in addition, the policy network structure of this method effectively enhances the modeling ability of single-step, local and overall action features through the local window sparse connection and LSTM cascade structure, and finally can effectively improve the success rate of the grasping task. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] Figure 1 is the overall block diagram of the algorithm;

[0015] Figure 2 is the schematic diagram of the improved policy network structure;

[0016] Figure 3 is the comparison diagram of the receptive field expansion of fully connected and sparse connections;

[0017] Figure 4 is the schematic diagram of the reinforcement learning trajectory recombination;

[0018] Figure 5 is the example diagram of the agent grasping unstably and then dropping and grasping again;

[0019] Figure 6 is the example diagram of the nut assembly task;

[0020] Figure 7 is the comparison diagram of the reward curves of the algorithm convergence speed;

[0021] Figure 8 is the comparison diagram of the reward curves of the algorithm stability; Detailed implementation manners

[0022] The present invention will be further described in detail below in conjunction with the accompanying drawings and specific implementation manners.

[0023] A grasping method for an arm-hand robot based on deep reinforcement learning, the method comprising the following steps:

[0024] 1) Define the state space S of the arm-hand robot (a 50-dimensional vector, including the angles s qpos and angular velocities s qvel of 18 joints, the 3D vector position s grip-pos of the grasping center of the five-finger hand, and the 4D vector orientation s grip-quat , the 3D vector position s object-pos and 4D vector orientation s object-quat ) of the object to be grasped, the action space A (a 12-dimensional vector, including the incremental form control of the 6D vector attitude a robot of the end tool center of the manipulator, and the 6D vector position a hand of the dexterous hand), and the reward function R(s t , a t ), where s t is the state at time t, and a t is the action to be executed by the agent at time t (where s t ∈S, a t ∈A);

[0025] 2) Initialize the value network Q θ , the policy network and related parameters (such as the regularization coefficient range λ min and λ max , the reward threshold R threshold , etc.);

[0026] 3) In each Episode, generate the action a t through the main policy network π(s t ), and obtain the state transition data (s t , a t , r t , s t+1 ) after interacting with the environment;

[0027] 4) Use the off-policy experience replay technique to assign a new target g to the experience data and modify the corresponding reward r, and the experience becomes (s t ||g, a t , r′ t , s t+1||(g), and stored in the hierarchical experience pool (buffer_1, buffer_2, buffer_3). The recombination module recombines the data in the experience pool to generate new experience samples for training;

[0028] 5) Calculate the average reward of the last 4 Episodes. Enter the update stage, randomly sample data from the experience pool, calculate the target TD value and TD error. After normalizing the TD error, dynamically adjust the regularization weight η according to the average reward value and TD error; Minimize the loss function of the regularization term to optimize Q θ and update the policy network by maximizing the value network value Q(s, π(s))

[0029] The overall block diagram of the algorithm of the present invention is as Figure 1 shown. Combining steps 1), 2), 3), 4), and 5), the complete process of the improved algorithm can be obtained:

[0030]

[0031]

[0032] The following mainly elaborates on steps 3), 4), and 5) in detail:

[0033] In step 3), in each Episode, the agent generates an action a t through the main policy network π(s t ), and after interacting with the environment, obtains the state transition data (s t , a t , r t , s t+1 ). Here, aiming at the problem that the traditional fully connected policy network structure is difficult to capture complex time-dependent relationships, a policy network structure based on a sparse causal time self-attention mechanism is proposed;

[0034] Reduce the attention calculation complexity through local window sparse connection and cascade with LSTM to construct a new policy structure. The improved new policy network structure is as Figure 2 shown. The core of the entire policy network structure lies in the sparse causal time attention mechanism (Sparse CT-MSA), which consists of multiple sub-modules and captures local and global time-dependent relationships by gradually expanding the receptive field layer by layer;

[0035] Specifically, it includes the following steps:

[0036] 1) Local window mechanism: The core part of the sparse causal temporal attention mechanism, mainly used to reduce the complexity of attention calculation. This mechanism divides the input time series into several non-overlapping windows of size W, calculates the attention within each window, and restricts the attention calculation to within the window range. This design significantly reduces the computational complexity while concentrating on capturing strong dependencies between local time steps. The attention calculation formula within the window is as follows:

[0037]

[0038] Among them, the query Q, key K, and value V are generated by the linear projection of the input feature X:

[0039] Q h = XW q , K h = XW k , V h = XW v (2)

[0040] Among them, is a learnable linear projection matrix, and d k is the dimension of the key vector.

[0041] 2) Causal masking mechanism: It is another key component in the design of the sparse causal temporal attention mechanism, used to ensure the causal constraint of the model. Through the masking matrix, the model can shield the information of future time steps, ensuring that the calculation of the current time step only depends on the current and previous time steps. The masking matrix is defined as:

[0042]

[0043] In the attention calculation, the masking matrix is added to the dot product result of the key and the query, thereby shielding the influence of future time steps. The updated attention matrix calculation formula is:

[0044]

[0045] This design ensures causality in time series tasks and provides a guarantee for causal modeling in the policy network and value network in reinforcement learning.

[0046] 3) Layer normalization and skip connection: The layer normalization (LayerNorm) and skip connection (Residual Connection) mechanisms are introduced in the sparse causal temporal attention mechanism module. Layer normalization helps to stabilize the training process, while skip connection effectively alleviates the vanishing gradient problem in deep networks and preserves the input information.

[0047] 4) Multi-layer perceptron sub-module: There are two layers of multi-layer perceptrons after the Mask calculation module, which further enhance the feature expression ability using non-linear activation functions. This design improves the non-linear modeling ability of the model, making it more flexible in time series modeling.

[0048] 5) Gradually expand the window size: In the multi-layer design of the sparse causal time attention mechanism, the window size expands layer by layer. The lower layers mainly capture short-term dependencies, while the higher layers capture global dependencies with long time spans by expanding the receptive field. This design not only optimizes the computational efficiency but also ensures the accuracy of dependency modeling.

[0049] Based on the original causal time self-attention mechanism, the present invention proposes a sparsification operation to further optimize the connection pattern: Different from the traditional fully connected design, the sparsification operation reduces the computational complexity by restricting the connections within each window to the dependencies between key time steps. Specifically, only the most critical time step dependencies within the window are retained, rather than performing a full connection for all time steps. The specific sparsification operation is as Figure 3 shown.

[0050] In the sparsified design, Block 1 with a window size of 2 mainly processes short-term dependencies, Block 2 with a window size of 4 processes medium-term dependencies, and Block 3 with a window size of 8 is responsible for capturing dependencies with longer time spans. Through this design, the time complexity of the module is O(ρTWC), where ρ represents the sparsity, usually ρ < 1; T is the sequence length, W is the window size, and C is the feature dimension.

[0051] Before the improvement of the policy network, the input required for each update of the policy network is usually "single-frame" data, that is, the batch tuple (s t , a t , r t , s t+1 ). After the improvement of the policy network, because the input needs to match the sequence input format of the LSTM and attention mechanism, the input of the improved policy network requires "multi-frame" data, that is, the input needs to be changed to the batch tuple (s t , a t , r t , s t+1 )...(s t+8 , a t+8 , r t+8 , s t+9 ).

[0052] The action trajectory of the entire reinforcement learning can be divided into single-step action features, local action features, and global action features. Combining the LSTM with the sparse causal temporal attention mechanism, the policy network can better extract the feature information in such trajectories. In this policy network, the LSTM is responsible for extracting the temporal dependence features in single-step, local, and global actions from the entire reinforcement learning trajectory, while the sparse causal temporal attention mechanism efficiently models the dependence relationship between local action features through sparse connections within the local window.

[0053] Step 4) By improving the storage strategy of the original experience pool of the SAC algorithm, a method of post hoc experience recombination is proposed, a phased experience pool structure is designed, and the "recombination" mechanism is combined to improve the utilization efficiency of the algorithm for successful samples. The specific steps are as follows:

[0054] When designing the experience pool, the Future strategy is selected. The Future strategy randomly selects k states s t satisfying t < i < T from the trajectory sequence where the current state s i is located as the target set G. In the grasping operation, the stage targets are first selected, which are the target g reach-stage in the approaching stage, the target g grasp-stage in the grasping stage, and the target g lift-stage in the lifting stage respectively; Therefore, the target set G of the Future strategy can be defined as:

[0055] G = [g reach-stage , g grasp-stage , g lift-stage (5)

[0056] Then, the reward is calculated for each stage target using Equation (6).

[0057] r t = [s t+1 = g] (6)

[0058] Through the above design, when the agent has not reached the current stage target after the exploration within a limited number of time steps, the reward value of the stage target after the current time step t can be used as the reward value of the current time step t, thus completing the modification of the experience pool data. The experience data processed by the HER algorithm is as shown in Equation (7):

[0059] (s t ||g, a t , r t , s t+1 ||g) (7)

[0060] If the agent explores to the end within a certain episode and belongs to the experience in the approaching stage or the successfully grasped experience, then according to the algorithm process of HER, they will be converted into the experience of the upper layer. However, for the experience of successfully lifting at the top layer, the HER (Hindsight Experience Replay) experience pool will not modify it. It is the experience of task success, but in the entire experience pool, the proportion of this part is the smallest, which also leads to insufficient utilization of this part of "successful" experience during algorithm training. At this time, the "reorganization" mechanism is introduced, and part of the successfully lifted experience and the successfully grasped experience are taken out for "reorganization".

[0061] As Figure 4 shown, the position k of two trajectories (experiences) is selected as the splitting point (the moment k is when the agent just grasps the target object), and the two trajectories are divided into the first half and the second half. Keep the first half of the successfully grasped trajectory (s1, a1, r1, s2),...,(s k , a k , r k , s k+1 ), intercept a corresponding second half with the starting point s k from the successfully lifted trajectory, and splice this part of the trajectory to the first half of the successfully grasped trajectory, so as to generate a "reorganized" trajectory.

[0062] To sum up, by combining hindsight experience replay (HER) and adding the reorganization method, a new experience pool can be obtained.

[0063] When updating the value network in step 5), aiming at the problem of policy oscillation in the later stage of algorithm training, an adaptive conservative Q-learning value network update method is designed, and the strength of the regularization term is dynamically adjusted through an adaptive adjustment mechanism based on temporal difference error and average reward. The specific steps are as follows:

[0064] The optimization objective function of the original conservative Q-learning is as shown in Equation (8), where the coefficient η of the regularization term is the weight of the regularization term, and it is a hyperparameter;

[0065]

[0066] The regularization term in Equation (8) consists of two terms. One is the □ s～D,a～μ(a∣S) [Q(s,a)] potential term. The potential term is the expectation of the Q value on the potential distribution. Generally, by adding random noise to the actions generated by the behavior policy distribution to generate a potential action distribution, actions outside the behavior policy distribution can be efficiently explored in the high-dimensional space; the commonly used formula for generating potential actions is:

[0067]

[0068] where a is the action generated by the behavioral policy; ò is Gaussian noise; N(0,σ 2 ) is a Gaussian distribution with a mean of 0 and a variance of σ 2 ; a′ is the action after adding noise, which is often used for policy smoothing or target policy generation;

[0069] Another term in the regularization term in Equation (8) is the actual term, and the actual term is the expected value of Q on the behavioral policy distribution;

[0070] The definition of TD error is based on the difference between the output of the current Q network and the TD target value:

[0071] TD target = r + γ·Q target (s′,π target (s′)) (10)

[0072] TD error = Q(s,a) - TD target (11)

[0073] where TD target is the Temporal Difference (TD) target value; r is the reward for the current step; γ is the discount factor, which controls the influence degree of future rewards; Q target is the output of the target Q network, representing the state-action value function; s′ is the next state; π target (s′) is the action selected by the target policy in the next state;

[0074] To facilitate dynamic adjustment of the regularization weight, TD error is normalized:

[0075]

[0076] where N is the size of the sampling batch, and the clamp(x,0,1) function can limit the mean of TD error within the range of [0,1];

[0077] The regularization term weight η is dynamically adjusted according to normalized_TD error , and the formula is:

[0078]

[0079] where: R avg is the average reward of multiple episodes of the agent. In actual application, it is the average of the rewards of 4 episodes; R threshold is the reward threshold, λ min and λmax are the minimum and maximum values of the regularization weights, set to 0 and 0.2 respectively;

[0080] Regarding R threshold The method selected in practical applications is as follows: The design of the reward function for a complex task is carried out in stages. After the agent interacts with the environment, the reward for a single time step needs to be normalized, which facilitates algorithm comparison. Usually, there is a scaling factor scale for each stage. Usually, this scale is the reward after the completion of this stage. The coefficients of each stage satisfy:

[0081]

[0082] n represents the last stage before the task is successfully completed. Therefore, in order to enable the agent to explore more stably when approaching the task completion, the selection of R threshold can be according to Equation (15):

[0083]

[0084] where episode steps represents the number of time steps in each episode, that is, the total number of steps the agent needs to interact with the environment within an episode;

[0085] When R avg ≤R threshold it is the initial stage of training, and the reward value is low. To enhance the exploration ability, the regularization weight η = λ min ; When R avg >R threshold it is the later stage of training, and the reward value is high. The agent gradually converges, and the regularization weight η is dynamically adjusted according to normalized_TD error ;

[0086] Combined with the dynamically adjusted η, the optimization objective function of the adaptive conservative Q-learning is:

[0087]

[0088] Example 1

[0089] The task scenarios provided by robosuite can be used to simulate the grasping operation of this method. Before the simulation, the relevant reward function needs to be designed. Taking the Lift scenario as an example, the design process of the reward function is as follows:

[0090] For the grasping task of the arm-mobile robot, a reward function combining sparse reward and dense reward is designed, and the specific logic is as follows: When the task is completed (the sparse reward condition is met), the sparse reward is directly given; if the task is not completed and the dense reward is enabled, the agent is guided to complete the task step by step through the proximity reward, orientation reward, and grasping reward. The overall definition of the reward function is as follows:

[0091]

[0092] Among them, R sparse represents the sparse reward; R shaping represents the dense reward. The following details each component of the reward function.

[0093] R sparse The sparse reward is used to provide clear feedback after the task is completed, indicating the achievement of the task goal. For the grasping operation, when the arm-mobile robot lifts the target object, a fixed reward value of 2.25 is provided. Its definition is as follows:

[0094]

[0095] R shaping The dense reward guides the agent to complete the task step by step by refining the intermediate process of the grasping task. The dense reward consists of the following three parts:

[0096] 1) Proximity reward R reaching

[0097] To encourage the agent to gradually approach the target object, a distance-based proximity reward is designed. The proximity reward is designed according to the Euclidean distance d between the grasping center and the center position of the target object. The specific definition is:

[0098] R reaching = 1 - tanh(10.0·d) (19)

[0099] where d is the Euclidean distance from the grasping center of the dexterous hand to the target object. When the grasping center of the dexterous hand gradually approaches the target object, the reward value gradually increases, thereby guiding the arm-mobile robot to gradually learn the approaching operation.

[0100] 2) Orientation reward R orientation

[0101] To ensure that the end effector can be adjusted to the desired pose of the target object, an orientation reward based on the rotation angle deviation is designed. The orientation reward is designed according to the relative rotation angle θ x in the x-axis direction between the grasping center and the target object. The specific definition is:

[0102] R orientation = (1 - tanh(0.1·|θ x-θ desired |))·0.25 (20)

[0103] where θ desired = 120° is the desired target angle. When θ desired = 120°, it can make the z-axis of the grasping center perpendicular to the desktop. This design can reduce the exploration of the agent in the orientation dimension. By rewarding orientations with smaller deviations, the end effector of the robotic arm is encouraged to adjust to the ideal pose.

[0104] 3) Grasping reward R grasp

[0105] To further guide the agent to complete the grasping action, a fixed reward of 0.25 is provided when the fingertip of the thumb of the dexterous hand touches the fingertip of any one of the other four fingers while grasping the cube. It is defined as:

[0106]

[0107] Combining the above dense reward parts, the dense reward R shaping can be defined as:

[0108] R shaping = R reaching + R orientation + R grasp (22)

[0109] The final reward value R is normalized and scaled as needed. The formula is:

[0110]

[0111] The role of reward_scale is to normalize and scale the reward value to ensure that the range of the reward value adapts to the optimization requirements of the reinforcement learning algorithm. In practical applications, it is generally necessary to normalize the reward. Therefore, reward_scale is set to 1 here to ensure that the reward value for a single time step does not exceed 1. This setting ensures the consistency of the reward range between different tasks, which not only helps the model to learn stably but also ensures the fairness of the comparative experiment.

[0112] After completing the above reward function design and training the algorithm process of the present invention in the task scenario of the simulation environment, the related grasping operations of the arm-mobile robot can be realized, such as Figure 5 the cube grasping task shown and Figure 6 the nut assembly task shown.

[0113] In addition Figure 7 the comparison of the algorithm convergence speed is shown in Figure 8The following shows the comparison of algorithm stability. The results of the two figures indicate that the present invention has a faster convergence rate and better stability.

[0114] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit them. Although the present invention has been described in detail with reference to the preferred embodiments, those of ordinary skill in the art should understand that the technical solutions of the present invention can be modified or equivalently replaced without departing from the purpose and scope of the technical solutions of the present invention, and they should all be covered within the scope of the claims of the present invention.

Claims

1. A grasping method for an arm-mobile robot based on deep reinforcement learning, characterized in that, The method includes the following steps: 1) Define the arm-mobile robot state space S, action space A, and reward function R(s t , a t ), where s t is the state at time t, and a t is the action to be executed by the agent at time t, where s t ∈ S, a t ∈ A; The state space S is: a 50-dimensional vector, the angles s of 18 joints qpos and the angular velocities s qvel , the 3-dimensional vector position s of the grasping center of the five-fingered hand grip-pos and the 4-dimensional vector orientation s grip-quat , the 3-dimensional vector position s of the object being grasped object-pos and the 4-dimensional vector orientation s object-quat ; The action space A is: a 12-dimensional vector, including the 6-dimensional vector of the attitude of the tool center at the end of the robotic arm a robot incremental form control, and the 6-dimensional vector position a of the dexterous hand hand ; 2) Initialize the value network Q θ , the policy network and related parameters (such as the regularization coefficient range λ min and λ max , the reward threshold R threshold etc.); 3) In each episode, the main policy network π(s t ) generates an action a t . After interacting with the environment, state transition data (s t , a t , r t , s t+1 ) is obtained; 4) Use the experience replay technique to assign a new target g to the experience data and modify the corresponding reward r, and the experience becomes (s t ||g,a t ,r t ′,s t+1 ||g), and store it in the hierarchical experience pool (buffer_1, buffer_2, buffer_3). The recombination module recombines the data in the experience pool to generate new experience samples for training; 5) Calculate the average reward of the last 4 Episodes. Enter the update phase, randomly sample data from the experience pool, calculate the target TD value and TD error. After normalizing the TD error, dynamically adjust the regularization weight η according to the average reward value and TD error. Optimize Q by minimizing the loss function of the regularization term, and at the same time update the policy network by maximizing the value network value Q(s,π(s)) θ while maximizing the value network value Q(s, π(s)) to update the policy network 2. The grasping method of an arm-mobile robot based on deep reinforcement learning according to claim 1, characterized in that In step 3), in each Episode, the agent generates an action a t through the main policy network π(s t ), and after interacting with the environment, obtains the state transition data (s t , a t , r t , s t+1 ). Here, aiming at the problem that the traditional fully connected policy network structure is difficult to capture complex time-dependent relationships, a policy network structure based on a sparse causal temporal self-attention mechanism is proposed; Reduce the attention calculation complexity through local window sparse connection and cascade it with LSTM to construct a new type of policy structure. The core of the entire policy network structure lies in the sparse causal temporal attention mechanism, which consists of multiple sub-modules. By gradually expanding the receptive field layer by layer, it captures local and global temporal dependencies; Specifically, it includes the following steps: 1) Local window mechanism: The core part of the sparse causal temporal attention mechanism, mainly used to reduce the complexity of attention calculation; this mechanism divides the input time series into several non-overlapping windows of size W, calculates the attention within each window, and restricts the attention calculation to within the window range. This design significantly reduces the calculation complexity while concentrating on capturing strong dependencies between local time steps. The attention calculation formula within the window is: Among them, the query Q (Query), key K (Key), and value V (Value) are generated by the linear projection of the input feature X: Q h = XW q , K h = XW k , V h = XW v (2) Among them, is a learnable linear projection matrix, and d k is the dimension of the key vector; 2) Causal mask mechanism: It is another key component in the design of the sparse causal temporal attention mechanism, used to ensure the causal constraint of the model; through the mask matrix, the model can shield the information of future time steps and ensure that the calculation of the current time step only depends on the current and previous time steps; the mask matrix is defined as: In the attention calculation, the mask matrix is added to the dot product result of the key and the query to shield the influence of future time steps. The updated attention matrix calculation formula is: This design ensures causality in time series tasks and provides a guarantee for causal modeling in the policy network and value network in reinforcement learning; 3) Layer normalization and skip connection: The layer normalization (LayerNorm) and skip connection (Residual Connection) mechanisms are introduced in the sparse causal temporal attention mechanism module; layer normalization helps to stabilize the training process, while skip connection effectively alleviates the vanishing gradient problem in deep networks and retains the input information; 4) Multi-layer perceptron sub-module: There are two layers of multi-layer perceptrons after the Mask calculation module, which uses non-linear activation functions to further enhance the feature expression ability; this design improves the non-linear modeling ability of the model and makes it more flexible in time series modeling; 5) Gradually expand the window size layer by layer: In the multi-layer design of the sparse causal temporal attention mechanism, the window size expands layer by layer. The lower layer mainly captures short-term dependencies, while the upper layer captures global dependencies with a long time span by expanding the receptive field; this design not only optimizes the calculation efficiency but also ensures the accuracy of dependency modeling; In the sparse design, Block 1 with a window size of 2 mainly processes short-term dependencies, Block 2 with a window size of 4 processes medium-term dependencies, and Block 3 with a window size of 8 is responsible for capturing dependencies with a longer time span; through this design, the time complexity of the module is O(ρTWC), where ρ represents the sparsity, and usually ρ < 1; T is the sequence length, W is the window size, and C is the feature dimension; Before the policy network is improved, the input required for each update of the policy network is usually "single-frame" data, that is, the batch tuple (s t , a t , r t , s t+1 ); After the policy network is improved, since the input needs to match the sequence input format of the LSTM and attention mechanisms, the input of the improved policy network requires "multi-frame" data, that is, the input needs to be changed to the batch tuple (s t , a t , r t , s t+1 )…(s t+8 , a t+8 , r t+8 , s t+9 ); The action trajectory of the entire reinforcement learning can be divided into single-step action features, local action features, and global action features. Combining the LSTM with the sparse causal temporal attention mechanism, the policy network can better extract the feature information in such trajectories. In this policy network, the LSTM is responsible for extracting the temporal dependence features in single-step, local, and global actions from the entire reinforcement learning trajectory, while the sparse causal temporal attention mechanism efficiently models the dependence relationship between local action features through the sparse connection method within the local window.

3. A grasping method for an arm-mobile robot based on deep reinforcement learning according to claim 1, characterized in that, In step 4), by improving the storage strategy of the original experience pool of the SAC algorithm, a method of post hoc experience recombination is proposed, a phased experience pool structure is designed, and the "recombination" mechanism is combined to improve the utilization efficiency of the algorithm for successful samples. The specific steps are as follows: Select the Future strategy when designing the experience pool. The Future strategy randomly selects k states s that satisfy t < i < T from the trajectory sequence where the current state s t is located as the target set G. In the grasping operation, the stage targets are first selected, which are the target g i in the approaching stage, the target g reach-stage in the grasping stage, and the target g grasp-stage in the lifting stage; therefore, the target set G of the Future strategy can be defined as: lift-stage ; G = [g reach-stage , g grasp-stage , g lift-stage (5) Then, use Equation (6) to calculate the rewards for the goals of each stage. r t = [s t+1 = g] (6) Through the above design, when the agent has not reached the goal of the current stage after the exploration ends within a limited number of time steps, the reward value of the goal of the stage after the current time step t can be used as the reward value of the current time step t, thus completing the modification of the experience pool data. The experience data processed by the HER algorithm is as shown in Equation (7): (s t ||g,a t ,r t ,s t+1 ||g) (7) If the agent's exploration ends within an episode and belongs to the experience of approaching the stage or successfully grasping the experience, then according to the algorithm process of HER, they will be converted into the experience of the upper layer. However, for the experience of successfully lifting at the top layer, the HER experience pool will not modify it. It is the experience of task success, but in the entire experience pool, the proportion of this part is the smallest, which also leads to insufficient utilization of this part of the "successful" experience during algorithm training. At this time, "recombination" is introduced, and part of the successfully lifted experience and the successfully grasped experience are taken out for "recombination". The position k of the two trajectories is selected as the splitting point. The moment k is the moment when the agent just grabs the target object. The two trajectories are divided into the first half and the second half. The first half of the successfully grabbed trajectory (s1, a1, r1, s2),...,(s k , a k , r k , s k+1 ) is retained. A section corresponding to the starting point s k of the second half is intercepted from the successfully lifted trajectory, and this part of the trajectory is spliced to the first half of the successfully grabbed trajectory, thereby generating a "recombined" trajectory. Combining the post hoc experience replay (HER) and adding the recombination method can obtain a new experience pool.

4. A method for a robotic arm-mobile robot grasping based on deep reinforcement learning according to claim 1, characterized in that In step 5), when updating the value network, aiming at the problem of policy oscillation in the later stage of algorithm training, an adaptive conservative Q-learning value network update method is designed. Through the adaptive adjustment mechanism based on the temporal difference error and the average reward, the intensity of the regularization term is dynamically adjusted. The specific steps are as follows: The optimization objective function of the original conservative Q-learning is as shown in Equation (8), where the coefficient η of the regularization term is the weight of the regularization term, and it is a hyperparameter. The regularization term in Equation (8) consists of two terms. One is the latent term. The latent term is the expected Q value on the latent distribution. Generally, by adding random noise to the actions generated by the behavioral policy distribution, a latent action distribution can be generated, which can efficiently explore actions outside the behavioral policy distribution in the high-dimensional space. The commonly used formula for generating latent actions is: where a is the action generated by the behavioral policy; ò is Gaussian noise; N(0,σ 2 ) is a Gaussian distribution with a mean of 0 and a variance of σ 2 ; a′ is the action after adding noise, which is often used for policy smoothing or target policy generation; Another term of the regularization term in Equation (8) is the actual term, which is the expected Q-value over the behavioral policy distribution; The definition of the TD error is based on the difference between the output of the current Q network and the TD target value: TD target = r + γ·Q target (s′, π target (s′)) (10) TD error = Q(s,a) - TD target (11) where TD target is the temporal difference (TD) target value; r is the reward at the current step; γ is the discount factor that controls the influence of future rewards; Q target is the output of the target Q-network, representing the state-action value function; s′ is the next state; π target (s′) is the action selected by the target policy in the next state; To facilitate dynamic adjustment of the regularization weight, TD error is normalized as follows: where N is the size of the sampling batch, and the clamp(x, 0, 1) function can limit the mean value of TD error within the range of [0, 1]; The weight η of the regularization term is dynamically adjusted according to normalized_TD error The formula is as follows: Where: R avg is the average reward of multiple episodes of the agent, which is the average of the rewards of 4 episodes in practical applications; R threshold is the reward threshold, λ min and λ max are the minimum and maximum values of the regularization weight, set to 0 and 0.2 respectively; Regarding R threshold The method selected in actual applications is as follows: The design of the reward function for a complex task is carried out in stages. After the agent interacts with the environment, the reward for a single time step needs to be normalized, which facilitates algorithm comparison; Usually, there is a proportionality coefficient scale for each stage. Usually, this scale is the reward after the completion of this stage. The coefficients of each stage satisfy: n represents the last stage before the successful completion of the task. Therefore, to enable the agent to explore more stably when approaching the task completion, the selection of R threshold can be based on Equation (15): where episode steps represents the number of time steps for each episode, that is, the total number of steps an agent needs to interact with the environment within one episode; When R avg ≤R threshold it is the initial stage of training, the reward value is low. To enhance the exploration ability, the regularization weight η = λ min ; When R avg >R threshold it is the later stage of training, the reward value is high, the agent gradually converges, and the regularization weight η is dynamically adjusted according to normalized_TD error ; Combined with the dynamically adjusted η, the optimization objective function of the adaptive conservative Q-learning is:

Images