A mechanical arm grabbing method based on an improved dreamer framework

Abstract

The application discloses a kind of mechanical arm grabbing method based on improved Dreamer framework, it is related to the field of intelligent production and manufacturing, first by modeling and observing action space and reward function design to build autonomous decision reinforcement learning problem model for mechanical arm.It is second to build a set of basic control interface and be equipped with mechanical arm operation simulation platform, it can be combined with various reinforcement learning algorithms.Then, the world model with forward prediction proposed effectively simulates and predicts the production environment, so that the world model learns forward and backward environment dynamics at the same time, improves the utilization rate of training data obtained by interaction.Finally, the improved world model is embedded in the Dreamer framework of deep reinforcement learning algorithm, and the mechanical arm is trained online on the platform built by interaction, and finally the convergence speed, sample efficiency and performance of the training are good, which is of great significance for realizing the autonomous decision-making of mechanical arm in intelligent manufacturing in reality.

Description

Technical Field

[0001] This invention relates to the field of intelligent manufacturing, specifically to a control decision-making method for a robotic arm intelligent agent based on environmental models and deep reinforcement learning. Background Technology

[0002] Intelligent manufacturing is a key direction for innovation-driven transformation and upgrading of my country's manufacturing industry. Robotic arms are widely used in various industrial manufacturing fields, including smart workshops where grasping scenarios are common. They possess significant theoretical research value and application prospects, and achieving autonomous decision-making by robotic arms is a requirement of intelligent manufacturing. In recent years, data-driven methods based on deep reinforcement learning have attracted widespread attention due to their end-to-end characteristics, environmental adaptability, and learnable strategies. However, to date, the sampling efficiency of general reinforcement learning algorithms remains relatively low, limiting their widespread application in robotics, especially for tasks with high-dimensional observations such as image input. Model-based reinforcement learning, through dynamic modeling of the environment, allows robots to interact with virtual environments, effectively improving sampling efficiency. Among the latest advances in image-based reinforcement learning algorithms, Dreamer, as an advanced framework, constructs an additional world model to simulate the environment while learning a strategy, representing a typical model-based reinforcement learning algorithm.

[0003] World models can effectively improve data utilization in applications such as robot simulation and video games. Learning world models from past experiences allows robots or intelligent agents to imagine future environments, reducing the number of interactions with real-world scenarios. Currently, world models are widely used in model-based reinforcement learning. The cyclic state-space-based world model proposed by Hafner et al. effectively predicts dynamic changes in the environment by modeling both the deterministic and stochastic components, enabling the training of strategies to solve tasks in a compact latent space.

[0004] In recent years, reinforcement learning algorithms based on world models have developed rapidly. In 2018, Hafner et al. proposed PlaNet, which constructed a Recurrent State-Space Models (RSSM) model to map the environmental image state to the latent space representation and combined it with the MPC algorithm to solve vision-based control problems, improving sample efficiency and achieving good results on DM_Control. Later, Hafner et al. (Dream to control: Learning behaviors by latent imagination[J]. arXiv preprint arXiv:1912.01603,2019) combined PlaNet with the Actor-Critic algorithm, eliminating the need for online planning, and proposed Dreamer. Later, they further proposed its improved version Dreamer-v2, which solved the pixel-based human-like motion problem for the first time. In 2022, Wu et al. (Daydreamer: World models for physical robot learning[C] / / Conference on RobotLearning.PMLR,2023:2226-2240) proposed DayDreamer, applying the Dreamer algorithm to actual robots and successfully completing tasks such as quadrupedal robot walking, robotic arm grasping and placement. In 2023, Hafner et al. (Mastering diverse domains through world models, arXiv preprint arXiv:2301.04104,2023) proposed the third version of the Dreamer algorithm, DreamerV3. Using the same set of parameters, it can achieve excellent results across a wide range of tasks and has long-term planning capabilities. In the Minecraft game environment, it realizes the process of learning from scratch and independently mining diamonds.

[0005] The decision-making models listed above can be applied to the training and execution of robot reinforcement learning tasks, but there are still some problems. The main problem is that the world model only focuses on forward transitions in the dynamic transition of the environment, while predicting backward transitions is also helpful in capturing the dynamic laws of the environment. Therefore, there is still room for improvement in terms of sample efficiency and convergence speed during training.

[0006] Therefore, improving the sample efficiency and convergence speed of the world model during training is an urgent problem to be solved. Summary of the Invention

[0007] In view of this, the present invention provides a robotic arm grasping method based on the improved Dreamer framework, which can add a forward prediction link to the world model, making the world model more effective in simulating and predicting the environment, while learning both forward and backward environmental dynamics, thereby improving sample efficiency and training convergence speed.

[0008] To achieve the above objectives, this invention provides a robotic arm grasping method based on an improved Dreamer framework. This method incorporates a forward prediction step into the world model. The forward prediction step predicts the latent space state of the previous time step using the current state and action. KL divergence is used to narrow the distance between the predicted latent space state of the previous time step and the posterior state distribution of the previous time step. Specifically, the method includes the following steps:

[0009] S1, randomly initialize the world model parameters φ, the actor network parameters θ, and the critic network parameters ψ.

[0010] Randomly generate the location and placement target of goods in the production environment; add the interaction data between the agent and the environment to dataset D.

[0011] Set hyperparameters, including the forward prediction loss function weights β. pre Sampling batch length L, policy training imagined trajectory length H, and model network learning rate α.

[0012] S2, if the current number of steps the robotic arm has interacted with the environment has reached the set maximum number of steps (MaxSteps), then training ends; otherwise, if the current round ends, the environment is reset, and observations are given. step If the round is in progress, proceed to the next step S3.

[0013] S3, determine whether the current number of steps the agent has interacted with the environment should be used for training. If training is required, extract trajectories of batch B and length L from the current data storage area D. Where a t o t r t These represent the actions, observations, and rewards collected during the interaction process, where t is time and k is the starting time of the currently extracted trajectory. The hidden state s of the extracted trajectory at each time step is calculated using a world model. t =(h t , z t );h t and z t These represent the deterministic and random components of the hidden state, respectively.

[0014] Based on the hidden state s at each time step calculated from the collected data t Using a world model to generate imagined trajectories in latent space Predict the reward for the corresponding state using reward prediction models and state value models. and state value v ψ (st), calculate the λ-Return estimate Update the Actor parameter θ based on Reinforce, and update the Critic parameter ψ based on the gradient descent method using the reward estimated by regression; proceed to the next step S4.

[0015] S4, Calculate the latent space state s of the current interaction step. step The action 'a' to be taken in the current interaction step is obtained by inputting it into the Actor model. step Add noise needed for exploration to the action, then execute the action to obtain the reward r for the current interaction step. step And the next step of observation. step+1 ; will o step a step r step and o step+1 Add to the data cache pool and increment the current interaction step count by one.

[0016] S5, repeatedly execute operations from S2 to S4 until the preset maximum number of steps is reached; finally, the optimized model parameters for the current scale problem include world model parameters φ, Actor network parameters θ, and Critic network parameters ψ. Using the optimized model parameters, the Dreamer framework is obtained, and its Actor policy network outputs reasonable actions to achieve successful grasping of the target by the robotic arm.

[0017] Furthermore, the world model's structure includes a sequence model, an encoder, a dynamic prediction model, a reward prediction model, and a decoder; the goal of the dynamic prediction model is to capture the patterns of environmental change, and the added forward prediction stage minimizes the Kullback-Leibler divergence between the forward prediction distribution and the posterior distribution of the previous state.

[0018] The forward prediction network is implemented using a multilayer perceptron (MLP), and the forward prediction loss function L is added to the loss function L(φ) in the world model. pre (φ), in calculating L pre When (φ) is reached, a strategy of stopping the gradient is adopted for the posterior distribution so that the predicted state distribution approximates the posterior distribution in one direction.

[0019] Furthermore, the loss function L(φ) in the world model is:

[0020]

[0021] L pre (φ)=max(1,KL[sg(q φ (z t|(h t x t )))||p φ (z t |(h t+1 Z t+1 a t )]);

[0022] in This indicates averaging; T represents the total number of time steps in the currently extracted trajectory; β pred L represents the prediction coefficient. pred (φ) represents the prediction loss function; β dvn L represents the dynamic coefficient; dyn (φ) represents the dynamic loss function; β rep L represents the characterization coefficient; rep (φ) represents the loss function; β pre L represents the weights of the forward prediction loss function; pre (φ) represents the forward prediction loss function; sg() represents the stopping gradient; p φ This represents an environmental dynamics prediction model, incorporating a multilayer perceptron (MLP) to represent h. t+1 , z t+1 a t Implement z as input t Forward prediction, x t For the original observation; q φ Indicates encoder, in h t x t Implement z as input t Forward prediction; KL[*] represents the KL divergence of *.

[0023] Furthermore, the weights of the forward prediction loss function are set to β. pre =1.0.

[0024] Furthermore, a t o t r t These are the actions, observations, and rewards collected during the interaction process; specifically:

[0025] Observations collected during the interaction t Includes: RGB images obtained from a third-person camera RGB The set of poses of each joint of the robotic arm, world coordinates of the robotic arm end effector, and opening and closing degrees of the end effector gripper;

[0026] Actions collected during the interaction a t Specifically, it employs a discrete motion space, including end-effector movement and gripper state switching;

[0027] Rewards collected during the interaction t Specifically: for scenarios where the robotic arm needs to place an object to a randomly specified target point, a dense reward scheme is adopted; the reward is determined based on the negative number of the Euclidean norm distance to the target point;

[0028] If the robotic arm places the object within the defined error range of the target point, it is considered to have reached the target position.

[0029] Furthermore, the observations collected during the interaction process... t Specifically:

[0030] When a task requires the robotic arm to place an object at a designated location, the target position coordinates are also added to the observation, and the observation is... t Defined as obs = [I RGB , θ, x e y e , z e , δ], where θ=[θ1,...,θ n [x] represents the n joint angles of an n-axis robotic arm. e y e , z e ] represents the world coordinates of the center of the end gripper, and δ∈[0,1] represents the degree of opening and closing of the gripper.

[0031] Beneficial effects:

[0032] This invention presents an autonomous decision-making method for robotic arms based on an improved Dreamer framework. By building a robotic arm operation platform, it provides control and reinforcement learning algorithm interfaces. In this method, robotic arm control is modeled as a Markov decision model, encompassing the design of observation, action, and reward functions. A forward prediction component is added to the world model, enabling it to more effectively simulate and predict the environment. Simultaneously, it learns both forward and backward environmental dynamics, improving sample efficiency and training convergence speed. Attached Figure Description

[0033] Figure 1 This is a schematic diagram of the overall scene logic and algorithm architecture in an autonomous decision-making method for robotic arms based on an improved Dreamer framework provided by the present invention.

[0034] Figure 2 This is a schematic diagram of a world model with forward prediction in an autonomous decision-making method for robotic arms based on an improved Dreamer framework provided by the present invention. Detailed Implementation

[0035] The present invention will now be described in detail with reference to the accompanying drawings and embodiments.

[0036] The overall scenario logic and algorithm architecture diagram of the robotic arm autonomous decision-making method based on the robotic arm operation platform and the improved Dreamer framework provided in this invention are shown below. Figure 1 As shown, the simulation platform built on Coppeliasim serves as a reinforcement learning environment. It can provide camera images and robotic arm poses as observations, and can also receive actions from the policy model via a remote API, execute them, and provide reward feedback. Observations, actions, and rewards are stored in a buffer for model training. The world model updates its parameters using representation learning and other methods based on extracted trajectory fragments, and then generates imagined trajectories in the latent space at each state node for the policy model to train. Model training and agent-environment interaction occur alternately.

[0037] The world model with forward prediction provided by this invention is as follows: Figure 2 As shown, at each time step, this model, in addition to positively predicting the state of the next time step, also needs to additionally predict the state of the previous time step. Specifically, the current hidden state and the action value of the previous time step are concatenated and passed through a feedforward network to obtain the predicted distribution of the previous state. During training, the distance between the predicted distribution and the posterior distribution of the previous state is narrowed. The optimization of the distribution distance is also achieved through KL divergence. The feedforward network is implemented using an MLP instead of a recurrent neural network because the deterministic state of later time steps already contains useful information from previous time steps through the GRU in the architecture. Specifically, the Dreamer framework, a deep reinforcement learning algorithm based on an improved world model, is provided in this embodiment of the invention. The world model structure includes a sequence model, an encoder, a dynamic prediction model, a reward prediction model, and a decoder. The goal of the dynamic prediction model is to capture the patterns of environmental changes. The added feedforward prediction step minimizes the Kullback-Leibler divergence between the feedforward prediction distribution and the posterior distribution of the previous state. The feedforward prediction network is implemented using a multilayer perceptron (MLP), and a feedforward prediction loss term l is added to the loss function L(φ) in the world model. pre (φ), with weights set to β pre =1.0, in calculating L pre When (φ), a gradient-stopping strategy is adopted for the posterior distribution so that the predicted state distribution can unidirectionally approximate the posterior distribution.

[0038] This invention provides an autonomous decision-making method for robotic arms based on an improved Dreamer framework, comprising the following steps:

[0039] S1: Randomly initialize the world model parameters φ, Actor network parameters θ, and Critic network parameters ψ. Randomly generate the locations of goods and placement targets in the production environment. Add the agent's interaction data with the environment to dataset D, and set the algorithm's hyperparameters: forward prediction loss function weights β.pre Sample batch length L, policy training imagined trajectory length H, model network learning rate α, etc.

[0040] S2, if the current agent's interaction steps with the environment have reached the set maximum number of steps (MaxSteps), then training ends. Otherwise, if the current round ends, the environment is reset, and observations are given. step If the round is in progress, the next step will be executed.

[0041] S3, determine whether the current number of steps the agent has interacted with the environment should be used for training. If training is required, extract trajectories of batch B and length L from the current data storage area D. Where a t o t r t These represent the actions, observations, and rewards collected during the interaction; the hidden state s at each step is calculated using a world model. t =(h t , z t ).

[0042] The loss function is defined as follows: The world model is updated through representation learning.

[0043] In the formula q φ Indicates encoder;

[0044] L pre (φ)=max(1,KL[sg(q φ (z t |(h t x t )))||p φ (z t |(h t+1 , z t+1 a t )]), where sg() represents stopping the gradient, p φ Represents an environmental dynamics prediction model, x t For the original observation, h t and z t These represent the deterministic and stochastic components of the hidden state, respectively; the hidden state s at each time step is calculated based on the collected data. t Using a world model to generate imagined trajectories in latent space Predict the reward for the corresponding state using reward prediction models and state value models. and state value v ψ (s t ), calculate the λ-Return estimate Update the Actor parameter θ based on Reinforce, and update the Critic parameter ψ based on the reward estimated by regression using gradient descent. Otherwise, proceed to the next step.

[0045] S4, Calculate the latent space state s of the current step. step The action 'a' to be taken in the current step is obtained by inputting it into the Actor model. step Add noise needed for exploration to the action, then execute the action to obtain the reward r for the current step. step And the next step of observation. step+1 . o step a step r step and o step+1 Add to the data cache pool and increment the current step count by one.

[0046] S5, repeat operations S2 to S4 until the agent's interaction steps with the environment reach the preset maximum number of steps. Finally, the optimized model parameters φ, θ, and ψ are obtained for the current scale of the problem.

[0047] In this embodiment of the invention, the design of the robotic arm modeling and observation, action space, and reward function is provided. The observation space for reinforcement learning is designed as an RGB image I obtained from a third-view camera. RGB This includes the poses of each joint of the robotic arm, the world coordinates of the robotic arm's end effector, and the opening and closing degrees of the end effector gripper. When the task requires the robotic arm to place an object in a specified position, the target position coordinates are also added to the observation, and the observation is defined as obs = [I RGB , θ, x e y e , z e , δ], where θ=[θ1,...,θ n [x] represents the n joint angles of an n-axis robotic arm. e y e , z e [ ] represents the world coordinates of the center of the end effector gripper, and δ∈[0,1] represents the degree of gripper opening and closing. The motion uses a discrete motion space, including end effector movement and gripper state switching. For scenarios where the robotic arm needs to place an object to a randomly specified target point, we employ a dense reward scheme. The reward is determined based on the negative Euclidean norm distance to the target point. If the robotic arm places the object within the defined error range from the target point, it is considered to have reached the target position.

[0048] In summary, the above are merely preferred embodiments of the present invention and are not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A robotic arm grasping method based on an improved Dreamer framework, characterized in that, A forward prediction stage is added to the world model. This stage predicts the latent space state of the previous time step based on the current state and action. KL divergence is used to narrow the distance between the predicted latent space state of the previous time step and the posterior state distribution of the previous time step. This capture method specifically includes the following steps: S1, Randomly initialize world model parameters Actor network parameters Critic network parameters ; Randomly generate cargo locations and placement targets in the production environment; add agent-environment interaction data to the dataset. D ; Configure hyperparameters, including the weights of the forward prediction loss function. Sampling batch length L Strategy training imagined trajectory length H Model network learning rate ; S2, if the current number of steps the robotic arm has interacted with the environment has reached the set maximum number of steps. MaxSteps If the current round ends, training ends; otherwise, if the current round ends, the environment is reset and observations are provided. If the round is in progress, proceed to the next step S3; S3, determine whether the current number of steps the agent has interacted with the environment should be used for training. If training is required, start from the current data storage area. D The sampling batch is B , length is L trajectory ,in These represent the actions, observations, and rewards collected during the interaction, where t is time and k is the starting time of the currently extracted trajectory. The hidden state at each time step in the extracted trajectory is calculated using a world model. ; and These represent the deterministic and random components of the hidden state, respectively. Based on the hidden states at each time step calculated from the collected data Using a world model to generate imagined trajectories in latent space The reward prediction model and the state value model are used to predict the reward for the corresponding state. and state value ,calculate -Return estimate ,based on Reinforce renew Actor parameter The reward estimated through regression is updated based on gradient descent. Critic parameter ; Proceed to the next step, S4; S4, Calculate the latent space state of the current interaction step. The action to be taken in the current interaction step is obtained by inputting it into the Actor model. Adding noise needed for exploration to an action before performing the action yields a reward for the current interaction step. and the next step of observation ; Will , , and Add to the data cache pool and increment the current interaction step count by one; S5, repeatedly execute operations S2 to S4 until the preset maximum number of steps is reached; ultimately, the optimized model parameters for the current scale problem include the world model parameters. Actor network parameters Critic network parameters The Dreamer framework, obtained using the optimized model parameters, outputs reasonable actions through its Actor policy network to enable the robotic arm to successfully grasp the target.

2. The robotic arm grasping method based on the improved Dreamer framework as described in claim 1, characterized in that, The world model structure includes a sequence model, an encoder, a dynamic prediction model, a reward prediction model, and a decoder; the goal of the dynamic prediction model is to capture the laws of environmental change, and the added forward prediction link is to minimize the Kullback-Leibler divergence between the forward prediction distribution and the posterior distribution of the previous state. The forward prediction network is implemented using a multilayer perceptron (MLP), and the loss function in the world model is... A forward prediction loss function was added. In calculation The strategy of stopping the gradient on the posterior distribution is adopted so that the predicted state distribution approximates the posterior distribution in one direction.

3. The robotic arm grasping method based on the improved Dreamer framework as described in claim 2, characterized in that, Loss function in the world model for: ； ； in This indicates averaging; T represents the total number of time steps for the currently extracted trajectory. Indicates the prediction coefficient; Represents the prediction loss function; Represents dynamic coefficients; Represents the dynamic loss function; Represents the characterization coefficient; This represents the loss function; Indicates the weights of the forward prediction loss function; This represents the forward prediction loss function; sg () indicates stopping the gradient; This represents an environmental dynamics prediction model, incorporating a multilayer perceptron. MLP ,by To implement input Forward prediction, For original observations; Indicates encoder, To implement input Forward prediction; This represents the KL divergence of *.

4. A robotic arm grasping method based on an improved Dreamer framework as described in claim 2 or 3, characterized in that, The weights of the forward prediction loss function are set as follows: .

5. The robotic arm grasping method based on the improved Dreamer framework as described in claim 3, characterized in that, These are the actions, observations, and rewards collected during the interaction process; specifically: Observations collected during the interaction Includes: RGB images obtained from a third-person camera The set of poses of each joint of the robotic arm, world coordinates of the robotic arm end effector, and opening and closing degrees of the end effector gripper; Actions collected during the interaction Specifically, it employs a discrete motion space, including end-effector movement and gripper state switching; Rewards collected during the interaction Specifically: for scenarios where the robotic arm needs to place an object to a randomly specified target point, a dense reward scheme is adopted; the reward is determined based on the negative number of the Euclidean norm distance to the target point; If the robotic arm places the object within the defined error range of the target point, it is considered to have reached the target position.

6. The robotic arm grasping method based on the improved Dreamer framework as described in claim 5, characterized in that, Observations collected during the interaction Specifically: When a task requires the robotic arm to place an object in a designated location, the target position coordinates are also added to the observation. Defined as ,in for n Axis robotic arm n Each joint angle That is, the world coordinates of the center of the end gripper. This indicates the degree to which the grippers are open or closed.

Images