Reinforcement learning method based on mechanical arm posture activity freedom degree

Abstract

The application provides a reinforcement learning method based on a mechanical arm posture activity degree of freedom, and relates to the field of intelligent control of a mechanical arm. The method first establishes a simulation model of the mechanical arm, and then establishes a motion model and a control model thereof; then, an observation space is established, and a reward function is set based on the observation space. A reinforcement learning algorithm is built based on an SAC algorithm, and a central joint included angle sine value is introduced to expand the observation space and the reward function. Finally, grabbing is performed, and grabbing data is recorded. According to the grabbed data, value estimation is predicted, and a decision network is optimized and trained by using a loss function. Compared with the prior art, the method has better convergence characteristics and task execution capability, and has better exploration performance, better smoothness and better space efficiency.

Description

Technical Field

[0001] This invention relates to the field of intelligent control of robotic arms, specifically a reinforcement learning method based on the degrees of freedom of a robotic arm's posture and movement. Background Technology

[0002] In 2021, Zehong et al. addressed the slow convergence speed and high convergence difficulty of reinforcement learning methods in training robotic arms by proposing a method using human experience for auxiliary evaluation. This method involves collecting a certain amount of operational data of the robotic arm performing specific tasks under human control, and training a judgement agent capable of assessing the rationality of operations using supervised learning. Then, during the reinforcement learning training phase, the feedback from this judgement agent is used to guide the agent to make decisions similar to those made by humans.

[0003] In 2021, Ali et al. simplified the decision space of the intelligent agent by breaking down the robotic arm task according to the processing flow and building a behavior tree based on human prior knowledge of the processing flow, thereby completing simple human-machine collaborative tasks.

[0004] In the field of space exploration, at the 2016 ICLR conference, Tom et al. proposed to use the TD-error method to adjust the sampling frequency of data replay in the empirical data pool, so as to sample data with larger TD-error with a higher probability, that is, to estimate a part of the data that differs greatly from the actual calculation.

[0005] In 2017, Andrychowicz proposed the HER algorithm, which sets a short-term achievement goal in the environment and improves the trial-and-error reward of some low-value data by modifying the target value in past experience data, so as to obtain correct experience from the erroneous data.

[0006] In summary, the existing technologies suffer from several problems: traditional reinforcement learning methods for solving robotic arm tasks suffer from high exploration difficulty, slow convergence speed, and poor convergence performance. Simply applying reinforcement learning algorithms to solve complex robotic arm tasks is challenging. To meet the demands of complex tasks, human experience is often used to simplify the processing flow and to some extent, the action space is simplified, such as controlling only the movement of the robotic arm's end effector. However, the specific control commands for these methods are still calculated using inverse kinematics, failing to fully leverage the flexibility of reinforcement learning decision-making. Furthermore, the exploration algorithms of reinforcement learning agents have significant limitations when applied to robotic arm tasks. The PER algorithm's improvements to its underlying principles do not fully consider the physical motion characteristics of the robotic arm, and the large decision space makes this strategy difficult to implement effectively. The HER algorithm's requirements for target values ​​are difficult to generalize across various robotic arm tasks. Summary of the Invention

[0007] To address the aforementioned technical problems, this invention proposes a reinforcement learning method based on the degrees of freedom of a robotic arm's posture. Based on an evaluation index for robotic arm dexterity, it can estimate the crouching state of a multi-degree-of-freedom robotic arm using single-frame robotic arm state data, providing a relatively general exploration and optimization method for reinforcement learning training of robotic arm tasks. The specific technical solution is as follows:

[0008] S1: Components construct a 3D simulation model of the robotic arm;

[0009] S2: Set anchor points at the center of each joint of the robotic arm, adjust and control the rotation of the anchor points, and observe the dynamic changes of the robotic arm in real time to establish the motion model and control model of the robotic arm.

[0010] S3: Set up the support platform and the target block to be grabbed in the simulation scene;

[0011] S4: Setting up a reinforcement learning environment:

[0012] S41: Establishing the observation space

[0013] [Δx target ,Δy target ,Δz target [angles]

[0014] In the formula, Δx target Δy target Δz garget These represent the relative positions of the robotic arm's end effector and the target block, respectively, while angles represent the included angles of each joint of the robotic arm.

[0015] S42: Observe the relative position of the robotic arm and the target block and the included angles of each joint of the robotic arm based on the observation space;

[0016] S43: By controlling the joints of the robotic arm to rotate counterclockwise, maintain the current position, and rotate clockwise, the motion space of the robotic arm is established;

[0017] S44: Set the basic reward function R T ,

[0018] R T =-e Distance ,

[0019] In the formula, Distance is the straight-line distance between the end effector of the robotic arm and the target block;

[0020] S5: Building a reinforcement learning model based on the SAC algorithm;

[0021] S51: Introducing the sine value λ of the central joint angle expands the observation space to

[0022] [Δx target Δy target Δz target ,angles,λ]

[0023] S52: Introduce λ into R T We obtain the extended reward function R.

[0024] R=ωλ+(1-ω)R T ,

[0025] In the formula, ω is the weight of λ in the crawling task;

[0026] S53: Perform the grabbing in the simulation environment and record the experience data of each grabbing;

[0027] S54: Use the SAC algorithm to interact with the simulation scenario, obtain the experience data and store it in the experience pool, and then collect experience data (s) from the experience pool. t a t s t+1 r t The SAC algorithm is then updated via network.

[0028] Among them, s t Let a be the state at time t. t Let s be the action at time t. t+1 Let r be the state at time t+1. t Let R be the reward value obtained at time t.

[0029] S55: Based on the optimal Bellman equation, using

[0030] U t =r t +γv(s t+1 )

[0031] As s t Estimation of the actual value of the action;

[0032] In the formula, γ is the reinforcement learning reward discount factor, v(s) t+1 ) for s t+1 Predicted state value estimation;

[0033] S56: Using the Q-network in the SAC algorithm, q(s t a t ) as s t Predictive value estimation;

[0034] S57: Using MSELoss as the loss function, U t With q(s) t at The SAC algorithm's decision network is then fitted and optimized based on the fitting results.

[0035] Furthermore, the grasping result is determined by the signals collected by the sensors installed at the end of the robotic arm; if the robotic arm fails to grasp the target object after a preset number of interactions with the target object, the grasping process ends and is judged as a grasping failure.

[0036] In this technical solution, compared with the existing technology, the optimization learning method based on the degrees of freedom of the robotic arm's posture has better convergence characteristics and task execution capability; because the optimization process targets multiple joints, it has better exploration performance than methods that simplify the decision space; and the introduction of reward function and predictive value estimation makes the trajectory of the robotic arm have better smoothness and spatial efficiency. Attached Figure Description

[0037] Figure 1 This is a flowchart illustrating one embodiment of the present invention.

[0038] Figure 2 This is a training reward comparison curve between the extended observation space and the basic method in one embodiment of the present invention.

[0039] Figure 3 This is a curve showing the number of interaction steps required to complete a single round of training in one embodiment of the present invention.

[0040] Figure 4 This is a comparison curve of the training reward function and the basic method in one embodiment of the present invention.

[0041] Figure 5 This is a curve showing the number of interaction steps required to complete a single round of training in one embodiment of the present invention.

[0042] Figure 6 This is a three-view drawing of the spatial position of the robotic arm end effector during the initial training phase of one embodiment of the present invention.

[0043] Figure 7 This is a spatial position trajectory diagram of the end effector of a robotic arm when performing a grasping task on the same target in one embodiment of the present invention. Detailed Implementation

[0044] Due to the significant challenges posed by the diverse tasks and complex interactions in industrial environments, next-generation robotic arm control methods must be intelligent. Therefore, Industry 4.0 demands that robots achieve intelligent movement in complex and changing environments. As a learning algorithm that learns from experience gained through interaction with the environment, reinforcement learning exhibits excellent adaptability to dynamic environments. Consequently, it has been widely applied in robotic arm control tasks in recent years.

[0045] Reinforcement learning is a class of agent training methods based on exploration and trial and error. Specifically, a partial state of the environment is passed to the agent. After preprocessing, observations for a single interaction step are obtained. Based on these observations, the agent makes a single control decision. After execution by the environment, a partial state of the next step is obtained, thus completing a single interaction between the agent and the environment. Throughout the training process of reinforcement learning control, the agent needs to adopt certain exploration strategies during the decision-making phase to achieve better convergence and performance. Therefore, consistency of the agent's exploration strategy is a crucial issue during the training process of reinforcement learning.

[0046] To address the problems in existing technologies, such as high exploration difficulty, slow convergence speed, and poor convergence effect in solving robotic arm-related tasks using traditional reinforcement learning methods; the inability to fully leverage the flexibility of reinforcement learning decision-making; and the significant limitations of reinforcement learning agent exploration algorithms when applied to robotic arm tasks, this invention proposes a reinforcement learning method based on the degrees of freedom of robotic arm posture activities, as detailed below:

[0047] By modeling simulation models of different multi-joint robotic arms, motion and control models of different multi-joint robotic arms are established.

[0048] Step 1: By modeling simulation models of different multi-joint robotic arms, establish motion and control models for different multi-joint robotic arms.

[0049] Step 2: Based on the motion and control models of the multi-joint robotic arm, construct a state-controllable and interactive robotic arm control simulation environment, and improve other environmental requirements in the interactive environment according to the relevant requirements of the robotic arm task.

[0050] Step 3: Based on reinforcement learning algorithms, build a neural network, select the observation space required to complete the current task and the joints that need to be controlled, and construct a multi-joint robotic arm task reinforcement learning model. By issuing decision commands to the selected joints in the task, the robotic arm can gradually complete the specific task objectives as required.

[0051] Step 4: Based on the robot arm's posture data, construct a robot arm posture evaluation index and add this index to the reinforcement learning training process to improve the robot arm's performance in environmental exploration.

[0052] Step 5: Based on reinforcement learning algorithms, train the decision model of the agent so that the robotic arm can complete the required task under the guidance of the reward function.

[0053] Furthermore, the specific process of step 1 is as follows:

[0054] Step 11: Based on the selected robotic arm model for the task, construct each component of the physical robotic arm and assemble them in the model file to obtain the digital simulation model of the current robotic arm. According to the joint settings of the robotic arm, model the motion modes (rolling, rotation, translation, etc.) of each joint and limit the effective range of motion to construct the motion model of the robotic arm.

[0055] Step 12: Based on the constructed robotic arm motion model, for each key motion point, take the standardized continuous motion control command [-1, 1] as input, and use a script to realize the motion response of specific joints to the input in the selected motion mode, thereby constructing the control model in the robotic arm simulation.

[0056] Furthermore, the specific process of step 2 is as follows:

[0057] Step 21: Based on the control model completed in S1, package the relevant instructions of the robotic arm in a standardized interface form, construct the method for obtaining the state of the robotic arm, and the external interface method for controlling each joint of the robotic arm.

[0058] Step 22: Determine the task objectives to be achieved in this training session, and supplement the control and interaction rules for other relevant objects in the simulation environment. For example, supplement the position generation rules for the robotic arm's interactive objects (blocks, joysticks, etc.).

[0059] Furthermore, the specific process of S3 is as follows:

[0060] Step 31: Select the observation state required to complete the task and select the observation space of the task. The setting of the observation space is highly correlated with the specific task and usually includes: the angle of each joint of the robotic arm, the angular velocity of each joint of the robotic arm, and the relative position of the end effector of the robotic arm and the manipulated object.

[0061] Step 32: Determine the control dimensions required to complete the task and declare the action space of the task. The complexity of the action space setting is strongly correlated with the difficulty of training, and it also determines the flexibility of the robotic arm during movement to a certain extent. It usually includes the direction and speed of movement of each joint.

[0062] Step 33: Based on the SAC algorithm, construct a reinforcement learning algorithm training model and build the basic reward function for the task.

[0063] Furthermore, the specific process of S4 is as follows:

[0064] Step 41: Construct a robotic arm posture evaluation index—degrees of freedom, denoted by λ. The relative value of the degrees of freedom characterizes the degree of retraction of the multi-axis robotic arm in space. The joint with the middle number among the joints of the multi-axis robotic arm is selected as the main joint for calculation, as it plays the greatest role in the process of the robotic arm changing from a fully extended state to a retracted state. The angles of the main joints are taken into consideration, and the sine value of the angle is calculated as the index of the degrees of freedom.

[0065] Step 42: Add the constructed activity freedom evaluation metric to the training process of the reinforcement learning algorithm. There are two main ways to add it:

[0066] First, it is added to the environmental observation space as additional observations. The new observation space s' is represented as:

[0067] s′=[s,λ]

[0068] In the formula, s represents the observation space;

[0069] Secondly, it serves as a supplement to the reward function, contributing to the basic reward function R constructed with the task objective. T Together, they form a composite reward function. The new reward function is expressed as:

[0070] R=ωλ+(1-λ)R T

[0071] Where R represents the adjusted composite reward function, and ω represents the proportion of activity freedom in the composite reward function, which is usually a positive value to indicate encouragement for higher activity freedom.

[0072] Furthermore, the specific process of S5 is as follows:

[0073] By interacting with the simulation environment through the algorithm module, a series of interactive experiences are obtained and stored in the experience pool. By resampling the data in the experience pool, the loss function of the algorithm is constructed. By minimizing the loss function, the reinforcement learning decision strategy is gradually optimized.

[0074] To more clearly illustrate the invention, the application principle of the invention will be further explained below with reference to specific embodiments:

[0075] Example 1: Exploration and analysis of the optimization process based on reinforcement learning of the degrees of freedom of a robotic arm's posture.

[0076] UR3 robotic arm control model:

[0077] In this example experiment, a robotic arm model is built using Unity's ArticulationBody component. The ArticulationBody component is a core function of Unity's robot simulation, accurately simulating kinematic chains and solving key simulation problems for robotic arms, manipulators, and mobile robots. ArticulationBody allows for the construction of physical connections using layered game objects.

[0078] In some embodiments, robotic arm simulation software can be developed based on OpenGL, and the software can be used to build a model of a specific robotic arm; or Coppeliasim can be used to build a simulation model of the robotic arm.

[0079] During simulation, Unity provides two properties for ArticulationBody: anchor point position and anchor point rotation, to assist in motion simulation. The default anchor point position of ArticulationBody is the object's center of mass. If the default center of mass position is (0, 0, 0), then when revolute is set, the object rotates according to its center of mass position. The anchor point position can be adjusted by changing the X, Y, and Z position information. In this example, the anchor point position is set at the center of the joint. Anchor point rotation is generally achieved by changing the XDrive; specifically, anchor point rotation is adjusting the XDrive. The corresponding changes can be seen in real-time through editjoints. The default coordinates are the world coordinates of the simulation environment; the model's pose can be adjusted by changing the three properties mentioned above.

[0080] Capture task interaction environment:

[0081] In addition to the simulation model of the robotic arm, task-related interactive objects and scene objects are added to the scene. In this embodiment, these include the target block to be grasped, the robotic arm, and the support platform for placing the target block.

[0082] In this embodiment, the task scenario is set as a rectangular table, with the robotic arm deployed in the center of the table. The target object to be grasped appears randomly on the table within the robotic arm's working space. In this embodiment, to limit the grasping action, the target object is set to be long and narrow, with a length longer than the maximum opening and closing distance of the gripper.

[0083] Building typical reinforcement learning training:

[0084] Setting up a reinforcement learning environment:

[0085] Round Reset: At the start of each round, reset the state of the robotic arm and the position of the target block, set all joints of the robotic arm to the center position, and the entire robotic arm is in a fully extended state, perpendicular to the table; randomly place the target block on the table within the robotic arm's activity space, and the block's posture is generated in a random manner;

[0086] Observation space:

[0087] In reinforcement learning experiments, all possible states can affect the experimental results. Therefore, when setting up the state space, all parameters that may affect the experimental results need to be considered. The learning objective of this embodiment is for the robotic arm's end effector to successfully grasp the object; therefore, it is necessary to observe the relative position of the gripper and the object. Furthermore, it is necessary to understand the current posture information of the robotic arm; therefore, it is necessary to observe the angles between the various joints of the robotic arm. In summary, the designed basic observation space includes [Δx...]. target Δy target Δz target [angles], where Δx target Δy target Δz target These represent the relative positions of the target position and the end effector in the world coordinate system, respectively, and angles represent the included angles of each joint of the robotic arm.

[0088] Action space:

[0089] In end-to-end control tasks, the movement of each joint of the robotic arm can be directly controlled. In this embodiment, the movement direction of each joint is used as the decision action. The decision of a single joint includes three options: [-1, 0, 1]. Here, -1 represents counterclockwise rotation, 0 represents maintaining the current position, and 1 represents clockwise rotation. The range of motion of each joint is limited to [-180°, 180°].

[0090] End of judgment:

[0091] The objective of the task is to successfully grasp the target object. Sensors located on the object's surface and inside the grippers are used to indicate the gripping effect. A successful grasp is determined when the gripper's interior simultaneously contacts both surfaces of the object. Otherwise, the round ends after 500 interactions and is considered a failure.

[0092] Reward function:

[0093] The goal of the experiment is to complete the grasping action in the shortest possible time within a specified time. The target state can be described as the contact between the end effector and the object, and the current state can be represented as the distance between the end effector position and the object position. Therefore, when setting the reward function, the distance between the end effector and the object is set as the reward function to encourage actively reducing the distance between them. Specifically, this is expressed as follows:

[0094] R T =-e Distance

[0095] Where Distance represents the straight-line distance between the end and the block.

[0096] Building a reinforcement learning algorithm:

[0097] The reinforcement learning model used is based on the SAC algorithm, which addresses reinforcement learning problems in discrete and continuous action spaces. It is an off-policy reinforcement learning algorithm. The SAC algorithm consists of five networks: one actor network, two V Critic networks (one V Critic network and one Target V Critic network), and two Q Critic networks.

[0098] The actor network takes the state as input and outputs the action probability π(a) t |s t (For discrete action spaces) or action probability distribution parameters (for continuous action spaces); the input of the critic network is the state, and the output is the value of the state. The output of the V Critic network is v(s), representing the estimate of the state value; the output of the Q Critic network is q(s, a), representing the estimate of the action-state pair value; in the SAC algorithm, to encourage exploration, the concept of entropy is added, therefore the training objectives of the actor and critic networks differ from those of conventional entropy-free algorithms (such as TD3, PPO).

[0099] Increase the measurement of the robot arm's pose and freedom of movement:

[0100] Expanding the observation space:

[0101] Supplementary information on the degrees of freedom of the agent is introduced at the observation location, treating the robot arm's degrees of freedom as an extension of the observation space. Specifically, the observation of the sine value of the central joint angle is added to the basic observation space. Although the various angles of the robot arm have already been observed in the basic observation space, it mainly focuses on expressing the local state of each joint. The extension of the observation of the central joint angle supplements the consideration of the overall flexibility of the robot arm. The extended observation space can be represented as [Δx target Δy target Δz target , angles, λ], where λ represents the sine of the included angle of the central joint.

[0102] Extended reward function:

[0103] The reward function is redesigned by adding supplementary information about the activity's degrees of freedom. Specifically, this involves adding a weighted average of the sine of the central joint angle to the basic reward function, which can be expressed as follows:

[0104] R=ωλ+(1-ω)R T

[0105] Reinforcement learning and training:

[0106] During reinforcement learning training, in the SAC algorithm training process, the algorithm module interacts with the simulation scenario to obtain experience pairs containing (observation, action, reward, termination state), accumulates them in the experience pool, and then extracts data (s) from the experience pool. t a t s t+1 r t The network is updated using U. Based on the optimal Bellman equation, U... t =r t +γv(s t+1 ) as s t The actual value of the action.

[0107] Among them, s t The state at time t; a t The action at time t; s t+1 The state at time t+1; r t v(s) represents the reward value obtained at time t based on R; γ is the reinforcement learning reward discount factor; t+1 ) for s t+1 Predicted state value estimation;

[0108] Through the Q-network in the SAC algorithm, q(s t a t ) as s t Predictive value estimation;

[0109] Using MSELoss as the loss function, U t With q(s) t a t The SAC algorithm decision network is then fitted using the fitting results, and optimized and trained accordingly. The fitting process can be implemented using tools such as Python, Matlab, and C++.

[0110] To verify the effectiveness and applicability of the algorithm, an algorithm evaluation system was designed. Experiments were designed to analyze and compare the implementation effect of the algorithm from four perspectives: the reward function curve during task training, the number of interaction steps required to complete the task, the exploration of the early end-effector position distribution, and the motion trajectory of the end-effector during task completion. The experimental results and data are presented. The training reward curves of the extended observation space and the basic method are shown below. Figure 2 As shown in the figure; the curve comparing the number of interaction steps required to complete a single round during training is as follows. Figure 3 As shown in the figure. The training reward comparison curves of the extended reward function and the basic method are as follows. Figure 4 As shown in the figure. The curve comparing the number of interaction steps required to complete a single round during training is shown in the figure. Figure 5 As shown. Compared with the basic method, both extended methods exhibit better convergence characteristics and task execution capabilities. The three-view distribution of the robotic arm's end effector exploring spatial positions in the initial training phase is shown in the figure. Figure 6 As shown, after introducing degrees of freedom, the robotic arm's end effector exhibits a wider exploration range and better contact with the tabletop, which is beneficial for grasping objects and demonstrates better exploration performance. Regarding task execution, tests were conducted on grasping the same target, and the spatial trajectory of the robotic arm's end effector is shown below. Figure 7 As shown. In Figures 2 to 5 In the diagram, the line value represents the average of the experimental data, while the shaded area represents the maximum and minimum values ​​of the experimental data.

[0111] After introducing the metrics, the trajectory exhibited better smoothness and spatial efficiency. In summary, based on the established evaluation system and corresponding metric requirements, the stability and accuracy of the algorithm were further verified.

[0112] In this technical solution, compared with the existing technology, the optimization learning method based on the degrees of freedom of the robotic arm's posture has better convergence characteristics and task execution capability; because the optimization process targets multiple joints, it has better exploration performance than methods that simplify the decision space; and the introduction of reward function and predictive value estimation makes the trajectory of the robotic arm have better smoothness and spatial efficiency.

[0113] The above are merely preferred embodiments of the present invention and are not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A reinforcement learning method based on the degrees of freedom of a robotic arm's posture movement, characterized in that, include: S1: Construct a 3D simulation model of the robotic arm; S2: Set anchor points at the center of each joint of the robotic arm, adjust and control the rotation of the anchor points, and observe the dynamic changes of the robotic arm in real time to establish the motion model and control model of the robotic arm. S3: Set up the support platform and the target block to be grabbed in the simulation scene; S4: Setting up a reinforcement learning environment: S41: Establishing the observation space [Δx target ,Δy target ,Δz target ,angles], In the formula, Δx target Δy target Δz target These represent the relative positions of the robotic arm's end effector and the target block, respectively, while angles represent the included angles of each joint of the robotic arm. S42: Observe the relative position of the robotic arm and the target block and the included angles of each joint of the robotic arm based on the observation space; S43: By controlling the counterclockwise rotation, maintaining the current position, and clockwise rotation of each joint of the robotic arm, the motion space of the robotic arm is established; the range of the counterclockwise rotation, maintaining the current position, and clockwise rotation is... S44: Set the basic reward function R T , R T =-e Distance , In the formula, Distance is the straight-line distance between the end effector of the robotic arm and the target block; S5: Building a reinforcement learning algorithm based on the SAC algorithm: S51: Introducing the sine value λ of the central joint angle expands the observation space to [Δx target ,No target ,Δz target ,angles,λ]; S52: Introduce λ into R T We obtain the extended reward function R. R=ωλ+(1-ω)R T , In the formula, ω is the weight of λ in the crawling task; S53: Perform the grabbing in the simulation environment and record the experience data of each grabbing; S54: Use the SAC algorithm to interact with the simulation scenario, obtain the experience data and store it in the experience pool, and then collect experience data (s) from the experience pool. t a t s t+1 r t The SAC algorithm is then updated via network. Among them, s t Let a be the state at time t. t Let s be the action at time t. t+1 Let r be the state at time t+1. t Let R be the reward value obtained at time t. S55: Based on the optimal Bellman equation, using U t =r t +γv(s t+1 ) As s t Estimate the true value of the action. In the formula, γ is the reinforcement learning reward discount factor, and ν(s) t+1 ) for s t+1 Predicted state value estimation; S56: Using the Q-network in the SAC algorithm, q(s t a t ) as s t Predictive value estimation; S57: Using MSELoss as the loss function, U t With q(s) t a t The SAC algorithm's decision network is then fitted and optimized based on the fitting results.

2. The reinforcement learning method based on the degrees of freedom of a robotic arm's posture movement as described in claim 1, characterized in that, The process of performing capture in a simulation environment and recording experience data for each capture includes: The grasping result is determined by the signals collected by sensors installed at the end of the robotic arm.

Images