A multi-degree-of-freedom robot arm operation reinforcement learning method from imitation to optimization

Abstract

The application discloses an imitation-to-optimization multi-degree-of-freedom robot arm operation reinforcement learning method, robot state information and a visual image about a robot arm operation process are input into a trained reinforcement learning network, and a robot arm joint angle change amount is output, and the robot arm is controlled through the robot arm joint angle change amount. The reinforcement learning network comprises a multi-modal encoder and an action decoder, the multi-modal encoder comprises a visual encoder, a state encoder and a bidirectional cross attention module, an initial strategy is constructed through an imitation learning mode by using offline teaching data, then online interaction is carried out in an environment, and fine tuning is carried out on the basis of the initial strategy through value-based reinforcement learning, and a trained reinforcement learning network is obtained. Through combination of the imitation learning and the reinforcement learning, the success rate of the robot in a multi-stage operation task can be improved under the condition of the same teaching sample, and operation control of the robot in a complex task environment is realized.

Description

Technical Field

[0001] This invention belongs to the field of robotics, and specifically relates to a reinforcement learning method for multi-degree-of-freedom robotic arm operations that progresses from imitation to optimization. Background Technology

[0002] Existing robotic arm manipulation technologies include template-based robotic arm vision manipulation, imitation learning-based end-to-end robotic arm manipulation, and reinforcement learning-based end-to-end robotic arm manipulation.

[0003] Traditional robotic arm operating systems typically rely on manual pre-programming, meaning that specific operational procedures and control logic need to be designed manually for each particular task. This approach is not only costly and inflexible, but also faces numerous challenges when dealing with highly complex or variable tasks, such as modeling difficulties and weak generalization capabilities.

[0004] With the development of deep learning and computer vision technologies, more and more robotic arm systems are integrating RGB-D (color and depth image) vision sensors to perceive their surroundings. By inputting the acquired visual information into a deep neural network for processing, the system can automatically extract key features and predict the robotic arm's next operational pose, thereby achieving closed-loop control. During the completion of a task, the robotic arm can sequentially reach multiple predefined key waypoints based on visual feedback to perform operations such as grasping, handling, and assembly.

[0005] To improve the automation and accuracy of grasping tasks, Shanghai Jiao Tong University proposed the GraspNet framework in 2020 (FANG HS, WANG C, GOU M, et al. GraspNet-1Billion: A Large-Scale Benchmark for General Object Grasping; proceedings of the 2020 IEEE / CVFConference on Computer Vision and Pattern Recognition (CVPR), F 13-19 June 2020, 2020 [C].). GraspNet is a large-scale neural network model trained on millions of real grasping data. It can directly predict executable 6D grasping poses (position and orientation) from RGB-D images without relying on manually set templates or grasping strategies. This method greatly promotes the grasping and manipulation capabilities of robotic arms in real industrial environments, especially suitable for flexible operation in unknown objects, complex backgrounds, or unstructured scenarios. GraspNet provides an end-to-end grasping detection mechanism, which is a key step in the transformation of robotic arms from traditional template-driven to perception-driven and autonomous intelligence. However, this method can only achieve end-to-end grasping, and for complex robotic arm operation tasks, manual programming and template definition are still required.

[0006] With the continuous advancement of deep learning algorithms and the sustained improvement of computing hardware performance, robotic arm operation technology is gradually shifting from traditional modular control methods to an end-to-end learning control paradigm. Compared to traditional methods that rely on manually set perception, planning, and control modules, the end-to-end approach can directly map raw sensory inputs (such as RGB images) to low-level control signals (such as joint angles), simplifying the system design process and improving the overall intelligence level and task generalization ability.

[0007] In 2023, Stanford University proposed the Action Chunking with Transformer (ACT) framework (ZHAO TZ, KUMAR V, LEVINE S, et al. Learning Fine-Grained BimanualManipulation with Low-Cost Hardware [J / OL] 2023). Based on the idea of ​​imitation learning, this method successfully achieves end-to-end prediction of robotic arm joint angles by introducing the Transformer architecture to perform sequence modeling on demonstration data. The core idea of ​​ACT is to decompose human or taught operations into multiple sub-actions (action chunks), and utilize the sequence modeling capabilities of the Transformer to learn the temporal relationships between these action units, thereby improving the understanding and execution efficiency of complex tasks. This method demonstrates strong generalization ability in several standard robotic arm operation tasks (such as object grasping, insertion, and door opening / closing). Building upon this foundation, the introduction of the Diffusion Policy (CHIC, XU Z, FENG S, et al. Diffusion Policy: Visuomotor Policy Learning via Action Diffusion [J / OL] 2023) has further propelled the development of end-to-end robotic arm manipulation technology. By modeling trajectory distribution in a continuous action space, the Diffusion Policy can efficiently sample high-quality action sequences from taught trajectories, exhibiting inherent diversity and stability. Compared to traditional regression strategies, the Diffusion Model demonstrates greater advantages in handling high-dimensional action spaces, exhibiting higher robustness and success rate.

[0008] While end-to-end methods based on imitation learning have made some progress in robotic arm manipulation, they still have significant shortcomings. First, these methods heavily rely on high-quality expert teaching data, which is costly to collect and has limited coverage. Second, the models have weak generalization ability in unfamiliar environments or states, making them prone to policy failure. Furthermore, imitation learning lacks an active exploration mechanism, making it difficult to address the error accumulation problem in long-term tasks.

[0009] Reinforcement learning, as a typical unsupervised decision-making method, has been widely used in robotic arm operations. Its core idea is to maximize long-term rewards by interacting with the environment, continuously trying and learning from mistakes, and accumulating experience. In practical applications, reinforcement learning algorithms are typically trained first in a high-fidelity simulation environment. After the policy training converges, it is then transferred and deployed to a real robotic arm to complete specific operational tasks.

[0010] As early as 2018, researchers proposed a self-supervised reinforcement learning method that enabled robotic arms to complete complex tasks such as object arrangement, pushing, and grasping through autonomous exploration and learning without human instruction (ZENG A, SONG S, WELKER S, et al. Learning Synergies Between Pushing and Grasping with Self-Supervised Deep Reinforcement Learning; proceedings of the 2018 IEEE / RSJ International Conference on Intelligent Robots and Systems (IROS), F 1-5 Oct. 2018, 2018 [C].). This work demonstrated the potential of reinforcement learning in multi-step decision-making and combinatorial tasks, breaking through the limitations of traditional imitation learning.

[0011] From the above, it's clear that imitation learning heavily relies on the quality and coverage of the teaching data, making it difficult to generalize to scenarios outside the training data. Its core idea is to use expert demonstrations for supervised learning, fitting a policy function so that the agent can imitate expert behavior. However, imitation learning is essentially passive learning, lacking active exploration capabilities and unable to make reasonable decisions in states not covered in the training data. Furthermore, due to state distribution shift (covariate shift), small errors are amplified in multi-step decision-making, leading to serious biases in model execution, especially in complex, multi-stage tasks, easily resulting in operational failures and task interruptions. Therefore, relying solely on imitation learning is insufficient to build robust and highly generalizable policy models. Reinforcement learning, on the other hand, has attracted increasing attention due to its active exploration and policy optimization capabilities. Compared to imitation learning, which can only replicate expert behavior within an existing policy space, reinforcement learning can discover policies superior to those taught in long-term interactions, thus possessing the potential to "outperform experts." It has strong exploration capabilities and long-term reward optimization capabilities, theoretically enabling the learning of complex policies, especially suitable for complex or undefined new task scenarios. However, reinforcement learning also faces a series of challenges, particularly in the field of robotic arms. Firstly, there are difficulties in transferring simulation to reality (Sim2Real) – a policy that performs well in simulation often becomes significantly less effective in real-world environments due to perceptual noise, dynamic biases, and other issues. Secondly, exploration efficiency is low – reinforcement learning requires a large number of interaction samples to learn effective policies, which is almost infeasible when training on real robots. Furthermore, in high-dimensional action spaces and sparse reward tasks, the policy optimization process is prone to getting stuck in local optima or experiencing training instability.

[0012] In summary, the existing technology has the following shortcomings that urgently need to be addressed:

[0013] ① It is difficult to learn robust policies from teaching data. Current imitation learning methods rely on limited and uniformly distributed teaching data, which is difficult to cover the diverse scenarios and state spaces in the target task, thus hindering the generalization and real-time adjustment of policies in complex environments. Once the environment changes or the target perturbation deviates significantly from the training samples, the model policy is prone to failure and lacks robustness in dealing with non-ideal states.

[0014] ② Reinforcement learning is difficult to apply in real-world environments. Currently, reinforcement learning-based policy optimization methods are challenging to deploy in real-world robotic systems. Their training process relies heavily on environmental interactions and requires frequent trial and error, which can lead to equipment wear and tear, inefficiency, and even safety risks in real physical systems. This hinders the efficient learning and dynamic adjustment of policies in practical systems. Furthermore, the latency, noise, and uncertainty in real-world feedback further exacerbate the instability of policy learning. Summary of the Invention

[0015] To address at least one of the problems existing in the prior art, this invention provides a reinforcement learning method for multi-degree-of-freedom robotic arm operations that combines imitation and optimization. By combining imitation learning with reinforcement learning, the success rate of the robot in multi-stage operation tasks can be improved under the same teaching samples, enabling the robot to operate and control in unstructured and complex task environments.

[0016] To achieve the purpose of this invention, this invention provides a reinforcement learning method for multi-degree-of-freedom robotic arm operation from imitation to optimization. The reinforcement learning network is trained by inputting robot state information and visual images of the robotic arm operation process collected by visual sensors, and outputs the change in joint angle of the robotic arm. The robotic arm operation is controlled by the change in joint angle of the robotic arm.

[0017] The reinforcement learning network includes a multimodal encoder and an action decoder. The multimodal encoder includes a visual encoder, a state encoder, and a bidirectional cross-attention module. The visual encoder is used to extract local spatial features from visual images of the robotic arm operation process acquired by the visual sensor. The state encoder is used to compress the robot's state information into a low-dimensional representation. The bidirectional cross-attention module is used to fuse the visual features output by the visual encoder and the robot's state information output by the state encoder to obtain a state representation vector after multimodal information fusion. The action decoder is used to receive the state representation vector and output the changes in the joint angles of the robotic arm.

[0018] In this process, an initial policy is constructed using offline teaching data through imitation learning. Then, online interaction is conducted in the environment, and the initial policy is fine-tuned through value-based reinforcement learning to obtain a trained reinforcement learning network.

[0019] The present invention also provides a reinforcement learning device for multi-degree-of-freedom robotic arm operation, which involves imitation to optimization.

[0020] The present invention also provides a computer device.

[0021] The present invention also provides a computer-readable storage medium.

[0022] This invention aims to address several core problems of current imitation learning and reinforcement learning in complex robotic arm operation tasks, and proposes an imitation-to-optimization (I2O) policy learning framework that integrates the advantages of both. The beneficial effects include:

[0023] 1. Learn task strategies better from teaching data

[0024] Existing imitation learning methods typically employ behavior cloning, directly regressing actions from the teaching data. However, this approach fails to distinguish between high-quality actions in the teaching data, easily degenerating into mechanical imitation in the presence of redundant or low-quality demonstrations, making it difficult for the model to generalize to unseen scenarios. This invention addresses this problem by introducing a value-based offline reinforcement learning mechanism, assigning different reward weighting levels to the teaching data. This facilitates the identification and priority imitation of high-quality trajectories, effectively extracting core policies and improving their expressive and generalization capabilities. Furthermore, training can be performed using only offline data, avoiding the costly online interaction and providing a feasible path for large-scale deployment.

[0025] 2. Improve the exploration efficiency of reinforcement learning

[0026] Traditional reinforcement learning requires random exploration in a high-dimensional action and state space, resulting in low training efficiency and a tendency to get trapped in local optima. This invention, based on a preliminary reasonable policy obtained through imitation learning, uses it as the initial or guiding policy for reinforcement learning, significantly shortening exploration time and improving training efficiency. By prioritizing the sampling of high-quality trajectories in the replay buffer, the convergence speed is significantly accelerated. Furthermore, this method effectively improves the performance of the policy in sparse reward or multi-stage tasks.

[0027] 3. Addressing the adaptability issue of value function-based reinforcement learning in continuous control.

[0028] In high-degree-of-freedom robotic arm operations, the motion space exhibits highly continuous and coupled characteristics, making it difficult to directly transfer traditional reinforcement learning methods based on discrete motion modeling. This invention employs value function modeling methods in continuous motion spaces (such as Q-functions with deterministic policy gradients) and constructs constraint priors by fusing imitation strategies, guiding the value function to learn more accurate expected reward estimates under critical states. Furthermore, by introducing expert motion constraints, this invention significantly improves the stability and performance of value function-based reinforcement learning algorithms in complex, multi-objective control tasks.

[0029] 4. Achieve an efficient combination of offline training and online exploration.

[0030] Offline data learning can save significant interaction costs, but it struggles to cope with dynamic environmental changes, while fully online reinforcement learning is costly to train and difficult to deploy. To resolve this contradiction, this invention constructs a training paradigm combining offline reinforcement learning training with online environment interaction. The initial policy is obtained through imitation learning, followed by online interaction to collect data and offline reinforcement learning in the environment, adapting in real time to changing task scenarios and object postures, thus achieving a balance between policy performance and environmental adaptability. Furthermore, this invention's policy correction mechanism ensures the policy is safely and reliably improved during online processing, further enhancing task completion rate and system robustness. Attached Figure Description

[0031] Figure 1 This is a framework diagram of an embodiment of the present invention.

[0032] Figure 2 This is a flowchart of an embodiment of the present invention.

[0033] Figure 3 This is a unit composition diagram of a multi-degree-of-freedom robotic arm operation reinforcement learning device that progresses from imitation to optimization, as described in an embodiment of the present invention. Detailed Implementation

[0034] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0035] Imitation learning suffers from problems such as over-reliance on teaching data and insufficient generalization ability. Reinforcement learning, on the other hand, optimizes its policies through continuous trial and error via interaction with the environment, possessing strong exploratory capabilities and long-term reward optimization abilities, and theoretically capable of learning complex policies. However, reinforcement learning faces significant challenges in practical robotic tasks. First, reinforcement learning often requires a large number of interaction samples to converge, resulting in low learning efficiency and high training costs. Especially in real physical systems, large-scale interaction processes are not only time-consuming but can also cause wear and tear or even damage to hardware. Second, the design of reward functions is complex and sensitive; an unreasonable reward design may lead the policy into local optima or exhibit behaviors that contradict human intuition. Furthermore, reinforcement learning policies are prone to exploratory dilemmas when facing high-dimensional state spaces and sparse reward problems, making it difficult to stably learn effective policies. This invention proposes a robot operation strategy learning method that progresses from imitation to optimization. This method addresses key issues in existing robotic arm operation tasks, such as poor generalization ability, heavy data dependence, and unrobust strategies. It also solves the problem of low success rate for robotic arms when learning complex operation tasks. Using reinforcement learning as the core framework, it introduces imitation learning as a mechanism for providing prior knowledge and combines a high-fidelity simulation platform with policy regression techniques to achieve intelligent operation of the robot from rough exploration to fine control.

[0036] This invention provides a reinforcement learning method for multi-degree-of-freedom robotic arm operations, which introduces the imitation learning approach into reinforcement learning. The method includes the following steps:

[0037] Step 1: Set up the robotic arm operating environment, including the simulation environment and the real machine environment.

[0038] In one embodiment, a six-DOF robotic arm (Songling Piper) is used as the execution subject in the RLbench simulation platform, with vision sensors installed at the end effector and directly above the robotic arm. In actual deployment, a six-DOF robotic arm (Songling Piper) is also selected as the execution subject, with an RGBD camera (Realsense D435i) installed at the end effector to acquire image data of the task scene, and a vision sensor (such as an RGB camera) fixed above the robotic arm.

[0039] In both the simulation and real machine environments, vision sensors were installed on the wrist joint of the robotic arm and directly above it, respectively, to ensure that the robotic arm could always obtain a complete field of view during its movement.

[0040] The simulation environment is primarily used to verify the effectiveness of the invention, while the real-world environment is mainly used for actual deployment and application. The strategies trained using data collected in the simulation environment are used in the simulation environment, and the data collected in the real-world environment is used in the real-world environment.

[0041] Step 2: Collect teaching data and assign value levels to the teaching data to construct a teaching dataset. The teaching data includes the joint angles of the robotic arm, the six-dimensional pose of the robotic arm end effector, the gripper status information, and visual images of the robotic arm operation process collected by a vision sensor.

[0042] In one embodiment, the joint angles of the robotic arm, the six-dimensional pose of the robotic arm's end effector, and the gripper status information can be directly obtained through the robotic arm's built-in SDK.

[0043] In one embodiment, during the task initialization phase, also known as the perception phase, after setting up the environment, the basic grasping operation task is first completed based on a traditional approach. This part is mainly used to collect teaching data in this step, requiring the acquisition of the pose of the object to be grasped and the construction of an original dataset for reinforcement learning network teaching. The robotic arm system is equipped with an RGBD camera to acquire image information of the operation process. The RGBD camera can be installed in either an "eye-in-hand" or "eye-to-hand" manner to ensure perception flexibility under different task requirements. Before startup, the hand-eye calibration of the RGBD camera is completed. The hand-eye calibration process mainly involves acquiring images of the calibration board observed by the robotic arm end effector in multiple different poses during the operation, and calculating the transformation relationship between the camera coordinate system and the robotic arm end effector coordinate system. The calibration process can use the Tsai-Lenz algorithm, which transforms the extrinsic parameters from multiple perspectives into a least-squares optimization problem to obtain accurate extrinsic parameters.

[0044] After hand-eye calibration is completed, the image information acquired by the RGBD camera can be used for 3D target recognition and localization. The pose of the object to be manipulated in the camera coordinate system is obtained through 6D pose estimation methods (such as FoundationPose). The desired 3D position of the robotic arm is obtained through hand-eye transformation and inverse kinematics solving, allowing the robotic arm to move to the target position and perform initial grasping actions, laying the foundation for subsequent teaching and training tasks.

[0045] In one embodiment, data acquisition can employ both manual programming and remote control via a gamepad. Manual programming is suitable for structured, simple scenarios, such as tasks involving repeatedly placed objects, where the robotic arm performs the operation by manually programming the steps. Remote control via a gamepad is suitable for unstructured environments or complex scenarios involving the grasping of multiple types of objects. By mapping the control buttons and joysticks of a gamepad to the six degrees of freedom of the robotic arm, dexterous manipulation of the end effector is achieved. Every action input during remote control is recorded in real time, including control signals, the current frame image, the joint angles of the robotic arm, the end effector pose, and environmental conditions (such as target labels). To ensure data diversity and perspective coverage, at least two cameras are configured to capture operation videos from different angles, capturing the entire process of the robotic arm's movements and providing a foundation for multimodal strategy learning.

[0046] In one embodiment, in a simulation environment, online teaching data is collected through online interaction between the policy and the environment during reinforcement learning network training, while offline teaching data is collected through predefined operation routes to gather data on the robotic arm's operation tasks. In a real environment, online teaching data consists of data obtained by deploying an initial policy (trained based on purely offline teaching data) onto the robotic arm and inferring from that initial policy. Offline teaching data requires the operator to use a PlayStation or Xbox controller as an input device, mapping the joystick of the input device to the X, Y, and Z directional position control of the robotic arm's end effector, and the buttons on the input device corresponding to end effector posture control and gripper opening / closing status. During data acquisition, each frame of the task operation process, the robotic arm joint angles, the six-dimensional pose of the end effector, and the gripper status information are recorded. Figure 1 As shown in (a), in addition to collecting online and offline teaching data, the teaching data is assigned a value level, with successful and unsuccessful operation data collected in a 1:1 ratio (success is judged by whether the operation task is successfully completed; for example, if the operation task is to open a door, then successfully opening the door is considered successful operation data). A reward of 1 is assigned to successful operation data, and a reward of 0 is assigned to unsuccessful operation data. The teaching process is executed multiple times to cover different objects, different postures, and different ambient lighting and background arrangements to enhance data diversity.

[0047] Step 3: Build a reinforcement learning network that moves from imitation to optimization.

[0048] Reinforcement learning networks include multimodal encoders and action decoders, such as Figure 1As shown in (b), the multimodal encoder is used to fuse multi-source information to achieve a high-dimensional representation of the robot's environmental state. The multi-source information includes robot state information and visual images of the robotic arm's operation process acquired from a vision sensor. The multimodal encoder includes a visual encoder, a state encoder, and a bidirectional cross-attention module. The visual encoder extracts local spatial features from the visual images acquired by the vision sensor. The state encoder compresses the robot state information into a low-dimensional representation through linear transformation, thus processing the multi-source information into vectors of the same size. The bidirectional cross-attention module fuses the visual features output by the visual encoder and the robot state information output by the state encoder to obtain a state representation vector after multimodal information fusion.

[0049] In one embodiment, the visual encoder employs a convolutional neural network (CNN), and the state encoder is a linear encoder. After the visual image is processed by the CNN to extract local spatial features, the output dimension is 1×25600. Low-dimensional information such as joint angles, gripper states, and gripper poses are mapped to a 1×128 feature space by the linear encoder. Subsequently, the two modal features are fused through bidirectional cross-attention and concatenation operations to form a unified state representation vector, providing input for the action decoder.

[0050] The action decoder employs a hierarchical reinforcement learning structure. First, the state representation vector obtained from multimodal fusion in the multimodal encoder is mapped through a linear layer to... The value space is hierarchically structured through repeated linear mappings. Value space, outputting each level (or interval) on a set of discrete value intervals. Value distribution, thereby enabling graded estimation of value. Based on Value distribution, through The operation and decoder obtain the motion (i.e., the change in the joint angle of the robotic arm).

[0051] In one embodiment, to ensure that the action selection process is differentiable with respect to network parameters, using Operational replacement of traditional This allows for smooth, differentiable index selection on discrete action distributions. For example... Figure 1 As shown in the action decoder section of (b), the reinforcement learning network simultaneously models the behavior distribution and value distribution, and can output actions and values. Value. For a set of values, after discretization... value , The weighted summation of the operation's output is in the form of:

[0052]

[0053] in , For discrete indexes, For the size of the vector, It includes a total of A vector, , Indicates the first The and the first A vector, This is a temperature coefficient; the higher the temperature, the closer the output is to the actual temperature. In one embodiment, .

[0054] Ultimately, the decoder uses a continuous decoding function. Decode discrete action indices into executable actions. The expression is:

[0055]

[0056] in and These represent the upper and lower limits of the action space, respectively. Represents the number of discretized intervals. Indicates the process The action number obtained from the operation. (Using...) The design of the output action ensures the differentiability of the reinforcement learning network while achieving a natural transition from imitation learning to reinforcement learning optimization, effectively improving the stability and accuracy of policy generation.

[0057] Step 4: Based on the visual images and robot state information collected in Step 2 regarding the robotic arm operation process, perform imitation learning and reinforcement learning training on the reinforcement learning network to obtain the trained reinforcement learning network, which can then be used to obtain the policy, including the network structure and weights.

[0058] The input to the reinforcement learning network is the robot's state and image information collected by the vision sensor. The robot's state information includes the joint angles of the robotic arm, the six-dimensional pose of the robotic arm's end effector, and the state information of the gripper. The output is the change in the joint angles of the robotic arm (i.e., the actions that can be performed).

[0059] The reinforcement learning network employs a two-stage perception-regression structure. The perception component, described in step 3, is a multimodal encoder used to extract image features and obtain a unified state representation vector. The regression component, also described in step 3, is an action decoder that receives the state representation vector encoded by the multimodal encoder and outputs the changes in the robotic arm's joint angles. This is achieved by using a discretized value range. Operation-to-action value distribution The value distribution is used for position regression, and then the decoder is used to obtain the final change in the joint angle of the robotic arm, thereby improving training stability.

[0060] First, an initial strategy is constructed using offline teaching data through imitation learning. The teaching data has different value levels; based on these value levels, high-quality teaching data is prioritized for playback into the playback pool for imitation learning. The loss function optimizes the action regression error, which can be expressed as:

[0061]

[0062] In the formula, This represents the regression error. To enhance the prediction of actions by the learning network, This represents the true action corresponding to the dataset.

[0063] Then, value-based reinforcement learning fine-tunes the initial policy. Reinforcement learning uses a time-difference-based loss for offline updates, and its loss equation can be expressed as:

[0064]

[0065] in, This indicates the loss from reinforcement learning. Indicates in the dataset Seeking hope in the middle Indicates the current state of the robot. Indicates an action, Indicates taking an action The reward received later This indicates the robot's state at the next moment. Represents actions in action space. This represents the discount factor (in one embodiment, the value is 0.99). Indicate target Network parameters, This indicates the network parameters to be updated. Indicates network parameters The calculated state-action pairs of value, This means finding an optimal action. Make The value is the largest. Indicates network parameters The calculated state-action pairs of Value. In an environment, reinforcement learning can achieve online fine-tuning by interacting with the surrounding environment and expanding the dataset online.

[0066] In one embodiment, a policy correction mechanism is introduced to ensure that the policy optimizes in the correct direction during reinforcement learning. This policy correction mechanism involves adjusting the expert's actions. Value greater than other actions The value limits the direction of updates during reinforcement learning.

[0067] Step 5: Deploy the strategy (the trained reinforcement learning network, including network structure and weights). In a real environment, input the robot's state information and visual images of the robotic arm's operation process collected by the visual sensors into the reinforcement learning network. This will output the change in the joint angle of the robotic arm, and the robotic arm's operation will be controlled by the change in the joint angle of the robotic arm.

[0068] To minimize the gap between simulation and reality, offline training can be performed using only data collected in the real environment (including robotic arm joint angles, six-dimensional pose of the robotic arm's end effector, and images acquired through vision sensors). The strategy trained on the real-world data is then deployed on a real robotic arm, enabling it to stably handle various target objects, multi-viewpoint changes, and scene interference. This achieves stable operational results with minimal data, significantly outperforming single imitation learning methods and demonstrating strong practicality and potential for wider application.

[0069] The overall framework of the embodiments of the present invention is as follows: Figure 2 As shown, it is divided into three stages: perception, imitation, and optimization.

[0070] During the policy training phase, an initial policy is constructed using imitation learning. An end-to-end neural network model is then built, processing visual input into action output. That is, reinforcement learning networks. Representative strategy, Represents model parameters, The action that represents the output. This represents visual observation, specifically images acquired by visual sensors. Representing the current state of the robot, the perception part, namely the visual encoder, uses a convolutional neural network to extract image features, while the action decoder uses a multilayer perceptron to regress the corresponding action state of the robotic arm. Value. To ensure that the output of the value network model is continuously differentiable, a value is introduced. Operation, can Value encoding is converted into specific actions, thereby supporting end-to-end backpropagation training.

[0071] Reinforcement learning networks trained using imitation learning possess basic operational capabilities, but they can only handle scenarios within the dataset and struggle to generalize well in complex environments. Therefore, a reinforcement learning-based optimization mechanism is designed for the final optimization stage. In this stage, the policy learning framework employs a value-based reinforcement learning approach, specifically using a Dueling DQN structure and a Coarse-to-Fine architecture. The Coarse-to-Fine architecture can be referenced as follows: Figure 2 In the optimization part, during the Coarse stage, the action space output by the reinforcement learning network is discretized into a fixed number of intervals, and a coarse target region for the action is extracted from the global image and robot state through the policy network. In the Fine stage, the local region discretized by the Coarse stage is further discretized into a fixed number of intervals, thereby outputting a fine action index. The action value is then output through the decoder, improving operational accuracy. Meanwhile, this embodiment of the invention employs a Dueling structure with a deep... The DQN (Direct Quality Network) framework will The value function is decomposed into two parallel branches: the state value network and the action advantage network. The state value network is used to estimate the overall value of the current robot state. This reflects the potential reward under the current strategy in this robot state; the AdvantageNet is used to estimate the relative advantage of different actions relative to the average level of this state. Subsequently, to avoid the unidentifiability problem and obtain a more stable estimate, the standard fusion method in Dueling DQN is adopted, and the outputs of the state value network and the action advantage network are reconstructed in the following form:

[0072]

[0073] in This indicates the current strategy in the current robot state. Next action The expected return that can be obtained Describe the current state of the robot Its long-term value is independent of specific actions. Indicates action Relative to the current robot state The relative advantage of the average level Represents the action space, This represents the size of the action space, i.e., the number of candidate actions. Represents actions in action space. Represents actions in action space Relative to the current robot state The relative advantage of the average level.

[0074] The value function is updated according to the Bellman equation, and experience replay and target network techniques are used to improve stability. During training, state transitions, reward feedback, and termination information are obtained through interaction with the environment, forming an experience pool. An imitation policy is used as the initial policy to guide the robotic arm to learn quickly, and reinforcement learning further optimizes the policy performance and robustness.

[0075] In one embodiment, different operational tasks are built in a simulation environment, data is collected, and policies are trained according to the methods provided in the aforementioned embodiments. The success rates of different tasks are verified and compared with other classic methods. The specific results are shown in Table 1. In Table 1, DrQ-v2 is a reinforcement learning method published at the International Conference on Learning Representations (ICLR) in 2022 (Yarats D, Fergus R, Lazaric A, et al. Mastering Visual Continuous Control: Improved Data Augmented Reinforcement Learning [J / OL] 2022), CQN is a reinforcement learning method published at the Conference on Robot Learning (CoRL) in 2024 (Seo Y, Uruc J, et al. Continuous Control with Coarse-to-fine Reinforcement Learning [J / OL] 2024), and ACT is a robotic arm operation imitation learning solution proposed by Stanford University (ZHAO TZ, KUMAR V, LEVINE S, et al. Learning Fine-Grained Bimanual Manipulation with Low-Cost Hardware). [J / OL]2023), I2O-RL is a reinforcement learning method for multi-degree-of-freedom robotic arm operation proposed in the embodiments of the present invention, which is a method from imitation to optimization. As can be seen from the table, the success rate of operation using the change in joint angle of the robotic arm output by the method provided in the embodiments of the present invention is much higher than that of other existing methods. That is, the present invention can achieve efficient operation capability under low data dependence conditions and has good practical application value.

[0076] Table 1. Success rates of different methods under different operational tasks (in %)

[0077]

[0078] In one embodiment, a multi-degree-of-freedom robotic arm manipulation reinforcement learning device is provided, which moves from imitation to optimization, to implement the methods described in the foregoing embodiments. Please refer to [link to previous document]. Figure 3 The device includes the following units:

[0079] The model training unit is used to construct an initial policy through imitation learning using offline teaching data, and then to fine-tune the initial policy through online interaction in the environment and value-based reinforcement learning to obtain the trained reinforcement learning network.

[0080] The motion output unit inputs robot state information and visual images of the robotic arm operation process collected by the vision sensor into a reinforcement learning network trained on it, and outputs the changes in the joint angles of the robotic arm. Based on these changes, the unit controls the operation of the robotic arm. The reinforcement learning network includes a multimodal encoder and a motion decoder. The multimodal encoder includes a visual encoder, a state encoder, and a bidirectional cross-attention module. The visual encoder extracts local spatial features from the visual images of the robotic arm operation process collected by the vision sensor. The state encoder compresses the robot state information into a low-dimensional representation. The bidirectional cross-attention module fuses the visual features output by the visual encoder and the robot state information output by the state encoder to obtain a state representation vector after multimodal information fusion. The motion decoder receives the state representation vector and outputs the changes in the joint angles of the robotic arm.

[0081] In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the methods described in the foregoing embodiments.

[0082] In one embodiment, a computer-readable storage medium is provided, the computer-readable storage medium storing a computer program that, when executed by a processor, implements the methods described in the foregoing embodiments.

[0083] Compared with the prior art, the functional differences of the foregoing embodiments of the present invention are mainly reflected in the following aspects:

[0084] Combining Reinforcement Learning and Imitation Learning: Background techniques typically rely on a single imitation learning or reinforcement learning approach, which suffers from poor generalization and low exploration efficiency in complex tasks or data-scarce scenarios. In contrast, this invention proposes a fusion strategy of "imitation-to-optimization." It first constructs an initial policy using imitation learning from teaching data, and then fine-tunes and enhances the policy through value-based reinforcement learning, significantly improving the policy's robustness and generalization ability. This method effectively solves the problem of "difficulty in learning robust policies from teaching data."

[0085] Direct policy regression based on value reinforcement learning: Traditional reinforcement learning methods often employ actor-critic structures or policy gradient methods, which suffer from policy instability and slow convergence in continuous action spaces. This invention adopts a value function-based approach... Learning architecture directly models the policy regression task as Operation, thereby achieving from The optimal action output is precisely regressed from the values. In terms of structural design, this invention employs a Coarse-to-Fine approach. The network architecture discretizes the action space multiple times to achieve a control effect from coarse to fine, improving the accuracy and computational efficiency of continuous control, thus effectively addressing the problem of "low efficiency in strategy optimization".

[0086] Combining offline and online reinforcement learning: Existing reinforcement learning methods often heavily rely on simulation environments for policy training, which is limited by the accuracy of the simulator and the gap with reality, resulting in poor "simulation-to-reality" transfer performance. This invention makes full use of offline teaching data and adopts offline reinforcement learning (step 4) (e.g., based on...). This invention employs a value-based behavioral cloning augmentation method for policy learning, achieving stable and efficient policy optimization without relying on online interaction. By simultaneously modeling behavioral distribution and value functions in offline teaching data, this invention effectively compensates for insufficient data coverage in the absence of real-world interaction, thereby obtaining policies with good generalization performance under purely offline conditions.

[0087] This invention can significantly improve the initial performance and learning efficiency of a policy. By assigning value to the teaching data and training with offline reinforcement learning, the initial policy has a high success rate, avoiding the inefficient exploration caused by random initialization.

[0088] The reinforcement learning training process of this invention is more stable and converges faster. Function-guided Regression operations and Coarse-to-Fine hierarchical modeling This effectively alleviates the problem of unstable continuous motion in traditional methods.

[0089] This invention can better adapt to changes in real-world environments and tasks. By combining offline data training with online interaction in limited real-world environments, the stability and robustness of the strategy in real-world operations are significantly improved, giving it strong practical deployment capabilities.

[0090] This invention enables better policy optimization under limited sample conditions. Through reinforcement learning, it achieves high-quality policy learning even in environments with limited data or high-cost data acquisition, reducing the difficulty of engineering implementation.

[0091] This invention supports end-to-end policy regression, reducing reliance on manual engineering. The process from perception to control no longer depends on intermediate manual setting modules, reducing reliance on traditional engineering processes such as geometric modeling and trajectory planning, and improving the overall autonomous learning ability and adaptability of the system.

[0092] The above description of the disclosed embodiments enables those skilled in the art to make or use the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined in this invention may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A reinforcement learning method for multi-degree-of-freedom robotic arm operations, characterized in that, The robot's state information and visual images of the robotic arm's operation process are input into the trained reinforcement learning network, which outputs the change in the joint angle of the robotic arm, and controls the operation of the robotic arm through the change in the joint angle of the robotic arm. The reinforcement learning network includes a multimodal encoder and an action decoder. The multimodal encoder includes a visual encoder, a state encoder, and a bidirectional cross-attention module. The visual encoder is used to extract local spatial features from visual images of the robotic arm's operation process. The state encoder is used to compress the robot's state information into a low-dimensional representation. The bidirectional cross-attention module is used to fuse the visual features output by the visual encoder and the robot's state information output by the state encoder to obtain a state representation vector after multimodal information fusion. The action decoder is used to receive the state representation vector and output the changes in the joint angles of the robotic arm. In this process, an initial policy is constructed using offline teaching data through imitation learning. Then, online interaction is conducted in the environment, and the initial policy is fine-tuned based on value-based reinforcement learning to obtain a trained reinforcement learning network. When performing value-based reinforcement learning, a Coarse-to-Fine architecture is adopted. In the Coarse stage, the action space output by the reinforcement learning network is discretized into a fixed number of intervals, and a coarse target region for actions is extracted from the global image and robot state through the policy network. In the Fine stage, the local regions discretized in the Coarse stage are further discretized into a fixed number of intervals, and a fine action index is output. Then, the action value is output through the decoder. Furthermore, a Dueling DQN structure is also employed for value-based reinforcement learning. The network will The value function is decomposed into a state value network and an action advantage network. The state value network is used to estimate the overall value of the current robot state, reflecting the potential reward in the robot state according to the current strategy. The action advantage network is used to estimate the relative advantage of different actions relative to the average level of the robot state.

2. The reinforcement learning method for multi-degree-of-freedom robotic arm operation from imitation to optimization according to claim 1, characterized in that, The offline teaching data is assigned a value level. When imitation learning is carried out, high-quality offline teaching data is preferentially played back to the playback pool based on the value level of the offline teaching data for imitation learning, and the action regression error is optimized.

3. The reinforcement learning method for multi-degree-of-freedom robotic arm operation from imitation to optimization according to claim 1, characterized in that, When learning by imitation, use The loss function is optimized for action regression error, and when performing value-based reinforcement learning, the loss based on temporal difference is updated offline. During reinforcement learning network training, the expert's actions are... Value greater than other actions The value limits the direction of updates during reinforcement learning.

4. The reinforcement learning method for multi-degree-of-freedom robotic arm operation from imitation to optimization according to claim 1, characterized in that, The motion decoder is configured to perform the following operations: Mapping the state representation vector through a linear layer to... The value space is hierarchically structured through repeated linear mappings. Value space, outputting each level or interval in a set of discrete value intervals. Value distribution, through Operation pair The value distribution is used for position regression, and then the change in the joint angle of the robotic arm is output through continuous decoding.

5. A reinforcement learning method for multi-degree-of-freedom robotic arm operation from imitation to optimization according to any one of claims 1-4, characterized in that, The visual encoder uses a convolutional neural network, and the state encoder is a linear encoder.

6. A reinforcement learning device for multi-degree-of-freedom robotic arm operations, characterized in that, For implementing the method according to any one of claims 1-5, the apparatus comprises the following units: The model training unit is used to construct an initial policy through imitation learning using offline teaching data, and then to fine-tune the initial policy through online interaction in the environment and value-based reinforcement learning to obtain the trained reinforcement learning network. The action output unit inputs robot state information and visual images of the robotic arm's operation process into a reinforcement learning network trained on it, and outputs the changes in the robotic arm's joint angles. Based on these changes, the unit controls the robotic arm's operation. The reinforcement learning network includes a multimodal encoder and an action decoder. The multimodal encoder comprises a visual encoder, a state encoder, and a bidirectional cross-attention module. The visual encoder extracts local spatial features from the visual images of the robotic arm's operation process. The state encoder compresses the robot's state information into a low-dimensional representation. The bidirectional cross-attention module fuses the visual features output by the visual encoder and the robot's state information output by the state encoder to obtain a fused state representation vector. The action decoder receives the state representation vector and outputs the changes in the robotic arm's joint angles.

7. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the method according to any one of claims 1-5.

8. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the method described in any one of claims 1-5.

Images