Mechanical arm reward-free reinforcement learning method based on expert state learning

Abstract

The invention provides a mechanical arm reward-free reinforcement learning method based on expert state learning, belongs to the technical field of automation, particularly focuses on operation control of a robot arm, and comprises two core modules: an expected state prediction network and a reward-free strategy network. The expected state prediction network is used for predicting state transition, and the non-reward strategy network optimizes action through an error between a prediction state of the expected state prediction network and a target state provided by an expert under the condition of no reward signal; according to the method, the mechanical arm can directly learn an effective grabbing strategy from state data observed by experts, complex reward function design is avoided, dependence on expert action data is not needed, and therefore understanding of dynamic characteristics of grabbing tasks is improved, and accuracy of action prediction is enhanced.

Description

Technical Field

[0001] The present invention relates to the field of automation technology, and in particular to a reward-free reinforcement learning method for a robotic arm based on expert state learning. Background Art

[0002] As a core component of intelligent manufacturing, the development of control technology for automated robotic arms is of great significance for improving production efficiency and quality. Robotic arms perform precise manipulation tasks such as grasping, handling, and assembly by simulating the movements of human arms. With the continuous advancement of sensor technology, computing power, and algorithm research, the autonomy and adaptability of robotic arms have been significantly improved. In the existing technology, rewarded reinforcement learning, as an important machine learning method, has been widely used in the field of robotic arm control to achieve adaptive operations in complex environments.

[0003] However, existing reinforcement learning methods face a series of challenges when applied to robotic arm control. First, the operating environment of the robotic arm is complex and changeable, and the uncertainty of the position and posture of the object increases the difficulty of learning. Second, designing an effective reward function requires deep domain knowledge and a lot of debugging work, which limits the generalization ability of the method and has a decisive impact on the learning process and final performance. In addition, existing methods usually rely on expert demonstration data, which requires access to high-quality expert action data, which may be difficult to obtain in practical applications. To address the above problems, researchers have proposed learning methods based on expert observation states. These methods reduce the dependence on expert action data by utilizing state data or videos provided by experts instead of direct action guidance. However, the performance of these methods is often limited by the objects to be imitated, and in practical applications they rely on the design of simulation data and reward functions, which is still a challenge in a changing environment.

[0004] Although deep learning technology has shown great potential in the state prediction and action decision-making of robotic arms in recent years, it still faces a series of challenges. These challenges include the complexity of reward function design, dependence on expert action data, and restrictions on imitation objects. These factors significantly limit the effectiveness of existing reinforcement learning methods in practical applications. Therefore, a new learning method is urgently needed to overcome these limitations and improve the operational performance of robotic arms in complex and dynamic environments. Summary of the invention

[0005] In view of the above problems existing in the prior art, the present invention is proposed.

[0006] Therefore, the present invention provides a reward-free reinforcement learning method for a robotic arm based on expert state learning to solve the challenges faced by existing robotic arms during task execution, such as difficulty in reward function design, dependence on expert action data, and limitations on imitation objects, thereby improving the operational performance of the robotic arm in complex and dynamic environments.

[0007] In order to solve the above technical problems, the present invention provides the following technical solutions: a non-reward reinforcement learning method for a robot arm based on expert state learning, comprising: collecting state, action and next state data in the robot arm environment to obtain a data set Buffer; constructing an expected state prediction network model ESP suitable for state changes after performing a given action in the current state; training the collected Buffer data in the established expected state prediction network model to obtain trained weights; collecting state data observed by experts to obtain Expert buffer Construct a non-reward strategy network model suitable for predicting the action of the robot arm without reward and expert action; according to the current state and the action predicted by the non-reward strategy network model, the expected state prediction network model with parameter freezing is used to obtain the predicted next state, and the error between the next state and the expert target state is obtained, and then the Train the reward-free policy network model using data.

[0008] As a preferred solution of the non-reward reinforcement learning method of a robotic arm based on expert state learning described in the present invention, wherein: the expected state prediction network model ESP includes a state-action fusion network module and a prediction output network module;

[0009] Among them, the state-action fusion network module is used to extract key features; the prediction output network module is used to obtain the final prediction target.

[0010] As a preferred solution of the non-reward reinforcement learning method of a robotic arm based on expert state learning described in the present invention, wherein: the state-action fusion network module includes a state processing stage and an action injection stage;

[0011] The network in the state processing stage is divided into a state embedder, a downsampling layer, and an upsampling layer. The process includes using several layers of convolutional neural networks and instance normalization layers for processing, sampling training data in the obtained buffer, inputting the state of the training data into the state embedder, converting the original state data into a low-dimensional representation, sending the embedded output to the downsampling layer, mapping the state data to a hidden representation of a fixed dimension, inputting the mapped output into the upsampling layer, and restoring the mapped state representation to the original dimension.

[0012] The action injection stage includes inputting the sampled actions in the buffer into the action embedder, converting the original action data into a low-dimensional representation, combining the action embedded value with the state representation processed by the upsampling layer, multiplying the action embedding with the state using element-by-element multiplication, and adding the original action embedded value directly to the state representation.

[0013] As a preferred solution of the non-reward reinforcement learning method of a robotic arm based on expert state learning described in the present invention, wherein: the prediction output network module includes using a fully connected layer to convert the processed hidden state into a predicted state with the same dimension as the input state as the final result.

[0014] As a preferred solution of the non-reward reinforcement learning method of a robotic arm based on expert state learning described in the present invention, wherein: the trained weights include making a mean square error function of the predicted state obtained by ESP and the actual next state sampled in the Buffer, and learning ESP by minimizing the loss.

[0015] As a preferred solution of the non-reward reinforcement learning method of a robotic arm based on expert state learning described in the present invention, wherein: the collecting state data observed by the expert includes collecting the expert state data by loading the expert model weight and interacting with the environment;

[0016] Collect expert status data frame by frame through expert videos;

[0017] The collected expert observation status data Store into ExpertBuffer;

[0018] The expert status data includes the status data of the expert, but does not include the action data between the expert statuses.

[0019] As a preferred solution of the non-reward reinforcement learning method of a robotic arm based on expert state learning described in the present invention, wherein: the non-reward strategy network model includes a strategy network, an error network, a target strategy network and a target error network.

[0020] As a preferred solution of the non-reward reinforcement learning method of a robotic arm based on expert state learning described in the present invention, wherein: the non-reward strategy network model is trained by data, including initializing strategy network, error network, target strategy network and target error network parameters;

[0021] For each round, RFSNet sets the current state s t As input to RFSNet, and output predicted actions through the policy network:

[0022]

[0023] Among them, θ μ represents the parameters of the policy network, μ(s t |θ μ ) represents the action output by the policy network, represents Gaussian noise;

[0024] The current state s t The action a obtained by the policy network t Put it into the ESP network with frozen parameters to get the new state predicted at this time,

[0025]

[0026] Among them, f represents the network parameters are frozen, and the predicted new state is With expert target status The negative mean square error is recorded as the error between the current round and the target:

[0027]

[0028] in, The final state for the expert to reach the goal;

[0029] The error network is used to calculate the error E of the current state-action pair. w (s t ,a t );

[0030] Predict the next action value through the target policy network output:

[0031] a t+1 =μ′(s i+1 |θ μ′ )

[0032] Among them, μ′ represents the target policy network, θ μ′ represents the parameters of the target policy network;

[0033] The error of the state-action pair at the next moment is obtained through the target error network Get the error in the current state,

[0034]

[0035] The parameters of the policy network are updated by minimizing the loss value. The loss function during network update is:

[0036]

[0037] Where N represents the total number of steps in each cycle, and i represents the i-th step in the cycle.

[0038] A computer device comprises a memory and a processor, wherein the memory stores a computer program, and wherein when the processor executes the computer program, the processor implements the steps of any one of the methods for reward-free reinforcement learning of a robotic arm based on expert state learning.

[0039] A computer-readable storage medium having a computer program stored thereon, characterized in that when the computer program is executed by a processor, the steps of any one of the methods of a robotic arm non-reward reinforcement learning method based on expert state learning are implemented.

[0040] The beneficial effects of the present invention are as follows: by storing the state information of T time steps, the next state information after executing the action, and the action information in the experience pool Buffer, and using these data to build and train the expected state prediction network model ESP; this allows ESP to obtain the predicted new state according to the state and action information of each time step, and train the ESP network through the predicted new state and the real new state, so as to obtain the trained weight; the design of this network enables the robot arm to optimize the action selection through the error between the predicted state and the expert target state without an external reward signal; subsequently, the state data can be collected by interacting with the environment by loading the expert model weight, or by collecting frame by frame through the expert video, and these data do not contain the action information between the expert states. Finally, the collected expert observation state data is stored in ExpertBuffer; a non-reward strategy network model RFSNet for predicting the action of the robot arm is established, and according to the current state and the action predicted by the non-reward strategy network model RFSNet, the predicted next state is obtained through the frozen expected state prediction network model ESP, at which time, the parameters of ESP remain unchanged, so as to calculate the error between the next state and the expert target state. Subsequently, the data of state, action, error and predicted next state are used to train the non-reward strategy network model RFSNet, and finally the trained weights are obtained. The non-reward reinforcement learning method proposed in the present invention effectively solves the challenges faced by the robot arm when performing tasks through the innovative expected state prediction network and non-reward strategy network: first, it optimizes the action selection by predicting the error between the state and the expert target state, avoiding the reliance on the design of complex reward functions in traditional reinforcement learning; second, the method directly learns from the state data observed by the expert, avoiding the reliance on the expert action data, and improving the autonomy and flexibility of learning; finally, the present invention enables the robot arm to optimize the action selection according to the current state and the predicted state through the training of the non-reward strategy network RFSNet, instead of simply imitating the behavior of the expert. This method breaks through the limitation of the imitation object and allows the intelligent agent to adjust the strategy according to the changes in the environment and its own predictions, rather than being restricted to a specific imitation object. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative work.

[0042] Figure 1 A task scenario diagram of a robot arm non-reward reinforcement learning method based on expert state learning in a Franka Emika Panda robot environment provided by an embodiment of the present invention, wherein A is the end effector of the robot arm and B is the target sphere.

[0043] Figure 2 An overall schematic diagram of a non-reward reinforcement learning method for a robotic arm based on expert state learning provided by an embodiment of the present invention.

[0044] Figure 3 An internal schematic diagram of an expected state prediction network model ESP of a robotic arm non-reward reinforcement learning method based on expert state learning provided in an embodiment of the present invention.

[0045] Figure 4 A schematic diagram of the training process of an expected state prediction network model ESP of a robotic arm non-reward reinforcement learning method based on expert state learning provided in an embodiment of the present invention.

[0046] Figure 5 A schematic diagram of expert state data collection for a reward-free reinforcement learning method for a robotic arm based on expert state learning provided in an embodiment of the present invention.

[0047] Figure 6 A robotic arm grasping training result curve of a robotic arm reward-free reinforcement learning method based on expert state learning provided in an embodiment of the present invention.

[0048] Figure 7 The present invention provides a bringball and bringpeg task graph of the Manipulator environment in the DM Control Suite of a robot arm non-reward reinforcement learning method based on expert state learning.

[0049] Figure 8 A performance diagram of the IQM values ​​of a robot arm without reward reinforcement learning method based on expert state learning provided by an embodiment of the present invention under five different baselines and methods. The horizontal axis represents each algorithm, and the vertical axis represents the IQM value range of each algorithm, where the short black line represents the IQM value.

[0050] Fig. 9 A result curve of a non-reward reinforcement learning method for a robotic arm based on expert state learning provided by an embodiment of the present invention compared with a method after adding expert actions. DETAILED DESCRIPTION

[0051] In order to make the above-mentioned purposes, features and advantages of the present invention more obvious and easy to understand, the specific implementation methods of the present invention are described in detail below in conjunction with the drawings of the specification. Obviously, the described embodiments are part of the embodiments of the present invention, but not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary persons in the art without creative work should fall within the scope of protection of the present invention.

[0052] Example 1

[0053] Reference Figure 1-Figure 6 , which is the first embodiment of the present invention, and provides a non-reward reinforcement learning method for a robotic arm based on expert state learning, comprising:

[0054] like Figure 1 As shown, this embodiment takes the 7-DOF Franka Emika Panda robot arm as an example, and the designed task scenario is to move the end effector of the robot arm to the target sphere position. This embodiment discloses a non-reward reinforcement learning method for a robot arm based on expert state learning. Figure 2 As shown, the following steps are included:

[0055] Step S1, collect the state s of the robot environment t 、Action a t and the next state s t+1 The data gets the data set Buffer.

[0056] The agent interacts with the environment randomly and adopts a random action sampling strategy. At each time step t, the agent takes action according to the environment observation value s. t Randomly sample action a t , and observe the new environment state s t+1 , thereby collecting the state-action-next state triplet data in various situations and storing the collected data in the data set Buffer.

[0057] Step S2, constructing an expected state prediction network model ESP suitable for state changes after performing a given action in the current state;

[0058] The expected state prediction network model ESP includes a state-action fusion network module and a prediction output network module; wherein the state-action fusion network module is used to extract key features; and the prediction output network module is used to obtain the final required prediction target.

[0059] The state-action fusion network module includes a state processing stage and an action injection stage. The purpose of the state processing stage is to extract features from the original state. The purpose of the action injection stage is to capture the association between the previous and next states and actions. Specifically, the network in the state processing stage is divided into a state embedder, a downsampling layer, and an upsampling layer. The purpose of the state processing stage is to extract features from the original state.

[0060] like Figure 3 As shown, the internal schematic diagram of the expected state prediction network model ESP of this embodiment, the specific steps include:

[0061] The state processing stage includes: sampling a batch of training data from the previously obtained Buffer, converting the state s of this batch of training data into t First, it is input into the state embedder to convert the original state data into a low-dimensional representation. Then, the embedded output h is sent to the downsampling layer to map the state data to a hidden representation h′ of a fixed dimension. Through this mapping, the dimension is reduced and key features are extracted, thereby reducing the complexity of the data and making it more suitable for processing in the model. The mapped output h′ is input into the upsampling layer to obtain h″, and the mapped state representation is restored to the original dimension.

[0062] The action injection phase includes: injecting the action sampled in the Buffer into t Input into the action embedder to convert the original action data into a low-dimensional representation a′; combine the action-embedded a′ with the state representation processed by the upsampling layer. Here, element-by-element multiplication is used to multiply the action embedding and the state, so that the model can adjust the state representation according to different actions. Subsequently, the original action-embedded a′ is directly added to the state representation, so that the model can consider the direct impact of the action on the state.

[0063] The output prediction stage includes: using a fully connected layer to convert the previously processed hidden state into a predicted state with the same dimension as the input state as the final result.

[0064] Step S3, in the expected state prediction network model established in S2, the Buffer data collected in S1 is trained to obtain the trained weights.

[0065] Specifically, Figure 4 As shown in the figure, the training process diagram of the expected state prediction network model ESP in this embodiment is as follows: The actual next state s sampled in the Buffer t+1 The mean square error (MSE) between is used as the loss function: ESP is learned by minimizing this loss.

[0066] Step S4, collect the state data observed by the expert, the data only contains the state data of the expert, and does not contain the action data between the expert states, and obtain Expertbuffer

[0067] Specifically, Figure 5 As shown in the figure, the expert status data collection schematic diagram of this embodiment, the status data observed by the expert can be collected in two ways, one is to collect the expert status data by loading the expert model weight and interacting with the environment; the other is to collect the expert status data frame by frame through the expert video, where the expert video information can be the expert video, or it can be the expert status data set when the status is picture data; finally, the collected status data observed by the expert is collected. Stored in ExpertBuffer.

[0068] Step S5, constructing a non-reward strategy network model suitable for predicting the robot arm action without reward and expert action.

[0069] Among them, the reward-free policy network model RFSNet includes four groups of neural networks: the policy network is used to output the predicted action, the error network is used to calculate the error of the current state-action pair, the target policy network is used to output the next action prediction value, and the target error network is used to obtain the error of the state-action pair at the next moment.

[0070] Step S6, according to the current state s t and the action a predicted by the no-reward policy network model t The predicted next state is obtained through the expected state prediction network model And get the next state With expert target status The error d t , and then collected The reward-free strategy network model is trained using the data, and steps S4 to S6 are repeatedly performed until the robotic arm completes the task in a timely and accurate manner according to the predicted action.

[0071] Specifically, the training process of the non-reward strategy network model RFSNet includes the following steps:

[0072] Step S6-1, initializing the policy network, error network, target policy network and target error network parameters;

[0073] Step S6-2, for each round, RFSNet sets the current state s t As input to RFSNet, and output predicted actions through the policy network:

[0074] where θ μrepresents the parameters of the policy network, μ(s t |θ μ ) represents the action output by the policy network, represents Gaussian noise;

[0075] Step S6-3, the current state s t The action a obtained by the policy network t Put it into the ESP network with frozen parameters and get the new state predicted at this time: Among them, f represents that the network parameters are frozen. The predicted new state With expert target status The negative mean square error is recorded as the error between the current round and the target: in, The final state for the expert to reach the goal;

[0076] Step S6-4, calculate the error E of the current state-action pair through the error network w (s t ,a t );

[0077] Step S6-5, predict the next action value through the target strategy network output: a t+1 =μ′(s i+1 |θ μ′ ), where μ′ represents the target policy network, θ μ′ represents the parameters of the target policy network;

[0078] Step S6-6, obtain the error of the state-action pair at the next moment through the target error network So we get the error in this state:

[0079] Step S6-7, update the parameters of the policy network by minimizing the loss value. The loss function during network update is: Where N represents the total number of steps in each cycle, and i represents the i-th step in the cycle;

[0080] Step S6-8, repeat S6-1 to S6-7 to optimize and adjust the various parameters of the function, and finally obtain an optimal solution, which is used as the weight of the non-reward strategy network model RFSNet.

[0081] The training curve obtained after the training is as follows Figure 6 As shown, it can be seen from the curve that the convergence speed is fast, wherein the environment score is a negative value, which represents how many steps it will take to complete the task. It can be seen from the curve that the target task is achieved in the later stage of training, verifying the feasibility of the present invention.

[0082] Embodiment 2 is the second embodiment of the present invention, referring to Figure 1 , Figure 6-Figure 9 This embodiment provides a non-reward reinforcement learning method for a robotic arm based on expert state learning, including:

[0083] The setting of the experimental environment is introduced, and the effectiveness of the method is verified by comparison with different baselines and performance evaluation, and an ablation study is performed for in-depth analysis. The experimental results show that the method performs well in multiple tasks of the DM Control Suite and FrankaEmikaPandarobot environments.

[0084] DM Control Suite (DMC). The DM Control package is a collection of Python libraries and task suites for reinforcement learning agents in articulated body simulation. The Manipulator environment in DMC is an environment in which a robotic arm simulates bringing an object to a target location. The bringball task under it is a robotic arm manipulation task involving a 6-degree-of-freedom robotic arm, a sphere, and a target point. The state contains a color image from a 64x64x3 camera, as well as data on the robotic arm position, speed, touch sensor, gripper position, object position, object speed, and target position. The action contains five continuous values, corresponding to the operations of the actuators such as translation and rotation of the robotic arm. The goal of the agent's robotic arm is to grab the sphere from its original position to the target position to obtain a reward. In addition to the official benchmark task bringball, the experiment also performed the bringpeg task (bringing the wooden peg to the target wooden peg position), such as Figure 7 As shown, this is the operation diagram of the two tasks.

[0085] FrankaEmikaPandarobot. The FrankaEmikaPandarobot environment is a reinforcement learning environment for the FrankaEmika Panda robot from OpenAI Gym. In the reach task in the Franka EmikaPandarobot environment, a robot arm with a two-fingered parallel gripper and 7 degrees of freedom is used in the environment using the MuJoCo physics engine. The state includes the angles and velocities of the robot's joints, as well as the position of the object. Since none of the problems considered require the gripper to rotate, the gripper is kept fixed, where the action space is three-dimensional, and the three dimensions specify the relative gripper position required for the next time step. Use The Euclidean distance operation is used as the basis for judging whether the target task is completed. If d≤0.05, it can be considered that the target is reached. The goal of the agent is to reach the target position in the shortest time step. Note that the score of the Franka Emika Panda robot environment is a negative value, which represents the number of steps to reach the target position. The maximum number of steps per round is 50. In the experiment, Hindsight Experience Replay (HER) was introduced. By adjusting the target position in the experience replay, the agent can learn from the failed experience, thereby accelerating the training process. Figure 1 The following is the operation diagram of this task.

[0086] Baselines. Five reward-free baselines are tested. The first is the Random baseline, which is a common strategy in most reinforcement learning tasks. The next baseline is the Plan2Explore (P2E) baseline, which optimizes the expected information gain of the model by training a single strategy. Then, a population-based version of P2E, called Population Plan2Explore (PP2E), is also compared, which trains a group of randomly initialized agents that independently maximize the expected information gain. The fourth baseline is Coordinated Active Sample Collection via Diverse Explorers (CASCADE), a self-supervised method that aims to solve the reward-free deployment efficiency problem. This method collects a large amount of useful data by training a group strategy and learns through the model. The last baseline is Inverse Preference Learning (IPL), an algorithm with efficient parameters, low computational cost and small performance fluctuations, which directly learns and optimizes policies from preference data without explicitly learning reward functions.

[0087] In the experiment, the performance of 3 tasks on 5 baselines was compared in the two different environments mentioned above. During the data collection process, each baseline collected 100k data per round for training, and a total of 5 rounds were conducted. After each round, the performance of the model was evaluated to promote the learning of specific behaviors.

[0088] The experimental results are analyzed as follows:

[0089] To verify the effectiveness of the proposed method, the performance of the RL agent in the robotic arm task was evaluated from two aspects: average return and data efficiency. The average return represents the long-term benefit of the strategy, while the data efficiency characterizes the efficiency of completing the task, that is, the amount of data required to complete the task. In addition, in order to further compare the performance of the method with the baseline, Figure 8In the DM Control Suite, the results of all tasks (Bring ball, Bringpeg) were combined and the interquartile mean (IQM) of the normalized scores was calculated. Table 1 shows the average return of completing different baseline related tasks in different environments. Table 2 shows the numerical efficiency of completing different baseline tasks in the Bring ball and Franka Emika Panda robot environments. Figure 6 The performance of the unrewarded policy network is shown.

[0090] Table 1. Average returns on three tasks evaluated in two environments relative to five baseline methods.

[0091]

[0092] Table 2. Data efficiency performance comparison of the Bringball and Panda Reach tasks trained with five baselines at different data amounts.

[0093]

[0094]

[0095] Here is the result analysis:

[0096] Average Return and Data Efficiency. In this section, we comprehensively evaluate the performance of our method against five baseline methods (Random, P2E, PP2E, CASCADE, and IPL) in terms of average return and data efficiency. As shown in Tables 1 and 2, our proposed expert-observed state-free reinforcement learning framework significantly outperforms these baseline methods in terms of average return for all tasks, and can effectively complete the tasks with only 300k time steps of data. The experimental results show that the Random, P2E, and PP2E baselines do not perform well. This may be because the Random baseline lacks directionality and goal orientation and cannot effectively learn policies in a reward-free environment. The P2E and PP2E baselines rely on environment models for self-supervised learning, but in the absence of reward signals, these models may not provide enough learning signals to complete the robotic grasping task. Although the CASCADE baseline performs well in two DMC tasks (Bringball and Bringpeg), it performs poorly in the Reach task. This is due to the lack of adaptability of CASCADE's reward-free framework to different environments and tasks, and insufficient attention to specific task goals, which leads to its low learning efficiency and unstable performance in the Reach task. In addition, although the IPL baseline learns policies through human preference data, avoids the complexity of traditional reward design, and performs well on all tasks, it relies on human feedback, which may be difficult to obtain and susceptible to noise in reward-free environments, and this implicit reward function still needs to be manually designed. Compared with these methods, our expected state prediction network (ESP) is able to adapt to different types of tasks, including numerical and image environments. By predicting the dynamic changes of the environment through ESP, we are able to provide accurate target state information to the reward-free policy network (RFSNet), and combined with the goal-driven optimization of the policy network, the agent can still explicitly optimize the policy in the absence of external reward signals.

[0097] The interquartile mean (IQM) of the normalized scores is used. The IQM provides an accurate measure of central tendency that is resistant to outliers. To compare the performance of the methods with the baselines (Random, P2E, PP2E, CASCADE, IPL), the results of all tasks in DMC (Bringball, Bringpeg) are aggregated and presented in Figure 8The quartile means of the normalized scores are shown in . The results show that the method of significantly outperforms all baselines in terms of IQM values. The framework of accurately identifies key states through ESP and combines the error between the state and the target state to help the agent use data more effectively. At the same time, it relies on expert states to directly learn strategies without designing complex reward functions, thus avoiding the uncertainty caused by reward design. These features ensure the stability and consistency of the scores of the method of in multiple tasks, making the framework of significantly outperform other baselines in IQM performance.

[0098] Performance of the Reward-Free Policy Network. In this section, the performance of the Reward-Free Policy Network (RFSNet) during training is shown to demonstrate the key contribution of the method. Taking the Reach task of the Franka Emika Panda robot environment as an example, Figure 6 As shown in the figure, by tracking the score changes of RFSNet in multiple training steps, it is observed that although the score fluctuates greatly in the initial stage, which may be due to the randomness of network initialization and exploration strategy, as the training progresses, the score gradually stabilizes and rises, indicating that RFSNet is learning effective strategies to predict and reach the target state. This trend proves that even in the absence of external reward signals, RFSNet can optimize the strategy by reducing the difference between the predicted state and the target state. This result highlights the advantages of the method: through the accurate state prediction provided by the expected state prediction network (ESP) and the goal-driven optimization of the reward-free policy network (RFSNet), RFSNet can effectively learn and perform tasks even in the absence of external reward signals.

[0099] Ablation studies:

[0100] Ablation study of expert observation module and HER module. Figure 1 In the Reach task in the Franka Emika Panda robot environment shown in Figure 3, we study the framework models after removing different components. In Table 3, “-E(·)” and “-HER(·)” correspond to the framework models after removing the expert observation module and the HER module from our framework, respectively. “-E(·)-HER(·)” represents the framework model after removing both the expert observation and HER modules from our method. We compare three cases: our framework, the framework after removing the HER module, and the framework after removing both the expert observation and HER modules. From Tables 3 and Figure 4The experimental results show that "E(·)" and "HER(·)" are crucial to the performance of the model. Specifically, in 1000 training rounds, the scores of the model after removing "HER(·)" in 50, 100, 200 and 1000 steps of training decreased by 11.80, 26.77, 38.04 and 8.89, respectively, while the scores of the model after removing "E(·)HER(·)" at the same time decreased by 14.62, 27.78, 38.51 and 14.47, respectively. This shows that the participation of expert observation significantly accelerates the training of the model and improves the efficiency of completing the robot arm task; at the same time, the HER module also plays a key role in the model.

[0101] Sensitivity analysis of parameters. Figure 1 In the Reach task of the Franka Emika Panda robot environment shown in Figure 3, we perform parameter sensitivity analysis by setting different hyperparameters. In order to analyze the impact of several key hyperparameters, the hyperparameters of RFSNet "α=0.001", "β=0.002" and "tau=0.005" (α is the Actor network learning rate, β is the Critic network learning rate, and tau is the target network update coefficient) are adjusted to "α=0.005", "β=0.010" and "tau=0.010" respectively. From the experimental results in Table 3, it can be seen that compared with the baselines of "α=0.005", "β=0.010" and "tau=0.010", these hyperparameters have less impact on the model performance, indicating that the model is not sensitive to hyperparameter changes.

[0102] Table 3. Sensitivity analysis of the parameters used

[0103]

[0104] Supplementary experiments:

[0105] Performance analysis after using expert actions. In this section, we use experiments to show that our framework is also applicable to the use of expert actions. Specifically, in order to effectively integrate expert actions, we collect action data between expert states while collecting expert state data. During each round of RFSNet training, we pass the expert action into RFSNet and calculate the error between the output action of the current policy network and the expert action. This error is introduced into the error network in a certain proportion to guide the training of the policy network. In addition, we replace the predicted state values ​​in the original framework with the actual state values ​​to further verify the accuracy of the expected state prediction network (ESP). At the same time, we introduce IPL as the second-best baseline for reference. The results are shown in Figure 2. Fig. 9As shown in Figure 3, our method performs well in the framework that combines expert actions, indicating that our framework is also applicable to this scenario. In addition, our method also performs well in the framework that uses the true state value, reflecting that our ESP network can accurately predict the true new state. In contrast, the performance of the IPL baseline is slightly insufficient, which is related to the instability of the learning process caused by its reliance on human preference feedback.

[0106] Embodiment 3: This embodiment further provides a computing device, which is applicable to a method for non-reward reinforcement learning of a robotic arm based on expert state learning, including:

[0107] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc., which can store program codes.

[0108] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as an ordered list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by an instruction execution system, device or apparatus (such as a computer-based system, a system including a processor, or other system that can fetch instructions from an instruction execution system, device or apparatus and execute instructions), or in conjunction with such instruction execution systems, devices or apparatuses. For the purposes of this specification, "computer-readable medium" can be any device that contains storage, communication, propagation or transmission of a program for use by an instruction execution system, device or apparatus, or in conjunction with such instruction execution systems, devices or apparatuses.

[0109] More specific examples of computer-readable media (a non-exhaustive list) include the following: an electrical connection with one or more wires (electronic device), a portable computer disk case (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), an optical fiber device, and a portable compact disk read-only memory (CDROM). In addition, the computer-readable medium may even be a paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, deciphering or, if necessary, processing in another suitable manner, and then stored in a computer memory.

[0110] It should be understood that the various parts of the present invention can be implemented by hardware, software, firmware or a combination thereof. In the above-mentioned embodiments, a plurality of steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

Claims

1. A non-reward reinforcement learning method for a robotic arm based on expert state learning, characterized in that: include, Collect the state, action and next state data in the robot environment to obtain the data set Buffer; Construct a network model for predicting the expected state after performing a given action in the current state; In the established expected state prediction network model, the collected buffer data is trained to obtain the trained weights; Collect the status data observed by experts to obtain the expert data set Expert buffer Construct a non-reward strategy network model suitable for predicting robot arm actions without rewards and expert actions; According to the current state and the action predicted by the no-reward strategy network model, the expected state prediction network model with the network parameters frozen is used to predict the next state, and the error between the next state and the expert target state is obtained, and then the Train the reward-free policy network model using data.

2. A non-reward reinforcement learning method for a robotic arm based on expert state learning as claimed in claim 1, characterized in that: The desired state prediction network model includes a state-action fusion network module and a prediction output network module; Among them, the state-action fusion network module is used to extract key features; the prediction output network module is used to obtain the final prediction target.

3. A non-reward reinforcement learning method for a robotic arm based on expert state learning as claimed in claim 2, characterized in that: The state-action fusion network module includes a state processing stage and an action injection stage; The network in the state processing stage is divided into a state embedder, a downsampling layer, and an upsampling layer. It uses several layers of convolutional neural networks and instance normalization layers for processing. According to the training data sampled in the buffer, the state of the training data is input into the state embedder, the original state data is converted into a low-dimensional representation, the embedded output is sent to the downsampling layer, the state data is mapped to a hidden representation of a fixed dimension, the mapped output is input into the upsampling layer, and the mapped state representation is restored to the original dimension; The action injection stage includes inputting the sampled actions in the buffer into the action embedder, converting the original action data into a low-dimensional representation, combining the action embedded value with the state representation processed by the upsampling layer, multiplying the action embedding with the state using element-by-element multiplication, and adding the original action embedded value directly to the state representation.

4. A non-reward reinforcement learning method for a robotic arm based on expert state learning as claimed in claim 3, characterized in that: The prediction output network module includes using a fully connected layer to convert the processed hidden state into a prediction state with the same dimension as the input state as the final result.

5. The method for non-reward reinforcement learning of a robotic arm based on expert state learning as claimed in claim 4, characterized in that: The obtained trained weights include performing a mean square error function on the predicted state obtained by the expected state prediction network and the actual next state sampled in the buffer, learning the expected state prediction network by minimizing the loss, and obtaining the trained weights of the expected state prediction network.

6. A non-reward reinforcement learning method for a robotic arm based on expert state learning as claimed in claim 5, characterized in that: The collecting of the state data observed by the expert includes collecting the expert state data by loading the expert model weights and interacting with the environment or collecting the expert state data frame by frame through the expert video; The collected expert observation status data Store into ExpertBuffer; The expert status data only includes the status data of the expert, and does not include the action data between the expert states.

7. A non-reward reinforcement learning method for a robotic arm based on expert state learning as claimed in claim 6, characterized in that: The non-reward policy network model includes a policy network, an error network, a target policy network and a target error network.

8. A non-reward reinforcement learning method for a robotic arm based on expert state learning as claimed in claim 7, characterized in that: The data-based training of the non-rewarded policy network model includes initializing policy network, error network, target policy network and target error network parameters; For each round, RFSNet sets the current state s t As the input of RFSNet, and output the predicted action through the policy network, it is expressed as: Among them, θ μ represents the parameters of the policy network, μ(s t |θ μ ) represents the action output by the policy network, represents Gaussian noise; The current state s t The action a obtained by the policy network t Put it into the ESP network with frozen parameters to get the new state of the current prediction, which is expressed as: Among them, f represents the network parameters are frozen, and the predicted new state is With expert target status The negative mean square error is recorded as the error between the current round and the target, expressed as: in, The final state for the expert to reach the goal; The error network is used to calculate the error E of the current state-action pair. w (s t ,a t ); The next action value is predicted by the target policy network output, expressed as: a t+1 =μ′(s i+1 |θ μ′ ) Among them, μ′ represents the target policy network, θ μ′ Represents the parameters of the target policy network; The error of the state-action pair at the next moment is obtained through the target error network The error in the current state is obtained, expressed as: The parameters of the policy network are updated by minimizing the loss value. The loss function during network update is expressed as: Where N represents the total number of steps in each cycle, and i represents the i-th step in the current cycle.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 8 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 8 are implemented.

Images