A mechanical arm reward-free reinforcement learning method based on expert state learning

Abstract

The application provides a mechanical arm non-reward reinforcement learning method based on expert state learning, belongs to the field of automation technology, and is particularly concerned about the operation control of a robot arm.The method comprises two core modules: an expected state prediction network and a non-reward policy network.The expected state prediction network is used for predicting state conversion, and the non-reward policy network optimizes actions through the error between the predicted state of the expected state prediction network and the target state provided by an expert without a reward signal; this method enables the mechanical arm to directly learn an effective grasping strategy from the state data observed by the expert, avoids complex reward function design, and does not need to rely on expert action data, thereby improving the understanding of the dynamic characteristics of the grasping task and enhancing the accuracy of action prediction.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of automation technology, and in particular to a mechanical arm reward-free reinforcement learning method based on expert state learning. BACKGROUND

[0002] As a core component of intelligent manufacturing, the development of control technology for automated mechanical arms is of great significance to improving production efficiency and quality. Mechanical arms simulate human arm movements to perform precise tasks such as grasping, carrying, and assembly. With the continuous progress of sensor technology, computing power, and algorithm research, the autonomy and adaptability of mechanical arms have been significantly improved. In existing technology, reward-based reinforcement learning, as an important machine learning method, has been widely applied in the field of mechanical arm control to achieve adaptive operation in complex environments.

[0003] However, existing reinforcement learning methods face a series of challenges when applied to mechanical arm control. First, the operating environment of mechanical arms is complex and variable, and the uncertainty of object position and posture increases the difficulty of learning. Second, designing an effective reward function requires deep domain knowledge and a lot of debugging work, which limits the generalization ability of the method and has a decisive impact on the learning process and final performance. In addition, existing methods usually rely on expert demonstration data, which requires access to high-quality expert action data, which may be difficult to obtain in practical applications. To solve the above problems, researchers have proposed learning methods based on expert observation states. These methods use expert-provided state data or videos instead of direct action guidance, reducing the dependence on expert action data. However, the performance of these methods is often limited by the imitation object, and in practical applications, they rely on the design of simulation data and reward functions, which is still a challenge in a variable environment.

[0004] Although deep learning technology has shown great potential in state prediction and action decision-making for mechanical arms in recent years, it still faces a series of challenges. These challenges include the complexity of reward function design, dependence on expert action data, and limitations of imitation objects. These factors significantly limit the effectiveness of existing reinforcement learning methods in practical applications. Therefore, there is an urgent need for a new learning method to overcome these limitations and improve the operating performance of mechanical arms in complex and dynamic environments. SUMMARY

[0005] In view of the problems existing in the prior art, the present application is proposed.

[0006] Therefore, this invention provides a rewardless reinforcement learning method for robotic arms based on expert state learning, in order to solve the challenges faced by existing robotic arms in task execution, such as the difficulty in designing reward functions, dependence on expert action data, and limitations of imitation objects, thereby improving the operational performance of robotic arms in complex and dynamic environments.

[0007] To address the aforementioned technical problems, this invention provides the following technical solution: a rewardless reinforcement learning method for robotic arms based on expert state learning, comprising: collecting state, action, and next state data from the robotic arm environment to obtain a dataset buffer; constructing an expected state prediction network model ESP suitable for predicting state changes after performing a given action in the current state; training the collected buffer data in the established expected state prediction network model to obtain trained weights; and collecting state data observed by experts to obtain an expert buffer. A rewardless policy network model is constructed to predict robotic arm movements in situations where there is no reward or expert action. Based on the current state and the movement predicted by the rewardless policy network model, the next state is predicted through a parameter-frozen desired state prediction network model. The error between the next state and the expert target state is then calculated. Subsequently, data is collected. Train a network model with a no-reward policy using data.

[0008] As a preferred embodiment of the expert state learning-based robotic arm no-reward reinforcement learning method described in this invention, the expected state prediction network model ESP includes a state-action fusion network module and a prediction output network module.

[0009] The state-action fusion network module is used to extract key features, while the prediction output network module is used to obtain the final prediction target.

[0010] As a preferred embodiment of the expert state learning-based robotic arm no-reward reinforcement learning method described in this invention, the state-action fusion network module includes a state processing stage and an action injection stage.

[0011] The network in the state processing stage is divided into a state embedder, a downsampling layer, and an upsampling layer. The process includes processing the data using several layers of convolutional neural networks and instance normalization layers, sampling the training data from the obtained buffer, inputting the state of the training data into the state embedder, converting the original state data into a low-dimensional representation, sending the embedded output into the downsampling layer, mapping the state data to a fixed-dimensional hidden representation, inputting the mapped output into the upsampling layer, and restoring the mapped state representation to the original dimension.

[0012] The action injection stage includes inputting the sampled action in the Buffer into the action embedder, converting the original action data into a low-dimensional representation, combining the action-embedded value with the state representation processed by the up-sampling layer, multiplying the action embedding and the state using element-wise multiplication, and directly adding the value of the original action embedding to the state representation.

[0013] As a preferred scheme of the mechanical arm reward-free reinforcement learning method based on expert state learning, the prediction output network module includes a full connection layer for converting the processed hidden state into a prediction state with the same dimension as the input state as the final result.

[0014] As a preferred scheme of the mechanical arm reward-free reinforcement learning method based on expert state learning, the trained weight includes an ESP prediction state and a real next state sampled in the Buffer, and the ESP is learned by minimizing the loss.

[0015] As a preferred scheme of the mechanical arm reward-free reinforcement learning method based on expert state learning, the collected expert observed state data includes expert state data collected by loading expert model weights and interacting with the environment.

[0016] The expert state data is collected frame by frame from the expert video.

[0017] The collected expert observed state data is stored in the ExpertBuffer.

[0018] The expert state data includes the state data of the expert, and does not include the action data between the expert states.

[0019] As a preferred scheme of the mechanical arm reward-free reinforcement learning method based on expert state learning, the reward-free policy network model includes a policy network, an error network, a target policy network and a target error network.

[0020] As a preferred scheme of the mechanical arm reward-free reinforcement learning method based on expert state learning, the data training of the reward-free policy network model includes initialization of the parameters of the policy network, the error network, the target policy network and the target error network.

[0021] For each round, the RFSNet takes the current state s t as the input and outputs the predicted action through the policy network:

[0022]

[0023] Where, θ μ The parameters representing the policy network, μ(s) t |θ μ ) represents the action output by the policy network. Represents Gaussian noise;

[0024] The current state s t Action a obtained from the policy network t By feeding the data into the ESP network with frozen parameters, we obtain the predicted new state.

[0025]

[0026] Where f represents the network parameters being frozen, resulting in the predicted new state. With expert target state The negative mean square error is denoted as the error between the current round and the target:

[0027]

[0028] in, This represents the final state in which experts reach their goal;

[0029] The error E of the current state-action pair is calculated using an error network. w (s t ,a t );

[0030] Predict the next action value by outputting the target policy network:

[0031] a t+1 =μ′(s i+1 |θ μ′ )

[0032] Where μ′ represents the target policy network, θ μ′ The parameters representing the target policy network;

[0033] The error of the state-action pair at the next moment is obtained through the target error network. To obtain the error under the current state,

[0034]

[0035] The parameters of the policy network are updated by minimizing the loss value. The loss function for network updates is:

[0036]

[0037] Where N represents the total number of steps in each cycle, and i represents the i-th step in that cycle.

[0038] A computer device comprises a memory and a processor, and the memory stores a computer program, characterized in that the processor implements the steps of any one of the methods in the expert state learning-based robot arm reward-free reinforcement learning method when executing the computer program.

[0039] A computer readable storage medium stores a computer program, characterized in that the computer program implements the steps of any one of the methods in the expert state learning-based robot arm reward-free reinforcement learning method when executed by a processor.

[0040] The present application has the following beneficial effects: by storing the state information of T time steps, the next state information after performing the action, and the action information into the experience pool Buffer, and using these data to build and train the expected state prediction network model ESP, the ESP can obtain the predicted new state according to the state and action information of each time step, and the ESP network is trained through the predicted new state and the real new state, so as to obtain the trained weight; the design of this network enables the robot arm to optimize the action selection through the error between the predicted state and the expert target state without external reward signals; subsequently, the collection of state data can be achieved by loading the expert model weight and interacting with the environment, or by collecting the data frame by frame from the expert video, and these data do not contain the action information between the expert states. Finally, the collected expert observation state data are stored into the ExpertBuffer; the reward-free policy network model RFSNet for predicting the robot arm action is established, and the predicted next state is obtained through the frozen expected state prediction network model ESP according to the current state and the action predicted by the reward-free policy network model RFSNet, at this time, the parameters of the ESP remain unchanged, so as to calculate the error between the next state and the expert target state. Subsequently, the data of the state, action, error, and predicted next state are used to train the reward-free policy network model RFSNet, and finally the trained weight is obtained. The reward-free reinforcement learning method proposed in the present application effectively solves the challenges faced by the robot arm in performing tasks through the innovative expected state prediction network and reward-free policy network: firstly, it optimizes the action selection through the error between the predicted state and the expert target state, avoiding the dependence on the design of complex reward functions in traditional reinforcement learning; secondly, the method directly learns from the expert observation state data, avoiding the dependence on the expert action data, and improving the autonomy and flexibility of learning; finally, the training of the reward-free policy network RFSNet enables the robot arm to optimize the action selection according to the current state and the predicted state, rather than simply imitating the behavior of the expert, which breaks through the limitation of the imitation object and allows the intelligent agent to adjust the strategy according to the changes in the environment and its own prediction, rather than being limited to a specific imitation object. BRIEF DESCRIPTION OF DRAWINGS

[0041] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings needed to be used in the embodiments will be briefly introduced as follows. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without any creative effort on the basis of these drawings.

[0042] Figure 1 A task scene graph of a mechanical arm reward-free reinforcement learning method based on expert state learning provided by the embodiment of the present application in a Franka Emika Panda robot environment, wherein A is a mechanical arm end effector, and B is a target ball.

[0043] Figure 2 A general schematic diagram of a mechanical arm reward-free reinforcement learning method based on expert state learning provided by the embodiment of the present application.

[0044] Figure 3 An internal schematic diagram of an expected state prediction network model ESP of a mechanical arm reward-free reinforcement learning method based on expert state learning provided by the embodiment of the present application.

[0045] Figure 4 A training process schematic diagram of an expected state prediction network model ESP of a mechanical arm reward-free reinforcement learning method based on expert state learning provided by the embodiment of the present application.

[0046] Figure 5 An expert state data collection schematic diagram of a mechanical arm reward-free reinforcement learning method based on expert state learning provided by the embodiment of the present application.

[0047] Figure 6 A mechanical arm grasping training result curve of a mechanical arm reward-free reinforcement learning method based on expert state learning provided by the embodiment of the present application.

[0048] Figure 7 A bringball and bringpeg task graph of a Manipulator environment in a DM Control Suite of a mechanical arm reward-free reinforcement learning method based on expert state learning provided by the embodiment of the present application.

[0049] Figure 8 A performance graph of IQM values under 5 different baseline methods of a mechanical arm reward-free reinforcement learning method based on expert state learning provided by the embodiment of the present application. The horizontal axis represents each algorithm, and the vertical axis represents the IQM value range of each algorithm, wherein the black short line represents the IQM value.

[0050] Figure 9 A result curve compared with adding expert actions of a mechanical arm no-reward reinforcement learning method based on expert state learning provided by an embodiment of the present application. DETAILED DESCRIPTION

[0051] In order to make the above objectives, characteristics and advantages of the present application more obvious and easy to understand, the specific embodiments of the present application will be described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor should fall within the scope of protection of the present application.

[0052] Embodiment 1

[0053] Reference Figures 1-6 For the first embodiment of the present application, the embodiment provides a mechanical arm no-reward reinforcement learning method based on expert state learning, comprising:

[0054] As shown in the figure, the embodiment takes a Franka Emika Panda robot arm with 7 degrees of freedom as an example, and the task scenario designed is to move the end effector of the robot arm to the target ball position. The embodiment discloses a mechanical arm no-reward reinforcement learning method based on expert state learning as shown in the figure, comprising the following steps: Figure 1 Figure 2 Step S1, collecting state s t , action a t and next state s t+1 data in the environment of the robot arm to obtain a data set Buffer.

[0055] Among them, the agent and the environment interact randomly, and a random action sampling strategy is adopted. At each time step t, the agent randomly samples an action a t according to the environment observation value s t , and observes a new environment state s t+1 , so as to collect state-action-next state triple data under various conditions, and store the collected data into the data set Buffer.

[0056] Step S2, constructing an expected state prediction network model ESP suitable for state change after executing a given action in the current state;

[0057] The expected state prediction network model ESP includes a state-action fusion network module and a prediction output network module; wherein the state-action fusion network module is used to extract key features; and the prediction output network module is used to obtain the final required prediction target.

[0058] The expected state prediction network model ESP includes a state-action fusion network module and a prediction output network module; wherein the state-action fusion network module is used to extract key features; and the prediction output network module is used to obtain the final required prediction target. ​

[0059] The state-action fusion network module comprises a state processing stage and an action injection stage. The state processing stage aims to extract features from the original state; the action injection stage aims to capture the correlation between previous and subsequent states and actions. Specifically, the network in the state processing stage is divided into a state embedder, a downsampling layer, and an upsampling layer. The purpose of the state processing stage is to extract features from the original state.

[0060] like Figure 3 As shown in the diagram, the internal structure of the desired state prediction network model ESP in this embodiment is illustrated. The specific steps include:

[0061] The state processing phase includes: sampling a batch of training data from the previously obtained buffer, and processing the state s of this batch of training data. t First, the data is fed into a state embedder to convert the original state data into a low-dimensional representation. Then, the embedded output h is fed into a downsampling layer to map the state data to a fixed-dimensional hidden representation h′. This mapping reduces the dimensionality and extracts key features, thereby reducing the complexity of the data and making it more suitable for processing in the model. The mapped output h′ is then fed into an upsampling layer to obtain h″, which restores the mapped state representation to the original dimension.

[0062] The action injection phase includes: sampling action a in the buffer. t The input is fed into the action embedder, which converts the original action data into a low-dimensional representation a′. The action-embedded a′ is then combined with the state representation processed by the upsampling layer. Here, element-wise multiplication is used to multiply the action embedding with the state, so that the model can adjust the state representation according to different actions. Then, the original action embedding a′ is directly added to the state representation, so that the model can consider the direct impact of actions on the state.

[0063] The output prediction stage includes: using a fully connected layer to transform the previously processed hidden states into predicted states of the same dimension as the input states as the final result.

[0064] Step S3: In the desired state prediction network model established in S2, train the buffer data collected in S1 to obtain the trained weights.

[0065] Specifically, such as Figure 4 As shown, this embodiment illustrates the training process of the Expected State Prediction Network (ESP) model, which predicts the state. The actual next state s sampled in the buffer t+1 The mean squared error (MSE) between the two sides is used as the loss function: ESP is learned by minimizing this loss.

[0066] Step S4, collect the state data observed by the expert, the data only contains the state data of the expert, and does not contain the action data between the states of the expert, and obtain Expertbuffer

[0067] Specifically, as shown in the expert state data collection schematic diagram of the embodiment, Figure 5 The state data observed by the expert can be collected in two ways. One way is to collect the state data of the expert by loading the expert model weight and interacting with the environment. The other way is to collect the state data of the expert frame by frame through the expert video. The expert video information here can be the expert video, or the expert state data set when the state is picture data. Finally, the collected state data observed by the expert is stored in ExpertBuffer.

[0068] Step S5, construct a reward-free policy network model suitable for predicting the action of the robot arm in the case of no reward and no expert action.

[0069] The reward-free policy network model RFSNet includes four groups of neural networks. They are a policy network for outputting a predicted action, an error network for calculating the error of the current state-action pair, a target policy network for outputting a next action prediction value, and a target error network for obtaining the error of the state-action pair at the next moment.

[0070] Step S6, according to the current state s t and the action a t predicted by the reward-free policy network model, obtain the predicted next state s through the expected state prediction network model, and obtain the error d between the next state s and the target state s t of the expert, and then collect the data through the data to train the reward-free policy network model, and repeat the steps S4 to S6 until the robot arm accurately completes the task in time according to the predicted action.

[0071] Specifically, the training process of the reward-free policy network model RFSNet includes the following steps:

[0072] Step S6-1, initialize the parameters of the policy network, the error network, the target policy network and the target error network;

[0073] Step S6-2, for each round, RFSNet takes the current state s t as the input of RFSNet, and outputs the predicted action a through the policy network:

[0074] where θ μrepresent the parameters of the policy network, μ(s t |θ μ ) represents the action output by the policy network, representing Gaussian noise;

[0075] Step S6-3, put the current state s t and the action a t obtained by the policy network into the ESP network with frozen parameters, to obtain the predicted new state at this time: wherein f represents that the network parameters are frozen. The obtained predicted new state and the negative mean square error of the expert target state is recorded as the error of the current round and the target: wherein, is the final state of the expert reaching the target;

[0076] Step S6-4, calculate the error E w (s t ,a t ) of the current state-action pair through the error network;

[0077] Step S6-5, output the predicted next action value a t+1 through the target policy network: a i+1 = μ'(s μ′ |θ μ′ ), wherein μ' represents the target policy network, and θ μ′ represents the parameters of the target policy network;

[0078] Step S6-6, obtain the error of the state-action pair at the next time through the target error network Thus, the error at this state is obtained:

[0079] Step S6-7, update the parameters of the policy network by minimizing the loss value, and the loss function during network updating is: wherein N represents the total number of steps in each period, and i represents the i-th step in the period;

[0080] Step S6-8, repeat S6-1 to S6-7 to optimize and adjust each parameter of the function, and finally obtain an optimal solution, which is used as the weight of the reward-free policy network model RFSNet.

[0081] The training curve obtained after training is shown in Figure 6 From the curve, it can be seen that the convergence speed is fast, wherein the environment score is negative, which represents how many steps can complete the task. It can be seen from the curve that the target task is achieved in the later training period, which verifies the feasibility of the present application.

[0082] Embodiment 2, which is a second embodiment of the present application, is described with reference to Figure 1 , Figures 6-9 This embodiment provides a mechanical arm reward-free reinforcement learning method based on expert state learning, comprising:

[0083] The setting of the experimental environment is introduced, and the effectiveness of the method is verified through comparison with different baselines and performance evaluation, and ablation studies are conducted for in-depth analysis. The experimental results show that the method performs well in multiple tasks in the DM Control Suite and Franka Emika Panda robot environments.

[0084] DM Control Suite (DMC). DM Control is a collection of Python libraries and task suites for reinforcement learning agents in simulated jointed body. The Manipulator environment in DMC is an environment for a simulated robotic arm to bring an object to a target location. The bringball task under it is a robotic arm manipulation task that contains a 6-DOF robotic arm, a ball, and a target location. The state contains a 64x64x3 color image of a camera, in addition to the data of the robotic arm position, velocity, tactile sensor, gripper position, object position, object velocity, and target position. The action contains five continuous values corresponding to the operations of the robotic arm's translational and rotational actuators. The goal of the agent robotic arm is to grasp the ball from the original position to the target position to obtain a reward. In addition to the official benchmark task bringball, the experiment also performs the bringpeg task (bring a wooden peg to the target peg position), as shown in Figure 7 Figure 2, which is an operation diagram of the two tasks.

[0085] Franka Emika Panda robot. The Franka Emika Panda robot environment is a reinforcement learning environment for the Franka Emika Panda robot of OpenAI Gym. In the reach task under the Franka Emika Panda robot environment, the MuJoCo physics engine is used, and a robotic arm with a double-finger parallel gripper and 7 degrees of freedom is used in the environment. The state includes the angles and velocities of the robot joints and the position of the object. Since the problem considered does not require the gripper to rotate, the gripper is kept fixed, and the action space is three-dimensional, which specifies the relative gripper position required for the next time step. The Euclidean distance is used as the criterion to determine whether the goal is achieved. If d≤0.05, it can be considered that the goal is reached. The goal of the agent is to reach the target position in the shortest time step. Note that the score of the Franka Emika Panda robot environment is negative, representing the number of steps to reach the target position, and the maximum number of steps per episode is 50. In the experiment, Hindsight Experience Replay (HER) is introduced, which adjusts the target position in the experience replay, enabling the agent to learn from failed experiences and accelerating the training process. Figure 1 The operation graph of the task is shown.

[0086] Baselines. Five reward-free baselines are tested. The first is the random policy (Random) baseline, which is commonly used in most reinforcement learning tasks. The next baseline is the Plan2Explore (P2E) baseline, which optimizes the expected information gain by training a single policy optimization model. Then, a population-based P2E version called Population Plan2Explore (PP2E) is also compared, which trains a group of randomly initialized agents that independently maximize the expected information gain. The fourth baseline is Coordinated Active Sample Collection via Diverse Explorers (CASCADE), a self-supervised method designed to address the reward-free deployment efficiency problem, which collects a large amount of useful data by training a group of strategies and learns through a model. The last baseline is Inverse Preference Learning (IPL), which is a parameter-efficient, computationally inexpensive, and performance-stable algorithm that directly learns and optimizes the strategy from preference data without explicitly learning the reward function.

[0087] In the experiment, the performance of 3 tasks on 5 baselines is compared in the above two different environments. During data collection, each baseline collects 100k data for training per episode, and a total of 5 episodes are performed. After each episode, the performance of the model is evaluated to promote the learning of specific behaviors.

[0088] The experimental results are analyzed as follows:

[0089] To verify the effectiveness of the method, the performance of the RL agent in the robot arm task is evaluated from two aspects: average return and data efficiency. The average return represents the long-term income of the strategy, while the data efficiency represents the efficiency of completing the task, i.e., measures the amount of data required to complete the task. In addition, to further compare the performance of the method and the baseline, the Figure 8In this case, the results of all tasks in the DM Control Suite (Bring ball, Bring peg) were combined, and the interquartile mean (IQM) of the normalized scores was calculated. Table 1 shows the average return for completing different baseline-related tasks in different environments. Table 2 shows the numerical efficiency for completing different baseline tasks in the Bring ball and Franka Emika Panda robot environments. Figure 6 Performance of the reward-free policy network is demonstrated.

[0090] Table 1, Average return for three tasks in two environments relative to five baseline methods.

[0091]

[0092] Table 2, Data efficiency performance comparison of the Bring ball and Panda Reach tasks for 5 baseline training at different data volumes.

[0093]

[0094]

[0095] Below is the result analysis:

[0096] Average return and data efficiency. In this section, we evaluate the performance of our method and five baseline methods (Random, P2E, PP2E, CASCADE, and IPL) from two aspects: average return and data efficiency. As shown in Tables 1 and 2, our proposed expert-observed-state-based reward-free reinforcement learning framework significantly outperforms these baseline methods in terms of average return on all tasks and effectively completes the tasks with only 300k time steps of data. The experimental results show that the Random, P2E, and PP2E baselines do not perform well. This may be because the Random baseline lacks directionality and goal orientation, making it difficult to learn a strategy effectively in a reward-free environment. The P2E and PP2E baselines rely on environment models for self-supervised learning, but in the absence of reward signals, these models may not provide sufficient learning signals to complete the robotic arm grasping task. Although the CASCADE baseline performs well in two tasks of DMC (Bringball and Bringpeg), it does not perform well in the Reach task. This is because the reward-free framework of CASCADE lacks adaptability to different environments and tasks, and insufficient attention to specific task goals leads to low learning efficiency, resulting in unstable performance in the Reach task. In addition, although the IPL baseline learns a strategy through human preference data, avoiding the complexity of traditional reward design, and performs well on all tasks, it relies on human feedback, which may be difficult to obtain and susceptible to noise in a reward-free environment, and this implicit reward function still needs to be designed by humans. Compared with these methods, our expected state prediction network (ESP) can adapt to different types of tasks, including numerical and image environments. By predicting the dynamic changes of the environment through ESP, we can provide accurate target state information for the reward-free strategy network (RFSNet) and combine the goal-driven optimization of the strategy network to enable the agent to explicitly optimize the strategy without external reward signals.

[0097] Interquartile mean of normalized scores (IQM). IQM provides a robust and accurate measure of central tendency that is resistant to outliers. To compare the performance of our method with the baselines (Random, P2E, PP2E, CASCADE, IPL), we aggregate the results of all tasks in DMC (Bringball, Bringpeg) and calculate the IQM of the normalized scores. Figure 8The quartile mean of the normalized scores is shown. The results show that the method of significantly outperforms all baselines in terms of IQM values. The framework of accurately identifies the key states through ESP and combines the error between the states and the target state, helping the agent to more effectively utilize the data. At the same time, relying on expert state to directly learn the strategy, without the need to design a complex reward function, thus avoiding the uncertainty brought by reward design. These characteristics ensure the score stability and consistency of the method in multiple tasks, making the framework of significantly outperform other baselines in terms of IQM performance.

[0098] Performance of the reward-free strategy network. In this section, the performance of the reward-free strategy network (RFSNet) during training is demonstrated to prove the key contributions of the method. Taking the Reach task of the Franka Emika Panda robot environment as an example, as shown in Figure 6 The score changes of RFSNet at multiple training steps are tracked, and it is observed that, although there are large fluctuations in the initial stage, which may be due to the randomness of network initialization and exploration strategy, the score gradually stabilizes and rises as training progresses, indicating that RFSNet is learning an effective strategy to predict and reach the target state. This trend proves that even without external reward signals, RFSNet can optimize the strategy by reducing the difference between the predicted state and the target state. This result highlights the advantages of the method: accurate state prediction provided by the expected state prediction network (ESP), and target-driven optimization of the reward-free strategy network (RFSNet), which can effectively learn and perform tasks even in the absence of external reward signals.

[0099] Ablation study:

[0100] Ablation study of the expert observation module and the HER module. In Figure 1 the Reach task in the Franka Emika Panda robot environment, we study the framework model after removing different components. In Table 3, “-E(·)” and “-HER(·)” correspond to the framework model after removing the expert observation module and the HER module from our framework, respectively. “-E(·)-HER(·)” represents the framework model after removing both the expert observation and the HER module from our method. We compare three cases: our framework, the framework after removing the HER module, and the framework after removing both the expert observation and the HER module. From Table 3 and Figure 4The experimental results show that "E(·)" and "HER(·)" are crucial to model performance. Specifically, in 1000 training rounds, the model with "HER(·)" removed scored 11.80, 26.77, 38.04, and 8.89 points lower at 50, 100, 200, and 1000 steps, respectively, while the model with "E(·)HER(·)" removed simultaneously scored 14.62, 27.78, 38.51, and 14.47 points lower, respectively. This indicates that the involvement of expert observation significantly accelerates model training and improves the efficiency of the robotic arm in completing tasks; at the same time, the HER module also plays a key role in the model.

[0101] Sensitivity analysis of the parameters. Figure 1 In the Reach task of the Franka Emika Panda robot environment shown, we conducted parameter sensitivity analysis by setting different hyperparameters. To analyze the impact of several key hyperparameters, the RFSNet hyperparameters "α = 0.001", "β = 0.002", and "tau = 0.005" (α is the learning rate of the Actor network, β is the learning rate of the Critic network, and tau is the update coefficient of the target network) were adjusted to "α = 0.005", "β = 0.010", and "tau = 0.010", respectively. The experimental results in Table 3 show that compared with the baseline of "α = 0.005", "β = 0.010", and "tau = 0.010", these hyperparameters have a smaller impact on model performance, indicating that the model is not sensitive to hyperparameter changes.

[0102] Table 3. Sensitivity analysis of the parameters used.

[0103]

[0104] Supplementary Experiment:

[0105] Performance Analysis with Expert Actions. In this section, we demonstrate through experiments that our framework is also applicable to the use of expert actions. Specifically, to effectively integrate expert actions, we collect action data between expert states along with expert state data. During each RFSNet training iteration, we feed expert actions into RFSNet and calculate the error between the current policy network's output action and the expert actions. This error is proportionally introduced into the error network to guide the training of the policy network. Furthermore, we replace the predicted state values ​​in the original framework with the true state values ​​to further validate the accuracy of the Expected State Prediction Network (ESP). We also introduce IPL as a second-best baseline for reference. The results are as follows: Figure 9As shown, our method performs well in the framework that incorporates expert actions, indicating that our framework is also applicable to this scenario. Moreover, our method also performs well in the framework that uses real state values, reflecting that our ESP network is able to accurately predict the real new state. In contrast, the performance of the IPL baseline is slightly worse, which is related to the instability of the learning process due to its reliance on human preference feedback.

[0106] Embodiment 3 further provides a computing device suitable for the case of the expert state learning based robotic arm reward-free reinforcement learning method, comprising:

[0107] If the functions are implemented in the form of software function units and sold or used as independent products, they can be stored in a computer readable storage medium. Based on this understanding, the technical solutions of the present application or the parts of the present application that essentially contribute to the prior art or the parts of the technical solutions can be embodied in the form of software products. The computer software product is stored in a storage medium and includes a plurality of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the method described in the embodiments of the present application. The aforementioned storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a magnetic disk or an optical disk, and various media that can store program codes.

[0108] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered a list of executable instructions for implementing logic functions, and can be specifically embodied in any computer-readable medium for use by an instruction execution system, apparatus or device, such as a computer-based system, a system including a processor or other system that can fetch and execute instructions from the instruction execution system, apparatus or device, or in conjunction with these instructions. For this specification, "computer-readable medium" can be any device containing storage, communication, propagation or transmission of programs for instruction execution systems, apparatus or devices, or in conjunction with these instruction execution systems, apparatus or devices.

[0109] More specific examples (a non-exhaustive list) of the computer-readable medium include the following: an electrical connection (electronic) having one or more wires, a portable computer diskette (magnetic), a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), an optical fiber device, and a portable compact disc read-only memory (CDROM). Additionally, the computer-readable medium can also be paper or another suitable medium upon which the program is printed, as the program can be electronically captured, for example, via an optical scanner, then compiled, interpreted, or otherwise processed, and stored in a computer memory in a form that is then employable by a computer. Examples of computer-readable media include but are not limited to prime and non-transitory computer-readable media. Non-transitory computer-readable media specifically include, but are not limited to, magnetic materials, optical media, and solid-state memories. Non- transitory computer-readable media do not include carrier waves.

[0110] It should be understood that aspects of the application can be implemented in hardware, software, firmware or combinations thereof. In the embodiments described above, various steps or methods can be implemented, for example, by software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, and in another embodiment, any forms of hardware, or combinations thereof, of the following can be employed: discrete logic circuitry having logic gates for implementing logic functions upon an application of data signals; application specific integrated circuits having appropriate combinational logic gates; programmable gate arrays (PGA), field programmable gate arrays (FPGA), and so forth.

Claims

1. A method for learning of a robot arm based on expert state without reward reinforcement learning, characterized in that: The method comprises the following steps: The state s in the environment of the collecting robot arm t , the action a t , and the next state s t+1 The data gets the data set Buffer; a desired state prediction network model suitable for predicting state changes after a given action is performed in a current state is constructed; the desired state prediction network model comprises a state-action fusion network module and a prediction output network module; the state-action fusion network module comprises a state processing stage and an action injection stage; in the established desired state prediction network model, the collected Buffer data is trained to obtain trained weights; Collecting the state data observed by the expert to obtain Expertbuffer a reward-free policy network model suitable for predicting robot arm actions in a reward-free and expert action-free environment is constructed; According to the current state s t And the action a predicted by the reward-free policy network model t The predicted next state is obtained by the expected state prediction network model frozen by the network parameters And the next state The error d with the expert target state t Subsequently, the The reward-free policy network model is trained by data;​ the reward-free policy network model comprises a policy network, an error network, a target policy network and a target error network; the reward-free policy network model is trained by data, which comprises initializing parameters of the policy network, the error network, the target policy network and the target error network; For each episode, RFSNet takes the current state s t as input and outputs a predicted action a through the policy network: where θ μ represent parameters of the policy network, μ(s t |θ μ ) represents an action output by the policy network, represents a Gaussian noise; The current state s t The action a t Into the ESP network with frozen parameters, get the predicted new state at this time, where f represents that the network parameters are frozen, the resulting predicted new state with the expert target state The negative mean squared error with the expert target state is recorded as the error for the current episode with the target: wherein, is the final state of the expert to reach the goal; calculating an error E for the current state-action pair through the error network w (s t ,a t ) a predicted next action value is output by the target policy network: a t+1 = μ'(s i+1 | θ μ′ ) wherein μ' represents the target policy network, θ μ′ represent the parameters of the target policy network. obtaining an error of the state-action pair at the next time through the target error network obtaining the error at the current state, the parameters of the policy network are updated by minimizing the loss value, and the loss function during network updating is: wherein N represents the total number of steps in each cycle, and i represents the i-th step in the cycle.

2. The method of claim 1, wherein the method is based on expert state learning for a robot arm without reward reinforcement learning. The state-action fusion network module is used for extracting key features, and the prediction output network module is used for obtaining the final prediction target.

3. The method of claim 2, wherein the method further comprises: The network of the state processing stage is divided into a state embedder, a down-sampling layer and an up-sampling layer, and the process comprises the following steps: the state processing stage is processed by using several convolutional neural networks and an instance normalization layer, the state of the training data is input into the state embedder according to the obtained sampling training data in the Buffer, the original state data is converted into a low-dimensional representation, the output after the embedding is input into the down-sampling layer, the state data is mapped to a fixed-dimensional hidden representation, the output after the mapping is input into the up-sampling layer, and the mapped state representation is restored to the original dimension. The action injection stage comprises the following steps: the sampled action in the Buffer is input into an action embedder, the original action data is converted into a low-dimensional representation, the action embedding value is combined with the state representation processed by the up-sampling layer, the action embedding is multiplied with the state by using element-wise multiplication, and the original action embedding value is directly added to the state representation.

4. The method of claim 3, wherein the method further comprises: The prediction output network module comprises the following step: a full connection layer is used to convert the processed hidden state into a prediction state with the same dimension as the input state as the final result.

5. The method of claim 4, wherein: The trained weights are obtained by the following steps: the mean square error function of the prediction state obtained by the ESP and the real next state sampled in the Buffer is calculated, the ESP is learned by minimizing the loss, and the trained weights of the ESP are obtained.

6. The method of claim 5, wherein: The state data observed by the expert is collected by the following steps: the state data is collected by loading the expert model weights and interacting with the environment, or the state data is collected frame by frame from the expert video. The collected expert observed state data is stored into ExpertBuffer; The state data comprises the state data of the expert, but does not comprise the action data between the states of the expert. 7.A computer device, comprising a memory and a processor, wherein the memory stores a computer program, and the computer device is configured to perform the method according to any one of claims 1-6 when the computer program is executed by the processor. The processor executes the computer program to realize the steps of the method in any one of claims 1 to 6.

8. A computer-readable storage medium having stored thereon a computer program, characterized in that, The computer program is executed by the processor to realize the steps of the method in any one of claims 1 to 6.

Images