A long-short time double history perception robot reinforcement learning training method and system

Abstract

The application discloses a long-short time double-history perception robot reinforcement learning training method and system, comprising the following steps: acquiring continuous N frames and M frames of observations as long-time and short-time sequences; extracting long-time motion features by using a time sequence feature network, and estimating a current linear velocity from the short-time sequence by using a state estimation network; inputting the short-time observation, the estimated velocity and the long-time features into a strategy network after splicing to output actions; constructing a total loss function, which comprises a strategy loss, a value loss, a state estimation loss and a gradient penalty loss, wherein the gradient penalty loss restricts the sensitivity of the strategy to observation noise; and jointly updating all network parameters. The application significantly improves the speed tracking accuracy and action smoothness, realizes end-to-end collaborative optimization of the estimator and the strategy, reduces joint buffeting, and improves the success rate of simulation-to-reality migration.

Description

Technical Field

[0001] This invention relates to the field of deep reinforcement learning technology, specifically to a robot reinforcement learning training method and system with long and short time dual history perception. Background Technology

[0002] With the development of deep reinforcement learning in the field of robot motion control, quadruped robots, humanoid robots, and other robots are able to learn robust walking and running strategies based on simulation-to-real-world transfer methods.

[0003] Real robots can only acquire proprioceptive information (such as joint encoders and IMUs) and cannot directly obtain global states such as linear velocity and ground friction. Existing methods typically employ additional state estimators (such as velocity observers or Kalman filters) to infer missing states. However, these estimators are trained independently of the policy network or are only preprocessed, leading to amplification of estimation errors in closed-loop control. The policy struggles to compensate for these errors, ultimately resulting in low velocity tracking accuracy and susceptibility to interference. Existing reinforcement learning policies directly optimize cumulative rewards during training, lacking explicit constraints on the smoothness of the policy function itself. The trained policies are often extremely sensitive to small changes in the observed input; that is, the gradient of the action log probability with respect to the observation is too large, causing sensor noise to be amplified by the policy, resulting in high-frequency jitter in the output action sequence and frequent abrupt changes in joint torque. This not only accelerates mechanical wear but also reduces the success rate of Sim-to-Real transfers. Existing methods typically treat state estimation as an independent module, failing to allow the estimator to be jointly optimized with the policy network during training. This results in the estimator outputting accurate velocities that are not suitable for the policy's decision-making needs (e.g., the estimated noise distribution is inconsistent with the policy's expected input distribution), leading to a degraded policy performance. It should be noted that there are existing techniques in the field of Generative Adversarial Networks (GANs) that utilize gradient penalty to constrain the Lipschitz continuity of the discriminator. However, the gradient penalty in GANs applies to the gradient of the discriminator's output with respect to the input samples, aiming to ensure the smoothness of the discriminator function for stable adversarial training. In contrast, the gradient penalty in this invention applies to the gradient of the log probability of the action network with respect to the observed input, aiming to reduce the policy's sensitivity to observation noise. This is a regularization method for policy robustness in the field of robot motion control. Summary of the Invention

[0004] The purpose of this invention is to provide a robot reinforcement learning training method and system with dual-history perception in both long and short time periods, in order to solve the problems mentioned in the background art, such as inaccurate state estimation under partial observation, non-smooth action, and mismatch between estimator and policy.

[0005] To achieve the above objectives, the present invention adopts the following technical solution: According to one aspect of the present invention, a robot reinforcement learning training method with long and short time dual history perception is provided, the method comprising: Acquire N consecutive frames of observations of the robot body as a long-term observation sequence, and M consecutive frames of observations as a short-term observation sequence, where M < N; The long-term observation sequence is input into a temporal feature extraction network to extract long-term motion features; the short-term observation sequence is input into a state estimation network to estimate the missing state vector at the current moment. The short-term observation sequence, the missing state vector, and the long-term motion features are concatenated into a fusion feature, which is then input into the policy network to output the action. In reinforcement learning training, trajectory data is collected, and the advantage function and reward are calculated. Construct a total loss function, which includes policy loss, value loss, state estimation loss, and gradient penalty loss; wherein, the state estimation loss is calculated based on the error between the missing state vector and the true state in the privileged observation; and the gradient penalty loss is calculated based on the norm of the gradient of the action log probability with respect to the observed input. The parameters of the temporal feature extraction network, the state estimation network, the policy network, and the value network are jointly updated using the total loss function.

[0006] Based on the aforementioned scheme, the acquisition of N consecutive frames of observation of the robot body as a long-term observation sequence and M consecutive frames of observation as a short-term observation sequence includes: collecting single-frame observation data in each control cycle and storing it in a circular buffer with a capacity of N, which is initialized to zero; when the cumulative number of frames is less than N or M, zero vectors are used for filling; the long-term observation sequence is all N frames of data in the buffer, and the short-term observation sequence is the last M frames of data in the buffer.

[0007] Based on the aforementioned scheme, the duration of the control cycle is Δt; N satisfies a preset proportion that N×Δt at least covers one complete gait cycle of the robot; and M satisfies that M×Δt is greater than the robot actuator delay time.

[0008] Based on the aforementioned scheme, the temporal feature extraction network is a one-dimensional convolutional neural network, including two convolutional layers, one flattening layer, and two fully connected layers; the number of input channels of the two convolutional layers are N and the number of output channels of the first convolutional layer, respectively, with the number of output channels decreasing layer by layer, the kernel size being a preset range, and the stride being greater than 1; the two fully connected layers compress the flattened features to a preset long-term feature dimension.

[0009] Based on the aforementioned scheme, the state estimation network is a multilayer perceptron, the input is the flattened vector of the short-time observation sequence, the hidden layer has at least three layers, the number of neurons decreases layer by layer, and the output layer dimension is equal to the dimension of the missing state vector; the missing state vector includes the linear velocity component of the robot in the body coordinate system.

[0010] Based on the aforementioned scheme, the output action includes: the dimension of the fused feature is equal to the sum of the flattened dimension of the short-term observation sequence, the dimension of the missing state vector, and the dimension of the long-term motion feature; the policy network is at least a three-layer fully connected network that outputs the action mean; the policy network maintains a learnable standard deviation parameter, which together with the action mean forms a diagonal Gaussian distribution for action sampling; after sampling, the action is mapped to the joint target angle through a scaling factor, and then converted into torque by the PD controller.

[0011] Based on the aforementioned scheme, the reinforcement learning training employs a proximal policy optimization algorithm; the advantage function and reward are calculated using generalized advantage estimation, with a discount factor of γ and generalized advantage estimation parameters of λ, where γ∈(0,1) and λ∈(0,1); the policy loss is the PPO cutoff loss, which takes the following form: ; in, For strategic losses, The importance sampling ratio, The log probability of the action under the old strategy. Let the logarithmic probability of the action be given by the current policy. To truncate the threshold, This is the dominant function.

[0012] Based on the aforementioned scheme, the formula for calculating the state estimation loss is as follows: ; in The loss is calculated for the state, where B is the number of samples in the mini-batch. The missing state vector output by the state estimation network. This is the true state vector extracted from privileged observations; The formula for calculating the gradient penalty loss is as follows: ; in, For gradient penalty loss, For observation input, For the corresponding action.

[0013] Based on the aforementioned scheme, the calculation of the gradient penalty loss includes: cloning the current mini-batch of observation tensors and enabling gradient tracking; inputting the cloned observation tensors into the current policy network and recalculating the log probability of actions; calling the automatic differentiation function to calculate the gradient of the sum of log probabilities of actions with respect to the cloned observation tensors, and setting the parameter of preserving the computation graph to true; calculating the square of the L2 norm for the gradient vector of each sample, and then taking the average of all samples.

[0014] According to another aspect of the present invention, a robot reinforcement learning training system with long and short time dual history perception is provided, the system comprising: The data acquisition module is used to acquire N consecutive frames of observations of the robot body as a long-term observation sequence and M consecutive frames of observations as a short-term observation sequence, where M < N; A temporal feature extraction network is used to receive the long-term observation sequence and extract long-term motion features; A state estimation network is used to receive the short-time observation sequence and estimate the missing state vector at the current time. A policy network is used to map the fused features obtained by concatenating the short-term observation sequence, the missing state vector, and the long-term motion features into actions; The motion mapping and execution module is used to map the motion to a target joint angle after scaling, and send it to the PD controller to generate joint torque; Value networks are used to estimate the value of a state. The training data collection module is used to collect trajectory data during reinforcement learning training and to calculate the advantage function and reward. The loss calculation module is used to construct the total loss function, which includes policy loss, value loss, state estimation loss and gradient penalty loss; wherein, the state estimation loss is calculated based on the error between the missing state vector and the true state in the privileged observation, and the gradient penalty loss is calculated based on the norm of the gradient of the action log probability with respect to the observed input; The joint optimization module is used to jointly update the parameters of the temporal feature extraction network, the state estimation network, the policy network, and the value network using the total loss function.

[0015] As can be seen from the above technical solutions, this invention has at least the following advantages and positive effects compared with the prior art: By constructing dual historical observation sequences of both long and short time periods, and utilizing CNN to extract long-term gait patterns and MLP to estimate linear velocity from short-term sequences, the policy network simultaneously obtains historical motion styles and current instantaneous velocity, effectively alleviating the problem of missing velocity information. The introduction of gradient penalty loss directly constrains the sensitivity of the action log probability to the observed input, making the policy insensitive to sensor noise; the trained policy exhibits smoother joint torque changes during deployment, significantly improving motion smoothness and extending robot lifespan. By jointly backpropagating the state estimation loss, policy loss, and value loss, the mismatch problem between the estimator and policy in traditional methods is eliminated, allowing the policy to better utilize estimation information and improving overall robustness. The use of a large-scale parallel environment and the PPO algorithm, combined with GAE advantage estimation, significantly improves sample efficiency.

[0016] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit the invention. Attached Figure Description

[0017] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the invention and, together with the description, serve to explain the principles of the invention. It is obvious that the drawings described below are merely some embodiments of the invention, and those skilled in the art can obtain other drawings based on these drawings without any inventive effort. In the drawings: Figure 1 This is a schematic diagram of a robot reinforcement learning training method with dual long and short time history perception according to the present invention; Figure 2 This is a flowchart illustrating a robot reinforcement learning training method with dual long and short-term historical perception according to the present invention. Detailed Implementation

[0018] To more clearly illustrate the purpose, technical solutions, and advantages of the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. The exemplary embodiments can be implemented in many forms and should not be construed as limited to the examples set forth herein. On the contrary, these embodiments are provided so that the present invention will be more comprehensive and complete, and fully convey the concept of the exemplary embodiments to those skilled in the art.

[0019] Furthermore, the described features, structures, or characteristics can be combined in any suitable manner in one or more embodiments. Numerous specific details are provided in the following description to give a full understanding of embodiments of the invention. However, those skilled in the art will recognize that the technical solutions of the invention can be practiced without one or more of the specific details, or other methods, components, apparatuses, steps, etc., can be employed. In other instances, well-known methods, apparatuses, implementations, or operations are not shown or described in detail to avoid obscuring various aspects of the invention.

[0020] The block diagrams shown in the accompanying drawings are merely functional entities and do not necessarily correspond to physically independent entities. That is, these functional entities can be implemented in software, in one or more hardware modules or integrated circuits, or in different network and / or processor devices and / or microcontroller devices.

[0021] The flowcharts shown in the accompanying drawings are merely illustrative and do not necessarily include all content and operations / steps, nor do they necessarily have to be performed in the described order. For example, some operations / steps can be broken down, while others can be combined or partially combined; therefore, the actual execution order may change depending on the specific circumstances.

[0022] The present invention will now be described in detail with reference to specific embodiments.

[0023] Example 1

[0024] like Figure 1 , 2 As shown, this embodiment provides a robot reinforcement learning training method with dual history perception in both long and short time periods. The specific steps of this method are as follows: S1: Obtain N consecutive frames of observations of the robot body as a long-term observation sequence, and M consecutive frames of observations as a short-term observation sequence, where M < N.

[0025] Within each control cycle, a set of raw sensor data is collected from the robot's sensor system to form a single-frame observation vector. This vector has a fixed dimension D and includes: joint position, joint velocity, output action from the previous moment, base angular velocity, base Euler angles, and external command signals. Each data item is linearly scaled and clipped, then mapped to a preset numerical range to form a standardized single-frame observation vector. The duration Δt of the control cycle is consistent with the robot's underlying control frequency and sensor sampling rate; in this embodiment, Δt = 0.01 seconds, corresponding to a 100Hz control frequency. At the end of each control cycle, the current frame observation vector is pushed into a fixed-length history queue.

[0026] Furthermore, a circular buffer is maintained as a history queue, with its capacity preset to the number of long-term window frames N. The circular buffer is initialized to a zero vector to represent the robot's static, zero-position state before startup. A write pointer is maintained in the buffer; each time a new observation is acquired, the oldest data at the pointer's location is overwritten, and the pointer is incremented cyclically. The circular buffer always stores the most recent N frames of valid observation data; if there are fewer than N frames, it is automatically filled with a zero vector without requiring additional branching logic.

[0027] When constructing neural network input in each control cycle, all N frames of data are read from the circular buffer and arranged in chronological order to form a long-term observation sequence with an N×D data shape. The last M frames of data are extracted from this long-term sequence to form a short-term observation sequence with an M×D data shape. Here, M is the preset number of short-term window frames, satisfying M < N. The number of long-term window frames N is determined based on the robot's gait cycle. The long-term window duration N×Δt covers at least a preset proportion of a complete gait cycle of the robot, preferably more than 50%, to ensure that the long-term observation sequence contains sufficient gait phase information, enabling the temporal feature extraction network to effectively capture periodic motion patterns. The number of short-term window frames M is determined based on the robot actuator delay time. The short-term window duration M×Δt is greater than the robot actuator delay time. The actuator delay includes communication delay, PD adjustment time, and physical lags such as friction. The short-term window duration M×Δt must be greater than this delay so that the state estimation network can infer the missing state at the current moment from historical trends.

[0028] In the initial stage after system startup or environment reset, the number of actual frames accumulated in the circular buffer may be less than N or M. At this time, a zero-padding strategy is automatically adopted. For the insufficient number of frames, zero vectors are added to the front of the time series to ensure that the long-term and short-term observation sequences always maintain fixed dimensions of N×D and M×D, respectively. The physical meaning of zero-padding is that it assumes the robot is initially stationary and has no motion tendency. This processing method ensures the constancy of the neural network input dimension and does not affect subsequent training convergence.

[0029] S2: Input the long-term observation sequence into the temporal feature extraction network to extract long-term motion features; input the short-term observation sequence into the state estimation network to estimate the missing state vector at the current moment.

[0030] After acquiring multi-scale historical observations, different feature extraction networks are used for long-term and short-term observation sequences to obtain complementary motion representations. Specifically, this includes: using a temporal feature extraction network to perform spatiotemporal compression and pattern encoding on the long-term observation sequence to obtain long-term motion features; and using a state estimation network to perform nonlinear mapping on the short-term observation sequence to obtain the missing state vector at the current moment.

[0031] S2.1: Temporal feature extraction of long-term observation sequences.

[0032] In this embodiment, the temporal feature extraction network adopts a one-dimensional convolutional neural network structure to extract motion pattern features from long-term observation sequences. The network includes: two convolutional layers, a flattening layer, and two fully connected layers. The first convolutional layer has an input channel number equal to the number of frames N in the long-term window, an output channel number of C1, a convolutional kernel size of K1, and a stride of S1. Among them, both the convolutional kernel size K1 and the stride S1 are preset positive integer values, and the stride S1 is greater than 1 to achieve downsampling in the time dimension. The second convolutional layer has an input channel number of C1 and an output channel number of C2 (C2 < C1), a convolutional kernel size of K2, and a stride of S2 (S2 > 1). Nonlinear activation functions are connected after both convolutional layers. By gradually decreasing the output channel number layer by layer, the network gradually compresses the feature dimension and extracts abstract motion patterns. The flattening layer flattens the multi-dimensional feature map output by the second convolutional layer into a one-dimensional vector. The first fully connected layer maps the flattened vector to a preset hidden layer dimension H1, and the second fully connected layer further compresses the H1-dimensional features to a preset long-term feature dimension. Nonlinear activation functions are connected after both fully connected layers.

[0033] Specifically, let the dimension of the long-term observation sequence obtained in step S1 be N×D, where N is the number of frames in the long-term window and D is the single-frame observation dimension. In this embodiment, N = 66 and D = 47, that is, the long-term observation sequence contains 66 frames, and each frame has 47-dimensional data. To adapt to the input format of the one-dimensional convolutional neural network (Conv1d), the sequence is reshaped into a three-dimensional tensor with a shape of (B, N, D), where B is the batch size (in the training stage, it is the number of parallel environments, such as 4096; in the inference stage, it is 1). In this tensor, the channel dimension corresponds to the number of time frames N = 66, and the sequence length corresponds to the single-frame feature dimension D = 47, so that the convolutional operation slides along the time frame dimension to extract local motion patterns at different time offsets.

[0034] The reshaped tensor is sequentially input into at least two one-dimensional convolutional layers, and a nonlinear activation function (such as ReLU) is connected after each convolutional layer. In this embodiment, two convolutional layers are adopted. The input channel number of the first convolutional layer is 66, the output channel number is 32, the convolutional kernel size is 6, and the stride is 3. The output tensor shape of this layer is , where ; The first convolutional layer is used to capture the local correlation of joint trajectories in a short time, corresponding to a time span of about 0.06 seconds. The input channel number of the second convolutional layer is 32, the output channel number is 16, the convolutional kernel size is 4, and the stride is 2. The output tensor shape of this layer is , where This layer further extracts higher-order motion patterns in the compressed time dimension, corresponding to a time span of approximately 0.04 seconds. Batch normalization and dropout regularization are not used in any of the convolutional layers to maintain the original scale and determinism of the temporal features. The activation function can also be LeakyReLU or ELU; ReLU is preferred in this embodiment.

[0035] The feature map of shape (B, 16, 7) output from the second convolutional layer is flattened into a one-dimensional vector with dimensions 16 × 7 = 112. This vector is then sequentially input into two fully connected layers, each followed by an ELU activation function. The first fully connected layer has an input of 112 dimensions and an output of 128 dimensions; the second fully connected layer has an input of 128 dimensions and an output of 64 dimensions. The final output is a long-term motion feature vector of length 64 dimensions. This vector represents the robot's gait phase, periodic motion pattern, and global dynamic characteristics over the past 0.66 seconds. Its dimension is much smaller than the original input dimension of 66 × 47 = 3102, achieving effective compression.

[0036] In this embodiment, the number of frames in the long-term window N=66 (corresponding to 0.66 seconds) covers about 2 / 3 of the gait cycle of the humanoid robot; the convolution kernel sizes 6 and 4 correspond to 0.06 seconds and 0.04 seconds respectively, which can capture the mid-frequency components of joint motion (about 16~25Hz); the output feature dimension 64 is the empirically optimal value, which achieves a good balance between retaining sufficient motion information and reducing overfitting.

[0037] S2.2: State estimation of short-time observation sequences.

[0038] From the short-time observation sequence obtained in step S1, a flattened one-dimensional vector is extracted. In this embodiment, the number of short-time window frames M=5, the single-frame observation dimension D=47, and the flattened vector dimension is 5×47=235; this vector contains the joint positions (12 active joints), joint velocities (12), the previous action (12), and the IMU angular velocity (12) within the last 5 control cycles (corresponding to 0.05 seconds). , , The base Euler angles (roll, pitch, yaw), gait phase signals (sin, cos), and desired velocity commands (3D) can reflect the robot's instantaneous motion trend.

[0039] The above 235-dimensional vector is input into a multilayer perceptron network, the structure of which is as follows: First hidden layer: fully connected, with 235-dimensional input and 256-dimensional output, followed by the ELU activation function; The second hidden layer is a fully connected layer with 256-dimensional input and 128-dimensional output, followed by an ELU activation function. The third hidden layer is a fully connected layer with 128-dimensional input and 64-dimensional output, followed by an ELU activation function. Output layer: Fully connected, 64-dimensional input, 3-dimensional output, no activation function.

[0040] The 3D vector of the output layer corresponds to the estimated three-axis velocities of the robot in the body coordinate system. , , The unit is meters per second. In other implementations, the output layer dimension can be extended to more unobservable states as needed, such as the vertical height of the base (1-dimensional), the coefficient of friction of the ground (1-dimensional), or the plantar contact force (6-dimensional).

[0041] The velocity vector output by the state estimation network has the same dimensions and coordinate system orientation as the robot's actual physical velocity: the x-axis is forward, the y-axis is to the left (or right, depending on the robot's definition), and the z-axis is upward. This output provides crucial instantaneous motion state information for subsequent policy decisions, effectively alleviating the problem of missing velocity information in some observable environments.

[0042] Before training begins, the parameters of each network are initialized. In this embodiment, the initialization strategy for each network includes: for the temporal feature extraction network (CNN encoder), convolutional layers are initialized using Kaiming uniform initialization, with biases set to 0, and fully connected layers are initialized using Kaiming uniform initialization; for the state estimation network, hidden layers are initialized using Xavier uniform initialization, and output layer weights and biases are all set to 0 (making the initial estimation speed 0); for the policy network (Actor) and value network (Critic), all fully connected layers are initialized using orthogonal initialization, with biases set to 0; the learnable standard deviation parameter σ is initialized as an all-1 vector. This initialization strategy collectively ensures that the action output is close to zero mean and the state estimation output is close to zero speed in the early stages of training, which is beneficial to the stability of the initial exploration.

[0043] It should be noted that the missing state vector refers to state variables that cannot be directly measured by sensors in partially observable environments, but are crucial for motion control. In this embodiment, the missing state vector includes at least the linear velocity component of the robot in the body coordinate system. The missing state vector is inferred from a short-time observation sequence by a state estimation network and input into a policy network for decision-making. In other embodiments, the missing state vector may further include unobservable states such as base vertical height, ground friction estimation, and foot contact force.

[0044] S3: Concatenate the short-term observation sequence, the missing state vector, and the long-term motion features into a fusion feature, input it into the policy network, and output the action.

[0045] After completing dual-path feature extraction, feature information from different sources and at different time granularities is fused, and robot joint movements are generated based on the fused features through a policy network.

[0046] Step S2 generates three types of feature data, including: 1) short-term observation features, which are the flattened vectors of the short-term observation sequences extracted in step S1, denoted as... In this embodiment, the short-time window frame count M=5, and the single-frame observation dimension D=47, therefore It is a 235-dimensional vector; this feature contains the joint position, joint velocity, previous action, angular velocity, Euler angle, and command signal within the last 5 control cycles (0.05 seconds), possessing high-frequency transient information. 2) Estimate the state vector, output by the state estimation network in step S2, denoted as... In this embodiment, a 3D vector is used. , , ), representing the estimated current linear velocity of the robot in the body coordinate system; this feature represents the motion state at the current moment. 3) Long-term motion features, output by the temporal feature extraction network (CNN encoder) in step S2, denoted as In this embodiment, it is a 64-dimensional vector, representing low-frequency global information such as gait phase and periodic motion patterns over the past 66 frames (0.66 seconds).

[0047] Estimating the state vector Although calculated from short-time history, its physical meaning is an estimate of the linear velocity at the current moment, therefore it is considered data contemporaneous with the latest frame in the short-time observations. Long-term motion characteristics. After convolution and pooling operations, the data no longer has a clear temporal index, but represents the global statistical pattern of the entire long-term window, which can be regarded as a global descriptor aligned with the current time step. The above three types of features are directly concatenated along the feature dimension to form a fused feature vector. : ; In this embodiment, the total dimension of the fused vector is 235+3+64=302; the concatenation operation is performed in parallel on the batch data, and the output shape is (B,302), where B is the batch size (the number of parallel environments during training, such as 4096; and 1 during inference).

[0048] The policy network (Actor) employs a multi-layer fully connected structure to map fused features to parameters of an action probability distribution. In this embodiment, the specific structure of the policy network is as follows: the input layer receives the fused feature vector, with a dimension of 302; the first hidden layer is fully connected, with an input of 302 dimensions and an output of 512 dimensions, followed by an ELU activation function; the second hidden layer is fully connected, with an input of 512 dimensions and an output of 256 dimensions, followed by an ELU activation function; the third hidden layer is fully connected, with an input of 256 dimensions and an output of 128 dimensions, followed by an ELU activation function; the output layer is fully connected, with an input of 128 dimensions and an output of... Dimensionless, with no activation function. Let be the dimension of the robot's motion space; for a humanoid robot with 12 active joints, this embodiment uses 12. The fused feature vectors will then be... The action mean vector is obtained by passing the above network layers sequentially. : ; The policy network maintains a learnable standard deviation parameter. This parameter is independent of the input, initially set to 1.0, and is optimized together with other network parameters during training. and We can jointly define a diagonal Gaussian distribution. This serves as the action sampling distribution. The learnable standard deviation parameter... As an independent parameter of the policy network (e.g., nn.Parameter in the PyTorch framework), it shares the same optimizer with other parameters of the temporal feature extraction network, state estimation network, policy network, and value network; its gradient is passed through the policy loss. and The entropy term is calculated indirectly, where the policy loss contributes to the gradient by influencing the logarithmic gradient of the action sampling probability, and the entropy term directly depends on the standard deviation. The optimizer automatically handles the gradient accumulation and update of this parameter, without requiring separate optimizer or update logic settings.

[0049] During the training phase, to encourage exploration, action vectors are sampled from the aforementioned Gaussian distribution. :

[0050] After sampling, each element of the action vector is clipped and restricted to a preset range. Inside; in this embodiment, However, due to the existence of subsequent action scaling factors, the actual effective range is much smaller than this value. The pruning operation is used to prevent numerical instability caused by extreme action values. During the deployment (inference) phase, sampling is no longer performed; instead, the action mean is used directly. As a deterministic output, randomness is eliminated, ensuring the repeatability of control.

[0051] Action vectors Each component corresponds to the angular offset (in radians) of one active joint of the robot. In this embodiment, the 12-dimensional motion is sequentially mapped to the left leg's hip pitch, hip roll, hip yaw, knee pitch, ankle pitch, and ankle roll, as well as the six joints corresponding to the right leg. To prevent the motion from exceeding the physical limits of the joints, the motion is linearly scaled before being sent to the underlying controller.

[0052] in, The default angles for each joint (e.g., joint angles in a standing posture). As the motion scaling factor, this embodiment takes... This scaling factor limits the actual joint angle offset to a reasonable range (e.g., within ±0.5 rad), much smaller than the physical limits of each joint (typically ±1.5~2.0 rad), ensuring motion safety. Ultimately, the target joint angle is... The data is sent to the robot's underlying PD controller, where it calculates torque based on joint speed feedback to drive the joint motors. The PD controller's proportional gain and derivative gain are preset according to the robot model, for example... , .

[0053] S4: Collect trajectory data during reinforcement learning training and calculate the advantage function and reward.

[0054] After completing action generation and environmental interaction, interaction trajectory data is collected, and the advantage function and reward are calculated based on Generalized Advantage Estimation (GAE), providing a data foundation for joint loss calculation and parameter update in step S5. A Proximal Policy Optimization (PPO) algorithm framework is adopted to improve sample efficiency through large-scale parallel environmental sampling.

[0055] This embodiment uses multiple parallel simulation environments (e.g., NVIDIA Isaac Gym), each independently simulating a robot instance. The number of parallel environments is denoted as num_envs, which is 4096 in this embodiment; this value allows for the acquisition of a large number of diverse trajectories in a single acquisition, reducing sample correlation. Each environment executes multiple control steps consecutively within a sampling phase, denoted as num_steps_per_env, which is 24 in this embodiment. That is, each environment acquires trajectory data for 24 time steps each time, resulting in a total sample size obtained in a single acquisition: total_samples=num_envs×num_steps_per_env=4096×24=98304 The data collection process is executed synchronously across multiple environments. All environments share the same policy network parameters, but each maintains its own independent state history queue and random seed.

[0056] At each control step (i.e., one time step per environment), after performing the following operations, the relevant data is stored in a buffer, and actions are obtained from the current policy network (sampled or deterministically output as in step S3); the actions are then executed in the simulation environment, and the reward value is obtained according to the predefined reward function. And the next moment's observation; record the following data items: observation (Long-term observation sequence constructed in step S1), privileged observations (Complete state information (such as actual velocity, contact force, etc.) is only available during training), action (Action vector obtained from step S3), reward (The scalar reward value returned by the environment, calculated according to a predetermined reward function), value estimation (Current Critic network's estimation of state value), log probability of actions. (Log likelihood of actions under the current policy), mean of actions (Mean vector of policy network output), action standard deviation (Learnable standard deviation parameter). All data is stored in order of environment index and time step, forming a tensor or flattened storage of shape (num_envs, num_steps_per_env, feature_dim).

[0057] Among them, reward value The reward is calculated based on a predefined dense reward function, which includes multiple reward terms corresponding to different optimization objectives of robot motion control. The immediate reward for each step is obtained by weighted summation of these terms. ; in, Let k be the weight of the k-th reward item. The corresponding reward value (positive value represents a reward, negative value represents a penalty). In this embodiment, the reward items and their weights are shown in Table 1:

[0058] The aforementioned weight values ​​were determined through experimental optimization and can be adjusted within a reasonable range in actual implementation based on the robot model, gait frequency, and task objectives.

[0059] After completing one acquisition phase (i.e., collecting num_steps_per_env steps of data), the advantage function is calculated using the GAE (Generalized Advantage Estimation) method. and discount rewards GAE achieves a balance between bias and variance. First, the time-series difference error is calculated: ; Where γ is the discount factor, γ∈(0,1), and in this embodiment γ=0.998; For the value estimation of the next state, for the last step t=T-1, The value is 0 (or the Critic output of the final state is used). The advantage function is calculated by exponentially weighted summation: ; in, For the GAE parameters, λ∈(0,1), in this embodiment λ=0.95; T=num_steps_per_env=24; this calculation is achieved through reverse recursion, and the advantage value can be obtained from the reward and Critic output at each time step. Discounted Return Calculated by the following formula: ; Alternatively, you can directly accumulate rewards through discounts: ,in Calculated and In subsequent updates, it serves as a supervisory signal; to stabilize training, the advantage value is usually normalized (subtract the mean and divide by the standard deviation). This embodiment uses batch normalization.

[0060] To support multiple updates (multiple epochs and mini-batches) in step S5, all collected data is organized into a format that facilitates random sampling. Data from each environment and time step is flattened into a large one-dimensional array with a length of total_samples = num_envs × num_steps_per_env. Each sample includes an observation vector, privileged observation vector, action vector, log probability of the old action, dominance value, reward value, mean of the old action, and standard deviation. In subsequent update phases, the data is randomly shuffled and divided into multiple mini-batches, each with a size of batch_size = total_samples / num_mini_batches. In this embodiment, num_mini_batches = 4, therefore each mini-batch contains 24576 samples.

[0061] Because the Critic network has not yet converged in the early stages of training, value estimation... The estimation may be inaccurate, leading to a large variance in the advantage estimate. Therefore, during the data collection phase, the current best Critic network is used (synchronized after each update), and larger γ and λ are employed to utilize long-term information. As training progresses, the accuracy of the value estimate improves, and the advantage estimate of GAE tends to stabilize. For terminating states (such as when the robot falls), the environment is reset early. In GAE calculations, the value estimate after a terminating state is set to 0, and backtracking is no longer continued.

[0062] In other implementations, the number of parallel environments (e.g., 2048, 8192) or the number of steps per environment (e.g., 16, 32) can be adjusted to accommodate different computing resources and task complexity; other advantage estimation methods (e.g., directly using temporal difference error) can also be used instead of GAE.

[0063] S5: Construct a total loss function, which includes policy loss, value loss, state estimation loss, and gradient penalty loss; wherein, the state estimation loss is calculated based on the error between the missing state vector and the true state in the privileged observation; and the gradient penalty loss is calculated based on the norm of the gradient of the action log probability with respect to the observed input.

[0064] A total loss function is constructed, which includes policy loss, value loss, state estimation loss, and gradient penalty loss. All trainable parameters of the temporal feature extraction network, state estimation network, policy network, and value network are updated jointly by backpropagation and the optimizer.

[0065] Define the total loss function for: ; in, The loss is calculated using the PPO truncation strategy (with a fixed coefficient of 1.0). For value loss (ClippedMSE); The entropy of the strategy; This is the mean square error loss for state estimation; The loss is a gradient penalty. , , For the corresponding hyperparameters. In this embodiment, the hyperparameter values ​​are: , , This value is determined through ablation experiments and can be adjusted according to actual conditions.

[0066] For the policy loss (PPO Clip loss), the log probabilities of actions under the old policy are read from storage. And recalculate the log probability of actions under the current policy using the current policy network. Define the importance sampling ratio: ; The loss of the PPO cutoff strategy is: ; in To truncate the threshold, The advantage function is calculated for step S4; this loss ensures that the policy update magnitude is limited to a reasonable range.

[0067] For the Clipped MSE (Value Loss), the value network estimates the value of the current state. and its target return Calculate the mean squared error. To prevent drastic changes in value estimation, a truncation mechanism is also employed: ; In this embodiment This aligns with the strategy loss.

[0068] The state estimation loss is used to supervise the state estimation network in step S2, so that the estimated velocity of the output approximates the true velocity extracted from the privileged observation.

[0069] In each mini-batch, from the privileged observation tensor Extract the true linear velocity values. Privileged observations contain the true linear velocities in the real body coordinate system provided by the simulation physics engine, stored in a fixed index range. In this embodiment, the total dimension of privileged observations is 81, and the true linear velocity occupies indices 45, 46, and 47 (corresponding to...). , , ).

[0070] The current small batch of short-term observations (from) Extracting the last M×D dimensions (i.e., 235 dimensions) from the input state estimation network yields the estimated velocity. , where B is the mini-batch size.

[0071] The state estimation loss uses mean squared error (MSE): ; in The loss is calculated for the state, where B is the number of samples in the mini-batch. The missing state vector output by the state estimation network. This is the true state vector extracted from privileged observations. In this embodiment, the three velocity components are given equal weights (all 1.0). In other embodiments, different weights can be assigned to each component (e.g., forward direction) according to task requirements. The weight is 2.0, horizontal. (Weight is 0.5). This loss term is directly added to the total loss, and its gradient is simultaneously backpropagated to the state estimation network and (indirectly affecting) the CNN encoder through observed inputs, achieving end-to-end collaborative optimization.

[0072] Gradient penalty loss mitigates the oversensitivity of the policy network to observation inputs, thereby reducing motion jitter caused by sensor noise and improving motion smoothness and Sim-to-Real transfer success rate during deployment. The calculation of gradient penalty loss includes: cloning the current mini-batch of observation tensors and enabling gradient tracking; inputting the cloned observation tensors into the current policy network and recalculating the log probabilities of actions; calling the automatic differentiation function to calculate the gradient of the sum of log probabilities of actions with respect to the cloned observation tensor, and setting the graph preservation parameter to True (create_graph=True); calculating the square of the L2 norm for the gradient vector of each sample, and then averaging over all samples.

[0073] Specifically, extract the observation tensor from the current small batch. (The shape is B×obs_dim, in this embodiment obs_dim=66×47=3102), create an independent copy, that is, clone the observation tensor. And by setting `requires_grad_(True)`, the gradient to be computed for this copy is required; this operation prepares the computation graph for subsequent automatic differentiation. Input the current policy network (Actor), re-obtain the action distribution (Gaussian distribution), and then use the old actions stored in the database. Calculate the log probability under the current policy. This calculation preserves the complete computational graph for differentiation with respect to observations. It calls the automatic differentiation function to calculate the sum of the log probabilities of all samples with respect to... gradient: ; in shape and The result is the same, i.e., B×3102. Setting `create_graph=True` preserves the computation graph, ensuring the gradient tensor itself retains a complete backpropagation path. This allows the gradient penalty loss to continue propagating back to the network parameters, effectively regularizing the policy network with the penalty term. The square of the L2 norm of the gradient vector for each sample is calculated, and then averaged over all samples. ; This scalar value measures the overall sensitivity of the logarithmic probability of an action to the observed input; The larger the value, the more sensitive the strategy is to changes in the observed data. Hyperparameter (In this embodiment, 1×10) -5Add the total loss afterward: ; This penalty encourages the policy function to be as flat as possible in the observation space, thereby reducing the magnitude of the policy's response to observation noise.

[0074] In this embodiment, the gradient penalty applies to the complete observation vector (i.e., all 3102 dimensions), including all observation features in both long-term and short-term scenarios. In other implementations, it may apply only to key physical dimensions (such as joint position and angular velocity), or to the fused features (302 dimensions) of the Actor network input instead of the original observations, to reduce computational cost.

[0075] S6: Use the total loss function to jointly update the parameters of the temporal feature extraction network, the state estimation network, the policy network, and the value network.

[0076] After completing the above loss calculations, parameter updates are performed. Specifically, `total_loss.backward()` is called to calculate the gradients of all trainable parameters (including the CNN encoder, state estimation MLP, Actor network, Critic network, and learnable standard deviation). To prevent gradient explosion, global norm clipping is performed on the gradients of all parameters. ; That is, when the sum of the L2 norms of the parameter gradients exceeds 1.0, it is scaled proportionally to this threshold.

[0077] Using the Adam optimizer (initial learning rate 3×10⁻⁶) -4 , , (Without weight decay) Perform one step of parameter update. This embodiment uses adaptive KL divergence scheduling. After each update, the KL divergence between the old and new strategies is calculated. If the KL divergence exceeds twice the target value (0.01), the learning rate is divided by 1.5; if it is less than half the target value but greater than zero, the learning rate is multiplied by 1.5. The learning rate is limited to [1×10]. -5 1×10 -2 Within the range.

[0078] To ensure stable joint training, all loss terms should be of the same order of magnitude. In this embodiment, the policy loss... Typically between 0.5 and 2.0; value loss After scaling by a factor of 1.0, the value ranges from 0.5 to 2.0; the entropy term... Approximately -0.01 to -0.05; State estimation loss (Square velocity error) is approximately 0.01~0.25; gradient penalty loss coefficient 1×10 -5After scaling, it is approximately 0.001 - 0.01. Each loss is balanced with each other, and the policy loss remains dominant. State estimation and gradient penalty provide effective regularization.

[0079] The method of the present invention can be stored in a computer-readable storage medium in the form of a computer program product. When the program is executed by a processor, all steps of the above method can be implemented.

[0080] Embodiment 2 This embodiment exemplarily presents a robot reinforcement learning training system with long-short term dual history perception, including a data acquisition module, a temporal feature extraction network, a state estimation network, a policy network, a value network, a training data collection module, a loss calculation module, and a joint optimization module.

[0081] The data acquisition module is used to obtain continuous N frames of observations of the robot body as a long-term observation sequence, and continuous M frames of observations as a short-term observation sequence, where M < N. Specifically, the data acquisition module includes a circular buffer with a buffer capacity of N (the number of frames in the long-term window); in each control cycle, raw data is read from sensors such as robot joint encoders and IMUs, and after scaling normalization and clipping, a single-frame observation vector (dimension D) is formed and written to the current pointer position of the circular buffer, and the pointer increments cyclically. When the number of accumulated frames in the buffer is less than N or M, zero vectors are automatically used for filling. In each control cycle, two data sequences are output, the long-term observation sequence: all N frames of data in the buffer, with a shape of N×D; the short-term observation sequence, the last M frames of data in the buffer, with a shape of M×D; the output of this module is connected to the temporal feature extraction network and the state estimation network.

[0082] The temporal feature extraction network is used to receive the long-term observation sequence and extract long-term motion features. The temporal feature extraction network uses a one-dimensional convolutional neural network (Conv1d), including: a first convolutional layer, a second convolutional layer, a flattening layer, a first fully connected layer, and a second fully connected layer connected in sequence. The first convolutional layer has an input channel number equal to N, an output channel number of C1, a convolutional kernel size of K1, and a stride of S1 (S1 > 1), followed by a non-linear activation function (such as ReLU); the second convolutional layer has an input channel number equal to C1, an output channel number of C2 (C2 < C1), a convolutional kernel size of K2, and a stride of S2 (S2 > 1), followed by a non-linear activation function; the flattening layer flattens the multi-dimensional feature map output by the second convolutional layer into a one-dimensional vector; the first fully connected layer maps the flattened vector to a hidden layer dimension H1, followed by an activation function (such as ELU); the second fully connected layer further compresses the H1-dimensional feature to a preset long-term feature dimension F_long and outputs a long-term motion feature vector. The output (long-term motion feature) of the temporal feature extraction network is connected to the policy network.

[0083] A state estimation network is used to receive a short-time observation sequence and estimate the missing state vector at the current time step. The state estimation network employs a multilayer perceptron (MLP), and its structure includes an input layer, at least three hidden layers, and an output layer. The input layer receives the flattened vector (dimension M×D) of the short-time observation sequence; the number of neurons in the hidden layers decreases layer by layer, and each layer is followed by an ELU activation function; the number of neurons in the output layer is equal to the dimension of the missing state vector, and there is no activation function. The missing state vector includes at least the robot's linear velocity component in its body coordinate system (…). ,, , The output of the state estimation network (the estimated missing state vector) is connected to the policy network.

[0084] The policy network is used to map the fused features (concatenated from short-term observation sequences, missing state vectors, and long-term motion features) into actions. Specifically, the policy network is a fully connected network with at least three layers. Each hidden layer is followed by an ELU activation function, and the number of neurons in the output layer is equal to the dimension of the robot's action space, outputting the mean action. The policy network also maintains a learnable standard deviation parameter, which, together with the mean action, forms a diagonal Gaussian distribution for action sampling during the training phase. During the deployment phase, the network directly outputs the mean action as a deterministic control command. The input to the policy network is a fused feature vector, which is formed by concatenating the short-term observation sequences (flattened) from the data acquisition module, the output of the state estimation network, and the output of the temporal feature extraction network along the feature dimension. The action mapping and execution module scales the output action of the policy network and maps it into joint target angles, sending them to the robot's underlying PD controller to generate joint torques.

[0085] The value network is used to estimate the value of a state. It employs a multi-layer fully connected structure, taking a privileged observation vector as input (containing state information such as true velocity, contact force, and terrain information, which is only available during training), and outputting a scalar value, i.e., the value estimate of the current state. The value network and policy network are trained independently and do not share parameters, but are updated simultaneously in the joint optimization module.

[0086] The training data collection module is used to collect trajectory data during reinforcement learning training and to calculate the advantage function and reward. The module runs in multiple parallel simulation environments, each independently executing the actions output by the policy network, acquiring rewards, and obtaining the next observation. At each time step, the module stores data including: observations, privileged observations, actions, rewards, value estimates, action log probabilities, action mean, and standard deviation. After completing a preset number of steps (e.g., 24 steps), the module uses generalized advantage estimation (GAE) to calculate the advantage function. and discount rewards All data is organized into a small-batch format that facilitates random sampling.

[0087] The loss calculation module is used to construct the total loss function, which includes policy loss, value loss, state estimation loss, and gradient penalty loss. The state estimation loss is calculated based on the error between the missing state vector and the true state in the privileged observation, and the gradient penalty loss is calculated based on the norm of the gradient of the action log probability with respect to the observed input.

[0088] The strategy loss uses PPO cutoff loss, calculated using the importance sampling ratio. The module includes a cutoff threshold ε, a value loss calculated using clipped mean squared error (MSE) to prevent drastic changes in value estimation, a state estimation loss calculated as the mean squared error between the estimated velocity output by the state estimation network and the true velocity extracted from privileged observations, and a gradient penalty loss calculated as the mean of the squared L2 norm of the gradient of the log probability of the action with respect to the observed input, to suppress the policy's sensitivity to observation noise. The module also includes an entropy regularization term and weighting coefficients for each loss term. The final output is the total loss.

[0089] The joint optimization module is used to jointly update the parameters of the temporal feature extraction network, state estimation network, policy network, and value network using the total loss function. This module includes an optimizer (such as Adam) and a gradient pruning unit. In each training iteration, the module receives the total loss output from the loss calculation module, calls backpropagation to calculate the gradients of all trainable parameters, performs global gradient pruning (with a maximum norm preset), and then calls the optimizer to update the parameters. The module can also implement adaptive learning rate scheduling (such as adjustment based on KL divergence) to stabilize the training process.

[0090] In this embodiment, the system can be deployed in a simulation environment for policy training. After training, the temporal feature extraction network, state estimation network, and policy network can be exported as a lightweight inference model and deployed to a real robot controller for operation.

[0091] Other embodiments of the invention will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of the invention that follow the general principles of the invention and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of the invention are indicated by the claims. It should be understood that the invention is not limited to the precise structures described above and shown in the drawings, and various modifications and changes can be made without departing from its scope. The scope of the invention is limited only by the appended claims.

Claims

1. A robot reinforcement learning training method with dual long and short-term history awareness, characterized in that, The method includes: Acquire N consecutive frames of observations of the robot body as a long-term observation sequence, and M consecutive frames of observations as a short-term observation sequence, where M < N; The long-term observation sequence is input into a temporal feature extraction network to extract long-term motion features; the short-term observation sequence is input into a state estimation network to estimate the missing state vector at the current moment. The short-term observation sequence, the missing state vector, and the long-term motion features are concatenated into a fusion feature, which is then input into the policy network to output the action. In reinforcement learning training, trajectory data is collected, and the advantage function and reward are calculated. Construct a total loss function, which includes policy loss, value loss, state estimation loss, and gradient penalty loss; wherein, the state estimation loss is calculated based on the error between the missing state vector and the true state in the privileged observation; and the gradient penalty loss is calculated based on the norm of the gradient of the action log probability with respect to the observed input. The parameters of the temporal feature extraction network, the state estimation network, the policy network, and the value network are jointly updated using the total loss function.

2. The robot reinforcement learning training method with dual long and short-term history perception according to claim 1, characterized in that, The acquisition of N consecutive frames of observation of the robot body as a long-term observation sequence and M consecutive frames of observation as a short-term observation sequence includes: collecting single-frame observation data in each control cycle and storing it in a circular buffer with a capacity of N, which is initialized to zero; when the cumulative number of frames is less than N or M, zero vectors are used for filling; the long-term observation sequence is all N frames of data in the buffer, and the short-term observation sequence is the last M frames of data in the buffer.

3. The robot reinforcement learning training method with dual long and short-term history perception according to claim 2, characterized in that, The duration of the control cycle is Δt; N satisfies a preset ratio that N×Δt at least covers one complete gait cycle of the robot; M satisfies that M×Δt is greater than the robot actuator delay time.

4. The robot reinforcement learning training method with dual long and short-term history perception according to claim 1, characterized in that, The temporal feature extraction network is a one-dimensional convolutional neural network, including two convolutional layers, one flattening layer, and two fully connected layers. The number of input channels of the two convolutional layers are N and the number of output channels of the first convolutional layer, respectively. The number of output channels decreases layer by layer. The kernel size is within a preset range, and the stride is greater than 1. The two fully connected layers compress the flattened features to a preset long-term feature dimension.

5. The robot reinforcement learning training method with dual long and short-term history perception according to claim 1, characterized in that, The state estimation network is a multilayer perceptron, with the input being the flattened vector of the short-time observation sequence. It has at least three hidden layers, with the number of neurons decreasing layer by layer. The output layer dimension is equal to the dimension of the missing state vector. The missing state vector includes the linear velocity component of the robot in the body coordinate system.

6. The robot reinforcement learning training method with dual long and short-term history perception according to claim 1, characterized in that, The output action includes: the dimension of the fused feature is equal to the sum of the flattened dimension of the short-term observation sequence, the dimension of the missing state vector, and the dimension of the long-term motion feature; the policy network is a fully connected network with at least three layers, outputting the action mean; the policy network maintains a learnable standard deviation parameter, which together with the action mean forms a diagonal Gaussian distribution for action sampling; after sampling, the action is mapped to the joint target angle through a scaling factor, and then converted into torque by the PD controller.

7. The robot reinforcement learning training method with dual long and short-term history perception according to claim 1, characterized in that, The reinforcement learning training employs a proximal policy optimization algorithm; the advantage function and reward are calculated using generalized advantage estimation, with a discount factor of γ and generalized advantage estimation parameters of λ, where γ∈(0,1) and λ∈(0,1); the policy loss is a PPO cutoff loss, in the form of: ； in, For strategic losses, The importance sampling ratio, The log probability of the action under the old strategy. Let the logarithmic probability of the action be given by the current policy. To truncate the threshold, This is the dominant function.

8. The robot reinforcement learning training method with dual long and short-term history perception according to claim 1, characterized in that, The formula for calculating the state estimation loss is as follows: ； in The loss is calculated for the state, where B is the number of samples in the mini-batch. The missing state vector output by the state estimation network. This is the true state vector extracted from privileged observations; The formula for calculating the gradient penalty loss is as follows: ； in, For gradient penalty loss, For observation input, For the corresponding action.

9. The robot reinforcement learning training method with long and short-term dual-history perception according to claim 1, characterized in that, The calculation of the gradient penalty loss includes: cloning the current mini-batch of observation tensors and enabling gradient tracking; inputting the cloned observation tensors into the current policy network and recalculating the log probability of actions; calling the automatic differentiation function to calculate the gradient of the sum of log probabilities of actions with respect to the cloned observation tensors, and setting the parameter of preserving the computation graph to true; calculating the square of the L2 norm for the gradient vector of each sample, and then taking the average of all samples.

10. A robot reinforcement learning training system with dual long and short-term history awareness, used to implement the method as described in any one of claims 1-9, characterized in that, include: The data acquisition module is used to acquire N consecutive frames of observations of the robot body as a long-term observation sequence and M consecutive frames of observations as a short-term observation sequence, where M < N; A temporal feature extraction network is used to receive the long-term observation sequence and extract long-term motion features; A state estimation network is used to receive the short-time observation sequence and estimate the missing state vector at the current time. A policy network is used to map the fused features obtained by concatenating the short-term observation sequence, the missing state vector, and the long-term motion features into actions; The motion mapping and execution module is used to map the motion to a target joint angle after scaling, and send it to the PD controller to generate joint torque; Value networks are used to estimate the value of a state. The training data collection module is used to collect trajectory data during reinforcement learning training and to calculate the advantage function and reward. The loss calculation module is used to construct the total loss function, which includes policy loss, value loss, state estimation loss and gradient penalty loss; wherein, the state estimation loss is calculated based on the error between the missing state vector and the true state in the privileged observation, and the gradient penalty loss is calculated based on the norm of the gradient of the action log probability with respect to the observed input; The joint optimization module is used to jointly update the parameters of the temporal feature extraction network, the state estimation network, the policy network, and the value network using the total loss function.

Images