Robot teleoperation control method based on delay-free data training in delay environment

Abstract

The application discloses a kind of robot teleoperation control methods based on no delay data training under delay environment, comprising the following steps: S1: the off-line data set obtained by robot in no delay environment executes task, off-line data set includes multiple trajectories, each trajectory includes state sequence, action sequence and reward sequence;S2: the state sequence and action sequence in off-line data set are reconstructed, and the information state sequence suitable for delay environment is generated;S3: state estimation is carried out to information state sequence, and the predicted state sequence is obtained;S4: based on information state sequence and predicted state sequence, offline reinforcement learning algorithm is used to train strategy model, and strategy model is used to output the action instruction of control robot based on predicted state sequence in delay environment;S5: the trained strategy model is used to control robot to execute teleoperation task in delay environment.The application realizes the safe, efficient and reliable control of robot teleoperation in delay environment.

Description

Technical Field

[0001] This invention relates to the field of robot teleoperation control, and in particular to a robot teleoperation control method based on delay-free data training in a delayed environment. Background Technology

[0002] Reinforcement learning (RL), as an important branch of artificial intelligence, has made groundbreaking progress in recent years in areas such as robot control, game agents, large-scale language model training, and financial decision-making. Particularly in robotics, RL provides new solutions for autonomous policy learning and complex control in robots. However, most current RL learning algorithms are based on the "instantaneous interaction assumption," which assumes that robots can achieve state perception, policy reasoning, and instruction execution without latency. This differs significantly from the actual operating environment of real-world robot systems.

[0003] In robot teleoperation scenarios, signal delay is a particularly prominent issue. Teleoperated robots typically interact with operators or remote control systems via communication links, and their state perception, action commands, and feedback transmission all experience varying degrees of delay. For example: (1) In underwater robots, the low rate of underwater acoustic communication and strong signal attenuation lead to significant delays in the transmission of state information; (2) In space exploration and lunar surface operation robots, long-distance communication delays can reach several seconds or even longer. If a strategy based on the assumption of no delay is still adopted for training, it is easy to cause control oscillations or execution deviations; (3) In medical remote surgical robots, delays in video and force feedback signals can seriously affect the accuracy and safety of operation. These delays prevent robots from making accurate decisions based on the latest state, resulting in a significant decrease in the performance of reinforcement learning strategies in actual deployment, and even causing task failures or safety accidents. Therefore, research on reinforcement learning algorithms under delay conditions has important theoretical value and practical significance.

[0004] In the process of robot interaction with the environment, signal delay can be mainly divided into three categories: (1) observation delay, which is the process delay from the generation of the environmental state to the robot receiving the state; (2) inference delay, which refers to the time required for the robot to make decision calculations; and (3) action delay, which is the response delay between the strategy output and the action of the actuator. Existing research has shown that all types of delay have an equal impact on the overall performance of the robot. Therefore, this embodiment of the invention only focuses on observation delay.

[0005] Currently, reinforcement learning solutions for addressing latency issues during robot teleoperation mainly fall into two categories: online reinforcement learning (Online RL) and offline reinforcement learning (Offline RL). Online reinforcement learning methods require extensive interaction with the latency environment and typically employ: (1) latency compensation based on auxiliary information, such as introducing expert policies or parallel latency-free environment-assisted training; (2) using mechanisms such as latency prediction or state enhancement to estimate or replace the real state. However, these methods are difficult to implement in teleoperation scenarios with significant latency because online robot interaction is costly, and communication latency is random and uncontrollable. Offline reinforcement learning methods utilize fixed datasets for training, enabling policy learning under non-interactive conditions, thereby reducing costs and risks. However, most existing offline datasets are collected in latency-free environments, resulting in a mismatch with the distribution of real-time latency tasks; simultaneously, increases or changes in latency values ​​can make it difficult for fixed datasets to support policy learning, leading to a decline in robot performance in real-time latency environments.

[0006] The above background information is provided only to aid in understanding the concept and technical solution of this invention. It does not necessarily belong to the prior art of this patent application. In the absence of clear evidence that the above information was disclosed on the filing date of this patent application, the above background information should not be used to evaluate the novelty and inventiveness of this application. Summary of the Invention

[0007] To address the aforementioned technical problems, this invention proposes a robot teleoperation control method based on delay-free data training in a delayed environment. This method achieves safe, efficient, and reliable remote operation of robots in a delayed environment without incurring high costs and risks in online trial and error.

[0008] To achieve the above objectives, the present invention adopts the following technical solution:

[0009] In a first aspect, the present invention discloses a robot teleoperation control method based on delay-free data training under delayed conditions, comprising the following steps:

[0010] S1: Collect offline datasets obtained by the robot performing tasks in a latency-free environment. The offline datasets include multiple trajectories, and each trajectory includes a state sequence, an action sequence, and a reward sequence.

[0011] S2: Reconstruct the state sequence and action sequence in the offline dataset to generate an information state sequence suitable for a delayed environment;

[0012] S3: Perform state estimation on the information state sequence to obtain the predicted state sequence;

[0013] S4: Based on the information state sequence and the predicted state sequence, an offline reinforcement learning algorithm is used to train a policy model. The policy model is used to output action commands to control the robot based on the predicted state sequence in a delayed environment.

[0014] S5: In a delayed environment, the trained strategy model is used to control the robot to perform teleoperation tasks.

[0015] Preferably, step S2 specifically includes: combining the current state in the offline dataset with the action sequence from several previous moments to form an information state sequence suitable for a delayed environment.

[0016] Preferably, step S3 specifically includes: performing state estimation on the information state sequence using a direct substitution function, a Gaussian sampling function, a neural network function, or a shortened augmented state sequence function as a mapping function to obtain the predicted state sequence.

[0017] Preferably, when using a direct replacement function as the mapping function, the actual state sequence in the offline dataset is used to directly replace the information state sequence as the predicted state sequence; when using a Gaussian sampling function as the mapping function, a Gaussian distribution is constructed based on the mean and variance of the state sequences in the offline dataset, and the predicted state sequence is sampled from the Gaussian distribution; when using a neural network function as the mapping function, a neural network function is used to perform a nonlinear mapping on the information state sequence to obtain the predicted state sequence; when using a shortened augmented state sequence function as the mapping function, a shorter historical state and action sequence is used to reconstruct the information state sequence to reduce the dimensionality of the information state sequence and obtain the predicted state sequence.

[0018] Preferably, the offline reinforcement learning algorithm used in step S4 is an algorithm based on the actor-critic framework. The input of the actor network is the predicted state sequence, and the output is the robot's action command. The input of the critic network is the predicted state sequence and the robot's action command, and the output is the value assessment of the robot's action command.

[0019] Preferably, a behavior cloning regularization term is added to the objective function of the actor network to constrain the learned policy to be close to the action distribution in the offline dataset.

[0020] Preferably, step S4, which uses an offline reinforcement learning algorithm to train the strategy model, specifically includes the following interaction process between the robot and the environment in a delayed environment: in the early interaction stage, the robot remains stationary; in the later interaction stage, the robot constructs an information state sequence and outputs action commands using the most recent non-empty observation and the corresponding action sequence.

[0021] Preferably, before step S1, a delay environment construction step is included, wherein the delay environment construction step includes: modeling the robot teleoperation environment based on a delayed Markov decision process, and in the modeling process, the observation lag caused by communication and control delays is also defined.

[0022] Preferably, the step of constructing the delayed environment specifically includes: defining a delayed Markov decision process tuple, wherein the tuple includes a measurable constant delay parameter, a robot observable state space, an augmented information state space consisting of historical observations and actions, and a reward function and a state transition function defined based on the augmented information state space.

[0023] In a second aspect, the present invention discloses a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the steps of the robot teleoperation control method as described in the first aspect.

[0024] Compared with existing technologies, the beneficial effects of this invention are as follows: The robot teleoperation control method based on delay-free data training in a delayed environment disclosed in this invention first collects an offline dataset obtained by the robot performing tasks in a delay-free environment, and performs state augmentation on the offline dataset to expand the original delay-free offline dataset into a data structure suitable for delayed tasks, thereby achieving delay reconstruction at the data level; and introduces a delayed state prediction model to perform state estimation on the augmented high-dimensional information state through, for example, dimensionality reduction processing, to reduce training complexity and improve policy generalization performance; by combining the state-augmented offline dataset with the predicted state sequence after state estimation, a policy model is trained, and the trained policy model is used to control the robot to perform teleoperation tasks. This can effectively improve the decision stability and control performance of reinforcement learning policies in delayed environments without additional interaction, thereby providing an efficient and feasible intelligent control solution for delay-sensitive applications such as robot teleoperation, underwater exploration, and space operations.

[0025] Other beneficial effects of the embodiments of the present invention will be further described below. Attached Figure Description

[0026] Figure 1 This is a flowchart of a robot teleoperation control method based on delay-free data training under a preferred embodiment of the present invention;

[0027] Figure 2 This is a general flowchart of a robot teleoperation control method based on delay-free data training in a delayed environment, as described in a specific embodiment of the present invention.

[0028] Figure 3 This is a data flow diagram in a specific embodiment of the present invention.

[0029] Figure 4a This is a structural diagram of the actor network in a specific embodiment of the present invention;

[0030] Figure 4b This is a structural diagram of the critic network in a specific embodiment of the present invention;

[0031] Figure 4c This is a diagram illustrating the calculation process of timing difference error in a specific embodiment of the present invention;

[0032] Figures 5a to 5d These are physical images of four types of multi-joint robots in the MuJoCo embodiment of the present invention. Detailed Implementation

[0033] The embodiments of the present invention will be described in detail below. It should be emphasized that the following description is merely exemplary and not intended to limit the scope and application of the present invention.

[0034] It should be noted that when a component is referred to as "fixed to" or "set on" another component, it can be directly on or indirectly on that other component. When a component is referred to as "connected to" another component, it can be directly connected to or indirectly connected to that other component. Furthermore, a connection can be used for both fixing and circuit / signal connectivity.

[0035] It should be understood that the terms "length", "width", "up", "down", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inner", and "outer" indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are only for the convenience of describing the embodiments of the present invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on the present invention.

[0036] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of embodiments of the present invention, "a plurality of" means two or more, unless otherwise explicitly specified.

[0037] The preferred embodiments of the present invention aim to solve the problems of high training cost, data mismatch and insufficient policy stability of existing reinforcement learning methods in real-world latency scenarios, and achieve safe, efficient and reliable control of robot teleoperation tasks.

[0038] Specifically, the present invention aims to solve the following three practical technical problems:

[0039] 1) To address the issues of online reinforcement learning algorithms relying on real-time interaction with the environment and high training costs, this invention employs a fixed, latency-free offline dataset to train the reinforcement learning model. This eliminates the need for frequent trial and error in real-world latency environments, reducing equipment wear and energy consumption, and avoiding safety risks caused by policy instability. This method can be directly applied to tasks requiring high stability, such as robot teleoperation and autonomous driving, improving system deployability and security.

[0040] 2) To address the issue that latency-free datasets cannot directly adapt to delayed environments, leading to control instability, the proposed state enhancement mechanism can structurally reconstruct latency-free offline data, enabling it to simulate state transition relationships under delayed environments. This solves the policy mismatch problem caused by "observation lag" in delayed environments. This method can significantly improve the response accuracy and control smoothness of systems such as teleoperated robotic arms and medical robots under delayed feedback conditions.

[0041] 3) To address the problem of sparse effective training samples and learning difficulties caused by increased latency, this invention introduces a latency state prediction module to estimate and model potential states under latency conditions, enabling the robot to maintain stable policy optimization capabilities even with limited data. This method can effectively improve control accuracy and mission success rate in high communication latency tasks (such as underwater and space teleoperation).

[0042] like Figure 1 As shown, a preferred embodiment of the present invention discloses a robot teleoperation control method based on delay-free data training in a delayed environment. Utilizing readily available delay-free offline data, it safely and efficiently trains a robot teleoperation control strategy that adapts to actual communication delays, thereby achieving safe, efficient, and reliable control of robot teleoperation tasks. The method includes the following steps:

[0043] S1: Collect offline datasets obtained by the robot performing tasks in a zero-latency environment. The offline datasets include multiple trajectories, and each trajectory includes a state sequence, an action sequence, and a reward sequence.

[0044] Among them, robots include multi-joint dynamic robots, whose tasks include motion control and trajectory tracking.

[0045] S2: Reconstruct the state and action sequences in the offline dataset to generate an information state sequence suitable for a delayed environment; realize state augmentation of offline data to expand the original delay-free offline dataset into a data structure suitable for delayed tasks, and achieve delay reconstruction at the data level.

[0046] Specifically, the current state in the offline dataset is combined with the action sequences from several previous moments to form an information state sequence suitable for delayed environments.

[0047] S3: Perform state estimation on the information state sequence to obtain the predicted state sequence; introduce a delayed state prediction model to reduce the dimensionality of the enhanced high-dimensional information state, thereby reducing training complexity and improving policy generalization performance.

[0048] Specifically, the predicted state sequence is obtained by using a direct substitution function, a Gaussian sampling function, a neural network function, or a shortened augmented state sequence function as a mapping function to estimate the state of the information state sequence.

[0049] Specifically, when using a direct substitution function as the mapping function, the actual state sequence from the offline dataset is directly used to replace the information state sequence as the predicted state sequence. When using a Gaussian sampling function as the mapping function, a Gaussian distribution is constructed based on the mean and variance of the state sequences in the offline dataset, and the predicted state sequence is sampled from the Gaussian distribution. When using a neural network function as the mapping function, a neural network function is used to perform a nonlinear mapping on the information state sequence to obtain the predicted state sequence. When using a shortened augmented state sequence function as the mapping function, a shorter historical state and action sequence is used to reconstruct the information state sequence to reduce the dimensionality of the information state sequence and obtain the predicted state sequence. When using a neural network function, the network parameters need to be trained through supervised learning, and the training data comes from the offline dataset.

[0050] It should be noted that dimensionality reduction or estimation of information states is an approximation process, aiming to balance computational complexity with the accuracy of state representation. Inaccuracies in the dimensionality reduction model introduce state estimation errors, which may affect the performance of subsequent policy learning. Therefore, in practical applications, an appropriate dimensionality reduction method and its model complexity should be selected based on the specific task's requirements for computational efficiency and accuracy.

[0051] S4: Based on the information state sequence and the predicted state sequence, an offline reinforcement learning algorithm is used to train the policy model. The policy model is used to output the action commands to control the robot based on the predicted state sequence in a delayed environment.

[0052] The offline reinforcement learning algorithm used is based on an actor-critic framework. The actor network takes the predicted state sequence as input and outputs the robot's action commands; the critic network takes the predicted state sequence and the robot's action commands as input and outputs the value assessment of the robot's action commands. Furthermore, a behavior cloning regularization term is added to the objective function of the actor network to ensure that the learned policy closely approximates the action distribution in the offline dataset.

[0053] The offline reinforcement learning algorithm is used to train the strategy model, which specifically includes the following interaction process between the robot and the environment in a delayed environment: In the early interaction stage, the robot remains stationary; in the later interaction stage, the robot constructs an information state sequence and outputs action commands using the most recent non-empty observation and the corresponding action sequence.

[0054] In a further embodiment, a delay environment construction step is included before step S1. This delay environment construction step includes: modeling the robot's teleoperation environment based on a delayed Markov decision process, whereby the observation lag caused by communication and control delays is defined during the modeling process. Further, the delay environment construction step specifically includes: defining a delayed Markov decision process tuple, which includes a measurable constant delay parameter, the robot's observable state space, an augmented information state space composed of historical observations and actions, and a reward function and a state transition function defined based on the augmented information state space.

[0055] The preferred embodiment of the present invention provides a robot teleoperation control method based on zero-delay data training in a delayed environment. Through data reconstruction and state prediction techniques, the offline dataset collected in a zero-delay environment is adapted to a real application scenario with communication delay. This enables the training of a robust control strategy that can adapt to delay without the need for high-risk online interaction, thereby achieving safe, efficient and reliable control of robot teleoperation tasks in a real delayed environment.

[0056] The following detailed description, in conjunction with specific embodiments, provides a preferred embodiment of the robot teleoperation control method based on delay-free data training under delayed conditions, as proposed in this invention.

[0057] like Figure 2 As shown in the figure, this specific embodiment proposes a robot teleoperation control method based on delay-free data training in a delayed environment, including the following steps:

[0058] A1: Delayed Environment Modeling. Addressing the perception and control delays in robot teleoperation, a delayed Markov decision process is defined to describe the time lag in the transmission and execution of control commands, providing a unified mathematical model for delayed policy learning.

[0059] The specific content can be further divided into three parts: building a standard reinforcement learning framework, describing the delayed environment, and constructing a delayed Markov decision process.

[0060] The first part is about building a standard reinforcement learning framework.

[0061] Reinforcement learning is typically modeled as a Markov Decision Process (MDP), which can be described as the following tuple: .in Represents the state space; Represents the action space; Represents the reward function. Moment reward is ; Represents the state transition function That is, from the current Moment State Action Pair Transition to the next state The probability of; Represents the initial state distribution function; This represents the discount factor. The goal of reinforcement learning is to find the optimal policy that maximizes the cumulative objective, i.e. ,in Representative strategy. Furthermore, the value function can be defined as... That is, in the state Execute action Expected return.

[0062] In robot teleoperation tasks, the various components of the MDP can be obtained by mapping actual sensors to control signals:

[0063] Moment State This includes the robot's end effector pose (position, attitude angle), joint angular velocity, force / torque sensor signals, and visual features from cameras or depth cameras;

[0064] Moment Action : Composed of target control signals output by the controller, such as joint angular velocity commands, end-effector displacement, or gripping force;

[0065] Moment Rewards Based on the definition of the operational task objectives, such as the negative value of the end position error, task completion rate, energy consumption minimization, or trajectory smoothness index;

[0066] State transition Determined by both robot dynamics and communication systems, it reflects the delay characteristics from control signals to physical responses.

[0067] This modeling enables robot teleoperation tasks to be analyzed and optimized within a standard reinforcement learning framework, providing a foundation for subsequent latency environment modeling and offline training.

[0068] The second part describes the latency environment.

[0069] In practical teleoperation systems, delays in multiple stages of the control link can significantly affect the effectiveness of reinforcement learning strategies. Delays caused by communication transmission, signal acquisition, and actuator response are generally equivalent to observation delays, and can be specifically categorized into the following two types:

[0070] (1) Constant delay This delay originates from fixed data transmission distances and signal processing times, such as communication delays between the ground control unit and the space robotic arm, or the fixed time delay between image acquisition and control execution in an industrial robot. This type of delay can be viewed as an observation lag at a fixed time step.

[0071] (2) Random delay ,in, The maximum value representing random delay originates from nondeterministic factors such as network congestion, communication jitter, or packet loss.

[0072] Therefore, the total delay can be expressed as:

[0073]

[0074] From a systems perspective, The state of environmental release at any moment The time observed by the robot should be Therefore, in Time Robot's Observation It can be represented as follows:

[0075]

[0076] in, represent Observe constantly The release moment, which is to satisfy The maximum value is determined by the fact that the closer the release time is to the current time, the more closely the state approximates the current state; therefore, the maximum value is taken. Representing an empty state, formula (3) represents observation. It may be empty or A state of constant release.

[0077] The third part is about constructing a delayed Markov decision process.

[0078] When dealing with signal delays, robots can measure constant delays, but random delays remain unknown. Therefore, at any given time, the robot may not observe any information, or it may observe a historical state with an unknown release time. Since an MDP augmented with information states can employ the same strategy as a plain MDP with delays, delayed reinforcement learning can be described as a tuple... The details are as follows:

[0079] (1) Represents a measurable constant delay. .

[0080] (2) This represents the state space observable by the robot. Observe constantly and .

[0081] (3) Represents the information state space, that is Time information status .and then, The definition is as follows:

[0082]

[0083] in, express When empty, it originally belonged to The position is closest to Non-empty observations at any given time are substituted. The process of defining the information state is the process of state enhancement.

[0084] (4) For the state space, and Equivalent.

[0085] (5) Represents the state transition function .

[0086] (6) represent The initial state distribution is defined as ,in The action specified when the state is unobservable.

[0087] (7) Represents the reward function. Moment Rewards It can be defined as follows:

[0088]

[0089] in That is, the reward function in the standard reinforcement learning framework.

[0090] (8) It is the discount factor, the same as in standard reinforcement learning.

[0091] Based on this, the objective function of delayed reinforcement learning should be modified to: ,in Representative strategy.

[0092] A2 (corresponding to steps S1 and S2 in the preferred embodiment of the present invention): Delay-free data reconstruction. Offline data of the robot performing tasks (such as grasping, carrying, and trajectory tracking) in a delay-free environment is collected. The original state and action sequence are combined into an information state through state enhancement, so that the delay-free data has delay characteristics, avoiding the need to recollect data in a real delay environment and reducing experimental costs and risks.

[0093] The offline dataset is collected in a latency-free environment. In this specific embodiment, state augmentation is further used to process it, enabling it to be used for delayed tasks. The offline dataset contains multiple trajectories, each recording the complete interaction process between the robot and the environment in a latency-free state within one round, including observation, action, post-action observation, reward, and task termination flag. It is assumed that the current task has a constant latency. For each trajectory ,in The trajectory elements can be adjusted as follows: the set of information states is defined as follows: The set of actions is defined as follows: The set of rewards is defined as follows: The set of termination flags is defined as follows: Therefore, the processed trajectory can be represented as The processed trajectory is stored in the dataset. .

[0094] To simulate observation lag under delayed conditions, at the first d time steps of each reconstructed trajectory, the "state" part of its information state is filled with a specific "null value" identifier (such as an all-zero vector), while the "action" part is filled with a preset initial action or zero action.

[0095] like Figure 3 As shown in the dataset section, the raw data retrieved from the offline dataset is latency-free, and its trajectory states and actions are in one-to-one correspondence, meaning the robot is in a state... Take action Afterwards, the environmental state changed The robot then took action. This process repeats. If the environmental latency is 1, the robot adapts to the current latency environment by augmenting the current state and actions into an information state, meaning the robot observes the information state. Make an action The environment becomes The robot then took action. And so it goes.

[0096] A3 (corresponding to step S3 in the preferred embodiment of the present invention): Delayed state prediction and dimensionality reduction. The dimensionality of information is reduced by utilizing the real states in the dataset, mitigating the exponential expansion of information state dimensions due to latency, and improving the learning efficiency and stability of the strategy in high-latency teleoperation scenarios (such as underwater or space operations).

[0097] A core challenge in offline reinforcement learning is handling extrapolation error, which involves evaluating the effectiveness of actions outside the existing dataset distribution. Value-based offline reinforcement learning algorithms address this issue through policy constraints, uncertainty estimation, and regularization. This architecture comprises two modules: (1) an actor network for decision-making, with input state... And output the action (1) It is used in both the training and validation processes; (2) It is used to evaluate value. Critic network, input state action pair Output value It is applied only during the training process and can evaluate the decision-making ability of the actor network.

[0098] When signal delay exists, observe It lags behind the actual state of the system, therefore it cannot be directly... As Instead of inputting actor and critic networks, the information status should be modified to... As Input actors and critics into a network. However, due to the information state space... The dimension is ( (used to solve for the dimension), while the original state space Dimension is Therefore, this correction method leads to an exponential increase in the state space. As latency increases, this difference will widen further, and the original latency-free dataset will be insufficient for delayed tasks.

[0099] In this specific embodiment, the critic network is only used during the training process and does not affect the verification process. Furthermore, the exponential growth of the state space primarily impacts the training process. Therefore, this specific embodiment employs delayed state prediction. Time information status Dimensionality reduction is achieved through a mapping function. get , making Therefore, as Figure 4a As shown, the input to the Actor network consists of the state Transform into information state The output remains unchanged; for example Figure 4b As shown, the critic network consists of responses to state actions. The evaluation is transformed into an assessment of information state action pairs. The evaluation is then transformed into predicted delayed state action pairs. The assessment.

[0100] In particular, mapping functions It can be defined in the following four ways:

[0101] (1) Direct replacement (abbreviated as Direct):

[0102]

[0103] This method uses the actual state Directly replace information status .

[0104] (2) Gaussian sampling (abbreviated as Gaussian):

[0105]

[0106] in, and It is a state front and back Step state sequence The mean and variance of the data. Sampling from the Gaussian distribution constructed above introduces greater data diversity.

[0107] (3) Neural network approximation (abbreviated as Dynamic):

[0108]

[0109] This method utilizes neural networks Approximate nonlinear function This is equivalent to using existing data to build a dynamic model to predict the state, thus achieving [the ability to predict the state]. Dimensional reduction.

[0110] (4) Shortening augmented state sequences (abbreviated as Short):

[0111] For the existence of constant delay And only a sequence of states and actions. It possesses the following properties: a. From state Begin by applying action sequences. Will eventually reach the state b. From the state Begin by applying action sequences. It will eventually reach its state. c. Regarding the state Application Actions Will reach state From the above characteristics, it can be seen that the information state It can be represented by shorter sequences of more recent states and actions:

[0112]

[0113] in, and This approach effectively reduces the dimensionality of information states while retaining the necessary historical information required to address latency issues.

[0114] In the specific embodiment of the present invention, the classic reinforcement learning framework Twin Delayed Deep Deterministic Policy Gradient (TD3) is used as the basic algorithm framework, and a behavior cloning regularization term is added to the objective function of the actor network. Therefore, the optimization problem of the actor network is as follows:

[0115]

[0116] Among them, the expected internal Representative value, Represents behavior cloning regularization. Hyperparameters The ratio used to balance reinforcement learning (maximizing value) and imitation learning (minimizing behavioral cloning loss) is defined as follows:

[0117]

[0118] in, Represents the contents of the sampling batch The quantity. Additionally, each information state needs to be regularized before use:

[0119]

[0120] in, Representative dataset All The mean, represent All variance It is a small constant used to prevent division by zero errors.

[0121] The timing difference error required for the critic network The calculation process is as follows Figure 4c As shown. Specifically, the truth value The calculation can be performed through the following process:

[0122]

[0123] in Representing the target actor (Actor) network, This represents the noise after trimming. Represents noise variance. This represents the clipping boundary. Based on this, the temporal difference objective can be constructed as follows:

[0124]

[0125] in, represent Momentary rewards Representing two critic networks The corresponding target critic network. Predictions are generated by... The calculation yields the following timing difference error values:

[0126]

[0127] This allows for the updating of the critic network. Additionally, Figure 4c Chinese commentators network 1 ( )correspond Target Commentator Network 1 ( ), Target Commentator Network 2 ( )Right now Actor Network ( )Right now .

[0128] Based on the above description, specific embodiments of the present invention combine a fixed offline no-delay dataset with state augmentation, delayed state prediction, etc., and on the basis of the TD3 framework with added behavior cloning regularization terms, finally propose a method fusion algorithm based on the actor-critic (MFAC).

[0129] A4 (corresponding to step S4 in the preferred embodiment of the present invention): Offline training of reinforcement learning strategy. Based on the reconstructed dataset and predicted state, offline reinforcement learning training is performed. This training process requires no online interaction, avoiding equipment wear and safety risks caused by frequent trial and error in high-latency control of the robot.

[0130] In this embodiment of the invention, the robot interacts with the environment only during the verification process, which can be divided into two stages:

[0131] (1) Preliminary interaction stage ( During this stage, due to observation delay, the robot's observations are all empty. Therefore, no action is taken during this stage, and the robot remains stationary. Taking a robot teleoperation task as an example, at this time, the control end has not yet received visual images, force / torque data, or joint angle feedback from the end effector, and therefore cannot accurately determine the current state. To avoid erroneous actions due to incomplete observation, this specific embodiment of the invention designs the robot to remain stationary during this stage, without sending any control commands, i.e., without performing any actions. Through this design, the robot can maintain safety and stability in the initial stage of communication establishment, preventing erroneous operations, collisions, or attitude instability caused by delay.

[0132] (2) Later interaction stage ( During this stage, the robot uses the nearest neighbor observation. And the corresponding action sequence to construct the information state If an observation is empty due to random delay, the nearest non-empty observation is used as an approximation of the current state. In robot teleoperation scenarios, the data sources at this stage are relatively abundant, potentially including visual features from cameras or depth cameras (image frames, pose estimation results), feedback from robotic arm joint angle and angular velocity sensors, force / torque sensor signals, and the execution status of the end effector. By constructing a state based on historical observations and action sequences, the robot can maintain decision-making consistency even with communication delays, achieving stable control of the teleoperated robotic arm.

[0133] A5 (corresponding to step S5 in the preferred embodiment of the present invention, where the robot executes a teleoperation task based on the trained strategy model): Strategy Verification and Application. The performance of the training strategy was verified on four multi-joint robots—"Ant," "HalfCheetah," "Hopper," and "Walker2d"—under the multi-joint dynamics robot simulation platform (MuJoCo). The results show that the method can significantly improve the control accuracy and task success rate of the robot under delayed communication conditions, and can be widely applied in fields such as remote robotic arm control, automated assembly, and space and underwater robot missions.

[0134] It should be noted that all the following experimental verifications were performed in a simulation environment. In a specific embodiment of the present invention, to verify the effectiveness of the proposed delayed environment offline reinforcement learning method for teleoperated robots, four multi-joint robots in the MuJoCo environment were selected for the experiment: "Ant", "HalfCheetah", "Hopper", and "Walker2d". Specifically, as follows... Figure 5a The "Ant" robot aims to learn to coordinate quadrupedal movements to move forward quickly while maintaining low energy consumption and body stability. Its state variables include trunk position, joint angles, and velocities, while its control variables are the motor torques driving the joints of its eight legs. For example... Figure 5b The "HalfCheetah" robot aims to mimic a cheetah's running motion, moving its torso forward as quickly as possible. Its state variables include torso orientation, joint position, and velocity; its control variables are the torques controlling the joints of its two hind legs. Figure 5c The "Hopper" robot aims to learn to propel itself forward continuously through consecutive jumps while maintaining balance. Its state variables include joint angles, body orientation, and velocity, while its control variables are the torques driving the joints of each leg. For example... Figure 5d The "Walker2d" robot aims to learn to walk steadily and coherently on two legs, like a human, and to travel as far as possible. Its state variables include body posture, joint information, and velocity, while the control variables are the torques that drive the joints of both legs. These tasks are widely applicable and representative in the field of reinforcement learning, simulating practical tasks such as robot motion control, joint coordination, and dynamic response.

[0135] The offline, latency-free dataset used in the experiment was derived from the Dataset for Data-Driven Deep Reinforcement Learning (D4RL), which contains the state, actions, and reward information of the robot performing tasks in a latency-free environment. To evaluate the performance of the method under different policy qualities, the following offline datasets were selected in this embodiment: (1) Medium-Expert (me): generated by a mixture of medium and expert policies, containing the robot's experience data in the high-performance task execution phase; (2) Medium (m): containing only the data generated by the robot performing tasks with medium policies, reflecting medium-level operational experience; (3) Medium-Replay (mr): containing all the data stored in the experience replay pool during the robot's training from the initial state to the medium robot level, covering the complete operational trajectory from low to medium levels. In the implementation, each offline dataset underwent state augmentation and latency reconstruction before the experiment: the latency-free state was combined with historical actions to construct an information state to simulate the observation input under latency conditions; the nearest non-empty observation was used to replace the missing observation to ensure the continuity and robustness of the policy under latency conditions.

[0136] To verify the effectiveness of MFAC, this embodiment of the invention compares it with offline RL algorithms such as DecisionTransformer (DT), Behavior Cloning (BC), Conservative Q-learning (CQL), and Twin Delayed Deep Deterministic Policy Gradient with Behavior Cloning (TD3_BC). In the experiments, a constant delay is set to... Random delay set to Simultaneously, the results of the optimal online RL algorithm, Belief-Projection-Based Q-learning (BPQL), and the results in a no-latency environment were also compared. Furthermore, the MFAC in the comparative experiments all employed direct replacement for dimensionality reduction. .

[0137] Table 1 below shows the average results for four different robots with different constant delays and different offline datasets. The bolded part represents the optimal result of the offline RL algorithm under the current task and delay settings (BPQL is an online RL algorithm and is not included in the comparison).

[0138] Table 1. Experimental results comparing the algorithm of the specific embodiment of the present invention with the baseline algorithm.

[0139]

[0140] As shown in Table 1, (1) In the teleoperation tasks of four different robot configurations, MFAC outperforms the other offline algorithms in both constant latency and random latency environments, and MFAC is more suitable for handling high latency tasks. (2) Although MFAC is weaker than the online RL algorithm BPQL, the training process of MFAC avoids the online interaction required by BPQL, and has great application potential in practical application scenarios such as remote control of space robots.

[0141] Furthermore, to verify the impact of different delayed state prediction methods on MFAC, specific embodiments of this invention apply four methods for dimensionality reduction: direct replacement, Gaussian sampling, dynamic neural network approximation, and shortening the augmented state sequence. To obtain Experiments were conducted on two robots, the "Hopper" and the "Walker2d," and the results are shown in Tables 2 and 3, respectively. Table 2 presents the ablation experiment results for delay state prediction using different methods in the "Hopper" task, and Table 3 presents the ablation experiment results for delay state prediction using different methods in the "Walker2d" task. "3-me" represents constant delay. Offline datasets use "me", and similar descriptions can be applied to other datasets.

[0142] Table 2 shows the ablation experimental results of the present invention using different methods for delayed state prediction in the "Jumper" task.

[0143]

[0144] Table 3 shows the ablation experimental results of the present invention using different methods for delayed state prediction in the "bipedal walker" task.

[0145]

[0146] From Tables 2 and 3, we can observe that: (1) for the "Hopper" robot, the Direct replacement and Shortened augmented state sequence methods outperform the other methods, with similar performance between the two; (2) for the "Walker2d" robot, the Direct replacement method outperforms the other methods, exhibiting optimal performance in two-thirds of the datasets. Therefore, using the Direct replacement method for dimensionality reduction is a good approach. get It is the optimal choice for MFAC.

[0147] Based on Tables 1 to 3, the results of the teleoperation experiments on four types of robots—Ant, Half-Cheetah, Hopper, and Walker2d—on the multi-joint dynamics robot simulation platform (MuJoCo) demonstrate that the method of this invention can effectively improve the decision-making stability and control performance of reinforcement learning strategies in latency environments without additional interaction. This provides an efficient and feasible intelligent control solution for latency-sensitive applications such as robot teleoperation, underwater exploration, and space operations.

[0148] The robot teleoperation control method based on delay-free data training in a delayed environment proposed in the preferred embodiment of the present invention has the following advantages:

[0149] (1) Offline training reduces risk and cost

[0150] By using a fixed offline data training strategy, repeated interactions in a real-time latency environment are eliminated, avoiding equipment wear, motion jitter, and safety risks that may occur during the training phase for remote robotic arms, autonomous vehicles, or underwater robots. At the same time, it reduces computing resource consumption and provides a feasible basis for actual system deployment.

[0151] (2) State enhancement enables data migration

[0152] This method involves enhancing and reconstructing the state of zero-latency data to adapt it to delayed environments. Data acquired from experimental platforms or simulations can then be directly used for remote control tasks, such as space robots or underwater robotic arms, achieving high-precision control under delayed conditions.

[0153] (3) Delayed state prediction guarantees policy robustness

[0154] Introducing delayed state prediction to reduce the dimensionality and estimate information states enables the strategy to be executed stably even when the delay increases or changes randomly, effectively improving the success rate of tasks and the continuity of actions. It is suitable for delay-sensitive scenarios such as teleoperation, autonomous driving and medical robots.

[0155] (4) The experimental verification results are excellent.

[0156] Under the conditions of four different robot configurations, namely "Ant", "HalfCheetah", "Hopper", and "Walker2d", on the multi-joint dynamics robot simulation platform (MuJoCo simulation platform) and with different latency and offline datasets, the method of this invention shows higher performance and stronger robustness compared with offline baseline algorithms such as DT, BC, CQL, and TD3_BC, verifying its application potential in high-latency continuous control tasks.

[0157] Another preferred embodiment of the present invention discloses a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the steps of the robot teleoperation control method based on delay-free data training under a delay environment as described in the preferred embodiment above.

[0158] Optionally, the aforementioned storage media may include, but are not limited to, USB flash drives, read-only memory (ROM), random access memory (RAM), portable hard drives, magnetic disks, or optical disks, and other media capable of storing computer programs.

[0159] The background section of this invention may include background information about the problems or circumstances surrounding the invention, rather than a description of prior art by others. Therefore, the content included in the background section is not an admission of prior art by the applicant.

[0160] The above description provides a further detailed explanation of the present invention in conjunction with specific / preferred embodiments, and it should not be construed that the specific implementation of the present invention is limited to these descriptions. For those skilled in the art, various substitutions or modifications can be made to these described embodiments without departing from the concept of the present invention, and all such substitutions or modifications should be considered within the scope of protection of the present invention. In the description of this specification, the reference to terms such as "an embodiment," "some embodiments," "preferred embodiment," "example," "specific example," or "some examples," etc., indicates that the specific features, structures, materials, or characteristics described in connection with that embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials, or characteristics described can be combined in any suitable manner in one or more embodiments or examples. Furthermore, those skilled in the art can combine and integrate the different embodiments or examples described in this specification and the features of different embodiments or examples without contradiction. Although the embodiments of the present invention and their advantages have been described in detail, it should be understood that various changes, substitutions, and modifications can be made herein without departing from the scope defined by the appended claims.

Claims

1. A method for robot teleoperation control based on delay-free data training in a delay environment, characterized in that, The method comprises the following steps: S1: collecting an offline data set obtained by a robot performing a task in a delay-free environment, the offline data set comprising a plurality of trajectories, each trajectory comprising a state sequence, an action sequence and a reward sequence; S2: reconstructing the state sequence and the action sequence in the offline data set to generate an information state sequence suitable for a delay environment; S3: performing state estimation on the information state sequence to obtain a predicted state sequence; S4: training a policy model based on the information state sequence and the predicted state sequence using an offline reinforcement learning algorithm, the policy model being used to output an action instruction for controlling the robot based on the predicted state sequence in the delay environment; S5: using the trained policy model to control the robot to perform a teleoperation task in the delay environment. In step S2, the state at the current time in the offline data set is combined with the action sequence at the previous time to form an information state sequence suitable for the delay environment.

2. The robotic teleoperation control method of claim 1, wherein, In step S3, the information state sequence is estimated by using a direct replacement function, a Gaussian sampling function, a neural network function or a shortened augmented state sequence function as a mapping function to obtain the predicted state sequence.

3. The robot teleoperation control method of claim 2, wherein when the direct replacement function is used as the mapping function, the actual state sequence in the offline data set is used to directly replace the information state sequence as the predicted state sequence; when the Gaussian sampling function is used as the mapping function, a Gaussian distribution is constructed based on the mean and variance of the state sequence in the offline data set, and the predicted state sequence is sampled from the Gaussian distribution; when the neural network function is used as the mapping function, the information state sequence is nonlinearly mapped by using the neural network function to obtain the predicted state sequence; when the shortened augmented state sequence function is used as the mapping function, the information state sequence is reconstructed by using shorter historical state and action sequences to reduce the dimension of the information state sequence, thereby obtaining the predicted state sequence.

4. The robotic teleoperation control method of claim 1, wherein, The offline reinforcement learning algorithm used in step S4 is an algorithm based on an actor-critic framework, the input of the actor network is the predicted state sequence, and the output is the action instruction of the robot; the input of the critic network is the predicted state sequence and the action instruction of the robot, and the output is the value evaluation of the action instruction of the robot.

5. The robotic teleoperation control method of claim 4, wherein, A behavior cloning regularization term is added to the objective function of the actor network to constrain the learned policy to be close to the action distribution in the offline data set.

6. The robotic teleoperation control method of claim 1, wherein, The training of the policy model in step S4 using the offline reinforcement learning algorithm comprises the following interaction process between the robot and the environment in the delay environment: In the early interaction stage, the robot remains stationary; in the later interaction stage, the robot constructs an information state sequence using the latest non-empty observation and the corresponding action sequence and outputs an action instruction.

7. The robotic teleoperation control method of claim 1, wherein, The step S1 is preceded by a step of building a delayed environment, wherein the step of building a delayed environment comprises modeling the robot teleoperation environment based on a delayed Markov Decision Process, in which the observation lag caused by the communication and control delays is also defined.

8. The robotic teleoperation control method of claim 7, wherein, The step of building a delayed environment comprises in particular defining a delayed Markov Decision Process tuple comprising a constant delay parameter that is measurable, a state space observable by the robot, an augmented information state space constituted by the history of observations and actions, and a reward function and a state transition function defined based on the augmented information state space.

9. A computer readable storage medium having stored thereon a computer program, characterized in that, The computer program, when executed by a processor, implements the steps of the robot teleoperation control method according to any one of claims 1 to 8.

Images