A teleoperation robot motion control method and system based on latent space alignment

Abstract

The application provides a teleoperation robot action control method and system based on hidden space alignment, and belongs to the technical field of robot control. The application aligns the augmented state and the real state at the same time step in the hidden space and the action value respectively by constructing an augmented state policy network and a hidden space value network, so that the precise action control of the teleoperation robot is realized. The application is used for solving the technical problems of inconsistent input structure, difficult unified representation and biased value estimation in the existing delayed reinforcement learning technology, fundamentally improving the decision stability and value evaluation accuracy in the delayed environment, and improving the reliability of the decision of the reinforcement learning used for performing action control under the condition of communication and perception delay.

Description

Technical Field

[0001] This invention belongs to the field of robot control technology, and in particular relates to a method and system for controlling the motion of a teleoperated robot based on latent space alignment. Background Technology

[0002] In recent years, with the deep integration of artificial intelligence and robotics, intelligent control methods based on deep reinforcement learning (DRL) have received widespread attention in fields such as automated manufacturing, service robots, unmanned systems, and teleoperated robots. Through continuous interaction with the environment, DRL can autonomously learn optimal control strategies in complex scenarios with high dimensions, nonlinearity, and dynamic uncertainty, thereby achieving autonomous decision-making and high-precision control.

[0003] In specific operational domains, teleoperated robot systems have become an important form of remote operation, environmental management, and complex task execution. A typical teleoperated robot system consists of an operator (human-machine interface device or ground control station) and a remote execution end (mobile platform, robotic arm, or unmanned system). The operator sends control commands in real time via a communication link and simultaneously receives feedback information from multimodal sensors (including vision, force, inertia, pose, etc.) to assist in human-machine collaborative decision-making and control command generation. However, in actual deployment, teleoperation systems typically involve multiple transmission links, including long-distance communication, signal compression and decoding, control calculation, and mechanical execution. These links inevitably introduce factors such as communication latency, perception latency, and execution latency. The accumulation of latency causes a time misalignment between the remote robot's perceived information and the control commands generated by the operator, making it impossible for the control loop to maintain real-time consistency. This latency characteristic severely limits the effectiveness of reinforcement learning-based intelligent control algorithms in teleoperation scenarios. Traditional reinforcement learning algorithms rely on the assumption of synchronous interaction between the agent and the environment. When there is a delay in observation or action, the state received by the agent can no longer accurately reflect the true dynamics of the environment, thus violating the Markov property of reinforcement learning. As a result, the agent's decision-making is distorted and control signals are lagging, leading to decreased operational accuracy, system oscillation, or task failure. For example, in tasks such as remote maintenance and regional search and rescue, if the instructions from the operating end are delayed by hundreds of milliseconds, it may cause the robotic arm's trajectory to deviate, target tracking to be interrupted, or attitude control to become unstable, and in severe cases, even lead to equipment damage and task failure. This performance degradation makes it difficult to directly apply existing DRL-based control strategies to teleoperated robot systems with significant time delays.

[0004] Existing research typically employs two types of methods to mitigate the latency problem: one is a prediction compensation strategy based on internal environment modeling, which estimates the current true state by introducing state prediction or delay compensation models. However, prediction errors accumulate rapidly in dynamic and highly variable environments, leading to a decrease in control stability. The other is a delay processing strategy based on historical information extension, which concatenates historical states with action sequences to form augmented state inputs to recover some Markov properties. While this method improves robustness at the policy level, it is prone to information mismatch problems in the value estimation stage: if the value network directly uses the augmented state as input, redundant or outdated information will be introduced; if only the current true state is used as input, it will be inconsistent with the input distribution of the policy network, resulting in training instability and insufficient generalization ability.

[0005] Therefore, addressing the prevalent communication and perception latency issues in teleoperated robots, there is an urgent need to propose a reinforcement learning network structure that can balance policy input integrity and value assessment consistency under latency conditions, achieving robust latency handling at the network representation level. This scheme should, without altering the robot's hardware architecture, leverage the principle of latent space alignment, employing a bi-branch state mapping and consistency constraint method, and adopting a joint optimization mechanism of latent representation alignment and value function to achieve stable decision-making and improved adaptive control performance under latency conditions. This would provide a deployable solution for teleoperated agent learning in complex environments. Summary of the Invention

[0006] To address the aforementioned technical problems, this invention proposes a teleoperated robot motion control method and system based on latent space alignment; aiming to overcome the problems of inconsistent input structure, difficulty in unifying representation, and value estimation bias in existing time-delay reinforcement learning techniques, and improve the decision reliability of reinforcement learning under conditions of communication and perception delays.

[0007] The technical problems solved by this invention include the following four aspects.

[0008] (1) Augmented state input suffers from redundancy and noise issues. In time-delay environments, to maintain Markov property, the policy network input is usually an augmented state containing historical observations and action sequences. This augmented state contains a large amount of historical information, including both decision-related dynamic features and redundant or outdated information. Existing technologies lack a mechanism to provide stable value references for key decision features in the augmented state, making policy network training susceptible to interference from redundant information. This invention introduces latent space encoding and alignment mechanisms into the value network, enabling the policy network to obtain stable value signals under augmented state input, thereby indirectly mitigating the impact of redundancy and noise on policy training, while the policy network itself does not need to participate in latent space encoding.

[0009] (2) Unstable estimation of the true state affects value assessment. Existing technologies often recover the true state through prediction or environmental models in delayed environments. However, due to communication delays, perception errors, and the dynamic complexity of the system, the predicted state is uncertain. This problem leads to increased bias in value network assessment and affects the accuracy of policy updates. This invention constructs a latent space encoding and alignment structure in the value network to map the true state to the latent space and align it, thereby reducing the impact of prediction errors on value assessment and ensuring that the policy network obtains reliable value feedback under augmented state input.

[0010] (3) Inconsistency between the policy network and the value network representations leads to value bias. The policy network takes augmented states as input, while the value network estimates based on the mapped latent space true states. The inputs differ in dimension, structure, and semantics. Existing technologies have not established an effective policy-value representation alignment mechanism, resulting in large value estimation biases and unstable policy updates. This invention aligns latent space features and action values ​​within the value network, ensuring consistency between the augmented state input policy network and the latent space value network in their optimization objectives, thereby reducing value bias caused by representation inconsistency.

[0011] (4) Lack of consistency constraints in policy-value coupling. Since the policy network and value network are trained based on augmented states and latent space mapping states respectively, there is a lack of a unified constraint mechanism. Existing technologies fail to ensure that the policy update direction is consistent with the value function evaluation, which can easily lead to instability in the training process. This invention introduces feature alignment loss and action value alignment loss into the latent space of the value network, so that the policy network obtains a consistent optimization signal under augmented state input, thereby achieving policy-value co-optimization.

[0012] The first aspect of this invention proposes a method for motion control of a teleoperated robot based on latent space alignment, the method comprising:

[0013] S1. Perform Markov modeling for the delayed environment;

[0014] In this study, Markov modeling is performed on the actual operating environment of the teleoperated robot, which has delays in communication and perception, to determine the time series dependency between the augmented state with delays and the real state without delays, and to further quantify the impact of the delays on the teleoperated robot's action execution and state observation.

[0015] S2. Design a strategy network based on the aforementioned latency environment;

[0016] The policy network takes the augmented state as input, and the teleoperated robot generates control commands based on historical observations and action sequences in the delayed environment, which are the outputs of the policy network. The control commands refer to continuous and stable actions to be executed.

[0017] S3. Construction and alignment of the implicit space of the execution value network;

[0018] Among them, a latent space encoding of the value network is designed to map the real state to the latent space. Through latent space feature alignment and action value alignment, the value network and the policy network are made to work together in terms of optimization objectives. The alignment mechanism is designed to address the input distribution differences caused by the delay.

[0019] S4. Train the policy using policy networks and value networks;

[0020] In the delayed environment, the policy network outputs continuous and stable actions in the augmented state to obtain feedback after the delay. The value network combines the loss of temporal difference, latent space feature alignment and action value alignment to optimize and train the policy.

[0021] S5. Validate and apply the trained strategy;

[0022] The trained strategy was applied to a teleoperated robot task set on a multi-joint contact dynamics robot platform. Experiments were conducted using fixed delays and different strategies to verify that the trained strategy could achieve motion control of the teleoperated robot under various delay scenarios.

[0023] In S1, during the task performed by the teleoperated robot, there is a delay in the communication link between the operator and the execution end. d This allows the operating terminal to be constantly t The received state has a delay. s t-d Lagging behind a state without delay s t The decision process in Markov modeling under delayed conditions consists of six tuples. Characterization is performed; where:

[0024] X Represents the delayed state space, Indicates time t The state of the time-delay state space, The non-delayed state space S and the past d It consists of a sequence of historical actions;

[0025] A Represents the action space; Indicates time t Actions in time, delayed state space X The state in , This is a historical sequence of actions, with subscripts indicating the time.

[0026] This represents the initial state distribution under a delayed environment. , This represents the initial state under a delayed environment. s 0 This represents the initial state in a non-delayed environment. This represents the initial state distribution under a non-delayed environment. Represents the Dirac function, This indicates the pre-interaction of a teleoperated robot with a non-delayed environment. d At a moment when a state is unobservable This represents a sequence of actions when the state is unobservable. This represents the sequence of actions in the action space when the state is unobservable; the initial state refers to... t =0;

[0027] P Let denote the transfer function, and we have:

[0028]

[0029] in, Indicates the state Next action Then transferred to state The probability, Indicates the state s t-d Next action Then transferred to state s t-d+1 The probability of;

[0030] in, Indicates the state Down t-d+j Momentary actions Indicates the state Down t-d+j Momentary actions , Indicates the state Down t Actions at any given moment;

[0031] R Represents the reward function; because in t Only observed at that time td The reward function is configured to reflect the state at time t. , Indicates the state Next action The rewards received Indicates the state s t-d Next action The rewards received;

[0032] Indicates the discount factor. ;

[0033] Wherein, the strategy is represented as , Represents the policy network.

[0034] In S2, under the delayed environment, the policy network generates control commands based on historical observations and action sequences. As input, with s t-d For a state with a delay, Given a historical action sequence, the policy network outputs continuous and stable actions. And there are , Indicates that it is composed of network parameters The policy network is composed of neural networks; where:

[0035] The teleoperated robot receives the state returned by the delayed environment, extracts the corresponding historical action sequence from the experience replay pool to construct an augmented state, and the policy network processes the constructed augmented state to obtain control instructions, which refer to continuous and stable actions to be executed.

[0036] In S3, a latent space encoding and alignment mechanism is introduced into the value network to map the real state to the latent space. Through latent space feature alignment and action-value alignment, the value network and the policy network achieve synergy in their optimization objectives. Specifically:

[0037] In the process of value network latent space encoding, the value network latent space encoder State Mapping to the latent space, we get :

[0038]

[0039] in, Belongs to the real number field, with a latent space dimension of . , This represents the parameters of the latent space encoder in the value network. The latent space encoding is used to extract the state. s t Value networks are characterized by decision-making-related features. and For input, output is for The assessed value.

[0040] In S3, latent space feature alignment includes:

[0041] Using augmented state encoders State Mapping to the latent space, we get :

[0042]

[0043] By minimizing and The Euclidean distance between them is used to align latent space features. The latent space feature alignment process is as follows. Represented as:

[0044]

[0045] in, D This represents the experience replay pool. Indicates from D Medium sampling data , Indicates from D Data obtained from mid-sampling Seeking expectations.

[0046] In S3, the action value alignment process Represented as:

[0047]

[0048] in, Represents a value network. Indicates that it is composed of network parameters The value network composed of neural networks Indicates from D Medium sampling data , Indicates to from D Data obtained from mid-sampling Seeking expectations.

[0049] In S4, the training process includes five stages: initialization, data acquisition, delayed environment interaction, network update, and iterative loop; among which:

[0050] During the initialization phase:

[0051] Configure the policy network to output the control commands at the current moment as actions;

[0052] Configure a value network to evaluate the value of actions and replicate the generated target value network to maintain training stability;

[0053] Create an encoder to map the augmented state and the real state to the latent space;

[0054] Set network update ratio ;

[0055] During the data acquisition phase:

[0056] Before the initial training period d Each time step interacts with the delayed environment using a random strategy:

[0057] Perform random actions and record the corresponding delay states;

[0058] Store the action and delay state in a temporary buffer. B , used to construct augmented states.

[0059] In S4, when the number of training steps exceeds d Then, the delayed environment interaction phase begins:

[0060] from B Extract s t-d and Thus constructing , ;

[0061] Policy network according to Output Execute in a delayed environment and obtain s t-d+1 and ;

[0062] Will s t-d+1 , , deposit B ;

[0063] When the number of training steps exceeds 2 d Afterwards, from B Extract relevant data to construct as well as( s t, s t-1 ), and will Stored as a piece of data D .

[0064] In S4:

[0065] During the network update phase:

[0066] from D Medium sampling N The data is represented as:

[0067]

[0068] in, Indicates the first k data, ;

[0069] calculate:

[0070]

[0071] Computational Value Bellman Objective :

[0072]

[0073] Represents a value network. Indicates that it is composed of network parameters A value network composed of neural networks will be formed by networks with parameters. The value network constructed from neural networks serves as the target value network. Indicates that The action with the greatest value;

[0074] Computational value Bellman loss :

[0075]

[0076] calculate and ;

[0077] Calculate total loss :

[0078]

[0079] in, and This is the adjustment coefficient;

[0080] by Update parameters , This represents the gradient descent method;

[0081] use Guidance strategy update;

[0082] by For parameters Update;

[0083] During the iterative loop phase:

[0084] Repeat the process of delaying the interaction between the environment and the network until training ends.

[0085] A second aspect of this invention provides a teleoperated robot motion control system based on latent space alignment, the system comprising a processing unit configured to execute:

[0086] Markov modeling is performed for delayed environments;

[0087] In this study, Markov modeling is performed on the actual operating environment of the teleoperated robot, which has delays in communication and perception, to determine the time series dependency between the augmented state with delays and the real state without delays, and to further quantify the impact of the delays on the teleoperated robot's action execution and state observation.

[0088] Design a strategy network based on the aforementioned latency environment;

[0089] The policy network takes the augmented state as input, and the teleoperated robot generates control commands based on historical observations and action sequences in the delayed environment, which are the outputs of the policy network. The control commands refer to continuous and stable actions to be executed.

[0090] Constructing and aligning the implicit space of the execution value network;

[0091] Among them, a latent space encoding of the value network is designed to map the real state to the latent space. Through latent space feature alignment and action value alignment, the value network and the policy network are made to work together in terms of optimization objectives. The alignment mechanism is designed to address the input distribution differences caused by the delay.

[0092] The policy is trained using a policy network and a value network;

[0093] In the delayed environment, the policy network outputs continuous and stable actions in the augmented state to obtain feedback after the delay. The value network combines the loss of temporal difference, latent space feature alignment and action value alignment to optimize and train the policy.

[0094] Validate and apply the trained strategy;

[0095] The trained strategy was applied to a teleoperated robot task set on a multi-joint contact dynamics robot platform. Experiments were conducted using fixed delays and different strategies to verify that the trained strategy could achieve motion control of the teleoperated robot under various delay scenarios.

[0096] In summary, this invention proposes a teleoperated robot motion control method and system based on latent space alignment. By constructing an augmented state policy network and a latent space value network, and aligning the augmented state and the corresponding real state at the same time step in terms of latent space representation and motion value, the method fundamentally improves the decision stability and value assessment accuracy in time-delay environments.

[0097] Compared with the prior art, the present invention has the following beneficial effects: (1) Improves the stability of value assessment under delayed conditions. By encoding the real state in the latent space and estimating the action value in this space, the noise interference introduced by communication and sensing lag is significantly reduced, providing a more stable and reliable value signal for policy updates. (2) Eliminates the input inconsistency problem between augmented state and real state. The present invention aligns the augmented state and the real state at the same time step, enabling the policy side and the value side to obtain semantically consistent supervision information, effectively avoiding training instability and policy drift caused by input distribution mismatch. (3) Suppresses dimensional expansion caused by state augmentation. By replacing the direct stacking of high-dimensional historical states with latent space compression representation, the present invention reduces the model size and computational overhead, and improves the scalability and real-time performance of the method in multi-degree-of-freedom robot systems. (4) Improves sample efficiency and convergence speed. Consistent latent space and value alignment enhance the effective contribution of each sample to policy and value learning, improves data utilization, shortens training convergence time, and obtains higher control performance under various delay conditions. Attached Figure Description

[0098] To more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0099] Figure 1 This is a flowchart illustrating the motion control method for a teleoperated robot based on latent space alignment according to an embodiment of the present invention.

[0100] Figure 2 This is a schematic diagram of data flow in a policy network according to an embodiment of the present invention;

[0101] Figure 3 This is a schematic diagram of data flow in a value network according to an embodiment of the present invention. Detailed Implementation

[0102] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0103] The first aspect of this invention proposes a method for motion control of a teleoperated robot based on latent space alignment, the method comprising:

[0104] S1. Perform Markov modeling for the delayed environment;

[0105] In this study, Markov modeling is performed on the actual operating environment of the teleoperated robot, which has delays in communication and perception, to determine the time series dependency between the augmented state with delays and the real state without delays, and to further quantify the impact of the delays on the teleoperated robot's action execution and state observation.

[0106] S2. Design a strategy network based on the aforementioned latency environment;

[0107] The policy network takes the augmented state as input, and the teleoperated robot generates control commands based on historical observations and action sequences in the delayed environment, which are the outputs of the policy network. The control commands refer to continuous and stable actions to be executed.

[0108] S3. Construction and alignment of the implicit space of the execution value network;

[0109] Among them, a latent space encoding of the value network is designed to map the real state to the latent space. Through latent space feature alignment and action value alignment, the value network and the policy network are made to work together in terms of optimization objectives. The alignment mechanism is designed to address the input distribution differences caused by the delay.

[0110] S4. Train the policy using policy networks and value networks;

[0111] In the delayed environment, the policy network outputs continuous and stable actions in the augmented state to obtain feedback after the delay. The value network combines the loss of temporal difference, latent space feature alignment and action value alignment to optimize and train the policy.

[0112] S5. Validate and apply the trained strategy;

[0113] The trained strategy was applied to a teleoperated robot task set on a multi-joint contact dynamics robot platform. Experiments were conducted using fixed delays and different strategies to verify that the trained strategy could achieve motion control of the teleoperated robot under various delay scenarios.

[0114] In S1, during the task performed by the teleoperated robot, there is a delay in the communication link between the operator and the execution end. d This allows the operating terminal to be constantly t The received state has a delay. s t-d Lagging behind a state without delay st The decision process in Markov modeling under delayed conditions consists of six tuples. Characterization is performed; where:

[0115] X Represents the delayed state space, Indicates time t The state of the time-delay state space, The non-delayed state space S and the past d It consists of a sequence of historical actions;

[0116] A Represents the action space; Indicates time t Actions in time, delayed state space X The state in , This is a historical sequence of actions, with subscripts indicating the time.

[0117] This represents the initial state distribution under a delayed environment. , This represents the initial state under a delayed environment. s 0 This represents the initial state in a non-delayed environment. This represents the initial state distribution under a non-delayed environment. Represents the Dirac function, This indicates the pre-interaction of a teleoperated robot with a non-delayed environment. d At a moment when a state is unobservable This represents a sequence of actions when the state is unobservable. This represents the sequence of actions in the action space when the state is unobservable; the initial state refers to... t =0;

[0118] P Let denote the transfer function, and we have:

[0119]

[0120] in, Indicates the state Next action Then transferred to state The probability, Indicates the state s t-d Next action Then transferred to state s t-d+1 The probability of;

[0121] in, Indicates the state Down t-d+j Momentary actions Indicates the state Down t-d+j Momentary actions , Indicates the state Down t Actions at any given moment;

[0122] R Represents the reward function; because in t Only observed at that time td The reward function is configured to reflect the state at time t. , Indicates the state Next action The rewards received Indicates the state s t-d Next action The rewards received;

[0123] Indicates the discount factor. ;

[0124] Wherein, the strategy is represented as , Represents the policy network.

[0125] In S2, under the delayed environment, the policy network generates control commands based on historical observations and action sequences. As input, with s t-d For a state with a delay, Given a historical action sequence, the policy network outputs continuous and stable actions. And there are , Indicates that it is composed of network parameters The policy network is composed of neural networks; where:

[0126] The teleoperated robot receives the state returned by the delayed environment, extracts the corresponding historical action sequence from the experience replay pool to construct an augmented state, and the policy network processes the constructed augmented state to obtain control instructions, which refer to continuous and stable actions to be executed.

[0127] In S3, a latent space encoding and alignment mechanism is introduced into the value network to map the real state to the latent space. Through latent space feature alignment and action-value alignment, the value network and the policy network achieve synergy in their optimization objectives. Specifically:

[0128] In the process of value network latent space encoding, the value network latent space encoder State Mapping to the latent space, we get :

[0129]

[0130] in, Belongs to the real number field, with a latent space dimension of . , This represents the parameters of the latent space encoder in the value network. The latent space encoding is used to extract the state. s t Value networks are characterized by decision-making-related features. and For input, output is for The assessed value.

[0131] In S3, latent space feature alignment includes:

[0132] Using augmented state encoders State Mapping to the latent space, we get :

[0133]

[0134] By minimizing and The Euclidean distance between them is used to align latent space features. The latent space feature alignment process is as follows. Represented as:

[0135]

[0136] in, D This represents the experience replay pool. Indicates from D Medium sampling data , Indicates to from D Data obtained from mid-sampling Seeking expectations.

[0137] In S3, the action value alignment process Represented as:

[0138]

[0139] in, Represents a value network. Indicates that it is composed of network parameters The value network composed of neural networks Indicates from DMedium sampling data , Indicates to from D Data obtained from mid-sampling Seeking expectations.

[0140] In S4, the training process includes five stages: initialization, data acquisition, delayed environment interaction, network update, and iterative loop; among which:

[0141] During the initialization phase:

[0142] Configure the policy network to output the control commands at the current moment as actions;

[0143] Configure a value network to evaluate the value of actions and replicate the generated target value network to maintain training stability;

[0144] Create an encoder to map the augmented state and the real state to the latent space;

[0145] Set network update ratio ;

[0146] During the data acquisition phase:

[0147] Before the initial training period d Each time step interacts with the delayed environment using a random strategy:

[0148] Perform random actions and record the corresponding delay states;

[0149] Store the action and delay state in a temporary buffer. B , used to construct augmented states.

[0150] In S4, when the number of training steps exceeds d Then, the delayed environment interaction phase begins:

[0151] from B Extract s t-d and Thus constructing , ;

[0152] Policy network according to Output Execute in a delayed environment and obtain s t-d+1 and ;

[0153] Will s t-d+1 , , deposit B;

[0154] When the number of training steps exceeds 2 d Afterwards, from B Extract relevant data to construct as well as( s t, s t-1 ), and will Stored as a piece of data D .

[0155] In S4:

[0156] During the network update phase:

[0157] from D Medium sampling N The data is represented as:

[0158]

[0159] in, Indicates the first k data, ;

[0160] calculate:

[0161]

[0162] Computational Value Bellman Objective :

[0163]

[0164] Represents a value network. Indicates that it is composed of network parameters A value network composed of neural networks will be formed by networks with parameters. The value network constructed from neural networks serves as the target value network. Indicates that The action with the greatest value;

[0165] Computational value Bellman loss :

[0166]

[0167] calculate and ;

[0168] Calculate total loss :

[0169]

[0170] in, and This is the adjustment coefficient;

[0171] by Update parameters , This represents the gradient descent method;

[0172] use Guidance strategy update;

[0173] by For parameters Update;

[0174] During the iterative loop phase:

[0175] Repeat the process of delaying the interaction between the environment and the network until training ends.

[0176] A second aspect of this invention provides a teleoperated robot motion control system based on latent space alignment, the system comprising a processing unit configured to execute:

[0177] Markov modeling is performed for delayed environments;

[0178] In this study, Markov modeling is performed on the actual operating environment of the teleoperated robot, which has delays in communication and perception, to determine the time series dependency between the augmented state with delays and the real state without delays, and to further quantify the impact of the delays on the teleoperated robot's action execution and state observation.

[0179] Design a strategy network based on the aforementioned latency environment;

[0180] The policy network takes the augmented state as input, and the teleoperated robot generates control commands based on historical observations and action sequences in the delayed environment, which are the outputs of the policy network. The control commands refer to continuous and stable actions to be executed.

[0181] Constructing and aligning the implicit space of the execution value network;

[0182] Among them, a latent space encoding of the value network is designed to map the real state to the latent space. Through latent space feature alignment and action value alignment, the value network and the policy network are made to work together in terms of optimization objectives. The alignment mechanism is designed to address the input distribution differences caused by the delay.

[0183] The policy is trained using a policy network and a value network;

[0184] In the delayed environment, the policy network outputs continuous and stable actions in the augmented state to obtain feedback after the delay. The value network combines the loss of temporal difference, latent space feature alignment and action value alignment to optimize and train the policy.

[0185] Validate and apply the trained strategy;

[0186] The trained strategy was applied to a teleoperated robot task set on a multi-joint contact dynamics robot platform. Experiments were conducted using fixed delays and different strategies to verify that the trained strategy could achieve motion control of the teleoperated robot under various delay scenarios.

[0187] In some embodiments, the present invention discloses a method for controlling the motion of a teleoperated robot based on latent space alignment, such as... Figure 1 As shown, the method mainly includes the following steps.

[0188] S1: Delayed environment Markov modeling.

[0189] Markov modeling is performed on the actual operating environment of teleoperated robots with communication and perception delays. The time series dependency between augmented state and real state is clarified, and the impact of delay on robot action execution and state observation is quantified, providing a theoretical basis for the subsequent design of policy network and value network.

[0190] S2: Policy Network Design.

[0191] By constructing a policy network with augmented states as input, the robot can generate control commands based on historical observations and action sequences in a delayed environment, thereby achieving continuous and stable action output.

[0192] S3: Construction and Alignment of Value Network Latent Space.

[0193] The design employs a latent space encoding for the value network, mapping the real state to the latent space. Through feature alignment and action-value alignment mechanisms, it ensures that the value network output aligns with the policy network input in terms of the optimization objective. This latent space alignment mechanism specifically addresses input distribution differences caused by time delay, enhancing the stability of action-value estimation and thus providing a reliable learning signal for the policy network.

[0194] S4: Policy Training.

[0195] Data acquisition and network updates are performed cyclically in a delayed environment: the agent outputs actions based on the augmented state, obtains delayed feedback, and stores it in the replay buffer; the value network combines temporal difference, latent space alignment, and action value alignment loss optimization to ensure that the policy network obtains stable value feedback under augmented state input.

[0196] S5: Strategy Validation and Application.

[0197] The trained strategy was applied to teleoperation tasks of four robots—Ant, Half-Cheetah, Swimmer, and Walker2d—set up on a multi-joint dynamics with contact robot platform (MuJoCo). Performance and sample efficiency were validated through experiments with fixed latency and different pre-trained expert strategies. Experimental results ensure that the strategy can be stably executed under various latency scenarios, providing a reliable decision-making scheme for real-world teleoperation systems.

[0198] In some embodiments, the modeling extension under teleoperation delay conditions includes the following process.

[0199] In teleoperation tasks, the communication link between the operator and the actuator introduces a non-negligible delay. Common sources of delay include network bandwidth limitations, encoding / decoding delays of video and sensor signals, and lag in feedback from remote mechanical execution. Let the delay be d, and these delays affect the state information received by the operator at time t. s t-d Lagging behind the actual environmental conditions This violates the Markov property that reinforcement learning algorithms rely on, thus breaking the Markov property.

[0200] To accurately characterize the delay features, the system is modeled as a Constant Delayed Markov Decision Process (CDMDP), defined as a six-tuple. The details are as follows:

[0201] X Represents the delayed state space, Indicates time t The state of the time-delay state space, The non-delayed state space S and the past d It consists of a sequence of historical actions;

[0202] A Represents the action space; Indicates time t Actions in time, delayed state space X The state in , This is a historical sequence of actions, with subscripts indicating the time.

[0203] This represents the initial state distribution under a delayed environment. , This represents the initial state under a delayed environment. s 0 This represents the initial state in a non-delayed environment. This represents the initial state distribution under a non-delayed environment. Represents the Dirac function, This indicates the pre-interaction of a teleoperated robot with a non-delayed environment. d At a moment when a state is unobservable This represents a sequence of actions when the state is unobservable. This represents the sequence of actions in the action space when the state is unobservable; the initial state refers to... t =0;

[0204] P Let denote the transfer function, and we have:

[0205]

[0206] in, Indicates the state Next action Then transferred to state The probability, Indicates the state s t-d Next action Then transferred to state s t-d+1 The probability of;

[0207] in, Indicates the state Down t-d+j Momentary actions Indicates the state Down t-d+j Momentary actions , Indicates the state Down t Actions at any given moment;

[0208] R Represents the reward function; because in t Only observed at that time td The reward function is configured to reflect the state at time t. , Indicates the state Next action The rewards received Indicates the state s t-d Next action The rewards received;

[0209] Indicates the discount factor. ;

[0210] Wherein, the strategy is represented as , Represents the policy network.

[0211] Based on this extended modeling, the agent's policy is represented as: Augmented states contain historical action information and can theoretically recover some Markov properties. CDMDP, while preserving resolvability, introduces latency characteristics, providing a unified framework for latent space encoding in value networks and training in policy networks, enabling intelligent control and robust decision-making for complex teleoperation tasks.

[0212] In some embodiments, such as Figure 2 As shown, the policy network design includes the following process.

[0213] In a delayed environment for teleoperated robots, the policy network is responsible for generating control commands based on historical observations and action sequences to ensure the continuity and accuracy of remote operation tasks. To address communication and perception delays, this invention designs a policy network that generates control commands based on historical observations and action sequences in the delayed environment. As input, with s t-d For a state with a delay, Given a historical action sequence, the policy network outputs continuous and stable actions. And there are , Indicates that it is composed of network parameters The policy network is composed of neural networks. When the robot receives a new state from the delayed environment, it extracts the corresponding historical action sequence from the experience replay pool to form an augmented state. Then, the designed policy network processes the newly constructed augmented state to obtain the decision action.

[0214] The characteristics of the policy network structure are as follows: (1) Input processing: For high-dimensional augmented states, the policy network uses an encoding module that can process sequential information (such as a multilayer perceptron or a temporal convolutional network) to extract dynamic features of the environment and action dependencies, maintain the integrity of historical information, and reduce the impact of redundant interference on decision-making. (2) Action generation: The policy network outputs continuous or discrete control commands, corresponding to robot actuators (joint drives, motor control quantities, or thrust commands). (3) Delay adaptability: Through augmented state input, the policy network can remember the effects and delay features of past actions, thereby maintaining the continuity and robustness of actions in remote operation tasks, and is suitable for remote operation scenarios in complex industrial operations. (4) Collaboration with the value network: The policy network itself does not perform latent space encoding, but during the training process, it provides a stable action value evaluation signal through the latent space alignment of the value network, ensuring that the policy update direction is consistent with the long-term control objective.

[0215] The design goal of this section is to ensure that the policy network can effectively utilize historical information under high latency conditions to generate high-precision and stable control actions, providing reliable policy outputs for teleoperated robots to perform complex tasks.

[0216] In some embodiments, the construction and alignment of the latent space of the value network includes the following processes.

[0217] In the latency environment of teleoperated robots, the policy network uses augmented state... Generate Actions To improve the stability of action value assessment, this invention introduces a latent space encoding and alignment mechanism into the value network, which incorporates the true state... s t By mapping to the latent space and using feature alignment and action value alignment constraints, the policy network and value network achieve synergistic consistency in optimization objectives, thereby improving the control accuracy and robustness of remote operation tasks.

[0218] Regarding the latent space encoding of the value network, the value network is based on the current real state. s t and actions For input, via encoder Mapping to latent space representation :

[0219]

[0220] in, Belongs to the real number field, with a latent space dimension of . , This represents the parameters of the latent space encoder in the value network. The latent space encoding is used to extract the state. s t Value networks are characterized by decision-making-related features. and For input, output is for The assessed value. Latent space coding can effectively extract the core features relevant to decision-making from the state, suppress noise and redundancy introduced by delayed information, thereby ensuring the accuracy of value assessment.

[0221] Regarding latent space feature alignment, to address the discrepancy between the augmented state input and the true state input of the value network, this invention designs an augmented state encoder. The augmented states are mapped to a latent space with the same dimensions as the value network:

[0222]

[0223] By minimizing and The Euclidean distance between them is used to align latent space features. The latent space feature alignment process is as follows. Represented as:

[0224]

[0225] in, D This represents the experience replay pool. Indicates from D Medium sampling data , Indicates to from D Data obtained from mid-sampling The expected value is calculated. This mechanism ensures consistency in representation between the policy network input and the value network latent space, thereby mitigating feature bias caused by augmented state redundancy and latency.

[0226] Regarding action value alignment, to ensure that the actions output by the policy network are consistent with the action value evaluations of the value network in the latent space, this invention introduces an action value alignment mechanism:

[0227]

[0228] in, Represents a value network. Indicates that it is composed of network parameters The value network composed of neural networks Indicates from D Medium sampling data , Indicates to from D Data obtained from mid-sampling The alignment mechanism ensures that, even in a delayed environment, the actions generated by the policy network receive reliable signals for value evaluation, thus enhancing the stability and robustness of policy updates.

[0229] like Figure 3 As shown, the corresponding augmented and real states are processed by their respective encoders (encoder 1 and encoder 2) to obtain corresponding latent space vector representations (augmented state latent representation and real state latent representation). Then, the two latent space features and action values ​​are aligned to constrain the update of the value network. Through latent space encoding and alignment mechanisms, this invention can improve the stability of value estimation and policy update in delayed environments: First, latent space encoding extracts the core features most relevant to decision-making from the real state, reducing noise interference caused by communication and perception delays; second, feature alignment ensures that the feature representation obtained by the policy network based on the augmented state is consistent with the latent space representation of the value network, thereby mitigating training bias caused by differences in their input distributions; furthermore, action value alignment ensures the consistency and reliability of actions generated by the policy network in value evaluation, enhancing the robustness of the training process and remote operation execution; simultaneously, this latent space alignment framework has good adaptability to different delay scales, different task scenarios, and different robot platforms, and can support high-precision and high-reliability control for complex teleoperation tasks. Based on the above mechanism, the policy network still directly generates control actions using augmented states, while the value network relies on latent space encoding and alignment to provide stable action value references, thus providing a solid theoretical foundation for subsequent training, policy optimization, and actual deployment.

[0230] In some embodiments, a reinforcement learning training process specifically designed for delayed environments is proposed. This process uses latent space encoding, representation alignment, and behavioral consistency constraints as core mechanisms, combined with delay modeling and augmented state representation, to construct a complete policy learning system. This process, named LSE-DA (Latent-State Encoding with Delay-AwareAlignment), is applicable to applications with significant communication delays or perception lags, such as remote robotic arm assembly, unmanned swarm navigation, and underwater exploration robot control. The training process includes five stages: initialization, data acquisition, environmental interaction, network update, and iterative iteration.

[0231] In the initialization phase, the core network structure of the algorithm is built and key parameters are set:

[0232] (1) Create a policy network (based on augmented state input): used to receive augmented state vectors containing historical information and output control commands at the current moment;

[0233] (2) Create two value networks to evaluate the value of actions, and replicate the target value network to maintain training stability;

[0234] (3) Create two encoders to map augmented states and real states to latent space;

[0235] (4) Set the target network soft update ratio wait.

[0236] This stage ensures that the agent has a trainable basic network structure and provides the conditions for efficient data collection and processing in latency environments.

[0237] During the data acquisition phase, in the early stages of training... d At each time step, the agent interacts with the environment using a stochastic strategy:

[0238] Perform random actions and record the corresponding delayed states. s t-d ;

[0239] Store the action and delay state in a temporary buffer. B , for building Make preparations.

[0240] This phase ensures that sufficient historical data is accumulated in the early stages of training to support the generation of augmented states and the calculation of belief states.

[0241] When the number of training steps exceeds d Afterwards, the agent enters the formal delayed environment interaction phase:

[0242] (1) From the temporary buffer B Extract historical delayed states and action sequences to construct augmented states. ;

[0243] (2) The policy network is based on the augmented state Output Action Execute in a delayed environment and obtain a new delayed state. s t-d+1 With rewards ;

[0244] (3) Store the new state, action and reward in a temporary buffer. B ;

[0245] (4) When the number of steps exceeds 2 d Afterwards, from B Extract earlier historical data to generate augmented states for the two consecutive time points. and the corresponding real state ( s t, s t-1 ), and will transfer the complete data. Store in the experience replay pool D .

[0246] Through this mechanism, the agent can continuously collect high-quality data on augmented states and belief states in a delayed environment, providing reliable samples for policy optimization.

[0247] During the network update phase, from D A batch of data was sampled:

[0248]

[0249] in, Indicates the first k data, .

[0250] The network is updated cyclically:

[0251] (1) Compute the implicit representation:

[0252]

[0253] (2) Calculating the value of Bellman's objective :

[0254]

[0255] And calculate the value of Bellman loss:

[0256]

[0257] (3) Calculate the latent space feature alignment loss separately. and ;

[0258] (4) The total loss of the combined value network is obtained as follows:

[0259]

[0260] use Update value network parameters ；

[0261] (5) Calculate the implicit representation corresponding to the strategy side, and utilize Guidance strategy update;

[0262] (6) Target network soft update: with For parameters Update.

[0263] During the iterative phase, the process of collecting data through delayed environment interaction and sampling from the replay buffer to update the network is repeated until training ends.

[0264] After final training, the system can output stable and accurate control commands in real time in scenarios with significant latency, and can be seamlessly deployed for tasks such as remote robotic arm grasping, unmanned vehicle path planning, and underwater robot operation.

[0265] In some embodiments, a method for policy verification and application is provided based on the trained latency policy. The verification process includes latency experiment comparison and performance and computational efficiency evaluation.

[0266] Eight robots from the MuJoCo simulation platform were selected for experiments, including Ant, HalfCheetah, Swimmer, and Walker2d. Specifically, Ant fully characterizes the kinematics and contact information of the torso and four legs in 111 dimensions, achieving high-speed straight-line running while maintaining dynamic stability through 8-dimensional hip and knee torques; HalfCheetah compresses the torso and six-joint angular velocities into 17 dimensions, driven by 6-dimensional hip and knee torques, aiming to maximize forward speed; Swimmer uses multi-segment joint angular velocities as states and multi-path joint torques as control, utilizing a wave propulsion mechanism to obtain maximum forward speed; Walker2d generates 6-dimensional hip and knee torques based on 17-dimensional torso and leg kinematic information, achieving a stable and efficient bipedal gait.

[0267] To evaluate the effectiveness of LSE-DA, this embodiment of the invention compares it with several state-of-the-art (SOTA) algorithms specifically designed for delayed reinforcement learning on four robots on the MuJoCo platform, including Belief-Projection-Based Q-learning (BPQL), Auxiliary-Delayed Reinforcement Learning (ADRL), Variational Delayed Policy Optimization (VDPO), and Behavior Cloning from Undelayed Free Policy (BC(pure)). The experimental results reflect that the LSE-DA method of this invention exhibits significant advantages under different delay conditions and different robot tasks: (1) Optimal performance: Low latency in all tasks ( d =4) The highest score was obtained in the setting, indicating that policy augmentation can effectively restore information dependencies that are disrupted by delay; (2) Strong delay robustness: in high latency ( d=8) The performance degradation is the smallest, which shows that the latent space alignment still maintains stable value assessment when facing severe observation lag; (3) The stability is the strongest: the standard deviation across tasks and across delays is the lowest, indicating that the latent space encoding and value consistency mechanism of the present invention has excellent training and execution stability; (4) It is completely dependent on delayed interaction data: it does not require demonstration or experts, and still significantly outperforms behavior cloning. Therefore, the present invention can form a more stable, more robust and more practical reinforcement learning strategy in remotely operated robot control under different delays and different dynamic conditions.

[0268] At a fixed delay d =4 The training time of four robots in the MuJoCo simulation platform was systematically measured, where BC (pure) included the pre-training time for acquiring the zero-delay policy. The following conclusions can be drawn: (1) The training time of LSE-DA is slightly longer than that of BPQL, but its performance is significantly better than BPQL and other baselines; (2) BC (pure) has the simplest network structure and the shortest training time, but its performance is the worst based on the results of the previous experiments; (3) ADRL and VDPO have longer training times due to auxiliary tasks or two-step iterative optimization processes. The overall results show that LSE-DA achieves a balance between efficiency and control effect by maintaining high performance and robustness while having a moderate training cost.

[0269] In summary, the proposed method and system for teleoperated robot motion control based on latent space alignment in this invention fundamentally improves decision stability and value assessment accuracy in time-delay environments by constructing an augmented state policy network and a latent space value network, and aligning the augmented state and the corresponding real state at the same time step in terms of latent space representation and motion value.

[0270] The main technical improvements brought about by this invention include: (1) Augmented state policy network: By constructing an augmented state containing historical observations and historical actions, the policy network maintains near-Markov property in a delayed environment, providing complete and continuous state information for agent decision-making. (2) Latent space value encoding and alignment mechanism: Latent space encoding is used to extract features from the real state, and the augmented state and the real state at the same time step are aligned in the latent space and action value to achieve stability and reliability of value assessment. (3) Policy-value network consistency constraint: Latent space and value consistency constraints are introduced in policy training to eliminate the semantic differences between the augmented state of the policy network and the real state input of the value network, ensuring the stability of the training process and the reliability of the gradient. (4) Dimension compression and computation optimization mechanism: By compressing high-dimensional historical information through latent space encoding, the state dimension expansion is suppressed, the computational overhead is reduced, and the scalability and real-time control capability of the multi-degree-of-freedom robot system in a delayed environment are improved. (5) Sample efficiency improvement and convergence acceleration strategy: By combining augmented state and latent space value alignment mechanism, each training sample can be efficiently utilized in policy and value learning, thereby improving training convergence speed and enhancing task execution performance under delayed conditions.

[0271] Compared with the prior art, the present invention has the following beneficial effects: (1) Improves the stability of value assessment under delayed conditions. By encoding the real state in the latent space and estimating the action value in this space, the noise interference introduced by communication and sensing lag is significantly reduced, providing a more stable and reliable value signal for policy updates. (2) Eliminates the input inconsistency problem between augmented state and real state. The present invention aligns the augmented state and the real state at the same time step, enabling the policy side and the value side to obtain semantically consistent supervision information, effectively avoiding training instability and policy drift caused by input distribution mismatch. (3) Suppresses dimensional expansion caused by state augmentation. By replacing the direct stacking of high-dimensional historical states with latent space compression representation, the present invention reduces the model size and computational overhead, and improves the scalability and real-time performance of the method in multi-degree-of-freedom robot systems. (4) Improves sample efficiency and convergence speed. Consistent latent space and value alignment enhance the effective contribution of each sample to policy and value learning, improves data utilization, shortens training convergence time, and obtains higher control performance under various delay conditions.

[0272] Please note that the technical features of the above embodiments can be combined arbitrarily. For the sake of brevity, not all possible combinations of the technical features in the above embodiments have been described. However, as long as the combination of these technical features does not contradict each other, it should be considered within the scope of this specification. The above embodiments only illustrate several implementations of the present invention, and their descriptions are relatively specific and detailed, but they should not be construed as limiting the scope of the invention patent. It should be pointed out that for those skilled in the art, several modifications and improvements can be made without departing from the concept of the present invention, and these all fall within the protection scope of the present invention. Therefore, the protection scope of the present invention should be determined by the appended claims.

Claims

1. A method for motion control of a teleoperated robot based on latent space alignment, characterized in that, The method includes: S1. Perform Markov modeling for the delayed environment; In this study, Markov modeling is performed on the actual operating environment of the teleoperated robot, which has delays in communication and perception, to determine the time series dependency between the augmented state with delays and the real state without delays, and to further quantify the impact of the delays on the teleoperated robot's action execution and state observation. S2. Design a strategy network based on the aforementioned latency environment; The policy network takes the augmented state as input, and the teleoperated robot generates control commands based on historical observations and action sequences in the delayed environment, which are the outputs of the policy network. The control commands refer to continuous and stable actions to be executed. S3. Construction and alignment of the implicit space of the execution value network; Among them, a latent space encoding of the value network is designed to map the real state to the latent space. Through latent space feature alignment and action value alignment, the value network and the policy network are made to work together in terms of optimization objectives. The alignment mechanism is designed to address the input distribution differences caused by the delay. S4. Train the policy using policy networks and value networks; In the delayed environment, the policy network outputs continuous and stable actions in the augmented state to obtain feedback after the delay. The value network combines the loss of temporal difference, latent space feature alignment and action value alignment to optimize and train the policy. S5. Validate and apply the trained strategy; The trained strategy was applied to a teleoperated robot task set on a multi-joint contact dynamics robot platform. Experiments were conducted using fixed delays and different strategies to verify that the trained strategy could achieve motion control of the teleoperated robot under various delay scenarios.

2. The teleoperated robot motion control method based on latent space alignment according to claim 1, characterized in that, In S1, during the task performed by the teleoperated robot, there is a delay in the communication link between the operator and the execution end. d This allows the operating terminal to be constantly t The received state has a delay. s t-d Lagging behind a state without delay s t The decision process in Markov modeling under delayed conditions consists of six tuples. Characterization is performed; where: X Represents the delayed state space, Indicates time t The state of the time-delay state space, The non-delayed state space S and the past d It consists of a sequence of historical actions; A Represents the action space; Indicates time t Actions in time, delayed state space X The state in , This is a historical sequence of actions, with subscripts indicating the time. This represents the initial state distribution under a delayed environment. , This represents the initial state under a delayed environment. s 0 This represents the initial state in a non-delayed environment. This represents the initial state distribution under a non-delayed environment. Represents the Dirac function, This indicates the pre-interaction of a teleoperated robot with a non-delayed environment. d At a moment when a state is unobservable This represents a sequence of actions when the state is unobservable. This represents the sequence of actions in the action space when the state is unobservable; the initial state refers to... t =0; P Let denote the transfer function, and we have: in, Indicates the state Next action Then transition to state The probability, Indicates the state s t-d Next action Then transition to state s t-d+1 The probability of; in, Indicates the state Down t-d+j Momentary actions Indicates the state Down t-d+j Momentary actions , Indicates the state Down t Actions at any given moment; R Represents the reward function; because in t Only observed at that time td The reward function is configured to reflect the state at time t. , Indicates the state Next action The rewards received Indicates the state s t-d Next action The rewards received; Indicates the discount factor. ; Wherein, the strategy is represented as , Represents the policy network.

3. The teleoperated robot motion control method based on latent space alignment according to claim 2, characterized in that, In S2, under the delayed environment, the policy network generates control commands based on historical observations and action sequences. As input, with s t-d For a state with a delay, Given a historical action sequence, the policy network outputs continuous and stable actions. And there are , Indicates that it is composed of network parameters The policy network is composed of neural networks; where: The teleoperated robot receives the state returned by the delayed environment, extracts the corresponding historical action sequence from the experience replay pool to construct an augmented state, and the policy network processes the constructed augmented state to obtain control instructions, which refer to continuous and stable actions to be executed.

4. The teleoperated robot motion control method based on latent space alignment according to claim 3, characterized in that, In S3, a latent space encoding and alignment mechanism is introduced into the value network to map the real state to the latent space. Through latent space feature alignment and action-value alignment, the value network and the policy network achieve synergy in their optimization objectives. Specifically: In the process of value network latent space encoding, the value network latent space encoder State Mapping to the latent space, we get : in, Belongs to the real number field, with a latent space dimension of . , This represents the parameters of the latent space encoder in the value network. The latent space encoding is used to extract the state. s t In terms of characteristics related to decision-making, value networks are based on and For input, output is for The assessed value.

5. The teleoperated robot motion control method based on latent space alignment according to claim 4, characterized in that, In S3, latent space feature alignment includes: Using augmented state encoders State Mapping to the latent space, we get : By minimizing and The Euclidean distance between them is used to align latent space features. The latent space feature alignment process is as follows. Represented as: in, D This represents the experience replay pool. Indicates from D Medium sampling data , Indicates to from D Data obtained from mid-sampling Seeking expectations.

6. The teleoperated robot motion control method based on latent space alignment according to claim 5, characterized in that, In S3, the action value alignment process Represented as: in, Represents a value network. Indicates that it is composed of network parameters The value network composed of neural networks Indicates from D Medium sampling data , Indicates to from D Data obtained from mid-sampling Seeking expectations.

7. The teleoperated robot motion control method based on latent space alignment according to claim 6, characterized in that, In S4, the training process includes five stages: initialization, data acquisition, delayed environment interaction, network update, and iterative loop; among which: During the initialization phase: Configure the policy network to output the control commands at the current moment as actions; Configure a value network to evaluate the value of actions and replicate the generated target value network to maintain training stability; Create an encoder to map the augmented state and the real state to the latent space; Set network update ratio ; During the data acquisition phase: Before the initial training period d Each time step interacts with the delayed environment using a random strategy: Perform random actions and record the corresponding delay states; Store the action and delay state in a temporary buffer. B , used to construct augmented states.

8. The teleoperated robot motion control method based on latent space alignment according to claim 7, characterized in that, In S4, when the number of training steps exceeds d Then, the delayed environment interaction phase begins: from B Extract s t-d and Thus constructing , ; Policy network according to Output Execute in a delayed environment and obtain s t-d+1 and ; Will s t-d+1 , , deposit B ; When the number of training steps exceeds 2 d Afterwards, from B Extract relevant data to construct as well as( s t, s t-1 ), and will Stored as a piece of data D .

9. The teleoperated robot motion control method based on latent space alignment according to claim 8, characterized in that, In S4: During the network update phase: from D Medium sampling N The data is represented as: in, Indicates the first k data, ; calculate: Computational Value Bellman Objective : Represents a value network. Indicates that it is composed of network parameters A value network composed of neural networks will be formed by networks with parameters. The value network constructed from neural networks serves as the target value network. Indicates that The action with the greatest value; Computational value Bellman loss : calculate and ; Calculate total loss : in, and This is the adjustment coefficient; by Update parameters , This represents the gradient descent method; use Guidance strategy update; by For parameters Update; During the iterative loop phase: Repeat the process of delaying the interaction between the environment and the network until training ends.

10. A teleoperated robot motion control system based on latent space alignment, characterized in that, The system includes a processing unit configured to perform: Markov modeling is performed for delayed environments; In this study, Markov modeling is performed on the actual operating environment of the teleoperated robot, which has delays in communication and perception, to determine the time series dependency between the augmented state with delays and the real state without delays, and to further quantify the impact of the delays on the teleoperated robot's action execution and state observation. Design a strategy network based on the aforementioned latency environment; The policy network takes the augmented state as input, and the teleoperated robot generates control commands based on historical observations and action sequences in the delayed environment, which are the outputs of the policy network. The control commands refer to continuous and stable actions to be executed. Constructing and aligning the implicit space of the execution value network; Among them, a latent space encoding of the value network is designed to map the real state to the latent space. Through latent space feature alignment and action value alignment, the value network and the policy network are made to work together in terms of optimization objectives. The alignment mechanism is designed to address the input distribution differences caused by the delay. The policy is trained using a policy network and a value network; In the delayed environment, the policy network outputs continuous and stable actions in the augmented state to obtain feedback after the delay. The value network combines the loss of temporal difference, latent space feature alignment and action value alignment to optimize and train the policy. Validate and apply the trained strategy; The trained strategy was applied to a teleoperated robot task set on a multi-joint contact dynamics robot platform. Experiments were conducted using fixed delays and different strategies to verify that the trained strategy could achieve motion control of the teleoperated robot under various delay scenarios.

Images