Operation control system for a delay environment

By employing time-sequential asynchronous transmission of control commands and status data in the robot teleoperation system, the problem of operational feedback lag under delayed environments is solved, enabling real-time response and precise synchronization of operations, thereby improving the efficiency and safety of teleoperation.

CN123103728APending Publication Date: 2026-11-17PNDBOTICS (NINGBO) CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202611279567.6
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2026-08-21
Publication Date
2026-11-17

AI Technical Summary

Technical Problem

In robot teleoperation systems, data transmission delays cause a significant lag between the feedback screen observed by the operator and the actual operation, affecting operational efficiency and task completion quality. This is especially true in delicate operation tasks, where situations such as overshooting, accidental contact with the workpiece, or repeated adjustments are likely to occur.

Method used

Control commands, status data, and 3D environment data are transmitted asynchronously to ensure timely response and feedback of control commands and status data. Independent of the transmission of 3D environment data, an operation feedback screen synchronized with the actual status of the remote execution device is generated.

Benefits of technology

Improving operational stability and responsiveness in delayed environments significantly enhances the efficiency and safety of teleoperation, while reducing the risk of misoperation.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN123103728A_ABST
    Figure CN123103728A_ABST
Patent Text Reader

Abstract

This disclosure relates to an operation control system for delayed environments. A local device generates control commands in response to user operations and sends them to a remote execution device. The remote execution device specifies the operation corresponding to the control command, collects its own status data and transmits it back to the local device, and collects and transmits the surrounding 3D environment data to the local device. The control commands, status data, and 3D environment data are transmitted asynchronously. The local device also updates locally stored 3D scene data based on the 3D environment data, generates a real state body representing the actual state of the remote execution device based on the status data, and generates an actual operation feedback screen based on the real state body and the 3D scene data of the local device at the current moment. Through this system, the real-time transmission of control commands and status data is not affected by the blocking of 3D environment data transmission, thereby improving the smoothness of operation and task execution efficiency in delayed environments.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This disclosure relates to the field of robot control, and more specifically, to an operation control system for delayed environments. Background Technology

[0002] Robot teleoperation systems have wide applications in industrial manufacturing, medical surgery, deep-sea exploration, and space exploration. In such systems, the operator sends motion control commands to a remote robot via a local control device. After executing these commands, the remote robot transmits its real-world data, such as joint angles and end-effector pose, back to the local device. Simultaneously, the remote robot also needs to transmit 3D environmental data (such as point clouds and depth images) of its surroundings to the local device for rendering and display.

[0003] However, in related technologies, the transmission of the aforementioned data has a significant delay, which may result in a noticeable lag between the feedback screen observed by the operator and the actual operation. This lag can prevent the operator from obtaining timely and accurate operational feedback during delicate operations such as gripping, insertion, alignment, and screwing, and can easily lead to overshooting, accidental contact with the workpiece, or repeated adjustments, which seriously affect operational efficiency and task completion quality. Summary of the Invention

[0004] To address the aforementioned technical issues, this disclosure provides an operation control system for delayed environments. By transmitting control commands, status data, and 3D environmental data asynchronously, the transmission of 3D environmental data does not block the issuance of control commands or the feedback of status data. This ensures timely response of the remote execution device to control commands and real-time status feedback, keeping the actual operation feedback screen generated by the local device synchronized with the actual status of the remote execution device. This improves operational stability and responsiveness in delayed environments.

[0005] To achieve the above objectives, this disclosure provides an operation control system for delayed environments, including a local terminal device and a remote execution device, wherein the local terminal device and the remote execution device are communicatively connected. The local terminal device is used to obtain operation data in response to the user's operation, generate control commands based on the operation data, and send the control commands to the remote execution device; The remote execution device is used to respond to the control command and execute the operation corresponding to the control command, collect its own status data when executing the operation and send it back to the local terminal device; The remote execution device is also used to collect three-dimensional environmental data of its own location and transmit it to the local terminal device, wherein the control commands, the status data and the three-dimensional environmental data are transmitted asynchronously over the communication connection. The local device is also used to update the three-dimensional scene data stored in the local device according to the three-dimensional environment data, generate a real state body according to the state data, and generate an actual operation feedback screen according to the real state body and the three-dimensional scene data stored in the local device at the current moment. The real state body is a three-dimensional virtual object of the remote execution device, and the real state body represents the actual state reached by the remote execution device in response to the control command.

[0006] Optionally, the local terminal device is further configured to acquire at least one of observation view data and operation command data, wherein the observation view data includes the pose data of the wearable device in the local terminal device; the operation command data includes at least one of the pose data of the handheld controller, hand tracking data, button input data, joystick input data, and touch input data; and the observation view data and / or the operation command data are parsed to obtain the operation data.

[0007] Optionally, the local terminal device is further configured to, when the operation data includes the current observation view obtained by parsing the observation view data, render the three-dimensional scene data and the real state volume according to the current observation view to generate an actual operation feedback screen corresponding to the current observation view.

[0008] Optionally, the communication connection between the local terminal device and the remote execution device is configured as an asynchronous communication link; The asynchronous communication link includes: The control command uplink channel is used to transmit the control commands sent by the local terminal device to the remote execution device; The status data downlink channel is used to transmit the status data returned by the remote execution device to the local device; The three-dimensional scene data downlink channel is used to transmit the three-dimensional environment data returned by the remote execution device to the local device.

[0009] Optionally, the remote execution device is further configured to collect its own status data in real time during the execution of the operation and transmit it back to the local device; and It is used to collect three-dimensional scene data in real time or at preset intervals, and to determine whether there are any changed areas in the three-dimensional scene where the remote execution device is located based on the collected three-dimensional scene data and historical scene data. If so, the 3D scene data of the changed area is sent back to the local terminal device to update the 3D scene cached by the local terminal device; If it does not exist, the currently collected scene data will not be returned.

[0010] Optionally, the control commands and the status data are transmitted with higher priority on the asynchronous communication link than the three-dimensional environment data. The three-dimensional environment data includes static environment data and dynamic environment data, with the transmission priority of the dynamic environment data being higher than that of the static environment data.

[0011] Optionally, the local terminal device is further configured to determine the credibility of each of the multiple regions based on at least one of the timestamp, data source, update frequency, dynamic and static attributes, and spatial change conditions in the metadata associated with each of the multiple regions in the 3D scene data. Based on the credibility of each region, the 3D environment data returned by the remote execution device is fused with the 3D scene data stored by the local device to update the 3D scene data stored by the local device.

[0012] Optionally, the local device is further configured to generate a command state body according to the control command, and generate and display a reference operation feedback screen based on the three-dimensional scene data stored at the current time and the command state body. The command state body is a three-dimensional virtual object of the remote execution device, and the command state body is used to characterize the target state to be achieved by the remote execution device.

[0013] Optionally, the local terminal device is further configured to generate deviation information based on the command status body and the actual status body, and to generate a target screen based on the deviation information, the command status body and the actual status body, wherein the target screen includes an actual operation feedback screen, a reference operation feedback screen and the deviation information.

[0014] Optionally, the remote execution device includes a robot, which includes at least one of a single-arm robot, a dual-arm robot, a fixed-base robot, a mobile robot, and a humanoid robot; the local device includes a local display device and a local input device, which includes at least one of a VR headset, AR glasses, a desktop computer monitor, a tablet computer, and a mobile phone, and the local input device includes at least one of a handheld controller, a hand tracking device, a touch device, a mouse, a keyboard, and a joystick.

[0015] The operation control system for delayed environments provided in this disclosure includes a local terminal device and a remote execution device connected via a communication link. The local terminal device is used to obtain operation data in response to user operations and generate control commands accordingly, which are then sent to the remote execution device. The remote execution device is used to execute corresponding operations in response to the control commands and collect its own status data, which is then transmitted back to the local terminal device. Furthermore, the remote execution device is also used to collect its own three-dimensional environment data and transmit it to the local terminal device. The control commands, the status data, and the three-dimensional environment data are transmitted asynchronously over the communication link. The local terminal device is also used to update its locally stored three-dimensional environment data based on the three-dimensional environment data. The system generates a real state body representing the actual state of the remote execution device based on the state data, and generates an actual operation feedback screen based on the real state body and the 3D scene data stored by the local device at the current moment. Through the above-mentioned time-series asynchronous transmission mechanism and the real-time rendering feedback mechanism based on the local scene data at the current moment, the issuance of control commands and the return of state data are not affected by the data transmission blockage in the 3D environment, and the generation of the actual operation feedback screen does not depend on the real-time arrival of the 3D environment data. Thus, in a delayed environment, the responsiveness of operation commands, the accurate synchronization of state feedback, and the smoothness and responsiveness of the operation screen are significantly improved, effectively ensuring the execution efficiency and safety of precise remote operation tasks.

[0016] Other features and advantages of this disclosure will be described in detail in the following detailed description section. Attached Figure Description

[0017] The accompanying drawings are provided to further illustrate the present disclosure and form part of the specification. They are used together with the following detailed description to explain the present disclosure, but do not constitute a limitation thereof. In the drawings: Figure 1 This is a schematic diagram of an operation control system for a delayed environment, illustrated according to an exemplary embodiment of the present disclosure.

[0018] Figure 2 This is another schematic diagram of an operation control system for a delayed environment, illustrated according to an exemplary embodiment of the present disclosure.

[0019] Figure 3 This is a schematic diagram of an operation feedback screen shown according to an exemplary embodiment of the present disclosure.

[0020] Figure 4 This is a schematic diagram illustrating the rendering stage of an operation feedback screen according to an exemplary embodiment of the present disclosure. Detailed Implementation

[0021] The specific embodiments of this disclosure will be described in detail below with reference to the accompanying drawings. It should be understood that the specific embodiments described herein are for illustration and explanation only and are not intended to limit this disclosure.

[0022] Please see Figure 1 , Figure 1 A connection block diagram of an operation control system for a delayed environment is shown. The operation control system for a delayed environment includes a local terminal device 10 and a remote execution device 20, wherein the local terminal device 10 and the remote execution device 20 communicate with each other through a communication link.

[0023] The local terminal device 10 is used to acquire the operation data of the local terminal device 10, generate control commands based on the operation data, and send the control commands to the remote execution device 20.

[0024] The remote execution device 20 is used to execute the operation corresponding to the control command in response to the control command, collect its own status data during the execution of the operation, and transmit it back to the local terminal device 10. The remote execution device 20 is also used to collect its own three-dimensional environment data and transmit it to the local terminal device 10. The control command, the status data, and the three-dimensional environment data are transmitted asynchronously over the communication connection.

[0025] The local device 10 updates the 3D scene data stored in the local device according to the 3D environment data, generates a real state body according to the state data, and generates an actual operation feedback screen according to the real state body and the 3D scene data stored in the local device 10 at the current moment. The real state body is a 3D virtual object of the remote execution device, and the real state body represents the actual state reached by the remote execution device in response to the control command.

[0026] By employing an operation control system designed for latency-prone environments, control commands, status data, and 3D environmental data are transmitted asynchronously over the communication connection. This allows the three types of data to be transmitted independently and in parallel, without waiting for each other. Consequently, the upward transmission of control commands is unaffected by the blockage in 3D environmental data transmission, enabling them to be sent instantly to remote execution devices to ensure timely response to operational intentions. Simultaneously, the feedback of status data is independent of 3D environmental data. Even if 3D environmental data is delayed or lost due to bandwidth constraints or network fluctuations, critical status information from the remote execution device can still be transmitted back to the local device in real time. This allows the local device to continuously receive accurate execution status feedback, and then, based on this... The status data generation represents the actual status of the remote execution device and combines it with the currently stored 3D scene data to generate an actual operation feedback screen. Since the generation of this screen only relies on the returned status data and the local scene data without waiting for the real-time arrival of the 3D environment data, even if the 3D environment data has not been updated due to network latency, the local device can still continuously render an accurate operation feedback screen based on its stored 3D scene data. Thus, it provides the operator with immediate, smooth and accurate operation feedback in a latency environment, significantly improving the responsiveness of teleoperation and task execution efficiency, while effectively reducing the risk of misoperation caused by feedback lag.

[0027] The local terminal device 10 refers to a device that displays operation feedback screens to the operator and receives user input to obtain operation data. The local terminal device 10 may include a local display device and a local input device. The local display device includes at least one of a head-up display, AR glasses, a desktop computer monitor, a tablet computer, and a mobile phone. The local input device includes at least one of a handheld controller, a hand tracking device, a touch device, a mouse, a keyboard, and a joystick. It should be understood that the local input device and the local display device can be integrated, such as a mobile phone, tablet computer, or head-up display.

[0028] The remote execution device 20 includes at least one of a robot, an intelligent mechanical workstation, and a remote operation tooling. The robot can be a single-arm robot, a dual-arm robot, a fixed-base robot, a mobile robot, or a humanoid robot.

[0029] The operation data of the local terminal device 10 refers to the data collected and parsed by the operator through operations performed on local input devices. Specifically, the operator can perform operations through local input devices such as VR headsets, controllers, hand tracking devices, buttons, joysticks, and haptic devices. The local terminal device 10 collects the original interaction signals of such operations and parses them to obtain operation data.

[0030] Optionally, the operation data of the local terminal device 10 may include at least one of viewing angle data and operation command data. The viewing angle data includes the pose data of the wearable device of the local terminal device 10 (e.g., the pose data of a VR headset), and the operation command data includes at least one of button input data, joystick input data, touch input data, pose data of a handheld controller, and hand tracking data. The operation data is obtained by parsing the viewing angle data and / or the operation command data.

[0031] Please refer to the following: Figure 2 , Figure 2 The diagram shows a local device 10 including a VR headset and controllers, and a remote execution device 20 as a single-arm robot with a gripper. Figure 2 In the scenario shown, when the operator wears a VR headset and holds a controller, the local device 10 obtains the VR headset's pose data as viewing angle data through the VR headset's positioning system, and obtains the controller's pose data and the input states of buttons and joysticks as operation command data through the controller. The local device 10 parses the above operation command data to obtain operation data including the operator's current viewing angle, hand pose, button trigger state, joystick deflection, etc. The pose data can include position information (e.g., three-dimensional spatial coordinates) and attitude information (e.g., Euler angles or quaternions).

[0032] For example, the pose data of a VR headset may include the VR headset's position (x, y, z) and orientation (roll, pitch, yaw) in the world coordinate system. The pose data of the controllers may include the position and orientation of the controllers in the world coordinate system. Hand tracking data may include the rotation angles of each finger joint relative to its parent joint in the local coordinate system (e.g., expressed as Euler angles or quaternions), and the three-dimensional coordinates of all joints in the hand's local coordinate system.

[0033] The world coordinate system is an absolute reference coordinate system in three-dimensional space, used to provide a unified position and orientation reference for all objects in space. The local coordinate system is a coordinate system attached to a specific object (such as a remote actuator, robotic arm end effector, or hand), moving and rotating synchronously with that object. Its origin represents the object's spatial coordinates in the world coordinate system, and its orientation represents the object's three-dimensional deflection relative to the world coordinate system. The hand local coordinate system is a special type of local coordinate system, typically with the wrist or base of the hand as its origin, used to uniformly describe the relative positions of all joints in the entire hand.

[0034] In kinematics, finger joints typically form a tree-like hierarchical structure (e.g., wrist → metacarpophalangeal joints → proximal interphalangeal joints → distal interphalangeal joints). The rotation of each joint is a local rotation relative to its parent joint (i.e., the joint at the next higher level). This method allows for the successive propagation and eventual determination of the pose of each joint in the world coordinate system.

[0035] Control commands are instructions that the remote execution device 20 can execute. Control commands may include joint position commands, joint velocity commands, end effector pose commands, gripper opening and closing commands, force control commands, trajectory segments, task-level control commands, etc.

[0036] When the local device 10 generates control commands based on the operation data, it can generate control commands through motion planning algorithms and / or inverse kinematics algorithms based on the operation data, the morphological data and kinematic model of the remote execution device 20.

[0037] The morphological data includes at least one of the degrees of freedom of each joint of the remote actuator 20, link length, and joint limits. The kinematic model includes a forward kinematic model and an inverse kinematic model. For example, when the operational data indicates a target pose of the end effector, the local device 10 maps the target pose to target angles of each joint of the remote actuator 20 using the inverse kinematics algorithm, and generates control commands containing the target angles of each joint based on the forward kinematic model and the morphological data. When the operational data indicates a target joint state, the local device 10 generates control commands containing at least one of joint angles, joint velocities, or joint torques based on the morphological data and the kinematic model. The motion planning algorithm is used to plan the motion trajectory from the current pose to the target pose, so that the motion of the remote actuator 20 is smooth and natural. The generated control commands include trajectory segments, which contain a series of intermediate states arranged in a time sequence.

[0038] When the remote execution device 20 includes a robot and a robot control module, the robot control module is used to receive control commands sent by the local terminal device 10 and drive the robot to perform corresponding actions according to the control commands. Depending on the type of control command, the robot control module uses a corresponding control method to drive the robot to perform actions, and the control method includes at least one of the following: Joint position control: The robot control module takes the target angle of each joint in the control command as input, and drives the joint motors to rotate to the target angle through the joint servo driver, so that the robot end effector reaches the target position. This control method is suitable for operation scenarios that require high end effector position accuracy and do not require consideration of speed planning during movement.

[0039] Joint speed control: The robot control module takes the target angular velocity of each joint in the control command as input and drives each joint to move continuously at the target angular velocity. This control method is suitable for scenarios that require smooth tracking of motion trajectories, such as continuous arc welding or spraying operations.

[0040] End-effector pose control: The robot control module takes the target end-effector position and pose from the control command as input, maps the end-effector pose to the target angles of each joint using the robot's built-in inverse kinematics solver, and then drives the robot's movement through joint servo drivers to bring the robot's end-effector to the target pose. This control method is suitable for master-slave teleoperation scenarios where the operator directly controls the robot's end-effector movement.

[0041] Inverse kinematics control: Based on the end-effector target pose in the control command and the robot's kinematic model, the robot control module uses an inverse kinematics algorithm to solve the target angle of each joint in real time and drives each joint to move to that target angle. This control method is essentially the same as end-effector pose control, the difference being that the inverse kinematics solution process is executed on the robot side, which can more fully utilize the robot's own kinematic constraints and real-time feedback for online optimization.

[0042] Force-position hybrid control: The robot control module coordinates the robot's position and contact force according to the desired pose and contact force in the control commands. That is, it controls the position in some degrees of freedom and the force in others, achieving compliant operation. This control method is suitable for scenarios that require interaction with the environment, such as precision assembly and polishing.

[0043] Gripper control: The robot control module drives the robot's end effector gripper or hand to perform opening, closing, or holding actions based on the target gripper opening degree in the control command. This control method is suitable for operations involving grasping and releasing target objects.

[0044] Full-body control: Based on multiple task objectives (such as end-effector pose, torso posture, and center of mass position) in the control commands, the robot control module simultaneously coordinates the movement of all joints of the robot using whole-body kinematics or dynamics optimization algorithms to generate joint drive commands that satisfy multi-task constraints. This control method is suitable for complex work scenarios requiring coordinated movement of multiple limbs, such as bipedal / quadrupedal robots and humanoid robots.

[0045] Task-level control: The robot control module parses high-level task instructions (such as "move to point A and grab the object") in the control commands into a series of low-level action sequences, and automatically plans the path, generates the trajectory, and coordinates the movement of each joint to complete the high-level task. This control method is suitable for interactive scenarios where the operator issues high-level instructions via voice, text, or simple gestures.

[0046] Optionally, during the execution of control commands, the remote execution device 20 can collect its real-time status data and transmit it back to the local device 10 via a communication link. The status data is obtained by the remote execution device 20 performing the operation corresponding to the control command, and the status data may include robot joint angles, joint velocities, actual end-effector pose, actual gripper opening / closing amount, motor current, force or torque data, contact status, controller execution status, etc.

[0047] For example, when the control command sent by the local device 10 is an end-effector pose command, the robot control module parses the end-effector target position and orientation, maps the target pose to the target angles of each joint using an inverse kinematics solver, and simultaneously drives the joint motors to rotate, so that the robot end-effector moves smoothly to the target pose. When the control command sent by the local device 10 is a gripper opening / closing command, the robot control module drives the gripper motors to perform the corresponding opening or closing actions to complete the grasping or releasing of the target object. When the control command sent by the local device 10 is a force-position hybrid control command, the robot control module drives the robot end-effector to approach the target position while monitoring the end-effector force sensor data in real time. When the contact force reaches the desired value, the position approach is stopped and the contact force is kept constant to achieve compliant contact operation.

[0048] In addition, during the process of the robot executing the above control commands, the robot's sensors collect its own state data in real time (such as the actual angle of each joint, actual speed, actual end pose, actual opening and closing amount of the gripper, motor current, force / torque data, contact state, etc.), and transmit it back to the local terminal device 10 through the communication link, so that the local terminal device 10 can generate a real state body, calculate deviation information, and trigger safety constraints.

[0049] Optionally, the local device 10 can receive status data transmitted back by the remote execution device 20, the status data including its own status data collected by the remote execution device 20 when performing the operation corresponding to the control command; generate a real state body based on the status data and a three-dimensional model of the remote execution device 20, the real state body being a virtual model of the remote execution device 20, the real state body representing the actual state reached by the remote execution device 20 in response to the control command; and generate an actual operation feedback screen based on the real state body.

[0050] Specifically, after receiving the state data, when the local device 10 generates a real state body based on the state data and the three-dimensional model of the remote execution device 20, it first parses the state data and extracts the actual state parameters; then, based on the type of the actual state parameters, it determines the actual pose of each moving part of the remote execution device 20 in the three-dimensional model, and drives the three-dimensional model to present the actual state through forward kinematics calculation or direct assignment, generating a virtual model corresponding to the remote execution device 20 as the real state body.

[0051] In other words, the process of generating the real state body can specifically be to map the numerical parameters (such as joint angles, end poses, etc.) that characterize the actual state of the device in the state data to each moving part (joint / kinematic chain) of the three-dimensional model, drive each part of the model to move to the actual state through positive kinematics, and finally generate a visualized virtual model in the three-dimensional operating space to characterize the actual state reached by the remote execution device 20 after executing the control command.

[0052] The real state entity is a virtual object displayed in the three-dimensional operating space. Its specific form corresponds to the actual form of the remote execution device 20. The pose of the real state entity is determined based on the state data transmitted back by the remote execution device 20. The real state entity is used to characterize the actual state reached by the remote execution device 20 during and / or after executing the control command. The actual state includes at least one of the following: actual pose information (e.g., actual position and / or actual posture), actual motion information (e.g., at least one of actual velocity, actual acceleration, and actual trajectory), actual force information (e.g., at least one of actual torque and actual contact force), actual actuator state information (actual gripper opening degree), and a timestamp corresponding to the actual state.

[0053] The actual pose information in the real state volume includes actual position and / or actual orientation. Actual position refers to the actual coordinates (e.g., x, y, z values) of the end effector 20 in three-dimensional space, and actual orientation refers to the actual orientation (e.g., Euler angles or quaternions) of the end effector 20. Together, they constitute the six-degree-of-freedom actual pose of the end effector 20 in space. The actual pose information is directly determined based on the state data transmitted back by the end effector 20. For example, after executing a control command, the end effector 20 collects end effector pose data through its onboard joint encoder, inertial measurement unit, or visual positioning system, and transmits the collected data back to the local device 10 as state data. The local device 10 then parses this state data to obtain the actual pose of the end effector 20.

[0054] The actual motion information in the real state body includes at least one of actual velocity, actual acceleration, and actual trajectory. Actual velocity refers to the actual displacement or angular displacement of the remote actuator 20 per unit time; actual acceleration refers to the actual velocity change per unit time; and actual trajectory refers to the actual position and attitude sequence of the remote actuator 20 over a period of time. This actual motion information can be obtained by performing differential or approximate operations on continuous actual pose information, or it can be directly measured by the velocity sensor or IMU of the remote actuator 20 itself. Actual force information includes actual torque and / or actual contact force. Actual torque refers to the actual output torque of each joint or actuator of the remote actuator 20; and actual contact force refers to the actual force exerted by the end effector of the remote actuator 20 when interacting with the environment. Actual force information can be collected by torque sensors or end effector force sensors at the joints of the remote actuator 20 and transmitted back to the local device 10 as part of the state data. Actual actuator state information includes actual gripper opening degree, i.e., the actual opening and closing degree of the end effector gripper of the remote actuator 20, which can be collected by displacement sensors or Hall sensors at the gripper. In addition, the real state entity also carries a second timestamp, which records the acquisition or transmission time of the state data corresponding to the real state entity. This second timestamp is used for subsequent comparison with the first timestamp of the command state entity to determine the communication delay deviation in the operation control system for delayed environments. All of the above state information collectively constructs the complete dataset of the real state entity, providing a data foundation for subsequently generating actual operation feedback screens and calculating various deviation information.

[0055] The asynchronous transmission of the control commands, status data, and 3D environment data over the communication connection means that each of these three data points is sent and received independently according to its own timing sequence, without waiting for or blocking the others. For example, the local device 10 can continuously generate and send new control commands without waiting for the remote execution device 20 to return status data; the remote execution device 20 can also return status data in real time without relying on the completion of 3D environment data acquisition; the acquisition and transmission of 3D environment data are also independent of the interaction between control commands and status data.

[0056] To achieve asynchronous transmission, the communication connection between the local device 10 and the remote execution device 20 is configured as an asynchronous communication link. This asynchronous communication link includes: an uplink channel for control commands, a downlink channel for status data, and a downlink channel for 3D scene data.

[0057] The control command uplink channel is used to transmit the control commands sent by the local device 10 to the remote execution device 20; the status data downlink channel is used to transmit the status data returned by the remote execution device 20 to the local device 10; and the three-dimensional scene data downlink channel is used to transmit the three-dimensional environment data returned by the remote execution device 20 to the local device 10.

[0058] By establishing multiple independent data channels on the same communication link, different channels can adapt to the differentiated transmission characteristics requirements of different types of data. For example, the control command uplink channel has high real-time requirements and can adopt a high-priority and high-frequency transmission strategy to ensure that the remote execution device 20 can respond to the operator's control intentions in a timely manner; the status data downlink channel has high requirements for data integrity and order and can adopt a reliable transmission protocol (such as TCP protocol) to avoid the real state from jumping or being distorted due to data loss or out-of-order; the 3D scene data downlink channel has high requirements for bandwidth and integrity, and can transmit complete scene model data at the initial connection and transmit incremental update data during operation according to the movement of the remote execution device 20 or changes in the environment to reduce bandwidth consumption.

[0059] Furthermore, the control commands and status data are transmitted with higher priority on the asynchronous communication link than the 3D environment data. The 3D environment data includes static and dynamic environment data, with the dynamic environment data having a higher transmission priority than the static environment data. This multi-level priority configuration ensures that the most critical control commands and status feedback are transmitted with minimal latency, while environmental data can be scheduled for differentiated transmission based on its network status.

[0060] Optionally, the local device 10 is further configured to generate a command state body based on the control command and the three-dimensional model of the remote execution device 20. The command state body is a virtual model of the remote execution device 20, and the pose of the command state body is determined according to the operation data. The command state body is used to characterize the target state to be achieved by the remote execution device 20. Based on the three-dimensional scene data stored in the local device 10 and the command state body, a reference operation feedback screen is generated.

[0061] Optionally, after generating a control command, the local device 10 generates a command state body based on the control command and the three-dimensional model of the remote execution device 20. Specifically, the local device 10 first parses the control command and extracts the target state parameters; then, based on the type of the target state parameters, it determines the target pose of each moving part of the remote execution device 20 in the three-dimensional model, and drives the three-dimensional model to present the target state to be achieved, i.e., the virtual model corresponding to the remote execution device 20, through forward kinematics calculation or direct assignment, and uses it as the command state body.

[0062] In other words, the generation process of the command state body can specifically involve mapping the abstract numerical parameters (such as joint angles, end-effector poses, etc.) in the control command to the moving parts (joints / kinematic chains) of the three-dimensional model, driving the three-dimensional model to move to the target state through positive kinematics, and finally generating a visualized virtual model in the three-dimensional operating space. This virtual model is the command state body, and its form corresponds to the target state to be achieved after the remote execution device 20 executes the control command. The target state includes at least one of the following: target pose information (such as target position and target posture), target motion information (such as at least one of target velocity, target acceleration, and target trajectory), target force information (such as at least one of target torque and target contact force), target actuator state information (target gripper opening degree), and the timestamp corresponding to the target state (i.e., the timestamp for generating the command state body).

[0063] For example, the target pose information in the command state body includes the target position and the target orientation. The target position refers to the coordinates (e.g., x, y, z values) to be reached by the end effector 20 in three-dimensional space, and the target orientation refers to the orientation (e.g., Euler angles or quaternions) to be reached by the end effector 20. Together, they constitute the six-degree-of-freedom target pose of the end effector 20 in space. The target pose information is directly determined based on the operation data of the local device 10. For example, when the operator manipulates the joystick, handle, or inputs through gestures on the local device 10, the local device 10 calculates the target position increment of the end effector 20 in three-dimensional space based on the magnitude and direction of the operation, and accumulates it with the current pose to obtain the target pose.

[0064] The target motion information in the command state body includes at least one of target velocity, target acceleration, and target trajectory. Target velocity refers to the displacement or angular displacement to be achieved by the remote execution device 20 per unit time; target acceleration refers to the change in velocity to be achieved per unit time; and target trajectory refers to the sequence of positions and attitudes to be achieved by the remote execution device 20 over a period of time. This target motion information can be obtained by performing differential or finite difference operations on continuous target pose information, or it can be directly generated by the upper-level trajectory planner.

[0065] The target force information includes target torque and target contact force. Target torque refers to the output torque to be achieved by each joint or actuator of the remote actuator 20, and target contact force refers to the force to be achieved when the end effector of the remote actuator 20 interacts with the environment. The target force information can be derived from operational data (such as input from a force feedback handle) or generated autonomously by the local device 10 according to task requirements.

[0066] The target actuator status information includes the target gripper opening degree, that is, the expected opening degree of the end gripper of the remote actuator 20, which can be directly determined based on the switch or analog input in the operation data.

[0067] For example, if the remote execution device 20 is a six-degree-of-freedom robotic arm, the operator controls the end effector to move to the target position via gesture input. The local device 10 generates an end effector pose command based on the gesture data, maps the target end effector pose to the target angles of the six joints using an inverse kinematics algorithm, and generates a control command containing joint angle commands. Subsequently, the local device 10 parses the control command, extracts the target angles of the six joints, and performs forward kinematics calculations based on the three-dimensional model of the robotic arm, driving the robotic arm in the three-dimensional model to achieve the target state where the end effector is at the target position and each joint is at the corresponding angle. Throughout the process, the virtual robotic arm model seen by the operator on the display screen of the local device 10 follows the operator's gestures or operations in real time, while the control commands are sent to the remote robotic arm for execution via the communication link after generation.

[0068] The local device 10 stores 3D scene data of the environment in which the remote execution device 20 is located. This 3D scene data can be pre-created and stored on the local device 10, or it can be acquired by the remote execution device 20 and asynchronously transmitted to the local device 10 for storage. This 3D scene data can be dynamically updated. The 3D scene data may include point clouds, meshes, depth maps, RGB graphics, texture maps, video projection surfaces, voxel maps, Gaussian splatting representations, object poses, dynamic object trajectories, safety boundaries, operable areas, obstacle locations, and object detection or segmentation results, etc. Each type of cached data can be accompanied by corresponding metadata, including: timestamp, spatial coordinate system, data source, reliability, validity period, update frequency, dynamic / static attributes, and whether it is related to the current operation.

[0069] Optionally, the remote execution device 20 collects 3D scene data of its location using its configured RGB camera, depth camera, binocular camera, LiDAR, wrist camera, external global camera, event camera, haptic or force sensor, and transmits the 3D scene data to the local device 10 via a communication link. After receiving the 3D scene data, the local device 10 writes it to its local 3D scene cache and adds a corresponding identifier ID and metadata to each type of cached data for subsequent credibility assessment and differentiated rendering. The metadata includes at least one of the following: timestamp, spatial coordinate system, data source, credibility, validity period, update frequency, dynamic / static attributes, and whether it is related to the current operation.

[0070] The local device 10 can generate a reference operation feedback screen based on its stored 3D scene data and command state body through a 3D rendering engine (such as Unity, Unreal Engine, or a custom rendering engine). The reference operation feedback screen is a visualized image containing the 3D scene and the command state body.

[0071] Optionally, when the operation data includes the current observation view obtained by parsing the observation view data, the local terminal device 10 is further configured to render the three-dimensional scene data stored in the local terminal device 10 and the command state body according to the current observation view, and generate a reference operation feedback screen corresponding to the current observation view.

[0072] For example, when the operation data includes viewing angle data, the reference operation feedback screen corresponds to the operator's current viewing angle. For example, when the local terminal device 10 includes a head-mounted display, when the operator's head rotates, the local terminal device 10 instantly generates the screen at the current viewing angle based on the real-time pose of the head-mounted display and the 3D scene data stored in the local 3D scene cache.

[0073] In other words, through the aforementioned remote operating system, after obtaining the user's operation data via the input device (such as a controller, hand tracking device, VR headset, etc.) of the local terminal device 10, the local terminal device 10 immediately generates a command state body and generates a reference operation feedback screen based on the 3D scene data stored in the local terminal device 10 and the command state body. Since the screen rendering relies on the locally stored 3D scene data and operation data, and does not rely on the 3D scene data transmitted in real time from the remote end, nor does it need to wait for the execution result of the remote execution device 20 to be returned, the operation feedback delay is effectively reduced, effectively eliminating the visual lag caused by waiting for remote feedback in traditional teleoperation, thereby significantly improving the smoothness of interaction on the user side.

[0074] Furthermore, the sending of control commands and the generation of command state bodies are independent processes. The sending of control commands does not need to wait for the generation and rendering of command state bodies to be completed, and the generation and rendering of command state bodies do not need to wait for the sending and confirmation of control commands, thus achieving decoupling between operation feedback and control execution.

[0075] After the local device 10 generates the actual operation feedback screen based on the real state entity, the reference operation feedback screen and the actual operation feedback screen can be displayed side by side. The actual operation feedback screen can also be obtained by updating the reference operation feedback screen based on the real state entity.

[0076] Optionally, the local terminal can generate deviation information based on the command status body and the actual status body; update the reference operation feedback screen based on the actual status body and the deviation information to obtain the actual operation feedback screen, wherein the actual operation feedback screen includes the command status body, the actual status body, and the deviation information between the command status body and the actual status body.

[0077] The deviation information between the command state body and the real state body may include at least one of the following: position deviation, attitude deviation, velocity deviation, trajectory deviation, gripper opening and closing deviation, torque deviation, contact state deviation, and time delay deviation.

[0078] Specifically, for any spatial or kinematic domain deviation among position deviation, attitude deviation, velocity deviation, trajectory deviation, gripper opening / closing deviation, torque deviation, and contact force deviation, the local terminal device 10 determines the deviation by calculating the difference between the corresponding target state parameter in the command state body and the corresponding actual state parameter in the real state body. For example, position deviation is the spatial distance vector between the target position and the actual position; attitude deviation is the rotation angle difference (such as Euler angle difference or quaternion deviation) between the target attitude and the actual attitude; velocity deviation is the vector difference between the target velocity and the actual velocity; trajectory deviation is the cumulative offset calculated by the position difference between the target trajectory point and the actual trajectory point at the same timestamp; gripper opening / closing deviation is the difference in opening degree between the target gripper opening degree and the actual gripper opening degree; torque deviation is the torque difference between the target torque and the actual torque; and contact force deviation is the force vector difference between the target contact force and the actual contact force. The above calculation methods are all direct difference operations on the corresponding dimension, that is, "target value minus actual value" or "actual value minus target value". The specific direction can be determined according to the needs of subsequent control strategies.

[0079] Regarding delay deviation, since delay reflects the misalignment of signal transmission or command response on the time axis, rather than a numerical difference at the spatial or mechanics level, delay deviation is not calculated through direct differences in state parameters. Instead, it is determined based on the first timestamp carried by the command state body and the second timestamp carried by the actual state body. For example, when generating the command state body, the local device 10 records the time of its issuance as the first timestamp. When receiving the state data returned by the remote execution device 20 and generating the actual state body, it records the time of receipt or the time of acquisition of the state data as the second timestamp. The delay deviation is the time difference between the second timestamp and the first timestamp. This time difference characterizes the total delay of the complete communication and response link from the local device 10 issuing the control command to the remote execution device 20, the remote execution device 20 performing the corresponding operation, and the remote execution device 20 returning the executed state data to the local device 10.

[0080] The aforementioned deviation information characterizes the degree of difference between the actual execution result of the remote execution device 20 and the expected target of the local device 10 from different dimensions. Specifically, position deviation, attitude deviation, velocity deviation, and trajectory deviation characterize the execution accuracy and tracking capability of the remote execution device 20 at the kinematic level, reflecting whether it moves according to the expected spatial position, orientation, velocity, and path; torque deviation and contact force deviation characterize the force output accuracy and environmental interaction capability of the remote execution device 20 at the dynamic level, reflecting whether it performs operations according to the expected force / torque output; gripper opening and closing deviation characterizes the positioning accuracy of the end effector (such as gripper) when performing grasping or releasing operations; and time delay deviation characterizes the communication delay status of the operation control system in a delayed environment. The above deviation information provides a multi-dimensional data foundation for subsequent operation feedback screen updates, control command corrections, and system status monitoring, helping operators to promptly perceive the actual execution status of the remote execution device 20 and the system communication status in a delayed environment.

[0081] By generating actual operation feedback screens, the difference between the expected state of the command and the actual operating state of the robot can be intuitively presented. With the help of visualized deviation information, operators can quickly and accurately judge the degree of synchronization of the robot following the command in real time, promptly detect abnormalities such as delay, trajectory deviation, and posture mismatch, reduce the risk of remote operation errors, and improve the accuracy of remote operation and the safety of operation.

[0082] Optionally, the local device 10 can also generate a safety constraint strategy based on the deviation information. The safety constraint strategy includes at least one of the following: generating and displaying a deceleration prompt message on the local device 10 to prompt the operator to slow down. For example, when the deviation exceeds a first threshold but does not exceed a second threshold, a text prompt or icon prompting "Please slow down" is displayed in the VR screen.

[0083] Control commands are limited to restrict their rate of change. For example, when the position or attitude deviation is large (e.g., exceeding a preset deviation threshold), it indicates that the operator is issuing a large and rapid movement command to the remote execution device 20. Directly issuing the control command in this situation could cause the remote execution device 20 to collide or become unstable due to excessively high joint angular velocity or end effector velocity. To avoid this risk, the local device 10 limits the joint angular velocity or end effector velocity in the control commands to a preset safe speed range, ensuring that the remote execution device 20 approaches the target state smoothly and safely.

[0084] The sending of control commands to the remote execution device 20 is suspended. For example, when at least one of the position deviation, attitude deviation, speed deviation, torque deviation, or contact force deviation exceeds a preset safety threshold (for example, an excessive position deviation indicates that the remote execution device 20 has not reached the target position as expected, and an excessive torque deviation indicates that the device may have had an accidental collision with the environment), the local device 10 suspends the sending of new control commands and continuously monitors the deviation information until the deviation returns to a safe range before resuming the sending of control commands, in order to avoid dangerous consequences caused by continuing to issue instructions under abnormal conditions.

[0085] The current pose of the command state body is locked. For example, when the confidence level of a target region in the 3D scene data is low (e.g., the corresponding 3D data has not been updated for a long time, data is missing, or the confidence score is below a preset threshold), it indicates that the actual environmental information of that region is unreliable. If the operator is allowed to continue moving the command state body to that region, it may lead to incorrect judgments based on outdated scene data. In this case, the local device 10 locks the pose of the command state body, prohibiting the operator from moving the command state body to the unreliable region until the 3D data of the target region becomes reliable or the user confirms the risk.

[0086] Regional constraints are added to the motion planning of the command state body. These regional constraints are used to restrict the remote execution device 20 from entering a preset region. For example, when the command state body approaches a low-confidence region or an outdated region (e.g., a region marked as low confidence or not updated in time in the 3D scene data), the local device 10 adds constraints to the motion planning to prohibit the remote execution device 20 from entering that region, thereby proactively avoiding potential risk regions at the planning level.

[0087] A retraction command is sent to the remote execution device 20 to cause the remote execution device 20 to respond to the retraction command and retract to a safe posture. For example, when it is determined based on the deviation information that the remote execution device 20 is approaching an unknown obstacle or entering a dangerous area (e.g., the position deviation persists and there is an abnormal offset between the actual state and the command state, or the torque deviation transmitted by the remote execution device 20 increases sharply), the local device 10 automatically generates a retraction command and sends it to the remote execution device 20, causing the device to retract to a preset safe posture, preventing the device from colliding with obstacles in the environment or being damaged due to exceeding the safety boundary.

[0088] The aforementioned safety protection measures can be implemented individually or in combination depending on the specific scenario. For example, when a large deviation is detected but has not yet exceeded the safety threshold, only the amplitude limiting processing of the control command can be performed to limit the rate of speed change; when the deviation exceeds the safety threshold, a combination of measures, including pausing the transmission of control commands and sending a rollback command to the remote execution device 20, can be implemented simultaneously to ensure that the remote execution device 20 can stop and roll back to a safe posture in a timely manner under extreme circumstances, thereby improving the safety of the operation control system for delayed environments.

[0089] Safety constraint policies are generated by using deviation information to enable timely intervention when abnormalities are detected by the local terminal device 10. Policies such as prompts, speed limits, pauses, freezes, regional restrictions, or automatic rollback are used to ensure the safety of operations and reduce the risk of misoperation and collisions.

[0090] It is worth mentioning that the remote execution device 20 can transmit 3D scene data to the local device 10 at preset intervals (e.g., every 500ms, every 1s, or every 2s), and can also send data in response to preset trigger conditions. These preset trigger conditions may include: the end effector of the remote execution device 20 entering a new spatial region, completing an operation (such as completing a grab), the environmental perception module detecting a new object or obstacle in the scene, the operator actively requesting an update of the 3D scene data for a certain region through the local device 10, or the data for a certain region in the 3D scene cache of the local device 10 exceeding its expiration date. By combining periodic transmission with event-triggered transmission, unnecessary communication overhead can be effectively reduced while ensuring the availability of scene data on the local device 10, making data transmission more flexible and efficient.

[0091] To reduce bandwidth consumption and communication frequency of 3D scene data transmission, the remote execution device 20 can also adopt at least one of the following strategies: First, an on-demand transmission strategy based on scene change detection, i.e., data of the changed area is transmitted only when the change in the currently acquired 3D data relative to the data of the previous moment exceeds a preset threshold; otherwise, transmission is not performed or is delayed. Second, a selective transmission strategy based on the observation perspective of the local terminal device 10, i.e., the visible area is determined according to the current observation perspective sent by the local terminal device 10, and the 3D scene data of that area is transmitted first, while the acquisition and transmission frequency is reduced for non-visible areas. Third, a differentiated transmission strategy based on semantic classification, i.e., transmission is based on the semantic category of different objects in the scene and their relationship with the current operation task. The transmission strategies employ different levels of relevance, using varying transmission frequencies and qualities (e.g., high-frequency, high-precision transmission for objects with high operational relevance, and simplified transmission for background data at lower frequencies). Fourth, an adaptive transmission strategy based on network status adjusts the transmission frequency, resolution, and compression rate of the 3D scene data according to the current communication link's bandwidth, latency, and packet loss rate. Fifth, an event-triggered conditional transmission strategy triggers data transmission in response to preset events such as the remote execution device 20 entering a new area, completing an operation, detecting a new object, or the local device 10 actively requesting data. Sixth, an incremental transmission and partial update strategy transmits only incremental data or data from changed areas that are already cached by the local device 10, rather than complete scene data. These various transmission strategies can be used individually or in combination according to the specific application scenario.

[0092] The remote execution device 20 can also be used to collect three-dimensional scene data in real time or at preset intervals, and determine whether there are any changed areas in the three-dimensional scene where the remote execution device 20 is located based on the collected three-dimensional scene data and historical scene data; if there are, the three-dimensional scene data of the changed area is sent back to the local terminal device to update the three-dimensional scene cached by the local terminal device 10; if there are no changes, the currently collected scene data is not sent back.

[0093] Specifically, the remote execution device 20 can use at least one of the following methods to determine scene changes: a point cloud density change detection method, which compares the point cloud density difference between the current frame point cloud and the historical frame point cloud within the same spatial voxel, and determines that the region has changed if the difference exceeds a preset density threshold; a depth image frame difference method, which performs pixel-by-pixel difference calculation between the current depth map and the historical depth map, and determines that the corresponding spatial region has changed if the difference value exceeds a preset depth threshold; an octree change detection method, which uses an octree data structure to spatially divide the 3D scene, and determines the changed region by comparing the occupancy status changes of each tree node at the current time and at historical time; and a semantic segmentation change detection method, which performs semantic segmentation on the acquired RGB image or point cloud data, identifies object instances in the scene, and determines that the corresponding region has changed when an object is added, removed, or the pose of an existing object changes.

[0094] If a changed area is detected in the 3D scene, the remote execution device transmits the 3D scene data of that changed area back to the local device to update the cached 3D scene on the local device. The transmitted changed area data can be the complete point cloud data of that area, updated voxel occupancy information, mesh data of the changed area, or an incremental data packet containing only the changed portion. If no changed area is detected in the 3D scene, the remote execution device does not transmit the currently acquired scene data, thus effectively avoiding the repeated transmission of the same static scene information.

[0095] By employing the aforementioned region-selective backhaul strategy based on change detection, the remote execution device only backhauls 3D data of the changed region when the scene changes. Static regions that have not changed are not repeatedly transmitted. This significantly reduces the amount of 3D environment data transmitted, lowers communication bandwidth usage and transmission frequency, and reserves more communication resources for the real-time transmission of control commands and status data, while ensuring the availability and timeliness of 3D scene data on the local device.

[0096] Optionally, the three-dimensional environment data includes static environment data and dynamic environment data, with the transmission priority of the dynamic environment data being higher than that of the static environment data.

[0097] Static environment data refers to spatial information in a scene that does not change over time or changes very infrequently. This includes, but is not limited to, 3D point clouds, mesh models, occupancy maps, or texture information of static structures such as fixed walls, floors, ceilings, fixed tooling fixtures, permanent workbenches, fixed bases, safety fences, and equipment supports. Static environment data hardly changes during the operation of remote execution devices, and once collected and successfully transmitted to the local device cache, it does not require repeated updates for a considerable period. Dynamic environment data refers to spatial information in a scene that changes over time. This includes, but is not limited to, the real-time pose and configuration of the remote execution device itself (such as the robotic arm, end effector, or mobile chassis), changes in the position and orientation of target objects, the position and trajectory of non-cooperative objects (such as workpieces moving on a conveyor belt, other mobile devices, or personnel), and obstacles that temporarily appear or are removed in the environment. Dynamic environment data is characterized by high timeliness, frequent changes, and a high correlation with operational safety, requiring timely updates to reflect the real-time state of the scene.

[0098] The specific implementation methods for prioritizing the transmission of dynamic environment data over static environment data include at least one of the following: when bandwidth is limited, the remote execution device prioritizes the transmission of dynamic environment data, while static environment data can be transmitted when bandwidth is idle or transmitted with a delay; dynamic environment data uses a higher transmission frequency, while static environment data uses a lower transmission frequency; when both dynamic and static environment data are waiting to be sent, the link scheduler prioritizes placing the dynamic environment data into the transmission queue; for dynamic environment data, incremental transmission can be used to promptly send its changes, while for static environment data, full transmission or on-demand transmission can be used.

[0099] By assigning different transmission priorities to dynamic and static environmental data, with dynamic environmental data taking precedence over static environmental data, the system ensures timely updates of dynamic environmental data (such as changes in target object position and the robot's own motion state) while guaranteeing the immediate issuance of control commands and real-time feedback of status data. Static environmental data is transmitted using available bandwidth. This refined multi-level priority scheduling mechanism maximizes operational safety and real-time feedback with limited communication resources, enabling the system to maintain stable and reliable operational control performance even in complex network environments.

[0100] Due to limitations in communication bandwidth fluctuations and transmission delays, the 3D data received by the local device 10 is often non-uniform over time. Therefore, when the 3D environment data is accompanied by corresponding metadata, including timestamps, spatial coordinate systems, data sources, reliability, validity periods, update frequencies, dynamic / static attributes, and relevance to the current operation, the local device 10 can dynamically assess the current reliability of each region based on the timestamps, data sources, update frequencies, dynamic / static attributes, and spatial variations of the locally cached 3D data.

[0101] That is, the local device is also used to determine the credibility of each of the multiple regions based on at least one of the timestamp, data source, update frequency, dynamic and static attributes, and spatial change conditions in the metadata associated with each region in the 3D scene data; and based on the credibility of each region, to fuse the 3D environment data returned by the remote execution device with the 3D scene data stored in the local device to update the 3D scene data stored in the local device.

[0102] Furthermore, the local terminal device is also used to perform differentiated rendering on the multiple regions in the updated 3D scene data stored by the local terminal device according to the confidence level corresponding to each of the multiple regions. The local terminal device is also used to generate and display the operation feedback screen based on the differentially rendered 3D scene data and the command status body; the differential rendering includes at least one of the following: areas with a confidence level higher than a first preset value are displayed normally; areas with a confidence level lower than the first preset value but higher than a second preset value are displayed semi-transparently, where the first preset value is higher than the second preset value; areas with a confidence level lower than the second preset value are displayed in a faded or weakened manner; areas that have exceeded their validity period are hidden or displayed with risk warnings based on the timestamp and update frequency; dynamic and static attribute indicators display historical trajectories or trails for dynamic object areas; and areas with spatial change indicators that are related to the current operation task are prioritized or highlighted.

[0103] For example, static desktops, fixed fixtures, walls, and other areas that have not changed for a long time can maintain high reliability; areas near the end effector require a high update frequency to maintain reliability, and reliability decreases if they are not updated for a long time; the reliability of dynamic objects, free objects, occluded areas, or areas that have not been observed for a long time decreases; the reliability of areas with consistent observations from multiple sensors increases; and areas directly related to the current operation task are displayed first. Based on the reliability assessment results, the local device 10 performs differentiated rendering of the cached data: high-reliability areas are displayed normally, providing clear spatial reference; medium-reliability areas are displayed semi-transparently, indicating to the operator that the information in this area may be inaccurate; low-reliability areas are displayed as dashed lines, faded, or weakened, reminding the operator to refer to them with caution; expired areas are hidden or have risk warnings displayed; dynamic objects display historical trajectories or trails to help the operator predict their movement trends; high-risk areas are highlighted to attract attention; and areas related to the current operation are displayed first.

[0104] Through the aforementioned differentiated rendering, operators can intuitively identify the credibility and timeliness of each area without relying on additional data annotations, effectively avoiding incorrect spatial judgments based on expired or low-quality data, thereby significantly improving the safety of teleoperation and the accuracy of task execution in a delayed environment.

[0105] The local terminal device 10 and the remote execution device 20 communicate via a communication link, which is used to transmit data between the local terminal device 10 and the remote execution device 20. This communication link can be configured as a bidirectional communication link or an asynchronous communication link.

[0106] In some embodiments of this application, the communication link is equipped with an uplink channel for control commands, a downlink channel for status data, and a downlink channel for 3D scene data; wherein: The control command uplink channel is used to transmit the control commands sent by the local terminal device 10 to the remote execution device 20; the status data downlink channel is used to transmit the status data returned by the remote execution device 20 to the local terminal device 10; and the three-dimensional scene data downlink channel is used to transmit the three-dimensional scene data of the environment in which the remote execution device 20 is located.

[0107] By establishing multiple independent data channels on the same communication link, different channels can adapt to the differentiated transmission characteristics requirements of different types of data. For example, the control command uplink channel, which has high real-time requirements, can employ a high-priority and high-frequency transmission strategy to ensure that the remote execution device 20 can respond promptly to the operator's control intentions; the status data downlink channel, which has high requirements for data integrity and order, can employ a reliable transmission protocol (such as TCP) to avoid jumps or distortions in the real state due to data loss or out-of-order delivery; the 3D scene data downlink channel, which has high requirements for bandwidth and integrity, can transmit complete scene model data during initial connection and incrementally update data during operation based on the movement of the remote execution device 20 or changes in the environment, thereby reducing bandwidth consumption. This multi-channel design allows different types of communication data to travel independently on the same communication link without interference, jointly providing a communication foundation for the real-time performance, reliability, and security of the operation control system in a latency-prone environment.

[0108] Optionally, if only one communication link is configured between the local terminal device 10 and the remote execution device 20, and the communication link is an asynchronous communication link, then the asynchronous communication link adopts different transmission priorities and / or transmission cycles when transmitting uplink control command data and downlink status data, so as to achieve differentiated transmission service quality on the same link.

[0109] By adopting the above system architecture, the local device 10 and the remote execution device 20 achieve data interaction through an asynchronous communication link. The three channels are independent and do not block each other, achieving decoupling of operation feedback and control execution, and improving the overall response performance, operational stability, and robustness of the system in a latency environment. Specifically, the control command uplink channel uses high priority and high frequency transmission to ensure that the user's control intentions can be delivered to the remote execution device 20 with minimal latency, effectively reducing instruction transmission latency; the status data downlink channel independently transmits real status data, enabling the local device 10 to continuously update the real state volume and perform deviation calculations and safety constraints without relying on scene data transmission, avoiding feedback delays caused by state data and scene data waiting for each other; the 3D scene data downlink channel uses differentiated update frequencies according to data type, so that static scenes, dynamic objects, the area near the robot end effector, and safety-related data are updated independently at appropriate frequencies, significantly reducing redundant data transmission. The asynchronous parallel working mechanism of the three channels ensures that delays or jitters in scene data transmission will not block the transmission of control commands and status data. Fluctuations in the communication quality of any link will not affect other links. This fundamentally solves the overall performance bottleneck problem caused by the strong coupling between control feedback, status transmission and scene rendering in traditional teleoperation systems, and significantly improves the availability and user experience of teleoperation systems in complex network environments.

[0110] like Figure 3 As shown, this embodiment uses the example of a local device 10 including a VR headset and a handheld controller, and a remote execution device 20 including a single-arm robot, where the operator controls the single-arm robot to grasp a target object on a table using the handheld controller. The device collects observation viewpoint data (headset pose) and operation command data (handle pose, button status) through the positioning systems of the head-mounted display and the controller. This data is then analyzed to obtain operation data containing the current observation viewpoint and the end-effector pose. Subsequently, based on the analyzed end-effector pose, the local device 10 combines the morphological data (degrees of freedom of each joint, link length, joint limits) and kinematic models (forward and inverse kinematic models) of the remote actuator 20. Using an inverse kinematics algorithm, it maps the end-effector pose to target angles for each joint, generating control commands containing joint angle commands. Simultaneously, the local device 10 generates a command state body (i.e., a virtual model of the target gripper representing the target state to be achieved by the end-effector) based on this control command and the 3D model of the remote actuator 20. Based on the 3D scene data stored in the local device 10 and this command state body, a reference operation feedback screen is generated using a local 3D rendering engine and displayed to the operator in the VR head-mounted display. The operator can see the virtual gripper representing their operational intention moving in real time following the controller, without waiting for any response from the remote robot. The reference operation feedback screen is the screen corresponding to the operator's current viewing angle.

[0111] Control commands are sent to the remote robot via the control command uplink channel. The robot control module receives the commands and drives the robot to perform the corresponding grasping actions. During execution, the remote robot's real-state acquisition module continuously collects state data such as the actual angles of each joint, the actual pose of the end effector, and the actual opening and closing amount of the gripper, and asynchronously transmits this data back to the local device 10 via the state data downlink channel. The local device 10 receives the state data transmitted back from the remote execution device 20, generates a real-state body (i.e., a virtual model of the actual gripper representing the actual execution state) based on this state data and the 3D model of the remote execution device 20, and generates an actual operation feedback screen based on this real-state body. Specifically, the local device 10 generates deviation information (including at least one of position deviation, posture deviation, and gripper opening and closing deviation) based on the command state body and the real-state body, and then updates the reference operation feedback screen based on the real-state body and this deviation information to obtain the actual operation feedback screen. In the actual operation feedback screen, the command state body, the real-state body, and the deviation information between them (such as deviation lines or color changes) are displayed side by side in the same VR 3D operation space.

[0112] When the command state is close to the target object but the actual state is still significantly lagging behind (e.g., due to network latency preventing the robot from reaching the target position), the operator can clearly judge the gap between the current operational intention and the actual execution state by visualizing the command state and the actual state displayed side-by-side in the actual operation feedback screen, as well as the deviation between them. If the deviation exceeds a preset safety threshold, the local device 10 generates a safety constraint strategy based on the deviation information. Specifically, the local device 10 generates and displays a "Please slow down" prompt, and simultaneously limits the amplitude of subsequent control commands to restrict the rate of change of control commands, preventing the robot from colliding or overshooting due to excessive speed. The operator adjusts the operation rhythm in a timely manner according to the above prompts. When the actual state finally reaches the target object position and the gripper's opening and closing state is consistent with the command state, the operator inputs the command to close the gripper through the handle button. The local device 10 generates the command state for closing the gripper and sends the corresponding control command to complete the entire grasping task.

[0113] like Figure 4 As shown, the example illustrates how an operator can quickly turn their head to observe the surrounding environment of a remote robot while performing the task of grasping the target object: The operator wears a VR headset and needs to frequently turn their head to observe different areas around the remote robot during operation. For example, before grasping a task, it is necessary to confirm whether there are obstacles around the target object, or to observe the spatial conditions on the robot's path during movement. Before performing head turning observation, the local device 10 has pre-acquired and cached the 3D scene data of the remote robot's environment through the 3D scene data downlink channel.

[0114] When the operator's head turns, the local device 10 collects the head-mounted display's pose data in real time through the head-mounted display's positioning system as observation viewpoint data, and analyzes this observation viewpoint data to obtain the current observation viewpoint. Subsequently, based on the stored 3D scene data and the generated command state body (such as the robot's current target pose), the local device 10 generates a reference operation feedback screen through the local 3D rendering engine. Specifically, the 3D scene data and command state body are rendered according to the current observation viewpoint to generate a reference operation feedback screen corresponding to the current observation viewpoint, which is then displayed to the operator in the VR head-mounted display. This process does not wait for the remote camera to re-capture the image, nor does it wait for new point clouds or new videos to arrive from the remote location. Instead, it directly generates the image from the new perspective based on the 3D scene data in the local 3D scene cache in real time—the operator can see the 3D scene and robot status from the new perspective almost instantly after turning their head, with smooth and lag-free image display.

[0115] Meanwhile, the remote environmental perception module continuously collects 3D environmental data and asynchronously sends it to the local device 10 via the 3D scene data downlink channel. The local device 10 writes the newly arrived data into the 3D scene cache and updates the timestamp and credibility information of the corresponding area (as described in the aforementioned local 3D scene cache module and credibility assessment and differentiated rendering module). When the remote execution device 20 transmits real-state data during the execution of control commands, the local device 10 receives the state data and generates a real-state body, and then generates an actual operation feedback screen based on the real-state body. Specifically, deviation information can be generated based on the command state body and the real-state body, and the reference operation feedback screen can be updated based on the real-state body and the deviation information to obtain an actual operation feedback screen containing the command state body, the real-state body, and the deviation information. Thus, while the operator turns their head to observe the scene, they can still continuously monitor the consistency between the robot's command intent and the actual execution state.

[0116] Furthermore, if a region already has data in its local cache but its timestamp is far removed from the current time, the credibility assessment module will determine its credibility based on the region's spatial attributes (such as whether it is a static wall or a fixed fixture) and update time. The differential rendering module will then display this information accordingly (as described in the aforementioned credibility assessment and differential rendering modules). For example, when an operator quickly turns to observe the area behind the robot, if the point cloud data for that area was collected 10 seconds ago and the area is a static wall, its credibility is considered high, and it is displayed normally for the operator's reference. If the area is where a dynamic object (such as another moving robot) has passed through and the data has expired, it will be displayed semi-transparently or as a dashed line with the message "Information for this area may be expired; please observe carefully." If the expired region is related to the current operation path and the deviation or risk exceeds a preset threshold, the local device 10 can also generate a safety constraint strategy based on the deviation information. For example, it can add regional constraints to the motion planning of the command state body to restrict the remote execution device 20 from entering the low-confidence or expired region, or suspend sending control commands to the remote execution device 20 until the 3D data of the region is updated and the confidence level is restored. After the new data arrives from the remote end, the system asynchronously updates the cache and refreshes the rendering effect, so that the operator can obtain the latest scene information.

[0117] Because the rendering of the head-view image is based entirely on the local 3D scene cache and does not rely on the collection and transmission of real-time data from a remote location, the head motion feedback latency is minimized, effectively reducing VR dizziness and operator fatigue. At the same time, through asynchronous updates and reliability assessments of 3D scene data, differentiated rendering, and the parallel display and deviation visualization of command and real state bodies, operators can intuitively judge which areas in the currently viewed 3D scene are reliable and which areas may pose risks. They can also continuously monitor the robot's command following status and intervene in a timely manner through safety constraint strategies when risks exist, thus ensuring both operational smoothness and operational safety.

[0118] By adopting the aforementioned operation control system designed for delayed environments, the traditional serial dependency relationship in teleoperation—where data must first be acquired remotely, then transmitted over a network, then rendered locally, and finally fed back to the operator—is fundamentally changed. In existing technologies, the view after the operator turns their head relies on a remote camera for re-acquisition and transmission. After the user inputs data, they must wait for the remote robot to execute and transmit the actual state before seeing the result. Furthermore, control feedback, state transmission, and scene rendering are in the same strongly coupled closed loop—a delay in any link directly leads to delayed head movement feedback, sluggish hand operations, and VR dizziness, severely limiting the success rate of delicate teleoperation tasks such as grasping, plugging, aligning, twisting, and assembling. This disclosure decouples the operation command state link from the 3D scene rendering link into two asynchronous, parallel, and non-blocking independent links. Combined with locally cached 3D scene data and real-time re-rendering of the head view, this allows the operator to immediately obtain a new spatial perspective based on existing local 3D data when turning their head, without waiting for remote data to arrive. This effectively reduces head movement feedback latency and significantly reduces VR dizziness and operator fatigue.

[0119] When the local device 10 specifically includes a VR headset, the operator can simultaneously observe the real-time generated command state (representing the current operational intent) and the asynchronously transmitted real state (representing the robot's actual execution state) in the virtual 3D operation space. These two are displayed side-by-side with accompanying deviation visualization prompts, allowing the user to confirm that the operational intent has been received by the system without waiting for the remote execution result. This effectively eliminates operational lag and significantly improves operational responsiveness. When the local device 10 includes a controller, hand tracking device, joystick, or haptic device, after the user inputs, the local device 10 immediately generates a control command based on the operation data and renders the command state. Simultaneously, the control command is sent to the remote execution device 20 via the uplink channel, and the real state is asynchronously transmitted via the downlink channel for correction and updating. Even if the real state is delayed due to network latency, the user can still obtain real-time operational feedback through the command state, continuously judging whether the robot accurately follows the current command, thereby significantly improving the success rate of fine-grained tasks.

[0120] Furthermore, this disclosure employs a local 3D scene cache to finely manage metadata such as timestamps, credibility, validity periods, and dynamic / static attributes attached to 3D data (point clouds, meshes, depth maps, etc.) asynchronously transmitted from remote locations. This allows historical 3D data to be continuously used for local re-rendering, spatial alignment, and operational reference, avoiding bandwidth waste caused by repeatedly transmitting the same static scene data. Simultaneously, a differentiated rendering mechanism based on credibility assessment (normal display of high-credibility areas, semi-transparent display of medium-credibility areas, faded dashed lines of low-credibility areas, and hiding or warning of expired areas) combined with multiple security constraint strategies (including deceleration warnings, amplitude limiting, pause transmission, pose freezing, regional constraints, and automatic rollback) enables operators to intuitively identify low-credibility and expired areas, effectively reducing the risk of misoperation and collisions, and significantly improving the overall robustness and operational safety of the system in bandwidth-constrained or high-latency network environments.

[0121] When performing remote control using an operation control system designed for a delayed environment, the following steps may be included: Step S1: Establish a local 3D operation space. The local device receives the initial environment data (such as point cloud, mesh, or depth map) returned by the remote execution device, initializes the 3D operation space, writes it to the local 3D scene cache, and attaches metadata such as timestamp, coordinate system, data source, and credibility.

[0122] Step S2, Local device collects user input: The operator generates operation input through VR headset, controller, hand tracking device or other local input device; the local device parses the collected raw interaction signals (including headset pose, controller pose, button / joystick status, hand joint data, etc.) into operation data, which represents the operation intention for the robot.

[0123] Step S3: The local device generates control commands and command status bodies. The local device generates control commands that can be executed by the remote execution device based on the operation data. At the same time, it generates command status bodies based on the control commands and the three-dimensional model of the remote execution device. The command status bodies can be displayed in the three-dimensional operation space, so that the operator can see the target status corresponding to the current operation intention in real time. This step does not wait for the remote execution device to execute or for the actual status to be returned.

[0124] Step S4: The local device sends control commands to the remote execution device: The local device sends control commands to the remote execution device through the control command uplink channel; The control module of the remote execution device receives the control commands and drives the robot to perform the corresponding actions.

[0125] Step S5, asynchronous transmission of status data from the remote execution device: During the execution of control commands, the real status acquisition module continuously collects real status data such as joint angles, end poses, gripper opening and closing amounts, force / torque data, and transmits them asynchronously back to the local device through the status data downlink channel.

[0126] Step S6: The local device generates a real state body and calculates the deviation: The local device generates a real state body based on the returned state data and the 3D model of the remote execution device, and calculates at least one of the following deviations between the command state body and the real state body: position deviation, attitude deviation, speed deviation, trajectory deviation, gripper opening and closing deviation, torque deviation, and time delay deviation. Based on the real state body and the deviation information, the reference operation feedback screen is updated to obtain the actual operation feedback screen. The deviation information is displayed in the actual operation feedback screen through deviation lines, color changes, or numerical prompts.

[0127] Step S7, Asynchronous Update of Remote 3D Scene Data: The remote environment perception module continuously collects 3D environmental data (point cloud, depth map, Mesh, etc.) and asynchronously sends it to the local device through the 3D scene data downlink channel; the local device writes the new data into the local 3D scene cache, updates the timestamp and reliability of the corresponding area, and performs differentiated storage according to the newness and type of the data.

[0128] Step S8: The local device performs real-time re-rendering of the head view based on the local cache: When the operator's head turns, the local device directly generates the VR screen of the current view based on the existing 3D scene data in the local 3D scene cache according to the real-time pose of the head display, without waiting for the arrival of new 3D data from the remote end; if new data from the remote end arrives later, the cache is updated asynchronously and the rendering effect of the corresponding area is refreshed.

[0129] Step S9, the local device executes safety constraints: safety constraints are triggered when the system detects any of the following situations—the deviation between the command state body and the real state body exceeds a preset threshold, the confidence level of the 3D data of the target area is too low, the target area has not been updated for a long time, the command state body enters a dangerous area or a low confidence area, or the user input speed exceeds the safe range; safety constraint strategies include prompting the operator to slow down, limiting the amplitude of control commands, pausing the transmission of control commands to the remote end, locking the current pose of the command state body, adding regional constraint conditions in motion planning, or sending a rollback command to the remote execution device to make it rollback to a safe pose.

[0130] It is worth mentioning that the above steps S7 (asynchronous update of remote 3D scene data) and S8 (real-time re-rendering of head view based on local cache) can be executed asynchronously in parallel. They are triggered by different events and run on their own independent time scales. Specifically, step S7 is triggered by the arrival of remote data. Its execution process involves writing the newly arrived 3D scene data into the local cache and updating the metadata (timestamp, credibility, etc.) of the corresponding area. This process is entirely performed in the background and does not depend on whether the user's head is turning or is not constrained by the rendering frame rate. Step S8 is triggered by the operator's head turning event (usually driven by the real-time pose update of the VR headset). Its execution process involves reading the existing 3D scene data from the local 3D scene cache according to the current headset pose and performing real-time rendering. This process is entirely completed locally and does not wait for the data to be written in step S7.

[0131] When step S7 completes data updates during step S8, the local device does not need to interrupt or restart the currently running rendering process; the latest data in the cache will be naturally adopted in the next rendering frame. When step S8 is triggered during step S7, step S8 reads the cached data before the update by step S7 (i.e., the latest data available in the previous frame), but the write operation of step S7 will not block the read operation of step S8. Through the above parallel execution method, the frame rate stability of head-view re-rendering depends on the local graphics rendering performance rather than the network transmission rate, while the update frequency of the remote 3D scene data is independently adapted to the network bandwidth and the speed of environmental changes. The two are independent of each other, fundamentally avoiding the head-view animation stuttering problem caused by scene data update delays in traditional solutions.

[0132] The preferred embodiments of this disclosure have been described in detail above with reference to the accompanying drawings. However, this disclosure is not limited to the specific details of the above embodiments. Within the scope of the technical concept of this disclosure, various simple modifications can be made to the technical solutions of this disclosure, and these simple modifications all fall within the protection scope of this disclosure.

[0133] It should also be noted that the various specific technical features described in the above embodiments can be combined in any suitable manner without contradiction. To avoid unnecessary repetition, this disclosure will not describe the various possible combinations separately.

[0134] Furthermore, various different embodiments of this disclosure can be combined in any way, as long as they do not violate the spirit of this disclosure, they should also be regarded as the content disclosed in this disclosure.

Claims

1. An operation control system for delayed environments, characterized in that, It includes a local terminal device and a remote execution device, and the local terminal device and the remote execution device are connected in communication: The local terminal device is used to obtain operation data in response to the user's operation, generate control commands based on the operation data, and send the control commands to the remote execution device; The remote execution device is used to respond to the control command and execute the operation corresponding to the control command, collect its own status data when executing the operation and send it back to the local terminal device; The remote execution device is also used to collect three-dimensional environmental data of its own location and transmit it to the local terminal device, wherein the control commands, the status data and the three-dimensional environmental data are transmitted asynchronously over the communication connection. The local device is also used to update the three-dimensional scene data stored in the local device according to the three-dimensional environment data, generate a real state body according to the state data, and generate an actual operation feedback screen according to the real state body and the three-dimensional scene data stored in the local device at the current moment. The real state body is a three-dimensional virtual object of the remote execution device, and the real state body represents the actual state reached by the remote execution device in response to the control command.

2. The system according to claim 1, characterized in that, The local terminal device is also used to acquire at least one of observation view data and operation command data. The observation view data includes the pose data of the wearable device in the local terminal device. The operation command data includes at least one of the pose data of the handheld controller, hand tracking data, button input data, joystick input data, and touch input data. The observation view data and / or the operation command data are parsed to obtain the operation data.

3. The system according to claim 2, characterized in that, The local terminal device is also used to render the three-dimensional scene data and the real state volume according to the current observation view when the operation data includes the current observation view obtained by parsing the observation view data, and generate an actual operation feedback screen corresponding to the current observation view.

4. The system according to claim 1, characterized in that, The communication connection between the local terminal device and the remote execution device is configured as an asynchronous communication link; The asynchronous communication link includes: The control command uplink channel is used to transmit the control commands sent by the local terminal device to the remote execution device; The status data downlink channel is used to transmit the status data returned by the remote execution device to the local device; The three-dimensional scene data downlink channel is used to transmit the three-dimensional environment data returned by the remote execution device to the local device.

5. The system according to claim 1, characterized in that, The remote execution device is also used to collect its own status data in real time when performing the operation and transmit it back to the local device. as well as It is used to collect three-dimensional scene data in real time or at preset intervals, and to determine whether there are any changed areas in the three-dimensional scene where the remote execution device is located based on the collected three-dimensional scene data and historical scene data. If so, the 3D scene data of the changed area is sent back to the local terminal device to update the 3D scene cached by the local terminal device; If it does not exist, the currently collected scene data will not be returned.

6. The system according to claim 5, characterized in that, The control commands and the status data are transmitted with higher priority than the three-dimensional environment data on the asynchronous communication link. The three-dimensional environment data includes static environment data and dynamic environment data, with the transmission priority of the dynamic environment data being higher than that of the static environment data.

7. The system according to claim 1, characterized in that, The local terminal device is also used to determine the credibility of each of the multiple regions based on at least one of the timestamps, data sources, update frequencies, dynamic and static attributes, and spatial changes in the metadata associated with each of the multiple regions in the 3D scene data. Based on the credibility of each region, the 3D environment data returned by the remote execution device is fused with the 3D scene data stored by the local device to update the 3D scene data stored by the local device.

8. The method according to claim 1, characterized in that, The local terminal device is also used to generate a command state body according to the control command, and to generate and display a reference operation feedback screen based on the three-dimensional scene data stored at the current time and the command state body. The command state body is a three-dimensional virtual object of the remote execution device, and the command state body is used to characterize the target state to be achieved by the remote execution device.

9. The system according to claim 8, characterized in that, The local terminal device is also used to generate deviation information based on the command status body and the actual status body, and to generate a target screen based on the deviation information, the command status body and the actual status body. The target screen includes an actual operation feedback screen, a reference operation feedback screen and the deviation information.

10. The system according to any one of claims 1-9, characterized in that, The remote execution device includes a robot, which includes at least one of a single-arm robot, a dual-arm robot, a fixed-base robot, a mobile robot, and a humanoid robot; the local device includes a local display device and a local input device, which includes at least one of a VR headset, AR glasses, a desktop computer monitor, a tablet computer, and a mobile phone, and the local input device includes at least one of a handheld controller, a hand tracking device, a touch device, a mouse, a keyboard, and a joystick.