Remote control robot system under high delay intercontinental network and control method

Abstract

This invention relates to the field of robot remote control technology, specifically to a remote-controlled robot system and control method under high-latency intercontinental networks. The system includes: acquiring operator input to generate remote control command data packets and writing them into operator-side perception data to form perception-marked command data packets; receiving the perception-marked command data packets and acquiring robot-side execution data and safety boundary data; calculating the state offset between the operator-side perception data and the robot-side execution data to generate command verification results, and converting them into local control commands according to control reconstruction rules; after executing the local control commands, verifying the consistency between the robot-side feedback data and the operator-side displayed update data, and adjusting the execution permission level of subsequent remote control command data packets accordingly. This invention achieves a technical method for matching remote control commands with the current execution state of the robot in a high-latency intercontinental network environment.

Description

Technical Field

[0001] This invention relates to the field of robot remote control technology, specifically to a remotely controlled robot system and control method under a high-latency intercontinental network. Background Technology

[0002] With the increasing application of remote robot control technology in multinational industrial inspection, overseas equipment maintenance, hazardous environment operations, remote emergency response, and unmanned operation and maintenance scenarios, a remote control relationship is gradually forming between operators and robots across regions, networks, and time zones. The remote control terminal generates control input based on the display status returned by the robot, and the robot drives the actuator to complete movement, posture adjustment, robotic arm movements, or end-effector operations based on the received remote control commands. In this type of system, the issuance of control commands, robot status feedback, operator display updates, and robot local execution together constitute a remote control closed loop. The stability of the control closed loop mainly depends on the matching relationship between the remote control commands and the real-time execution status of the robot, as well as the consistency between the robot's feedback status and the operator's display status. In the remote control of robots under high-latency intercontinental networks, there are also problems such as misalignment between remote control commands and the current execution conditions of the robot, and the continuous impact of delayed feedback on subsequent control inputs from the operator. Specifically, the issues are as follows: On the one hand, when the operator issues remote control commands, it is based on the robot status displayed on the remote control terminal. After being transmitted back through the intercontinental network, this displayed status is already lagging behind the robot's actual status. By the time the remote control command reaches the robot, the robot may have already undergone changes in pose, motion state, actuator state, or task stage, resulting in a mismatch between the original remote control command and the current execution conditions of the robot. This can easily lead to excessive forward movement, posture deviation, robotic arm overstepping boundaries, premature action of the end effector, or path deviation. On the other hand, there is a time misalignment between the feedback data from the robot after it performs a control action and the updated display data from the remote control terminal. The operator may continue to input subsequent control actions based on an outdated or distorted display state, causing the robot to continuously receive repeated correction commands, reverse correction commands, or excessive control commands based on delayed feedback. This can lead to control oscillations, action suspension, permission mismatch, or frequent triggering of safety hold in the remote control closed loop. Summary of the Invention

[0003] The purpose of this invention is to provide a remote-controlled robot system and control method under high-latency intercontinental networks, so as to solve the problems mentioned in the background art, such as the misalignment between the remote control command and the current execution conditions of the robot, and the continuous impact of delayed feedback on subsequent control inputs from the operator.

[0004] To achieve the above objectives, the invention aims to provide a remotely controlled robot system operating over a high-latency intercontinental network, comprising: The remote control terminal is used to acquire the operator's control input, generate remote control instruction data packets, and send the remote control instruction data packets to the robot's controller. The robot end controller is used to receive remote control command data packets, obtain robot end execution data, and drive the robot actuator according to local control commands. The sensing and marking unit is used to write the sensing data from the operating end into the remote control command data packet when the remote control command data packet is generated, thus forming a sensing and marking command data packet. The instruction verification unit is used to read the operator end perception data in the perception mark instruction data packet and obtain the robot end execution data. It performs state offset calculation on the operator end perception data and the robot end execution data to obtain instruction offset data. The instruction offset data and safety boundary data are input into the instruction verification table to generate instruction verification results. The control reconfiguration unit is used to determine the control reconfiguration rules from the control reconfiguration rule table based on the instruction verification results, manipulation action data and robot-side execution data, and convert the perception tag instruction data packet into local control instructions according to the control reconfiguration rules. The feedback correction unit is connected to both the robot-end controller and the remote control terminal. It is used to acquire robot-end feedback data after the robot executes local control commands, and to acquire corresponding operator terminal display update data from the remote control terminal. It performs feedback consistency verification on the robot-end feedback data and the operator terminal display update data, generates feedback verification results, and adjusts the execution permission level of the remote control command data packet based on the feedback verification results.

[0005] Preferably, the robot end controller includes an instruction receiving module, an execution data acquisition module, a safety boundary acquisition module, and a local drive module; wherein, the instruction receiving module is used to receive perception tag instruction data packets and generate reception timing data; the execution data acquisition module is used to synchronously acquire robot end execution data when generating reception timing data, the robot end execution data including robot motion state data, actuator state data, and task stage data; the safety boundary acquisition module is used to synchronously acquire safety boundary data when generating reception timing data, the safety boundary data including robot motion boundary data, actuator action boundary data, and task stage allowable boundary data; the local drive module is used to drive the robot actuator according to local control instructions, robot end execution data, and safety boundary data.

[0006] Preferably, the perception marking unit includes a perception state reading module, an instruction timing marking module, a same-packet binding module, and a mark integrity verification module; wherein, the perception state reading module is used to read operator-end perception data from the display cache data of the remote control terminal when the remote control instruction data packet is generated, the operator-end perception data includes the operator-end display state identifier and the operator-end task stage data; the instruction timing marking module is used to associate the display timing data corresponding to the operator-end perception data with the instruction timing data in the remote control instruction data packet to generate perception timing association data; the same-packet binding module is used to write the operator-end perception data, perception timing association data, control action data, and instruction timing data into the same remote control instruction data packet to form a perception mark instruction data packet; the mark integrity verification module is used to perform field integrity verification on the operator-end perception data, perception timing association data, control action data, and instruction timing data in the perception mark instruction data packet, and send the perception mark instruction data packet that passes the field integrity verification to the robot-end controller.

[0007] Preferably, the instruction verification unit includes a timing offset calculation module, a motion offset calculation module, a stage offset calculation module, and an offset data generation module. The timing offset calculation module generates timing offset data based on the perception timing association data in the perception marker instruction data packet and the received timing data generated by the robot end controller. The motion offset calculation module generates motion offset data based on the operator end display status identifier in the operator end perception data and the robot motion status data in the robot end execution data. The stage offset calculation module generates stage offset data based on the operator end task stage data in the operator end perception data and the task stage data in the robot end execution data. The offset data generation module performs field aggregation and encoding encapsulation on the timing offset data, motion offset data, and stage offset data to generate instruction offset data and sends the instruction offset data to the instruction verification table.

[0008] Preferably, the instruction verification table includes an offset index field, a safety boundary index field, and an execution code field; wherein, the offset index field is used to receive instruction offset data and establish a verification index according to timing offset data, motion offset data, and stage offset data; the safety boundary index field is used to receive safety boundary data and match the safety boundary data with the verification index to form an instruction execution qualification determination item; the execution code field is used to output the instruction verification result according to the instruction execution qualification determination item, and the instruction verification result includes direct execution code, downgraded execution code, corrected execution code, suspended execution code, and invalid code; the direct execution code, downgraded execution code, corrected execution code, suspended execution code, and invalid code respectively correspond to different control reconstruction entry points in the control reconstruction unit.

[0009] Preferably, the control reconfiguration unit includes a rule matching module, a control parameter generation module, an instruction encapsulation module, and a boundary verification module. The rule matching module receives instruction verification results, manipulation action data, and robot-side execution data, and determines control reconfiguration rules in the control reconfiguration rule table using the instruction verification results as a first matching index, the manipulation action data as a second matching index, and the robot-side execution data as a third matching index. The control parameter generation module performs localized conversion on the manipulation action data according to the control reconfiguration rules, generating robot motion control data, actuator control data, and motion timing data. The instruction encapsulation module encapsulates the robot motion control data, actuator control data, motion timing data, and safety constraint data to generate local control instructions. The boundary verification module performs boundary verification on the local control instructions based on safety boundary data and sends the boundary-verified local control instructions to the local drive module of the robot-side controller.

[0010] Preferably, the control reconfiguration rule table includes direct execution rules, degraded execution rules, corrective execution rules, suspended execution rules, and failure handling rules; wherein, the degraded execution rule is used to limit the control action data based on safety boundary data and generate a local control instruction containing safety restriction data when the instruction verification result is a degraded execution code; the corrective execution rule is used to perform state adaptation processing on the control action data based on the robot-side execution data and generate a local control instruction corresponding to the robot-side execution data when the instruction verification result is a corrective execution code; the suspended execution rule is used to generate a local control instruction that maintains the current execution state when the instruction verification result is a suspended execution code; and the failure handling rule is used to generate a processing identifier to terminate the execution path of the current remote control instruction data packet when the instruction verification result is a failure code.

[0011] Preferably, the feedback correction unit includes a feedback data synchronization module, a feedback state normalization module, a feedback difference calculation module, and a feedback result generation module; wherein, the feedback data synchronization module is used to perform timing alignment of robot-side feedback data and operator-side display update data according to the action timing data corresponding to the local control command, and generate feedback alignment data; the feedback state normalization module is used to convert robot-side feedback data and operator-side display update data into unified state fields based on the feedback alignment data, and generate a feedback state field group; the feedback difference calculation module is used to perform field difference calculation on the robot-side feedback fields and operator-side display update fields in the feedback state field group, and generate feedback difference data; the feedback result generation module is used to input the feedback difference data into a feedback verification table, generate a feedback verification result, and send the feedback verification result to the execution permission level adjustment path.

[0012] Preferably, the execution permission level adjustment path includes a continuous control permission path, a limited control permission path, a staged control permission path, and a safety maintenance permission path; wherein, the continuous control permission path is used to allow subsequent remote control instruction data packets to enter the state offset calculation path of the instruction verification unit when the feedback verification result meets the continuous control condition; the limited control permission path is used to write the control action data in the subsequent remote control instruction data packets into the downgrade execution rule of the control reconstruction unit when the feedback verification result meets the limited control condition; the staged control permission path is used to limit the subsequent remote control instruction data packets to the task stage data corresponding to the robot-side execution data for verification and reconstruction when the feedback verification result meets the staged control condition; the safety maintenance permission path is used to enable the control reconstruction unit to generate local control instructions to maintain the current execution state when the feedback verification result meets the safety maintenance condition.

[0013] On the other hand, the present invention provides a remotely controlled robot control method under a high-latency intercontinental network, used in the aforementioned remotely controlled robot system under a high-latency intercontinental network, comprising the following steps: S10.1 Obtain the control input data formed by the operator's input, generate a remote control instruction data packet based on the control input data, and write the operator's perception data when the remote control instruction data packet is generated to form a perception mark instruction data packet. S10.2 Receive the perception tag instruction data packet, obtain the robot-side execution data and safety boundary data, and read the operator-side perception data in the perception tag instruction data packet; S10.3 Calculate the state offset between the sensing data from the operator end and the execution data from the robot end to obtain the instruction offset data. Input the instruction offset data and safety boundary data into the instruction verification table to generate the instruction verification result. Based on the instruction verification result, the control input data and the execution data from the robot end, determine the control reconstruction rule from the control reconstruction rule table. Based on the control reconstruction rule, convert the sensing tag instruction data packet into a local control instruction. S10.4 Drive the robot actuator according to the local control command, obtain the robot end feedback data and the operator end display update data after the robot executes the local control command, perform feedback consistency verification on the robot end feedback data and the operator end display update data, generate feedback verification result, and adjust the execution permission level of the remote control command data packet according to the feedback verification result.

[0014] Compared with the prior art, the above-mentioned technical solution of the present invention has the following beneficial technical effects: 1. In this invention, based on the state offset verification between the operator-end perception data and the robot-end execution data, it is possible to determine whether the remote control command is still suitable for the current robot state after it arrives at the robot end, thereby avoiding the robot directly executing the original control command that has been delayed and reducing the risk of excessive forward movement, posture deviation, path deviation and actuator malfunction under high latency conditions. 2. In this invention, the local control instruction is reconstructed by driving the instruction verification result, and the subsequent execution permission level is adjusted by combining feedback consistency verification. This enables remote control instructions to be converted according to the current execution conditions of the robot, reducing repeated corrections, reverse corrections and excessive control caused by the operator's display lag on the control terminal, and improving the control stability of high-latency intercontinental remote control processes. Attached Figure Description

[0015] Figure 1 This is a schematic diagram of one embodiment of the present invention; Reference numerals in the attached diagram: 1. Remote control terminal; 2. Robot end controller; 3. Sensing and marking unit; 4. Command verification unit; 5. Control reconfiguration unit; 6. Feedback correction unit. Detailed Implementation

[0016] Example 1, as Figure 1 As shown, a remotely controlled robot system for high-latency intercontinental networks is provided, comprising: Remote control terminal 1 is used to acquire the operator's control input, generate remote control instruction data packets, and send the remote control instruction data packets to the robot end controller 2. Robot end controller 2 is used to receive remote control instruction data packets, obtain robot end execution data, and drive the robot actuator according to local control instructions. The sensing tagging unit 3 is used to write the sensing data of the operation terminal into the remote control command data packet when the remote control command data packet is generated, so as to form a sensing tagging command data packet. Command verification unit 4 is used to read the operator end perception data in the perception mark command data packet and obtain the robot end execution data. It performs state offset calculation on the operator end perception data and the robot end execution data to obtain command offset data. It inputs the command offset data and safety boundary data into the command verification table to generate command verification results. The control reconfiguration unit 5 is used to determine the control reconfiguration rules from the control reconfiguration rule table based on the instruction verification results, manipulation action data and robot end execution data, and convert the perception tag instruction data packet into local control instructions according to the control reconfiguration rules. Feedback correction unit 6 is connected to robot end controller 2 and remote control terminal 1 respectively. It is used to obtain robot end feedback data after the robot executes local control commands, and obtain the corresponding operation terminal display update data of remote control terminal 1. It performs feedback consistency verification on robot end feedback data and operation terminal display update data, generates feedback verification result, and adjusts the execution permission level of remote control command data packet according to feedback verification result.

[0017] In this embodiment, the remote control terminal 1 serves as an input device for the operator to initiate remote control actions. The remote control terminal 1 can be configured as a remote control console, portable control terminal, industrial workstation, or control computing device with a human-machine interface. The remote control terminal 1 receives control input from the operator via a direction controller, attitude controller, button assembly, touch assembly, force feedback hand, or a combination of control devices. The control input is converted into control action data in a unified format within the remote control terminal 1. The control action data characterizes the corresponding action category, action direction, action amplitude, action duration, and action triggering state. Action categories include movement actions and postures. The data includes adjustment actions, actuator actions, stop actions, and hold actions; the action direction indicates the spatial or joint direction corresponding to the control input; the action amplitude indicates the velocity, displacement, angle, or actuator action corresponding to the control input; the action duration indicates the time range from triggering to ending the control input; the action trigger state indicates whether the control input is pressed, continuous, released, or interrupted; the remote control terminal 1 samples, debounces, converts formats, and normalizes fields for control inputs from different sources, so that both continuous and discrete control inputs are organized into the same control action data structure, facilitating the generation of remote control command data packets according to unified fields.

[0018] In this embodiment, after generating control action data, the remote control terminal 1 generates a remote control instruction data packet based on the control action data. The remote control instruction data packet includes at least instruction identification data, control action data, and instruction timing data. The instruction identification data is used to distinguish different remote control instruction data packets in the same control task and to record the generation order of the remote control instruction data packets. The control action data is used to carry the action content formed by the operator's input. The instruction timing data is used to record the generation time, encapsulation time, and transmission time of the remote control instruction data packet on the remote control terminal 1 side. The remote control terminal 1 encapsulates the instruction identification data, control action data, and instruction timing data according to a preset encapsulation format. The sequence data is encapsulated; the preset encapsulation format includes a header field, an instruction identifier field, an action data field, a timing data field, and a field verification field; the header field is used to indicate the data packet type; the instruction identifier field is used to write instruction identifier data; the action data field is used to write control action data; the timing data field is used to write instruction timing data; the field verification field is used to record the field integrity verification results of the remote control instruction data packet; after the remote control terminal 1 completes the encapsulation, it performs field integrity checks, instruction identifier duplication checks, and action data validity checks on the remote control instruction data packet, so that the remote control instruction data packet has a definite data structure and identifiable control content before being sent.

[0019] In this embodiment, before sending remote control command data packets, the remote control terminal 1 performs sending queue management on the remote control command data packets. The sending queue management includes enqueue processing, sequence marking, priority marking, and dequeue processing. Enqueue processing is used to write the encapsulated remote control command data packets into the sending queue. Sequence marking is used to record the arrangement position of the remote control command data packets in the sending queue according to the generation order of the remote control command data packets. Priority marking is used to determine the sending priority of the remote control command data packets according to the action category corresponding to the control action data. Dequeue processing is used to output the remote control command data packets to the sending interface according to the sending priority and generation order. Remote control command data packets corresponding to stop and hold actions are configured to have a higher sending priority than movement actions, attitude adjustment actions, and actuator action actions. During the sending process, the remote control terminal 1 records the dequeue time and sending status of the remote control command data packets. The sending status includes the waiting to be sent status, sent status, sending interruption status, and retransmission record status. Through the above sending queue management process, the remote control terminal 1 enables multiple remote control command data packets in the same control task to be output in a determined order and with a determined priority, avoiding the chaos of the sending order when multiple control inputs are generated continuously.

[0020] In this embodiment, the remote control terminal 1 is also used to provide a display interface for the operator to use when making control inputs; the display interface is used to display robot feedback images, robot pose prompts, robot motion prompts, task stage prompts, actuator status prompts, and operation prompts; the remote control terminal 1 writes the display data in the display interface into a display buffer; the display buffer stores display status data according to the display frame sequence and configures display timing data for each display status data; the display status data is used to represent the robot display status visible to the operator on the remote control terminal 1; the display timing data is used to represent the time when the display status data enters the display buffer and is called by the display interface on the remote control terminal 1 side; when the operator generates control input based on the display interface, the remote control terminal 1 can record the display status data and display timing data corresponding to the formation of the control input; the display status data and display timing data are stored in the cache data area of ​​the remote control terminal 1; the cache data area has a data call relationship with the remote control command data packet generation process; through the above display caching process, the remote control terminal 1 can retain the display basis corresponding to the operator's control input, so that subsequent remote control command data packets can be associated with the display status when the control input occurs.

[0021] In this embodiment, the robot end controller 2 includes an instruction receiving module, an execution data acquisition module, a safety boundary acquisition module, and a local drive module. The instruction receiving module receives perception tag instruction data packets and generates reception timing data. The execution data acquisition module acquires robot end execution data synchronously while generating the reception timing data; the robot end execution data includes robot motion state data, actuator state data, and task stage data. The safety boundary acquisition module acquires safety boundary data synchronously while generating the reception timing data; the safety boundary data includes robot motion boundary data, actuator action boundary data, and task stage allowable boundary data. The local drive module drives the robot actuator based on local control instructions, robot end execution data, and safety boundary data.

[0022] In this embodiment, the robot end controller 2 serves as the robot-side control processing device, used to receive perception tag instruction data packets sent by the remote control terminal and after perception tagging is completed. The robot end controller 2 is equipped with an instruction receiving module. After receiving the perception tag instruction data packet, the instruction receiving module performs header recognition, instruction identifier reading, field integrity checking, and reception sequence registration on the perception tag instruction data packet. Upon completion of reception registration, it generates reception timing data, which includes the reception time, reception sequence number, and reception status of the perception tag instruction data packet received by the robot end. The reception sequence number is used to distinguish multiple perception tag instruction data packets arriving consecutively in the same control task. The reception status records whether the perception tag instruction data packet is in the state of received, pending processing, or abnormal reception. The instruction receiving module establishes an association between the reception timing data and the corresponding perception tag instruction data packet, enabling subsequent robot end execution data to correspond to the instruction reception status at the same reception time. The robot end controller 2 is configured to acquire execution data. The execution data acquisition module synchronously acquires robot-side execution data while the instruction receiving module generates the receiving timing data. This robot-side execution data includes robot motion state data, actuator state data, and task stage data. The robot motion state data represents the robot's pose, velocity, direction of motion, and attitude state when receiving the perception-marking instruction data packet. The actuator state data represents the action state, drive state, and operable state of the robot chassis, robotic arm, joints, gripper, end effector, or other actuators at the same receiving moment. The task stage data represents the robot's current task stage, such as approach, alignment, contact, operation, withdrawal, or holding. The execution data acquisition module reads this data through the robot body control bus, actuator drive interface, sensor state buffer, and task state register, and synchronously marks the reading time with the receiving timing data, ensuring that the robot-side execution data reflects the robot's actual execution state when the perception-marking instruction data packet arrives at the robot side.

[0023] In this embodiment, the robot end controller 2 is equipped with a safety boundary acquisition module. The safety boundary acquisition module is used to synchronously acquire safety boundary data when generating and receiving timing data. The safety boundary data includes robot motion boundary data, actuator action boundary data, and task stage permitted boundary data. The robot motion boundary data is used to represent the range of movement, speed, and posture changes that the robot is allowed to execute in its current position, current posture, and current task area. The actuator action boundary data is used to represent the amplitude, speed, angle, and duration of the action that the actuator is allowed to execute in the current driving state. The task stage permitted boundary data is used to represent the types of actions that are allowed to be triggered and the types of actions that are prohibited from being triggered in the current task stage. The safety boundary acquisition module reads the safety boundary data from the robot's local safety configuration table, the task stage constraint table, the actuator limit table, and the field area constraint data, and provides the safety boundary data and the robot end execution data together to the subsequent control processing, so that the robot end controller 2 can obtain the corresponding execution state constraints after receiving remote control commands.

[0024] In this embodiment, the robot end controller 2 is equipped with a local drive module. The local drive module receives local control commands and drives the robot actuator based on these commands, robot end execution data, and safety boundary data. The local control commands include motion control data for driving robot movement or posture adjustment, actuator control data for driving actuator actions, motion timing data for limiting action execution time, and safety constraint data for limiting the action range. When executing local control commands, the local drive module first matches the motion control data and actuator control data in the local control commands with the robot end execution data. Then, it constrains the motion amplitude, motion direction, motion duration, and actuator action range based on the safety boundary data. Subsequently, it outputs the constrained drive control quantities to the corresponding robot actuator. The robot actuator includes a mobile chassis, robotic arm joints, an end effector, a gripper, or a special working tool. The local drive module... After outputting the drive control quantity, the start time, execution status, and execution end status of the local control instruction are recorded so that the robot end controller 2 can form a status record corresponding to the local execution process. The robot end controller 2 performs unified scheduling of the instruction receiving module, execution data acquisition module, safety boundary acquisition module, and local drive module. After the instruction receiving module completes the reception of the perception mark instruction data packet and generates the reception timing data, the execution data acquisition module and the safety boundary acquisition module acquire the robot end execution data and safety boundary data respectively with the reception timing data as the synchronization reference. After receiving the local control instruction, the local drive module uses the robot end execution data as the basis for the current execution status and the safety boundary data as the basis for the action constraints to drive and control the robot actuator. Through the above processing, the robot end controller 2 makes the reception time of the perception mark instruction data packet, the robot end execution status, the safety boundary conditions, and the local drive process form a corresponding relationship in the same control link.

[0025] In this embodiment, the perception marking unit 3 includes a perception state reading module, an instruction timing marking module, a same-packet binding module, and a marking integrity verification module. The perception state reading module reads operator-end perception data from the display cache data of the remote control terminal 1 when the remote control instruction data packet is generated. The operator-end perception data includes the operator-end display state identifier and the operator-end task stage data. The instruction timing marking module associates the display timing data corresponding to the operator-end perception data with the instruction timing data in the remote control instruction data packet to generate perception timing association data. The same-packet binding module writes the operator-end perception data, perception timing association data, control action data, and instruction timing data into the same remote control instruction data packet to form a perception marking instruction data packet. The marking integrity verification module performs field integrity verification on the operator-end perception data, perception timing association data, control action data, and instruction timing data in the perception marking instruction data packet and sends the perception marking instruction data packet that passes the field integrity verification to the robot end controller 2.

[0026] In this embodiment, the sensing tagging unit 3 is used to establish a data correspondence between control action data and operator terminal sensing data during the remote control command data packet generation stage. The sensing tagging unit 3 is set on the remote control terminal 1 side and is connected to the display cache data and command generation process of the remote control terminal 1. When the remote control command data packet is formed, the sensing tagging unit 3 synchronously reads the display cache data that the remote control terminal 1 is calling when the operator generates control input, and extracts operator terminal sensing data from the display cache data. The operator terminal sensing data includes operator terminal display status identifier and operator terminal task stage data. The operator terminal display status identifier is used to mark the corresponding robot display status in the display interface when the operator issues control input. The operator terminal task stage data is used to mark the corresponding robot task stage in the display interface when the operator issues control input. The task stage includes at least one stage type among approach, alignment, contact, operation, withdrawal and holding. Through the above processing, the sensing tagging unit 3 enables the remote control command data packet to carry the source of the display status on which the operator generates control input.

[0027] In this embodiment, the perception state reading module is used to read the display cache data of the remote control terminal 1 when the remote control command data packet is generated. The display cache data is stored according to the display frame sequence. Each display cache data corresponds to the configuration of display timing data. The display timing data includes the time when the display cache data enters the display cache area, the time when the display cache data is called by the display interface, and the display sequence number corresponding to the display cache data. The perception state reading module uses the generation time of the remote control command data packet as the reading reference to determine the display cache data corresponding to the time when the control input is formed from the display cache data, and reads the operation terminal display status identifier and operation terminal task stage data from the display cache data. The operation terminal display status identifier can be represented by a display frame identifier, status frame identifier, or display status number. The operation terminal task stage data can be represented by a task stage code, thereby giving the operation terminal perception data a definite data source and a definite timing position.

[0028] In this embodiment, the instruction timing marker module is used to associate the display timing data corresponding to the operator's perceived data with the instruction timing data in the remote control instruction data packet to generate perceived timing association data. The instruction timing data includes the generation time, encapsulation time, and pre-send time of the remote control instruction data packet. The display timing data includes the display call time and display sequence number corresponding to the operator's perceived data. The instruction timing marker module establishes an association between the display call time and the instruction generation time, and establishes an association between the display sequence number and the instruction identification data. The associated data is then written into the perceived timing association data. The perceived timing association data is used to represent the timing correspondence between the display state on which the operator generates control input and the remote control instruction data packet, so that subsequent processing can identify the display basis when the remote control instruction data packet is generated based on the same perceived timing association data.

[0029] In this embodiment, the same-packet binding module is used to write the operator-end perception data, perception timing association data, control action data, and instruction timing data into the same remote control instruction data packet to form a perception marker instruction data packet. The same-packet binding module writes the operator-end display status identifier, operator-end task stage data, the association result between display timing data and instruction timing data, control action data, and instruction timing data into the corresponding fields of the remote control instruction data packet according to a preset field order. This ensures that the same perception marker instruction data packet simultaneously contains the operator's input control action content, the display status corresponding to the generation of the control action content, and the timing correspondence between the two. During the writing process, the same-packet binding module keeps the instruction identifier data of the remote control instruction data packet unchanged and binds the newly added operator-end perception data and perception timing association data to the instruction identifier data. This avoids perception status mismatch in multiple remote control instruction data packets generated consecutively in the same control task.

[0030] In this embodiment, the mark integrity verification module is used to perform field integrity verification on the perception mark instruction data packet. The field integrity verification includes field existence verification, field format verification, and field correspondence verification on the operator terminal perception data, perception timing association data, control action data, and instruction timing data. The field existence verification is used to confirm that the perception mark instruction data packet contains the operator terminal display status identifier, operator terminal task stage data, perception timing association data, control action data, and instruction timing data. The field format verification is used to confirm that each field conforms to the preset data format. The field correspondence verification is used to confirm that the operator terminal perception data and perception timing association data correspond to the instruction identifier data of the same remote control instruction data packet. The mark integrity verification module generates a mark pass status for the perception mark instruction data packet that passes the field integrity verification and sends the perception mark instruction data packet with the mark pass status to the robot end controller 2.

[0031] In this embodiment, the perception marking unit 3 sequentially completes the display status reading, timing association, same-packet writing, and field verification processing through the perception status reading module, instruction timing marking module, same-packet binding module, and marking integrity verification module. This transforms the remote control instruction data packet into a perception marking instruction data packet before it leaves the remote control terminal 1. The perception marking instruction data packet simultaneously retains the control action data, instruction timing data, operator end perception data, and perception timing association data. This enables the robot end controller 2 to identify the operator end display status and display timing basis corresponding to the control action data after receiving the perception marking instruction data packet, providing a unified data foundation for subsequent state offset calculation based on operator end perception data and robot end execution data.

[0032] In this embodiment, the instruction verification unit 4 includes a timing offset calculation module, a motion offset calculation module, a stage offset calculation module, and an offset data generation module. The timing offset calculation module generates timing offset data based on the perception timing association data in the perception mark instruction data packet and the received timing data generated by the robot end controller 2. The motion offset calculation module generates motion offset data based on the operator end display status identifier in the operator end perception data and the robot motion status data in the robot end execution data. The stage offset calculation module generates stage offset data based on the operator end task stage data in the operator end perception data and the task stage data in the robot end execution data. The offset data generation module performs field collection and encoding encapsulation on the timing offset data, motion offset data, and stage offset data to generate instruction offset data and sends the instruction offset data to the instruction verification table.

[0033] In this embodiment, the instruction verification unit 4 is used to calculate the state difference between the operation terminal perception data and the robot terminal execution data carried in the perception mark instruction data packet after the perception mark instruction data packet arrives at the robot terminal, and generate instruction offset data for entering the instruction verification table. The instruction verification unit 4 includes a timing offset calculation module, a motion offset calculation module, a stage offset calculation module, and an offset data generation module. The timing offset calculation module reads the perception timing association data in the perception mark instruction data packet and reads the receiving timing data generated by the robot terminal controller 2. It matches the display timing data, instruction timing data and receiving timing data in the perception timing association data with the receiving timing data to obtain the timing difference between the remote control instruction data packet from the operation terminal display basis to the robot terminal reception, and forms timing offset data. The timing offset data is used to represent the time relationship between the display time corresponding to the operator forming the control input, the remote control instruction data packet generation time and the robot terminal reception time, so that the subsequent verification process can identify whether the control action data corresponding to the perception mark instruction data packet comes from the lagging operation terminal display state.

[0034] In this embodiment, the motion offset calculation module is used to generate motion offset data based on the operator terminal display status identifier in the operator terminal perception data and the robot motion state data in the robot terminal execution data. The operator terminal display status identifier corresponds to the robot display status record in the display cache data of the remote control terminal 1. The robot display status record includes the robot's display pose, display motion direction, and display motion state in the operator terminal display interface. The robot motion state data includes the robot's actual pose, actual motion direction, and actual motion state when the robot receives the perception marker instruction data packet. The motion offset calculation module maps the robot display status record corresponding to the operator terminal display status identifier to the robot motion state data, and calculates the differences between the display pose and the actual pose, the display motion direction and the actual motion direction, and the display motion state and the actual motion state to generate motion offset data. The motion offset data is used to represent the difference between the robot display motion state on which the operator forms the control input and the actual motion state when the robot receives the instruction. Stage offset calculation. The module generates stage offset data based on the task stage data in the operator's perception data and the task stage data in the robot's execution data. The operator's task stage data represents the corresponding task stage displayed on the remote control terminal 1 when the operator generates control input. The task stage data in the robot's execution data represents the actual task stage corresponding to the robot receiving the perception marker instruction data packet. The stage offset calculation module compares the stage positions of the operator's task stage data and the robot's execution data according to a preset task stage sequence, which includes approach, alignment, contact, operation, withdrawal, and hold. When the operator's task stage data and the robot's execution data are at the same stage position, the stage offset data is recorded as the same stage offset result. When they are at different stage positions, the stage offset data records the stage interval and stage direction between them. The stage offset data represents the degree of misalignment between the task stage data generated by the remote control instruction data packet and the actual task stage of the robot.

[0035] In this embodiment, the offset data generation module is used to collect and encode the time-series offset data, motion offset data, and stage offset data to generate instruction offset data. The field collection process writes the time-series offset data into the time-series offset field of the instruction offset data, the motion offset data into the motion offset field of the instruction offset data, and the stage offset data into the stage offset field of the instruction offset data. It also writes the instruction identifier data from the perception marker instruction data packet into the associated field of the instruction offset data. The encoding and encapsulation process uniformly encodes the above fields according to the input format of the instruction verification table, enabling the instruction offset data to be read by the offset index field in the instruction verification table. After generating the instruction offset data, the offset data generation module sends the instruction offset data to the instruction verification table, allowing the instruction verification table to generate the corresponding instruction verification result in conjunction with the security boundary data.

[0036] In this embodiment, the instruction verification table includes an offset index field, a security boundary index field, and an execution code field. The offset index field receives instruction offset data and establishes a verification index based on timing offset data, motion offset data, and stage offset data. The security boundary index field receives security boundary data and matches it with the verification index to form an instruction execution qualification determination item. The execution code field outputs the instruction verification result based on the instruction execution qualification determination item. The instruction verification result includes direct execution code, downgraded execution code, corrected execution code, suspended execution code, and invalidation code. The direct execution code, downgraded execution code, corrected execution code, suspended execution code, and invalidation code correspond to different control reconstruction entry points in the control reconstruction unit 5.

[0037] In this embodiment, the instruction verification table is used to determine the instruction verification result corresponding to the perception marker instruction data packet based on the instruction offset data and the security boundary data. The instruction verification table includes an offset index field, a security boundary index field, and an execution encoding field. The offset index field is used to receive the instruction offset data and establish a verification index according to the timing offset data, motion offset data, and stage offset data in the instruction offset data. The verification index is used to represent the offset combination state of the current perception marker instruction data packet in the three dimensions of timing, motion, and task stage. When establishing the verification index, the offset index field uses the timing offset data as the first index item, the motion offset data as the second index item, and the stage offset data as the third index item, and binds the three types of index items together. Under the same instruction identifier data, the multi-dimensional offset states corresponding to the same perception marker instruction data packet can enter the same verification path; the safety boundary index field is used to receive safety boundary data and match the safety boundary data with the verification index established by the offset index field to form an instruction execution qualification judgment item; the safety boundary data includes robot motion boundary data, actuator action boundary data, and task stage allowed boundary data; the robot motion boundary data is used to limit the allowed motion range in the current robot end execution state; the actuator action boundary data is used to limit the allowed motion range in the current actuator state; the task stage allowed boundary data is used to limit the allowed action category corresponding to the current task stage data; the safety index field will... The aforementioned safety boundary data is matched against the timing offset data, motion offset data, and stage offset data in the verification index, and an instruction execution eligibility determination item is generated based on the matching results. This item indicates the processable category of the current perception-marked instruction data packet under the current execution state and safety boundary conditions on the robot. The execution encoding field outputs the instruction verification result based on the instruction execution eligibility determination item. The instruction verification result includes direct execution encoding, downgraded execution encoding, corrected execution encoding, suspended execution encoding, and failure encoding. When the instruction execution eligibility determination item indicates that the instruction offset data is within the allowed execution range, and the safety boundary data allows the current manipulation action data to be executed, the execution encoding field outputs the direct execution encoding. The execution code field outputs a downgraded execution code when the instruction execution eligibility judgment item indicates that the control action data can enter the execution path but is subject to limitations on action amplitude, action speed, or action range; when the instruction execution eligibility judgment item indicates that there is a state deviation between the control action data and the robot-side execution data, but the data can enter the execution path through state adaptation, the execution code field outputs a corrected execution code; when the instruction execution eligibility judgment item indicates that the control action data needs to wait for the robot-side task stage or the actuator state to meet the processing conditions, the execution code field outputs a suspended execution code; when the instruction execution eligibility judgment item indicates that the control action data does not match the current safety boundary data or the task stage allowed boundary data, the execution code field outputs a failure code.

[0038] In this embodiment, direct execution code, downgraded execution code, corrected execution code, suspended execution code, and failure code correspond to different control reconstruction entry points in the control reconstruction unit 5; direct execution code corresponds to the entry point of the direct execution rule; downgraded execution code corresponds to the entry point of the downgraded execution rule; corrected execution code corresponds to the entry point of the corrected execution rule; suspended execution code corresponds to the entry point of the suspended execution rule; failure code corresponds to the entry point of the failure handling rule; the instruction verification table converts the instruction execution qualification judgment item into an instruction verification result that can be recognized by the control reconstruction unit 5 through the execution code field, so that the control reconstruction unit 5 can determine the subsequent control reconstruction rule based on the same instruction verification result, and convert the perception mark instruction data packet into the corresponding local control instruction or processing identifier.

[0039] In this embodiment, the control reconfiguration unit 5 includes a rule matching module, a control parameter generation module, an instruction encapsulation module, and a boundary verification module. The rule matching module receives instruction verification results, manipulation action data, and robot-side execution data, and determines control reconfiguration rules in the control reconfiguration rule table using the instruction verification results as a first matching index, the manipulation action data as a second matching index, and the robot-side execution data as a third matching index. The control parameter generation module performs localized conversion on the manipulation action data according to the control reconfiguration rules, generating robot motion control data, actuator control data, and action timing data. The instruction encapsulation module encapsulates the robot motion control data, actuator control data, action timing data, and safety restriction data to generate local control instructions. The boundary verification module performs boundary verification on the local control instructions based on safety boundary data and sends the boundary-verified local control instructions to the local drive module of the robot-side controller 2.

[0040] In this embodiment, the control reconstruction unit 5 is used to convert the perception marker instruction data packet into a local control instruction that the robot can directly execute, based on the instruction verification result, the manipulation action data, and the robot-side execution data. The control reconstruction unit 5 includes a rule matching module, a control parameter generation module, an instruction encapsulation module, and a boundary verification module. The rule matching module receives the instruction verification result output by the instruction verification table and reads the manipulation action data in the perception marker instruction data packet, while simultaneously reading the robot-side execution data obtained by the robot-side controller 2. The rule matching module uses the instruction verification result as the first matching index, the manipulation action data as the second matching index, and the robot-side execution data as the third matching index. The index determines the corresponding control reconstruction rule in the control reconstruction rule table. The first matching index is used to determine the processing entry point for the remote control instruction data packet, which is either direct execution, degraded execution, modified execution, suspended execution, or failure handling. The second matching index is used to determine the action category, action direction, action amplitude, and action duration corresponding to the control action data. The third matching index is used to determine the motion state, actuator state, and task stage state of the robot when receiving the perception marker instruction data packet. Through the above three matching indices, the rule matching module ensures that the control reconstruction rule output by the control reconstruction rule table is simultaneously constrained by the instruction verification result, the control action content, and the current execution state of the robot.

[0041] In this embodiment, the control parameter generation module is used to perform localized conversion on the manipulation action data according to the control reconstruction rules determined by the rule matching module, generating robot motion control data, actuator control data, and action timing data. The robot motion control data represents the speed, direction, posture, or position adjustment parameters of the robot actuator in the current execution state on the robot end. The actuator control data represents the action parameters of the robotic arm, joint, gripper, end tool, or other actuator. The action timing data represents the start time, duration, and end condition of the local control command execution on the robot end. When performing localized conversion, the control parameter generation module matches the action category in the manipulation action data with the task stage data in the robot end execution data, matches the action amplitude in the manipulation action data with the motion state data in the robot end execution data, and matches the actuator action content in the manipulation action data with the actuator state data in the robot end execution data, so that the remote control action formed by the operator is converted into control data with a definite action object and definite action parameters in the current execution state on the robot end.

[0042] In this embodiment, the instruction encapsulation module encapsulates robot motion control data, actuator control data, motion timing data, and safety restriction data to generate local control instructions. The robot motion control data, actuator control data, and motion timing data are generated by the control parameter generation module. The safety restriction data is generated by the control reconfiguration unit 5 based on control reconfiguration rules, safety boundary data, and robot-side execution data. The local control instructions are encapsulated according to control fields recognizable by the robot-side controller 2. These control fields include a local instruction identifier field, a motion control field, an actuator control field, a motion timing field, a safety restriction field, and an execution status record field. The local instruction identifier field is used to associate the local control instruction with the corresponding perception marker instruction data packet. The motion control field is used to write robot motion control data. The actuator control field is used to write actuator control data. The motion timing field is used to write motion timing data. The safety restriction field is used to write safety restriction data. The execution status record field is used to record the processing status of the local control instruction after it enters the robot-side controller 2. Through the above encapsulation process, the remote control content in the perception marker instruction data packet is converted into local control instructions that can be read by the robot-side local drive link.

[0043] In this embodiment, the boundary verification module is used to perform boundary verification on the local control commands based on safety boundary data. The boundary verification module reads the robot motion control data, actuator control data, motion timing data, and safety limit data from the local control commands, and reads the robot motion boundary data, actuator motion boundary data, and task stage allowable boundary data from the safety boundary data. The boundary verification module matches the robot motion control data with the robot motion boundary data, matches the actuator control data with the actuator motion boundary data, and matches the motion timing data and safety limit data with the task stage allowable boundary data. When all control fields in the local control commands are within the allowable range of the corresponding safety boundary data, the boundary verification module generates a verification pass status and sends the local control commands to the local drive module of the robot end controller 2. When any control field in the local control commands exceeds the allowable range of the corresponding safety boundary data, the boundary verification module generates a verification blockage status and returns the verification blockage status to the rule matching module, so that the rule matching module re-enters the control reconstruction rule table to determine the corresponding downgrade execution rule, correct execution rule, or suspend execution rule.

[0044] In this embodiment, the control reconfiguration unit 5 forms a conversion link from instruction verification results to local control instructions through a rule matching module, a control parameter generation module, an instruction encapsulation module, and a boundary verification module. The rule matching module determines the control reconfiguration rules, the control parameter generation module generates control parameters executable by the robot based on the control reconfiguration rules, the instruction encapsulation module encapsulates the control parameters into local control instructions, and the boundary verification module performs a safety boundary verification on the local control instructions. This conversion link ensures that the perception marker instruction data packet does not directly drive the robot actuator as the original remote control instruction, but is converted into a local control instruction corresponding to the robot's execution data and safety boundary data through the control reconfiguration rule table. This ensures that the control content received by the robot actuator has a clear execution object, action parameters, action sequence, and safety restrictions.

[0045] In this embodiment, the control reconfiguration rule table includes direct execution rules, degraded execution rules, corrective execution rules, suspended execution rules, and failure handling rules. Specifically, the degraded execution rule is used to limit the manipulation action data based on safety boundary data and generate a local control instruction containing safety restriction data when the instruction verification result is a degraded execution code. The corrective execution rule is used to perform state adaptation processing on the manipulation action data based on the robot-side execution data and generate a local control instruction corresponding to the robot-side execution data when the instruction verification result is a corrective execution code. The suspended execution rule is used to generate a local control instruction that maintains the current execution state when the instruction verification result is a suspended execution code. The failure handling rule is used to generate a processing identifier to terminate the execution path of the current remote control instruction data packet when the instruction verification result is a failure code.

[0046] In this embodiment, the control reconstruction rule table is used to determine the corresponding control reconstruction method based on different instruction verification results. The control reconstruction rule table includes direct execution rules, degraded execution rules, modified execution rules, suspended execution rules, and failure handling rules. The direct execution rules, degraded execution rules, modified execution rules, suspended execution rules, and failure handling rules correspond to the direct execution code, degraded execution code, modified execution code, suspended execution code, and failure code in the instruction verification results, respectively. The control reconstruction rule table uses the instruction verification results as the rule entry point, uses the manipulation action data and robot-side execution data as the rule matching basis, and outputs control reconstruction rules for generating local control instructions. Through this control reconstruction rule table, the control reconstruction unit 5 can convert the same type of remote manipulation action into local control content under different processing paths according to the different execution categories of the instruction verification results.

[0047] In this embodiment, the direct execution rule is used to convert the manipulation action data into corresponding local control instructions when the instruction verification result is direct execution encoding. The direct execution rule reads the action category, action direction, action amplitude, and action duration from the manipulation action data, and reads the robot motion state data, actuator state data, and task stage data from the robot-side execution data. The direct execution rule matches the action category with the current task stage of the robot, matches the action direction with the current motion state of the robot, matches the action amplitude with the allowed action range of the robot, and matches the action duration with the action timing data. After the matching is completed, the direct execution rule generates robot motion control data, actuator control data, and action timing data corresponding to the manipulation action data, and the instruction encapsulation module forms local control instructions. The local control instructions corresponding to the direct execution rule maintain the action direction and action category of the manipulation action data, and complete the mapping of local control fields according to the robot-side execution data.

[0048] In this embodiment, the degraded execution rule is used to limit the amplitude of the manipulation action data based on safety boundary data when the instruction verification result is a degraded execution code, and to generate a local control instruction containing safety restriction data. The degraded execution rule reads the amplitude, direction, and duration of the manipulation action data, and reads the robot motion boundary data, actuator motion boundary data, and task stage allowable boundary data from the safety boundary data. The degraded execution rule limits the velocity, displacement, angle, actuator motion, and duration of the manipulation action data according to the safety boundary data, so that the manipulation action data is converted into robot motion control data and actuator control data within the range of the safety boundary data. The safety restriction data records the amplitude-limited motion range, motion speed, motion duration, and actuator motion range. The instruction encapsulation module encapsulates the amplitude-limited robot motion control data, actuator control data, motion timing data, and safety restriction data into a local control instruction, so that the perception marker instruction data packet corresponding to the degraded execution code enters the restricted execution path.

[0049] In this embodiment, the modified execution rule is used to perform state adaptation processing on the control action data based on the robot-side execution data when the instruction verification result is a modified execution code, and generate local control instructions corresponding to the robot-side execution data. The modified execution rule reads the robot motion state data, actuator state data, and task stage data from the robot-side execution data, and maps the above data to the action category, action direction, action amplitude, and action duration in the control action data. The state adaptation processing includes action direction adaptation, action amplitude adaptation, actuator state adaptation, and task stage adaptation. Action direction adaptation is used to convert the action direction in the control action data into an executable direction in the current coordinate system of the robot. Action amplitude adaptation is used to adjust the action amplitude in the control action data according to the current motion state of the robot. Actuator state adaptation is used to determine the actuator control data according to the actuator state data. Task stage adaptation is used to determine the action entry corresponding to the control action data according to the current task stage data. The local control instructions generated by the modified execution rule are used to enable the remote control action to complete the control conversion according to the current execution state of the robot.

[0050] In this embodiment, the suspended execution rule is used to generate a local control instruction to maintain the current execution state when the instruction verification result is a suspended execution code. The suspended execution rule reads the robot motion state data, actuator state data, and task stage data from the robot's execution data and generates a hold-type control field corresponding to the current execution state. The hold-type control fields include a hold motion state field, a hold actuator state field, a hold task stage field, and an action wait field. The hold motion state field is used to maintain the robot's current speed, posture, or position hold state. The hold actuator state field is used to maintain the actuator's current drive state or stop state. The hold task stage field is used to prevent the current task stage data from entering the next task stage. The action wait field is used to record the waiting state after the perception marker instruction data packet enters the suspended execution path. The instruction encapsulation module encapsulates the hold-type control fields into a local control instruction, enabling the robot actuator... The system maintains a defined execution state until the current task phase or actuator status meets the subsequent processing conditions. Failure handling rules are used to generate a processing identifier to terminate the execution path of the current remote control instruction data packet when the instruction verification result is a failure code. The failure handling rules read the instruction identifier data and instruction verification result from the perception marker instruction data packet, bind the failure code to the corresponding instruction identifier data, and generate a failure handling identifier. The failure handling identifier is used to record that the current perception marker instruction data packet does not enter the local control instruction generation path, and is used by the robot controller 2 to register the termination of the execution path for the perception marker instruction data packet. The failure handling rules do not generate action execution type local control instructions and send the failure handling identifier to the instruction receiving module of the robot controller 2 for processing and recording. Through the failure handling rules, the robot controller 2 can recognize that the current remote control instruction data packet has completed verification processing and will no longer enter the actuator drive path.

[0051] In this embodiment, the feedback correction unit 6 includes a feedback data synchronization module, a feedback state normalization module, a feedback difference calculation module, and a feedback result generation module. The feedback data synchronization module is used to perform timing alignment of the robot-side feedback data and the operator-side display update data according to the action timing data corresponding to the local control command, generating feedback aligned data. The feedback state normalization module is used to convert the robot-side feedback data and the operator-side display update data into unified state fields based on the feedback aligned data, generating a feedback state field group. The feedback difference calculation module is used to perform field difference calculation on the robot-side feedback fields and the operator-side display update fields in the feedback state field group, generating feedback difference data. The feedback result generation module is used to input the feedback difference data into a feedback verification table, generate a feedback verification result, and send the feedback verification result to the execution permission level adjustment path.

[0052] In this embodiment, the feedback correction unit 6 is used to process the robot-side feedback data and the operator-side display update data after the robot executes the local control command, and generate a feedback verification result for adjusting the processing path of subsequent remote control command data packets. The feedback correction unit 6 includes a feedback data synchronization module, a feedback state normalization module, a feedback difference calculation module, and a feedback result generation module. The robot-side feedback data is used to represent the actual feedback state after the robot executes the local control command, including robot motion feedback data, actuator feedback data, task stage feedback data, and execution state feedback data. The operator-side display update data is used to represent the updated state displayed by the remote control terminal 1 after the robot executes the local control command, including operator-side display motion data, operator-side display actuator data, operator-side display task stage data, and operator-side display status identifier. The feedback correction unit 6 uses the action timing data corresponding to the local control command as the feedback correspondence benchmark, incorporates the robot-side feedback data and the operator-side display update data into the same feedback verification process, so that the actual state of the robot after execution and the state displayed on the operator can be compared according to the same control action.

[0053] In this embodiment, the feedback data synchronization module is used to perform timing alignment of robot-side feedback data and operator-side display update data according to the action timing data corresponding to the local control command, generating feedback alignment data. The action timing data includes the execution start time, action duration, and execution end condition of the local control command. The feedback data synchronization module reads the action timing data corresponding to the local control command, extracts the feedback record corresponding to the action timing data from the robot-side feedback data, and extracts the display update record corresponding to the execution process of the local control command from the operator-side display update data. The feedback data synchronization module establishes an alignment relationship according to the feedback timing of the feedback record and the display timing of the display update record, and writes robot-side feedback data and operator-side display update data belonging to the same local control command execution process into the same feedback alignment data. The feedback alignment data includes a feedback command identifier, a feedback timing field, a display timing field, a robot-side feedback field, and an operator-side display update field.

[0054] In this embodiment, the feedback state normalization module is used to convert robot-side feedback data and operator-side display update data into unified state fields based on feedback alignment data, generating a feedback state field group. The robot-side feedback data and operator-side display update data come from different sources and differ in field names, state granularity, and data format. The feedback state normalization module maps the motion feedback field, actuator feedback field, task stage feedback field, and execution state feedback field in the robot-side feedback data to unified state fields according to a preset feedback field mapping table, and maps the display motion field, display actuator field, display task stage field, and display state field in the operator-side display update data to their corresponding unified state fields. The feedback state field group includes a robot-side feedback field group and an operator-side display update field group. The robot-side feedback field group and the operator-side display update field group use the same field names and the same field order, enabling the feedback difference calculation module to perform difference calculations on similar fields.

[0055] In this embodiment, the feedback difference calculation module is used to perform field difference calculation on the robot-side feedback field and the operator-side display update field in the feedback status field group to generate feedback difference data. The feedback difference calculation module compares the robot-side feedback field and the operator-side display update field according to the field type, which includes motion status field, actuator status field, task stage field, and execution status field. For the motion status field, the feedback difference calculation module compares the motion status in the robot-side feedback field with the displayed motion status in the operator-side display update field to generate motion feedback difference data. For the actuator status field, the feedback difference calculation module compares the robot-side feedback field with the displayed motion status in the operator-side display update field to generate motion feedback difference data. The actuator status in the feedback field is compared with the displayed actuator status in the operator's display update field to generate actuator feedback difference data. For the task stage field, the feedback difference calculation module compares the task stage in the robot's feedback field with the displayed task stage in the operator's display update field to generate task stage feedback difference data. For the execution status field, the feedback difference calculation module compares the execution status in the robot's feedback field with the display status in the operator's display update field to generate execution status feedback difference data. The feedback difference data is formed by aggregating motion feedback difference data, actuator feedback difference data, task stage feedback difference data, and execution status feedback difference data.

[0056] In this embodiment, the feedback result generation module is used to input feedback difference data into the feedback verification table, generate feedback verification results, and send the feedback verification results to the execution permission level adjustment path. The feedback verification table includes a feedback difference index field, a feedback condition field, and a permission path field. The feedback difference index field is used to read motion feedback difference data, actuator feedback difference data, task phase feedback difference data, and execution status feedback difference data from the feedback difference data. The feedback condition field is used to store continuous control conditions, amplitude limiting control conditions, phase control conditions, and safety maintenance conditions. The permission path field is used to output the feedback verification results corresponding to the feedback condition field. The feedback result generation module matches the feedback difference data with the feedback condition field and generates feedback verification results based on the matching results. The feedback verification results serve as input data for the execution permission level adjustment path, used to determine whether subsequent remote control command data packets enter the continuous control permission path, amplitude limiting control permission path, phase control permission path, or safety maintenance permission path.

[0057] In this embodiment, the execution permission level adjustment path includes a continuous control permission path, a limited control permission path, a staged control permission path, and a safety maintenance permission path. Specifically, the continuous control permission path is used to allow subsequent remote control instruction data packets to enter the state offset calculation path of the instruction verification unit 4 when the feedback verification result meets the continuous control condition; the limited control permission path is used to write the control action data in the subsequent remote control instruction data packets into the downgrade execution rule of the control reconstruction unit 5 when the feedback verification result meets the limited control condition; the staged control permission path is used to limit the subsequent remote control instruction data packets to the task stage data corresponding to the robot's execution data for verification and reconstruction when the feedback verification result meets the staged control condition; and the safety maintenance permission path is used to cause the control reconstruction unit 5 to generate a local control instruction that maintains the current execution state when the feedback verification result meets the safety maintenance condition.

[0058] In this embodiment, the execution permission level adjustment path is used to determine the processing permission of subsequent remote control command data packets based on the feedback verification results. The execution permission level adjustment path includes a continuous control permission path, a limited amplitude control permission path, a staged control permission path, and a safety maintenance permission path. The continuous control permission path is used to allow subsequent remote control command data packets to enter subsequent processing according to the normal state offset calculation path. The limited amplitude control permission path is used to apply an amplitude limit to the control action data in the subsequent remote control command data packets. The staged control permission path is used to limit the subsequent remote control command data packets to the task stage data corresponding to the robot's execution data for verification and reconstruction. The safety maintenance permission path is used to allow subsequent remote control command data packets to enter a processing path that maintains the current execution state. The execution permission level adjustment path transforms the feedback verification results into processing rules for subsequent remote control command data packets through the above four types of permission paths.

[0059] In this embodiment, the continuous control permission path is used to allow subsequent remote control command data packets to enter the state offset calculation path of the command verification unit 4 when the feedback verification result meets the continuous control conditions. The continuous control conditions include that the robot-side feedback data and the operator-side displayed update data all form a corresponding relationship in the motion state field, actuator state field, task stage field, and execution state field, and the feedback difference data is within the field range corresponding to the continuous control conditions. After receiving the feedback verification result, the continuous control permission path writes the continuous control permission identifier into the permission field of the subsequent remote control command data packet, so that the subsequent remote control command data packet can enter the robot-side control link according to the original processing flow of perception marking, state offset calculation, command verification, and control reconstruction. The continuous control permission path does not change the action category and action amplitude of the control action data, but only retains the permission identifier required for subsequent state offset calculation.

[0060] In this embodiment, the amplitude-limited control permission path is used to write the control action data in the subsequent remote control instruction data packet into the downgraded execution rule of the control reconfiguration unit 5 when the feedback verification result meets the amplitude-limited control conditions. The amplitude-limited control conditions include differences in motion feedback, actuator feedback, or execution status feedback between the robot-side feedback data and the operator-side displayed update data, and the task stage corresponding to the feedback difference data still allows the execution of the same type of control action. After receiving the feedback verification result, the amplitude-limited control permission path writes the amplitude-limited control permission identifier into the permission field of the subsequent remote control instruction data packet, and marks the action amplitude, action speed, action duration, or actuator action amount in the control action data as restricted fields. After reading the amplitude-limited control permission identifier, the control reconfiguration unit 5 causes the subsequent remote control instruction data packet to enter the downgraded execution rule, and generates local control instructions containing safety restriction data based on the safety boundary data.

[0061] In this embodiment, the stage control permission path is used to limit the subsequent remote control instruction data packets to the task stage data corresponding to the robot-side execution data for verification and reconstruction when the feedback verification result meets the stage control conditions. The stage control conditions include a difference in the task stage field between the robot-side feedback data and the operator-side displayed update data, or a task stage corresponding to the operator-side displayed update data lagging behind the task stage corresponding to the robot-side feedback data. After receiving the feedback verification result, the stage control permission path writes the stage control permission identifier into the permission field of the subsequent remote control instruction data packets and uses the task stage data in the robot-side execution data as the stage constraint field for subsequent verification and reconstruction. After reading the stage control permission identifier, the instruction verification unit 4 uses the task stage data corresponding to the robot-side execution data as the task stage comparison benchmark. After reading the stage control permission identifier, the control reconstruction unit 5 determines the local control instruction only within the control reconstruction rules allowed in the current task stage. The permission path is used to enable the control reconfiguration unit 5 to generate a local control instruction to maintain the current execution state when the feedback verification result meets the safety maintenance conditions. The safety maintenance conditions include mismatches between the robot-side feedback data and the operator-side updated data, such as mismatches in the execution state field, task stage field, or actuator state field, and the feedback difference data falls within the field range corresponding to the safety maintenance conditions. After receiving the feedback verification result, the safety maintenance permission path writes the safety maintenance permission identifier into the permission field of the subsequent remote control instruction data packet and guides the subsequent remote control instruction data packet to the suspended execution rule. The control reconfiguration unit 5 generates a local control instruction to maintain the current execution state according to the suspended execution rule. The local control instruction includes a maintain motion state field, maintain actuator state field, maintain task stage field, and action wait field, enabling the robot-side controller 2 to drive the robot actuator in a manner that maintains the current execution state until the feedback state is re-corresponded.

[0062] In this embodiment, the effect of the execution permission level adjustment path on subsequent remote control command data packets is realized through the permission field. The permission field enters the subsequent control flow along with the remote control command data packet and serves as the basis for processing path selection in the instruction verification unit 4 and the control reconstruction unit 5. When the permission field is a continuous control permission identifier, the subsequent remote control command data packet enters the state offset calculation path. When the permission field is a limited control permission identifier, the subsequent remote control command data packet enters the downgrade execution rule. When the permission field is a stage control permission identifier, the subsequent remote control command data packet is verified and reconstructed according to the task stage data corresponding to the robot's execution data. When the permission field is a safety maintenance permission identifier, the subsequent remote control command data packet enters the processing path that maintains the current execution state. Through the writing and reading of the permission field, the feedback verification result generated by the feedback correction unit 6 can be applied to the processing of subsequent remote control command data packets.

[0063] Example 2: This invention proposes a remote-controlled robot control method under a high-latency intercontinental network, used in the remote-controlled robot system under a high-latency intercontinental network described in Example 1 above, comprising the following steps: S10.1 Obtain the control input data formed by the operator's input, generate a remote control instruction data packet based on the control input data, and write the operator's perception data when the remote control instruction data packet is generated to form a perception mark instruction data packet. S10.2 Receive the perception tag instruction data packet, obtain the robot-side execution data and safety boundary data, and read the operator-side perception data in the perception tag instruction data packet; S10.3 Calculate the state offset between the sensing data from the operator end and the execution data from the robot end to obtain the instruction offset data. Input the instruction offset data and safety boundary data into the instruction verification table to generate the instruction verification result. Based on the instruction verification result, the control input data and the execution data from the robot end, determine the control reconstruction rule from the control reconstruction rule table. Based on the control reconstruction rule, convert the sensing tag instruction data packet into a local control instruction. S10.4 Drive the robot actuator according to the local control command, obtain the robot end feedback data and the operator end display update data after the robot executes the local control command, perform feedback consistency verification on the robot end feedback data and the operator end display update data, generate feedback verification result, and adjust the execution permission level of the remote control command data packet according to the feedback verification result.

[0064] The embodiments of the present invention have been described in detail above with reference to the accompanying drawings. However, the present invention is not limited thereto. Various changes can be made within the scope of knowledge possessed by those skilled in the art without departing from the spirit of the present invention.

Claims

1. A teleoperated robotic system over a high latency transcontinental network, comprising: include: Remote control terminal (1) is used to obtain the operator's control input, generate remote control instruction data packets, and send the remote control instruction data packets to the robot end controller (2). Robot end controller (2) is used to receive remote control instruction data packets, obtain robot end execution data, and drive robot actuators according to local control instructions; The sensing tag unit (3) is used to write the sensing data of the operation terminal into the remote control instruction data packet when the remote control instruction data packet is generated, so as to form the sensing tag instruction data packet; The instruction verification unit (4) is used to read the operator end perception data in the perception mark instruction data packet and obtain the robot end execution data. It performs state offset calculation on the operator end perception data and the robot end execution data to obtain instruction offset data. It inputs the instruction offset data and safety boundary data into the instruction verification table to generate instruction verification results. The control reconfiguration unit (5) is used to determine the control reconfiguration rules from the control reconfiguration rule table based on the instruction verification results, manipulation action data and robot end execution data, and convert the perception tag instruction data packet into local control instructions according to the control reconfiguration rules. Feedback correction unit (6) is connected to robot end controller (2) and remote control terminal (1) respectively. It is used to obtain robot end feedback data after the robot executes local control instructions, and obtain the corresponding operation terminal display update data of remote control terminal (1). It performs feedback consistency verification on robot end feedback data and operation terminal display update data, generates feedback verification result, and adjusts the execution permission level of remote control instruction data packet according to feedback verification result.

2. The tele-robotic system for remote manipulation across a high latency intercontinental network according to claim 1, wherein, The robot end controller (2) includes an instruction receiving module, an execution data acquisition module, a safety boundary acquisition module, and a local drive module; wherein, the instruction receiving module is used to receive perception tag instruction data packets and generate reception timing data; the execution data acquisition module is used to synchronously acquire robot end execution data when generating reception timing data, the robot end execution data includes robot motion state data, actuator state data, and task stage data; the safety boundary acquisition module is used to synchronously acquire safety boundary data when generating reception timing data, the safety boundary data includes robot motion boundary data, actuator action boundary data, and task stage allowable boundary data; the local drive module is used to drive the robot actuator according to local control instructions, robot end execution data, and safety boundary data.

3. The tele-robotic system for remote manipulation across a high latency intercontinental network according to claim 2, wherein, The perception marking unit (3) includes a perception state reading module, an instruction timing marking module, a same-packet binding module, and a marking integrity verification module. The perception state reading module is used to read the operator perception data from the display cache data of the remote control terminal (1) when the remote control instruction data packet is generated. The operator perception data includes the operator display state identifier and the operator task stage data. The instruction timing marking module is used to associate the display timing data corresponding to the operator perception data with the instruction timing data in the remote control instruction data packet to generate perception timing association data. The same-packet binding module is used to write the operator perception data, perception timing association data, control action data, and instruction timing data into the same remote control instruction data packet to form a perception marking instruction data packet. The marking integrity verification module is used to perform field integrity verification on the operator perception data, perception timing association data, control action data, and instruction timing data in the perception marking instruction data packet, and send the perception marking instruction data packet that has passed the field integrity verification to the robot end controller (2).

4. The tele-robotic system for remote manipulation across a high latency intercontinental network according to claim 3, wherein, The instruction verification unit (4) includes a timing offset calculation module, a motion offset calculation module, a stage offset calculation module, and an offset data generation module. The timing offset calculation module generates timing offset data based on the sensing timing association data in the sensing mark instruction data packet and the receiving timing data generated by the robot end controller (2). The motion offset calculation module generates motion offset data based on the operator end display status identifier in the operator end sensing data and the robot motion status data in the robot end execution data. The stage offset calculation module generates stage offset data based on the operator end task stage data in the operator end sensing data and the task stage data in the robot end execution data. The offset data generation module performs field collection and encoding encapsulation on the timing offset data, motion offset data, and stage offset data to generate instruction offset data and sends the instruction offset data to the instruction verification table.

5. The remotely controlled robot system under a high-latency intercontinental network according to claim 4, characterized in that, The instruction verification table includes an offset index field, a security boundary index field, and an execution code field. The offset index field is used to receive instruction offset data and establish a verification index according to the timing offset data, motion offset data, and stage offset data. The security boundary index field is used to receive security boundary data and match the security boundary data with the verification index to form an instruction execution qualification determination item. The execution code field is used to output the instruction verification result according to the instruction execution qualification determination item. The instruction verification result includes direct execution code, downgraded execution code, corrected execution code, suspended execution code, and failure code. The direct execution code, downgraded execution code, corrected execution code, suspended execution code, and failure code correspond to different control reconstruction entry points in the control reconstruction unit (5).

6. The remotely controlled robot system under a high-latency intercontinental network according to claim 5, characterized in that, The control reconfiguration unit (5) includes a rule matching module, a control parameter generation module, an instruction encapsulation module, and a boundary verification module. The rule matching module receives instruction verification results, manipulation action data, and robot-side execution data, and uses the instruction verification results as the first matching index, the manipulation action data as the second matching index, and the robot-side execution data as the third matching index to determine control reconfiguration rules in the control reconfiguration rule table. The control parameter generation module performs localization conversion on the manipulation action data according to the control reconfiguration rules to generate robot motion control data, actuator control data, and action timing data. The instruction encapsulation module encapsulates the robot motion control data, actuator control data, action timing data, and safety restriction data to generate local control instructions. The boundary verification module performs boundary verification on the local control instructions based on safety boundary data and sends the local control instructions that have passed the boundary verification to the local drive module of the robot-side controller (2).

7. The remotely controlled robot system under a high-latency intercontinental network according to claim 6, characterized in that, The control reconfiguration rule table includes direct execution rules, degraded execution rules, corrective execution rules, suspended execution rules, and failure handling rules. Specifically, the degraded execution rule, when the instruction verification result is a degraded execution code, performs amplitude limiting processing on the control action data based on safety boundary data and generates a local control instruction containing safety limitation data. The corrective execution rule, when the instruction verification result is a corrective execution code, performs state adaptation processing on the control action data based on the robot-side execution data and generates a local control instruction corresponding to the robot-side execution data. The suspended execution rule, when the instruction verification result is a suspended execution code, generates a local control instruction that maintains the current execution state. The failure handling rule, when the instruction verification result is a failure code, generates a processing identifier to terminate the execution path of the current remote control instruction data packet.

8. The remotely controlled robot system under a high-latency intercontinental network according to claim 7, characterized in that, The feedback correction unit (6) includes a feedback data synchronization module, a feedback state normalization module, a feedback difference calculation module, and a feedback result generation module. The feedback data synchronization module is used to perform timing alignment of the robot-side feedback data and the operator-side display update data according to the action timing data corresponding to the local control command, generating feedback alignment data. The feedback state normalization module is used to convert the robot-side feedback data and the operator-side display update data into unified state fields based on the feedback alignment data, generating a feedback state field group. The feedback difference calculation module is used to perform field difference calculation on the robot-side feedback field and the operator-side display update field in the feedback state field group, generating feedback difference data. The feedback result generation module is used to input the feedback difference data into a feedback verification table, generate a feedback verification result, and send the feedback verification result to the execution permission level adjustment path.

9. The remotely controlled robot system under a high-latency intercontinental network according to claim 8, characterized in that, The execution permission level adjustment path includes a continuous control permission path, a limited control permission path, a staged control permission path, and a safety maintenance permission path. The continuous control permission path is used to allow subsequent remote control instruction data packets to enter the state offset calculation path of the instruction verification unit (4) when the feedback verification result meets the continuous control condition. The limited control permission path is used to write the control action data in the subsequent remote control instruction data packets into the downgrade execution rule of the control reconstruction unit (5) when the feedback verification result meets the limited control condition. The staged control permission path is used to limit subsequent remote control instruction data packets to the task stage data corresponding to the robot-side execution data for verification and reconstruction when the feedback verification result meets the staged control condition. The safety maintenance permission path is used to allow the control reconstruction unit (5) to generate a local control instruction that maintains the current execution state when the feedback verification result meets the safety maintenance condition.

10. A remotely controlled robot control method under a high-latency intercontinental network, used in a remotely controlled robot system under a high-latency intercontinental network as described in any one of claims 1-9, characterized in that: Includes the following steps: S10.1 Obtain the control input data formed by the operator's input, generate a remote control instruction data packet based on the control input data, and write the operator's perception data when the remote control instruction data packet is generated to form a perception mark instruction data packet. S10.2 Receive the perception tag instruction data packet, obtain the robot-side execution data and safety boundary data, and read the operator-side perception data in the perception tag instruction data packet; S10.3 Calculate the state offset between the sensing data from the operator end and the execution data from the robot end to obtain the instruction offset data. Input the instruction offset data and safety boundary data into the instruction verification table to generate the instruction verification result. Based on the instruction verification result, the control input data and the execution data from the robot end, determine the control reconstruction rule from the control reconstruction rule table. Based on the control reconstruction rule, convert the sensing tag instruction data packet into a local control instruction. S10.4 Drive the robot actuator according to the local control command, obtain the robot end feedback data and the operator end display update data after the robot executes the local control command, perform feedback consistency verification on the robot end feedback data and the operator end display update data, generate feedback verification result, and adjust the execution permission level of the remote control command data packet according to the feedback verification result.

Images