Robot remote control method and system and storage medium

Abstract

The invention relates to the technical field of robot control, in particular to a robot remote control method and system and a storage medium, and the method comprises the steps that a robot side sends data including a timestamp to an operation side through a server side; after the robot end receives the data including the timestamp and the operation signal of the operation end, the time of the timestamp and the time of receiving the operation signal are compared, and a data transmission delay value is obtained; if the data transmission delay value is not greater than the first preset threshold value, the robot end executes operation according to the operation signal; and if the data transmission delay value is greater than the first preset threshold value, the robot end starts an emergency processing strategy. The calculation of the data transmission delay value is from the time of the robot end, so that network clock synchronization is not needed, the interference of errors generated by the network clock synchronization on the calculation of the network delay is avoided, and the data transmission delay value can accurately reflect the network delay time.

Description

Technical Field

[0001] This invention relates to the field of robot motion control, and more specifically, to a robot remote control method, system, and storage medium. Background Technology

[0002] A remote-controlled robot system typically includes an operator terminal, a server terminal, and a robot terminal. The robot terminal and the operator terminal communicate via a network, with the robot sending data to the server terminal. The operator terminal sends operation signals to the robot terminal, while the robot terminal sends feedback signals and collected environmental information back to the operator terminal.

[0003] However, network transmission latency often leads to untimely data delivery, a problem particularly prominent in applications requiring high control accuracy. To reduce operational errors caused by network latency, it is typically necessary to measure and verify network latency. Current technologies generally calculate network latency by including timestamps in the transmitted data. However, since the local clocks of the operator and the robot are not synchronized, resulting in clock discrepancies, network clock synchronization is required before calculating latency to ensure consistency between the two clocks. Then, the network latency and transmission speed are calculated based on the timestamp information.

[0004] However, the network clock synchronization process itself is affected by network latency and transmission speed, exhibiting fluctuations and making perfectly accurate synchronization difficult to achieve. This means that clock synchronization errors are still introduced when calculating network latency. In general robotic applications, while this synchronization error does affect latency calculations, it usually still meets the basic requirements of remote control. However, in specialized fields with high precision and real-time requirements, such as surgical robot systems, the robot needs to transmit high-definition surgical field images to the operator in real time, while the operator needs to issue precise operational commands based on the image information and robot feedback signals. Such scenarios demand extremely high precision and real-time performance in control; network latency can lead to serious consequences. For example, a doctor's "stop moving" command might fail to be delivered and executed in time due to delay, causing the robotic arm to move excessively and damage the patient's delicate tissues. In these applications, the operator's signal latency is typically required to be strictly controlled within 80 milliseconds. Under such extremely strict time constraints, errors generated by network clock synchronization can significantly interfere with the accurate calculation and control of network latency, thereby affecting the system's reliability and security. Summary of the Invention

[0005] To overcome the problem that the accuracy of network latency calculation is affected by network clock synchronization errors in the prior art, this invention provides a robot remote control method, system, and storage medium that can obtain accurate network latency information without network clock synchronization.

[0006] To solve the above-mentioned technical problems, the technical solution adopted by the present invention is: a remote control method for a robot, comprising: The robot sends timestamped data to the operator via the server. After the robot receives data from the operator including the timestamp and the operation signal, it compares the time of the timestamp with the time of receiving the operation signal to obtain the data transmission delay value. If the data transmission delay value is not greater than the first preset threshold, the robot performs the operation according to the operation signal; if the data transmission delay value is greater than the first preset threshold, the robot activates the emergency handling strategy.

[0007] In the above technical solution, the timestamp is sent from the robot, passes through the server, and reaches the operator. The operator does not generate a new timestamp or calculate network latency based on the timestamp. Instead, it sends the timestamp back to the robot along with the operation signal. Upon receiving the returned timestamp, the robot compares it with the time it received the original timestamp to obtain the data transmission latency value. Because both the timestamp and the data transmission latency calculation are based on the robot's time, network time synchronization is unnecessary. This avoids errors caused by fluctuations in network conditions, ensuring that the obtained data transmission latency value accurately reflects the network latency.

[0008] Different control strategies are adopted based on the different data transmission delay values. If the data transmission delay value is not greater than a first preset threshold, it indicates that the network delay is within an acceptable range and will not affect the robot's control accuracy. In this case, the robot executes the operation according to the operation signal. However, if the data transmission delay value is greater than the first preset threshold, it indicates that the network delay has affected the robot's control accuracy or precision, and emergency handling is required.

[0009] Preferably, the timestamp is generated by reading the current time from the robot; the operator extracts the received timestamp and sends it to the server along with the operation signal. The timestamp is generated by reading the current time from the robot, and the time of the received timestamp comes from the same device clock, ensuring that the time source is the same. The operator simply extracts the timestamp without processing it before sending it to the server along with the operation signal, ensuring that the time recorded by the timestamp does not change.

[0010] Preferably, the emergency response strategy includes the robot not executing the operation signal and stopping its movement, or executing an exit operation, or executing a local control strategy.

[0011] Preferably, if the data transmission delay value is greater than the first preset threshold but not greater than the second preset threshold, it may be caused by network fluctuations during the current data transmission. It can recover on its own during the next data transmission. Therefore, the robot can stop executing the operation signal and stop moving. After receiving the operation signal again, if the data transmission delay value is determined to be no greater than the first preset threshold, the robot can be made to move again.

[0012] If the data transmission delay value exceeds a second preset threshold, it is determined that the network connection is broken. Even if the network is subsequently reconnected or the data transmission delay value does not exceed a first preset threshold, due to the significant data loss during the network connection break, the robot's actions based on the operation signals may be distorted or have large errors. Therefore, in this case, the robot performs an exit operation or executes a local control strategy. After performing an exit operation, the robot can wait for the network to reconnect and stabilize before resuming actions based on the operation signals. The local control strategy uses the operation signals from the robot's built-in controller and executes the corresponding actions. The local control strategy allows the robot to continue operating even when remote control is not possible.

[0013] Preferably, the server sends empty packets to the operator and obtains the packet loss rate; if the packet loss rate is higher than a preset packet loss threshold, the server will switch the data transmission network to a backup network.

[0014] Preferably, the robot verifies that the duration for which the data transmission delay value is greater than the first preset threshold is greater than a preset time threshold, and then executes a local control strategy. If the data transmission delay value is continuously greater than the first preset threshold, it means that the robot remains in a stopped state and does not perform any action. A prolonged stop will affect the execution of the entire task. Therefore, when it is verified that the duration for which the data transmission delay value is greater than the first preset threshold is greater than the preset time threshold, the robot switches to executing a local control strategy to allow the robot's actions to continue.

[0015] A robot remote control system includes: a robot end, a server end, and an operator end; the robot end sends device data including timestamps to the server end; the operator end sends operation data including the timestamps and operation signals to the server end; the robot end extracts the timestamps from the operation data and compares them with the time the robot end receives the operation data, and according to the comparison result, instructs the robot end to perform an action based on the operation signals or only perform an emergency handling strategy; the server end transmits the operation data to the robot end and the device data to the operator end.

[0016] In this technical solution, the timestamp originates from the robot. After the server transmits the device data from the robot to the operator, the operator sends the timestamp from the device data along with the operation signal to the server. The server then transmits this data to the robot. The robot receives the operation data at a specific time, which can be the robot's current time. This time is compared with the timestamp to obtain the data transmission delay value. Based on the magnitude of the data transmission delay value, the robot performs different actions, including performing actions based on the operation signal or only executing a pre-defined emergency handling strategy.

[0017] Since the timestamp and the calculation of the data transmission delay value are both based on the robot's time, there is no need for network time synchronization. Therefore, the network time synchronization error caused by fluctuations in the network status itself will not occur. The obtained data transmission delay value can truly reflect the network delay time and obtain an accurate network delay.

[0018] Preferably, the robot end includes a controller module, a data acquisition module, a data transmission module, a data receiving module, and an execution device; the operation end includes an information display module, a data acquisition and processing module, and an operation device; the server end includes a data transmission module. The data acquisition module collects the status information of the execution device and the current time of the execution device, and forms device data including timestamps; The data receiving module receives and extracts the timestamp from the operation data and compares it with the time of receiving the operation data to obtain the data transmission delay value. The controller module sends an operation signal to the execution device based on the data transmission delay value and instructs the execution device to execute or to execute an emergency handling strategy. The data transmission module is used to transmit the device data to the server. The information display module is used to display the status information of the execution device; the data acquisition and processing module is used to extract the timestamp and the operation signal of the acquisition device to form operation data, and transmit the operation data to the server. The data transmission module is used to transmit the device data from the robot to the operating terminal and the operation data from the operating terminal to the robot.

[0019] Preferably, the server side further includes an empty packet verification module and a network switching module; the empty packet verification module sends empty packets to the operating terminal and detects whether the packet loss rate is higher than a preset packet loss threshold; the network switching module switches to a backup network when the packet loss rate is higher than the preset packet loss threshold.

[0020] A storage medium for storing a computer program; wherein the computer program, when executed by a processor, implements the above-described remote robot control method.

[0021] Compared with existing technologies, the advantages of this invention are as follows: the robot generates a timestamp and sends it to the operator, who then sends the timestamp back to the robot. The robot compares the returned timestamp with the time of receiving the operation signal from the operator to obtain the data transmission delay value. Since the calculation of the data transmission delay value is based entirely on the robot's time, network clock synchronization is unnecessary, avoiding interference from errors caused by network clock synchronization in the calculation of network delay. This ensures that the data transmission delay value accurately reflects the network delay time. The robot's ability to execute different control strategies based on the data transmission delay value ensures accurate execution of its actions. Attached Figure Description

[0022] Figure 1 This is a flowchart of a robot remote control method according to the present invention; Figure 2 This is a schematic diagram of the framework of a robot remote control system according to the present invention. Detailed Implementation

[0023] The accompanying drawings are for illustrative purposes only and should not be construed as limiting this patent. To better illustrate this embodiment, some components in the drawings may be omitted, enlarged, or reduced, and do not represent the actual dimensions of the product. It is understandable to those skilled in the art that some well-known structures and their descriptions may be omitted in the drawings. The positional relationships described in the drawings are for illustrative purposes only and should not be construed as limiting this patent.

[0024] In the accompanying drawings of the embodiments of the present invention, the same or similar reference numerals correspond to the same or similar components. In the description of the present invention, it should be understood that if terms such as "upper," "lower," "left," "right," "long," and "short" indicate the orientation or positional relationship based on the orientation or positional relationship shown in the drawings, they are only for the convenience of describing the present invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, the terms used to describe positional relationships in the drawings are only for illustrative purposes and should not be construed as limiting the present patent. For those skilled in the art, the specific meaning of the above terms can be understood according to the specific circumstances.

[0025] The technical solution of the present invention will be further described in detail below through specific embodiments and in conjunction with the accompanying drawings: Example 1 like Figure 1 The following is an embodiment 1 of a robot remote control method, comprising: The robot sends timestamped data to the operator via the server. After receiving data containing timestamps and operation signals from the operator, the robot compares the timestamp time with the time the operation signal was received to obtain the data transmission delay value. If the data transmission delay is not greater than the first preset threshold, the robot performs the operation according to the operation signal; if the data transmission delay is greater than the first preset threshold, the robot activates the emergency handling strategy.

[0026] In this embodiment, the emergency handling strategy includes the robot not executing the operation signal and stopping its movement, or executing an exit operation, or executing a local control strategy. Specifically, if the data transmission delay value is greater than a first preset threshold but not greater than a second preset threshold, the robot not executing the operation signal and stopping its movement; if the data transmission delay value is greater than the second preset threshold, the robot executing an exit operation or executing a local control strategy.

[0027] Specifically, if the data transmission delay value is greater than the first preset threshold but not greater than the second preset threshold, the robot will not execute the operation signal and will stop moving; if the data transmission delay value is greater than the second preset threshold, the robot will execute an exit operation or execute a local control strategy.

[0028] If the data transmission delay value is greater than the first preset threshold but not greater than the second preset threshold, it may be due to network fluctuations during the current data transmission. It can recover on its own during the next data transmission, so the robot can stop executing the operation signal and stop moving. After receiving the operation signal again, if the data transmission delay value is not greater than the first preset threshold, the robot can be made to move again.

[0029] If the data transmission delay exceeds the second preset threshold, the robot will either exit or execute a local control strategy. If the network connection is determined to be broken, even if the network subsequently reconnects or the data transmission delay does not exceed the first preset threshold, significant data loss during a network connection break may cause the robot to distort or significantly erroneous actions based on the operation signals. Therefore, in this case, the robot will either exit or execute a local control strategy. After exiting, the robot can wait for the network to reconnect and stabilize before resuming actions based on the operation signals. The local control strategy utilizes the operation signals from the robot's built-in controller and executes the corresponding actions. This allows the robot to continue operating even when remote control is unavailable.

[0030] In this embodiment, an application to a surgical robot is used as an example. The first preset threshold can be 100-150ms, and the second preset threshold can be 400-600ms. Since the data transmission delay includes the network delay for the timestamp to reach the operating end and the network delay for returning to the robot end, it differs from the one-way network delay in the prior art. The uplink and downlink network delays also differ, but the difference is not significant. Therefore, within 100-150ms, the actual network delay from the operating end to the robot end will be within 80ms, meeting the usage requirements. To further enhance safety redundancy, this embodiment sets the first preset threshold to 100ms and the second preset threshold to 500ms.

[0031] In this embodiment, if the duration for which the data transmission delay value is greater than a first preset threshold exceeds a preset time threshold, the robot will execute a local control strategy. If the data transmission delay value continuously exceeds the first preset threshold, it means the robot remains in a stopped state without performing any actions. A prolonged stop will affect the execution of the entire task. Therefore, when it is verified that the duration for which the data transmission delay value is greater than the first preset threshold exceeds the preset time threshold, the robot switches to the local control strategy, allowing the robot's actions to continue. In this embodiment, the preset time threshold is 3 seconds, but other times can be set according to actual conditions.

[0032] In other implementations, it can also be verified whether the number of times the data transmission delay value is greater than the first preset threshold is greater than a preset number limit.

[0033] The working principle or workflow of this embodiment is as follows: A timestamp is sent from the robot, and the time recorded by the timestamp is t0. After passing through the server, it reaches the operator. The operator does not generate a new timestamp or calculate the network latency based on the timestamp. Instead, it sends the timestamp along with the operation signal back to the robot. After receiving the returned timestamp, the robot compares it with the time t1 when it received the timestamp to obtain the data transmission latency value (t1-t0). Since both the timestamp and the calculation of the data transmission latency value are based on the robot's time, network time synchronization is not required. Therefore, the errors caused by network time synchronization itself are avoided due to fluctuations in the network state. The obtained data transmission latency value can accurately reflect the network latency time, resulting in an accurate network latency.

[0034] Different control strategies are adopted depending on the data transmission delay value. If the data transmission delay value is not greater than the first preset threshold, it means that the network delay is within an acceptable range and will not affect the robot's control accuracy. In this case, the robot executes the operation according to the operation signal. However, if the data transmission delay value is greater than the first preset threshold, it means that the network delay has affected the robot's control accuracy or precision, and emergency handling is required.

[0035] The beneficial effects of this embodiment are as follows: The robot generates a timestamp and sends it to the operator. The operator then sends the timestamp back to the robot. The robot compares the returned timestamp with the time it receives the operation signal from the operator to obtain the data transmission delay value. Since the calculation of the data transmission delay value is based entirely on the robot's time, there is no need for network clock synchronization. This avoids interference from errors caused by network clock synchronization in the calculation of network delay, ensuring that the data transmission delay value accurately reflects the network delay time. The robot can execute different control strategies based on the data transmission delay value to ensure that the robot's actions are executed accurately.

[0036] Example 2 An embodiment 2 of a robot remote control method is based on embodiment 1. The difference between embodiment 1 and embodiment 1 is that the server sends empty packets to the operator and obtains the packet loss rate; if the packet loss rate is higher than the preset packet loss threshold, the server will switch the data transmission network to the backup network.

[0037] Example 3 like Figure 2 The illustration shows an embodiment of a robot remote control system, comprising: a robot end, a server end, and an operator end; the robot end sends device data including timestamps to the server end; the operator end sends operation data including timestamps and operation signals to the server end; the robot end extracts the timestamps from the operation data and compares them with the time the robot end receives the operation data, and based on the comparison result, instructs the robot end to perform an action based on the operation signals or only perform an emergency handling strategy; the server end transmits the operation data to the robot end and the device data to the operator end.

[0038] Specifically, the robot end includes a controller module, a data acquisition module, a data transmission module, a data receiving module, and an execution device; the operation end includes an information display module, a data acquisition and processing module, and an operation device; the server end includes a data transmission module. The execution device includes a robotic arm and an actuator mounted on the end effector of the robotic arm. For ease of localized operation, the robot end also includes a local manipulator, which may include a handheld manipulator and / or a foot-operated manipulator. The operation device includes a handheld manipulator and / or a foot-operated manipulator, and the operation signals are generated by the operation device.

[0039] The data acquisition module collects the status information and current time of the execution device and generates device data including timestamps. The status information of the execution device may include the robot's motor positions, posture, and target object images. In the field of surgical robots, the target object image may be the location of a lesion or an eye, etc. The data acquisition module may include sensors for collecting motor positions and postures, and a camera device for collecting target object images.

[0040] The data receiving module receives and extracts the timestamp from the operation data and compares it with the time of receiving the operation data to obtain the data transmission delay value. The controller module sends operation signals to the execution device based on the data transmission delay value, instructing the execution device to perform the operation or to execute an emergency handling strategy. Specifically, if the data transmission delay value is not greater than a first preset threshold, the robot performs the operation based on the operation signal; if the data transmission delay value is greater than the first preset threshold but not greater than a second preset threshold, the robot does not execute the operation signal and stops moving; if the data transmission delay value is greater than the second preset threshold, the robot performs an exit operation or executes a local control strategy. The first preset threshold can be 100-150ms, and the second preset threshold can be 400-600ms. When executing a local control strategy, the robotic arm is operated through a handheld manipulator and / or a foot-operated manipulator on the robot. The exit operation involves the robotic arm returning to its original position or moving away from the target object.

[0041] The data transmission module is used to transmit device data to the server. The information display module is used to display the status information of the execution device, which can be a screen; the data acquisition and processing module is used to extract timestamps and collect operation signals of the operation device to form operation data, and transmit the operation data to the server; the operation terminal includes a handheld operator and / or a foot-operated operator, and the operation signals are generated by the handheld operator and / or the foot-operated operator.

[0042] The data transmission module is used to transmit equipment data from the robot to the operator and to transmit operation data from the operator to the robot.

[0043] The working principle or workflow of this embodiment is as follows: The timestamp originates from the robot. After the server transmits the device data from the robot to the operator, the operator sends the timestamp from the device data along with the operation signal to the server. The server then transmits this data to the robot. The robot receives the operation data at a specific time, which can be the robot's current time. This time is compared with the timestamp to obtain the result; the difference between the two is the data transmission delay value. Based on the magnitude of the data transmission delay value, the robot performs different actions, including performing actions based on the operation signal or only executing a pre-defined emergency handling strategy.

[0044] Since the timestamp and the calculation of the data transmission delay value are both based on the robot's time, there is no need for network time synchronization. Therefore, the network time synchronization error caused by fluctuations in the network status itself will not occur. The obtained data transmission delay value can truly reflect the network delay time and obtain an accurate network delay.

[0045] The beneficial effects of this implementation are as follows: The robot generates a timestamp and sends it to the operator. The operator then sends the timestamp back to the robot. The robot compares the returned timestamp with the time it receives the operation signal from the operator to obtain the data transmission delay value. Since the calculation of the data transmission delay value is based entirely on the robot's time, there is no need for network clock synchronization. This avoids the interference of errors caused by network clock synchronization in the calculation of network delay, ensuring that the data transmission delay value accurately reflects the network delay time. The robot can execute different control strategies based on the data transmission delay value to ensure that the robot's actions are executed accurately.

[0046] Example 4 Another embodiment of a robot remote control system, based on embodiment 3, differs from embodiment 3 in that the server side further includes a data verification module, a data encryption module, a network switching module, and an empty packet verification module. The empty packet verification module sends empty packets to the operator and detects whether the packet loss rate is higher than a preset packet loss threshold. The network switching module switches to a backup network when the packet loss rate is higher than the preset packet loss threshold. The data verification module verifies the received data, verifying information such as the data source device number and IP address. The data encryption module encrypts the data using an encryption algorithm.

[0047] Example 5 An embodiment of a storage medium for storing a computer program; wherein the computer program, when executed by a processor, implements the robot remote control method of any of the above embodiments.

[0048] Obviously, the above embodiments of the present invention are merely examples for clearly illustrating the present invention, and are not intended to limit the implementation of the present invention. Those skilled in the art can make other variations or modifications based on the above description. It is neither necessary nor possible to exhaustively describe all embodiments here. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the scope of protection of the claims of the present invention.

Claims

1. A robot remote control method characterized by, The robot end sends data including a timestamp to the server end, and the server end sends the data to the operation end. The robot end receives data including the timestamp and operation signal from the operation end, compares the time of the timestamp with the time of receiving the operation signal, and obtains a data transmission delay value. If the data transmission delay value is not greater than a first preset threshold, the robot end executes the operation signal; if the data transmission delay value is greater than the first preset threshold, the robot end starts an emergency processing strategy. The timestamp is generated by reading the current time of the robot end; the operation end extracts the received timestamp and sends it to the server end together with the operation signal.

2. The robot remote control method according to claim 1, characterized by, The emergency processing strategy includes that the robot end does not execute the operation signal and stops moving, or executes an exit operation, or executes a local control strategy.

3. The robot remote control method according to claim 1, wherein, If the data transmission delay value is greater than the first preset threshold and not greater than a second preset threshold, the robot end does not execute the operation signal and stops moving; if the data transmission delay value is greater than the second preset threshold, the robot end executes an exit operation or executes a local control strategy.

4. The robot remote control method according to claim 2, wherein The server end sends empty packets to the operation end and obtains a packet loss rate; if the packet loss rate is higher than a preset packet loss threshold, the server end switches the network for data transmission to a backup network.

5. The robot remote control method according to claim 3, wherein If the robot end verifies that the duration of the data transmission delay value being greater than the first preset threshold is greater than a preset time threshold, the robot end executes an exit operation or executes a local control strategy.

6. The method of claim 1-5, wherein, The robot end, the server end and the operation end; the robot end sends device data including a timestamp to the server end; the operation end sends operation data including the timestamp and an operation signal to the server end; the robot end extracts the timestamp in the operation data and compares it with the time of receiving the operation signal of the robot end, and according to the comparison result, the robot end executes an action based on the operation signal or only executes an emergency processing strategy; 7. A robot remote control system characterized by comprising: The server end transmits the operation data to the robot end and transmits the device data to the operation end. The robot end includes a controller module, a data acquisition module, a data transmission module, a data receiving module and an execution device; the operation end includes an information display module, a data acquisition and processing module and an operation device; the server end includes a data transmission module; The data acquisition module acquires state information of the execution device and the current time of the execution device and forms device data including a timestamp; 8. The robotic telepresence system of claim 7, wherein, The data receiving module receives and extracts the timestamp in the operation data and compares it with the current time of the execution device to obtain a data transmission delay value; The controller module transmits the operation signal to the execution device according to the data transmission delay value and makes the execution device execute or makes the execution device execute an emergency processing strategy; The data transmission module is used for transmitting the device data to the server end; The information display module is used for displaying the state information of the execution device; the data acquisition and processing module is used for extracting the timestamp and acquiring the operation signal of the operation device to form operation data, and transmitting the operation data to the server end; ​ ​ The data transmission module is configured to transmit device data of the robot end to the operation end and transmit operation data of the operation end to the robot end.

9. The robotic telepresence system of claim 8, wherein, The server end further comprises an empty packet verification module and a network switching module; the empty packet verification module sends empty packets to the operation end and detects whether a packet loss rate is higher than a preset packet loss threshold; the network switching module switches to a backup network when the packet loss rate is higher than the preset packet loss threshold.

10. A storage medium, characterized by A computer program product for storing a computer program; wherein the computer program is executed by a processor to implement the robot remote control method according to any one of claims 1-6.

Images