Robot remote control method, electronic equipment, storage medium and computer program product

Abstract

The embodiment of the invention provides a robot remote control method, electronic equipment, a storage medium and a computer program product. The robot remote control method comprises the steps that first communication connection based on MQTT is established between a plurality of main control devices and a plurality of controlled robots with force feedback capacity through a back-end server; after the first communication connection is successfully established, WebSocket authentication is carried out on the plurality of main control devices and the plurality of controlled robots through the back-end server, and after the authentication is passed, the plurality of main control devices and the plurality of controlled robots are connected to a public network; and respectively acquiring and storing structure attribute information and control software version information corresponding to the plurality of main control devices and the plurality of controlled robots, so as to match a target controlled robot for executing a control instruction from the plurality of controlled robots when the control instruction of at least one main control device is received through a public network. According to the scheme, large-scale deployment of robot remote control in a public network environment can be realized.

Description

TECHNICAL FIELD

[0001] Embodiments of the present application relate to the technical field of robots, and in particular to a robot remote control method, an electronic device, a storage medium and a computer program product. BACKGROUND

[0002] Current intelligent robot technology is developing rapidly, and the working scenarios of robots are becoming more and more extensive. For example, various factory industrial operations, object handling, hotel services, hospital cleaning and the like. The current fully autonomous robot technology is still under development, on the one hand, it needs sufficient data for training, and on the other hand, due to the limitations of technology, it cannot completely rely on algorithms in actual use scenarios. Therefore, the current solution of remote control operation can solve the above tasks, and can save more enterprise and employee costs.

[0003] However, the current robot remote control operation solution can usually only be deployed within a local area network, and is generally a one-to-one fixed remote control operation, which is difficult to meet the current robot use requirements. SUMMARY

[0004] Therefore, embodiments of the present application provide a robot remote control solution to at least partially solve the above problems.

[0005] According to a first aspect of embodiments of the present application, a robot remote control method is provided, for a backend server, the method comprising: establishing, by the backend server, a first communication connection based on Message Queue Telemetry Transport (MQTT) between a plurality of master control devices and a plurality of controlled robots with force feedback capability; after the first communication connection is successfully established, performing, by the backend server, WebSocket authentication for full-duplex communication for a plurality of master control devices and a plurality of controlled robots, and after the authentication is passed, connecting the plurality of master control devices and the plurality of controlled robots to a public network; obtaining the structure attribute information and the control software version information corresponding to the plurality of master control devices and the plurality of controlled robots respectively, and storing the structure attribute information and the control software version information, so as to match a target controlled robot from the plurality of controlled robots to execute a control instruction of at least one master control device when the control instruction is received through the public network.

[0006] According to a second aspect of the embodiments of the present application, a robot remote control method is provided, for a target master device in a plurality of master devices, the method comprising: establishing, by a backend server, a first communication connection based on a message queue telemetry transport protocol (MQTT) between a plurality of controlled robots with force feedback capability; after the first communication connection is successfully established, performing, by the backend server, WebSocket authentication for full-duplex communication, and after the authentication is passed, connecting to a public network by the backend server, so that when it is determined that a target controlled robot needs to be controlled, sending a control instruction to the target controlled robot via the public network through the backend server.

[0007] According to a third aspect of the embodiments of the present application, a robot remote control method is provided, for a target controlled robot in a plurality of controlled robots with force feedback capability, the method comprising: establishing, by a backend server, a first communication connection of a message queue telemetry transport protocol (MQTT) with a plurality of master devices; after the first communication connection is successfully established, performing, by the backend server, WebSocket authentication for full-duplex communication, and after the authentication is passed, connecting to a public network by the backend server, so that after receiving a control instruction of a target master device in the plurality of master devices, executing the operation instruction, and feeding back an execution result to the target master device through the backend server.

[0008] According to a fourth aspect of the embodiments of the present application, an electronic device is provided, comprising: a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory complete communication with each other through the communication bus; the memory is used to store a computer program; the processor is used to execute the robot remote control method of any one of the first aspect, the second aspect and the third aspect by running the computer program stored on the memory.

[0009] According to a fifth aspect of the embodiments of the present application, a computer storage medium is provided, having a computer program stored thereon, the computer program being executed by a processor to implement the robot remote control method of any one of the first aspect, the second aspect and the third aspect.

[0010] According to a sixth aspect of the embodiments of the present application, a computer program product is provided, comprising a computer program, the computer program being executed by a processor to implement the robot remote control method of any one of the first aspect, the second aspect and the third aspect.

[0011] According to the robot remote control scheme provided in the embodiments of the present application, on the one hand, a first communication connection based on a message queue telemetry transport protocol MQTT can be established between the plurality of master devices and the plurality of controlled robots with force feedback capability through the backend server, and after the first communication connection is successfully established, full-duplex communication WebSocket authentication is performed on the plurality of master devices and the plurality of controlled robots through the backend server, and after the authentication is passed, the plurality of master devices and the plurality of controlled robots are connected to the public network, thereby the present scheme can effectively realize the public network deployment of the plurality of master devices and the plurality of controlled robots, and further effectively realize the large-scale deployment of robot remote control in a public network environment, and can make the system have a lower delay, since the present scheme does not need to establish a private network, the deployment cost of robot remote control can also be reduced, and the dependence on training data and algorithms is relatively low when the robot is deployed in a large scale in a public network environment, so the cost of robot large-scale deployment can be further solved; on the other hand, the structural attribute information and the control software version information corresponding to the plurality of master devices, and the structural attribute information and the control software version information corresponding to the plurality of controlled robots can also be acquired respectively, and the structural attribute information and the control software version information are stored, thereby when the control instruction of at least one master device is received through the public network, the target controlled robot that executes the control instruction can be accurately matched from the plurality of controlled robots, which makes it possible to accurately determine the target controlled robot that is remotely controlled based on the actual structural attribute state and the software version state of the master device and the controlled robot when the robot is remotely controlled, and ensures the condition adaptation between the control end and the controlled end, thereby the convenience, reliability and operation adaptability of the robot remote control deployed in a large scale in a public network environment can be improved, and the robot remote control can more accurately meet the needs of users; on the other hand, since the plurality of controlled robots deployed in a large scale in a public network environment in the present scheme have force feedback capability, the user (such as an operator, etc.) can obtain the force feedback information of the controlled robot in operation when needed in the process of remotely controlling the controlled robot, thereby the user can be more intuitively and perceptually presented the tactile state of the controlled robot in operation, which has a high application value in various use scenarios of the controlled robot, and can help to complete the remote control operation task with high accuracy and high reliability. BRIEF DESCRIPTION OF DRAWINGS

[0012] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings needed to be used in the embodiments or the prior art description will be briefly introduced as follows. Obviously, the drawings in the following description are only some embodiments described in the embodiments of the present application, and other drawings can also be obtained by those skilled in the art based on these drawings.

[0013] Figure 1A Fig. 1 shows a schematic diagram of a robot remote control system according to an embodiment of the present application.

[0014] Figure 1A Fig. 2 shows a schematic diagram of an example communication connection between a master device and a controlled robot and a backend server.

[0015] Figure 2 Fig. 3 shows a flow chart of steps of a robot remote control method according to an embodiment of the present application.

[0016] Figure 3 Fig. 4 shows a flow chart of one optional step of matching a target controlled robot to execute a control instruction in an embodiment of the present application.

[0017] Figure 4 Fig. 5 shows a flow chart of steps of another robot remote control method according to an embodiment of the present application.

[0018] Figure 5 Fig. 6 shows a flow chart of steps of still another robot remote control method according to an embodiment of the present application.

[0019] Figure 6 Fig. 7 shows a structural schematic diagram of an electronic device according to an embodiment of the present application.

[0020] Fig. 1 shows a schematic diagram of a robot remote control system according to an embodiment of the present application. DETAILED DESCRIPTION

[0021] In order to make personnel in the art better understand the technical solutions in the embodiments of the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only some of the embodiments of the present application, rather than all the embodiments of the present application. Based on the embodiments in the embodiments of the present application, all other embodiments obtained by those of ordinary skill in the art should belong to the scope of protection of the embodiments of the present application.

[0022] The specific implementation of the embodiments of the present application will be further described below in conjunction with the drawings of the embodiments of the present application.

[0023] Figure 1A Fig. 1 shows a schematic diagram of a robot remote control system according to an embodiment of the present application. Figure 1AAs shown, the robot remote control system 100 can include a backend server 102, a plurality of master devices 104, and a plurality of force feedback capable controlled robots 106.

[0024] Optionally, the backend server 102 can be any appropriate device for storing information, data, programs, and / or any other suitable type of content, including but not limited to distributed storage system devices, server clusters, computing cloud server clusters, etc. In some optional embodiments, the backend server 102 can perform any appropriate functions. Optionally, the backend server 102 can perform a robot remote control method.

[0025] In some optional embodiments, when the backend server 102 performs the robot remote control method of the embodiments of the present application, a first communication connection based on the MQTT (Message Queuing Telemetry Transport) protocol can be established between the plurality of master devices 104 and the plurality of force feedback capable controlled robots 106 through the backend server 102; after the first communication connection is successfully established, full-duplex communication WebSocket authentication can be performed on the plurality of master devices 104 and the plurality of controlled robots 106 through the backend server 102, and after the authentication is passed, the plurality of master devices 104 and the plurality of controlled robots 106 are connected to the public network; then, the structure attribute information and the control software version information corresponding to the plurality of master devices 104 and the structure attribute information and the control software version information corresponding to the plurality of controlled robots 106 can be acquired respectively, and the structure attribute information and the control software version information can be stored, so that when a control instruction of at least one master device 104 is received through the public network, a target controlled robot that executes the control instruction can be matched from the plurality of controlled robots 106. Then, optionally, the target controlled robot can execute the control instruction, so as to realize remote control of the target controlled robot by the master device 104.

[0026] Optionally, the master device 104 in the embodiments of the present application can interact with a user (such as an operator, etc.), and can receive the operation of the user, and send a control instruction to the backend server 102 based on the operation, so as to facilitate remote control of the controlled robot 106. The master device 104 that sends the control instruction can be a target master device. The master device 104 can be constructed in any structure, which is not specifically limited in the embodiments of the present application. In some optional embodiments, the master device 104 can have the ability of force conversion, and when the master device 104 receives force feedback information (which can be torque information) from the controlled robot 106, a haptic state can be presented to the user based on the force feedback information.

[0027] For example, in an example of a user remotely controlling a controlled robot 106 via a master control device 104, the user (e.g., an operator) can hold the end effector of the master control device 104's robotic arm to drive the end effector of the controlled robot 106 (the target controlled robot) to execute the same trajectory. The force feedback information (e.g., torque information) obtained by the end effector of the controlled robot 106 after contacting an object will also act on the end effector of the master control device 104's robotic arm, allowing the user to directly feel the contact force. This provides a more intuitive and perceptible view of the tactile state of the controlled robot 106 during operation. Throughout the entire process of this embodiment, high-frequency message transmission enables low-latency, continuous remote control operation and force feedback transmission.

[0028] In some optional embodiments, the master control device 104 can execute a robot remote control method. Optionally, the master control device 104 can be a target master control device. When the target master control device executes the robot remote control method, it can establish a first communication connection based on the Message Queuing Telemetry Transport Protocol (MQTT) with multiple controlled robots 106 that have force feedback capabilities through the backend server 102. Then, after the first communication connection is successfully established, full-duplex WebSocket authentication is performed through the backend server 102. After successful authentication, the device connects to the public network through the backend server 102 so that when it is determined that the target controlled robot needs to be controlled, the control command is sent to the target controlled robot through the public network via the backend server 102. Optionally, the target controlled robot can then execute the control command, thereby realizing the remote control of the target controlled robot by the target master control device.

[0029] Optionally, the controlled robot 106 in this embodiment may be equipped with a hardware module (e.g., a sensor) capable of providing force feedback. This allows the controlled robot 106 to provide force feedback when needed during operation, thus providing a more intuitive and perceptible view of the robot's tactile state. The controlled robot 106 can be constructed in any structure, including but not limited to robotic arms, self-moving devices, and biomimetic robots.

[0030] In some optional embodiments, the controlled robot 106 can execute a robot remote control method. Optionally, the controlled robot 106 can be a target controlled robot. When the target controlled robot executes the robot remote control method, it can establish a first communication connection with multiple master control devices 104 via the MQTT protocol through the backend server 102. Then, after the first communication connection is successfully established, it performs full-duplex communication WebSocket authentication through the backend server 102. After successful authentication, it connects to the public network through the backend server 102 to execute operation instructions after receiving control instructions from the target master control device among the multiple master control devices 104, and feeds back the execution results to the target master control device through the backend server 102.

[0031] Optionally, Figure 1B A schematic diagram illustrating an example communication connection between the master control device 104, the controlled robot 106, and the backend server 102 is shown. Figure 1B As shown, in some optional embodiments, through the system of this solution, an MQTT connection can be established between any master control device 104 and the backend server 102, and a first public network transmission link 201 and a second public network transmission link 202 can be established based on the WebSocket protocol, thereby connecting to the public network through the backend server 102. Furthermore, as... Figure 1B As shown, in some optional embodiments, an MQTT connection can be established between any controlled robot 106 and the backend server 102, and a first public network transmission link 201 and a second public network transmission link 202 can be established based on the WebSocket protocol, thereby connecting to the public network through the backend server 102. The first public network transmission link 201 is used to transmit control commands from the master control device 104 to the public network of the controlled robot 106, and the second public network transmission link 202 is used to transmit feedback information from the controlled robot 106 to the public network of the master control device 104. For ease of explanation of the embodiments of this application, the above-mentioned parts will be described in detail in the robot remote control method embodiments below, and will not be repeated here.

[0032] Based on the robot remote control system 100 described above, this application provides a robot remote control scheme, which will be described below through several embodiments.

[0033] Figure 2 A flowchart illustrating the steps of a robot remote control method according to an embodiment of this application is shown. According to a first aspect of an embodiment of this application, a robot remote control method is provided, which can be used in a backend server 102, see reference... Figure 2 As shown, the method includes steps S202, S204, and S206, specifically: S202: Establish the first communication connection of the MQTT protocol between multiple master control devices and multiple controlled robots with force feedback capability through the backend server.

[0034] MQTT is a lightweight, publish / subscribe IoT messaging protocol that provides stable and reliable network services for devices. The publish / subscribe model used in MQTT is a decoupled communication model; message senders (publishers) and receivers (subscribers) do not need to know of each other's existence, communicating only with an intermediary broker. Therefore, using MQTT makes the system easily scalable. It should be understood that in this embodiment, a first communication connection based on the MQTT protocol is established between multiple master control devices 104 and multiple controlled robots with force feedback capabilities through a backend server 102, facilitating the large-scale deployment of robot remote control.

[0035] Optionally, in this embodiment of the application, the backend server 102 may deploy an intermediate broker server, which can treat multiple master control devices 104 and multiple controlled robots 106 as MQTT clients. The intermediate broker server can establish connections with each MQTT client respectively, thereby establishing a first communication connection based on MQTT between multiple master control devices 104 and multiple controlled robots 106 with force feedback capabilities.

[0036] In some optional embodiments, step S202 may include: after determining that the multiple master control devices 104 and the multiple controlled robots 106 have the conditions to access the public network, performing MQTT authentication on the multiple master control devices 104 and the multiple controlled robots 106 through the backend server 102 based on MQTT authentication information; after the MQTT authentication is passed, enabling the multiple master control devices 104 and the multiple controlled robots 106 to establish a first communication connection based on MQTT with the backend server 102 respectively.

[0037] In this embodiment, the backend server 102 can determine whether multiple master control devices 104 and multiple controlled robots 106 have the conditions to access the public network in any way. For example, when a device has the conditions to access the public network, it can meet the following conditions: normal hardware (e.g., normal network interface, processor, power supply, etc.), normal network (e.g., sufficient bandwidth, etc.), normal data security (e.g., normal authentication, encryption, firewall, etc.), normal software, and compliant access to the public network (e.g., obtaining operator permission, meeting data privacy requirements, etc.). For example, in some embodiments, the backend server 102 can perform MQTT authentication on multiple master control devices 104 and multiple controlled robots 106 when it is determined that multiple master control devices 104 and multiple controlled robots 106 all have the conditions to access the public network, and the backend server 102 itself has the conditions to access the public network.

[0038] Optionally, when the master control device 104 / controlled robot 106 has the conditions to access the public network, the backend server 102 can know in any way that the master control device 104 / controlled robot 106 has the conditions to access the public network. For example, in some embodiments, the master control device 104 / controlled robot 106 can continuously send status information indicating that it has the conditions to access the public network to the backend server 102 through the network when it has the conditions to access the public network; in other embodiments, when the master control device 104 / controlled robot 106 is in the state of having the conditions to access the public network, the backend server 102 can receive setting information from the administrator regarding whether the master control device 104 / controlled robot 106 has the conditions to access the public network, and determine that the master control device 104 / controlled robot 106 has the conditions to access the public network through the setting information. Of course, it is not limited to this, and other methods of determining whether the master control device 104 and controlled robot 106 have the conditions to access the public network are also applicable to the schemes of the embodiments of this application.

[0039] In some embodiments, the MQTT authentication information can be built into the device's storage unit, or it can be obtained by the device through other means such as a network. For example, optionally, for each master device 104 / controlled robot 106, the master device 104 / controlled robot 106, as an MQTT client, can establish an MQTT connection with the intermediate broker server deployed on the backend server 102 based on the built-in MQTT authentication information. The intermediate broker server deployed on the backend server 102 performs MQTT authentication on the master device 104 / controlled robot 106 based on the built-in MQTT authentication information to determine whether to establish an MQTT connection. After successful MQTT authentication, a first MQTT-based communication connection is established between the master device 104 / controlled robot 106 and the backend server 102. Thus, multiple master devices 104 and multiple controlled robots 106 can each establish a first MQTT-based communication connection with the backend server 102.

[0040] It should be understood that, through the above optional method, after determining that multiple master control devices 104 and multiple controlled robots 106 have the conditions to access the public network, the backend server performs MQTT authentication on the multiple master control devices 104 and multiple controlled robots 106 based on MQTT authentication information. After the MQTT authentication is passed, the multiple master control devices 104 and multiple controlled robots 106 respectively establish a first communication connection based on MQTT with the backend server 102. This effectively establishes a first communication connection based on the MQTT protocol between the multiple master control devices 104 and multiple controlled robots 106 through the backend server 102, thus providing conditions for the large-scale deployment of robot remote control in a public network environment.

[0041] S204: After the first communication connection is successfully established, the backend server performs full-duplex WebSocket authentication on multiple master control devices and multiple controlled robots, and after successful authentication, connects multiple master control devices and multiple controlled robots to the public network.

[0042] Full-duplex communication is a communication mode that allows data to be transmitted independently in both directions simultaneously. The WebSocket protocol is a communication protocol that enables full-duplex communication over a single TCP connection, allowing for real-time, bidirectional communication.

[0043] Optionally, after multiple master control devices 104 and multiple controlled robots 106 have successfully established a first communication connection with the backend server 102 based on the MQTT protocol, the backend server 102 can perform full-duplex communication WebSocket authentication on the multiple master control devices 104 and multiple controlled robots 106. After the WebSocket authentication is successful, a WebSocket connection can be established between the multiple master control devices 104 and multiple controlled robots 106 and the backend server 102, thereby connecting the multiple master control devices 104 and multiple controlled robots 106 to the public network through the backend server 102.

[0044] In some optional embodiments, step S204 may include: performing full-duplex WebSocket authentication on multiple master control devices 104 and multiple controlled robots 106 based on WebSocket authentication information via the backend server 102; and after successful authentication, establishing a first public network transmission link 201 and a second public network transmission link 202 between the multiple master control devices 104 and multiple controlled robots 106 and the backend server 102, respectively, to connect the multiple master control devices 104 and multiple controlled robots 106 to the public network through the backend server 102. The first public network transmission link 201 is used for transmitting control commands from the master control devices 104 to the public network of the controlled robots 106, and the second public network transmission link 202 is used for transmitting feedback information from the controlled robots 106 to the public network of the master control devices 104.

[0045] Optionally, refer to Figure 1B As shown, for each master control device 104, the backend server 102 can perform WebSocket authentication with any master control device 104. After successful authentication, a first public network transmission link 201 and a second public network transmission link 202 based on the WebSocket protocol are established between the backend server 102 and the master control device 104, thereby enabling the master control device 104 to connect to the public network through the backend server 102. Additionally, optionally, as... Figure 1B As shown, for each controlled robot 106, the backend server 102 can perform WebSocket authentication with any controlled robot 106, and after successful authentication, establish a first public network transmission link 201 and a second public network transmission link 202 based on the WebSocket protocol between the backend server 102 and the controlled robot 106, thereby enabling the controlled robot 106 to connect to the public network through the backend server 102.

[0046] For example, such as Figure 1BAs shown, when the control commands from the master control device 104 are transmitted to the controlled robot 106, the commands can first be transmitted to the backend server 102 via the first public network transmission link 201 established between the master control device 104 and the backend server 102, and then transmitted to the controlled robot 106 via the first public network transmission link 201 established between the backend server 102 and the controlled robot 106. The controlled robot 106 can then execute the control commands to achieve remote control operation. For example, as... Figure 1B As shown, when the feedback information of the controlled robot 106 is transmitted to the master control device 104, the feedback information is first transmitted to the back-end server 102 through the second public network transmission link 202 established between the controlled robot 106 and the back-end server 102, and then transmitted to the master control device 104 through the second public network transmission link 202 established between the back-end server 102 and the master control device 104, thereby realizing information feedback.

[0047] Optionally, the feedback information of the controlled robot 106 may include, but is not limited to, at least one of force feedback information and operational status feedback information. It is understood that the controlled robot 106 transmits force feedback information to the main control device 104 via the second public network transmission link 202. This allows the user (e.g., the operator) to receive force feedback information from the controlled robot 106 when needed during actual remote control of the robot 106. This provides a more intuitive and perceptible view of the tactile state of the robot 106 during operation. This has high application value in various usage scenarios of the controlled robot 106, especially in scenarios requiring precise operations via remote control, and can help ensure the high accuracy and reliability of remote control tasks.

[0048] It should be understood that, in the above optional implementation, after WebSocket authentication is passed, a first public network transmission link 201 and a second public network transmission link 202 are established between multiple master control devices 104 and multiple controlled robots 106 and the backend server 102, respectively. This effectively connects multiple master control devices 104 and multiple controlled robots 106 to the public network through the backend server 102, so as to effectively realize the large-scale deployment of robot remote control in a public network environment and make the system have low latency. In addition, since the first public network transmission link 201 based on the WebSocket protocol can be used for master control devices 104 and multiple controlled robots 106, the second public network transmission link 201 is established between the master control devices 104 and multiple controlled robots 106 and the backend server 102. The control commands from 04 are transmitted to the public network of the controlled robot 106. The establishment of a second public network transmission link 202 based on the WebSocket protocol can be used to transmit the feedback information from the controlled robot 106 to the public network of the master control device 104. Therefore, it is possible to realize the information closed loop of the robot remote control deployed on a large scale in the public network environment, from the master control device 104 to the controlled robot 106 issuing control commands, and the controlled robot 106 to the master control device 104 returning feedback information. This is beneficial to ensuring the remote control effect of the robot and helps to complete the remote control operation task with high accuracy and high reliability, so as to better meet the remote control usage needs of the robot.

[0049] After the first communication connection based on the MQTT protocol is established in step S202 and the public network connection is established in step S204, the multiple master control devices 104 and the multiple controlled robots 106 with force feedback capability can establish connections with the backend server 102 and all can reach the online state.

[0050] Optionally, such as Figure 1B As shown, the robot remote control system 100 may further include a front-end 108 that communicates with the back-end server 102. Information regarding multiple online master control devices 104 and multiple controlled robots 106 can be displayed through the front-end 108. Optionally, the front-end 108 can communicate with the back-end server 102 via the HTTP protocol, but this is not a limitation. For example, the front-end 108 may include a front-end page through which information can be displayed. Optionally, the front-end 108 can also receive operations performed by users (e.g., operators, administrators, etc.) on the online master control devices 104 and multiple controlled robots 106, generate corresponding information based on the operations, and transmit it to the back-end server 102 for processing.

[0051] S206: Obtain the structural attribute information and control software version information corresponding to multiple master control devices, as well as the structural attribute information and control software version information corresponding to multiple controlled robots, and store the structural attribute information and control software version information so as to match the target controlled robot to execute the control instruction from multiple controlled robots when receiving control instructions from at least one master control device through the public network.

[0052] Structural attribute information indicates the physical structure of the corresponding device, while control software version information indicates the control software version of the corresponding device. For different master control devices 104 and different controlled robots 106, the physical structural attributes and control software versions differ, meaning that a particular master control device 104 may not be compatible with all controlled robots 106. This poses a challenge to the large-scale deployment of multiple master control devices 104 and multiple controlled robots 106 in a public network environment. Therefore, in this embodiment, the structural attribute information and control software version information corresponding to multiple master control devices 104 and multiple controlled robots 106 can be obtained and considered to accurately match the target controlled robot to execute control commands from among the multiple controlled robots, thereby facilitating the operational adaptability of robot remote control in a planned deployment environment under public network conditions.

[0053] For example, taking the structural attribute information of the controlled robot 106 as an example, some controlled robots 106 have heads, while others do not; and among the controlled robots 106 with heads, some of their heads are rotatable, while others are not; or, for example, some controlled robots 106 have a movable body structure, while others are not; and so on. Similarly, the structural attribute information of the main control device 104 is also like this: some main control devices 104 have a device for controlling the rotation of the robot's head, while others do not; or, some main control devices 104 have a device for controlling the movement of the robot, while others do not; and so on. Of course, these are just some examples for ease of understanding and are not limited to this.

[0054] For example, regarding the control software version information of the controlled robot 106, some controlled robots 106 use version 1, while others use version 2, version 3, and so on. Similarly, the control software version information of the master control device 104 is also the same; some master control devices 104 use version 1, while others use version 2, version 3, and so on.

[0055] Therefore, structural attribute information and control software version information corresponding to multiple master control devices 104 and multiple controlled robots 106 can be acquired and stored. This allows the backend server 102 to match the appropriate target controlled robot from among the multiple controlled robots 106 when it receives control commands from at least one master control device 104. This achieves effective adaptation between the master control device 104 and the controlled robot 106, and also enables the master control device 104 to perform dimensionality-reduction adaptation to the controlled robot 106 when the structural attributes and / or control software version of the controlled robot 106 are lower than those of the master control device 104. This expands the applicability of the solution and saves on implementation costs.

[0056] Based on this, the optional implementation of steps S202~S206 in the embodiments of this application can be as follows: On the one hand, a first communication connection based on the MQTT protocol can be established between multiple master control devices and multiple controlled robots with force feedback capabilities through a backend server. After the first communication connection is successfully established, the backend server performs full-duplex WebSocket authentication on the multiple master control devices and multiple controlled robots. After successful authentication, the multiple master control devices and multiple controlled robots are connected to the public network. Thus, this solution can effectively realize the public network deployment of multiple master control devices and multiple controlled robots, thereby effectively realizing the large-scale deployment of robot remote control in a public network environment. It can also make the system have low latency. Since this solution does not require the establishment of a private network, it can also reduce the deployment cost of robot remote control. Furthermore, the large-scale deployment of robot remote control in a public network environment has relatively low dependence on training data and algorithms, thus further solving the cost problem when deploying robots on a large scale. On the other hand, the structural attribute information and control software version information corresponding to multiple master control devices, as well as the structural attribute information and control software version information corresponding to multiple controlled robots, can be obtained separately, and the structural attribute information can be stored. By integrating information on the control software version, this solution can accurately match the target robot from multiple controlled robots when receiving control commands from at least one master control device via the public network. This allows for precise identification of the target robot during remote control based on the actual structural attributes and software version status of both the master control device and the controlled robot, ensuring conditional compatibility between the control and controlled ends. This improves the convenience, reliability, and operational adaptability of large-scale robot remote control deployments in a public network environment, enabling more precise fulfillment of user needs. Furthermore, since the multiple controlled robots deployed on a large scale in this solution have force feedback capabilities, users (e.g., operators) can obtain force feedback information from the controlled robots during operation when needed. This provides a more intuitive and perceptible view of the tactile state of the controlled robots, which is highly valuable for various usage scenarios, especially those requiring precise operations via remote control. This contributes to the high accuracy and reliability of remote control tasks.

[0057] Optionally, the backend server 102 can automatically filter multiple controlled robots 106 after receiving the control command, and match the target controlled robot to execute the control command based on the filtering results.

[0058] In some optional embodiments, for each master control device 104 that issues an operation command, refer to Figure 3The flowchart shown illustrates steps S2062-S2064, which are used to match the target robot that executes the control commands: S2062: Based on the structural attribute information and control software version information of the main control device, the structural attribute information and control software version information of multiple controlled robots, and preset filtering conditions, filter multiple controlled robots to determine at least one candidate controlled robot.

[0059] The preset filtering conditions can be set as needed. For example, in some embodiments, the preset filtering conditions can be: filtering out controlled robots 106 whose structural attribute information and control software version information are incompatible with the main control device 104. Then, after filtering the multiple controlled robots 106, at least one controlled robot 106 whose structural attribute information and control software version information are compatible with the main control device 104 can be retained and identified as at least one candidate controlled robot.

[0060] Optionally, the adaptation of the structural attribute information of the controlled robot 106 to the master control device 104 may mean that the structural attribute information of the master control device 104 indicates that the master control device 104 has a device capable of controlling the corresponding structure of the controlled robot 106. For example, in some embodiments, the master control device 104 includes a device for controlling the movement of the robot's head, and a certain controlled robot 106 has a rotatable head, then the structural attribute information of the controlled robot 106 is adapted to the master control device 104.

[0061] Optionally, the compatibility of the control software version information of the controlled robot 106 with that of the main control device 104 can mean that the control software version information of the main control device 104 indicates that the control commands generated by the control software of the main control device 104 can be executed by the control software of the controlled robot 106. For example, in some embodiments, the control software version of the main control device 104 is the same as that of the controlled robot 106, or the control software version of the main control device 104 is higher than that of the controlled robot 106, in which case the control software version information of the controlled robot 106 is compatible with that of the main control device 104. In some optional embodiments, when the control software version of the main control device 104 is the same as that of the controlled robot 106, the main control device 104 can normally control the controlled robot 106 through control commands; while when the control software version of the main control device 104 is higher than that of the controlled robot 106, the main control device 104 can perform dimensional reduction control of the controlled robot 106 through control commands.

[0062] In some optional embodiments, step S2062 may include: filtering the multiple controlled robots 106 based on the structural attribute information and control software version information corresponding to the main control device 104, the structural attribute information and control software version information corresponding to the multiple controlled robots 106, the usage information corresponding to the controlled robots 106, and preset filtering conditions, to determine at least one candidate controlled robot; wherein, the usage information includes at least one of usage permission information and usage area information.

[0063] The usage permission information for different controlled robots 106 may differ. For example, some controlled robots 106 may be configured with high-level permissions, while others may be configured with low-level permissions, and so on. Alternatively, some controlled robots 106 may be configured to be accessible to the first master control device 104, while others may be configured to be accessible to the second master control device 104, and so on. Furthermore, some controlled robots 106 may be configured to be accessible to operator A, while others may be configured to be accessible to operator B, and so on. These are merely examples for ease of understanding and are not limited to this. Optionally, master control devices 104 or operators that do not meet the usage permissions of a controlled robot 106 will not be able to remotely control that robot 106.

[0064] The usage area information of different controlled robots 106 may differ. For example, some controlled robots 106 may be configured to use in factories, while others may be configured to use in hospitals, and so on. Of course, these are just some examples for ease of understanding and are not limited to this. Optionally, the main control device 104 cannot remotely operate a controlled robot 106 if it does not conform to the usage area of ​​the controlled robot 106.

[0065] In some example embodiments, when considering usage information (including at least one of usage permission information and usage area information), the preset filtering condition can be: filtering out controlled robots 106 that do not conform to the usage information and whose structural attribute information and control software version information are incompatible with the main control device 104. Thus, after filtering the multiple controlled robots 106, at least one controlled robot 106 that conforms to the usage information, whose structural attribute information is compatible with the main control device 104, and whose control software version information is compatible with the main control device 104 can be retained and identified as at least one candidate controlled robot.

[0066] It is understood that in this embodiment of the application, the structural attribute information and control software version information corresponding to the main control device 104, the structural attribute information and control software version information corresponding to multiple controlled robots 106, the usage information including at least one of the usage permission information and usage area information, and the preset filtering conditions are comprehensively considered to filter multiple controlled robots 106. This allows the matched candidate controlled robots to effectively adapt to the current usage permission and / or usage area, avoiding the problem of exceeding the usage permission or incorrectly controlling the controlled robot 106 in the wrong usage area during subsequent robot remote control operations. This effectively improves the accuracy, reliability, and operational adaptability of robot remote control deployed on a large scale in the public network environment, which is conducive to ensuring the remote control effect of the robot and helps to complete the remote control operation task with high precision and high reliability, so as to better meet the remote control usage needs of the robot.

[0067] S2064: Based on the pairing information between the master control device and the candidate controlled robot, pair the master control device with at least one candidate controlled robot.

[0068] The pairing information is used to pair the master control device 104 with the candidate controlled robot, so that the master control device 104 can subsequently remotely control the candidate controlled robot.

[0069] In some embodiments, pairing the master control device 104 with at least one candidate controlled robot can be achieved by pairing the master control device 104 with one candidate controlled robot, thereby realizing one-to-one remote control. Alternatively, if the remote control requirements can be met, the master control device 104 can be paired with multiple candidate controlled robots, thereby realizing one-to-many remote control.

[0070] Optionally, the pairing information can be generated based on a pairing operation received from a user (e.g., an operator). For example, the backend server 102 can be connected to the frontend 108, which can include a frontend page that displays the candidate controlled robots determined in step S2062. The frontend 108 can then transmit the information of the user's pairing operation between the main control device 104 and one or more candidate controlled robots to the backend server 102. The backend server 102 can then pair the main control device 104 with at least one candidate controlled robot based on the pairing information between the main control device 104 and the candidate controlled robots.

[0071] Alternatively, the pairing information can also include default pairing information. This can be selected as needed; there is no single limitation. For example, default pairing information may include, but is not limited to: default pairing information for directly pairing the master control device 104 with all candidate controlled robots; default pairing information for randomly pairing the master control device 104 with a specific candidate controlled robot; default pairing information for pairing the master control device 104 with candidate controlled robots listed in a preset robot list, and so on.

[0072] S2066: The candidate controlled robot that has successfully paired with the main control device is identified as the target controlled robot for executing control commands.

[0073] Based on this, the embodiments of this application, through the optional implementation of the above steps S2062~S2066, can comprehensively consider the structural attribute information and control software version information corresponding to each master control device 104 that issues operation commands, the structural attribute information and control software version information corresponding to multiple controlled robots 106, and preset filtering conditions to filter multiple controlled robots 106, thereby accurately determining the target controlled robot to be remotely controlled, ensuring the conditional compatibility between the control end and the controlled end, thereby improving the convenience, reliability and operational adaptability of robot remote control deployed on a large scale in a public network environment, enabling robot remote control to more accurately meet user needs, and helping to complete remote control operation tasks with high precision and high reliability, so as to better meet the remote control usage needs of robots.

[0074] In some optional embodiments, the robot remote control method of this application further includes: sending remote control commands to the master control device 104 that issues the control command and the target controlled robot that executes the control command through a first communication connection, so that the master control device 104 that issues the control command and the target controlled robot that executes the control command respectively open the first public network transmission link 201 and the second public network transmission link 202 between themselves and the backend server 102.

[0075] It should be understood that the first communication connection based on the MQTT protocol opens the first public network transmission link 201 and the second public network transmission link 202 between the master control device 104 and the target controlled robot. This allows the control commands from the master control device 104 to be effectively transmitted to the target controlled robot via the first public network transmission link 201, enabling the target controlled robot to execute the control commands and achieve remote control operation. Furthermore, the feedback information from the target controlled robot can be effectively transmitted to the master control device 104 via the second public network transmission link 202, thus achieving information feedback. Therefore, a complete closed loop of the entire robot command link and information feedback link can be achieved.

[0076] For example, in an example of a user remotely controlling a controlled robot 106 via a master control device 104, the user (e.g., an operator) can hold the end effector of the master control device 104's robotic arm to drive the end effector of the controlled robot 106 (the target controlled robot) to execute the same trajectory. The force feedback information (e.g., torque information) obtained by the end effector of the controlled robot 106 after contacting an object will also act on the end effector of the master control device 104's robotic arm, allowing the user to directly feel the contact force. This provides a more intuitive and perceptible view of the tactile state of the controlled robot 106 during operation. Throughout the entire process of this embodiment, high-frequency message transmission enables low-latency, continuous remote control operation and force feedback transmission.

[0077] In some optional embodiments, the robot remote control method of this application further includes: obtaining preview data of the surrounding environment of the target controlled robot before the target controlled robot executes the control command, and sending the preview data to the front end 108, wherein the preview data is non-interactive data.

[0078] Since the target robot, which is a physical entity, is the object of remote control operation, it is not safe to directly remotely control it without confirming the surrounding environmental conditions of the target robot's control site. Therefore, in this embodiment, before the target robot executes the control command, a preview of the target robot's surrounding environment in non-interactive data form is obtained, and then the preview data is sent to the front end 108 for display. This allows users (such as the operator of the main control device 104) to see the preview data and understand the surrounding environmental status of the target robot, so that users can make safety decisions before the actual remote control operation, thereby better ensuring the safety of the robot's remote control operation.

[0079] The preview data in non-interactive data form can be in any form. For example, the preview data can include, but is not limited to, at least one of the following: preview video data that can display the environment around the target controlled robot, preview image data that can display the environment around the target controlled robot, and text data that can describe the environment around the target controlled robot.

[0080] For example, taking video data as preview data, the front end 108 can enter the remote control preview page after the main control device 104 and the target controlled robot are paired. The target controlled robot can push the preview video data to the front end 108 through the back end server 102. The front end 108 can render and display the preview video data in the remote control preview page.

[0081] Optionally, the target controlled robot can be equipped with one or more cameras to collect video data of the surrounding environment in at least one direction for video streaming. In some examples, the target controlled robot can be equipped with three cameras, which can be installed on the robot's head and two hands respectively, thereby collecting three channels of video data of the surrounding environment. The backend server 102 pushes the preview video data to the frontend 108, and the frontend 108 can render and display the three channels of preview video data on the remote control preview page for user viewing. It is understood that, provided that the video data acquisition requirements can be met, the cameras can also be set outside the controlled robot; there is no unique limitation in this embodiment.

[0082] Understandably, when preview data begins to be displayed on the front end 108, the target controlled robot cannot yet execute control commands until the triggering conditions for executing control commands are met. For example, the triggering condition could be the receipt of a trigger signal. This trigger signal could be generated by the front end 108 receiving user (e.g., operator) input, and then received by the backend server 102 from the front end 108 and sent to the target controlled robot. Alternatively, the triggering condition could be that the target controlled robot continuously receives control commands from the main control device 104 for a preset duration, at which point the triggering condition is considered met. Of course, this is not the only possible scenario.

[0083] In some optional embodiments, the robot remote control method of this application further includes: after matching the target controlled robot, sending configuration information to the target controlled robot to indicate that the first communication connection corresponding to the target controlled robot is occupied; and sending configuration information to the front end 108 so that when the target controlled robot executes the control command, it pushes the obtained first video data through the back end server 102 to the front end 108 configured with the configuration information.

[0084] Since the target robot needs to execute control commands from a master control device 104 after being matched, in order to prevent the target robot from being remotely controlled by other master control devices 104 while it is being remotely controlled, configuration information indicating that the first communication connection corresponding to the target robot is occupied can be sent to the target robot in this embodiment of the application. This prevents the control commands from other master control devices 104 from remotely controlling the target robot, thereby avoiding interference during robot remote control and helping to complete the remote control operation task of the target robot with high accuracy and high reliability.

[0085] Furthermore, in this embodiment, the backend server 102 sends configuration information to both the target controlled robot and the frontend 108, thereby facilitating the target controlled robot to push video streams to the frontend 108 through the backend server 102. This allows the target controlled robot to effectively push the obtained first video data to the frontend 108, which has been configured using the configuration information, when executing control commands. This enables the frontend 108 to render and display the first video data, which is beneficial for users (e.g., operators) to refer to the first video data rendered and displayed by the frontend 108 when remotely controlling the target controlled robot through the main control device 104. This makes remote control operations more convenient and can further contribute to the high accuracy and high reliability of remote control operation tasks for the target controlled robot.

[0086] Optionally, the first video data can be the video data obtained by the target controlled robot when executing the control command. The specific acquisition process can be referred to the acquisition process of the preview video data mentioned above. Therefore, the relevant content about the camera acquiring video data can be understood in conjunction with the above text, and will not be repeated here.

[0087] In some optional embodiments, the configuration information may include channel number configuration information.

[0088] It should be understood that configuring the front-end 108 and the target controlled robot through the channel number configuration information enables the target controlled robot to accurately push the obtained first video data to the front-end 108 through the back-end server 102 when executing control commands, so that the front-end 108 can render and display the first video data.

[0089] Optionally, the first video data can also be used to ensure the safety of robot remote control operations. For example, when the first video data is sent to the front end 108 for rendering and display, if the user (operator) controlling the main control device 104 or other users (such as system administrators) view the first video data through the page of the front end 108 and determine that there is a safety risk in the robot remote control operation, they can stop the robot remote control operation in time.

[0090] In some optional embodiments, the robot remote control method of this application further includes: acquiring execution data generated during the execution of control commands by the target controlled robot and sending it to the front end 108, wherein the execution data includes unified robot description format information corresponding to the execution of control commands by the target controlled robot, control commands, and feedback information of the target controlled robot, wherein the unified robot description format information is used to generate first visual data in combination with control commands and to generate second visual data in combination with feedback information.

[0091] The Unified Robot Description Format (URDF) is an XML-based information format used to describe the structure of a robot. The URDF information of a target controlled robot can describe its structure; optionally, modeling of the target controlled robot can be achieved based on URDF information.

[0092] Optionally, both the first and second visual data may include the rendered robot model. The first visual data can be generated by the front-end 108 combining unified robot description format information and control commands, and can be used to indicate the theoretical operating form of the target controlled robot based on the control commands. The second visual data can be generated by the front-end 108 combining unified robot description format information and feedback information from the target controlled robot, and can be used to indicate the actual operating form of the target controlled robot based on the control commands.

[0093] It should be understood that by rendering the target robot in two forms (i.e., theoretical operating form and actual operating form) based on the execution data mentioned above through the front end 108, users (such as operators) can intuitively see the difference between the actual operating form and the theoretical operating state of the target robot from a third perspective. This makes it easier for users to make decisions on subsequent operations based on this difference, thereby facilitating the subsequent remote control operation of the target robot, effectively improving the robot remote control effect, and helping to complete the remote control operation task of the target robot with high accuracy and high reliability.

[0094] In some optional embodiments, the robot remote control method of this application further includes: receiving health status information of the target controlled robot through a first communication connection during the execution of control commands by the target controlled robot, and sending the health status information to the front end 108 so as to display the health status information through the front end 108.

[0095] Optionally, during the process of the target controlled robot executing control commands to achieve remote control of the robot, the target controlled robot can send its own health status information to the backend server 102 through the first communication connection based on the MQTT protocol. The backend server 102 can communicate with the frontend 108 through the HTTP protocol and send the health status information of the target controlled robot it receives to the frontend 108 through the HTTP protocol. The frontend 108 can display the health status information on the frontend page.

[0096] Therefore, through the above optional implementation methods, the health status information of the target controlled robot can be displayed intuitively to users (such as the operator of the main control device 104, the system administrator, etc.), so that users can deal with health problems of the target controlled robot in a timely manner to ensure the smooth progress of the robot remote control task.

[0097] Optionally, when displaying the health status information of the target controlled robot on the front end 108, if a health problem is determined to exist, corresponding alarm processing can be performed according to the different levels of the health problem. The levels of health problems can be preset as needed. For example, if the health problem is determined to be a low-level health problem with a minor impact on the remote control task, an alarm can be performed by displaying a warning icon on the front end page; while if the health problem is determined to be a high-level health problem with a significant impact on the remote control task, in addition to displaying a warning icon on the front end page, more direct methods such as audible and visual alarms can be used for alarm processing. Of course, this is just an example and is not limited to this; any alarm method that meets the requirements can be used.

[0098] In some optional embodiments, the robot remote control method of this application further includes: obtaining second video data containing control information of the user controlling the main control device 104 from the main control device 104, and sending the second video data to the front end 108.

[0099] Optionally, in this embodiment, considering the security risks of remote robot operation, a second video data stream (e.g., data collected via a camera) can be set to monitor the control information of the user controlling the main control device 104 (e.g., the operator of the main control device 104). This control information may include the user's control actions. Optionally, the backend server 102 can obtain the second video data from the main control device 104 and send it to the frontend 108 for rendering and display. When the user controlling the main control device 104 or other users (e.g., system administrators) view the second video data through the frontend 108 page and determine that the robot remote control operation poses a security risk, they can stop the robot remote control operation in a timely manner.

[0100] For example, if the operator controlling the main control device 104 discovers a safety risk in their control actions by viewing the second video data displayed on the front end 108, they can stop the robot remote control operation in time. Or, if the system administrator discovers a safety risk in the operator's control actions by viewing the second video data displayed on the front end 108, they can send a stop command to the system immediately, causing the system to disconnect the robot remote control link (e.g., turn off the main control device 104, turn off the target controlled robot, etc.) so that the robot remote control operation stops.

[0101] Therefore, by acquiring and sending the second video data containing the user's control information of the main control device 104 to the front end 108, the safety of robot remote control can be improved. This is conducive to the widespread and safe application of the robot large-scale deployment scheme in the public network environment of this application embodiment in various work scenarios.

[0102] In some optional embodiments, after the robot remote control operation task is completed, the front end 108 can receive a termination operation from a user (e.g., an operator) to terminate the remote control task. The front end 108 can then send the termination operation information to the back end server 102. Optionally, the back end server 102 can send termination remote control commands to the master control device 104 and the target controlled robot respectively (e.g., via a first communication connection based on the MQTT protocol) based on the termination operation information. This causes the master control device 104 and the target controlled robot to disconnect their WecSocket connections with the back end server 102 based on the termination remote control commands, thereby cutting off the first public network transmission link 201 and the second public network transmission link 202 to terminate the robot remote control. Further optionally, the push of second video data from the master control device 104 to the front end 108 via the back end server 102, and the push of first video data from the target controlled robot to the front end 108 via the back end server 102, can also be terminated simultaneously.

[0103] It is understood that in the entire system of this application embodiment, the control commands of the master control device 104 and the feedback information of the controlled robot 106 are relayed through the backend server 102. However, using this technical solution, the overall forwarding process is relatively fast, resulting in very low transmission latency. Furthermore, the benefit is that both the master control device 104 and the controlled robot 106 only need to establish a connection with the backend server 102, such as an MQTT connection, a WebSocket link, and video streaming. This facilitates large-scale robot deployment in a public network environment and improves the robot's remote control effect. Optionally, from Figure 1B As can be seen from the figure, in the system architecture of this application embodiment, the main control device 104 and the controlled robot 106 are similar in technical implementation. The software configuration of the two can be achieved through remote configuration in the cloud, which is also convenient and quick.

[0104] The following is based on Figure 1A and Figure 1B The robot remote control method of this application embodiment will be described with reference to a scenario example. This scenario example can be understood by substituting it for the remote control operation of a handling robot in a factory. Optionally, it can be combined with... Figure 1AAs shown in the example scenario, the robot remote control system 100 may include multiple master control devices 104, a backend server 102, and multiple controlled robots 106 (i.e., handling robots) with force feedback capabilities. The operator (i.e., the user) can operate the master control devices 104 to control the controlled robots 106.

[0105] Combination Figure 1B As shown, each master control device 104 can establish a first communication connection based on the MQTT protocol with the backend server 102, and each controlled robot 106 can establish a first communication connection based on the MQTT protocol with the backend server 102. After the first communication connection is successfully established, each master control device 104 and each controlled robot 106 performs full-duplex WebSocket authentication with the backend server 102. After successful authentication, a first public network transmission link 201 is established for transmitting control commands from the master control device 104 to the controlled robot 106 via the public network, and a second public network transmission link 202 is established for transmitting feedback information from the controlled robot 106 to the master control device 104 via the public network. This allows multiple master control devices 104 and multiple controlled robots 106 to connect to the public network through the backend server 102.

[0106] An operator can remotely control the robot using a target master control device among multiple master control devices 104. The target master control device receives the operator's commands, generates corresponding control instructions, and transmits these instructions to the backend server 102 via the first public network transmission link 201. Then, the backend server 102 combines the structural attribute information and control software version information of the target master control device, the structural attribute information and control software version information of the multiple controlled robots 106, the usage information of the controlled robots 106, and preset filtering conditions to filter the multiple controlled robots 106. Usage information may include, for example, usage permission information and usage area information. This filtering process identifies at least one candidate controlled robot that matches the target master control device in terms of structural attribute information, control software version information, and the operator has usage permission for, and is used within the factory area. The operator then selects the candidate controlled robot through the front end 108 to pair the target master control device with the selected candidate controlled robot. The successfully paired candidate controlled robot becomes the target controlled robot used to execute the control instructions of the target master control device. Subsequently, the target controlled robot can execute control commands to achieve remote control operation tasks. For example, in one example operation, the operator can hold the end effector of the target master control device to drive the end effector of the target controlled robot's end effector to execute the same trajectory. Thus, the target controlled robot can carry objects through the end effector. The force feedback information (e.g., torque information) obtained by the end effector of the target controlled robot after contacting the object is also transmitted through the second public network transmission link 202 and applied to the end effector of the target master control device's end effector. This allows the operator to directly feel the contact force, thereby providing the operator with a more intuitive and perceptible demonstration of the tactile state of the target controlled robot during operation.

[0107] Optionally, before the target controlled robot executes control commands, the target controlled robot can transmit preview data of the surrounding environment (e.g., preview video data) to the front end 108 for display via the backend server 102. Optionally, when the target controlled robot executes control commands, the target controlled robot can push the obtained first video data to the front end 108 for display via the backend server 102. Optionally, the target master control device can also transmit second video data containing operator control information to the front end 108 for display via the backend server 102. This enables multi-level and multi-purpose video data streaming. Optionally, during the process of the target controlled robot executing control commands (i.e., during the process of the robot moving objects), it can also transmit its own health status information to the front end 108 for display via the backend server 102. Through the above optional methods, relevant video data and relevant status data can be displayed to the operator of the target master control device and the system administrator via the front end 108.

[0108] Optionally, the front end 108 can also receive URDF information, control instructions, and feedback information of the target controlled robot when the target controlled robot executes the control instructions from the back end server 102, thereby rendering first visual data and second visual data, so as to show the operator the difference between the actual operating form and the theoretical operating form of the target controlled robot based on the control instructions, so as to facilitate the operator's subsequent remote control operation of the robot.

[0109] After the object handling is completed, the robot remote control task can be terminated. The operator can terminate the remote control task through the front end 108. The front end 108 can send the termination operation information to the back end server 102. Based on the termination operation information, the back end server 102 sends termination remote control commands to the target master control device and the target controlled robot respectively, so as to finally end the robot remote control.

[0110] It should be understood that the above scenario examples are only for the purpose of facilitating understanding of the embodiments of this application, and are not intended to limit the embodiments of this application. Therefore, it can be seen that the robot remote control scheme in the embodiments of this application can effectively realize the large-scale deployment of robot remote control in a public network environment, which is beneficial to improving the robot remote control effect.

[0111] It is understood that the foregoing description of the robot remote control method is merely an exemplary description of the embodiments of this application and is not intended to limit the embodiments of this application in any way.

[0112] Figure 4 A flowchart illustrating the steps of another robot remote control method according to an embodiment of this application is shown. According to a second aspect of an embodiment of this application, a robot remote control method is provided, which can be used for a target master control device among a plurality of master control devices 104, referring to… Figure 4 As shown, the method includes steps S402 and S404, specifically: S402: Establish the first communication connection based on the Message Queuing Telemetry Transport Protocol (MQTT) between the backend server and multiple controlled robots with force feedback capabilities; S404: After the first communication connection is successfully established, full-duplex WebSocket authentication is performed through the backend server. After successful authentication, the backend server connects to the public network so that when it is determined that the target robot needs to be controlled, the control command is sent to the target robot through the backend server via the public network.

[0113] Based on this, the optional implementation of steps S402-S404 in the embodiments of this application can, on the one hand, establish a first communication connection based on the MQTT protocol between a backend server and multiple controlled robots with force feedback capabilities. After the first communication connection is successfully established, the backend server accesses the public network, so that when it is necessary to control the target controlled robot, the control command can be sent to the target controlled robot via the backend server through the public network. Thus, this solution can effectively realize the public network deployment of multiple master control devices and multiple controlled robots, thereby effectively realizing the large-scale deployment of robot remote control in a public network environment, and can make the system have low latency. Since this solution does not require the establishment of a private network, it can also reduce the deployment cost of robot remote control. In a public network environment, the large-scale deployment of remote-controlled robots has a relatively low dependence on training data and algorithms, thus further reducing the cost of large-scale robot deployment. On the other hand, since the multiple controlled robots deployed on a large scale in the public network environment in this solution have force feedback capabilities, users (such as operators) can obtain force feedback information of the controlled robots when needed during actual remote control operations. This allows for a more intuitive and perceptible display of the tactile state of the controlled robots during operation, which has high application value in various usage scenarios of controlled robots, especially in scenarios where precise operations need to be performed by remotely controlling the controlled robots. This can help to complete remote control operation tasks with high accuracy and high reliability.

[0114] In some optional embodiments, the robot remote control method further includes: after connecting to the public network through the backend server 102, sending the structural attribute information and control software version information corresponding to the target master control device to the backend server 102, so that when the backend server 102 receives the control command of the target master control device, it matches the target controlled robot to execute the control command from the multiple controlled robots 106 based on the structural attribute information and control software version information corresponding to the target master control device and the structural attribute information and control software version information corresponding to the multiple controlled robots 106.

[0115] Based on this, when remotely controlling a robot, the target robot can be accurately determined according to the actual structural attributes and software version status of the master control device and the controlled robot. This ensures the conditional compatibility between the control end and the controlled end, thereby improving the convenience, reliability, and operational adaptability of robot remote control deployed on a large scale in a public network environment, and enabling robot remote control to more accurately meet the needs of users.

[0116] In some optional embodiments, "matching the target controlled robot for executing control commands from the multiple controlled robots 106 based on the structural attribute information and control software version information corresponding to the target master control device, and the structural attribute information and control software version information corresponding to the multiple controlled robots 106" includes: filtering the multiple controlled robots 106 based on the structural attribute information and control software version information corresponding to the target master control device, the structural attribute information and control software version information corresponding to the multiple controlled robots 106, and preset filtering conditions to determine at least one candidate controlled robot; pairing the target master control device with at least one candidate controlled robot based on the pairing information between the target master control device and the candidate controlled robot; and determining the candidate controlled robot that is successfully paired with the target master control device as the target controlled robot for executing control commands.

[0117] In some optional embodiments, multiple controlled robots 106 are filtered based on the structural attribute information and control software version information corresponding to the target master control device, the structural attribute information and control software version information corresponding to multiple controlled robots 106, and preset filtering conditions to determine at least one candidate controlled robot. This includes filtering multiple controlled robots 106 based on the structural attribute information and control software version information corresponding to the target master control device, the structural attribute information and control software version information corresponding to multiple controlled robots 106, the usage information corresponding to the controlled robots 106, and preset filtering conditions to determine at least one candidate controlled robot. The usage information includes at least one of usage permission information and usage area information.

[0118] In some optional embodiments, step S404, which involves performing full-duplex WebSocket authentication via a backend server and connecting to the public network via the backend server after successful authentication, may include: performing full-duplex WebSocket authentication via the backend server 102 based on WebSocket authentication information, and establishing a first public network transmission link 201 and a second public network transmission link 202 between the backend server 102 and the backend server 102 after successful authentication, so as to connect to the public network via the backend server 102; wherein, the first public network transmission link 201 is used for transmitting the control commands of the master control device 104 to the public network of the controlled robot 106, and the second public network transmission link 202 is used for transmitting the feedback information of the controlled robot 106 to the public network of the master control device 104.

[0119] In some optional embodiments, the robot remote control method further includes: receiving remote control commands sent by a backend server 102 through a first communication connection, and opening a first public network transmission link 201 and a second public network transmission link 202 between the robot and the backend server 102 based on the remote control commands.

[0120] In some optional embodiments, step S402 includes: after obtaining access to the public network, performing MQTT authentication based on MQTT authentication information through the backend server 102; and establishing a first MQTT-based communication connection with the backend server 102 after successful MQTT authentication; wherein, the backend server 102 also establishes first MQTT-based communication connections with multiple controlled robots 106 respectively. Optionally, the backend server 102 also establishes first MQTT-based communication connections with other master control devices 104 besides the target master control device among the multiple master control devices 104.

[0121] In some optional embodiments, the robot remote control method further includes: sending second video data containing control information of a user controlling a target master control device to a backend server 102, so as to send the second video data to the frontend 108 through the backend server 102.

[0122] It should be understood that the robot remote control method embodiment of the second aspect has been described in detail in the robot remote control method embodiment of the first aspect. Therefore, its related content and beneficial effects can be understood by referring to the method embodiment of the first aspect described above, and will not be repeated here.

[0123] Figure 5 A flowchart illustrating the steps of another robot remote control method according to an embodiment of this application is shown. According to a third aspect of an embodiment of this application, a robot remote control method is provided, which can be used for a target robot among a plurality of controlled robots 106 with force feedback capability, referring to… Figure 5 As shown, the method includes steps S502 and S504, specifically: S502: Establishes the first communication connection with multiple master control devices via the MQTT protocol through the backend server; S504: After the first communication connection is successfully established, full-duplex WebSocket authentication is performed through the backend server. After successful authentication, the system connects to the public network through the backend server. Upon receiving control commands from the target master device among multiple master devices, the system executes the operation commands and feeds back the execution results to the target master device through the backend server.

[0124] Based on this, the optional implementation of steps S502~S504 in the embodiments of this application involves, on the one hand, establishing a first communication connection with multiple master control devices via a backend server using the MQTT protocol. After the first communication connection is successfully established, full-duplex WebSocket authentication is performed via the backend server. After successful authentication, the system connects to the public network via the backend server. This allows the system to execute operation commands received from the target master control device among the multiple master control devices, and to feed back the execution results to the target master control device via the backend server. Therefore, this solution effectively enables the public network deployment of multiple master control devices and multiple controlled robots, thereby effectively achieving large-scale deployment of robot remote control in a public network environment. Furthermore, it allows the system to have low latency. Since this solution does not require the establishment of a private network, it also... This approach can reduce the deployment cost of robot remote control, and in a public network environment, the large-scale deployment of robot remote control has a relatively low dependence on training data and algorithms, thus further addressing the cost of large-scale robot deployment. On the other hand, since the multiple controlled robots deployed on a large scale in a public network environment in this solution have force feedback capabilities, users (such as operators) can obtain force feedback information from the controlled robots when needed during actual remote control operations. This allows for a more intuitive and perceptible display of the tactile state of the controlled robots during operation, which has high application value in various usage scenarios of controlled robots, especially in scenarios where precise operations need to be performed by remotely controlling the controlled robots. This can help to complete remote control tasks with high accuracy and high reliability.

[0125] In some optional embodiments, the target controlled robot is the controlled robot selected by the backend server 102 from multiple controlled robots 106 to execute the control instructions when it receives the control instructions from the target master control device. This is based on the structural attribute information and control software version information corresponding to the target master control device, as well as the structural attribute information and control software version information corresponding to multiple controlled robots 106.

[0126] Based on this, when remotely controlling a robot, the target robot can be accurately determined according to the actual structural attributes and software version status of the master control device and the controlled robot. This ensures the conditional compatibility between the control end and the controlled end, thereby improving the convenience, reliability, and operational adaptability of robot remote control deployed on a large scale in a public network environment, and enabling robot remote control to more accurately meet the needs of users.

[0127] In some optional embodiments, the target controlled robot is a candidate controlled robot that has been successfully paired with the target controlled robot by the backend server 102 after pairing the target main control device with at least one candidate controlled robot based on the pairing information between the target main control device and the candidate controlled robot. The at least one candidate controlled robot is the at least one controlled robot determined by the backend server 102 by filtering multiple controlled robots 106 based on the structural attribute information and control software version information corresponding to the target main control device, the structural attribute information and control software version information corresponding to multiple controlled robots 106, and preset filtering conditions.

[0128] In some optional embodiments, at least one candidate controlled robot is determined by the backend server 102 through filtering multiple controlled robots 106 based on the structural attribute information and control software version information corresponding to the target master control device, the structural attribute information and control software version information corresponding to multiple controlled robots 106, the usage information corresponding to the controlled robots 106, and preset filtering conditions. The usage information includes at least one of usage permission information and usage area information.

[0129] In some optional embodiments, step S504, which involves performing full-duplex WebSocket authentication via a backend server and connecting to the public network via the backend server after successful authentication, includes: performing full-duplex WebSocket authentication via the backend server 102 based on WebSocket authentication information, and establishing a first public network transmission link 201 and a second public network transmission link 202 between the backend server 102 and the backend server 102 after successful authentication, so as to connect to the public network via the backend server 102; wherein, the first public network transmission link 201 is used for transmitting the control commands of the master control device 104 to the public network of the controlled robot 106, and the second public network transmission link 202 is used for transmitting the feedback information of the controlled robot 106 to the public network of the master control device 104.

[0130] In some optional embodiments, the robot remote control method further includes: receiving remote control commands sent by a backend server 102 through a first communication connection, and opening a first public network transmission link 201 and a second public network transmission link 202 between the robot and the backend server 102 based on the remote control commands.

[0131] In some optional embodiments, step S502 includes: after obtaining access to the public network, performing MQTT authentication based on MQTT authentication information through the backend server 102; and establishing a first MQTT-based communication connection with the backend server 102 after successful MQTT authentication; wherein, the backend server 102 also establishes first MQTT-based communication connections with multiple master control devices 104 respectively. Optionally, the backend server 102 also establishes first MQTT-based communication connections with other controlled robots 106 besides the target controlled robot.

[0132] In some optional embodiments, the robot remote control method further includes: sending execution data generated during the execution of control instructions to a backend server 102, so as to send the execution data to the frontend 108 through the backend server 102. The execution data includes unified robot description format information corresponding to the target controlled robot executing the control instructions, the control instructions, and the feedback information of the target controlled robot. The unified robot description format information is used to generate first visual data in combination with the control instructions and to generate second visual data in combination with the feedback information.

[0133] In some optional embodiments, the robot remote control method further includes: sending preview data of the surrounding environment of the target controlled robot to a backend server 102 before executing the control command, so as to send the preview data to the frontend 108 through the backend server 102, wherein the preview data is non-interactive data.

[0134] In some optional embodiments, the robot remote control method further includes: receiving configuration information sent by the backend server 102 indicating that the first communication connection corresponding to the target controlled robot is occupied; and when executing the control command, pushing the obtained first video data through the backend server 102 to the frontend 108 after it has been configured using the configuration information.

[0135] In some optional embodiments, the configuration information includes channel number configuration information.

[0136] In some optional embodiments, the robot remote control method further includes: sending the health status information of the target controlled robot to the backend server 102 when executing the control command, so that the health status information can be sent to the frontend 108 for display through the backend server 102.

[0137] It should be understood that the robot remote control method embodiment of the third aspect has been described in detail in the robot remote control method embodiment of the first aspect. Therefore, its related content and beneficial effects can be understood by referring to the method embodiment of the first aspect described above, and will not be repeated here.

[0138] According to a fourth aspect of the present application, an electronic device is provided, comprising: a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other via the communication bus; the memory is used to store a computer program; and the processor is used to execute the aforementioned robot remote control method by running the computer program stored in the memory.

[0139] Optionally, the electronic device 1000 can be implemented as a backend server 102, a target master control device, or a data processing device in a target controlled robot. When the electronic device 1000 is implemented as a data processing device in the backend server 102, its processor can run a computer program stored in the memory to execute the robot remote control method in the first aspect described above. When the electronic device 1000 is implemented as a data processing device in the target master control device, its processor can run a computer program stored in the memory to execute the robot remote control method in the second aspect described above. When the electronic device 1000 is implemented as a data processing device in the target controlled robot, its processor can run a computer program stored in the memory to execute the robot remote control method in the third aspect described above.

[0140] Figure 6 A structural block diagram of an optional electronic device according to an embodiment of this application is shown. This application does not limit the specific implementation of the electronic device 1000; however, as an example, reference is made to... Figure 6 The electronic device 1000 provided in this application embodiment includes: a processor 1002, a communications interface 1004, a memory 1006, and a communication bus 1008. Wherein: The processor 1002, communication interface 1004, and memory 1006 communicate with each other via communication bus 1008.

[0141] Communication interface 1004 is used to communicate with other electronic devices or servers.

[0142] The processor 1002 is used to execute the computer program 1010, which can specifically execute the relevant steps in any of the aforementioned robot remote control method embodiments.

[0143] Specifically, computer program 1010 may include program code that includes computer operation instructions.

[0144] The processor 1002 may be a CPU, a GPU (Graphics Processing Unit), an Application Specific Integrated Circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of this application. The one or more processors included in the smart device may be processors of the same type, such as one or more CPUs; or they may be processors of different types, such as one or more CPUs and one or more ASICs.

[0145] Memory 1006 is used to store computer program 1010. Memory 1006 may include high-speed RAM memory, and may also include non-volatile memory, such as at least one disk storage device.

[0146] Specifically, computer program 1010 can be used to cause processor 1002 to execute the robot remote control method in any of the foregoing embodiments.

[0147] The specific implementation of each step in computer program 1010 can be found in the corresponding steps and units described in any of the foregoing robot remote control method embodiments, and will not be repeated here. Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working process of the devices and modules described above can be referred to the corresponding process descriptions in the foregoing method embodiments, and will not be repeated here.

[0148] The electronic device 1000 in this application embodiment has been described in detail in the foregoing robot remote control method embodiment. Therefore, its related content and beneficial effects can be understood by referring to the above method embodiment, and will not be repeated here.

[0149] According to a fifth aspect of the embodiments of this application, the embodiments of this application also provide a computer storage medium storing a computer program thereon, which, when executed by a processor, implements the robot remote control method described in any of the foregoing method embodiments. The computer storage medium includes, but is not limited to, compact disc read-only memory (CD-ROM), random access memory (RAM), floppy disk, hard disk, or magneto-optical disk, etc.

[0150] According to a sixth aspect of the embodiments of this application, the embodiments of this application also provide a computer program product, including a computer program that, when executed by a processor, implements the robot remote control method as described in any of the embodiments of the plurality of method embodiments above.

[0151] The electronic device 1000 / computer storage medium / computer program product embodiment in this application has been described in detail in the foregoing robot remote control method embodiment. Therefore, its related content and beneficial effects can be understood by referring to the above method embodiment, and will not be repeated here.

[0152] Furthermore, it should be noted that the user-related information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to sample data used for training the model, data used for analysis, stored data, displayed data, etc.) involved in the embodiments of this application are all information and data authorized by the user or fully authorized by all parties. Moreover, the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and corresponding operation entry points are provided for users to choose to authorize or refuse.

[0153] It should be noted that, depending on the implementation needs, the various components / steps described in the embodiments of this application can be broken down into more components / steps, or two or more components / steps or parts of the operation of components / steps can be combined into new components / steps to achieve the purpose of the embodiments of this application.

[0154] The methods described in the embodiments of this application can be implemented in hardware, firmware, or as software or computer code that can be stored in a recording medium (such as a CD-ROM, RAM, floppy disk, hard disk, or magneto-optical disk), or as computer code downloaded over a network that is originally stored in a remote recording medium or a non-transitory machine-readable medium and will be stored in a local recording medium. Thus, the methods described herein can be stored on a recording medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware (such as an Application Specific Integrated Circuit (ASIC) or a Field Programmable Gate Array (FPGA)). It is understood that the computer, processor, microprocessor controller, or programmable hardware includes storage components (e.g., Random Access Memory (RAM), Read-Only Memory (ROM), Flash Memory, etc.) capable of storing or receiving software or computer code, which, when accessed and executed by the computer, processor, or hardware, implements the methods described herein. Furthermore, when a general-purpose computer accesses code used to implement the methods shown herein, the execution of the code transforms the general-purpose computer into a dedicated computer for executing the methods shown herein.

[0155] Those skilled in the art will recognize that the units and method steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for specific applications, but such implementations should not be considered beyond the scope of the embodiments of this application.

[0156] The term "comprising" and its variations as used herein are open-ended, meaning "including but not limited to". The term "based on" means "at least partially based on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments". It should be noted that the concepts of "first", "second", etc., mentioned in the embodiments of this application are only used to distinguish different devices, modules, or units, and are not used to limit the order of functions performed by these devices, modules, or units or their interdependencies. It should be noted that the modifications of "a" and "a plurality" mentioned in the embodiments of this application are illustrative rather than restrictive, and those skilled in the art should understand that unless otherwise expressly indicated in the context, they should be understood as "one or more".

[0157] The above embodiments are only used to illustrate the embodiments of this application, and are not intended to limit the embodiments of this application. Those skilled in the art can make various changes and modifications without departing from the spirit and scope of the embodiments of this application. Therefore, all equivalent technical solutions also fall within the scope of the embodiments of this application, and the patent protection scope of the embodiments of this application should be defined by the claims.

Claims

1. A robot remote control method for a backend server, the method comprising: Through the backend server, a first communication connection based on the Message Queue Telemetry Transmission Protocol (MQTT) is established between multiple master control devices and multiple controlled robots with force feedback capabilities. After the first communication connection is successfully established, the backend server performs full-duplex WebSocket authentication on multiple master control devices and multiple controlled robots, and after the authentication is passed, the multiple master control devices and multiple controlled robots are connected to the public network. The structural attribute information and control software version information corresponding to multiple master control devices, as well as the structural attribute information and control software version information corresponding to multiple controlled robots, are obtained respectively. The structural attribute information and control software version information are stored so that when a control command is received from at least one of the master control devices through the public network, a target controlled robot that executes the control command is matched from the multiple controlled robots.

2. The method according to claim 1, wherein, The step of performing full-duplex WebSocket authentication on multiple master control devices and multiple controlled robots through the backend server, and connecting the multiple master control devices and multiple controlled robots to the public network after successful authentication, includes: The backend server performs full-duplex WebSocket authentication on multiple master control devices and multiple controlled robots based on WebSocket authentication information. After the authentication is successful, a first public network transmission link and a second public network transmission link are established between the multiple master control devices and multiple controlled robots and the backend server, so as to connect the multiple master control devices and multiple controlled robots to the public network through the backend server. The first public network transmission link is used to transmit control commands from the master control device to the public network of the controlled robot, and the second public network transmission link is used to transmit feedback information from the controlled robot to the public network of the master control device.

3. The method according to claim 2, wherein, The method further includes: Through the first communication connection, remote control commands are sent to the master control device that issues the control command and the target controlled robot that executes the control command, so that the master control device that issues the control command and the target controlled robot that executes the control command respectively open the first public network transmission link and the second public network transmission link with the backend server.

4. The method according to claim 1, wherein, When receiving a control command from at least one of the master control devices via the public network, the step of matching a target controlled robot from among the multiple controlled robots to execute the control command includes: For each master control device that issues a control command: Based on the structural attribute information and control software version information of the main control device, the structural attribute information and control software version information of the multiple controlled robots, and preset filtering conditions, the multiple controlled robots are filtered to determine at least one candidate controlled robot. Based on the pairing information between the main control device and the candidate controlled robot, the main control device is paired with at least one candidate controlled robot; The candidate controlled robot that successfully pairs with the main control device is identified as the target controlled robot for executing the control commands.

5. The method according to claim 4, wherein, The step of filtering multiple controlled robots based on the structural attribute information and control software version information corresponding to the main control device, the structural attribute information and control software version information corresponding to multiple controlled robots, and preset filtering conditions to determine at least one candidate controlled robot includes: Based on the structural attribute information and control software version information of the main control device, the structural attribute information and control software version information of the multiple controlled robots, the usage information of the controlled robots, and preset filtering conditions, the multiple controlled robots are filtered to determine at least one candidate controlled robot. The usage information includes at least one of usage permission information and usage area information.

6. The method according to any one of claims 1-5, wherein, The method further includes: The execution data generated during the execution of the control command by the target controlled robot is acquired and sent to the front end. The execution data includes the unified robot description format information corresponding to the execution of the control command by the target controlled robot, the control command, and the feedback information of the target controlled robot. The unified robot description format information is used to generate first visual data in combination with the control command and to generate second visual data in combination with the feedback information.

7. The method according to any one of claims 1-5, wherein, The method further includes: Before the target controlled robot executes the control command, preview data of the surrounding environment of the target controlled robot is obtained and sent to the front end. The preview data is non-interactive data.

8. The method according to any one of claims 1-5, wherein, The method further includes: After the target controlled robot is matched, configuration information is sent to the target controlled robot to indicate that the first communication connection corresponding to the target controlled robot is occupied; Furthermore, the configuration information is sent to the front end so that when the target controlled robot executes the control command, the obtained first video data is pushed through the backend server to the front end after being configured using the configuration information.

9. The method according to any one of claims 1-5, wherein, The method further includes: During the execution of the control command by the target controlled robot, the health status information of the target controlled robot is received through the first communication connection, and the health status information is sent to the front end for display.

10. The method according to any one of claims 1-5, wherein, The method further includes: The second video data containing the user's control information for operating the main control device is obtained from the main control device, and the second video data is sent to the front end.

11. A robot remote control method for a target master control device among multiple master control devices, the method comprising: Establish the first communication connection based on the Message Queue Telemetry Transport Protocol (MQTT) between the backend server and multiple controlled robots with force feedback capabilities; After the first communication connection is successfully established, full-duplex WebSocket authentication is performed through the backend server. After the authentication is successful, the system connects to the public network through the backend server so that when it is determined that the target robot needs to be controlled, the control command is sent to the target robot through the public network via the backend server.

12. A robot remote control method for a target controlled robot among multiple controlled robots with force feedback capability, the method comprising: Establish the first communication connection with multiple master control devices via the backend server using the Message Queue Telemetry Transport Protocol (MQTT). After the first communication connection is successfully established, full-duplex WebSocket authentication is performed through the backend server. After the authentication is successful, the system connects to the public network through the backend server. Upon receiving the control command from the target master device among the multiple master control devices, the system executes the operation command and feeds back the execution result to the target master control device through the backend server.

13. An electronic device, comprising: The processor, the communication interface, the memory, and the communication bus are provided, wherein the processor, the communication interface, and the memory communicate with each other via the communication bus. The memory is used to store computer programs; The processor is configured to perform the method of any one of claims 1-12 by running the computer program stored in the memory.

14. A computer storage medium having a computer program stored thereon, which, when executed by a processor, implements the method as described in any one of claims 1-12.

15. A computer program product comprising a computer program that, when executed by a processor, implements the method as described in any one of claims 1-12.

Images