Method and related device for remotely controlling a robot

Abstract

The method and related device for remotely controlling a robot provided in the embodiment of the present application relate to the field of robot control technology. The method includes: obtaining a real-time three-dimensional space image of the operator's hand; identifying the feature points of the operator's hand according to the three-dimensional space image; determining the coordinates of the feature points according to the three-dimensional space image; determining the posture information of the operator's hand according to the coordinates of the feature points, generating a control signal according to the posture information, and sending the control signal to the robot, and the control signal is used to drive the robot's mechanical arm to synchronize with the operator's hand operation. In this way, the posture information of the operator's hand can be determined based on the three-dimensional space image, and then the control signal can be obtained, which reduces the cost of controlling the robot; the posture information includes the direction of the palm and the angle between the two knuckles of each finger, which can achieve fine control of the robot.

Description

Technical Field

[0001] The present application relates to the field of robotics technology, and in particular to a method for remotely controlling a robot and a related device. Background Art

[0002] With the development of industrial automation, robots are being used more and more widely. Robots can replace workers to work in dangerous, harmful, toxic, low-temperature, high-heat and other harsh environments.

[0003] Using data gloves to remotely control robots is currently a common means of controlling robots. The high cost of data gloves restricts the application of robots, and the control accuracy of related operations is not high, resulting in the inability of robots to perform delicate operations. Summary of the invention

[0004] In view of this, an embodiment of the present application provides a method and related device for remotely controlling a robot, which can reduce the cost of remotely controlling the robot and improve the control accuracy of the robot.

[0005] In the first aspect, the embodiment of the present application proposes a method for remotely controlling a robot: the method includes: obtaining a real-time three-dimensional spatial image of an operator's hand; identifying the characteristic points of the operator's hand according to the three-dimensional spatial image, the characteristic points including: the fingertips of each finger, the distal fingertip joints of each finger, the proximal fingertip joints of each finger, and the joints of the palms corresponding to each finger; determining the coordinates of the characteristic points according to the three-dimensional spatial image; determining the posture information of the operator's hand according to the coordinates of the characteristic points, the posture information of the operator's hand including: the orientation of the operator's palm, the finger angle at the distal fingertip joints of each finger, and the finger angle at the proximal fingertip joints of each finger; generating a control signal according to the posture information of the operator's hand, and sending the control signal to the robot, the control signal is used to drive the robot's mechanical arm to synchronize with the operator's hand operation. By adopting the technical solution provided by this embodiment, the posture information of the operator's hand can be determined based on the three-dimensional spatial image, and then the control signal is obtained, which reduces the cost of controlling the robot. At the same time, the posture information includes the palm orientation and the angle between the two finger joints of each finger, which can achieve fine control of the robot.

[0006] In some possible implementations, the position information of the operator's hand is determined based on the coordinates of the feature points, including: taking the normal vector of the plane determined by the palm joint point of the thumb and the palm joint points corresponding to any two fingers of the other four fingers as the orientation of the operator's palm. In this way, by accurately determining the position of the hand, fine control can be performed. It is understandable that in order to improve accuracy, the palm can provide a direction as a whole, or it can be divided into multiple areas, each area corresponding to a direction, which is conducive to more precise control.

[0007] In some possible implementations, the method for remotely controlling the robot further includes: obtaining the initial position of the operator's hand; and determining the origin of the operator side and the initial position of the operator's hand according to the initial position. It is understandable that after the initial position is determined, the coordinates of the characteristic points of the operator's hand can be further calibrated, which is conducive to clearly understanding the position information of the operator's hand.

[0008] In some possible implementations, determining the origin on the operator side according to the initial position includes: using the joint point of the palm corresponding to the thumb in the initial position of the operator's hand as the origin on the operator side. This helps the operator to intuitively understand and perceive the position of the hand, and the operator can more accurately determine the position and direction of the hand and fingers in space, which is conducive to improving the accuracy and precision of the operation.

[0009] In some possible implementations, the control signal includes a first difference between the real-time position of the operator's hand and the initial position; and a second difference between the orientation of the palm corresponding to the real-time position of the operator's hand and the orientation of the palm corresponding to the initial position. It can be understood that by using the difference between the real-time position and the initial position of the operator's hand, and the difference in the palm orientation as the control signal, the hand dynamics can be accurately captured, the motion tracking can be achieved, and the accuracy of the remote control can be improved.

[0010] In some possible implementations, the method for remotely controlling the robot further includes: displaying a real-time three-dimensional spatial image of the operator's hand. It is understandable that when the operator remotely controls the robot, by observing the displayed real-time three-dimensional spatial image, the operator can be provided with intuitive visual feedback, which is conducive to improving the operation accuracy, and is also conducive to assisting the operator to make correct decisions for subsequent operations and reduce misoperations.

[0011] In some possible implementations, the method for remotely controlling the robot further includes: superimposing a line drawing of the operator's hand structure formed according to the identified feature points of the operator's hand on the displayed real-time three-dimensional space image. In this way, the hand structure can be clearly displayed, assisting the operator to intuitively understand the mapping relationship between each feature point of the hand and the hand, thereby enhancing the intuitiveness of the operation.

[0012] In some possible implementations, the method for remotely controlling a robot further includes: displaying an image on the robot side in real time, including a robotic arm and a target object operated by the robotic arm in the image on the robot side; displaying optional operation options for the target object; determining the target operation according to the selected operation instruction; displaying an operation reminder mark in combination with the target operation and the positional relationship between the robotic arm and the target object; the operation reminder mark includes one or more of the following prompts: a reminder mark for indicating the direction of movement of the operator's hand, a reminder mark for indicating that the operator's hand performs finger grasping, and a mark for indicating that the operator's hand performs finger extension. In this way, by displaying the robot side image and operation options in real time, the operation decision is simplified, and the intuitiveness and real-time performance are improved. The operation reminder mark can guide the operator's actions, reduce the difficulty of operation, and improve the operation efficiency.

[0013] In a second aspect, an embodiment of the present application provides a device for remotely controlling a robot, comprising: a communication circuit, one or more processors and a memory; the communication circuit and the memory are coupled to the one or more processors, the communication circuit is used to obtain a real-time three-dimensional image of the operator's hand, and to send a control signal to the robot; the memory is used to store computer program code, the computer program code includes computer instructions, and one or more processors call the computer instructions to enable the device for remotely controlling the robot to implement the method described in the first aspect or any possible implementation of the first aspect.

[0014] In a third aspect, an embodiment of the present application provides a robot comprising: a robotic arm, a communication circuit, and a control circuit, wherein the communication circuit is used to receive a control signal generated by a device for remotely controlling the robot according to the method described in the first aspect or any possible implementation of the first aspect; and the control circuit is used to drive the robotic arm in synchronization with the operator's hand operation according to the control signal.

[0015] In a fourth aspect, an embodiment of the present application provides a remote control system, comprising: a first camera, a robot, and a device for remotely controlling the robot; the device for remotely controlling the robot is in communication connection with the first camera and the robot; wherein the first camera is used to obtain a three-dimensional spatial image of the operator's hand; the device for remotely controlling the robot comprises: a communication circuit, a memory, and one or more processors; the communication circuit, the memory are coupled with the one or more processors, the memory is used to store computer program code, the computer program code includes computer instructions, and the one or more processors call the computer instructions to make the electronic device execute the method described in the first aspect or any possible implementation of the first aspect. The robot comprises: a mechanical arm, a communication circuit, and a control circuit; the communication circuit is used to receive a control signal generated by the device for remotely controlling the robot according to the method described in the first aspect or any possible implementation of the first aspect; the control circuit is used to drive the mechanical arm to synchronize with the operator's hand operation according to the control signal.

[0016] In some possible implementations, the remote control system also includes a second camera for acquiring images on the robot side in real time, the images on the robot side including the robotic arm and the target object operated by the robotic arm; the device for remotely controlling the robot is also used to display the robot's optional operation options for the target object; determine the target operation according to the selected operation instruction; and display an operation reminder mark in combination with the target operation and the positional relationship between the robotic arm and the target object; the operation reminder mark includes one or more of the following prompts: a reminder mark for indicating the direction of movement of the operator's hand, a reminder mark for indicating the operator's hand to perform finger grasping, and a mark for indicating the operator's hand to perform finger stretching.

[0017] In the fifth aspect, an embodiment of the present application provides a chip system, which is applied to an electronic device, and the chip system includes one or more processors, and the one or more processors are used to call computer instructions so that the electronic device executes the method described in the first aspect or any possible implementation of the first aspect.

[0018] In a sixth aspect, an embodiment of the present application provides a computer-readable storage medium, which includes computer instructions. When the computer instructions are executed on an electronic device, the electronic device executes the method described in the first aspect or any possible implementation of the first aspect.

[0019] In the seventh aspect, an embodiment of the present application provides a computer program product, which includes a computer program code. When the computer program code runs on an electronic device, the electronic device executes the method described in the first aspect or any possible implementation of the first aspect.

[0020] It can be understood that the beneficial effects that can be achieved by the device for remotely controlling a robot of the second aspect, the robot of the third aspect, the remote control system of the fourth aspect, the chip system of the fifth aspect, the computer-readable storage medium of the sixth aspect, and the computer program product of the seventh aspect provided above can be referred to the beneficial effects of the first aspect and any possible implementation manner thereof, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1A It is a schematic diagram of an application scenario provided by an embodiment of the present application;

[0022] Figure 1B is a schematic diagram of the positions of characteristic points of a hand provided in an embodiment of the present application;

[0023] Figure 1C is a schematic diagram of an application scenario provided by another embodiment of the present application;

[0024] Figure 2 is a flowchart of a method for remotely controlling a robot provided in one embodiment of the present application;

[0025] Figure 3 is a schematic diagram of a robot picking up a water cup in one embodiment of the present application;

[0026] Figure 4A is a schematic structural diagram of a device for remotely controlling a robot provided in one embodiment of the present application;

[0027] Figure 4B is a schematic structural diagram of a device for remotely controlling a robot provided in another embodiment of the present application;

[0028] Figure 5 It is a schematic diagram of the structure of a robot provided in one embodiment of the present application. DETAILED DESCRIPTION

[0029] In order to make the objectives, technical solutions and advantages of the present application clearer, the implementation methods of the present application will be further described in detail below in conjunction with the accompanying drawings.

[0030] It should be understood that the "multiple" mentioned in this application refers to two or more. In the description of this application, unless otherwise specified, " / " means or, for example, A / B can mean A or B; "and / or" in this article is only a description of the association relationship of associated objects, indicating that there can be three relationships, for example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. In addition, in order to facilitate the clear description of the technical solution of this application, the words "first" and "second" are used to distinguish between the same items or similar items with basically the same functions and effects. Those skilled in the art can understand that the words "first" and "second" do not limit the quantity and execution order, and the words "first" and "second" do not limit them to be different.

[0031] In applications related to remote control of robots, when using data gloves to control the robot's mechanical arm, the hand movement information is captured by the sensors on the data gloves and converted into digital signals for transmission to the computer. After the computer analyzes these signals, it converts them into control instructions to drive the mechanical arm to perform the corresponding actions. In order to improve the control accuracy, data gloves need to have built-in high-precision sensors, and related devices (such as computers) need to use complex data processing algorithms, which correspondingly require higher manufacturing and R&D costs. In addition, data gloves have transmission delays when transmitting data, and sensors have large errors when recognizing complex gestures. These have limited the application of using data gloves to control robots.

[0032] An application scenario of the method for remotely controlling a robot provided by an embodiment of the present application is as follows: Figure 1A As shown, the device for remotely controlling the robot (such as Figure 1A The computer 101 in the robot 104 obtains a real-time three-dimensional image of the operator's hand 102, identifies feature points of the operator's hand 102 according to the three-dimensional image, determines the coordinates of the feature points according to the three-dimensional image, determines the position and posture information of the operator's hand 102 according to the coordinates of the feature points, generates a control signal according to the position and posture information, and sends the control signal to the robot 104. The control signal drives the mechanical arm of the robot 104 to synchronize with the operator's hand operation, so as to realize operations such as picking up, moving, or putting down the target object (such as the water cup 105).

[0033] The following introduces a method for remotely controlling a robot provided by an embodiment of the present application through a possible implementation process.

[0034] For example, Figure 2 As shown, a flow chart of a method for remotely controlling a robot according to an embodiment of the present application is shown, which may include steps: 201 to 205.

[0035] 201. Obtain real-time three-dimensional spatial images of the operator's hands.

[0036] For example, a three-dimensional camera module (such as a depth camera, etc.) can be used to obtain a three-dimensional spatial image of the operator's hand, and then transmitted to a device for remotely controlling the robot by wire or wireless means. The three-dimensional camera module can capture depth of field images to form a 3D stereo model. The three-dimensional camera module can include: an infrared camera, a dot matrix projector, and a distance sensor, etc., or an Apple mobile phone with a faceID camera module can be used to capture the operator's hand. It can be understood that this embodiment can realize the relevant functions using a general terminal with a three-dimensional camera module without the need to configure a special data glove.

[0037] In order to further improve the accuracy of operator hand recognition, multiple three-dimensional camera modules can be set, for example, a group of cameras can be added and placed in the vertical direction of the existing camera, and the images of the two groups of cameras are fused to obtain a three-dimensional spatial image of the operator's hand. It is understandable that more than two three-dimensional camera modules can be set, and the angles between different three-dimensional camera modules can be set according to actual conditions.

[0038] 202. Recognize the feature points of the operator's hand based on the three-dimensional space image.

[0039] In an embodiment of the present application, the operator's hand feature points may include: the fingertips of each finger, the distal fingertip joints of each finger, the proximal fingertip joints of each finger, and the palm joints corresponding to each finger. The distal fingertip joints are positions corresponding to the joints between the middle phalanx and the distal phalanx. The distal fingertip joints can cooperate with the hand to flex and extend, help to get closer, and increase grip strength. The proximal fingertip joints are positions corresponding to the joints between the proximal phalanx and the middle phalanx close to the palm. Figure 1B As shown, the fingertips of the thumb, index finger, middle finger, ring finger and little finger are a1, b1, c1, d1 and e1 respectively; the distal fingertip joints of the thumb, index finger, middle finger, ring finger and little finger are a2, b2, c2, d2 and e2 respectively; the proximal fingertip joints of the thumb, index finger, middle finger, ring finger and little finger are a3, b3, c3, d3 and e3 respectively; the joints of the palm of the thumb, index finger, middle finger, ring finger and little finger are A, B, C, D and E respectively.

[0040] It is understandable that in order to perform more precise recognition of the operator's hand, more feature points can be set, such as Figure 1B As shown in point a4 etc.

[0041] 203. Determine the coordinates of the feature points based on the three-dimensional space image.

[0042] In an embodiment of the present application, before determining the coordinates of the feature points according to the three-dimensional space image, the method may further include: obtaining the initial position of the operator's hand. Exemplarily, when the three-dimensional space image is obtained by the depth camera, if the obtained three-dimensional space image has no displacement within a preset time or the displacement is less than a preset value, the operator's hand position at that time may be determined as the initial position.

[0043] In some possible implementations, the joint point of the palm corresponding to the thumb in the initial position of the operator's hand can be used as the origin on the operator side, such as Figure 1B Point A in the image. Based on the determined origin and the three-dimensional space image, the coordinates of each feature point can be determined.

[0044] 204. Determine the position and posture information of the operator's hand according to the coordinates of the feature points.

[0045] In the embodiment of the present application, the operator's hand posture information includes: the orientation of the operator's palm, the finger angle at the distal fingertip joint of each finger, and the finger angle at the proximal fingertip joint of each finger. For example, the normal vector of the plane determined by the palm joints corresponding to any two fingers of the index finger, middle finger, ring finger and little finger can be used as the orientation of the operator's palm. For example, Figure 1B The normal vector of the triangle corresponding to the feature point A, feature point B, and feature point C is used as the orientation of the operator's palm. It is understandable that the normal vector of the triangle corresponding to the feature point A, feature point B, and feature point E can also be used as the orientation of the operator's palm. In order to obtain a more accurate palm posture, the orientation of the palm can include multiple, for example, the palm can be divided into multiple areas, each area corresponds to an orientation, and can be represented by the orientation of the plane corresponding to the feature point in the area. For example, the normal vector of the triangle corresponding to the feature point A, feature point a3, and feature point B can be used as the orientation of a part of the operator's palm area. The normal vector of the triangle corresponding to the feature point A, feature point B, and feature point C is used as the orientation of a part of the operator's palm area. The normal vector of the triangle corresponding to the feature point A, feature point C, and feature point D is used as the orientation of a part of the operator's palm area. And it is feasible to use the normal vector of the triangle corresponding to the feature point A, feature point D, and feature point E as the orientation of a part of the operator's palm area.

[0046] 205. Generate a control signal according to the position information of the operator's hand, and send the control signal to the robot.

[0047] In an embodiment of the present application, the characteristic points of the operator's hand have corresponding points on the robot's manipulator, and the control signal is used to drive the robot's manipulator arm to synchronize with the operator's hand operation. For example, when the operator's hand performs a moving operation, the control signal drives the robot's manipulator arm to perform a moving operation driven by the manipulator arm, and the direction and distance of movement are the same as the direction and distance of movement of the operator's hand. When the operator's hand performs a gripping operation, the control signal drives the robot's manipulator fingers to also perform a gripping operation. When the operator's hand performs an unfolding operation, the control signal drives the robot's manipulator fingers to also perform an unfolding operation. It should be noted that, compared to the simple linkage operation of the finger joints in the related art, the control signal of the present application can drive the angles of each joint corresponding to the characteristic points on the manipulator hand and the orientation of the manipulator palm to be consistent with the posture of the operator's hand. In this way, the control is more precise, and precise operations can be performed. For example, in Figure 1A In the scenario shown, when the operator wants the robot 104 to pick up the cup 105 next to it, the operator can control his hand 102 to do a grabbing operation, and the depth camera 103 transmits the captured real-time image to the computer 101. The computer 101 generates a control signal based on the acquired real-time three-dimensional space image to drive the robot 104's manipulator to move synchronously with the user's hand 102 until the manipulator grabs the cup. Figure 3 As shown, further, the operator's hand can move upward, and accordingly, the robot arm is controlled to pick up the water cup.

[0048] In some possible implementations, the control signal includes a first difference between the real-time position of the operator's hand and the initial position; and a second difference between the orientation of the palm corresponding to the real-time position of the operator's hand and the orientation of the palm corresponding to the initial position. It can be understood that by using the difference between the real-time position and the initial position of the operator's hand, and the difference in the orientation of the palm as the control signal, the hand dynamics can be accurately captured, accurate motion tracking can be achieved, and the accuracy of remote control can be improved.

[0049] As an improvement, in some possible implementations, the method for remotely controlling the robot may further include: Figure 1A The real-time three-dimensional spatial image of the operator's hand is displayed on the display screen of the computer 101. It is understandable that when the operator remotely controls the robot, by observing the displayed real-time three-dimensional spatial image, the operator can be provided with intuitive visual feedback, which is conducive to improving the operation accuracy, and is also conducive to assisting the operator to make correct decisions for subsequent operations and reduce misoperation.

[0050] In some possible real-time modes, the display screen of the computer 101 may overlay a line drawing of the operator's hand structure formed based on the identified feature points of the operator's hand when displaying the real-time three-dimensional space image. In this way, the hand structure can be clearly displayed, helping the operator to intuitively understand the mapping relationship between each feature point of the hand and the hand, thereby enhancing the intuitiveness of the operation.

[0051] In some possible implementations, such as Figure 1C As shown, the robot side may also include a shooting device 106, which is used to obtain a real-time image of the robot side. The real-time image of the robot side includes the robot's manipulator and the target object (such as Figure 1C The water cup 105 in the robot). The shooting device 106 transmits the acquired real-time image to the computer 101 by wired or wireless means, and displays it on the display of the computer 101. In some possible implementations, optional operation options for the target object can also be displayed on the display of the computer 101. For example, the optional operation options may include: moving, grabbing, measuring, pinching, hooking, pushing, hanging, hitting, pinching with two fingertips, pinching with three fingertips, or touching. The target operation is determined according to the selected operation instruction; in combination with the target operation and the positional relationship between the robot arm and the target object, an operation reminder mark can be displayed on the display screen; the operation reminder mark includes one or more of the following prompts: a reminder mark for indicating the direction of movement of the operator's hand, a reminder mark for indicating the operator's hand to perform finger grasping, a mark for indicating the operator's hand to perform finger extension, a mark for indicating the further bending or unfolding angle of a certain finger, etc. In this way, by displaying the robot side image and operation options in real time, the operation decision is simplified, and the intuitiveness and real-time performance are improved. The operation reminder mark can guide the operator's actions, reduce the difficulty of operation, and improve the operation efficiency.

[0052] like Figure 4AAs shown, the embodiment of the present application also provides a device 400 for remotely controlling a robot, comprising: a communication circuit 401, a memory 402, and one or more processors 403; the communication circuit 401, the memory 402, and the one or more processors 403 are coupled, the communication circuit 401 is used to obtain a real-time three-dimensional image of the operator's hand, and send a control signal to the robot; the memory 402 is used to store computer program codes, the computer program codes include computer instructions, and the one or more processors 403 call the computer instructions to enable the device to perform the following operations: identify the feature points of the operator's hand according to the three-dimensional space image, and the feature points It includes: the fingertips of each finger, the distal fingertip joints of each finger, the proximal fingertip joints of each finger, and the joints of the palm corresponding to each finger; determine the coordinates of the feature points according to the three-dimensional space image; determine the posture information of the operator's hand according to the coordinates of the feature points, the posture information of the operator's hand includes: the direction of the operator's palm, the finger angles at the distal fingertip joints of each finger, and the finger angles at the proximal fingertip joints of each finger; generate a control signal according to the posture information of the operator's hand, send the control signal to the robot through the communication circuit, and the control signal is used to drive the robot's mechanical arm to synchronize with the operator's hand operation.

[0053] In some embodiments, in determining the posture information of the operator's hand based on the coordinates of the feature points, the processor 403 is specifically used to use the normal vector of the plane determined by the palm joint point of the thumb and the palm joint points corresponding to any two of the other four fingers as the orientation of the operator's palm.

[0054] In some embodiments, the processor 403 is further used to obtain the initial position of the operator's hand; and determine the origin of the operator side and the initial posture of the operator's hand according to the initial position.

[0055] In some embodiments, in determining the origin of the operator side according to the initial position, the processor is specifically configured to use the joint point of the palm corresponding to the thumb in the initial position of the operator's hand as the origin of the operator side.

[0056] In some embodiments, the control signal includes a first difference between the real-time position of the operator's hand and the initial position; and a second difference between the orientation of the operator's palm corresponding to the real-time position of the operator's hand and the orientation of the operator's palm corresponding to the initial position.

[0057] In some possible embodiments, relative to Figure 4A For the device 400 for remotely controlling a robot, Figure 4BThe device 400 for remotely controlling a robot shown also includes a display 404 for displaying an image on the robot side in real time, wherein the image on the robot side includes a robotic arm and a target object operated by the robotic arm; the display 404 is also used to display optional operation options for the target object; after obtaining the selection operation instruction for the optional operation option, the processor 403 determines the target operation according to the selection operation instruction; and displays an operation reminder mark in combination with the target operation, the positional relationship between the robotic arm and the target object; the operation reminder mark includes one or more of the following prompts: a reminder mark for indicating the direction of movement of the operator's hand, a reminder mark for indicating the operator's hand to perform finger grasping, a mark for indicating the operator's hand to perform finger stretching, etc.

[0058] like Figure 5 As shown, an embodiment of the present application further provides a robot 500, comprising: a communication circuit 501, a control circuit 502, and a robotic arm 503, wherein the communication circuit 501 is used to receive a control signal generated by a device for remotely controlling the robot according to any of the above-mentioned method embodiments; the control circuit 502 is used to drive the robotic arm 503 to be synchronized with the operator's hand operation according to the control signal.

[0059] The present application also provides a remote control system. Figure 1A As shown, it includes: a device 101 for remotely controlling a robot, a first camera 103 and a robot 104; the device 101 for remotely controlling a robot is connected to the first camera 103 and the robot 104 in communication, and specifically, the relevant data can be transmitted by wire or wirelessly; the first camera 103 is used to obtain a three-dimensional spatial image of the operator's hand 102; the device 101 for remotely controlling a robot includes: a communication circuit, a memory and one or more processors; the communication circuit, the memory are coupled with one or more processors; the communication circuit is used to obtain a real-time three-dimensional image of the operator's hand, and send a control signal to the robot 103; the memory is used to store computer program codes, the computer program codes include computer instructions, and one or more processors call computer instructions to enable the electronic device to execute any of the above method embodiments. The robot 104 includes: a mechanical arm, a communication circuit and a control circuit; the communication circuit is used to receive a control signal generated by the device for remotely controlling a robot according to any of the above method embodiments; the control circuit is used to drive the mechanical arm to synchronize with the operator's hand operation according to the control signal.

[0060] like Figure 1CAs shown, in some possible embodiments, the remote control system further includes a second camera 106 for acquiring images on the robot side in real time, and the images on the robot side include the mechanical arm and the water cup 105 operated by the mechanical arm. The device 101 for remotely controlling the robot is also used to receive and display the images acquired by the second camera 106, and to display optional operation options for the water cup 105, determine the target operation according to the selected operation instruction, and display the operation reminder mark in combination with the target operation, the position relationship between the mechanical arm and the water cup 105; the operation reminder mark includes one or more of the following prompts: a reminder mark for indicating the direction of movement of the operator's hand, a reminder mark for indicating the operator's hand to perform finger grasping, a mark for indicating the operator's hand to perform finger extension, etc.

[0061] An embodiment of the present application also provides a chip system, which is applied to an electronic device. The chip system includes one or more processors, and the one or more processors are used to call computer instructions so that the electronic device executes any of the above method embodiments.

[0062] An embodiment of the present application further provides a computer-readable storage medium, which includes computer instructions. When the computer instructions are executed on an electronic device, the electronic device executes any of the above method embodiments.

[0063] An embodiment of the present application also provides a computer program product, which includes a computer program code. When the computer program code runs on an electronic device, the electronic device executes a method for remotely controlling a robot as described in the first aspect or any possible implementation of the first aspect.

[0064] It can be understood that the beneficial effects that can be achieved by the above-mentioned device for remotely controlling a robot, the robot, the remote control system, the chip system, the computer-readable storage medium and the computer program product can be referred to the beneficial effects described in the method embodiments and will not be repeated here.

[0065] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented by software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network or other programmable device. The computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium, for example, the computer instructions may be transmitted from a website site, computer, server or data center by wired (such as: coaxial cable, optical fiber, data subscriber line (Digital Subscriber Line, DSL)) or wireless (such as: infrared, wireless, microwave, etc.) mode to another website site, computer, server or data center. The computer-readable storage medium may be any available medium that a computer can access, or a data storage device such as a server or data center that includes one or more available media integrated. The available medium may be a magnetic medium (such as a floppy disk, a hard disk, a magnetic tape), an optical medium (such as a digital versatile disc (DVD)), or a semiconductor medium (such as a solid state disk (SSD)).

[0066] The above are optional embodiments provided for the present application and are not intended to limit the present application. Any modifications, equivalent substitutions, improvements, etc. made within the technical scope disclosed in the present application shall be included in the protection scope of the present application.

Claims

1. A method for remotely controlling a robot, characterized in that: The method comprises: Use a depth camera to obtain a real-time three-dimensional spatial image of the operator's hand; Identifying feature points of the operator's hand according to the three-dimensional space image, the feature points including: fingertips of each finger, distal fingertip joints of each finger, proximal fingertip joints of each finger, and palm joints corresponding to each finger; Determine the coordinates of the feature points according to the three-dimensional space image; Determine the posture information of the operator's hand according to the coordinates of the feature points, the posture information of the operator's hand including: the orientation of the operator's palm, the finger angle at the distal fingertip joint point of each finger, and the finger angle at the proximal fingertip joint point of each finger; the orientation of the operator's palm includes multiple orientations, the operator's palm is divided into multiple areas, each area corresponds to a orientation, and is represented by the orientation of the plane corresponding to the feature points in each area; Generate a control signal according to the posture information of the operator's hand, and send the control signal to the robot, wherein the control signal is used to drive the robot's mechanical arm to synchronize with the operator's hand operation; Displaying a real-time three-dimensional spatial image of the operator's hand, and superimposing a line drawing of the operator's hand structure formed according to the identified feature points of the operator's hand on the displayed real-time three-dimensional spatial image; when acquiring a three-dimensional spatial image through the depth camera, when the acquired three-dimensional spatial image has no displacement within a preset time or the displacement is less than a preset value, determining the operator's hand position at that time as the initial position; determining the initial posture of the operator's hand according to the initial position; and taking the joint point of the palm corresponding to the thumb in the initial position of the operator's hand as the origin on the operator side; Displaying an image of the robot side in real time, wherein the image of the robot side includes a robot arm and a target object operated by the robot arm; and displaying optional operation options for the target object; Determine the target operation according to the selected operation instruction; In combination with the target operation and the positional relationship between the robotic arm and the target object, an operation reminder mark is displayed; the operation reminder mark includes one or more of the following prompts: a reminder mark for indicating the direction of movement of the operator's hand, a reminder mark for instructing the operator's hand to perform finger grasping, and a mark for instructing the operator's hand to perform finger extension.

2. The method according to claim 1, characterized in that The control signal includes: a first difference between the real-time position of the operator's hand and the initial position, and a second difference between the orientation of the palm corresponding to the real-time position of the operator's hand and the orientation of the palm corresponding to the initial position.

3. A device for remotely controlling a robot, characterized in that: include: A communication circuit, a memory and one or more processors; the communication circuit and the memory are coupled to the one or more processors, the communication circuit is used to obtain a real-time three-dimensional image of the operator's hand and send a control signal to the robot; the memory is used to store a computer program code, the computer program code includes computer instructions, and the one or more processors call the computer instructions to enable the device to perform the method as described in any one of claims 1 to 2.

4. A robot, characterized in that: include: A robotic arm, a communication circuit, and a control circuit, wherein: The communication circuit is used to receive a control signal generated by a device for remotely controlling the robot according to any one of the methods of claims 1 to 2; The control circuit is used to drive the robot arm to synchronize with the operator's hand operation according to the control signal.

5. A remote control system, characterized in that: include: A first camera, a robot, and a device for remotely controlling the robot; The device for remotely controlling the robot is in communication connection with the first camera and the robot; wherein, The first camera is used to obtain a three-dimensional image of the operator's hand; The device for remotely controlling a robot comprises: a communication circuit, a memory and one or more processors; the communication circuit, the memory and the one or more processors are coupled; the communication circuit is used to obtain a real-time three-dimensional image of an operator's hand and send a control signal to the robot; the memory is used to store a computer program code, the computer program code comprises a computer instruction, and the one or more processors call the computer instruction to make the device for remotely controlling a robot perform the method as described in any one of claims 1 to 2; The robot comprises: a mechanical arm, a communication circuit and a control circuit; the communication circuit is used to receive a control signal generated by a device for remotely controlling the robot; the control circuit is used to drive the mechanical arm to synchronize with the operator's hand operation according to the control signal; The remote control system also includes a second camera, The second camera is used to acquire an image of the robot side in real time, wherein the image of the robot side includes the robotic arm and a target object operated by the robotic arm; The device for remotely controlling the robot is also used to receive and display the image acquired by the second camera, display the robot's optional operation options for the target object; determine the target operation according to the selected operation instruction; and display an operation reminder mark in combination with the target operation and the positional relationship between the robotic arm and the target object; the operation reminder mark includes one or more of the following prompts: a reminder mark for indicating the direction of movement of the operator's hand, a reminder mark for instructing the operator's hand to perform finger grasping, and a mark for instructing the operator's hand to perform finger stretching.

Images