Vision-based multi-mode teleoperation grasping method and system for humanoid robotic arm

Abstract

The present application discloses a vision-based multi-mode teleoperation grasping method and system for a humanoid robot arm, the method comprising: determining the dexterous hand control data and the robot arm control data according to each frame of the operator's hand motion image; in the full teleoperation control mode, mapping the robot arm control data to the robot arm as a whole, and controlling the robot arm as a whole to perform a second action; the second action is determined according to the operator's palm posture; in the semi-automatic operation control mode, according to the first coordinate of the object to be grasped relative to the robot arm base, controlling the robot arm as a whole to perform a third action; mapping the robot arm control data to the end of the robot arm, and controlling the end of the robot arm to perform a fourth action; completing the teleoperation grasping through the coordination of the first action and the second action, or through the coordination of the first action, the third action and the fourth action. The overall steps of the embodiments of the present application improve the teleoperation grasping efficiency and performance of the humanoid robot arm, and can be widely used in the field of computer technology.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a vision-based multi-mode teleoperation grasping method and system for a humanoid robotic arm. Background Art

[0002] Teleoperation is a technology for remotely controlling robots. It allows operators to control the movements and behaviors of robots through communication devices far away from the actual working place of the robots. Robot teleoperation can be used to achieve object grasping and collect robot-related data. At present, most robot arm operating systems use wearable devices (such as data gloves, motion capture clothing, virtual reality (VR) helmets, etc.) to capture the operator's movements and physiological signals. Although this method improves the intuitiveness of operation, the wearable device itself has a certain size. In addition to being difficult to carry, it may also restrict the operator's movements to a certain extent. The current vision-based teleoperation method usually uses a full-body posture estimation algorithm to extract the bone point coordinates of the palm and arm, and further controls the robot arm through a bone redirection algorithm. The relatively complex algorithm will introduce a large amount of calculation and it is difficult to meet the real-time requirements. Summary of the invention

[0003] In order to solve at least one of the above problems, the embodiments of the present application propose a vision-based humanoid robotic arm multi-mode teleoperation grasping method and system with high efficiency and good performance.

[0004] To achieve the above-mentioned purpose, one aspect of an embodiment of the present application proposes a multi-mode teleoperation grasping method of a humanoid robot arm based on vision, the method comprising:

[0005] Determine dexterous hand control data and robotic arm control data according to each frame of the operator's hand motion image; wherein the dexterous hand control data is used to control the dexterous hand to perform a first motion; the first motion is determined according to the operator's finger angle;

[0006] In the full teleoperation control mode, the robot arm control data is mapped to the entire robot arm, and the entire robot arm is controlled to perform a second action; the second action is determined according to the palm posture of the operator;

[0007] In the semi-automatic operation control mode, according to the first coordinate of the object to be grasped relative to the base of the robot arm, the robot arm is controlled to perform a third action as a whole; the robot arm control data is mapped to the end of the robot arm, and the end of the robot arm is controlled to perform a fourth action; wherein the third action is to automatically move to a preset distance range around the object to be grasped; wherein the fourth action is determined according to the palm posture of the operator;

[0008] The remote control grasping is completed by the cooperation of the first action and the second action, or by the cooperation of the first action, the third action and the fourth action.

[0009] In some embodiments, determining the dexterous hand control data and the robotic arm control data according to each frame of the operator's hand motion image includes the following steps:

[0010] Obtain the operator's hand motion image for each frame;

[0011] Determining key three-dimensional coordinates of key points of the hand according to the hand motion image;

[0012] Determine finger angles and palm positions according to the key three-dimensional coordinates;

[0013] Quantifying the finger angles to obtain six-dimensional dexterous hand control data;

[0014] The palm posture is subjected to Euler angle conversion processing to obtain six-dimensional robotic arm control data.

[0015] In some embodiments, determining the finger angle and palm position according to the key three-dimensional coordinates comprises the following steps:

[0016] Determine the finger joint angle according to the key three-dimensional coordinates;

[0017] Determine the degree of finger bending and the degree of lateral opening and closing of the thumb according to the joint angle to obtain the finger angle;

[0018] Determine the finger joint position according to the key three-dimensional coordinates;

[0019] The palm posture is calculated according to the joint positions.

[0020] In some embodiments, mapping the robot arm control data to the entire robot arm to control the entire robot arm to perform a second action includes the following steps:

[0021] Determine the joint angle of the robot arm according to the target posture data in the robot arm control data;

[0022] Calculating the movement speed and movement acceleration according to the relationship between the change of the joint angle of the robot arm and time;

[0023] Obtaining a motion trajectory optimization result according to the motion speed, the motion acceleration and the six-degree-of-freedom data in the robot arm control data;

[0024] According to the transparent transmission time and the optimization result of the motion trajectory, the robot arm is controlled to perform a second action as a whole.

[0025] In some embodiments, the controlling the robotic arm as a whole to perform the third action according to the first coordinate of the object to be grasped relative to the base of the robotic arm comprises the following steps:

[0026] Obtaining the object label of the object to be grasped determined by the operator;

[0027] According to the object label, obtaining the object depth information of the object to be grasped through the depth camera at the end of the robotic arm;

[0028] Calculating a first coordinate of the object to be grasped relative to a base of the robotic arm according to the object depth information and a current state of the robotic arm;

[0029] According to the first coordinate, the robotic arm is controlled to automatically move to a preset distance range around the object to be grasped to complete a third action.

[0030] In some embodiments, before the step of obtaining the object label of the object to be grasped determined by the operator, the following steps are also included:

[0031] Perform open scene target detection based on the depth camera at the end of the robotic arm to obtain the target object;

[0032] The robot arm joint angle state and object label corresponding to the target object are stored in a database.

[0033] In some embodiments, the open scene target detection based on the depth camera at the end of the robotic arm to obtain the target object includes the following steps:

[0034] The end of the robotic arm equipped with a depth camera is moved around the robotic arm to take pictures of the scene;

[0035] The yolo-world target detection model is used to perform target detection on the scene photo to obtain the target object.

[0036] To achieve the above purpose, another aspect of the embodiment of the present application proposes a vision-based multi-mode teleoperated grasping system for a humanoid robotic arm, the system comprising:

[0037] The first module is used to determine the dexterous hand control data and the robotic arm control data according to each frame of the operator's hand action image; wherein the dexterous hand control data is used to control the dexterous hand to perform a first action; and the first action is determined according to the operator's finger angle;

[0038] The second module is used to map the control data of the robot arm to the entire robot arm in the full teleoperation control mode, and control the entire robot arm to perform a second action; the second action is determined according to the palm posture of the operator;

[0039] The third module is used to control the robot arm as a whole to perform a third action according to the first coordinate of the object to be grasped relative to the robot arm base in a semi-automatic operation control mode; map the robot arm control data to the end of the robot arm, and control the end of the robot arm to perform a fourth action; wherein the third action is to automatically move to a preset distance range around the object to be grasped; wherein the fourth action is determined according to the palm posture of the operator;

[0040] The fourth module is used to complete remote control grasping through the coordination of the first action and the second action, or through the coordination of the first action, the third action and the fourth action.

[0041] It should be noted that, in some embodiments, the system may also include the following modules:

[0042] The fifth module performs open scene target detection based on the depth camera at the end of the robotic arm to obtain the target object; and stores the robotic arm joint angle state and object label corresponding to the target object in the database.

[0043] To achieve the above objective, another aspect of an embodiment of the present application provides an electronic device, the electronic device comprising a memory and a processor, the memory storing a computer program, and the processor implementing the above-mentioned method when executing the computer program.

[0044] To achieve the above objective, another aspect of an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method described above is implemented.

[0045] The embodiments of the present application include at least the following beneficial effects: The present application provides a vision-based humanoid robotic arm multi-mode remote-operated grasping method and system, which determines the dexterous hand control data and the robotic arm control data according to each frame of the operator's hand movement image, and is further used to control the dexterous hand and the robotic arm to perform related actions in different operation control modes. Through the coordination of the first action and the second action, or through the coordination of the first action, the third action and the fourth action, remote-operated grasping can be performed through different control modes according to actual task requirements, thereby improving the efficiency of remote-operated grasping. The data required to control the robotic arm and the dexterous hand can be obtained only through hand movements, which reduces the amount of calculation and improves the performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] The accompanying drawings are used to provide further understanding of the technical solution of the present application and constitute a part of the specification. Together with the embodiments of the present application, they are used to explain the technical solution of the present application and do not constitute a limitation on the technical solution of the present application.

[0047] Figure 1 It is a step diagram of a multi-mode teleoperation grasping method of a humanoid robot arm based on vision provided in an embodiment of the present application;

[0048] Figure 2 It is a robotic arm teleoperation framework provided in an embodiment of the present application;

[0049] Figure 3 is a flow chart of a multi-mode teleoperation grasping method of a humanoid robot arm based on vision provided in an embodiment of the present application;

[0050] Figure 4 is a schematic diagram of visual remote control grabbing provided in an embodiment of the present application;

[0051] Figure 5 21 hand joint example diagrams provided in the embodiments of the present application;

[0052] Figure 6 is a schematic diagram of a palm plane provided in an embodiment of the present application;

[0053] Figure 7 is a schematic diagram of palm posture provided in an embodiment of the present application;

[0054] Figure 8 It is a target detection result diagram of yolo-world provided in an embodiment of the present application;

[0055] Fig. 9 is a schematic diagram of the grasping process of the semi-automatic control mode provided in an embodiment of the present application;

[0056] Fig.10 is a schematic diagram of calculating the position of an object according to depth provided in an embodiment of the present application;

[0057] Fig.11 It is a module schematic diagram of a vision-based multi-mode teleoperated grasping system of a humanoid robotic arm provided in an embodiment of the present application;

[0058] Fig.12 It is a schematic diagram of the hardware structure of the electronic device provided in the embodiment of the present application. DETAILED DESCRIPTION

[0059] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application is further described in detail below in conjunction with the accompanying drawings and examples. It should be understood that the specific embodiments described herein are only used to explain the present application and are not intended to limit the present application. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the embodiments of the present application. They are only examples of devices and methods consistent with some aspects of the embodiments of the present application as detailed in the attached claims.

[0060] Although the functional modules are divided in the system schematic diagram and the logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the system or the order in the flowchart. The terms "first / S100", "second / S200", etc. in the specification, claims and the above drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence.

[0061] It is understood that the terms "first", "second", etc. used in this application can be used to describe various concepts in this article, but unless otherwise specified, these concepts are not limited by these terms. These terms are only used to distinguish one concept from another concept. For example, without departing from the scope of the embodiment of the present application, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Depending on the context, the words "if" and "if" as used herein can be interpreted as "at the time of" or "when" or "in response to determination".

[0062] The terms "at least one", "multiple", "each", "any", etc. used in this application, at least one includes one, two or more, multiple includes two or more, each refers to each of the corresponding multiple, and any refers to any one of the multiple.

[0063] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0064] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0065] It should be noted that, for the sake of convenience, formulas with similar functions in this article may be represented by the same formula mark. For example, "Formula (2)" may represent a formula combination consisting of several formulas.

[0066] Before describing the embodiments of the present application in detail, some nouns and terms involved in the embodiments of the present application are first described. The nouns and terms involved in the embodiments of the present application are subject to the following explanations.

[0067] Teleoperation is a technology for remotely controlling robots, which allows the operator to control the robot's movements and behaviors through communication equipment at a distance from the robot's actual working location.

[0068] In the related technology, common robotic arm grasping usually requires a fixed-angle overhead camera to estimate the 2D posture of the target, and then uses a robotic arm with a gripper to automatically grasp it. However, this grasping has low flexibility and cannot complete grasping for some objects with special shapes and materials. Although the existing remote operation methods based on wearable devices have good real-time performance, most of the equipment is expensive. Advanced wearable devices such as data gloves or full-body suits with sensors are usually expensive and have limited coverage. At the same time, wearable devices may affect the user's comfort and freedom, especially when used for a long time.

[0069] In comparison, vision-based teleoperation methods are low-cost, easy to deploy, and more flexible and natural to operate. However, current vision-based teleoperation methods usually use a full-body posture estimation algorithm to extract the bone point coordinates of the palm and arm, and further control the robotic arm through a bone redirection algorithm. The relatively complex algorithm introduces a lot of calculations and is difficult to meet real-time requirements.

[0070] In addition, visual teleoperation requires a certain level of operator proficiency. It usually takes a period of training to achieve relatively smooth operation. An obvious way to simplify the operation is to let the robot arm automatically approach the object to be grasped, and then the operator will perform fine control. However, the shortcomings of existing technologies stem from their inherent limitations. Traditional target detection algorithms rely on a predefined set of fixed categories for training, which leads to the lack of flexibility of the system and the inability to identify targets of unknown categories. When faced with new target categories, a lot of resources are needed to re-collect data and train models, which is not only costly, but also seriously affects the system's ability to adapt in complex and dynamic environments.

[0071] In view of this, the embodiments of the present application provide a vision-based humanoid robotic arm multi-mode remote control grasping method and system. Based on visual remote control, this scheme simplifies and improves the control scheme of the robotic arm and the robotic hand, combined with the lightweight posture estimation algorithm Mediapipe Hands, to solve the problems of large computational complexity and poor real-time performance of robot visual remote control. At the same time, a semi-automatic control method is proposed, combined with the open scene target detection algorithm yolo-world, to solve the limitations of traditional target detection algorithms in the semi-automatic process of remote control systems.

[0072] The vision-based multi-mode teleoperation grasping method of a humanoid robot arm provided in an embodiment of the present application relates to the field of computer technology. The vision-based multi-mode teleoperation grasping method of a humanoid robot arm provided in an embodiment of the present application can be applied to a terminal, can also be applied to a server, and can also be software running in a terminal or a server. In some embodiments, the terminal can be a smart phone, a tablet computer, a laptop computer, a desktop computer, a smart speaker, a smart watch, and a car terminal, etc., but is not limited to this; the server side can be configured as an independent physical server, or a server cluster or distributed system composed of multiple physical servers, and can also be configured as a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms. The server can also be a node server in a blockchain network; the software can be an application that implements a vision-based multi-mode teleoperation grasping method for a humanoid robot arm, etc., but is not limited to the above forms.

[0073] The present application can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, etc. The present application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present application can also be practiced in distributed computing environments, in which tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media including storage devices.

[0074] It should be noted that in each specific implementation of the present application, when it comes to the need to perform relevant processing based on data related to user identity or characteristics such as user information, user behavior data, user historical data, and user location information, the user's permission or consent will be obtained first, and the collection, use, and processing of these data will comply with relevant laws, regulations, and standards. In addition, when the embodiment of the present application needs to obtain the user's sensitive personal information, the user's separate permission or consent will be obtained through a pop-up window or by jumping to a confirmation page. After clearly obtaining the user's separate permission or consent, the necessary user-related data for the normal operation of the embodiment of the present application will be obtained.

[0075] Figure 1 is an optional flow chart of a multi-mode teleoperation grasping method of a humanoid robot arm based on vision provided in an embodiment of the present application. Figure 1 The method may include but is not limited to steps S100 to S400.

[0076] Step S100, determining dexterous hand control data and robotic arm control data according to each frame of the operator's hand motion image; wherein the dexterous hand control data is used to control the dexterous hand to perform a first action; and the first action is determined according to the operator's finger angle.

[0077] Step S200, in the full teleoperation control mode, mapping the robot arm control data to the entire robot arm, and controlling the entire robot arm to perform a second action; the second action is determined according to the palm posture of the operator.

[0078] Step S300, in the semi-automatic operation control mode, according to the first coordinate of the object to be grasped relative to the base of the robot arm, the robot arm as a whole is controlled to perform a third action; the robot arm control data is mapped to the end of the robot arm, and the end of the robot arm is controlled to perform a fourth action; wherein, the third action is to automatically move to a preset distance range around the object to be grasped; wherein, the fourth action is determined according to the palm posture of the operator.

[0079] Step S400: completing remote control grabbing through the coordination of the first action and the second action, or through the coordination of the first action, the third action and the fourth action.

[0080] Steps S100 to S400 shown in the embodiment of the present application determine the dexterous hand control data and the robotic arm control data according to the operator's hand motion image, and then control the dexterous hand and the robotic arm to perform related actions in different operation control modes, thereby realizing the function of visual teleoperation of the robotic arm to grasp. In addition, multi-mode teleoperation control is realized, and teleoperation grasping can be performed through different control modes according to actual task requirements, thereby improving the efficiency of teleoperation grasping.

[0081] In some embodiments, step S100 may include but is not limited to steps S110 to S150:

[0082] Step S110, obtaining each frame of the operator's hand movement image.

[0083] Step S120, determining key three-dimensional coordinates of key points of the hand according to the hand motion image.

[0084] Step S130, determining the finger angle and palm posture according to the key three-dimensional coordinates.

[0085] Step S140, quantizing the finger angles to obtain six-dimensional dexterous hand control data.

[0086] Step S150, performing Euler angle conversion processing on the palm posture to obtain six-dimensional robot arm control data.

[0087] In step S130 of some embodiments, determining the finger angle and palm posture according to the key three-dimensional coordinates includes but is not limited to the following steps S131 to S134:

[0088] Step S131, determining the finger joint angle according to the key three-dimensional coordinates.

[0089] Step S132, determining the degree of finger bending and the degree of lateral opening and closing of the thumb according to the joint angle to obtain the finger angle.

[0090] Step S133, determining the finger joint position according to the key three-dimensional coordinates.

[0091] Step S134, calculating the palm posture according to the joint positions.

[0092] In some embodiments, step S200 includes but is not limited to the following steps S210 to S240:

[0093] Step S210, determining the joint angle of the robot arm according to the target posture data in the robot arm control data.

[0094] Step S220, calculating the motion speed and motion acceleration according to the relationship between the change of the joint angle of the robot arm and time.

[0095] Step S230, obtaining a motion trajectory optimization result according to the motion speed, the motion acceleration and the six-degree-of-freedom data in the robot arm control data.

[0096] Step S240, controlling the entire robotic arm to perform a second action according to the transparent transmission time and the action trajectory optimization result.

[0097] In step S300 of some embodiments, the step of controlling the robot arm as a whole to perform the third action according to the first coordinate of the object to be grasped relative to the robot arm base includes but is not limited to the following steps S310 to S340:

[0098] Step S310: obtaining the object label of the object to be grasped determined by the operator.

[0099] Step S320: acquiring object depth information of the object to be grasped through a depth camera at the end of the robotic arm according to the object label.

[0100] Step S330: Calculate the first coordinate of the object to be grasped relative to the base of the robotic arm according to the object depth information and the current state of the robotic arm.

[0101] Step S340: According to the first coordinate, the robot arm is controlled to automatically move to a preset distance range around the object to be grasped, thereby completing a third action.

[0102] Before step S310 in some embodiments, the following target detection steps S410 to S420 may also be included:

[0103] Step S410: performing open scene target detection based on the depth camera at the end of the robotic arm to obtain the target object.

[0104] Step S420: storing the robot arm joint angle state and object label corresponding to the target object in a database.

[0105] In some embodiments, step S410 includes but is not limited to the following steps S411 to S412:

[0106] Step S411, the end of the robotic arm equipped with the depth camera is moved around the robotic arm to take a picture of the scene.

[0107] Step S412, target detection is performed on the scene photo through the yolo-world target detection model to obtain the target object. Next, in combination with a specific application example of remote operation grabbing scene, the scheme of the embodiment of the present application is described in detail and explained:

[0108] In the embodiment of the present application, a multi-mode teleoperation grasping method for a humanoid robot arm based on vision is provided. The method can be applied to grasping an object by teleoperating a robot arm. The dual-stream visual robot arm teleoperation based on the dexterous hand control stream and the robot arm control stream detects the operator's hand motion image through a camera, and uses finger data and palm data to control the dexterous hand and the robot arm respectively. The specific teleoperation framework is as follows Figure 2 As shown in the implementation flow chart Figure 3 shown.

[0109] Furthermore, the operator faces the depth camera, which records the operator's hand movement images in each frame. The computer obtains the 3D coordinates of the key points of the hand through the input image, and divides the 3D coordinates of the palm and fingers into two control flows of the dexterous hand and the robotic arm for processing. In the dexterous hand control flow, the quantized data of a total of 6 dimensions is obtained by calculating the finger angle and other information for controlling the dexterous hand. In the robotic arm control flow, the hand posture is calculated to obtain different 6-dimensional data for controlling the robotic arm. Finally, the robotic arm is controlled through the communication module to achieve remote control grasping.

[0110] Further, refer to Figure 4 The operator faces the camera for remote control. When the operator makes a movement, the camera obtains the operator's hand posture information and then controls the robotic arm to make the same movement in real time, thereby grabbing objects within the moving range of the robotic arm.

[0111] In some embodiments, the main steps of remote operation in the embodiment of the present application can be performed as follows 1 to 6:

[0112] Step 1: Connect the operating end to the robot arm. For full remote control, proceed directly to step 5. For semi-automatic control, start from step 2.

[0113] Step 2: The robotic arm is adjusted to a suitable angle, and the end moves around the robotic arm. The Realsense depth camera at the end of the robotic arm is used to detect open scene targets. All detected objects are stored in the local txt file database in the form of [robotic arm joint angle status, label]. After the detection is completed, the robotic arm is reset.

[0114] Step 3: The operator specifies the label of the object to be grasped. The robot arm returns to the state when the specified object is detected according to the angle status information in the txt file, and aims the camera at the object to obtain the depth information of the object.

[0115] Step 4: Calculate the coordinates of the object relative to the base of the robotic arm through the object depth information and the current state of the robotic arm. According to the object coordinates, the robotic arm automatically moves to the vicinity of the object along a certain trajectory and in a fixed posture.

[0116] Step 5: The operator performs remote control grasping. For full remote control, the operator's palm position (position and posture) will be directly mapped to the robotic arm, which can achieve more flexible grasping. For semi-automatic grasping, the operator's palm position corresponds to the end position of the robotic arm, which can achieve faster and more accurate grasping in a smaller range.

[0117] Step 6. For the full remote control mode, the operator can end the grasping at any time and the robot arm will reset. For the semi-automatic control mode, after the operator ends the current grasping action, he / she will confirm whether to continue grasping. If he / she continues to grasp, the process returns to step 3. Otherwise, the robot arm will reset and the grasping will end.

[0118] Furthermore, the embodiment of the present application detects key points of the hand and processes posture data to facilitate subsequent calculation of finger angles and palm posture.

[0119] Specifically, first, the operator's frontal video image is acquired through a Realsense 435i depth camera, and then Mediapipe Hands is used to estimate the hand posture.

[0120] Mediapipe Hands is a relatively efficient real-time two-stage hand posture tracking method. By inputting an RGB image, it can obtain the 2.5D coordinates of the human hand skeleton points. Mediapipe Hands consists of two modules, namely the palm detection module and the hand key point detection module. The palm detection module inputs an RGB image and outputs the bounding box of the palm as the input of the hand key point detection module. It then returns the 2.5D coordinates of the 21 joints of the hand. Indicates, where t represents the t-th frame image, n∈0,1,K,20, represents the joint point number, and the corresponding joint is as follows Figure 5 It should be noted that the hand joints are sorted according to the output order of Mediapipe. You can change other expressions. When sorting, you need to ensure that each point corresponds to the corresponding joint data.

[0121] Since the depth coordinates of the 2.5D coordinates of the hand skeleton points obtained by Mediapipe Hands are obtained by estimation and have low accuracy, the depth information obtained by the depth camera can be used to correct the corresponding joint point positions. The depth-corrected skeleton coordinates can be expressed as the following formula (1):

[0122]

[0123] in, Represents the corrected bone coordinates; W and H represent the pixel width and height of the image, which can be 640 and 480 by default and can be adjusted according to actual conditions. In the image The depth of the corresponding pixel coordinates is used in the subsequent expressions to simplify the expression. express.

[0124] Since the palm can be regarded as a rigid body, joints 0, 5, 9, 13, and 17 can be regarded as being on the same plane. Therefore, the hand pose is calculated using joints 0, 5, 9, and 17. First, the normal vector of the palm plane is calculated. and the direction vector of the palm like Figure 6 and Figure 7 shown.

[0125] It can be calculated by vector cross product, It can be directly represented by the vector from joint 0 to joint 9, expressed as follows:

[0126]

[0127] in, Represents a vector cross product operation.

[0128] use The arc cosine can also be used to calculate the angle between the first and second joints of each finger, and then quantify the degree of bending of each finger and the degree of lateral opening and closing of the thumb, as shown in the following formula (3):

[0129]

[0130] Where arccos(g) is the inverse cosine function, k = 1, 2, K, 5, They represent the five fingers from the little finger to the thumb. represents the ath joint coordinate of the corresponding finger starting from the root, for example, Respectively equivalence.

[0131] The lateral opening and closing degree of the thumb is calculated by the angle between the vertical line between the No. 2 joint and the line connecting the No. 0-5 joints and the palm plane, as shown in the following formula (4):

[0132]

[0133] in, is the perpendicular vector from joint 2 to the line connecting joints 0-5. So far, the 6-DOF data for controlling the dexterous hand has been obtained.

[0134] Before controlling the robot arm, the target position of the end of the robot arm needs to be obtained. In this embodiment, only the depth-corrected coordinates of the palm joint 0 are used to calculate the end position of the robot arm, thereby simplifying the data. The pixel position of the palm in the image is used to obtain the y-axis coordinate y of the end of the robot arm. r and the z-axis coordinate z r , use the depth information to obtain the x-axis coordinate x of the end of the robot arm r In order to adapt to the size of the robot arm RM-65 used, the specific coordinate transformation is shown in the following formula (5):

[0135]

[0136] Among them, δ is the depth correction coefficient, which is used to adjust the operator's appropriate distance from the camera during teleoperation. γ and λ are the distance scale change coefficients of the y-axis and z-axis, respectively, which are used to adjust the scale of the operator's palm movement distance in the image to the robot arm movement distance. The values ​​of the three can be adjusted to adapt to the operating habits of different operators.

[0137] Then, based on the normal vector obtained by formula (2), we can and Solve the rotation matrix between the robot end coordinate system and the current palm posture coordinate system. The specific process is shown in the following formula (6):

[0138]

[0139] Among them, T a ,T b ,T c V 1 ,V 2 The unit vectors of the three axes of the coordinate system formed by the cross product of two vectors, and the rest of the symbols are the same. x0 and V z0 They represent the direction vectors of the x-axis and y-axis of the zero-position state of the end of the robot (i.e., the robot is fully extended and the robot joint angle is zero). Then, the Euler angle can be obtained from the rotation matrix, as shown in the following formula (7):

[0140]

[0141] in, They represent the Euler angles of the x, y, and z axes of the robot base at time t, respectively. arctan2(x,y) is used to calculate the arc from the point (x,y) to the origin. At this point, the 6-DOF data used to control the robot arm is calculated.

[0142] However, since the hand skeletal coordinates detected by Mediapipe Hands are jittery, if not processed, jitter will also occur during actual robot operation. Therefore, in the embodiment of the present application, two threshold processing methods are used for the robotic arm and the dexterous hand to avoid jitter. After each hand skeletal coordinate is detected and the dexterous hand control data is calculated, the current value is recorded and the difference is calculated with the last recorded value, as shown in the following formula (8):

[0143]

[0144] in, and are the state changes of the dexterous hand and the robotic arm respectively, and then the angle threshold processing function Threshold is designed h (g) and weighted threshold processing function Threshold a (g) If the following formula (9) holds true, it is considered that the data change is not caused by jitter and is a normal movement, and the data is sent to the robot control module:

[0145]

[0146] Among them, i represents the subscript of the corresponding element of the vector, γ h ,γ a are the corresponding thresholds, which can be appropriately lowered according to the operation sensitivity requirements (it is recommended to be no less than 0.02).

[0147] Furthermore, in the embodiment of the present application, the remote control of the robot arm based on the hand posture can be controlled by transparent transmission. Transparent transmission is to send the posture information directly to the robot arm without the internal algorithm planning of the robot arm. The robot arm directly executes. The shorter the transparent transmission period, the smoother the movement of the robot arm. Transparent transmission has high requirements for each point, so trajectory planning is required. However, it is more troublesome to plan the posture directly, so it is necessary to first inversely calculate the posture to the angle, then plan the angle, and then use the angle transparent transmission to operate the robot arm.

[0148] First, given the moving target pose p end , combined with the current pose p start By performing the inverse solution, we can get the corresponding joint angles q of each robot arm at the target point. 1Then, the trajectory is optimized by using fifth-order polynomial interpolation. The relationship between the angle of each joint and time represented by the fifth-order interpolation polynomial q(t) is shown in the following formula:

[0149] q(t)=k 0 +k 1 (tt 0 )+k 2 (tt 0 ) 2 +k 3 (tt 0 ) 3 +k 4 (tt 0 ) 4 +k 5 (tt 0 ) 5 (10)

[0150] Where t represents time, t 0 That is the starting time. Taking the first-order derivative can get the corresponding speed v(t):

[0151] v(t)=q′(t)=k 1 +2k 2 (tt 0 )+3k 3 (tt 0 ) 2 +4k 4 (tt 0 ) 3 +5k 5 (tt 0 ) 4 (11)

[0152] Similarly, taking the second-order derivative can get the corresponding acceleration a(t):

[0153] a(t)=q″(t)=2k 2 +6k 3 (tt 0 )+12k 4 (tt 0 ) 2 +20k 5 (tt 0 ) 3 (12)

[0154] Based on equations (4), (5), and (6), since the velocity and acceleration are continuous, plus the starting angle q 0 , setting the initial velocity, acceleration, stop velocity, and acceleration to zero, we can obtain the constraint solution shown in (13):

[0155]

[0156] Among them, t 1 is the end time, and its specific value is determined by the maximum absolute value of the angle difference of each joint of the robot. Substituting the above constraint solution back into equations (10), (11), and (12), the final trajectory optimization result can be obtained. After obtaining the motion trajectory, by setting the transparent time step dt, the robot can move to the corresponding position of the operator's hand. The specific value of dt is determined by the maximum joint deflection angle of the robot.

[0157] Furthermore, the embodiment of the present application performs open scene target positioning and capture based on yolo-world. Yolo-world is a new open vocabulary target detection method that combines the yolo detector and the visual-language model (VLM). Its overall architecture includes three main modules: the YOLO detector, the text encoder, and the reparameterizable visual-language path aggregation network (RepVL-PAN). The text encoder encodes the input text into a text embedding. The image encoder (YOLO backbone network) is used to extract multi-scale image features. RepVL-PAN realizes cross-modal fusion between image features and text embeddings. In general, YOLO-World extends the traditional YOLO detector to the field of open vocabulary detection by introducing visual-language modeling and large-scale pre-training, while maintaining a high reasoning efficiency. This design gives it strong flexibility and generalization capabilities in practical applications. The specific recognition effects are as follows Figure 8 shown.

[0158] At the end of the robotic arm, there is a Realsense depth camera of the same model to obtain the depth of the object.

[0159] The specific process of semi-automatic mode is as follows Fig. 9 When the semi-automatic mode is enabled, the robot arm will rotate around once, and the image captured by the camera at the end of the robot arm will be detected by yolo-world, and the label of the detected object and the angle state of each joint of the robot arm at this time will be recorded. After the rotation is completed, all the recorded information will be in the form of {[φ 1 ,φ 2 ,φ 3 ,φ 4 ,φ 5 ,φ 6], label} in the local txt database. The operator then selects the label of the object to be grasped, and the robot arm immediately returns to the angle at which it was recorded, aligns the center of the camera with the center of the object detection frame (Bouding Box), and then the depth camera returns the depth information of the object to calculate the relative position of the target.

[0160] like Fig.10 As shown in the figure, the small ball represents the object, and the black line segment represents the robot arm link. Respectively represent the rotation angles of the three rotation joints of the robot arm, l is the length of the visual connector, and the distance between the ball and the robot arm can be obtained by the following formula (14):

[0161] d o =d arm +d ep sinφ-l cosφ (14)

[0162] in, is the distance between the end of the robot arm and the base, d ep That is, the depth distance obtained by the camera, φ is the overall angle of the end of the robotic arm, At this time, the rotation angle of the robot arm base is ω 0 , then the x of the object o ,y o The coordinates are:

[0163]

[0164] Similarly, the z of the object can be calculated o The coordinates are:

[0165] z o =z c -d ep cosφ-l sinφ (16)

[0166] So far, the specific coordinates of the object relative to the base of the robot arm (x o ,y o ,z o ).

[0167] After obtaining the object coordinates, the robotic arm can move to the vicinity of the target through the set program. Since the object to be grasped is close to the dexterous hand at this time, the grasping accuracy can be improved by reducing the moving range of the teleoperation. It can be obtained by weighted calculation of the current position and the target position. The calculation formula is shown in formula (17):

[0168]

[0169] Among them, w is the weight coefficient, the value range is 0.005~0.05, is the current 6-element posture data of the robotic arm, is the corrected target posture data.

[0170] In summary, the embodiments of the present application at least include the following beneficial effects:

[0171] (1) The embodiment of the present application proposes a dual-stream arm remote control framework, which calculates the global 6-DOF position of the palm through visual detection of the hand posture and maps it to the end of the robotic arm; calculates the five bending angles of the fingers and one rotation angle of the thumb through the hand posture, and maps them to the five-finger dexterous hand, thereby realizing the visual remote control function of the robotic arm.

[0172] (2) The embodiment of the present application proposes a multi-mode remote control method, which can perform remote grasping through different control modes according to actual task requirements.

[0173] (3) The embodiment of the present application proposes a lightweight robotic arm remote-operated grasping solution, which adopts a 6-DOF robotic arm and a five-finger dexterous hand to achieve more flexible grasping. At the same time, a real-time hand posture detection algorithm and a data-simplified visual remote-operation control algorithm are used to achieve intuitive and precise control of the robotic arm by identifying the posture of a single human hand.

[0174] (4) The embodiment of the present application proposes a simple hand coordinate preprocessing process to improve the stability of the robot arm movement while ensuring real-time performance.

[0175] (5) The embodiment of the present application adopts a target detection algorithm for open scenes, which can detect objects in most scenes and perform semi-automatic grasping of specified objects in combination with the camera at the end of the robotic arm.

[0176] (6) Compared with the common robot visual remote control method, the embodiment of the present application uses a 6-DOF robotic arm and a five-finger dexterous hand, which has better flexibility. The proposed method only uses the skeletal point data of the palm to control the robot, and has better real-time performance. In addition, the embodiment of the present application also proposes a semi-automatic grasping mode based on the open scene target detection algorithm, which improves the remote control grasping efficiency.

[0177] See also Fig.11 The embodiment of the present application also provides a vision-based humanoid robotic arm multi-mode teleoperation grasping system, which can implement the vision-based humanoid robotic arm multi-mode teleoperation grasping method, and the system includes:

[0178] The first module 101 is used to determine the dexterous hand control data and the robotic arm control data according to each frame of the operator's hand action image; wherein the dexterous hand control data is used to control the dexterous hand to perform a first action; and the first action is determined according to the finger angle of the operator;

[0179] The second module 102 is used to map the robot arm control data to the entire robot arm in the full teleoperation control mode, and control the entire robot arm to perform a second action; the second action is determined according to the palm posture of the operator;

[0180] The third module 103 is used to control the robot arm as a whole to perform a third action according to the first coordinate of the object to be grasped relative to the robot arm base in the semi-automatic operation control mode; map the robot arm control data to the end of the robot arm, and control the end of the robot arm to perform a fourth action; wherein the third action is to automatically move to a preset distance range around the object to be grasped; wherein the fourth action is determined according to the palm posture of the operator;

[0181] The fourth module 104 is used to complete the remote control grasping through the coordination of the first action and the second action, or through the coordination of the first action, the third action and the fourth action.

[0182] It should be noted that, in some embodiments, the system may also include the following modules:

[0183] The fifth module performs open scene target detection based on the depth camera at the end of the robotic arm to obtain the target object; and stores the robotic arm joint angle state and object label corresponding to the target object in the database.

[0184] It can be understood that the contents of the above method embodiments are all applicable to the present system embodiments, the functions specifically implemented by the present system embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0185] The embodiment of the present application also provides an electronic device, the electronic device includes a memory and a processor, the memory stores a computer program, and the processor implements the above-mentioned vision-based humanoid robot arm multi-mode teleoperation grasping method when executing the computer program. The electronic device can be any intelligent terminal including a tablet computer, a car computer, etc.

[0186] It can be understood that the contents of the above method embodiments are all applicable to the present device embodiments, the functions specifically implemented by the present device embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0187] See also Fig.12, Fig.12 The hardware structure of an electronic device of another embodiment is illustrated, and the electronic device includes:

[0188] The processor 201 may be implemented by a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present application;

[0189] The memory 202 can be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 202 can store an operating system and other application programs. When the technical solution provided in the embodiment of this specification is implemented by software or firmware, the relevant program code is stored in the memory 202, and the processor 201 calls and executes the vision-based humanoid robot arm multi-mode teleoperation grasping method of the embodiment of this application;

[0190] Input / output interface 203, used to implement information input and output;

[0191] The communication interface 204 is used to realize the communication interaction between the device and other devices. The communication can be realized through a wired manner (such as USB, network cable, etc.) or a wireless manner (such as mobile network, WIFI, Bluetooth, etc.);

[0192] Bus 205 , which transmits information between various components of the device (e.g., processor 201 , memory 202 , input / output interface 203 , and communication interface 204 );

[0193] The processor 201 , the memory 202 , the input / output interface 203 and the communication interface 204 are connected to each other in communication within the device via the bus 205 .

[0194] An embodiment of the present application also provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, it implements the above-mentioned vision-based humanoid robotic arm multi-mode teleoperation grasping method.

[0195] It can be understood that the contents of the above method embodiments are all applicable to the present storage medium embodiments, the functions specifically implemented by the present storage medium embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0196] The memory, as a non-transient computer-readable storage medium, can be used to store non-transient software programs and non-transient computer executable programs. In addition, the memory may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory may optionally include a memory remotely disposed relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0197] The embodiments described in the embodiments of the present application are intended to more clearly illustrate the technical solutions of the embodiments of the present application and do not constitute a limitation on the technical solutions provided in the embodiments of the present application. Those skilled in the art will appreciate that with the evolution of technology and the emergence of new application scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0198] Those skilled in the art will appreciate that the technical solutions shown in the figures do not constitute a limitation on the embodiments of the present application, and may include more or fewer steps than shown in the figures, or a combination of certain steps, or different steps.

[0199] The system embodiments described above are merely illustrative, and the units described as separate components may or may not be physically separated, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the present embodiment.

[0200] Those skilled in the art will appreciate that all or some of the steps in the methods disclosed above, and the functional modules / units in the systems and devices may be implemented as software, firmware, hardware, or a suitable combination thereof.

[0201] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0202] It should be understood that in the present application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0203] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the system embodiments described above are only schematic. For example, the division of the above units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0204] The units described above as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0205] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0206] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including multiple instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, referred to as ROM), random access memory (Random Access Memory, referred to as RAM), disk or optical disk and other media that can store programs.

[0207] The preferred embodiments of the present invention are described above with reference to the accompanying drawings, but the scope of the rights of the present invention is not limited thereto. Any modification, equivalent substitution and improvement made by a person skilled in the art without departing from the scope and essence of the present invention should be within the scope of the rights of the present invention.

Claims

1. A multi-mode teleoperation grasping method for a humanoid robot arm based on vision, characterized in that: The following steps are involved: Determine dexterous hand control data and robotic arm control data according to each frame of hand motion image of the operator; wherein the dexterous hand control data is used to control the dexterous hand to perform a first motion; the first motion is determined according to the finger angle of the operator; In the full teleoperation control mode, the robot arm control data is mapped to the entire robot arm, and the entire robot arm is controlled to perform a second action; the second action is determined according to the palm posture of the operator; In the semi-automatic operation control mode, according to the first coordinate of the object to be grasped relative to the base of the robot arm, the robot arm is controlled to perform a third action as a whole; the robot arm control data is mapped to the end of the robot arm, and the end of the robot arm is controlled to perform a fourth action; wherein the third action is to automatically move to a preset distance range around the object to be grasped; wherein the fourth action is determined according to the palm posture of the operator; The remote control grasping is completed by the cooperation of the first action and the second action, or by the cooperation of the first action, the third action and the fourth action; The method of determining the dexterous hand control data and the robotic arm control data according to each frame of the operator's hand motion image comprises the following steps: Obtain the operator's hand motion image for each frame; Determining key three-dimensional coordinates of key points of the hand according to the hand motion image; Determine finger angles and palm positions according to the key three-dimensional coordinates; Quantifying the finger angles to obtain six-dimensional dexterous hand control data; Performing Euler angle conversion processing on the palm posture to obtain six-dimensional robotic arm control data; Wherein, determining the finger angle and palm posture according to the key three-dimensional coordinates includes the following steps: Determine the finger joint angle according to the key three-dimensional coordinates; Determine the degree of finger bending and the degree of lateral opening and closing of the thumb according to the joint angle to obtain the finger angle; Determine the finger joint position according to the key three-dimensional coordinates; Calculating the palm posture according to the joint positions; The step of mapping the robot arm control data to the entire robot arm and controlling the entire robot arm to perform a second action comprises the following steps: Determine the joint angle of the robot arm according to the target posture data in the robot arm control data; Calculating the movement speed and movement acceleration according to the relationship between the change of the joint angle of the robot arm and time; Obtaining a motion trajectory optimization result according to the motion speed, the motion acceleration and the six-degree-of-freedom data in the robot arm control data; According to the transparent transmission time and the motion trajectory optimization result, the robot arm is controlled to perform a second action as a whole.

2. The method according to claim 1, characterized in that The controlling the robotic arm as a whole to perform the third action according to the first coordinate of the object to be grasped relative to the base of the robotic arm comprises the following steps: Obtaining the object label of the object to be grasped determined by the operator; According to the object label, obtaining the object depth information of the object to be grasped through the depth camera at the end of the robotic arm; Calculating a first coordinate of the object to be grasped relative to a base of the robotic arm according to the object depth information and a current state of the robotic arm; According to the first coordinate, the robotic arm is controlled to automatically move to a preset distance range around the object to be grasped to complete a third action.

3. The method according to claim 2, characterized in that Before the step of obtaining the object label of the object to be grasped determined by the operator, the following steps are also included: Perform open scene target detection based on the depth camera at the end of the robotic arm to obtain the target object; The robot arm joint angle state and object label corresponding to the target object are stored in a database.

4. The method according to claim 3, characterized in that The method of performing open scene target detection based on the depth camera at the end of the robotic arm to obtain the target object includes the following steps: The end of the robotic arm equipped with a depth camera is moved around the robotic arm to take pictures of the scene; The yolo-world target detection model is used to perform target detection on the scene photo to obtain the target object.

5. A vision-based multi-mode teleoperated grasping system for humanoid robotic arms, characterized in that: include: The first module is used to determine the dexterous hand control data and the robotic arm control data according to each frame of the operator's hand action image; wherein the dexterous hand control data is used to control the dexterous hand to perform a first action; and the first action is determined according to the operator's finger angle; The second module is used to map the control data of the robot arm to the entire robot arm in the full teleoperation control mode, and control the entire robot arm to perform a second action; the second action is determined according to the palm posture of the operator; The third module is used to control the robot arm as a whole to perform a third action according to the first coordinate of the object to be grasped relative to the robot arm base in a semi-automatic operation control mode; map the robot arm control data to the end of the robot arm, and control the end of the robot arm to perform a fourth action; wherein the third action is to automatically move to a preset distance range around the object to be grasped; wherein the fourth action is determined according to the palm posture of the operator; A fourth module is used to complete remote control grasping through the cooperation of the first action and the second action, or through the cooperation of the first action, the third action and the fourth action; The method of determining the dexterous hand control data and the robotic arm control data according to each frame of the operator's hand motion image comprises the following steps: Obtain the operator's hand motion image for each frame; Determining key three-dimensional coordinates of key points of the hand according to the hand motion image; Determine finger angles and palm positions according to the key three-dimensional coordinates; Quantifying the finger angles to obtain six-dimensional dexterous hand control data; Performing Euler angle conversion processing on the palm posture to obtain six-dimensional robotic arm control data; Wherein, determining the finger angle and palm posture according to the key three-dimensional coordinates includes the following steps: Determine the finger joint angle according to the key three-dimensional coordinates; Determine the degree of finger bending and the degree of lateral opening and closing of the thumb according to the joint angle to obtain the finger angle; Determine the finger joint position according to the key three-dimensional coordinates; Calculating the palm posture according to the joint positions; The step of mapping the robot arm control data to the entire robot arm and controlling the entire robot arm to perform a second action comprises the following steps: Determine the joint angle of the robot arm according to the target posture data in the robot arm control data; Calculating the movement speed and movement acceleration according to the relationship between the change of the joint angle of the robot arm and time; Obtaining a motion trajectory optimization result according to the motion speed, the motion acceleration and the six-degree-of-freedom data in the robot arm control data; According to the transparent transmission time and the motion trajectory optimization result, the robot arm is controlled to perform a second action as a whole.

6. An electronic device, characterized in that: including a processor and a memory; The memory is used to store programs; The processor executes the program to implement the method according to any one of claims 1 to 4.

7. A computer storage medium storing a program executable by a processor, characterized in that: The program executable by the processor is used to implement the method according to any one of claims 1 to 4 when executed by the processor.

Images