A robot gesture remote operation method and related device
By collecting and processing image information of RGB-D cameras, accurately distinguishing user gesture categories, solving the problem that remote operation in the existing technology is difficult to distinguish left-handed hands, and achieving efficient robot gesture remote operation, improving user experience and operation efficiency.
Patent Information
- Application Number
- CN202210081710.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-01-24
- Publication Date
- 2024-12-03
- Estimated Expiration
- 2042-01-24
AI Technical Summary
The existing remote operation technology is difficult to effectively distinguish left-hand gestures, which leads to inconvenience in user operation, especially in scenarios such as post-disaster rescue and remote surgery that require complex operations.
By collecting user's gesture image information, using RGB-D camera to obtain gesture box information, gesture three-dimensional position information, gesture confidence and gesture posture information, combined with depth images and color images, estimate key point center information and gesture direction information, and judge the key point center information and gesture direction information. Gesture category information, thereby controlling the robot's robotic movements.
It can accurately distinguish left and right hands without wearing interactive devices, improve the efficiency and user experience of remote operation, and efficiently control the two-arm robot for complex operations.
Smart Images

Figure CN114495273B_ABST
Abstract
Description
Technical Field
[0001] The embodiments of the present application relate to the field of vision, and in particular to a robot gesture remote operation method and related devices. Background Art
[0002] Teleoperation refers to controlling remote equipment to complete complex operations in an environment far away from the object of operation under human control and participation. In special scenarios such as post-disaster rescue sites and remote surgery, teleoperation highlights its value to society. Among them, teleoperation that simulates the gestures of human hands has the flexibility of human hands and can remotely control mechanical equipment to complete complex operations.
[0003] The existing solutions, based on data gloves and myoelectric bracelets, can remotely control robots to complete corresponding actions. Vision-based detection and differentiation of both hands is a difficult point in visual teleoperation technology. Because when both hands show different gestures, the difference in gestures is much greater than the difference in asymmetry of human hands. However, the existing methods can only extract the features of human hands. These methods cannot distinguish between left and right hands well, which brings inconvenience to users. Summary of the invention
[0004] The embodiments of the present application provide a robot gesture remote operation method and related devices.
[0005] A robot gesture teleoperation method, comprising:
[0006] Collecting user's gesture image information, where the gesture image information is image information representing the user's hand movements;
[0007] Obtaining gesture box information, gesture three-dimensional position information, gesture confidence, and gesture posture information according to the gesture image information, wherein the gesture box information is information indicating the range of the user's hand, the gesture confidence indicates the probability that the gesture posture information is consistent with the user's finger posture, and the gesture posture information is information indicating the posture in the gesture database that has the highest probability of being consistent with the user's finger posture;
[0008] Estimate key point center information and gesture direction information through the gesture image information, wherein the key point center information is information indicating the estimated center position of the hand, and the gesture direction information is information indicating the orientation of the user's hand;
[0009] Obtaining gesture category information according to the gesture box information, the gesture confidence and the key point center information, wherein the gesture category information indicates the left and right category of the user's hand;
[0010] The manipulator action of the robot is controlled by the first target gesture information, wherein the first target gesture information includes the gesture category information, the gesture three-dimensional position information, the gesture direction information and the gesture posture information.
[0011] Optionally, obtaining gesture box information, gesture three-dimensional position information, gesture confidence and gesture posture information according to the gesture image information includes:
[0012] Extracting features from the color image to obtain a feature map, wherein the color image is part of the gesture image information;
[0013] Based on the depth image, the feature map is detected to obtain the three-dimensional position information of the gesture and the gesture frame information, wherein the depth image is part of the gesture image information;
[0014] The gesture box information is classified to obtain the gesture posture information and the gesture confidence.
[0015] Optionally, estimating key point center information and gesture direction information through the gesture image information includes:
[0016] Determine two-dimensional information of key points according to the color image;
[0017] Based on the depth image, determining the three-dimensional information of the key points through the two-dimensional information of the key points;
[0018] The key point center information and the gesture direction information are calculated through the key point three-dimensional information.
[0019] Optionally, obtaining gesture category information according to the gesture box information, the gesture confidence and the key point center information includes:
[0020] Determining whether the gesture confidence is greater than a confidence threshold, where the confidence threshold is a preset constant value;
[0021] If yes, determining whether the key point center information matches the gesture box information;
[0022] If there is a match, determining the gesture category information according to the key point center information that matches the gesture box information;
[0023] If not, determining whether the distance between the estimated center position of the hand in the key point center information and the center position of the hand determined in the gesture three-dimensional position information is less than a distance threshold;
[0024] If it is less than, the gesture category information is determined according to the category of the estimated hand center position whose distance from the determined hand center position is less than the distance threshold.
[0025] Optionally, controlling the manipulator action of the robot through the first target gesture information includes:
[0026] The position information of the manipulator is obtained by calculating the three-dimensional position information of the gesture;
[0027] The robot direction information is obtained by calculating the gesture direction information;
[0028] The robot movement is controlled by second target gesture information, wherein the second target gesture information includes the gesture category information, the robot position information, the robot direction information and the gesture posture information.
[0029] Optionally, based on the depth image, determining the three-dimensional information of the key point by using the two-dimensional information of the key point includes:
[0030] The three-dimensional information of the key points is calculated by the following formula:
[0031]
[0032]
[0033]
[0034] z i 、x i and i is the three-dimensional coordinate information of the i-th key point, that is, the three-dimensional information of the key point;
[0035] d i is the depth value of the i-th key point, obtained from the depth image;
[0036] s is the depth range of the depth image;
[0037] u i and v i is the two-dimensional coordinate information of the i-th key point, that is, the two-dimensional information of the key point;
[0038] cx, cy are the coordinate values of the center point of the depth image;
[0039] fx is the focal length of the camera on the x-axis, and fy is the focal length of the camera on the y-axis.
[0040] Optionally, calculating the key point center information through the key point three-dimensional information includes:
[0041] The key point center information is calculated by the following formula:
[0042]
[0043]
[0044]
[0045] X, Y, and Z are the estimated center coordinate information of the hand of the corresponding category;
[0046] z i 、x i and i is the three-dimensional coordinate information of the i-th key point of the hand of the corresponding category;
[0047] k is the total number of key points of the hand in the corresponding category.
[0048] A robot gesture teleoperation device, comprising:
[0049] A collection unit, used to collect user's gesture image information, wherein the gesture image information is image information representing the user's hand movement;
[0050] a processing unit, configured to obtain gesture box information, gesture three-dimensional position information, gesture confidence, and gesture posture information according to the gesture image information, wherein the gesture box information is information indicating the range of the user's hand, the gesture confidence indicates the probability that the gesture posture information is consistent with the user's finger posture, and the gesture posture information is information indicating the posture in a gesture database that has the highest probability of being consistent with the user's finger posture;
[0051] an estimating unit, configured to estimate key point center information and gesture direction information from the gesture image information, wherein the key point center information is information indicating an estimated center position of a hand, and the gesture direction information is information indicating a direction of the user's hand;
[0052] The processing unit is further used to obtain gesture category information according to the gesture box information, the gesture confidence and the key point center information, wherein the gesture category information indicates the left and right category of the user's hand;
[0053] A control unit is used to control the manipulator action of the robot through first target gesture information, wherein the first target gesture information includes the gesture category information, the gesture three-dimensional position information, the gesture direction information and the gesture posture information.
[0054] A robot gesture teleoperation device, comprising:
[0055] CPU, memory and input / output interface;
[0056] The memory is a short-term storage memory or a persistent storage memory;
[0057] The central processing unit is configured to communicate with the memory and execute instructions in the memory to perform the above method.
[0058] A computer-readable storage medium includes instructions. When the instructions are executed on a computer, the computer executes the above method.
[0059] It can be seen from the above technical solutions that the embodiments of the present application have the following advantages:
[0060] After collecting the user's gesture image information, the gesture box information, gesture three-dimensional position information, gesture confidence and gesture posture information are obtained according to the gesture image information, and then the key point center information and gesture direction information are estimated through the gesture image information. Then the gesture category information is obtained according to the gesture box information, gesture confidence and key point center information. Finally, the robot's manipulator action is controlled by the first target gesture information. The left and right hands can be distinguished by the gesture box information, gesture confidence and key point center information, and the left and right categories of the user's hands, that is, the gesture category information, can be obtained. Combined with other information, the left and right manipulators of the robot are remotely operated accordingly, which improves efficiency and gives users a better experience. BRIEF DESCRIPTION OF THE DRAWINGS
[0061] Figure 1 This is a schematic diagram of gesture remote operation in an embodiment of the present application;
[0062] Figure 2 This is a schematic diagram of an embodiment of the robot gesture remote operation method of the present application;
[0063] Figure 3 This is a schematic diagram of another embodiment of the robot gesture remote operation method of the present application embodiment;
[0064] Figure 4 It is a topological diagram of key points of the human body according to an embodiment of the present application;
[0065] Figure 5 This is a schematic diagram of an embodiment of a robot gesture teleoperation device according to an embodiment of the present application;
[0066] Figure 6 This is a schematic diagram of another embodiment of the robot gesture teleoperation device according to an embodiment of the present application. DETAILED DESCRIPTION
[0067] The embodiments of the present application provide a robot gesture remote operation method and related devices.
[0068] The teleoperation method of traditional robots can control traditional mechanical arms through devices such as mice and keyboards. However, traditional devices cannot easily and effectively control a two-arm robot equipped with a mechanical arm, and cannot distinguish between the user's left and right hands. The existing remote control method of a two-arm robot that imitates human hands and arms is based on remote control using wearable interactive devices, but the interactive device limits the movement of the user's hands, and the method can only extract the features of the human hand. These methods cannot distinguish between the left and right hands well. The gesture teleoperation method provided in the embodiments of the present application allows users to no longer be restricted by interactive devices, and can distinguish between the user's left and right hands for efficient control.
[0069] See also Figure 1 The remote operation of the embodiment of the present application collects the user's gesture image information through an RGB-D camera, processes the collected gesture image information, and maps it to the manipulator of the dual-arm robot, that is, the five-finger dexterous hand. The five-finger dexterous hand grabs objects on the table according to the user's gesture operation to complete the remote operation. Among them, the dual-arm robot can be a Baxter robot or other robots, which are not limited here. Each arm of the Baxter dual-arm robot has 6 degrees of freedom, which can realize basic path planning and motion control functions, and the two arms have a collaborative function, which can avoid collisions between the two arms in complex operations. The manipulator at the end also has 6 degrees of freedom, including 5 bending degrees of freedom of 5 fingers and 1 rotational degree of freedom of the thumb. It is possible to grasp or operate different objects through different gestures.
[0070] For a detailed description of the gesture remote operation method of the present application, please refer to Figure 2 , an embodiment of the robot gesture teleoperation method of the present application includes:
[0071] 201. Collecting user's gesture image information;
[0072] The user's gesture image information is collected. The gesture image information is the image information representing the user's hand movements. Specifically, the user does not need to wear interactive equipment, and can use the RGB-D camera to collect relevant image information by making gestures with bare hands within the photographic range of the RGB-D camera. The RGB-D camera can collect RGB color images and depth images of the user's hands to generate relevant three-dimensional information. The gesture image information here includes color images and depth images.
[0073] It is understandable that the type of RGB-D camera can be a realsense d435i camera, or a camera such as a kinect camera that can capture depth images and two-dimensional color images, and is not specifically limited here.
[0074] 202. Obtain gesture box information, gesture three-dimensional position information, gesture confidence, and gesture posture information according to the gesture image information;
[0075] The gesture image information is processed to obtain gesture box information, gesture three-dimensional position information, gesture confidence and gesture posture information. Specifically, feature extraction is performed on the color image in the gesture image information to obtain a feature map, and the feature map is detected in combination with the depth image to obtain the gesture box information and gesture three-dimensional position information. The gesture box information is input into the EfficientNet-B5 classifier for classification, and compared with the gesture postures in the gesture posture library, and the most similar gesture posture is selected to determine the gesture posture information. The corresponding gesture confidence can be obtained according to the gesture posture information. Among them, the gesture box information is information indicating the range of the user's hand, the gesture confidence indicates the probability that the gesture posture information matches the user's finger posture, and the gesture posture information is information indicating the posture with the highest probability of matching the user's finger posture in the gesture database.
[0076] 203. Estimate key point center information and gesture direction information based on the gesture image information;
[0077] The key point center information and gesture direction information are estimated through gesture image information. Specifically, the BlazePose algorithm network can be used to analyze the color image to determine the two-dimensional information of the key points, and then the three-dimensional information of the key points can be obtained by combining the depth image. Among them, the BlazePose algorithm network is a lightweight convolutional neural network architecture for human posture estimation and real-time inference. During the inference process, the network generates 33 body key points for the human body and runs at a speed of more than 30 frames per second. The center position information of the hand, that is, the key point center information, can be estimated based on the three-dimensional position information of the key points of the hand. A coordinate system can be established based on the three-dimensional information of the key points, and the gesture direction information can be determined using the quaternion rotation principle. The gesture direction information is information representing the direction of the user's hand.
[0078] 204. Obtain gesture category information according to the gesture box information, the gesture confidence and the key point center information;
[0079] The gesture category information is obtained by analyzing and judging the gesture box information, gesture confidence and key point center information. Among them, the gesture category information represents the left and right categories of the user's hand, which is used to distinguish the left and right hands of the user. Specifically, the gesture category information is obtained by comparing the gesture confidence with the preset confidence threshold, and judging whether the key point center information matches the gesture box information or judging whether the distance between the estimated hand center position in the key point center information and the hand center position determined in the gesture three-dimensional position information is less than the distance threshold.
[0080] 205. Controlling the manipulator movement of the robot through the first target gesture information;
[0081] After obtaining the gesture category information, the robot's manipulator action is controlled by the first target gesture information. The first target gesture information includes gesture category information, gesture three-dimensional position information, gesture direction information, and gesture posture information. After processing the first target gesture information, the second target gesture information is obtained, and the second target gesture information is mapped to the manipulator of the dual-arm robot to control the manipulator to perform corresponding actions.
[0082] In the embodiment of the present application, after collecting the user's gesture image information, the gesture box information, the gesture three-dimensional position information, the gesture confidence and the gesture posture information are obtained according to the gesture image information, and then the key point center information and the gesture direction information are estimated through the gesture image information. Then the gesture category information is obtained according to the gesture box information, the gesture confidence and the key point center information. Finally, the manipulator action is controlled by the first target gesture information. The left and right hands can be distinguished by the gesture box information, the gesture confidence and the key point center information, and the left and right categories of the user's hands, that is, the gesture category information, can be obtained. Combined with other information, the left and right manipulators of the robot are remotely operated accordingly, which improves efficiency. In addition, since the hand movements without wearable interactive devices are collected, that is, the gesture image information is collected, and then the information is processed to obtain the first target gesture information, the manipulator is remotely controlled to perform corresponding actions according to the first target gesture information, so that when remote operation is possible, the hand can gesture without restriction, giving the user a better experience.
[0083] See also Figure 3 Another embodiment of the robot gesture teleoperation method of the present application embodiment includes:
[0084] 301. Collecting user's gesture image information;
[0085] The user's gesture image information is collected. The gesture image information is the image information representing the user's hand movements. Specifically, the user does not need to wear interactive equipment, and can use the RGB-D camera to collect relevant image information by making gestures with bare hands within the photographic range of the RGB-D camera. The RGB-D camera can collect RGB color images and depth images of the user's hands to generate relevant three-dimensional information. The gesture image information here includes color images and depth images.
[0086] It is understandable that the type of RGB-D camera can be a realsense d435i camera, or a camera such as a kinect camera that can capture depth images and two-dimensional color images, and is not specifically limited here.
[0087] 302. Extract features from the color image to obtain a feature map;
[0088] Feature extraction is performed on the color image to obtain a feature map. Specifically, the color image is input to the input layer of the convolutional neural network, and then to the convolutional layer for feature extraction, and finally a feature map of the user's hand is obtained. It is understood that the type of convolutional neural network can be a network that can be used for detection and feature extraction, such as a DLA-34 network, and is not specifically limited here.
[0089] 303. Based on the depth image, detect the feature map to obtain the three-dimensional position information of the gesture and the gesture frame information;
[0090] Based on the depth image, the feature map is detected to obtain the three-dimensional position information of the gesture and the gesture box information. Specifically, a coordinate system is established, and the feature map is detected to obtain the two-dimensional center coordinates of the hand and the width and height of the box representing the range of the hand. The offset can be set according to practical experience. The depth image is then used for coordinate mapping to obtain the three-dimensional position information of the gesture and the gesture box information. For example, the two-dimensional center coordinates of the hand detected from the feature map are (x, y), the width of the box is w, the height of the box is h, the offset is (Δx, Δy), and the depth value obtained from the depth image is z. Then the two-dimensional center coordinates of the hand are (x+Δx, y+Δy), and the coordinates of one diagonal point of the hand box are (x+Δx+w / 2, y+Δy+h / 2), and the coordinates of the other three diagonal points are deduced accordingly. The three-dimensional position information of the gesture is (x+Δx, y+Δy, z), and the gesture box information H_box(x, y, z, w, h) represents the coordinate range of the hand with (x+Δx, y+Δy, z) as the center coordinate and w, h as the width and height.
[0091] 304. Classify the gesture box information to obtain the gesture posture information and the gesture confidence;
[0092] The gesture box information is classified to obtain gesture information. Specifically, after determining the range of the hand box, the feature map is cropped according to the box, and then the cropped hand image is input into the EfficientNet-B5 classifier for classification. Among them, a gesture library is pre-set, which may include a variety of existing gestures, such as 24 English letters or 10 Arabic numerals made by hand. Compare in the gesture library to find the gesture with the highest probability similar to the user's hand, and determine it as gesture information, and determine its probability as gesture confidence. For example, the gesture in the cropped image is a gesture of the number 1, and there is just no gesture of the number 1 in the gesture library, only the gestures of the number 2, the number 3, the number 4 and the number 5, and the probability of the number 2 is 90%, the probability of the number 3 is 80%, the probability of the number 4 is 70%, and the probability of the number 5 is 60%. By comparison, the gesture of the number 2 with the highest probability is determined as the gesture information, and the gesture confidence is determined to be 90%.
[0093] 305. Determine two-dimensional information of key points according to the color image;
[0094] Determine the two-dimensional information of key points based on the color image. The BlazePose algorithm network can be used to analyze the color image to determine the two-dimensional information of key points. The BlazePose algorithm network is a lightweight convolutional neural network architecture used for human pose estimation and real-time inference. See Figure 4 , the network can generate two-dimensional key point information based on the key point topology map of the human body. During the inference process, the network generates 33 body key points for the human body and runs at a speed of more than 30 frames per second. In practice, it is not necessary to obtain the 10 key points of the face, but only 22 key points, especially the key points of the left and right hands.
[0095] 306. Determine the three-dimensional information of the key point based on the depth image and the two-dimensional information of the key point;
[0096] Based on the depth image, the three-dimensional information of the key points is determined by the two-dimensional information of the key points. Specifically, the three-dimensional information of the key points is calculated by the following formula:
[0097]
[0098]
[0099]
[0100] z i 、x i and i is the three-dimensional coordinate information of the i-th key point, that is, the three-dimensional information of the key point;
[0101] d i is the depth value of the i-th key point, obtained from the depth image;
[0102] s is the depth range of the depth image;
[0103] u i and v i is the two-dimensional coordinate information of the i-th key point, that is, the two-dimensional information of the key point;
[0104] cx, cy are the coordinate values of the center point of the depth image;
[0105] fx is the focal length of the camera on the x-axis, and fy is the focal length of the camera on the y-axis.
[0106] 307. Calculate the key point center information and the gesture direction information through the key point three-dimensional information;
[0107] The key point center information and gesture direction information are calculated through the key point three-dimensional information. Specifically, the key point center information is calculated through the following formula:
[0108]
[0109]
[0110]
[0111] X, Y, and Z are the estimated center coordinate information of the hand of the corresponding category;
[0112] z i 、x i and i is the three-dimensional coordinate information of the i-th key point of the hand of the corresponding category;
[0113] k is the total number of key points of the hand in the corresponding category.
[0114] For the left hand, see Figure 4 , the key points of the left hand are the 15th, 17th, 19th and 21st points, then the value of k is 4, and the center coordinate information of the left hand is:
[0115]
[0116]
[0117]
[0118] And so on for the right hand.
[0119] A coordinate system is established based on the three-dimensional information of the key points. Specifically, the z-axis is defined as the unit normal vector of the plane formed by the 15th, 17th and 19th points, the x-axis is defined as the unit direction vector from the 15th point to the center of the left hand, and the y-axis is defined as the unit direction vector of the cross product of the x-axis and the z-axis. Based on this coordinate system, the gesture direction information can be obtained using the quaternion rotation principle.
[0120] 308. Determine whether the gesture confidence is greater than the confidence threshold. If so, execute step 309. If not, execute step 310. The confidence threshold is a preset constant value ranging from 0.5 to 1. It is generally 0.85 based on experience.
[0121] 309, determining whether the key point center information matches the gesture box information, if so, executing step 312, if not, executing step 311;
[0122] Specifically, it is determined whether the estimated center position of the hand in the key point center information is within the box of the hand. If it is within the box, the category of the hand can be determined.
[0123] 310, determining whether the distance between the estimated hand center position in the key point center information and the hand center position determined in the gesture three-dimensional position information is less than a distance threshold, if so, executing step 313, if not, executing step 311, wherein the distance threshold is a constant value pre-set based on practical experience;
[0124] 311. End the control of the manipulator;
[0125] End control of the robot. Unable to determine the type of action, end remote control to prevent the robot from performing incorrect actions.
[0126] 312. Determine the gesture category information according to the key point center information matching the gesture frame information;
[0127] The gesture category information is determined based on the key point center information that matches the gesture box information. If the category of the key point center information that matches the gesture box information is left, the gesture category information represents the information of the left hand. If the category of the key point center information that matches the gesture box information is right, the gesture category information represents the information of the right hand. For example, if the estimated center position of the left hand is within the hand box, the gesture category information is determined as the left category, that is, the user's hand is the left hand. If the estimated center position of the right hand is within the hand box, the gesture category information is determined as the right category, that is, the user's hand is the right hand.
[0128] 313. Determine the gesture category information according to the category of the estimated hand center position whose distance from the determined hand center position is less than the distance threshold;
[0129] The gesture category information is determined based on the category of the estimated center position of the hand whose distance from the determined center position of the hand is less than the distance threshold. For example, the estimated center position of the left hand in the key point center information is point P1, the estimated center position of the right hand is point P2, and the determined center position of the hand in the gesture three-dimensional position information is point P3, and the distance threshold is set to 5. If the distance between P1 and P3 is 2, and the distance between P2 and P3 is 7, 2<5<7, the gesture category information is determined to be the left category, that is, the user's hand is the left hand.
[0130] 314. Calculate the position information of the manipulator through the three-dimensional position information of the gesture;
[0131] The position information of the manipulator is calculated by the three-dimensional position information of the gesture. In order to efficiently control the manipulator, the three-dimensional position information of the gesture needs to be processed to obtain the position information of the manipulator. Specifically, the position information of the manipulator is calculated by the following formula:
[0132]
[0133]
[0134]
[0135] and represents the three-dimensional center coordinate information of the manipulator in the i-th frame, that is, the position information of the manipulator, where i ≥ 2;
[0136] and Represents the 3D center coordinate information of the user's hand in the i-th frame, that is, the 3D position information of the gesture;
[0137] k p is the position increment coefficient, which can be pre-set based on practical experience and is generally set to 1.
[0138] 315. Calculate the direction information of the robot arm through the gesture direction information;
[0139] The direction information of the manipulator is calculated by the gesture direction information. In order to efficiently control the manipulator, the gesture direction information needs to be processed to obtain the manipulator direction information. Specifically, the manipulator direction information is calculated by the following formula:
[0140]
[0141]
[0142]
[0143]
[0144] represents the direction information of the manipulator in the i-th frame, that is, the direction information of the manipulator;
[0145] Indicates the direction information of the user's hand in the i-th frame, that is, the gesture direction information;
[0146] k o is the directional increment coefficient, which can be pre-set based on practical experience and is generally set to 1.
[0147] 316. Controlling the movement of the robot arm through the second target gesture information;
[0148] The manipulator movement is controlled by the second target gesture information. The second target gesture information includes gesture category information, manipulator position information, manipulator direction information and gesture posture information. Specifically, the second target gesture information is incrementally mapped to the manipulator on the dual-arm robot to remotely control the manipulator movement.
[0149] In this embodiment, after collecting the user's gesture image information, the gesture box information, gesture three-dimensional position information, gesture confidence and gesture posture information are obtained according to the gesture image information, and then the key point center information and gesture direction information are estimated through the gesture image information. Then the gesture category information is obtained according to the gesture box information, gesture confidence and key point center information. Finally, the robot action is controlled by the second target gesture information. Since the hand action without a wearable interactive device is collected, that is, the gesture image information is collected, and then the information is processed to obtain the first target gesture information, the robot is remotely controlled to perform the corresponding action according to the first target gesture information, so that the hand can gesture without restriction when remote operation is possible, giving the user a better experience. By introducing the key point center information and gesture confidence for judgment, the gesture box information and gesture three-dimensional position information can be detected to judge the left and right hand categories, and the action accuracy can be improved.
[0150] See also Figure 5 , an embodiment of the robot gesture teleoperation device of the present application includes:
[0151] A collection unit 501 is used to collect user's gesture image information, where the gesture image information is image information representing the user's hand movements;
[0152] A processing unit 502 is used to obtain gesture box information, gesture three-dimensional position information, gesture confidence and gesture posture information according to the gesture image information, wherein the gesture box information is information indicating the range of the user's hand, the gesture confidence indicates the probability that the gesture posture information is consistent with the user's finger posture, and the gesture posture information is information indicating the posture in the gesture database that has the highest probability of being consistent with the user's finger posture;
[0153] An estimation unit 503, configured to estimate key point center information and gesture direction information from the gesture image information, wherein the key point center information is information indicating an estimated center position of the hand, and the gesture direction information is information indicating the orientation of the user's hand;
[0154] The processing unit 502 is further configured to obtain gesture category information according to the gesture box information, the gesture confidence and the key point center information, wherein the gesture category information indicates the left and right category of the user's hand;
[0155] The control unit 504 is used to control the manipulator action of the robot through the first target gesture information, wherein the first target gesture information includes the gesture category information, the gesture three-dimensional position information, the gesture direction information and the gesture posture information.
[0156] In the embodiment of the present application, after the acquisition unit 501 acquires the user's gesture image information, the processing unit 502 obtains the gesture box information, the gesture three-dimensional position information, the gesture confidence and the gesture posture information according to the gesture image information, and the estimation unit 503 then estimates the key point center information and the gesture direction information according to the gesture image information. Then the gesture category information is obtained according to the gesture box information, the gesture confidence and the key point center information. Finally, the control unit 504 controls the manipulator action according to the first target gesture information. The left and right hands can be distinguished by the gesture box information, the gesture confidence and the key point center information, and the left and right categories of the user's hands, that is, the gesture category information, can be obtained. Combined with other information, the left and right manipulators of the robot are remotely operated accordingly, which improves efficiency. In addition, since the hand movements without the wearable interactive device are collected, that is, the gesture image information is collected, and then the information is processed to obtain the first target gesture information, the manipulator is remotely controlled to perform the corresponding action according to the first target gesture information, so that when the remote operation is possible, the hand gestures are not restricted, giving the user a better experience.
[0157] The functions and processes performed by each unit in the robot gesture teleoperation device of this embodiment are similar to those described above. Figures 1 to 5 The functions and processes performed by the gesture remote control device are similar and will not be repeated here.
[0158] Figure 61 is a schematic diagram of the structure of a gesture remote control device provided in an embodiment of the present application. The gesture remote control device 600 may include one or more central processing units (CPU) 601 and a memory 605. The memory 605 stores one or more application programs or data.
[0159] The memory 605 may be a volatile storage or a persistent storage. The program stored in the memory 605 may include one or more modules, each of which may include a series of instruction operations in the gesture teleoperation device. Furthermore, the central processor 601 may be configured to communicate with the memory 605 and execute a series of instruction operations in the memory 605 on the gesture teleoperation device 600.
[0160] The gesture remote control device 600 may also include one or more power supplies 602, one or more wired or wireless network interfaces 603, one or more input and output interfaces 604, and / or one or more operating systems, such as Windows ServerTM, Mac OS XTM, UnixTM, LinuxTM, FreeBSDTM, etc.
[0161] The CPU 601 can execute the aforementioned Figures 1 to 5 The operations performed by the gesture remote control device in the illustrated embodiment will not be described in detail here.
[0162] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.
[0163] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.
[0164] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.
[0165] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.
[0166] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, read-only memory), random access memory (RAM, random access memory), disk or optical disk and other media that can store program code.
Claims
1. A robot gesture teleoperation method, characterized in that: include: Collecting user's gesture image information, where the gesture image information is image information representing the user's hand movements; Obtaining gesture box information, gesture three-dimensional position information, gesture confidence, and gesture posture information according to the gesture image information, wherein the gesture box information is information indicating the range of the user's hand, the gesture confidence indicates the probability that the gesture posture information is consistent with the user's finger posture, and the gesture posture information is information indicating the posture in the gesture database that has the highest probability of being consistent with the user's finger posture; estimating key point center information and gesture direction information through the gesture image information, wherein the key point center information is information indicating the estimated center position of the hand, and the gesture direction information is information indicating the orientation of the user's hand; Obtaining gesture category information according to the gesture box information, the gesture confidence and the key point center information, wherein the gesture category information indicates the left and right category of the user's hand; The manipulator movement of the robot is controlled by the first target gesture information, wherein the first target gesture information includes the gesture category information, the gesture three-dimensional position information, the gesture direction information and the gesture posture information.
2. The gesture remote operation method according to claim 1, characterized in that: According to the gesture image information, gesture box information, gesture three-dimensional position information, gesture confidence and gesture posture information are obtained, including: Extracting features from the color image to obtain a feature map, wherein the color image is part of the gesture image information; Based on the depth image, the feature map is detected to obtain the three-dimensional position information of the gesture and the gesture frame information, wherein the depth image is part of the gesture image information; The gesture box information is classified to obtain the gesture posture information and the gesture confidence.
3. The gesture remote operation method according to claim 2, characterized in that: Estimating key point center information and gesture direction information through the gesture image information includes: Determine two-dimensional information of key points according to the color image; Based on the depth image, determining the three-dimensional information of the key points through the two-dimensional information of the key points; The key point center information and the gesture direction information are calculated through the key point three-dimensional information.
4. The gesture remote operation method according to claim 3, characterized in that: Obtaining gesture category information according to the gesture box information, the gesture confidence and the key point center information, including: Determining whether the gesture confidence is greater than a confidence threshold, where the confidence threshold is a preset constant value; If yes, determining whether the key point center information matches the gesture box information; If there is a match, determining the gesture category information according to the key point center information that matches the gesture box information; If not, determining whether the distance between the estimated center position of the hand in the key point center information and the center position of the hand determined in the gesture three-dimensional position information is less than a distance threshold; If it is less than, the gesture category information is determined according to the category of the estimated hand center position whose distance from the determined hand center position is less than the distance threshold.
5. The gesture remote operation method according to claim 4, characterized in that: The robot's manipulator motion is controlled by the first target gesture information, including: The position information of the manipulator is obtained by calculating the three-dimensional position information of the gesture; The robot direction information is obtained by calculating the gesture direction information; The robot movement is controlled by second target gesture information, wherein the second target gesture information includes the gesture category information, the robot position information, the robot direction information and the gesture posture information.
6. The gesture remote operation method according to claim 3, characterized in that: Based on the depth image, determining the three-dimensional information of the key point by using the two-dimensional information of the key point includes: The three-dimensional information of the key points is calculated by the following formula: z i 、x i and i is the three-dimensional coordinate information of the i-th key point, that is, the three-dimensional information of the key point; d i is the depth value of the i-th key point, obtained from the depth image; s is the depth range of the depth image; u i and v i is the two-dimensional coordinate information of the i-th key point, that is, the two-dimensional information of the key point; cx, cy are the coordinate values of the center point of the depth image; fx is the focal length of the camera on the x-axis, and fy is the focal length of the camera on the y-axis.
7. The gesture remote operation method according to claim 3, characterized in that: Calculating the key point center information through the key point three-dimensional information includes: The key point center information is calculated by the following formula: X, Y, and Z are the estimated center coordinate information of the hand of the corresponding category; z i 、x i and i is the three-dimensional coordinate information of the i-th key point of the hand of the corresponding category; k is the total number of key points of the hand in the corresponding category.
8. A robot gesture teleoperation device, characterized in that: include: A collection unit, used to collect user's gesture image information, wherein the gesture image information is image information representing the user's hand movement; a processing unit, configured to obtain gesture box information, gesture three-dimensional position information, gesture confidence, and gesture posture information according to the gesture image information, wherein the gesture box information is information indicating the range of the user's hand, the gesture confidence indicates the probability that the gesture posture information is consistent with the user's finger posture, and the gesture posture information is information indicating the posture in a gesture database that has the highest probability of being consistent with the user's finger posture; an estimating unit, configured to estimate key point center information and gesture direction information from the gesture image information, wherein the key point center information is information indicating an estimated center position of a hand, and the gesture direction information is information indicating a direction of the user's hand; The processing unit is further used to obtain gesture category information according to the gesture box information, the gesture confidence and the key point center information, wherein the gesture category information indicates the left and right category of the user's hand; A control unit is used to control the manipulator action of the robot through first target gesture information, wherein the first target gesture information includes the gesture category information, the gesture three-dimensional position information, the gesture direction information and the gesture posture information.
9. A robot gesture teleoperation device, characterized in that: include: CPU, memory and input / output interface; The memory is a short-term storage memory or a persistent storage memory; The central processing unit is configured to communicate with the memory and execute instructions in the memory to perform the method according to any one of claims 1 to 7.
10. A computer-readable storage medium, characterized in that: The method comprises instructions, which, when executed on a computer, cause the computer to execute the method according to any one of claims 1 to 7.
