Method and device for controlling robot to move based on three-dimensional gestures

Abstract

The invention provides a method and a device for controlling a robot to move based on a three-dimensional gesture. The method for controlling the robot to move based on the three-dimensional gestures comprises the steps that the corresponding relation between the three-dimensional gestures and robot movement is preset; collecting three-dimensional data of the three-dimensional gesture through the mobile AR equipment, wherein the three-dimensional data is represented through three-dimensional coordinates of five finger joints and one wrist joint; performing matching processing on the three-dimensional data and the robot motion based on the corresponding relation; performing model training by taking the three-dimensional data as input and taking robot motion matched with the three-dimensional data as output to obtain a robot control model; the robot control model controls the robot to move based on the three-dimensional gestures of the user. According to the method for controlling the robot to move based on the three-dimensional gestures, the effect of large-range movement and even remote control can be achieved while the accuracy of robot control is improved.

Description

Technical Field

[0001] The present invention relates to the field of computer graphics technology, and in particular, to a method and device for controlling robot motion based on three-dimensional gestures. Background Art

[0002] Gesture recognition technology refers to the computer's ability to infer the information and user intentions transmitted through captured gestures. Classified from the perspective of gesture acquisition sources, gesture recognition methods can be divided into vision-based and sensor-based. Vision-based methods mainly include obtaining color (RGB) images, depth images, etc. of gestures from a monocular camera, a stereo camera, or a professional motion capture device such as Kinect or Leap Motion. Sensor-based methods involve sensor devices such as data gloves and accelerometers. Classified from the expression of gesture information, it can be divided into two-dimensional shapes and three-dimensional models. For example, RGB images can record the two-dimensional shape of gestures, while methods such as skeletons, meshes, and voxels can represent the three-dimensional structure of gestures. Designing a simple, intuitive, and natural robot control system in an unknown environment is an important topic in the field of human-computer interaction. Among the optional control signals, gesture signal control is considered an intuitive way that can well meet the task of users transmitting control information to the robot.

[0003] Robots can be simply classified into fixed robots and mobile robots. For example, robotic arms are usually fixed in an area to complete specified tasks, while wheeled robots can move over a large range to complete tasks such as transportation, transfer, and inspection. Traditional robot control mainly refers to research in dynamics. After the emergence of deep learning technology, neural networks have also been widely used in problems such as robot control optimization and simulation. From the perspective of interactive control methods, using gestures to control robots is an important branch topic. Gestures can intuitively convey the user's intentions and can achieve a natural and intuitive control method compared to mice, keyboards, and buttons.

[0004] Augmented reality is a technology that superimposes virtual objects created by a computer onto the real environment, enabling users to immerse themselves in a virtual-real fusion environment. People can use an air finger interaction method to freely manipulate virtual objects and obtain the most realistic feedback from the environment. Due to its close combination with real life, augmented reality technology is regarded as a new generation of stereoscopic display technology, truly realizing human-machine intelligent interaction.

[0005] There are some existing technical methods for controlling a robot based on gesture recognition. The general idea is to combine a vision-based recognition algorithm to convert gestures into control commands for the robot, thereby achieving the purpose of controlling the robot. From the perspective of the sensor device for collecting gestures, the first type is the method based on an RGB camera. The captured RGB images can be used to train a convolutional neural network for recognition, or the classical OpenCV library algorithm can be called for preprocessing and then recognition. However, since the RGB images obtained by taking the same gesture from different angles may be different, and coupled with the complex real environment, the accuracy of this type of method is very limited. The second type of technical solution is based on a depth camera such as Kinect. The captured depth images can also be used to train a neural network for gesture recognition, and there is also a more traditional solution, that is, the postures of standard gestures are predefined in advance, and then thresholds are set to judge the accuracy of the gestures. The main problem with this type of method is caused by the fixed sensor, resulting in limited movement ranges for the user and the robot. Summary of the Invention

[0006] In view of this, the present invention provides a method and device for controlling a robot's movement based on three-dimensional gestures to solve the above problems.

[0007] In the first aspect of the present invention, a method for controlling a robot's movement based on three-dimensional gestures is provided, including: presetting the correspondence between three-dimensional gestures and the robot's movement; collecting three-dimensional data of the three-dimensional gestures through a mobile AR device, where the three-dimensional data is represented by the three-dimensional coordinates of 5 finger joints and 1 wrist joint; based on the correspondence, performing matching processing on each piece of three-dimensional data and the robot's movement; using each piece of three-dimensional data as input and the robot's movement matched with each piece of three-dimensional data as output for model training to obtain a robot control model; and the robot control model controlling the robot's movement based on the user's three-dimensional gestures.

[0008] In another implementation manner of the present invention, the robot's movement includes 5 basic actions: forward, backward, right turn, left turn, and stop; combining the basic actions to obtain complex action sequences; presetting the correspondence between three-dimensional gestures and the robot's movement, including: presetting the correspondence between three-dimensional gestures and basic actions based on the different directions of the fingers and the changes in the number of fingers, where different three-dimensional gestures correspond to different basic actions.

[0009] In another implementation manner of the present invention, the method for controlling a robot's movement based on three-dimensional gestures further includes: the mobile AR device preprocessing the three-dimensional data using a floating calibration strategy; where the floating calibration strategy is to set a virtual coordinate system above the three-dimensional gesture so that the three-dimensional data is represented in the same coordinate system.

[0010] In another implementation of the present invention, the method for controlling robot movement based on three-dimensional gestures further includes: preprocessing the three-dimensional data through an anchor alignment strategy, where the anchor alignment strategy is to align the three-dimensional gestures to the same characteristic joint through translational transformation, making the characteristics of the three-dimensional gestures easier to identify in model training.

[0011] In a second aspect of the present invention, there is provided a device for controlling robot movement based on three-dimensional gestures, including: a data acquisition module: used to preset the correspondence between three-dimensional gestures and robot movement; collect three-dimensional data of three-dimensional gestures through a mobile AR device, and the three-dimensional data is represented by the three-dimensional coordinates of 5 finger joints and 1 wrist joint; perform matching processing on each three-dimensional data and robot movement based on the correspondence; a model training module: used to take each three-dimensional data as input and the robot movement matched with each three-dimensional data as output for model training to obtain a robot control model; a motion control module: used to control the robot movement based on the user's three-dimensional gestures by the robot control model.

[0012] The method for controlling robot movement based on three-dimensional gestures of the present invention uses a mobile AR device as a sensor and an interaction interface, and at the same time adopts a gesture control scheme represented only by 6 finger joints. Compared with the traditional two-dimensional gesture strategy, it can provide depth information to improve the recognition accuracy; compared with a fixed sensor device, it can achieve the effect of large-range movement or even remote control; moreover, the three-dimensional gesture representation, combined with the floating calibration and anchor alignment strategies, reduces the computational amount while supporting real-time and accurate control of the robot. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] In order to more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following will briefly introduce the drawings required for use in the description of the embodiments or the prior art. By reading the detailed description of the following embodiments, the advantages and benefits in the solutions will become clear to those skilled in the art. The drawings are only used for the purpose of showing the preferred embodiments and are not considered to be a limitation of the present invention. In the drawings:

[0014] Figure 1 It is a schematic flowchart of the steps of the method for controlling robot movement based on three-dimensional gestures according to an embodiment of the present invention.

[0015] Figure 2 It is a schematic technical flowchart of the method for controlling robot movement based on three-dimensional gestures according to an embodiment of the present invention.

[0016] Figure 3 It is a schematic diagram of the effect of the method for controlling robot movement based on three-dimensional gestures according to an embodiment of the present invention.

[0017] Figure 4A three - dimensional gesture schematic diagram of an embodiment of the present invention. Detailed implementation manners

[0018] In order to enable those skilled in the art to better understand the technical solutions in the embodiments of the present invention, the following will clearly and detailedly describe the technical solutions in the embodiments of the present invention in combination with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all of the embodiments. Based on the embodiments in the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art shall fall within the protection scope of the embodiments of the present invention.

[0019] Figure 1 A step - flow diagram of a method for controlling a robot's movement based on three - dimensional gestures provided by an embodiment of the present invention, as Figure 1 shown, this embodiment mainly includes the following steps:

[0020] S101. Preset the correspondence between three - dimensional gestures and the movement of the robot.

[0021] S102. Collect three - dimensional data of the three - dimensional gesture through a mobile AR device. The three - dimensional data is represented by the three - dimensional coordinates of 5 finger joints and 1 wrist joint.

[0022] Exemplarily, after the movement relationship between the three - dimensional gesture and the robot is determined, a neural network is built by collecting a data set to solve a function that maps the user's gesture to the robot's movement. The mobile AR device can use a mixed - reality head - mounted display (HoloLens). In the AR environment created by HoloLens, the user makes a standard gesture corresponding to the robot's movement according to the previous gesture design. HoloLens's own depth sensor can obtain and track the three - dimensional model of the hand, sampling the positions of 5 finger joints and 1 wrist joint, a total of 6 joints, to represent a gesture in a lightweight manner. That is, the input of each piece of data in the data set is a tuple formed by splicing the three - dimensional coordinates of 6 joints, and the output is a string representing the robot's movement.

[0023] S103. Based on the correspondence, perform matching processing on each three - dimensional data and the robot's movement.

[0024] S104. Use each three - dimensional data as the input and the robot's movement matched with each three - dimensional data as the output for model training to obtain a robot control model.

[0025] Exemplarily, three datasets were collected for model training, which were used as Training Set 1, Test Set 1, and Test Set 2 respectively. Each dataset contains 175 records, that is, each gesture has 35 records. The neural network architecture adopted is a Multiplayer Perceptron (MLP) model. Specifically, the MLP model includes 7 non-linear activation (ReLu) layers and 1 output (Softmax) layer. Among them, the 7 ReLu layers are all 256-dimensional, and the Softmax layer is 5-dimensional, corresponding to 5 robot motions. The loss function used during training is Cross-entropy Loss.

[0026] S105. The robot control model controls the robot motion based on the user's three-dimensional gestures.

[0027] Exemplarily, as Figure 2 shown, a computer is configured for the robot as a node for data transceiver and processing. At the same time, a local area network is built to connect the HoloLens with the robot, and the trained neural network model is deployed on the computer. As Figure 3 shown, when the user makes a gesture under the HoloLens, the corresponding coordinates will be sent from the HoloLens to the computer. After the neural network makes a real-time prediction and obtains the control command for the robot, it is then sent to the robot's operating system to direct the robot to move.

[0028] The method for controlling a robot motion based on three-dimensional gestures of the present invention uses a mobile AR device as a sensor and an interaction interface, and at the same time adopts a gesture control scheme represented by only 6 finger joints. Compared with the traditional two-dimensional gesture strategy, it can provide depth information to improve the recognition accuracy; compared with a fixed sensor device, it can achieve the effect of large-range movement or even remote control. Moreover, the three-dimensional gesture representation, combined with the floating calibration and anchor point alignment strategy, can reduce the computational complexity while supporting real-time and accurate control of the robot.

[0029] In another implementation manner of the present invention, the robot motion includes 5 basic actions: forward, backward, right turn, left turn, and stop; complex action sequences are obtained by combining the basic actions; the corresponding relationship between the preset three-dimensional gestures and the robot motion is included, including: the corresponding relationship between the preset three-dimensional gestures and the basic actions based on the different directions and the changes in the number of fingers. Among them, different three-dimensional gestures correspond to different basic actions.

[0030] Exemplarily, to determine the correspondence between three-dimensional gestures and robot movements, for a mobile robot, its movements are decomposed into five actions: forward, backward, right turn, left turn, and stop. These five actions are regarded as primitive actions, that is, based on these five basic actions, more complex action sequences can be combined to complete specific tasks in a real environment. For the design of gestures, the principle that the direction of the fingers indicates the trend of movement is followed, and at the same time, the change in the number of fingers is combined to make each type of finger more characteristic. Specifically, as Figure 4 shown, all five fingers pointing forward correspond to the forward movement of the robot, four fingers pointing backward correspond to the backward movement of the robot, three fingers pointing to the right correspond to the right turn of the robot, two fingers pointing to the left correspond to the left turn of the robot, and all fingers curled up (fist posture) represent the stop action.

[0031] The present invention proposes a gesture control scheme represented by only six finger joints, which achieves lightweight in computational processing, allows real-time control, and at the same time uses three-dimensional gesture information as a control signal, which can eliminate ambiguity, can effectively cope with complex real environments, and improve the recognition accuracy.

[0032] In another implementation manner of the present invention, the method for controlling robot movement based on three-dimensional gestures further includes: the mobile AR device preprocesses the three-dimensional data of the three-dimensional gestures by adopting a floating calibration strategy; wherein, the floating calibration strategy is to set a virtual coordinate system above the three-dimensional gesture so that the three-dimensional data is represented in the same coordinate system.

[0033] Exemplarily, in the network training stage, a data correction module is adopted to improve the accuracy of network prediction. First, every time HoloLens is started, the origin of the world coordinate is initialized, which results in the coordinate data collected at different times and locations not being in the same coordinate system, so the corresponding gesture meanings are incorrect. Therefore, the floating calibration strategy is adopted to set a virtual coordinate system floating above the hand, so that all gesture data can be represented in the same coordinate system, ensuring the consistency of data distribution.

[0034] Aiming at the problem of limited movement space existing when using a fixed sensor as a signal acquisition device to control a robot, the present invention uses the head-mounted AR device HoloLens2 as a sensor to collect three-dimensional gesture data to control the robot, supports large-range movement and user remote control of the robot, without being limited to a certain area, and realizes improving the accuracy of controlling the movement of the robot in a complex real environment.

[0035] In another implementation of the present invention, the method for controlling a robot's movement based on three-dimensional gestures further includes: preprocessing the three-dimensional data through an anchor alignment strategy; wherein, the anchor alignment strategy is to align the three-dimensional gestures to the same characteristic joint through translational changes, making the features of the three-dimensional gestures easier to recognize during model training.

[0036] Exemplarily, since the three-dimensional gesture is a lightweight representation of 6 joints, the coincidence and stacking of different joints result in a linearly inseparable gesture space, and it is difficult for network training to fit an ideal hyperplane. Therefore, an anchor alignment preprocessing strategy is adopted. According to the previous gesture design, the little finger with the largest movement amplitude is used as the anchor to recalibrate all gestures, which can make the original data space more separated. Thus, through the floating calibration and anchor alignment strategy, the prediction ability of the neural network is significantly improved, and the accuracy of gesture recognition is guaranteed. Through the data alignment and correction strategy of finger joints, the gesture recognition has a high accuracy rate.

[0037] In a second aspect of the present invention, there is provided a device for controlling a robot's movement based on three-dimensional gestures, including:

[0038] A data acquisition module: used to preset the correspondence between three-dimensional gestures and the robot's movement; collect the three-dimensional data of the three-dimensional gestures through a mobile AR device, and the three-dimensional data is represented by the three-dimensional coordinates of 5 finger joints and 1 wrist joint; based on the correspondence, perform matching processing on each three-dimensional data and the robot's movement.

[0039] A model training module: used to take each three-dimensional data as input and the robot's movement matched with each three-dimensional data as output for model training to obtain a robot control model.

[0040] A motion control module: used to control the robot's movement based on the user's three-dimensional gestures by the robot control model.

[0041] The device for controlling a robot's movement based on three-dimensional gestures of the present invention uses a mobile AR device as a sensor and an interaction interface, and at the same time adopts a gesture control scheme represented by only 6 finger joints. Compared with the traditional two-dimensional gesture strategy, it can provide depth information to improve the recognition accuracy; compared with a fixed sensor device, it can achieve the effect of large-range movement or even remote control; moreover, the three-dimensional gesture representation, combined with the floating calibration and anchor alignment strategy, reduces the computational amount while enabling real-time and accurate control of the robot.

[0042] Thus far, specific embodiments of the present invention have been described. Other embodiments are within the scope of the appended claims. In some cases, the acts recited in the claims may be performed in a different order and still achieve the desired result. Additionally, the processes depicted in the figures do not necessarily require the particular order or sequential order shown to achieve the desired result. In certain embodiments, multitasking and parallel processing may be advantageous.

[0043] It should be noted that all directional indications (such as up, down, left, right, rear, etc.) in the embodiments of the present invention are only used to explain the relative positional relationship and movement conditions between components in a specific posture (as shown in the drawings). If the specific posture changes, the directional indications will change accordingly.

[0044] In the description of the present invention, the terms "first" and "second" are only used for convenience in describing different components or names, and cannot be construed as indicating or implying an order relationship, relative importance, or implicitly indicating the quantity of the indicated technical features. Thus, the features defined with "first" and "second" may explicitly or implicitly include at least one of such features.

[0045] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by those of ordinary skill in the technical field to which the present invention pertains. The terms used in the description of the present invention herein are only for the purpose of describing specific embodiments and are not intended to limit the present invention.

[0046] It should be noted that although the specific embodiments of the present invention have been described in detail in conjunction with the accompanying drawings, it should not be construed as a limitation on the protection scope of the present invention. Within the scope described in the claims, various modifications and variations that can be made by those of ordinary skill in the art without creative efforts still fall within the protection scope of the present invention.

[0047] The examples of the embodiments of the present invention are intended to briefly illustrate the technical features of the embodiments of the present invention, so that those of ordinary skill in the art can intuitively understand the technical features of the embodiments of the present invention, and are not an improper limitation on the embodiments of the present invention.

[0048] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that they can still modify the technical solutions described in the foregoing embodiments, or perform equivalent replacements for some of the technical features; and these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for controlling the movement of a robot based on three-dimensional gestures, characterized in that, Comprising: The corresponding relationship between preset three-dimensional gestures and robot movements; Collecting three-dimensional data of the three-dimensional gestures through a mobile AR device, where the three-dimensional data is represented by the three-dimensional coordinates of 5 finger joints and 1 wrist joint; Based on the corresponding relationship, performing matching processing on each three-dimensional data and robot movement; Using each three-dimensional data as input and the robot movement matched with each three-dimensional data as output for model training to obtain a robot control model; The robot control model controls the robot movement based on the user's three-dimensional gesture; 2. The method according to claim 1, characterized in that, The robot movement includes 5 basic actions: forward, backward, right turn, left turn, and stop; Combining the basic actions to obtain complex action sequences; The corresponding relationship between the preset three-dimensional gestures and robot movements includes: Presetting the corresponding relationship between three-dimensional gestures and basic actions based on the different directions of fingers and the changes in the number of fingers, where different three-dimensional gestures correspond to different basic actions.

3. The method according to claim 1, characterized in that, Further comprising: The mobile AR device preprocesses the three-dimensional data using a floating calibration strategy; Wherein, the floating calibration strategy is to set a virtual coordinate system above the three-dimensional gesture to represent the three-dimensional data in the same coordinate system.

4. The method according to claim 3, characterized in that, Further comprising: Preprocessing the three-dimensional data through an anchor point alignment strategy; Wherein, the anchor point alignment strategy is to align the three-dimensional gesture to the same characteristic joint through translational transformation, making the characteristics of the three-dimensional gesture easier to identify in model training.

5. A device for controlling the movement of a robot based on three-dimensional gestures, characterized in that, Comprising: Data acquisition module: used to preset the corresponding relationship between three-dimensional gestures and robot movements; Collecting three-dimensional data of the three-dimensional gestures through a mobile AR device, where the three-dimensional data is represented by the three-dimensional coordinates of 5 finger joints and 1 wrist joint; Based on the corresponding relationship, performing matching processing on each three-dimensional data and robot movement; Model training module: used to use each three-dimensional data as input and the robot movement matched with each three-dimensional data as output for model training to obtain a robot control model; Motion control module: used for the robot control model to control the robot movement based on the user's three-dimensional gesture.

Images