Method, device and electronic equipment for controlling robot

Abstract

The present invention provides a method, device, and electronic device for controlling a robot, wherein the method comprises: obtaining current gesture data of a target user, the current gesture data including the coordinates of multiple key points of the hand; determining the gesture type of the current gesture data based on the coordinates of the multiple key points in the current gesture data; when the gesture type of the current gesture data is a motion gesture, determining the degree of change of the current gesture data and generating a continuous control instruction for controlling the motion of the robot; the type of the continuous control instruction is consistent with the gesture type of the current gesture data, and there is a positive correlation between the control amount of the continuous control instruction and the degree of change of the current gesture data. Through the technical solution provided by the embodiment of the present invention, a gain mapping relationship is established between the degree of change of the motion gesture and the change of the physical quantity of the robot, so that the target user can continuously, freely, flexibly, and efficiently control the robot in front of the robot.

Description

Technical Field

[0001] The present invention relates to the field of human-computer interaction technology, and in particular to a method, device, electronic device and computer-readable storage medium for controlling a robot. Background Art

[0002] With the increasing popularity of digital cameras and the growing power of mobile phone cameras, photography has become a vital way for everyone to record and share their lives, and a crucial component of daily life. To facilitate remote selfies, robot-assisted photography, such as drone tracking, is gaining popularity. Because robots lack aesthetic appreciation or compositional skills, users must control the robot's position and camera parameters to achieve these remote selfies.

[0003] Remote control of assistive photography robots currently requires external devices, such as remote controls and applications on smart devices. Some solutions offer voice control, but this approach is inefficient for precise adjustments to camera angle and position, and can easily lead to inaccurate or even incorrect adjustments. Summary of the Invention

[0004] To solve existing technical problems, embodiments of the present invention provide a method, device, electronic device, and computer-readable storage medium for controlling a robot.

[0005] In a first aspect, an embodiment of the present invention provides a method for controlling a robot, comprising:

[0006] Acquire current gesture data of the target user, where the current gesture data includes coordinates of multiple key points of the hand;

[0007] determining a gesture type of the current gesture data according to the coordinates of the plurality of key points in the current gesture data;

[0008] When the gesture type of the current gesture data is a motion gesture, the degree of change of the current gesture data is determined, and a continuous control instruction for controlling the movement of the robot is generated; the type of the continuous control instruction is consistent with the gesture type of the current gesture data, and there is a positive correlation between the control amount of the continuous control instruction and the degree of change of the current gesture data.

[0009] In a second aspect, an embodiment of the present invention further provides a device for controlling a robot, comprising:

[0010] An acquisition module, configured to acquire current gesture data of a target user, wherein the current gesture data includes coordinates of multiple key points of a hand;

[0011] a type determination module, configured to determine a gesture type of the current gesture data according to coordinates of a plurality of key points in the current gesture data;

[0012] A processing module is used to determine the degree of change of the current gesture data and generate continuous control instructions for controlling the movement of the robot when the gesture type of the current gesture data is a motion gesture; the type of the continuous control instruction is consistent with the gesture type of the current gesture data, and there is a positive correlation between the control amount of the continuous control instruction and the degree of change of the current gesture data.

[0013] In a third aspect, an embodiment of the present invention provides an electronic device comprising a bus, a transceiver, a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the transceiver, the memory, and the processor are connected via the bus, and when the computer program is executed by the processor, the steps in any one of the above-described methods for controlling a robot are implemented.

[0014] In a fourth aspect, an embodiment of the present invention further provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the steps in any one of the above-mentioned methods for controlling a robot.

[0015] The robot control method, apparatus, electronic device, and computer-readable storage medium provided by embodiments of the present invention pre-set motion gestures. By executing these motion gestures, a target user can input corresponding continuous control instructions, thereby controlling the robot's movement. The degree of change in the motion gesture can be determined using current gesture data, and the control amount of the continuous control instruction is positively correlated with this degree of change. This establishes a gain mapping relationship between the degree of change in the motion gesture and the change in the robot's physical quantity. For example, a mapping relationship can be established between the movement speed or rotation speed of the motion gesture and the movement speed or rotation speed of the robot. This allows the target user to control the robot continuously, freely, flexibly, and efficiently while in front of the robot. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] In order to more clearly illustrate the technical solutions in the embodiments of the present invention or the background technology, the drawings required for use in the embodiments of the present invention or the background technology will be described below.

[0017] Figure 1 A schematic diagram showing a process of assisting shooting scenes provided by an embodiment of the present invention is shown;

[0018] Figure 2 A schematic diagram showing the problems existing in traditional control methods;

[0019] Figure 3A flowchart of a method for controlling a robot provided by an embodiment of the present invention is shown;

[0020] Figure 4 A schematic diagram of hand joints provided by an embodiment of the present invention is shown;

[0021] Figure 5 Shows the control system structure diagram of the auxiliary photography scene;

[0022] Figure 6 A schematic diagram showing a user gesture in the method for controlling a robot provided by an embodiment of the present invention;

[0023] Figure 7 A detailed schematic diagram of a user's gesture in the method for controlling a robot provided by an embodiment of the present invention is shown;

[0024] Figure 8 A schematic diagram illustrating establishing a relative coordinate system in a method for controlling a robot provided by an embodiment of the present invention is shown;

[0025] Figure 9 A schematic structural diagram of a device for controlling a robot provided by an embodiment of the present invention is shown;

[0026] Figure 10 A schematic structural diagram of an electronic device for executing a method for controlling a robot provided by an embodiment of the present invention is shown. DETAILED DESCRIPTION

[0027] In the auxiliary photography scenario, it is generally necessary for the user to trigger the command multiple times to take a photo. In the embodiment of the present invention, the user's behavior of completing the photo taking is divided into three stages, following a process of coarse adjustment, fine adjustment, and finally confirming the shot. Figure 1 The user's behavioral characteristics in these three stages are demonstrated. The first stage is the rough shooting picture: the user makes the robot move in a large way to reach the vicinity of the target point according to the conception of the picture in his mind. The second stage is the Control-Feedback Loop: the user uses the real-time feedback of the camera picture and his own brain's aesthetic prejudgment to guide the robot to make fine adjustments. This cycle is repeated several times until the fine-tuning effect is satisfactory, and then the shooting is carried out. This is also similar to the real-life photography process. When real-life photographers and models take pictures, they often make rough adjustments to determine the general content and structure of the picture first, and then make fine adjustments. The third stage is the completion of the shooting: the user confirms that the real-time camera picture meets his requirements, issues instructions such as taking pictures and recording videos, and completes the photo interaction.

[0028] The first and third stages can be implemented by matching gesture commands with command sets, or voice commands with command sets, and are not difficult. However, the difficulty of user-robot interaction in assisted photography scenarios lies in the second stage of image fine-tuning. Assisting users in fine-tuning the image is a key issue in assisted photography and also a key difficulty in interaction in this scenario.

[0029] Currently, there are voice-controlled robots. If this method is used for the second stage of fine-tuning, the user must issue voice commands to the camera robot. However, the inventors discovered that when the camera robot's commands involve directions, such as "a little to the left," the user must independently distinguish between left and right. This can lead to slight hesitation when using commands involving left and right, and even the misinterpretation of left and right. Furthermore, voice commands like "a little to the left" are difficult for the camera robot to accurately respond to, as the robot cannot accurately determine the exact "a little" the user expresses.

[0030] Although users can issue accurate voice commands, such as "1 meter forward" or "20 cm back", the user being filmed cannot directly obtain a clear numerical value of how many meters the robot should move or how many degrees it should rotate, making it difficult to achieve accurate control. For details, see Figure 2 There are two main reasons for this. First, the person being photographed (i.e., the user) does not know the exact amount of movement required to achieve the desired effect and can only make judgments based on the feedback from the camera image on the screen. Second, humans cannot accurately measure specific values ​​like robots and can only make estimates based on past experience.

[0031] This voice control method is difficult to achieve the desired effect through one or a few fine-tuning. Generally, users are required to input voice commands intermittently and multiple times to control the movement of the robot, which is time-consuming and labor-intensive, and it is difficult to accurately achieve the desired effect.

[0032] In an embodiment of the present invention, the user inputs control instructions through gestures. The characteristic of gestures that can express directions intuitively and quickly can avoid the user distinguishing between left and right. Gestures can also intuitively, quickly and continuously express the control quantity that needs to be controlled (such as the size of the displacement distance, the size of the rotation angle, etc.). The use of gestures that express continuously can achieve efficient control.

[0033] The embodiments of the present invention are described below with reference to the accompanying drawings.

[0034] An embodiment of the present invention provides a method for controlling a robot. The method can be executed by the controlled robot or by other devices, such as a cloud server, a user's smart terminal, etc. Figure 3FIG. 1 is a flow chart showing a method for controlling a robot provided by an embodiment of the present invention. Figure 3 As shown, the method includes:

[0035] Step 301: Acquire current gesture data of the target user, where the current gesture data includes coordinates of multiple key points of the hand.

[0036] In an embodiment of the present invention, when a user needs to control a robot using gestures, that user is designated as the target user, and the coordinates of multiple key points on the target user's hand are collected to generate corresponding gesture data. Since gesture data can be collected at different times, this embodiment refers to the currently collected gesture data as current gesture data. For example, a robot equipped with a depth camera can collect the coordinates of multiple key points on the target user's hand to generate gesture data. The target user is defined as the user within the depth camera's field of view. Specifically, the robot can be a robot with auxiliary photography capabilities, or a drone, for example.

[0037] Step 302: Determine the gesture type of the current gesture data according to the coordinates of multiple key points in the current gesture data.

[0038] In an embodiment of the present invention, a user is allowed to control a robot based on at least one type of gesture, that is, a gesture corresponds to a corresponding gesture type. In an embodiment of the present invention, the gesture type is a type classified based on the control function that the gesture can play, and different gesture types implement different control functions. For example, the gesture type includes at least one of a move gesture, a rotation gesture, a zoom gesture, and a stop gesture. The move gesture is used to control the movement of the robot, the rotation gesture is used to control the rotation of the robot, the zoom gesture is used to control the size of the robot's captured image (when the robot includes a camera device), and the stop gesture is used to pause the control of the robot.

[0039] Among them, in different gesture types, the coordinates of multiple key points have different characteristics. In order to be able to distinguish multiple gesture types more accurately, multiple key points with different relative postures under different gesture types can be selected. In an embodiment of the present invention, different gesture types can be distinguished by the different postures of the fingers. For example, the multiple key points involved in the embodiment of the present invention include at least key points with fixed postures under different gesture types (such as palm key points, wrist key points, etc.) and key points on at least one finger (such as fingertip key points).

[0040] For example, the key points required in this embodiment can be selected from the joints of the hand. Figure 4As shown in FIG, the hand contains a total of 24 joints, including wrist joints (Wrist), palm joints (Palm) and joints on the five fingers; among them, the thumb (Thumb) involves 5 joints from the thumb root (Thumb0) to the thumb tip (ThumbTip), the index finger (Index) involves 4 joints from the index finger root (Index1) to the index finger tip (IndexTip), the middle finger (Middle) involves 4 joints from the middle finger root (Middle1) to the middle finger tip (MiddleTip), the ring finger (Ring) involves 4 joints from the ring finger root (Ring1) to the ring finger tip (RingTip), and the pinky finger (Pinky) involves 5 joints from the pinky root (Pinky0) to the pinky tip (PinkyTip). The joints numbered 0 or 1 can be used as the base joints, and the joints numbered Tip can be used as the fingertip joints. For example, the base joint of the thumb can be Thumb0 or Thumb1, and the interphalangeal joint of the thumb can be ThumbTip. Each joint can be used as a corresponding key point.

[0041] The gesture data can be represented by the coordinates of multiple key points. For example, the coordinates of key point i are s i , and s i =(x i ,y i ,z i ), i = 1, 2, ..., n, n is the number of key points, for example, n = 24. The gesture data can be represented as a set of n key point coordinates, that is, the gesture data S satisfies: S = {s1, s2, ..., s n When a user performs a certain type of gesture, the current gesture data collected meets the characteristics of that type of gesture, so that the corresponding gesture type can be determined based on the current gesture data, and then the type of gesture currently being performed by the target user can be determined.

[0042] Step 303: When the gesture type of the current gesture data is a motion gesture, determine the degree of change of the current gesture data and generate a continuous control instruction for controlling the movement of the robot; the type of the continuous control instruction is consistent with the gesture type of the current gesture data, and there is a positive correlation between the control amount of the continuous control instruction and the degree of change of the current gesture data.

[0043] In this embodiment of the present invention, gesture types can be divided into at least motion gestures and stop gestures. A motion gesture refers to a gesture used by a user to control the movement of a robot. Correspondingly, a stop gesture refers to a gesture used by a user to control the stationary state of a robot. If the gesture type of the current gesture data is a motion gesture, a continuous control instruction for controlling the movement of the robot is generated. This continuous control instruction is used to achieve continuous control of the robot.

[0044] Continuous control instructions include at least two attributes: type and control amount. In this embodiment, the type of continuous control instructions is consistent with the gesture type, meaning each gesture type corresponds to a type of continuous control instruction. For example, a motion gesture can be a movement gesture or a rotation gesture. Accordingly, a continuous control instruction is an instruction to continuously control the robot's movement or rotation. By continuously controlling the robot, users can improve interaction efficiency.

[0045] Specifically, from the perspective of the control system, the human-robot interaction in the auxiliary photography scene can be summarized as a double-feedback closed-loop control system. The control block diagram of the control system is as follows: Figure 5 As shown. Assume that the initial target position of the robot is The target position of the robot at time i is The user's estimated position of the robot at time i is The actual position of the robot at time i is X i .

[0046] The inner feedback closed-loop control loop is where the user controls the robot's actual position based on the robot's target position. At time i, the difference that the controller wants to control can be expressed as follows:

[0047]

[0048] The actual position of the robot at time i is obtained by the controller G(X) as X i , X i The estimated position of the robot at time i is obtained by user measurement H(X) An inner feedback closed-loop control loop is formed.

[0049] The external feedback closed-loop control loop allows the user to continuously adjust the robot's target position based on the real-time camera image. The output of the inner feedback closed-loop control loop is X i , the final image output is obtained through the camera image presentation C(x), and the target position of the robot at time i+1 is determined through the user's subjective aesthetic judgment S(x) That is to say, the given quantity of the inner feedback closed-loop control loop is determined by the outer feedback closed-loop control loop.

[0050] It can be seen that the target position of the robot in the auxiliary photography scene is different from the control of the robot position in other scenes. The spatial manipulation of the photography robot must be continuous to allow for timely changes in control of the robot based on changes in its target position. The embodiments of the present invention utilize gesture-based continuous control commands to continuously control a robot (e.g., a photography robot), thereby improving control efficiency.

[0051] Furthermore, the current gesture data is based on the coordinates of multiple key points collected when the target user triggers a motion gesture. Since motion gestures are dynamic, the current gesture data also has a degree of change, such as the movement speed or rotation speed corresponding to the current gesture data, or the coordinate change between the current gesture data and previous historical gesture data. In this embodiment, the control amount of the continuous control instruction is determined by the degree of change of the current gesture data, and there is a positive correlation between the control amount and the degree of change, that is, the greater the degree of change of the current gesture data, the greater the control amount of the generated continuous control instruction. For example, if the gesture type of the current gesture data is a moving gesture and its degree of change is a moving speed, a control instruction for controlling the robot's moving speed can be generated; and, by using gesture data at multiple consecutive moments, the robot's moving speed can be continuously controlled within these multiple consecutive moments.

[0052] Currently, some solutions utilize the duration of speech to achieve continuous control; alternatively, they design a variety of static gestures, using the duration of these gestures to achieve continuous control of the robot. For example, a user's open palm facing forward controls the robot's forward movement. The duration of this open palm gesture determines the robot's forward movement duration, with the robot's movement speed typically being fixed. In assisted photography scenarios, the second stage involves refined fine-tuning control. For precise spatial positioning, the robot's movement speed is typically set moderately to avoid missing the target point.

[0053] At its core, the underlying logic of this method of repeating the same gesture for robot navigation is to trade time for space, controlling the robot's movement distance by the duration of the gesture. This inevitably leads to low efficiency. This is similar to the method of controlling the robot's movement distance by the duration of the voice or the interval between commands, both of which are linear relationships. Specifically, let the robot's distance to the target position be D, then the time MT required to control the robot to reach the target position is ref As follows:

[0054] MT ref =a ref +b ref ·D (2)

[0055] where b ref is the inverse of the robot's moving speed, a ref is a constant. However, this linear efficiency is not efficient for interaction. Furthermore, this control method is not flexible enough. Because the robot's transformation speed is constant, it cannot be adjusted at any speed like directly controlling a camera or tripod.

[0056] According to Fitts's Law, the difficulty index ID of a task is related to the distance D between the current robot and the target position, as well as the target size S, as shown in the following formula:

[0057]

[0058] The time MT to reach the target location can be expressed as follows:

[0059] MT=a+b·ID (4)

[0060] Taking the current gesture as a moving gesture as an example, let the time for the moving gesture to control the robot to reach the target position be MT TF , which should be the sum of the time overhead for preparation, triggering, swiping, de-triggering, and retraction. The time for preparation, triggering, de-triggering, and retraction is related to the system response and the user itself and can be considered constant. The swiping time can be considered as the time it takes to move the robot to a target using a hand, which conforms to Fitts's law in one-dimensional space. Therefore, the time MT required to control the robot to reach the target position using a mobile gesture is TF It can be expressed as the following formula:

[0061]

[0062] Among them, a TF 、b TF is a constant, D TF The distance the robot's hand needs to move to reach the target position, for example, the degree of change in the movement gesture; S TF The width of the target area in the user's mind.

[0063] It can be seen that the time MT to reach the target position using the mobile gesture TF The distance D that the hand needs to move when the robot reaches the target position TF The logarithmic relationship is better than the linear relationship and can achieve higher control efficiency.

[0064] Optionally, the method further includes generating a stop instruction for pausing control of the robot when the gesture type of the current gesture data is a stop gesture. By providing a stop gesture, the embodiment of the present invention allows the user to adjust the position of the hand by triggering the stop gesture after triggering a motion gesture, and then continue to execute the motion gesture, thereby providing the user with more operating space.

[0065] When the target user triggers a motion gesture, the robot enters a controlled state, meaning the target user can control the robot's position, angle, and other aspects based on the motion gesture. When the target user triggers a stop gesture, the robot enters an idle state, meaning the robot does not respond to the target user's gestures. The target user can switch between the controlled and idle states by triggering motion and stop gestures.

[0066] An embodiment of the present invention provides a method for controlling a robot. Preset motion gestures are used by a target user to input corresponding continuous control instructions by executing the motion gestures, thereby controlling the robot's movement. The degree of change in the motion gestures can be determined using current gesture data, and the control amount of the continuous control instructions is positively correlated based on the degree of change. This establishes a gain mapping relationship between the degree of change in the motion gestures and changes in physical quantities of the robot. For example, a mapping relationship is established between the movement speed or rotational speed of the motion gestures and the movement speed or rotational speed of the robot. This allows the target user to control the robot continuously, freely, flexibly, and efficiently while in front of the robot.

[0067] Optionally, in an embodiment of the present invention, the gesture type includes at least one of a move gesture, a rotation gesture, a zoom gesture, and a stop gesture.

[0068] Among them, the move gesture is a gesture that moves along the palm when multiple fingers are spread out; the rotation gesture is a gesture that keeps the target finger in position unchanged and rotates around the target finger when the target finger is spread out; the zoom gesture is a gesture that moves along the palm when multiple fingers are spread out, and the zoom gesture and move gesture are gestures triggered by the left and right hands respectively; the stop gesture is a gesture with all fingers folded.

[0069] In the auxiliary photography scenario, the target user mainly needs to control the movement of the robot and the orientation of the robot (especially the camera), so it is necessary to set movement gestures and rotation gestures; in addition, zoom gestures can also be set to be able to zoom in and out of the image size captured by the robot. Among them, movement gestures, rotation gestures, and zoom gestures are all motion gestures. In order to facilitate the target user to intuitively determine the control direction, such as determining the movement direction, rotation direction, etc., this embodiment uses the movement direction corresponding to the palm facing as the movement direction of the movement gesture, and uses the rotation direction when rotating along the target finger as the rotation direction of the rotation gesture.

[0070] Specifically, see Figure 6 As shown, Figure 6 The left picture in the figure shows a moving gesture, in which multiple fingers are stretched out (for example, five fingers are stretched out) and naturally closed; after the user triggers the moving gesture, the palm is facing the direction (i.e. Figure 6 The robot can be controlled to move by moving along the palm direction (such as Figure 6 Move your palm to the left), or move it against the direction of your palm (e.g. Figure 6 (Move the palm to the right in the middle), which is not limited in this embodiment.

[0071] Figure 6 The middle figure in the figure shows a rotation gesture, that is, a finger is selected as the target finger, the target finger is stretched out, and the other fingers except the target finger are retracted; the entire palm is rotated with the target finger as the axis, and the position of the target finger remains basically unchanged during the rotation; generally, the target finger can be the index finger or the thumb. Figure 6 In the figure, the index finger is taken as the target finger as an example.

[0072] Figure 6 The right picture in the figure shows a stop gesture, with all five fingers clenched in the palm of the hand, a fist-like gesture.

[0073] In addition, the left and right hands can also represent different gestures. In the embodiments of the present invention, the zoom gesture is similar to the move gesture. Both are gestures that move along the palm with multiple fingers spread out. The difference is that they are triggered by different palms. For example, moving the dominant hand (e.g., the right hand) triggers the move gesture, while moving the non-dominant hand (e.g., the left hand) triggers the zoom gesture.

[0074] For example, a gesture set provided by an embodiment of the present invention can be found in Figure 7 As shown, Figure 7In the example, the direction between the target user and the robot is the y-axis direction. For the moving gesture, the palm of the target user is facing along the y-axis, and the moving gesture of moving forward or backward can be triggered by moving the palm along the y-axis; the palm of the target user is facing along the x-axis, and the moving gesture of moving left or right can be triggered by moving the palm along the x-axis; the palm of the target user is facing along the z-axis, and the moving gesture of moving up or down can be triggered by moving the palm along the z-axis. Those skilled in the art will understand that when moving forward or backward, the palm of the hand can face forward (negative direction of the y-axis) or backward (positive direction of the y-axis); when moving up or down, the palm of the hand can face upward (positive direction of the z-axis) or downward (negative direction of the z-axis), and then the specific direction of the moving gesture is determined by the change in the coordinates of the moving gesture. For example, when the palm of the target user is facing upward, it can be performed to move upward or downward.

[0075] For the rotation gesture, the target finger (for example, the index finger) is facing the positive direction of the z-axis, and the palm is rotated with the target finger at this time as the axis to generate a rotation gesture of direction rotation to control the yaw angle (Yaw) of the robot; the target finger (for example, the index finger) is facing the negative direction of the y-axis, and the palm is rotated with the target finger at this time as the axis to generate a rotation gesture of tilt rotation to control the roll angle (Roll) of the robot; the target finger (for example, the index finger) is facing the negative direction of the x-axis, and the palm is rotated with the target finger at this time as the axis to generate a rotation gesture of pitch rotation to control the pitch angle (Pitch) of the robot.

[0076] The zoom gesture is triggered by the left hand and is similar to the move gesture, requiring multiple fingers to be spread out. Specifically, moving the left hand to the left is a zoom-in gesture, and moving the left hand to the right is a zoom-out gesture.

[0077] In addition, in order to be able to distinguish between movement gestures and rotation gestures, the multiple key points involved in the embodiments of the present invention include at least key points with fixed postures under different gesture types (such as palm key points, wrist key points, etc.), key points on the target finger (such as the fingertip key points of the target finger), and key points on other fingers (such as the fingertip key points of other fingers); for example, the multiple key points include key points on five fingers.

[0078] Although there are many gesture postures in three-dimensional space, the move gesture, zoom gesture, stop gesture, etc. adopted in the embodiments of the present invention use the direction of the palm to represent the movement direction, and use the rotation direction when the target finger rotates to indicate the rotation of the robot. These motion gestures are consistent with the user's daily operating habits and can ensure the naturalness of the interaction.

[0079] Optionally, in an embodiment of the present invention, the coordinates of the key points are absolute coordinates in an absolute coordinate system. For example, the coordinates of each key point are determined by a depth camera on the robot side, and the coordinates are coordinates in the three-dimensional absolute coordinate system of the depth camera, which is an absolute coordinate. When the user's palm moves or rotates, even if the gesture type does not change, the absolute coordinates of the corresponding key points will change, thereby affecting the determination of the gesture type of the current gesture data. In an embodiment of the present invention, the gesture type of the current gesture data is determined by relative coordinates. Specifically, the above step 302 "Determining the gesture type of the current gesture data based on the coordinates of multiple key points in the current gesture data" includes steps A2 and A4:

[0080] Step A2: converting the absolute coordinates of multiple key points in the current gesture data into relative coordinates in a relative coordinate system; when the gesture type remains unchanged, the relative coordinate system can remain unchanged.

[0081] Step A4: Determine the gesture type of the current gesture data according to the relative coordinates of the multiple key points.

[0082] In an embodiment of the present invention, each gesture type corresponds to a corresponding gesture posture. When a user performs a gesture of a certain gesture type, the gesture posture of his hand remains unchanged (the position of the hand may change); wherein the gesture posture includes the relative posture between multiple key points of the hand. The relative coordinate system established by the embodiment of the present invention can also remain unchanged when the gesture type remains unchanged. For example, two of the key points can be selected as one of the axes of the relative coordinate system, and the other two axes of the relative coordinate system can be determined based on this, thereby establishing a three-dimensional relative coordinate system. In this relative coordinate system, when the user continues to perform a gesture of a certain gesture type, the relative coordinates of the key points can remain unchanged; that is, for each gesture type, the key point has a relative coordinate, so that the gesture type can be represented more concisely and accurately based on the relative coordinates of the key point, thereby improving the accuracy of determining the gesture type.

[0083] Optionally, the relative coordinate system is a coordinate system determined based on the palm surface and the palm center orientation, where the palm center orientation is a direction perpendicular to the palm surface.

[0084] This embodiment of the present invention determines a relative coordinate system based on the palm surface and palm orientation. When a target user performs different gestures, they typically only change the position of their fingers, such as from extending their fingers to tucking them. However, the palm surface and palm orientation remain largely unchanged when changing between gestures. This embodiment establishes a relative coordinate system based on the palm surface and palm orientation, which better represents the relative coordinates of key points and allows for rapid determination of the relative coordinates of key points even when the gesture type changes.

[0085] Furthermore, the relative coordinate system for gestures of different types is essentially the same, even if the gesture moves or rotates freely in three-dimensional space. In other words, within the same relative coordinate system, the coordinates of multiple key points in different gesture types can be effectively and uniformly expressed. This allows accurate differentiation of different gesture types based on the relative coordinates of these key points, and allows accurate identification of the gesture type of the current gesture data.

[0086] Optionally, before the above step A2, a process of establishing a relative coordinate system is also included, and the process specifically includes steps A11-A14:

[0087] Step A11: Determine a first vector in the palm surface.

[0088] Step A12: Determine a second vector on the palm surface that is located in a plane with the first vector as a normal vector.

[0089] Step A13: Determine a third vector perpendicular to the first vector and the second vector.

[0090] Step A14: Determine a unit vector basis corresponding to the first vector, the second vector, and the third vector, and establish a relative coordinate system based on the unit vector basis.

[0091] In this embodiment of the present invention, the relative coordinate system is established based on the palm surface and palm orientation. The palm orientation is perpendicular to the palm surface, so two perpendicular vectors can be first determined within the palm surface. Because it is difficult to directly extract two perpendicular vectors from key points of the hand, this embodiment uses projection to determine two perpendicular vectors, namely, a first vector and a second vector.

[0092] Specifically, the first vector is determined based on two key points in the palm surface, for example, see Figure 4As shown, taking joint points as key points as an example, when a user completes various gestures, their wrist key points (Wrist), palm key points (Palm), and key points at the base of each finger (e.g., Thumb0, Pinky0, etc.) are generally always located on the palm surface. In this case, two key points can be selected from them, and the vector between these two key points is used as the first vector. In addition, a plane with the first vector as the normal vector is determined, and the collinearity between this plane and the palm surface can be used as the second vector. For example, based on two other key points on the palm surface, another vector A that is not parallel to the first vector is determined, and the projection of this other vector A onto the plane with the first vector as the normal vector is used as the second vector. After determining the first and second vectors on the palm surface, a third vector perpendicular to the palm surface can be determined; for example, the third vector can be determined by vector cross product. Furthermore, based on the mutually perpendicular first, second, and third vectors, a unit vector basis of a three-dimensional coordinate system can be determined. The coordinate system determined based on this unit vector basis is the relative coordinate system.

[0093] For example, see Figure 8 As shown, this embodiment takes the wrist joint (Wrist) as the coordinate origin to construct a relative coordinate system. Specifically, the direction from the wrist joint (Wrist) to the palm joint (Palm) is used as the x-axis, the direction on the palm surface that is perpendicular to the x-axis and points to the side of the thumb is used as the y-axis; and the direction perpendicular to the palm surface is used as the z-axis. Among them, the vector from the wrist joint (Wrist) to the palm joint (Palm) can be used as the first vector x; and the vector between the wrist joint (Wrist) and the thumb base joint (Thumb0) can be used as another vector A, and the projection of the vector A to the plane with the first vector x as the normal vector is the second vector y. Let s0 represent the absolute coordinates of the wrist joint, s1 represent the absolute coordinates of the palm joint, and s2 represent the absolute coordinates of the thumb base joint, then the three vectors x, y, and z satisfy the following formula:

[0094]

[0095] Where <,> represents vector inner product, and × represents vector cross product. The unit vector basis obtained by normalizing the three vectors x, y, and z is {e x =x / ||x||,e y =y / ||y||,e z =z / ||z||}, and then a relative coordinate system can be established.

[0096] Alternatively, in an embodiment of the present invention, the origin of the relative coordinate system is generally selected from a key point on the palm surface, such as a wrist key point, a palm key point, or a finger base key point. In this case, any key point in the original absolute coordinate system can be projected into the relative coordinate system, and the relative coordinates of multiple key points in the current gesture data satisfy:

[0097] p i =( i -s0,e x >, i -s0,e y >, i -s0,e z >) (6)

[0098] Among them, p i Represents the relative coordinates of key point i, s i represents the absolute coordinates of the key point i, s0 represents the absolute coordinates of the wrist key point, palm key point or finger base key point, {e x ,e y ,e z} is the unit vector basis, <,> represents the vector inner product; the wrist key point, palm key point, and finger base key point are each one of the multiple key points.

[0099] The relative coordinates of multiple key points are combined to form the basic features used to determine the gesture type. Figure 4 The 24 joint points shown represent key points, so the basic feature It can be expressed as:

[0100] Alternatively, the gesture type of the current gesture data may be determined based on a preset classification model. Alternatively, in the case of determining the relative coordinates of key points, the above step A4 "determining the gesture type of the current gesture data based on the relative coordinates of multiple key points" includes steps A41-A42:

[0101] Step A41: Determine the similarity between the relative coordinates of multiple key points in the current gesture data and the standard posture corresponding to each gesture type, where the standard posture includes the standard coordinates of multiple key points in the relative coordinate system under the corresponding gesture type.

[0102] Step A42: taking the gesture type corresponding to the similarity greater than a preset threshold as the gesture type of the current gesture data.

[0103] ​​​In embodiments of the present invention, after a target user performs a gesture of a certain type, the relative coordinates of the key points remain unchanged even if the target user moves or rotates their hand. Therefore, standard coordinates of multiple key points corresponding to each gesture type can be pre-determined, and these standard coordinates can be used as a reference to determine which gesture type the current gesture data belongs to.

[0104] For example, the embodiment of the present invention sets a move gesture, a rotation gesture and a stop gesture (such as Figure 6 For each gesture, the relative coordinates of key points collected by multiple users at different angles can be pre-collected and averaged to obtain the standard coordinates of each key point under the corresponding gesture type. The relative coordinates of multiple key points in the current gesture data are used as the basic features of the current gesture data. The standard coordinates of multiple key points under each gesture type are used as the standard features of the gesture type For example, if the number of key points is 24, the standard feature can be recorded as: Among them, h i Indicates the standard coordinates of each key point.

[0105] Determine the basic characteristics Standard features for each gesture type The similarity between them, then the gesture type recognition is a basic binary classification problem. If the basic features of the current gesture data Standard features of a gesture type When the similarity between the two gestures is greater than a preset threshold, the gesture is considered to be of the gesture type. For example, the Euclidean similarity can be used as a metric to calculate the Euclidean distance of the corresponding joint points of the two gestures and average it, and then normalize it to the range of [0,1] to get the similarity between the two gestures.

[0106]

[0107] Based on any of the above embodiments, the inventors have discovered that experimental results show that users prefer a constant speed gain in auxiliary photography scenarios. Therefore, the speed mapping can be converted into a distance (or angle) mapping, and continuous control of the robot can be achieved through changes in distance (or angle). The above step 303 "Determining the degree of change in the current gesture data" includes step B1 and / or step B2:

[0108] Step B1: When the gesture type of the current gesture data is a moving gesture, determine the displacement change between the current gesture data and the first historical gesture data in the moving direction of the current gesture data, and use the displacement change as the change degree of the current gesture data.

[0109] Step B2: When the gesture type of the current gesture data is a rotation gesture, determine the angle change between the current gesture data and the second historical gesture data in the rotation direction of the current gesture data, and use the angle change as the degree of change of the current gesture data.

[0110] In embodiments of the present invention, both a move gesture and a rotation gesture are motion gestures. For different types of motion gestures, the degree of change in the current gesture data is determined in different ways. Specifically, as shown in step B1, if the current gesture data is a move gesture, i.e., when the target user performs a move gesture, historical gesture data prior to the current gesture data, i.e., first historical gesture data, is determined. The displacement change between the current gesture data and the first historical gesture data is used as the degree of change in the current gesture data. Specifically, the displacement change is used to generate the control variable for the continuous control instruction.

[0111] Alternatively, as shown in step B2, if the current gesture data is a rotation gesture, that is, when the target user performs a rotation gesture, the historical gesture data before the current gesture data, that is, the second historical gesture data, is determined, and the angle change between the current gesture data and the second historical gesture data is used as the degree of change of the current gesture data, that is, the control amount of the continuous control instruction is generated using the angle change.

[0112] Optionally, the moving gesture is a gesture moving in the direction of the palm; for example, see Figure 6 As shown, the moving gesture is a gesture in which multiple fingers are spread out and move in the direction of the palm. The above step B1 "determining the displacement change between the current gesture data and the first historical gesture data in the moving direction of the current gesture data" may include steps B11-B13:

[0113] Step B11: Determine the palm direction d corresponding to the current gesture data.

[0114] In an embodiment of the present invention, when the movement gesture is a gesture that moves along the palm orientation, the palm orientation can represent the direction of movement. In this case, it is necessary to determine the vector d corresponding to the palm orientation. For example, when establishing the above-mentioned relative coordinate system, the palm orientation d can be represented by a third vector z, that is, d = z.

[0115] Step B12: Determine the displacement of a valid key point between the current gesture data and the first historical gesture data. The valid key point is one of the multiple key points, and the displacement of the valid key point satisfies:

[0116]

[0117] Among them, the current gesture data is the gesture data at time T, and the first historical gesture data is the gesture data at time t1. Indicates the absolute coordinates of the valid key point i in the current gesture data, Represents the absolute coordinates of the valid key point i in the first historical gesture data, <,> represents the vector inner product, Distance i (T; t1, d) represents the displacement of the effective key point i.

[0118] In the embodiment of the present invention, the current gesture data is the gesture data at time T, and the first historical gesture data is the gesture data at time t1, which is earlier than time T; for example, the first historical gesture data can be the m frames of data before the current gesture data (for example, m=1). In this embodiment, at least one key point is selected from multiple key points as a valid key point, and the displacement vector of the same valid key point in the two gesture data (i.e., the current gesture data and the first historical gesture data) is used to calculate the key point. To determine the displacement of the effective key point along the palm toward d, that is, from time t1 to time T, the moving distance of the effective key point is expressed as Distance i (T; t1, d).

[0119] Those skilled in the art will appreciate that if the distance i If (T; t1, d) is positive, it means that the target user's palm is moving towards the center of the palm in the positive direction d; on the contrary, if the distance Distance i If (T; t1, d) is negative, it means that the target user's palm is moving in the opposite direction of the palm direction d.

[0120] Step B13: Determine the displacement change between the current gesture data and the first historical gesture data based on the displacement of at least one valid key point.

[0121] In this embodiment of the present invention, one key point can be selected as a valid key point, that is, the number of key points is one. For example, the palm key point (or wrist key point) can be selected as a valid key point. In this case, the displacement of the valid key point can be directly used as the displacement difference between the current gesture data and the first historical gesture data. Alternatively, if there are multiple valid key points, the average displacement of the multiple valid key points can be used as the displacement difference between the current gesture data and the first historical gesture data.

[0122] In the embodiment of the present invention, the displacement change (eg, Distance iThe data (T; t1, d)) and palm orientation d accurately describe the direction and amount of movement of the gesture, thereby generating corresponding continuous control instructions. For example, if palm orientation d indicates the left side, the sign of the displacement change determines whether the movement is left or right. The magnitude of the displacement change also determines the control amount of the continuous control instruction, thus achieving left and right movement control of the robot.

[0123] Optionally, the control amount of the continuous control instruction is directly proportional to the degree of change of the current gesture data. In an embodiment of the present invention, when the current gesture is a moving gesture, the control amount of the generated continuous control instruction represents the moving distance of the robot, and the control amount can be expressed as:

[0124] f t =w t ·x

[0125] Where x is the displacement change, f t is the mapped robot moving distance, i.e. the control amount; coefficient w t is an adjustable hyperparameter constant.

[0126] If the zoom gesture is similar to the move gesture, except for the distinction between the left and right hands, then when the current gesture is a zoom gesture, the displacement change can also be determined in the same way, and the direction of the displacement can be used to determine whether the current gesture is a zoom-in gesture or a zoom-out gesture. This embodiment will not go into details about this.

[0127] Optionally, the rotation gesture is a gesture that rotates around a target finger; for example, see Figure 6 As shown, the rotation gesture is a gesture that rotates around the target finger while keeping the target finger in a fixed position. The above step B2 "determining the angular change between the current gesture data and the second historical gesture data in the rotation direction of the current gesture data" may include steps B21-B24:

[0128] Step B21: Determine the direction d of the target finger in the current gesture data rota .

[0129] In an embodiment of the present invention, the current gesture data includes at least two key points located in the target finger, and the orientation of the target finger can be determined based on the absolute coordinates of these two key points. In general, the current gesture data may include the coordinates of the finger base key point and the fingertip key point. For example, the index finger is used as the target finger, that is, the gesture is triggered with the index finger as the rotation axis. At this time, the orientation of the index finger can be obtained by subtracting the absolute coordinates of the finger base key point (Index1) and the fingertip key point (IndexTip) on the index finger, which is recorded as d rota .

[0130] Step B22: Determine the direction d according to the current gesture data rota The current vector is not parallel to the direction d, and is determined according to the second historical gesture data. rota Non-parallel history vectors; the current vector and the history vector are vectors determined from the corresponding gesture data in the same way.

[0131] Step B23: Determine the current vector and the history vector are projected to the direction d rota The current projection vector determined by the plane of the normal vector and the historical projection vector

[0132] Step B24: The current projection vector With the historical projection vector The included angle between them is used as the angle change between the current gesture data and the second historical gesture data.

[0133] In the embodiment of the present invention, in order to conveniently calculate the rotation angle around the target finger, based on the direction d rota Non-parallel vectors are moving in the direction d rota Specifically, as shown in steps B22-B24 above, for the current gesture data and the previous second historical gesture data, the same method is used to select the direction d from them. rota Non-parallel vectors; for example, select the vector from the index finger base joint (Index1) to the palm joint (Palm). The vector determined from the current gesture data is called the current vector, and the vector determined from the second historical gesture data is called the historical vector. Based on the projection formula, it can be determined that the current vector and the historical vector are in the direction d rota is the projection vector of the normal vector in the plane. For example, select the vector from the index finger base joint (Index1) to the palm joint (Palm): palm -s index1 =(x palm -x index1 ,y palm -y index1 ,z palm -z index1 ), which is projected onto the direction d rota On the plane of the normal vector, the projection vector v rota for:

[0134]

[0135] Accordingly, this embodiment uses Represents the current projection vector, Represents a historical projection vector. The current gesture data is the gesture data at time T, and the second historical gesture data is the gesture data at time t2, where time t2 is earlier than time T. For example, the second historical gesture data may be the m frames of data preceding the current gesture data (e.g., m=1), or the second historical gesture data may be the gesture data collected at the initial moment of triggering the rotation gesture.

[0136] From time t2 to time T, the user moves the target finger in the direction d rota The rotation angle of the axis is the current projection vector With the historical projection vector The angle between them satisfies:

[0137]

[0138] The included angle RotationAngle(T; t2) can be used as the angle change between the current gesture data and the second historical gesture data.

[0139] In addition, optionally, since the angle determined by the above formula (10) is positive, the rotation direction of the rotation gesture cannot be determined based on this alone. Therefore, the embodiment of the present invention may further include:

[0140] Step B25: Determine the direction indication value, and determine the rotation direction of the rotation gesture based on the positive or negative sign of the direction indication value; the direction indication value satisfies:

[0141]

[0142] The current gesture data is the gesture data at time T, the second historical gesture data is the gesture data at time t2, RotationDirection(T; t2) represents the direction indication value, and × represents vector cross product.

[0143] In an embodiment of the present invention, a rotation gesture can be determined based on the direction indication value. Specifically, when the direction indication value is positive, it indicates that the rotation direction is counterclockwise, otherwise it indicates that the rotation direction is clockwise.

[0144] Similarly, for a rotation gesture, the control amount of the continuous control instruction is directly proportional to the degree of change in the current gesture data. In an embodiment of the present invention, when the current gesture is a rotation gesture, the control amount of the generated continuous control instruction represents the rotation angle of the robot, which can be expressed as:

[0145] f r =w r ·θ

[0146] Where θ is the angle change, f r is the mapped robot rotation angle, i.e. the control quantity; coefficient wr is a tunable hyperparameter constant; for example, w r =1.

[0147] A method for controlling a robot provided by an embodiment of the present invention can accurately and quickly determine the displacement change or angle change using the absolute coordinates of key points contained in gesture data, thereby achieving rapid response.

[0148] The above describes in detail the method for controlling a robot provided by an embodiment of the present invention. This method can also be implemented by a corresponding device. The following describes in detail the device for controlling a robot provided by an embodiment of the present invention.

[0149] Figure 9 FIG. 1 shows a schematic diagram of the structure of a device for controlling a robot provided by an embodiment of the present invention. Figure 9 As shown, the device for controlling the robot includes:

[0150] An acquisition module 91 is configured to acquire current gesture data of a target user, wherein the current gesture data includes coordinates of multiple key points of a hand;

[0151] a type determination module 92, configured to determine a gesture type of the current gesture data according to coordinates of a plurality of key points in the current gesture data;

[0152] The processing module 93 is used to determine the degree of change of the current gesture data and generate a continuous control instruction for controlling the movement of the robot when the gesture type of the current gesture data is a motion gesture; the type of the continuous control instruction is consistent with the gesture type of the current gesture data, and there is a positive correlation between the control amount of the continuous control instruction and the degree of change of the current gesture data.

[0153] In a possible implementation, the coordinates of the key points are absolute coordinates in an absolute coordinate system;

[0154] The type determination module 92 includes:

[0155] a conversion unit, configured to convert the absolute coordinates of the plurality of key points in the current gesture data into relative coordinates in a relative coordinate system; the relative coordinate system is a coordinate system determined based on the palm surface and the palm orientation, wherein the palm orientation is a direction perpendicular to the palm surface;

[0156] A determining unit is configured to determine a gesture type of the current gesture data according to relative coordinates of the plurality of key points.

[0157] In a possible implementation, the type determination module 92 further includes: a coordinate unit;

[0158] Before converting the absolute coordinates of the plurality of key points in the current gesture data into relative coordinates in a relative coordinate system, the coordinate unit is used to:

[0159] determining a first vector in the palm surface;

[0160] determining a second vector on the palm surface that is located in a plane having the first vector as a normal vector;

[0161] determining a third vector perpendicular to the first vector and the second vector;

[0162] A unit vector basis corresponding to the first vector, the second vector, and the third vector is determined, and the relative coordinate system is established based on the unit vector basis.

[0163] In a possible implementation, the relative coordinates of the multiple key points in the current gesture data satisfy:

[0164] p i =( i -s0,e x >, i -s0,e y >, i -s0,e z >)

[0165] Among them, p i Represents the relative coordinates of key point i, s i represents the absolute coordinates of the key point i, s0 represents the absolute coordinates of the wrist key point, palm key point or finger base key point, {e x ,e y ,e z} is the unit vector basis, <,> represents the vector inner product; the wrist key point, the palm key point, and the finger base key point are each one of the multiple key points.

[0166] In a possible implementation, the determining unit determines the gesture type of the current gesture data according to the relative coordinates of the plurality of key points, including:

[0167] Determining similarities between relative coordinates of the plurality of key points in the current gesture data and a standard gesture corresponding to each gesture type, the standard gesture including standard coordinates of the plurality of key points in the relative coordinate system under the corresponding gesture type;

[0168] The gesture type corresponding to the similarity greater than a preset threshold is used as the gesture type of the current gesture data.

[0169] In a possible implementation, the processing module 93 includes a moving unit and / or a rotating unit;​​​

[0170] The moving unit is configured to: when the gesture type of the current gesture data is a moving gesture, determine a displacement change amount between the current gesture data and the first historical gesture data in a moving direction of the current gesture data, and use the displacement change amount as a change degree of the current gesture data;

[0171] The rotation unit is configured to: when the gesture type of the current gesture data is a rotation gesture, determine an angle change between the current gesture data and the second historical gesture data in a rotation direction of the current gesture data, and use the angle change as a degree of change of the current gesture data;

[0172] The moving gesture and the rotating gesture are both motion gestures.

[0173] In a possible implementation, the movement gesture is a gesture moving in the direction of the palm; and the movement unit determines, in the moving direction of the current gesture data, an amount of displacement change between the current gesture data and the first historical gesture data, including:

[0174] Determine the palm direction d corresponding to the current gesture data;

[0175] Determine a displacement of a valid key point between the current gesture data and the first historical gesture data, where the valid key point is one of the plurality of key points, and the displacement of the valid key point satisfies:

[0176]

[0177] The current gesture data is the gesture data at time T, and the first historical gesture data is the gesture data at time t1. represents the absolute coordinates of the valid key point i in the current gesture data, Represents the absolute coordinates of the valid key point i in the first historical gesture data, <,> represents the vector inner product, Distance i (T; t1, d) represents the displacement of the effective key point i;

[0178] The displacement change between the current gesture data and the first historical gesture data is determined based on the displacement of at least one of the valid key points.

[0179] In a possible implementation, the rotation gesture is a gesture rotating around a target finger; and the rotation unit determines, in a rotation direction of the current gesture data, an angular change between the current gesture data and the second historical gesture data, including:

[0180] Determine the direction d of the target finger in the current gesture data rota ;

[0181] Determine the direction d according to the current gesture data rota The current vector is not parallel to the direction d, and the second historical gesture data is determined based on the second historical gesture data. rota Non-parallel historical vectors; the current vector and the historical vector are vectors determined from corresponding gesture data in the same manner;

[0182] Determine the current vector and the history vector are projected to the direction d rota The current projection vector determined by the plane of the normal vector and the historical projection vector

[0183] The current projection vector With the historical projection vector The included angle between the current gesture data and the second historical gesture data is used as the angle change between the current gesture data and the second historical gesture data.

[0184] In a possible implementation, the rotating unit is further configured to:

[0185] Determine a direction indication value, and determine a rotation direction of the rotation gesture based on the positive or negative sign of the direction indication value; the direction indication value satisfies:

[0186]

[0187] The current gesture data is the gesture data at time T, the second historical gesture data is the gesture data at time t2, RotationDirection(T; t2) represents the direction indication value, and × represents vector cross product.

[0188] In a possible implementation, there is a positive proportional relationship between the control amount of the continuous control instruction and the degree of change of the current gesture data.

[0189] In a possible implementation, the processing module 93 is further configured to:

[0190] In a case where the gesture type of the current gesture data is a stop gesture, a stop instruction for pausing control of the robot is generated.

[0191] In a possible implementation, the gesture type includes at least one of a move gesture, a rotation gesture, a zoom gesture, and a stop gesture;

[0192] The moving gesture is a gesture in which multiple fingers are spread out and moved in the direction of the palm;

[0193] The rotation gesture is a gesture of keeping the target finger in a constant position and rotating around the target finger when the target finger is open;

[0194] The zoom gesture is a gesture in which multiple fingers are spread out and moved along the palm direction, and the zoom gesture and the move gesture are gestures triggered by the left and right hands respectively;

[0195] The stop gesture is a gesture in which all fingers are put away.

[0196] In addition, an embodiment of the present invention also provides an electronic device, including a bus, a transceiver, a memory, a processor, and a computer program stored in the memory and runnable on the processor. The transceiver, the memory, and the processor are respectively connected via a bus. When the computer program is executed by the processor, each process of the above-mentioned method embodiment for controlling a robot is implemented, and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0197] For details, see Figure 10 As shown, an embodiment of the present invention further provides an electronic device, which includes a bus 1110 , a processor 1120 , a transceiver 1130 , a bus interface 1140 , a memory 1150 and a user interface 1160 .

[0198] In an embodiment of the present invention, the electronic device further includes: a computer program stored in the memory 1150 and executable on the processor 1120, and when the computer program is executed by the processor 1120, each process of the above-mentioned method embodiment for controlling a robot is implemented.

[0199] The transceiver 1130 is configured to receive and send data under the control of the processor 1120 .

[0200] In an embodiment of the present invention, a bus architecture (represented by bus 1110) may include any number of interconnected buses and bridges, and bus 1110 connects various circuits including one or more processors represented by processor 1120 and a memory represented by memory 1150.

[0201] Bus 1110 represents one or more of any of several types of bus structures, including a memory bus and memory controller, a peripheral bus, an Accelerated Graphical Port (AGP), a processor, or a local bus using any of a variety of bus architectures. By way of example and not limitation, such architectures include an Industry Standard Architecture (ISA) bus, a Micro Channel Architecture (MCA) bus, an Enhanced ISA (EISA) bus, a Video Electronics Standards Association (VESA), and a Peripheral Component Interconnect (PCI) bus.

[0202] The processor 1120 can be an integrated circuit chip with signal processing capabilities. During implementation, each step of the above method embodiment can be completed by an integrated logic circuit of hardware in the processor or an instruction in the form of software. The above-mentioned processor includes: a general-purpose processor, a central processing unit (CPU), a network processor (NP), a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA), a complex programmable logic device (CPLD), a programmable logic array (PLA), a microcontroller unit (MCU) or other programmable logic devices, discrete gates, transistor logic devices, discrete hardware components. The various methods, steps and logic block diagrams disclosed in the embodiments of the present invention can be implemented or executed. For example, the processor can be a single-core processor or a multi-core processor, and the processor can be integrated into a single chip or located on multiple different chips.

[0203] The processor 1120 can be a microprocessor or any conventional processor. The method steps disclosed in conjunction with the embodiments of the present invention can be directly executed by a hardware decoding processor, or can be executed by a combination of hardware and software modules in the decoding processor. The software module can be located in a readable storage medium known in the art, such as a random access memory (RAM), a flash memory (Flash Memory), a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), or a register. The readable storage medium is located in a memory, and the processor reads the information in the memory and performs the steps of the above method in conjunction with its hardware.

[0204] The bus 1110 may also connect various other circuits, such as peripheral devices, voltage regulators, or power management circuits. The bus interface 1140 provides an interface between the bus 1110 and the transceiver 1130. These are all well known in the art and are therefore not further described in this embodiment of the present invention.

[0205] The transceiver 1130 can be a single component or multiple components, such as multiple receivers and transmitters, providing a means for communicating with various other devices over a transmission medium. For example, the transceiver 1130 receives external data from other devices and transmits data processed by the processor 1120 to other devices. Depending on the nature of the computer system, a user interface 1160 may also be provided, such as a touch screen, physical keyboard, display, mouse, speaker, microphone, trackball, joystick, or stylus.

[0206] It should be understood that in an embodiment of the present invention, the memory 1150 may further include a memory remotely located relative to the processor 1120, and these remotely located memories may be connected to a server via a network. One or more parts of the aforementioned network may be an ad hoc network, an intranet, an extranet, a virtual private network (VPN), a local area network (LAN), a wireless local area network (WLAN), a wide area network (WAN), a wireless wide area network (WWAN), a metropolitan area network (MAN), the Internet, a public switched telephone network (PSTN), a plain old telephone service network (POTS), a cellular telephone network, a wireless network, a wireless fidelity (Wi-Fi) network, or a combination of two or more of the aforementioned networks. For example, the cellular telephone network and the wireless network can be a Global System for Mobile Communications (GSM) system, a Code Division Multiple Access (CDMA) system, a Worldwide Interoperability for Microwave Access (WiMAX) system, a General Packet Radio Service (GPRS) system, a Wideband Code Division Multiple Access (WCDMA) system, a Long Term Evolution (LTE) system, an LTE Frequency Division Duplex (FDD) system, an LTE Time Division Duplex (TDD) system, an Advanced Long Term Evolution (LTE-A) system, a Universal Mobile Telecommunications (UMTS) system, an Enhanced Mobile Broadband (eMBB) system, a Massive Machine Type of Communication (mMTC) system, an Ultra Reliable Low Latency Communications (uRLLC) system, and the like.

[0207] It should be understood that the memory 1150 in the embodiment of the present invention may be a volatile memory or a non-volatile memory, or may include both volatile and non-volatile memories. Non-volatile memories include read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory.

[0208] Volatile memory includes random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of RAM are available, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM), and direct RAM bus random access memory (DRRAM). The memory 1150 of the electronic device described in the embodiments of the present invention includes, but is not limited to, the above and any other suitable types of memory.

[0209] In the embodiment of the present invention, the memory 1150 stores the following elements of the operating system 1151 and the application 1152: executable modules, data structures, or subsets thereof, or extended sets thereof.

[0210] Specifically, the operating system 1151 includes various system programs, such as a framework layer, a core library layer, and a driver layer, which are used to implement various basic services and process hardware-based tasks. The application 1152 includes various application programs, such as a media player and a browser, which are used to implement various application services. The program that implements the method of the embodiment of the present invention may be included in the application 1152. The application 1152 includes applets, objects, components, logic, data structures, and other computer system executable instructions that perform specific tasks or implement specific abstract data types.

[0211] In addition, an embodiment of the present invention also provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the various processes of the above-mentioned method embodiment for controlling a robot are implemented and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0212] Computer-readable storage media include: permanent and non-permanent, removable and non-removable media, which are tangible devices that can retain and store instructions for use by instruction execution devices. Computer-readable storage media include: electronic storage devices, magnetic storage devices, optical storage devices, electromagnetic storage devices, semiconductor storage devices, and any suitable combination of the above. Computer-readable storage media include: phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), non-volatile random access memory (NVRAM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassette storage, magnetic disk storage or other magnetic storage devices, memory sticks, mechanical encoding devices (such as punched cards or raised structures with grooves in which instructions are recorded), or any other non-transmission medium that can be used to store information that can be accessed by a computing device. As defined in the embodiments of the present invention, computer-readable storage media does not include temporary signals themselves, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides or other transmission media (such as light pulses passing through fiber optic cables), or electrical signals transmitted through wires.

[0213] In the several embodiments provided in this application, it should be understood that the disclosed devices, electronic devices and methods can be implemented in other ways. For example, the device embodiments described above are merely schematic. For example, the division of the modules or units is only a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, or can be an electrical, mechanical or other form of connection.

[0214] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, and may be located in a single location or distributed across multiple network units. Some or all of these units may be selected based on actual needs to address the issues addressed by the embodiments of the present invention.

[0215] In addition, the functional units in the various embodiments of the present invention may be integrated into a single processing unit, each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0216] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the embodiment of the present invention is essentially or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for enabling a computer device (including: a personal computer, a server, a data center or other network device) to perform all or part of the steps of the method described in each embodiment of the present invention. The above-mentioned storage medium includes the various media that can store program codes as listed above.

[0217] In describing the embodiments of the present invention, those skilled in the art should understand that the embodiments of the present invention can be implemented as methods, apparatuses, electronic devices, and computer-readable storage media. Therefore, the embodiments of the present invention can be specifically implemented in the following forms: complete hardware, complete software (including firmware, resident software, microcode, etc.), or a combination of hardware and software. In addition, in some embodiments, the embodiments of the present invention can also be implemented in the form of a computer program product in one or more computer-readable storage media, wherein the computer-readable storage medium contains computer program code.

[0218] The above-mentioned computer-readable storage medium may adopt any combination of one or more computer-readable storage media. Computer-readable storage media include: electrical, magnetic, optical, electromagnetic, infrared or semiconductor systems, devices or components, or any combination thereof. More specific examples of computer-readable storage media include: portable computer disks, hard disks, random access memories (RAM), read-only memories (ROM), erasable programmable read-only memories (EPROM), flash memories (Flash Memory), optical fibers, compact disc read-only memories (CD-ROMs), optical storage devices, magnetic storage devices or any combination thereof. In an embodiment of the present invention, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, device, or component.

[0219] The computer program code contained in the computer-readable storage medium may be transmitted using any appropriate medium, including wireless, wire, optical cable, radio frequency (RF), or any suitable combination thereof.

[0220] The computer program code for performing the operations of the embodiments of the present invention can be written in assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, integrated circuit configuration data, or in one or more programming languages ​​or a combination thereof, wherein the programming language includes an object-oriented programming language, such as Java, Smalltalk, C++, and also includes a conventional procedural programming language, such as C language or a similar programming language. The computer program code can be executed entirely on the user's computer, partially on the user's computer, as a stand-alone software package, partially on the user's computer and partially on a remote computer, and entirely on a remote computer or server. In the case of a remote computer, the remote computer can be connected to the user's computer or to an external computer via any type of network, including a local area network (LAN) or a wide area network (WAN).

[0221] The embodiments of the present invention describe the provided methods, devices, and electronic devices through flowcharts and / or block diagrams.

[0222] It should be understood that each block in the flowchart and / or block diagram, as well as combinations of blocks in the flowchart and / or block diagram, can be implemented by computer-readable program instructions. These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, thereby producing a machine. These computer-readable program instructions are executed by the computer or other programmable data processing device to produce a device that implements the functions / operations specified in the blocks in the flowchart and / or block diagram.

[0223] These computer-readable program instructions may also be stored in a computer-readable storage medium that enables a computer or other programmable data processing device to operate in a specific manner. In this way, the instructions stored in the computer-readable storage medium produce an instruction device product that implements the functions / operations specified in the blocks in the flowchart and / or block diagram.

[0224] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device, so that a series of operational steps are performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby enabling the instructions executed on the computer or other programmable data processing apparatus to provide a process that implements the functions / operations specified by the blocks in the flowchart and / or block diagram.

[0225] The above description is merely a specific implementation of the embodiments of the present invention, but the scope of protection of the embodiments of the present invention is not limited thereto. Any changes or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in the embodiments of the present invention should be included in the scope of protection of the embodiments of the present invention. Therefore, the scope of protection of the embodiments of the present invention should be based on the scope of protection of the claims.

Claims

1. A method for controlling a robot, characterized in that: include: Acquire current gesture data of the target user, where the current gesture data includes coordinates of multiple key points of the hand; determining a gesture type of the current gesture data according to the coordinates of the plurality of key points in the current gesture data; When the gesture type of the current gesture data is a motion gesture, determining a degree of change of the current gesture data and generating a continuous control instruction for controlling the motion of the robot; the type of the continuous control instruction is consistent with the gesture type of the current gesture data, and there is a positive correlation between a control amount of the continuous control instruction and the degree of change of the current gesture data; The coordinates of the key points are absolute coordinates in the absolute coordinate system; The determining the gesture type of the current gesture data according to the coordinates of the plurality of key points in the current gesture data includes: Converting the absolute coordinates of the plurality of key points in the current gesture data into relative coordinates in a relative coordinate system; when the gesture type remains unchanged, the relative coordinate system can remain unchanged; determining a gesture type of the current gesture data according to the relative coordinates of the plurality of key points; The determining the gesture type of the current gesture data according to the relative coordinates of the plurality of key points includes: Determining similarities between relative coordinates of the plurality of key points in the current gesture data and a standard gesture corresponding to each gesture type, the standard gesture including standard coordinates of the plurality of key points in the relative coordinate system under the corresponding gesture type; The gesture type corresponding to the similarity greater than a preset threshold is used as the gesture type of the current gesture data.

2. The method according to claim 1, characterized in that The relative coordinate system is a coordinate system determined based on the palm surface and the palm center orientation, where the palm center orientation is a direction perpendicular to the palm surface.

3. The method according to claim 2, characterized in that Before converting the absolute coordinates of the plurality of key points in the current gesture data into relative coordinates in a relative coordinate system, the method further includes: determining a first vector in the palm surface; determining a second vector on the palm surface that is located in a plane having the first vector as a normal vector; determining a third vector perpendicular to the first vector and the second vector; A unit vector basis corresponding to the first vector, the second vector, and the third vector is determined, and the relative coordinate system is established based on the unit vector basis.

4. The method according to claim 3, characterized in that The relative coordinates of the key points in the current gesture data satisfy: p i =(<s i -s0,e x >,<s i -s0,e y >,<s i -s0,e z >) Among them, p i Represents the relative coordinates of key point i, s i represents the absolute coordinates of the key point i, s0 represents the absolute coordinates of the wrist key point, palm key point or finger base key point, {e x ,e y ,e z } is the unit vector basis, <,> represents the vector inner product; the wrist key point, the palm key point, and the finger base key point are each one of the multiple key points.

5. The method according to claim 1, characterized in that The determining the change degree of the current gesture data includes: If the gesture type of the current gesture data is a moving gesture, determining a displacement change between the current gesture data and the first historical gesture data in a moving direction of the current gesture data, and using the displacement change as a degree of change of the current gesture data; When the gesture type of the current gesture data is a rotation gesture, determining an angular change between the current gesture data and the second historical gesture data in a rotation direction of the current gesture data, and using the angular change as a degree of change of the current gesture data; The moving gesture and the rotating gesture are both motion gestures.

6. The method according to claim 5, characterized in that The moving gesture is a gesture moving in the direction of the palm; Determining a displacement change between the current gesture data and the first historical gesture data in the moving direction of the current gesture data includes: Determine the palm direction d corresponding to the current gesture data; Determine a displacement of a valid key point between the current gesture data and the first historical gesture data, where the valid key point is one of the plurality of key points, and the displacement of the valid key point satisfies: The current gesture data is the gesture data at time T, and the first historical gesture data is the gesture data at time t1. represents the absolute coordinates of the valid key point i in the current gesture data, Represents the absolute coordinates of the valid key point i in the first historical gesture data, <,> represents the vector inner product, Distance i (T; t1, d) represents the displacement of the effective key point i; The displacement change between the current gesture data and the first historical gesture data is determined based on the displacement of at least one of the valid key points.

7. The method according to claim 5, characterized in that The rotation gesture is a gesture that rotates around a target finger; Determining an angle change between the current gesture data and the second historical gesture data in the rotation direction of the current gesture data includes: Determine the direction d of the target finger in the current gesture data rota ; Determine the direction d according to the current gesture data rota The current vector is not parallel to the direction d, and the second historical gesture data is determined based on the second historical gesture data. rota Non-parallel historical vectors; the current vector and the historical vector are vectors determined from corresponding gesture data in the same manner; Determine the current vector and the history vector are projected to the direction d rota The current projection vector determined by the plane of the normal vector and the historical projection vector The current projection vector With the historical projection vector The included angle between the current gesture data and the second historical gesture data is used as the angle change between the current gesture data and the second historical gesture data.

8. The method according to claim 7, characterized in that Also includes: Determine a direction indication value, and determine a rotation direction of the rotation gesture based on the positive or negative sign of the direction indication value; the direction indication value satisfies: The current gesture data is the gesture data at time T, the second historical gesture data is the gesture data at time t2, RotationDirection(T; t2) represents the direction indication value, and × represents vector cross product.

9. The method according to claim 1, characterized in that There is a positive proportional relationship between the control amount of the continuous control instruction and the degree of change of the current gesture data.

10. The method according to any one of claims 1 to 9, characterized in that: Also includes: In a case where the gesture type of the current gesture data is a stop gesture, a stop instruction for pausing control of the robot is generated.

11. The method according to any one of claims 1 to 9, characterized in that: The gesture type includes at least one of a move gesture, a rotation gesture, a zoom gesture, and a stop gesture; The moving gesture is a gesture in which multiple fingers are spread out and moved in the direction of the palm; The rotation gesture is a gesture of keeping the target finger in a constant position and rotating around the target finger when the target finger is open; The zoom gesture is a gesture in which multiple fingers are spread out and moved along the palm direction, and the zoom gesture and the move gesture are gestures triggered by the left and right hands respectively; The stop gesture is a gesture in which all fingers are put away.

12. A device for controlling a robot, characterized in that: include: An acquisition module, configured to acquire current gesture data of a target user, wherein the current gesture data includes coordinates of multiple key points of a hand; a type determination module, configured to determine a gesture type of the current gesture data according to coordinates of a plurality of key points in the current gesture data; a processing module, configured to, when the gesture type of the current gesture data is a motion gesture, determine a degree of change in the current gesture data and generate a continuous control instruction for controlling the motion of the robot; the type of the continuous control instruction is consistent with the gesture type of the current gesture data, and a control amount of the continuous control instruction is positively correlated with the degree of change in the current gesture data; The coordinates of the key points are absolute coordinates in the absolute coordinate system; The type determination module includes: a conversion unit, configured to convert the absolute coordinates of the plurality of key points in the current gesture data into relative coordinates in a relative coordinate system; when the gesture type remains unchanged, the relative coordinate system can remain unchanged; a determining unit, configured to determine a gesture type of the current gesture data according to the relative coordinates of the plurality of key points; The determining unit is specifically configured to determine similarities between the relative coordinates of the plurality of key points in the current gesture data and a standard gesture corresponding to each gesture type, the standard gesture including the standard coordinates of the plurality of key points in the relative coordinate system under the corresponding gesture type; The gesture type corresponding to the similarity greater than a preset threshold is used as the gesture type of the current gesture data.

13. An electronic device comprising a bus, a transceiver, a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the transceiver, the memory, and the processor are connected via the bus, wherein: When the computer program is executed by the processor, the steps of the method for controlling a robot according to any one of claims 1 to 11 are implemented.

14. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method for controlling a robot according to any one of claims 1 to 11 are implemented.

Images