Gesture recognition method, device, equipment, medium and program

Abstract

The embodiment of the invention provides a gesture recognition method and device, equipment, a medium and a program, and the method comprises the steps: obtaining the key point information of a to-be-recognized gesture, determining the recognition information of the to-be-recognized gesture according to the key point information of the to-be-recognized gesture, the recognition information comprises one or more of the following information: the gesture of the hand, the orientation of the hand and the contact relation of the skeleton of the hand, the gesture type of the gesture to be recognized is determined according to the recognition information, and the recognition method can quickly and accurately recognize the gesture.

Description

[0001] This application is a divisional application for the Chinese patent application submitted to the China Patent Office on October 31, 2022, with application number 2022113509090 and invention name "Gesture Recognition Method, Device, Equipment, Medium and Program". Technical Field

[0002] The embodiments of the present application relate to the field of artificial intelligence, and in particular, to a gesture recognition method, apparatus, device, medium and program. Background Art

[0003] Extended Reality (XR) refers to the combination of reality and virtuality through computers to create a virtual environment for human-computer interaction. XR is also a general term for multiple technologies such as Virtual Reality (VR), Augmented Reality (AR) and Mixed Reality (MR). By integrating the visual interaction technologies of the three, it brings the experiencer an "immersive feeling" of seamless transition between the virtual world and the real world.

[0004] In XR scenarios, users can interact with electronic devices through gaze control, handheld hardware device (such as controller) control, gesture control, wearable device (such as wristband) control, voice control, etc., so as to control virtual objects in the virtual environment corresponding to the electronic device. Currently, with the development of motion capture technology, using gestures for control will become a more mainstream control method, but the existing gesture recognition technology has a low recognition accuracy and cannot meet user needs. Summary of the invention

[0005] The embodiments of the present application provide a gesture recognition method, apparatus, device, medium and program, which can improve the recognition accuracy of gestures.

[0006] In a first aspect, an embodiment of the present application provides a gesture recognition method, the method comprising:

[0007] Get the key point information of the gesture to be recognized;

[0008] Determining recognition information of the gesture to be recognized according to key point information of the gesture to be recognized, wherein the recognition information includes one or more of the following: hand posture, hand orientation, and contact relationship of hand bones;

[0009] The gesture type of the gesture to be recognized is determined according to the recognition information.

[0010] In some embodiments, the hand posture parameter includes at least one of the following parameters: a first bending parameter of each finger, a second bending parameter and an abduction parameter of each finger;

[0011] The first bending parameter is a bending parameter of the metacarpophalangeal joint, the second bending parameter is a bending parameter of the distal joint of each finger, and the abduction parameter is used to describe the opening and closing state between two fingers.

[0012] In some embodiments, when the recognition information of the gesture to be recognized includes the posture of the hand, determining the recognition information of the gesture to be recognized according to the key point information of the gesture to be recognized includes:

[0013] Determining the values ​​of the hand gesture parameters according to the key point information of the gesture to be recognized;

[0014] The step of determining the gesture type of the gesture to be recognized according to the recognition information of the gesture to be recognized includes:

[0015] Comparing the value of the hand gesture parameter with the threshold of the gesture parameter of the preset gesture;

[0016] The gesture type of the gesture to be recognized is determined according to the comparison result.

[0017] In some embodiments, the first bending parameter and the second bending parameter include identification information of whether to bend, and the opening and closing parameter includes identification information of whether to open and close.

[0018] In some embodiments, the first bending parameter and the second bending parameter include bending angles, and the opening and closing parameter includes an opening and closing distance.

[0019] In some embodiments, when the recognition information of the gesture to be recognized includes the contact relationship of the bones of the hand, determining the recognition information of the gesture to be recognized according to the key point information of the gesture to be recognized includes:

[0020] Determine the coordinates of the skeleton of the gesture to be recognized according to the key point information of the gesture to be recognized;

[0021] According to the coordinates of the bones of the gesture to be recognized, the contact relationship between the bones of the hand is determined.

[0022] In some embodiments, the skeleton of the hand includes 23 bones, wherein the little finger and thumb each include 5 bones, the index finger, middle finger and ring finger each include 4 bones, and the wrist includes 1 bone.

[0023] In some embodiments, when the recognition information of the gesture to be recognized includes the direction of the hand, determining the recognition information of the gesture to be recognized according to the key point information of the gesture to be recognized includes:

[0024] Determining the starting position and detection direction of the gesture to be recognized according to the key point information of the gesture to be recognized;

[0025] Establishing a three-dimensional coordinate system according to the starting position of the gesture to be recognized and the detection direction;

[0026] Mapping a reference direction of a preset gesture into the coordinate system to obtain a mapping direction of the reference direction in the coordinate system;

[0027] The mapping direction and the detection direction are compared to obtain the orientation of the hand.

[0028] In some embodiments, mapping the reference direction of the preset gesture to the coordinate system to obtain the mapping direction of the reference direction in the coordinate system includes:

[0029] The reference direction is projected into a preset plane of the coordinate system to obtain a projection direction, where the projection direction is a mapping direction of the reference direction in the coordinate system.

[0030] In some embodiments, comparing the mapping direction and the detection direction to obtain the orientation of the hand includes:

[0031] Determining an angle between the mapping direction and the detection direction;

[0032] When the included angle is less than or equal to a preset angle, the direction of the hand is determined to be the reference direction.

[0033] In some embodiments, the starting position of the gesture to be recognized is any one of the following positions: a palm position, a wrist position, a position corresponding to the palm position plus a first offset, and a position corresponding to the wrist position plus a second offset.

[0034] In some embodiments, the detection direction is any one of the following directions: the direction pointed by four fingers except the thumb, the direction the palm faces, and the direction pointed by the thumb.

[0035] In some embodiments, the reference direction is any one of the following directions: a direction facing the face, a direction away from the face, a direction toward the sky, and a direction toward the ground.

[0036] In some embodiments, determining the gesture type of the gesture to be recognized according to the recognition information of the gesture to be recognized includes:

[0037] Get the recognition result of the gesture type at the previous moment;

[0038] If the recognition result of the gesture type at the current moment is that the gesture type at the previous moment is different from the preset gesture type, the threshold of the recognition information of the preset gesture type is increased by a threshold width to obtain a target threshold range of the recognition information of the preset gesture type, wherein the preset gesture types at the previous moment and the current moment are the same;

[0039] Determining the gesture type of the gesture to be recognized according to the value of the recognition information of the gesture to be recognized and the target threshold range of the recognition information of the preset gesture type;

[0040] If the recognition result of the gesture type at the current moment is that the gesture type at the previous moment is the same as the preset gesture type, the gesture type of the gesture to be recognized is determined according to the value of the recognition information of the gesture to be recognized and the threshold of the recognition information of the preset gesture type.

[0041] In some embodiments, after determining the gesture type of the gesture to be recognized according to the recognition information of the gesture to be recognized, the method further includes:

[0042] Obtaining the duration of the gesture type of the gesture to be recognized;

[0043] When the duration of the gesture type of the gesture to be recognized is less than the preset duration, determining that the gesture type of the gesture to be recognized is not effective;

[0044] When the duration of the gesture type of the gesture to be recognized is greater than or equal to the preset duration, it is determined that the gesture type of the gesture to be recognized is effective.

[0045] In some embodiments, the method further comprises:

[0046] When the gesture type of the gesture to be recognized is recognized for the first time, starting to time the duration of the gesture type of the gesture to be recognized;

[0047] When it is detected that the gesture type of the gesture to be recognized has changed, the timing of the duration of the gesture type of the gesture to be recognized is stopped.

[0048] In some embodiments, obtaining key point information of the gesture to be recognized includes:

[0049] Get image data;

[0050] Using a hand detection algorithm to detect whether a hand exists in the image data;

[0051] When a hand exists in the image data, a key point detection algorithm is used to detect key point information of the hand.

[0052] In some embodiments, obtaining key point information of the gesture to be recognized includes:

[0053] Acquire sensor data collected by a wearable device on the hand, wherein the sensor data includes motion data of joint points of the hand;

[0054] Determine key point information of the gesture to be recognized according to the sensor data.

[0055] On the other hand, an embodiment of the present application provides a gesture recognition device, the device comprising:

[0056] An acquisition module is used to obtain key point information of the gesture to be recognized;

[0057] A determination module, configured to determine recognition information of the gesture to be recognized based on key point information of the gesture to be recognized, wherein the recognition information includes one or more of the following: hand posture, hand orientation, and contact relationship of hand bones;

[0058] The recognition module is used to determine the gesture type of the gesture to be recognized according to the recognition information.

[0059] On the other hand, an embodiment of the present application provides an electronic device, comprising: a processor and a memory, wherein the memory is used to store a computer program, and the processor is used to call and run the computer program stored in the memory to execute any of the methods described above.

[0060] On the other hand, an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium is used to store a computer program, and the computer program enables a computer to execute any of the methods described above.

[0061] On the other hand, an embodiment of the present application provides a computer program product, including a computer program, which implements any of the methods described above when executed by a processor.

[0062] The gesture recognition method, apparatus, device, medium and program provided in the embodiments of the present application include: obtaining key point information of the gesture to be recognized, determining recognition information of the gesture to be recognized based on the key point information of the gesture to be recognized, and the recognition information includes one or more of the following: hand posture, hand orientation and contact relationship of hand bones, and determining the gesture type of the gesture to be recognized based on the recognition information. The recognition method can quickly and accurately recognize gestures. BRIEF DESCRIPTION OF THE DRAWINGS

[0063] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0064] Figure 1 This is the process of the gesture recognition method provided in Example 1 of the present application;

[0065] Figure 2 is a schematic diagram of the flexion state of the finger;

[0066] Figure 3 Schematic diagram of the bending state of the finger;

[0067] Figure 4 To show intention for the appearance of the finger;

[0068] Figure 5 This is a contact diagram of the bones of the hand;

[0069] Figure 6 A flowchart of a gesture recognition method provided in Embodiment 2 of the present application;

[0070] Figure 7 Schematic diagram of hand posture parameters;

[0071] Figure 8 A flowchart of a method for determining the direction of a hand provided in Embodiment 3 of the present application;

[0072] Fig. 9 A schematic diagram of a coordinate system;

[0073] Fig.10 A schematic diagram of the reference direction;

[0074] Fig.11 Another schematic diagram for reference direction;

[0075] Fig.12 This is a schematic diagram after the reference direction is mapped to the coordinate system;

[0076] Fig.13 It is a schematic diagram of the projection of the reference direction into the plane of the coordinate system;

[0077] Fig.14 A flowchart of a gesture recognition method provided in Embodiment 4 of the present application;

[0078] Fig.15 A schematic diagram of the bones of the hand;

[0079] Fig.16 A flowchart of the anti-shake method provided in Embodiment 5 of the present application;

[0080] Fig.17 Schematic diagram of the threshold range of gestures before and after adopting the anti-shake strategy;

[0081] Fig.18 A flowchart of the anti-shake method provided in Embodiment 6 of the present application;

[0082] Fig.19 A schematic diagram of the structure of a gesture recognition device provided in Embodiment 7 of the present application;

[0083] Fig. 20 A structural schematic diagram of an electronic device provided in Example 8 of the present application. DETAILED DESCRIPTION

[0084] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0085] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product, or server that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products, or devices.

[0086] The embodiments of the present application provide a gesture recognition method, which can be applied to electronic devices, servers that perform cloud rendering on electronic devices, smart wearable devices, smart homes, and any other electronic devices that can be controlled by gestures.

[0087] XR is a general term for VR, AR and AR technology, and electronic devices include but are not limited to VR devices, AR devices and MR devices. At present, electronic devices use one or more interaction methods such as gaze control, handheld hardware device (such as handle) control, gesture control, wearable device (such as wristband) control, voice control, etc. When using gesture control, it is necessary to accurately identify the gesture type. By using the gesture recognition method of the embodiment of the present application for gesture recognition, the accuracy of gesture recognition can be improved, and more types of gestures can be recognized.

[0088] In order to facilitate understanding of the embodiments of the present application, before describing each embodiment of the present application, some concepts involved in all embodiments of the present application are first appropriately explained, as follows:

[0089] VR: Technology for creating and experiencing a virtual world. It generates a virtual environment by computation. It is a multi-source information (the virtual reality mentioned in this article includes at least visual perception, and can also include auditory perception, tactile perception, motion perception, and even taste perception, olfactory perception, etc.). It realizes the simulation of the fusion and interactive three-dimensional dynamic vision and entity behavior of the virtual environment, immersing users in a simulated virtual reality environment, and realizing applications in various virtual environments such as maps, games, videos, education, medical care, simulation, collaborative training, sales, assisted manufacturing, maintenance and repair.

[0090] VR device refers to a terminal that realizes virtual reality effects, which can usually be provided in the form of glasses, helmet-mounted displays (Head Mount Display, HMD), and contact lenses to realize visual perception and other forms of perception. Of course, the form of virtual reality devices is not limited to this, and can be further miniaturized or enlarged as needed.

[0091] AR: AR scenery refers to a simulated scenery in which at least one virtual object is superimposed on a physical scenery or a representation thereof. For example, an electronic system may have an opaque display and at least one imaging sensor for capturing images or videos of a physical scenery, which are representations of the physical scenery. The system combines the image or video with the virtual object and displays the combination on the opaque display. An individual uses the system to indirectly view the physical scenery via an image or video of the physical scenery and observes the virtual object superimposed on the physical scenery. When the system uses one or more image sensors to capture images of the physical scenery and uses those images to present the AR scenery on an opaque display, the displayed image is referred to as video transmission. Alternatively, an electronic system for displaying an AR scenery may have a transparent or translucent display through which an individual can directly view the physical scenery. The system may display virtual objects on a transparent or translucent display so that an individual uses the system to observe virtual objects superimposed on the physical scenery. As another example, the system may include a projection system that projects virtual objects into a physical scenery. Virtual objects can be projected, for example, on a physical surface or as a hologram, so that individuals using the system observe virtual objects superimposed on a physical setting. Specifically, a technology that calculates the camera's posture parameters in the real world (or three-dimensional world, real world) in real time during the process of the camera capturing images, and adds virtual elements to the images captured by the camera based on the camera posture parameters. Virtual elements include, but are not limited to: images, videos, and three-dimensional models. The goal of AR technology is to integrate the virtual world into the real world on the screen for interaction.

[0092] MR: By presenting virtual scene information in the real scene, an interactive feedback information loop is established between the real world, the virtual world and the user to enhance the realism of the user experience. For example, computer-created sensory input (e.g., virtual objects) is integrated with sensory input from the physical scene or its representation in the simulated scene. In some MR scenes, the computer-created sensory input can adapt to changes in sensory input from the physical scene. In addition, some electronic systems used to present MR scenes can monitor the orientation and / or position relative to the physical scene so that virtual objects can interact with real objects (i.e., physical elements from the physical scene or their representations). For example, the system can monitor movement so that virtual plants appear stationary relative to physical buildings.

[0093] Virtual reality devices (VR devices), terminals for realizing virtual reality effects, can usually be provided in the form of glasses, head mount displays (Head Mount Display, referred to as HMD), contact lenses, for realizing visual perception and other forms of perception. Of course, the form of virtual reality devices is not limited to this, and can be further miniaturized or enlarged according to actual needs.

[0094] Optionally, the virtual reality device (i.e., electronic device) described in the embodiments of the present application may include but is not limited to the following types:

[0095] 1) Mobile virtual reality devices support the configuration of mobile terminals (such as smart phones) in various ways (such as head-mounted displays with dedicated card slots). Through wired or wireless connection with the mobile terminal, the mobile terminal performs relevant calculations of virtual reality functions and outputs data to the mobile virtual reality device, such as watching virtual reality videos through the mobile terminal's APP.

[0096] 2) All-in-one virtual reality devices have a processor for performing relevant calculations of virtual functions, and thus have independent virtual reality input and output functions. They do not need to be connected to a PC or mobile terminal and have a high degree of freedom in use.

[0097] 3) Computer-based virtual reality (PCVR) devices use the PC to perform relevant calculations and data output for virtual reality functions. External computer-based virtual reality devices use the data output by the PC to achieve virtual reality effects.

[0098] Gesture recognition is generally divided into the following two types:

[0099] Gesture recognition based on external devices, for example, using wearable devices to directly detect the angles and spatial positions of each joint of the hand and arm. The wearable device sends the measured gesture data to the gesture recognition system for recognition. Typical wearable devices include data gloves, which collect motion data of each joint of the hand and send it to the recognition system.

[0100] Another example is the optical marking method, which replaces data gloves by wearing optical markers on human hands, and transmits changes in the position of human hands and fingers to the recognition system through infrared rays.

[0101] Vision-based gesture recognition: Visual gesture recognition refers to the process of processing image sequences containing gestures captured by video acquisition devices through computer vision technology to identify gestures.

[0102] The development of vision-based gesture recognition technology is a process from two-dimensional (2D) to three-dimensional (3D). Two-dimensional hand shape recognition can only recognize a few static gestures, such as clenching a fist or spreading five fingers, but cannot perceive the "continuous changes" of gestures. Compared with two-dimensional gesture recognition, three-dimensional gesture recognition adds a Z-axis information, which can recognize various hand shapes, gestures and movements.

[0103] Figure 1This is the process of the gesture recognition method provided in the first embodiment of the present application, which is applied in an electronic device, which may be an electronic device, a server for rendering the electronic device, a smart home device, a smart wearable device, etc. Figure 1 As shown, the method provided in this embodiment includes the following steps.

[0104] S101, obtaining key point information of a gesture to be recognized.

[0105] The method of this embodiment can be applied to the above-mentioned gesture recognition based on external devices and the gesture recognition scenario based on vision, and is mainly aimed at 3D gesture recognition.

[0106] In a vision-based gesture recognition scenario, for example, the key point information of the gesture to be recognized can be obtained in the following manner: the electronic device obtains image data, and then uses a hand detection algorithm to detect whether there is a hand in the image data; when a hand exists in the image data, the key point detection algorithm is used to detect the key point information of the hand.

[0107] For example, image data is collected through the binocular camera of a VR device. The image data may include hands or may not include hands. It is necessary to further detect whether there are hands in the image data. When there are hands in the image, a region of interest (ROI) detection is performed to obtain the hand area and further extract the key points of the hand.

[0108] In a gesture recognition scenario based on an external device, for example, the key point information of the gesture to be recognized can be obtained in the following manner: sensor data collected by a wearable device on the hand is obtained, the sensor data including motion data of the joints of the hand, and the key point information of the gesture to be recognized is determined based on the sensor data.

[0109] The key point information includes the coordinates of the key point, which are the three-dimensional coordinates of the key point, x, y, and z, and the coordinates of the key point are the coordinates in the world coordinate system.

[0110] It can be understood that the above are just two examples of the methods for obtaining the key point information of the gesture to be recognized, and do not constitute a limitation. The key point information of the gesture to be recognized can also be obtained by any existing method.

[0111] S102: Determine recognition information of the gesture to be recognized based on key point information of the gesture to be recognized, where the recognition information includes one or more of the following: hand posture, hand orientation, and contact relationship of hand bones.

[0112] The hand posture is used to describe the bending of fingers, the opening and closing of fingers, etc. Exemplarily, the hand posture parameters include at least one of the following parameters: a first bending parameter of each finger, a second bending parameter of each finger, and an abduction parameter. Among them, the first bending parameter is the bending parameter of the metacarpophalangeal joint, the second bending parameter is the bending parameter of the distal joint of each finger, and the abduction parameter is used to describe the opening and closing state between two fingers.

[0113] The bending of the metacarpophalangeal joints of the fingers corresponds to the flexion of the fingers in anatomy, see Figure 2 , Figure 2 (a) is the flexion state of the finger, θ represents the flexion angle of the finger, and the value range of the flexion angle θ of the finger can be 90-180 degrees, which is determined by the maximum flexion degree of the finger. The maximum bending between the finger and the palm is usually 90 degrees, that is, the palm and the finger are 90 degrees. Figure 2 (b) is a schematic diagram of θ when the fingers are stretched out. The stretched out finger state can be understood as the flexion angle of the finger is 180 degrees.

[0114] The flexion of the distal joints of the fingers corresponds to the anatomical curl of the fingers, see Figure 3 , Figure 3 (a) is the bending state of the finger, θ represents the bending angle of the finger, and the value range of the bending angle θ of the finger can be 0-180 degrees. Figure 3 (b) is a schematic diagram of θ when the finger is in the straight state. The straight state of the finger can be understood as the bending angle of the finger is 180 degrees.

[0115] Abduction of the fingers means that the fingers are separated and there is a certain distance between them. Figure 4 , Figure 4 To indicate the intention of the finger.

[0116] The direction of the hand is used to describe the direction of the hand. For example, the direction of the hand is defined as facing the face, away from the face, toward the sky, or toward the ground. It can be understood that the direction of the hand includes but is not limited to the four directions listed. Different from the prior art, the direction of the hand in the embodiment of the present application is determined by the following three parameters: Axis, Space + Target, where Space is a three-dimensional space and Target is the reference direction expected by the gesture, including but not limited to facing the face, away from the face, toward the sky, or toward the ground.

[0117] The contact relationship of the hand bones is used to describe the relative relationship between different parts of the hand, that is, whether two parts of the hand are in contact. For example, when a user makes a thumb and index finger pinch gesture, the bones of the thumb and index finger are in contact. Figure 5In the gesture shown, the user forms a heart shape with both hands, and the fingers of the left and right hands are in contact with each other, that is, the thumbs of the left and right hands are in contact, the index fingers of the left and right hands are in contact, and the middle fingers of the left and right hands are in contact.

[0118] S103: Determine the gesture type of the gesture to be recognized according to the recognition information.

[0119] In one implementation, the value of the identification information of the gesture to be identified is compared with the threshold of the predefined identification information. If the difference between the value of the identification information of the gesture to be identified and the threshold of the predefined identification information is less than a certain error range, it is considered that the type of the gesture to be identified is the same as the type of the preset gesture corresponding to the threshold of the predefined identification information. The predefined threshold of the identification information can be a specific value or a value range.

[0120] After determining the type of gesture to be recognized, a corresponding operation is performed. For example, when the type of gesture to be recognized is the same as the type of the preset gesture, the object in the virtual space is controlled to move, open, etc. When the type of gesture to be recognized is different from the type of the preset gesture, no operation may be performed, and the gesture at the next moment may be recognized, or the user may be prompted that the gesture recognition is wrong, or the target gesture is not recognized, or the user may be prompted to adjust the gesture.

[0121] In this embodiment, the electronic device obtains key point information of the gesture to be recognized, and determines recognition information of the gesture to be recognized based on the key point information of the gesture to be recognized, and the recognition information includes one or more of the following: hand posture, hand orientation, and contact relationship of hand bones. Based on the recognition information, the gesture type of the gesture to be recognized is determined. The recognition method can quickly and accurately recognize gestures.

[0122] Based on the first embodiment, the second embodiment of the present application provides a gesture recognition method, which performs gesture recognition based on the posture of the hand. Figure 6 This is a flowchart of a gesture recognition method provided in Embodiment 2 of the present application, such as Figure 6 As shown, the gesture recognition method includes the following steps.

[0123] S201: Obtain key point information of the gesture to be recognized.

[0124] S202. Determine the value of the hand posture parameter according to the key point information of the gesture to be recognized, where the posture parameter includes at least one of the following parameters: a bending parameter of the metacarpophalangeal joint of each finger, a bending parameter and an abduction parameter of the distal joint of each finger.

[0125] Figure 7 is a schematic diagram of the hand posture parameters, such as Figure 7As shown, the posture of the hand is defined by the posture parameters of the five fingers of the hand. The parameters of each finger except the little finger include: flexion parameter (i.e., first bending parameter), bending parameter (i.e., second bending parameter) and abduction parameter; the little finger only has flexion parameter and bending parameter, no abduction parameter, or the value of the abduction parameter of the little finger is empty. It can be understood that when the abduction parameter is determined based on the little finger, the thumb has no abduction parameter, or the value of the abduction parameter of the thumb is empty, and the other four fingers all have abduction parameters.

[0126] In one implementation, the first bending parameter and the second bending parameter include identification information of whether to bend, and the abduction parameter includes identification information of whether to open or close. In the hand posture defined in this way, the bending angle and the opening and closing interval have little influence on the posture.

[0127] Taking the first bending parameter as an example, illustratively, when determining whether to bend, select at least three key points, for example, the key points at the metacarpophalangeal joints of the fingers, the key points between the metacarpophalangeal joints and the distal joints, and the key points between the metacarpophalangeal joints and the palms, and determine whether these three key points are in a straight line. If they are in a straight line, it means that the metacarpophalangeal joints of the fingers are not bent. If they are not in a straight line, it means that the metacarpophalangeal joints of the fingers are bent.

[0128] In another implementation, the first bending parameter and the second bending parameter include bending angles, and the abduction parameter includes an opening and closing distance, that is, the bending angles of the first bend and the second bend need to be determined, and different bending angles may correspond to different gestures.

[0129] Taking the first bending parameter as an example, illustratively, when calculating the bending angle, at least three key points are selected. illustratively, the key points at the metacarpophalangeal joints of the fingers, the key points between the metacarpophalangeal joints and the distal joints, and the key points between the metacarpophalangeal joints and the palms are selected, and the coordinates of the selected key points are used to calculate the bending angle of the metacarpophalangeal joints.

[0130] Exemplarily, when calculating the opening and closing distance between two fingers, you can select the key points of the two fingers at the same position, calculate the distance between the two key points, and use the calculated distance as the opening and closing distance. For example, select the distal joint points of the index finger and the middle finger, calculate the distance between the distal joint points of the index finger and the middle finger, and use the calculated distance as the opening and closing distance between the index finger and the middle finger.

[0131] S203: Compare the values ​​of the hand gesture parameters with the thresholds of the preset gesture gesture parameters, and determine the gesture type of the gesture to be recognized according to the comparison result.

[0132] The threshold of the posture parameter of the preset gesture is predefined. The preset gesture can be understood as a gesture with known posture parameters. The value of the posture parameter of the gesture to be recognized is compared with the threshold of the posture parameter of the preset gesture to determine whether the two are consistent. The threshold of the posture parameter of the preset gesture can be a specific value or a value range.

[0133] Exemplarily, when the hand posture parameter is the first bending angle of each finger, the first bending angle of each finger of the gesture to be recognized is compared with the first bending angle range of each finger of the preset gesture to obtain a first comparison result. If the first bending angle of each finger of the gesture to be recognized falls within the first bending angle range of each finger of the preset gesture, it is determined that the gesture type of the gesture to be recognized is the gesture type of the preset gesture.

[0134] When the hand posture parameter is the second bending angle of each finger, the second bending angle of each finger of the gesture to be recognized is compared with the second bending angle range of each finger of the preset gesture to obtain a second comparison result. If the second bending angle of each finger of the gesture to be recognized falls within the second bending angle range of each finger of the preset gesture, it is determined that the gesture type of the gesture to be recognized is the gesture type of the preset gesture.

[0135] When the hand gesture parameter is the opening and closing spacing of each finger, the opening and closing spacing of each finger of the gesture to be recognized is compared with the opening and closing spacing range of each finger of the preset gesture to obtain a third comparison result. If the opening and closing spacing of each finger of the gesture to be recognized falls within the opening and closing spacing range of each finger of the preset gesture, it is determined that the gesture type of the gesture to be recognized is the gesture type of the preset gesture.

[0136] When the hand posture parameter is the bending mark of the first bend of each finger, the bending mark of the first bend of each finger of the gesture to be identified is compared with the bending mark of the first bend of each finger of the preset gesture. If the bending marks of the two are the same, the gesture type of the gesture to be identified is determined to be the gesture type of the preset gesture. Similarly, when the hand posture parameter is the bending mark of the second bend of each finger or the opening and closing mark of each finger, the bending mark or the opening and closing mark is compared, which will not be explained in detail here.

[0137] In this embodiment, the first bending parameter of each finger, the second bending parameter of each finger and the abduction parameter of each finger can each independently define a gesture, or any two or three parameters can be combined to define a gesture.

[0138] It should be noted that, in some embodiments of the present application, the hand posture parameters include the following three types of parameters: a first bending parameter of each finger, a second bending parameter and an abduction parameter of each finger. However, in different gestures, the values ​​of some parameters may not be taken into account. For example, the hand posture parameter mainly used in gesture 1 is the first bending parameter of each finger. When comparing the value of the hand posture parameter with the threshold of the posture parameter of the preset gesture, only the value of the first bending parameter of each finger is compared with the threshold of the first bending parameter of the preset gesture, and the value of the second bending parameter and the value of the abduction parameter of each finger do not need to be compared. Accordingly, the values ​​of the second bending parameter and the abduction parameter of each finger can be set to arbitrary values.

[0139] The method of this embodiment determines the value of the hand posture parameter based on the key point information of the gesture to be recognized, and the posture parameter includes at least one of the following parameters: the bending parameter of the metacarpophalangeal joint of each finger, the bending parameter and abduction parameter of the distal joint of each finger, and compares the value of the hand posture parameter with the threshold of the gesture posture parameter of the preset gesture, and determines the gesture type of the gesture to be recognized based on the comparison result. The bending parameters of the metacarpophalangeal joint of each finger, the bending parameters and abduction parameters of the distal joint of each finger can define a large number of gestures alone or in combination, basically covering the common gestures in life, and enriching the gesture types that the device can recognize.

[0140] Based on the first embodiment, the third embodiment of the present application provides a method for determining the direction of a hand, which is used to explain the step S102 in the first embodiment in detail. Figure 8 This is a flow chart of a method for determining the direction of a hand provided in Embodiment 3 of the present application, such as Figure 8 As shown, the method for determining the direction of the hand includes the following steps.

[0141] S301 : Determine the starting position and detection direction of the gesture to be recognized according to the key point information of the gesture to be recognized.

[0142] The change of the hand direction mainly involves the following three parts: Axis (coordinate system) content, Space (space) content and Target (target) content.

[0143] The Axis content is used to establish a coordinate system for the gesture to be recognized. Axis defines the origin and direction of the coordinate system. The Target content is used to predefine a reference direction for the gesture, which is the expected direction of the gesture. The Space content is used to map the reference direction to the established coordinate system, and use the mapped direction and the detection direction of the gesture to be recognized to determine the direction of the hand of the gesture to be recognized. The detection direction is the direction defined in the Axis content.

[0144] The Axis (coordinate system) content includes position and direction. The position is the starting position of the gesture, and the direction is the detection direction of the gesture.

[0145] Exemplarily, the starting position of the gesture may be any one of the following positions: the palm position, the wrist position, the position corresponding to the palm position plus the first offset (offset), and the position corresponding to the wrist position plus the second offset (offset). The first offset and the second offset may be the same or different, and the sizes of the first offset and the second offset may be set as needed.

[0146] Exemplarily, the detection direction may be any one of the following directions: the direction pointed by the four fingers except the thumb, the direction the palm faces, and the direction pointed by the thumb. The direction pointed by the four fingers except the thumb refers to the direction of the fingers facing or away from the wrist, and similarly, the direction pointed by the thumb refers to the direction of the thumb facing or away from the wrist.

[0147] After obtaining the key point information of the gesture to be recognized, the palm position or wrist position of the gesture to be recognized and the detection direction of the gesture to be recognized can be obtained from the key point information of the gesture to be recognized. If the detection direction of the gesture is defined in the Axis content as the direction pointed by the four fingers except the thumb, the direction pointed by the four fingers except the thumb in the gesture to be recognized is determined according to the key point information of the gesture to be recognized, and the direction is used as the detection direction of the gesture to be recognized.

[0148] S302: Establish a three-dimensional coordinate system according to the starting position and detection direction of the gesture to be recognized.

[0149] After obtaining the starting position and detection direction of the gesture to be identified, the starting position of the gesture to be identified is used as the origin of the coordinate system, and the detection direction is used as an axis of the coordinate system to establish a three-dimensional coordinate system. Different detection directions correspond to different axes. For example, when the detection direction is the direction pointed by the four fingers except the thumb, the detection direction is used as the direction of the X axis of the coordinate system; when the detection direction is the direction of the palm, the detection direction is used as the direction of the Y axis of the coordinate system; when the detection direction is the direction pointed by the thumb, the detection direction is used as the direction of the Z axis of the coordinate system.

[0150] After determining the origin of the coordinate system and the direction of one axis of the coordinate system, a coordinate system can be uniquely determined. For example, when the detection direction is the direction of the palm, when the origin is the palm, the detection direction is the direction of the Y axis of the coordinate system. After the direction of the Y axis is determined, the right-hand rule can be used to uniquely determine the directions of the Y axis and the Z axis.

[0151] Fig. 9 is a schematic diagram of the coordinate system, such as Fig. 9As shown, the starting position of the gesture is the palm, and the detection direction of the gesture can be any direction of the three axes of the coordinate system, wherein the X-axis (i.e., X-Fingers in the figure) is the direction pointed by the four fingers except the thumb, the Y-axis (i.e., Y-Palm in the figure) is the direction of the palm, and the Z-axis (i.e., Z-Thumb in the figure) is the direction pointed by the thumb.

[0152] This coordinate system can be understood as a rotating coordinate system that rotates with the rotation of the user's hand. Fig. 9 , assuming that the detection direction is the direction of the palm, and the detection direction is the Y axis of the coordinate system, but when the user's hand rotates, the direction of the Y axis also rotates with the hand, and the direction of the Y axis in the world coordinate system is different. However, in the gesture to be recognized, the coordinate system does not rotate.

[0153] S303: Map the reference direction of the preset gesture into the coordinate system to obtain the mapping direction of the reference direction in the coordinate system, compare the mapping direction with the detection direction, and obtain the direction of the hand.

[0154] The reference direction of the gesture is defined in the Target content. Exemplarily, the reference direction may be any one of the following directions: a direction facing the face, a direction away from the face, a direction toward the sky, and a direction toward the ground.

[0155] Fig.10 A schematic diagram of the reference direction, such as Fig.10 As shown, the direction facing the face and the direction away from the face are shown by arrows. The direction facing the face and the direction away from the face are directions relative to the user, which change with the user's posture. They are not absolute directions but relative directions. For example, when the user faces the electronic device and turns away from the electronic device, the direction facing the face is opposite, but for the user, the direction facing the face does not change.

[0156] Fig.11 Another schematic diagram of the reference direction is Fig.11 As shown, the direction toward the sky and the direction toward the ground are shown by arrows. The direction toward the sky and the direction toward the ground are directions in a world coordinate system, which are absolute directions and do not change with changes in the user's posture.

[0157] In one implementation, the reference direction is mapped to the coordinate system to obtain the mapping direction of the reference direction in the coordinate system. Fig.12 This is a schematic diagram of the reference direction mapped to the coordinate system, as shown in Fig.12As shown, the direction of ray f is the mapping direction of the reference direction in the coordinate system, and the mapping direction is a direction in a three-dimensional space. In the three-dimensional space, the mapping direction is compared with the detection direction of the gesture to be recognized to obtain the direction of the hand.

[0158] In another implementation, the reference direction is projected into a preset plane of the coordinate system to obtain a projection direction, which is a mapping direction of the reference direction in the coordinate system. The projection direction is compared with the detection direction of the gesture to be identified to obtain the direction of the hand.

[0159] The projection direction within the preset plane of the coordinate system is a direction within a two-dimensional plane, in which the projection direction is compared with the detection direction of the gesture to be recognized. The preset plane can be any one of the three planes formed by the three-dimensional coordinate system, thereby providing more dimensional direction detection.

[0160] Fig.13 This is a schematic diagram of the projection of the reference direction into the plane of the coordinate system. Fig.13 , Fig.13 (a) is a schematic diagram of projecting the reference direction onto the XY plane. In the figure, ray l is the mapping direction of the reference direction, and ray l1 is the projection direction of the reference direction. In this method, it is equivalent to comparing the reference direction and the detection direction only in the XY plane. Fig.13 (b) is a schematic diagram of projecting the reference direction onto the YZ plane. In the figure, ray m is the mapping direction of the reference direction, and ray m1 is the projection direction of the reference direction. In this method, it is equivalent to comparing the reference direction and the detection direction only in the YZ plane. Fig.13 (c) is a schematic diagram of the projection of the reference direction onto the XZ plane. In the figure, ray n is the mapping direction of the reference direction, and ray n1 is the projection direction of the reference direction. In this method, it is equivalent to comparing the reference direction and the detection direction only in the XZ plane.

[0161] In some scenarios, we only need to pay attention to the movement of the reference direction and the detection direction in a certain direction of the three-dimensional coordinate system, and ignore the movement in other directions. Then we can map the reference direction to the plane where the direction we need to pay attention to is located. For example, in a VR scenario, suppose there are three virtual characters in the virtual space, and the three virtual characters are respectively in front of the user, in front of the left, and in front of the right. The user needs to point his finger at the virtual character in front of him to Fig.13Take the coordinate system shown as an example. Assuming that the user is currently pointing to the virtual character in front of him, if the user moves his finger left or right, the finger may point to the virtual characters in front of the left or right. If the user moves his finger up or down, no matter whether the user points to the head, body or feet of the virtual character in front of him, the direction pointed by the user is considered to be in front of him. In this scenario, when detecting the direction pointed by the user's finger, it is not necessary to pay attention to the movement of the finger in the up and down direction (i.e., the Y direction), that is, the finger can move arbitrarily in the Y direction, and it is only necessary to pay attention to the movement of the finger in the X and Z directions. Therefore, the direction of the finger can be projected into the XZ plane to determine whether the projection direction in the XZ plane points to the front.

[0162] Optionally, the direction of the hand can be obtained in the following manner: determine the angle between the mapping direction and the detection direction, and when the angle is less than or equal to a preset angle, determine the direction of the hand as a reference direction. The preset angle can be understood as the allowable error range between the actual direction of the gesture and the reference direction. If the angle between the two is less than or equal to the preset angle, the reference direction is determined as the direction of the hand. If the angle between the two is greater than the preset angle, determine that the direction of the hand is not the reference direction.

[0163] In this embodiment, the starting position and detection direction of the gesture to be identified are determined based on the key point information of the gesture to be identified, a three-dimensional coordinate system is established based on the starting position and detection direction of the gesture to be identified, the reference direction of the preset gesture is mapped into the coordinate system, the mapping direction of the reference direction in the coordinate system is obtained, the mapping direction and the detection direction are compared, and the direction of the hand is obtained. By placing the detection direction and the reference direction of the gesture in three-dimensional space for comparison, the accuracy of the hand direction detection is improved. In addition, by projecting the reference direction into a preset plane of the coordinate system and comparing the reference direction and the detection direction in the preset plane, more dimensional direction detection is provided.

[0164] Based on the first embodiment, the fourth embodiment of the present application provides a gesture recognition method, which performs gesture recognition based on the contact relationship of the hand bones. Fig.14 This is a flowchart of a gesture recognition method provided in Embodiment 4 of the present application, such as Fig.14 As shown, the gesture recognition method includes the following steps.

[0165] S401: Obtain key point information of the gesture to be recognized.

[0166] S402: Determine the coordinates of the skeleton of the gesture to be recognized based on the key point information of the gesture to be recognized, and determine the contact relationship between the skeletons of the hand based on the coordinates of the skeleton of the gesture to be recognized.

[0167] Fig.15 A schematic diagram of the skeleton of the hand, such as Fig.15As shown, the skeleton of the hand includes 23 bones, of which the little finger and thumb each include 5 bones, the index finger, middle finger and ring finger each include 4 bones, and the wrist includes 1 bone. It can be understood that Fig.15 This is just a schematic diagram. The skeleton of the hand may also include more or fewer bones, and the embodiments of the present application are not limited to this.

[0168] The key points of the hand may correspond to the bones one-to-one. In this way, the coordinates of the key points can be directly used as the coordinates of the bones of the hand. The key points of the hand may not correspond to the bones one-to-one. The coordinates of the bones need to be determined based on the coordinates of the key points of the hand.

[0169] The positional relationship between the bones can be determined based on the coordinates of the bones, and based on the positional relationship, it can be determined whether the bones are in contact.

[0170] S403: Determine the gesture type of the gesture to be recognized according to the contact relationship between the bones of the hand of the gesture to be recognized.

[0171] In some gestures, the bones of the hand touch each other, for example, when the user makes a pinching gesture with the thumb and index finger. Fig.15 Taking the bone structure as an example, the bones at the fingertips of the thumb and index finger are in contact, that is, the two bones at the tips of the index finger and thumb are in contact.

[0172] In this embodiment, the coordinates of the skeleton of the gesture to be recognized are determined based on the key point information of the gesture to be recognized, the contact relationship between the bones of the hand is determined based on the coordinates of the skeleton of the gesture to be recognized, the gesture type of the gesture to be recognized is determined based on the contact relationship between the bones of the hand of the gesture to be recognized, and the gesture is recognized based on the positional relationship of the hand bones, thereby enriching the types of gesture recognition, so that more gesture types can be recognized.

[0173] It should be noted that the methods of embodiments 2 to 4 can be used separately for gesture recognition, or can be combined arbitrarily for gesture recognition. For example, gestures can be recognized based on the direction of the hand and the contact relationship of the hand bones, or gestures can be recognized based on the hand posture and the direction of the hand, thereby being able to recognize more gesture types.

[0174] Some gestures are relatively close to each other. For example, gesture A and gesture B are two similar gestures. When performing gesture recognition, when the user's gesture is close to the critical point of gesture A and gesture B, jitter will occur between the data of the two gestures, resulting in incorrect gesture recognition results or not meeting user expectations.

[0175] The data jitter of the above two gestures may be introduced by the errors of user posture and the errors captured by the algorithm. The errors of user posture are due to the different habits and other factors of different users when performing gesture A, resulting in different postures or directions of gesture A. The errors captured by the algorithm include the errors introduced by the hand pictures taken by the user, the errors introduced when detecting the motion data of the hand, or the errors introduced when detecting the key points of the gesture, the errors introduced when detecting the posture of the hand, the direction of the hand and the contact relationship of the hand bones, etc. In the process of gesture recognition, each processing flow may bring errors.

[0176] In order to prevent the above-mentioned jitter of the data of the two gestures, an embodiment of the present application provides an anti-jitter strategy, which adjusts the threshold range of the gesture recognition information so that the threshold range of the gesture recognition information is larger, thereby making the gesture recognition result more accurate.

[0177] On the basis of the above embodiments, the fifth embodiment of the present application provides an anti-shake strategy in a gesture recognition method, which is used to perform anti-shake processing when determining the gesture type of the gesture to be recognized according to the recognition information of the gesture to be recognized, Fig.16 This is a flow chart of the anti-shake method provided in Embodiment 5 of the present application, such as Fig.16 As shown, the method provided in this embodiment includes the following steps.

[0178] S501: Obtain the recognition result of the gesture type at the previous moment.

[0179] In this embodiment, after the gesture type at each moment is recognized, the recognition result is saved, and the recognition result is used to indicate whether the recognized gesture type is the same as the preset gesture type.

[0180] S502: Determine whether the gesture type at the previous moment is the same as the preset gesture type.

[0181] According to the recognition result of the gesture type at the previous moment, determine whether the gesture type at the previous moment is the same as the preset gesture type. If the gesture type at the previous moment is the same as the preset gesture type, execute step S503; if the gesture type at the current moment is not the same as the preset gesture type, execute step S504.

[0182] S503: Determine the gesture type of the gesture to be recognized according to the value of the recognition information of the gesture to be recognized and the threshold of the recognition information of the preset gesture type.

[0183] In this embodiment, the preset gesture type at the previous moment and the current moment is the same, that is, the previous moment and the current moment are for identifying the same gesture type. When the gesture type at the previous moment is the same as the preset gesture type, it means that the recognition result is stable, and the same threshold as the previous moment is still used for recognition.

[0184] S504: Increase the threshold of the recognition information of the preset gesture type by a threshold width to obtain a target threshold range of the recognition information of the preset gesture type.

[0185] When the gesture type at the current moment is different from the preset gesture type, it means that the recognition result is unstable and jitter may occur, so a threshold width is added on the basis of the threshold at the previous moment. The threshold width is set in advance. By increasing the threshold width, the threshold range of the recognition information of the preset gesture type is larger when recognizing at the current moment, thereby making the recognition result more stable.

[0186] S505: Determine the gesture type of the gesture to be recognized according to the value of the recognition information of the gesture to be recognized and the target threshold range of the recognition information of the preset gesture type.

[0187] Fig.17 Schematic diagram of the threshold range of gestures before and after the anti-shake strategy is adopted, such as Fig.17 As shown in (a), the threshold range of gesture A is shown in the gray rectangular box, and the threshold range of gesture B is shown in the white rectangular box. When performing gesture recognition, if the recognition information of the gesture to be recognized exceeds the maximum value of the threshold range of gesture A (i.e., the position shown in the right border), the gesture to be recognized is recognized as gesture B. If the recognition information of the gesture to be recognized does not exceed the maximum value of the threshold range of gesture A, the gesture to be recognized is recognized as gesture A.

[0188] Fig.17 (b) Fig.17 On the basis of the threshold range shown in (a), a preset threshold width is added to the threshold range of gesture A. Then, when performing gesture recognition, when the recognition information of the gesture to be recognized exceeds the maximum value of the threshold width (i.e., the position shown by the right border of the increased threshold width), the gesture to be recognized is recognized as gesture B. When the recognition information of the gesture to be recognized does not exceed the maximum value of the threshold width, the gesture to be recognized is recognized as gesture A.

[0189] In the method of this embodiment, if the gesture type at the current moment is different from the preset gesture type, the threshold of the recognition information of the preset gesture type is increased by a threshold width to obtain a target threshold range for the recognition information of the preset gesture type. When identifying the gesture type at the current moment, the adjusted target threshold range is used for identification, thereby making the recognition result more stable.

[0190] In a gesture recognition scenario, when a user switches from a current gesture to a new gesture, there may be a transition gesture during the switching process. The transition gesture is not the gesture expected by the user, but the electronic device will still capture and recognize the transition gesture, which may cause the electronic device to make some erroneous instructions based on the transition gesture.

[0191] In order to avoid errors introduced by transition gestures during gesture switching, embodiment 6 of the present application provides an anti-shake strategy for performing the anti-shake processing of this embodiment after the electronic device determines the gesture type of the gesture to be recognized based on the recognition information of the gesture to be recognized.

[0192] Fig.18 This is a flow chart of the anti-shake method provided in Embodiment 6 of the present application, such as Fig.18 As shown, the method provided in this embodiment includes the following steps.

[0193] S601: Obtain the duration of the gesture type of the gesture to be recognized.

[0194] In this embodiment, a duration is introduced for the gesture. The duration of the gesture refers to the shortest length of time that the new gesture needs to be maintained when switching from the current gesture to the new gesture. In actual usage scenarios, the user's gestures are usually continuous and will last for a certain period of time. In the process of switching from the current gesture to the new gesture, the time of the introduced transition gesture is usually very short. Based on this, the duration of the gesture is introduced, and the duration of the gesture is used to determine whether the gesture is effective.

[0195] In this embodiment, when a new gesture is recognized, the duration of the new gesture starts to be counted, and the new gesture is continuously recognized and counted until the new gesture switches to another gesture, at which time the count stops. The timing of the new gesture after the switch starts, and the above process is repeated.

[0196] When the gesture type of the gesture to be recognized (as a new gesture) is recognized for the first time, the duration of the gesture type of the gesture to be recognized starts to be timed, and each time after gesture recognition is performed and a recognition result is obtained, the timing value of the duration of the gesture is read.

[0197] S602: Determine whether the duration of the gesture type of the gesture to be recognized is less than a preset duration.

[0198] A preset duration is configured for each gesture type, wherein the durations of the gesture types may be the same or different. When the duration of the gesture type of the gesture to be identified is less than the preset duration, step S603 is executed; when the duration of the gesture type of the gesture to be identified is greater than or equal to the preset duration, step S604 is executed.

[0199] S603: Determine that the gesture type of the gesture to be recognized is not valid.

[0200] When the gesture type of the gesture to be recognized is not effective, the instruction corresponding to the gesture type of the gesture to be recognized will not be executed, the electronic device continues to perform gesture recognition, and the duration of the gesture type of the gesture to be recognized continues to be timed.

[0201] S604: Determine whether the gesture type of the gesture to be recognized is effective.

[0202] When the duration of the gesture type of the gesture to be recognized is greater than or equal to the preset duration, it means that the gesture to be recognized lasts for a long time, and a transition gesture usually does not last so long, thereby excluding a transition gesture in the gesture switching process.

[0203] After steps S603 and S604, when it is detected that the gesture type of the gesture to be recognized has changed, the duration of the gesture type of the gesture to be recognized is stopped.

[0204] S605: Execute an instruction corresponding to the gesture type of the gesture to be recognized.

[0205] After determining that the gesture type of the gesture to be recognized is effective, execute the instruction corresponding to the gesture type of the gesture to be recognized. For example, in a VR device, when it is detected that the user has made a target gesture, control the object in the virtual reality space to move, or click a certain control, or display a preset UI.

[0206] In this embodiment, after determining the gesture type of the gesture to be identified, the duration of the gesture type of the gesture to be identified is obtained. When the duration of the gesture type of the gesture to be identified is less than the preset duration, it is determined that the gesture type of the gesture to be identified is not effective. When the duration of the gesture type of the gesture to be identified is greater than or equal to the preset duration, it is determined that the gesture type of the gesture to be identified is effective. During the gesture switching process, the duration of the transition gesture from the current gesture to the new gesture is short. The method of this embodiment can filter out the identified transition gesture, that is, the transition gesture will not be effective due to its short duration, thereby avoiding errors introduced by the transition gesture during the gesture switching process.

[0207] Embodiment 5 and embodiment 6 provide different anti-shake strategies for different shakes. During the gesture recognition process, the two anti-shake strategies can be used alone or in combination, and can be set according to specific usage scenarios.

[0208] In order to better implement the gesture recognition method of the embodiment of the present application, the embodiment of the present application also provides a gesture recognition device. Fig.19 This is a structural diagram of a gesture recognition device provided in Embodiment 7 of the present application, such as Fig.19As shown, the gesture recognition device 100 may include:

[0209] An acquisition module 11 is used to acquire key point information of the gesture to be recognized;

[0210] A determination module 12, configured to determine recognition information of the gesture to be recognized based on key point information of the gesture to be recognized, wherein the recognition information includes one or more of the following: hand posture, hand orientation, and contact relationship of hand bones;

[0211] The recognition module 13 is used to determine the gesture type of the gesture to be recognized according to the recognition information.

[0212] In some embodiments, the hand posture parameter includes at least one of the following parameters: a first bending parameter of each finger, a second bending parameter and an abduction parameter of each finger;

[0213] The first bending parameter is a bending parameter of the metacarpophalangeal joint, the second bending parameter is a bending parameter of the distal joint of each finger, and the abduction parameter is used to describe the opening and closing state between two fingers.

[0214] In some embodiments, when the identification information of the gesture to be identified includes the posture of the hand, the determination module 12 is specifically used to:

[0215] Determining the values ​​of the hand gesture parameters according to the key point information of the gesture to be recognized;

[0216] The identification module 13 is specifically used for:

[0217] Comparing the value of the hand gesture parameter with the threshold of the gesture parameter of the preset gesture;

[0218] The gesture type of the gesture to be recognized is determined according to the comparison result.

[0219] In some embodiments, the first bending parameter and the second bending parameter include identification information of whether to bend, and the opening and closing parameter includes identification information of whether to open and close.

[0220] In some embodiments, the first bending parameter and the second bending parameter include bending angles, and the opening and closing parameter includes an opening and closing distance.

[0221] In some embodiments, when the recognition information of the gesture to be recognized includes the contact relationship of the bones of the hand, the determination module 12 is specifically used to:

[0222] Determine the coordinates of the skeleton of the gesture to be recognized according to the key point information of the gesture to be recognized;

[0223] According to the coordinates of the bones of the gesture to be recognized, the contact relationship between the bones of the hand is determined.

[0224] In some embodiments, the skeleton of the hand includes 23 bones, wherein the little finger and thumb each include 5 bones, the index finger, middle finger and ring finger each include 4 bones, and the wrist includes 1 bone.

[0225] In some embodiments, when the identification information of the gesture to be identified includes the direction of the hand, the determination module 12 is specifically configured to:

[0226] Determining the starting position and detection direction of the gesture to be recognized according to the key point information of the gesture to be recognized;

[0227] Establishing a three-dimensional coordinate system according to the starting position of the gesture to be recognized and the detection direction;

[0228] Mapping a reference direction of a preset gesture into the coordinate system to obtain a mapping direction of the reference direction in the coordinate system;

[0229] The mapping direction and the detection direction are compared to obtain the orientation of the hand.

[0230] In some embodiments, the determination module 12 is specifically used to:

[0231] The reference direction is projected into a preset plane of the coordinate system to obtain a projection direction, where the projection direction is a mapping direction of the reference direction in the coordinate system.

[0232] In some embodiments, the determination module 12 is specifically used to:

[0233] Determining an angle between the mapping direction and the detection direction;

[0234] When the included angle is less than or equal to a preset angle, the direction of the hand is determined to be the reference direction.

[0235] In some embodiments, the starting position of the gesture to be recognized is any one of the following positions: a palm position, a wrist position, a position corresponding to the palm position plus a first offset, and a position corresponding to the wrist position plus a second offset.

[0236] In some embodiments, the detection direction is any one of the following directions: the direction pointed by four fingers except the thumb, the direction the palm faces, and the direction pointed by the thumb.

[0237] In some embodiments, the reference direction is any one of the following directions: a direction facing the face, a direction away from the face, a direction toward the sky, and a direction toward the ground.

[0238] In some embodiments, the identification module 13 is specifically used for:

[0239] Get the recognition result of the gesture type at the previous moment;

[0240] If the recognition result of the gesture type at the current moment is that the gesture type at the previous moment is different from the preset gesture type, the threshold of the recognition information of the preset gesture type is increased by a threshold width to obtain a target threshold range of the recognition information of the preset gesture type, wherein the preset gesture types at the previous moment and the current moment are the same;

[0241] Determining the gesture type of the gesture to be recognized according to the value of the recognition information of the gesture to be recognized and the target threshold range of the recognition information of the preset gesture type;

[0242] If the recognition result of the gesture type at the current moment is that the gesture type at the previous moment is the same as the preset gesture type, the gesture type of the gesture to be recognized is determined according to the value of the recognition information of the gesture to be recognized and the threshold of the recognition information of the preset gesture type.

[0243] In some embodiments, the identification module 13 is further used for:

[0244] Obtaining the duration of the gesture type of the gesture to be recognized;

[0245] When the duration of the gesture type of the gesture to be recognized is less than the preset duration, determining that the gesture type of the gesture to be recognized is not effective;

[0246] When the duration of the gesture type of the gesture to be recognized is greater than or equal to the preset duration, it is determined that the gesture type of the gesture to be recognized is effective.

[0247] In some embodiments, the identification module is further configured to:

[0248] When the gesture type of the gesture to be recognized is recognized for the first time, starting to time the duration of the gesture type of the gesture to be recognized;

[0249] When it is detected that the gesture type of the gesture to be recognized has changed, the timing of the duration of the gesture type of the gesture to be recognized is stopped.

[0250] In some embodiments, the acquisition module 11 is specifically used for:

[0251] Get image data;

[0252] Using a hand detection algorithm to detect whether a hand exists in the image data;

[0253] When a hand exists in the image data, a key point detection algorithm is used to detect key point information of the hand.

[0254] In some embodiments, the acquisition module 11 is specifically used for:

[0255] Acquire sensor data collected by a wearable device on the hand, wherein the sensor data includes motion data of joint points of the hand;

[0256] Determine key point information of the gesture to be recognized according to the sensor data.

[0257] It should be understood that the device embodiment and the method embodiment may correspond to each other, and similar descriptions may refer to the method embodiment. To avoid repetition, they will not be described here.

[0258] The above describes the device 100 of the embodiment of the present application from the perspective of the functional module in conjunction with the accompanying drawings. It should be understood that the functional module can be implemented in hardware form, can be implemented by instructions in software form, and can also be implemented by a combination of hardware and software modules. Specifically, the steps of the method embodiment in the embodiment of the present application can be completed by the hardware integrated logic circuit and / or software form instructions in the processor, and the steps of the method disclosed in the embodiment of the present application can be directly embodied as a hardware decoding processor to perform, or a combination of hardware and software modules in the decoding processor to perform. Optionally, the software module can be located in a mature storage medium in the field such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory, an electrically erasable programmable memory, a register, etc. The storage medium is located in a memory, and the processor reads the information in the memory, and completes the steps in the above method embodiment in conjunction with its hardware.

[0259] An embodiment of the present application also provides an electronic device. Fig. 20 A structural diagram of an electronic device provided in Embodiment 8 of the present application, such as Fig. 20 As shown, the electronic device 200 may include:

[0260] The memory 21 and the processor 22, the memory 21 is used to store the computer program and transmit the program code to the processor 22. In other words, the processor 22 can call and run the computer program from the memory 21 to implement the method in the embodiment of the present application.

[0261] For example, the processor 22 may be configured to execute the above method embodiments according to instructions in the computer program.

[0262] In some embodiments of the present application, the processor 22 may include but is not limited to:

[0263] General-purpose processor, digital signal processor (DSP), application-specific integrated circuit (ASIC), field programmable gate array (FPGA) or other programmable logic device, discrete gate or transistor logic device, discrete hardware components, etc.

[0264] In some embodiments of the present application, the memory 21 includes but is not limited to:

[0265] Volatile memory and / or non-volatile memory. Among them, the non-volatile memory can be read-only memory (ROM), programmable ROM (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM) or flash memory. The volatile memory can be random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of RAM are available, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link DRAM (SLDRAM), and direct RAM bus random access memory (DR RAM).

[0266] In some embodiments of the present application, the computer program may be divided into one or more modules, which are stored in the memory 21 and executed by the processor 22 to complete the method provided by the present application. The one or more modules may be a series of computer program instruction segments capable of completing specific functions, and the instruction segments are used to describe the execution process of the computer program in the electronic device.

[0267] like Fig. 20 As shown, the electronic device may further include: a transceiver 23 , which may be connected to the processor 22 or the memory 21 .

[0268] The processor 22 may control the transceiver 23 to communicate with other devices, specifically, to send information or data to other devices, or to receive information or data sent by other devices. The transceiver 23 may include a transmitter and a receiver. The transceiver 23 may further include an antenna, and the number of antennas may be one or more.

[0269] Understandable, although Fig. 20 Not shown, the electronic device 200 may also include a camera module, a wireless fidelity WIFI module, a positioning module, a Bluetooth module, a display, a controller, etc., which will not be repeated here.

[0270] It should be understood that the various components in the electronic device are connected via a bus system, wherein the bus system includes not only a data bus but also a power bus, a control bus and a status signal bus.

[0271] The present application also provides a computer storage medium on which a computer program is stored, and when the computer program is executed by a computer, the computer can perform the method of the above method embodiment. In other words, the present application embodiment also provides a computer program product containing instructions, and when the instructions are executed by a computer, the computer can perform the method of the above method embodiment.

[0272] The present application also provides a computer program product, which includes a computer program stored in a computer-readable storage medium. The processor of the electronic device reads the computer program from the computer-readable storage medium, and the processor executes the computer program, so that the electronic device executes the corresponding process in the method for controlling the user position in the virtual scene in the embodiment of the present application, which will not be described in detail for the sake of brevity.

[0273] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the module is only a logical function division. There may be other division methods in actual implementation, such as multiple modules or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or modules, which can be electrical, mechanical or other forms.

[0274] The modules described as separate components may or may not be physically separated, and the components displayed as modules may or may not be physical modules, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. For example, each functional module in each embodiment of the present application may be integrated into a processing module, or each module may exist physically separately, or two or more modules may be integrated into one module.

[0275] The above are only specific implementations of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A gesture recognition method, characterized in that: include: Get the key point information of the gesture to be recognized; According to the key point information of the gesture to be identified, determining the identification information of the gesture to be identified, the identification information including the posture of the hand, the direction of the hand and the contact relationship of the bones of the hand, wherein the posture parameter of the hand includes at least one of the following parameters: a first bending parameter of each finger, a second bending parameter and an abduction parameter of each finger; wherein the first bending parameter is a bending parameter of a metacarpophalangeal joint, the second bending parameter is a bending parameter of a distal joint of each finger, and the abduction parameter is used to describe the opening and closing state between two fingers; The gesture type of the gesture to be recognized is determined according to the recognition information.

2. The method according to claim 1, characterized in that When the recognition information of the gesture to be recognized includes the posture of the hand, determining the recognition information of the gesture to be recognized according to the key point information of the gesture to be recognized includes: Determining the values ​​of the hand gesture parameters according to the key point information of the gesture to be recognized; The step of determining the gesture type of the gesture to be recognized according to the recognition information of the gesture to be recognized includes: Comparing the value of the hand gesture parameter with the threshold of the gesture parameter of the preset gesture; The gesture type of the gesture to be recognized is determined according to the comparison result.

3. The method according to claim 1 or 2, characterized in that: The first bending parameter and the second bending parameter include identification information of whether to bend, and the abduction parameter includes identification information of whether to open or close.

4. The method according to claim 1 or 2, characterized in that: The first bending parameter and the second bending parameter include bending angles, and the abduction parameter includes an opening and closing distance.

5. The method according to claim 1, characterized in that When the recognition information of the gesture to be recognized includes the contact relationship of the bones of the hand, determining the recognition information of the gesture to be recognized according to the key point information of the gesture to be recognized includes: Determine the coordinates of the skeleton of the gesture to be recognized according to the key point information of the gesture to be recognized; According to the coordinates of the bones of the gesture to be recognized, the contact relationship between the bones of the hand is determined.

6. The method according to claim 5, characterized in that The skeleton of the hand includes 23 bones, among which the little finger and thumb each include 5 bones, the index finger, middle finger and ring finger each include 4 bones, and the wrist includes 1 bone.

7. The method according to claim 1, characterized in that When the recognition information of the gesture to be recognized includes the direction of the hand, determining the recognition information of the gesture to be recognized according to the key point information of the gesture to be recognized includes: Determining the starting position and detection direction of the gesture to be recognized according to the key point information of the gesture to be recognized; Mapping the reference direction of the preset gesture to the coordinate system corresponding to the starting position and the detection direction to obtain the mapping direction of the reference direction in the coordinate system; The mapping direction and the detection direction are compared to obtain the orientation of the hand.

8. The method according to claim 7, characterized in that The mapping of the reference direction of the preset gesture to the coordinate system corresponding to the starting position and the detection direction to obtain the mapping direction of the reference direction in the coordinate system includes: The reference direction is projected into a preset plane of the coordinate system to obtain a projection direction, where the projection direction is a mapping direction of the reference direction in the coordinate system.

9. The method according to claim 7 or 8, characterized in that: Comparing the mapping direction with the detection direction to obtain the direction of the hand includes: Determining an angle between the mapping direction and the detection direction; When the included angle is less than or equal to a preset angle, the direction of the hand is determined to be the reference direction.

10. The method according to claim 7 or 8, characterized in that: The starting position of the gesture to be recognized is any one of the following positions: a palm position, a wrist position, a position corresponding to the palm position plus a first offset, and a position corresponding to the wrist position plus a second offset.

11. The method according to claim 7 or 8, characterized in that: The detection direction is any one of the following directions: the direction pointed by four fingers except the thumb, the direction the palm faces, and the direction pointed by the thumb.

12. The method according to claim 7 or 8, characterized in that: The reference direction is any one of the following directions: a direction facing the face, a direction away from the face, a direction toward the sky, and a direction toward the ground.

13. The method according to claim 1, characterized in that The step of determining the gesture type of the gesture to be recognized according to the recognition information of the gesture to be recognized includes: Get the recognition result of the gesture type at the previous moment; If the recognition result of the gesture type at the current moment is that the gesture type at the previous moment is different from the preset gesture type, the threshold of the recognition information of the preset gesture type is increased by a threshold width to obtain a target threshold range of the recognition information of the preset gesture type, wherein the preset gesture types at the previous moment and the current moment are the same; Determining the gesture type of the gesture to be recognized according to the value of the recognition information of the gesture to be recognized and the target threshold range of the recognition information of the preset gesture type; If the recognition result of the gesture type at the current moment is that the gesture type at the previous moment is the same as the preset gesture type, the gesture type of the gesture to be recognized is determined according to the value of the recognition information of the gesture to be recognized and the threshold of the recognition information of the preset gesture type.

14. The method according to claim 1 or 13, characterized in that: After determining the gesture type of the gesture to be recognized according to the recognition information of the gesture to be recognized, the method further includes: Obtaining the duration of the gesture type of the gesture to be recognized; When the duration of the gesture type of the gesture to be recognized is less than the preset duration, determining that the gesture type of the gesture to be recognized is not effective; When the duration of the gesture type of the gesture to be recognized is greater than or equal to the preset duration, it is determined that the gesture type of the gesture to be recognized is effective.

15. The method according to claim 14, characterized in that Also includes: When the gesture type of the gesture to be recognized is recognized for the first time, starting to time the duration of the gesture type of the gesture to be recognized; When it is detected that the gesture type of the gesture to be recognized has changed, the timing of the duration of the gesture type of the gesture to be recognized is stopped.

16. The method according to claim 1, characterized in that The step of obtaining key point information of the gesture to be recognized includes: Get image data; Using a hand detection algorithm to detect whether a hand exists in the image data; When a hand exists in the image data, a key point detection algorithm is used to detect key point information of the hand.

17. The method according to claim 1, characterized in that The step of obtaining key point information of the gesture to be recognized includes: Acquire sensor data collected by a wearable device on the hand, wherein the sensor data includes motion data of joint points of the hand; Determine key point information of the gesture to be recognized according to the sensor data.

18. A gesture recognition device, characterized in that: include: An acquisition module is used to obtain key point information of the gesture to be recognized; A determination module, used to determine the identification information of the gesture to be identified according to the key point information of the gesture to be identified, wherein the identification information includes the posture of the hand, the direction of the hand and the contact relationship of the bones of the hand, wherein the posture parameter of the hand includes at least one of the following parameters: a first bending parameter of each finger, a second bending parameter and an abduction parameter of each finger; wherein the first bending parameter is a bending parameter of a metacarpophalangeal joint, the second bending parameter is a bending parameter of a distal joint of each finger, and the abduction parameter is used to describe the opening and closing state between two fingers; The recognition module is used to determine the gesture type of the gesture to be recognized according to the recognition information.

19. An electronic device, characterized in that: include: A processor and a memory, the memory being used to store a computer program, and the processor being used to call and run the computer program stored in the memory to execute the method according to any one of claims 1 to 17.

20. A computer-readable storage medium, characterized in that: Used to store a computer program, the computer program causing a computer to execute the method according to any one of claims 1 to 17.

21. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the method according to any one of claims 1 to 17 is implemented.

Images