Gesture recognition method and device, and electronic device

Abstract

The application is suitable for the technical field, and provides a gesture recognition method and device and electronic equipment, including: acquiring a to-be-recognized image; if a hand is detected in the to-be-recognized image, outputting a joint point coordinate of the hand; determining a hand state according to the joint point coordinate of the hand, the hand state including a hand overall state, a finger bending degree and an inter-finger state; and recognizing a classification result of a hand gesture in the to-be-recognized image according to the hand state, the hand state including the hand overall state, the finger bending degree and the inter-finger state being determined through the joint point coordinate to recognize the hand gesture classification, so that a more complex static gesture recognition scene can be coped with, the accuracy of gesture recognition is improved, and the problems of low recognition accuracy and few applicable scenes of the existing gesture recognition method are solved.

Description

Technical Field

[0001] This application belongs to the technical field, and in particular relates to a gesture recognition method, device, and electronic device. Background Technology

[0002] In computer science, gesture recognition refers to a method that uses mathematical algorithms to enable computers to recognize human gestures. Users can use gestures to control or interact with devices. In recent years, gesture-based human-computer interaction has been increasingly applied to devices such as mobile phones and televisions, enabling air-to-ground interaction.

[0003] Current gesture recognition methods generally fall into two categories: RGB image-based gesture recognition and hand knuckle-based gesture recognition. RGB image-based methods typically acquire an RGB image of the user's gesture and then use a deep learning network to classify the RGB image to determine the gesture category. However, RGB image-based gesture recognition cannot accurately distinguish between different gesture categories for similar gestures, thus easily leading to misidentification.

[0004] To accurately define the boundaries of each gesture category, gesture recognition methods based on hand joints can be used. These methods typically first detect the hand joints and then perform gesture recognition based on them, thus accurately defining the boundaries of each gesture category. Current hand joint-based gesture recognition methods usually encode the hand using 0-1 codes based on the curvature of the hand joints to determine the gesture category. This method is only suitable for forward gesture recognition scenarios and cannot be applied to complex gesture recognition scenarios. Summary of the Invention

[0005] This application provides a gesture recognition method, device, and electronic device, which can solve the problems of low recognition accuracy and limited applicable scenarios in existing gesture recognition methods.

[0006] In a first aspect, embodiments of this application provide a gesture recognition method, including: Acquire the image to be recognized; If a human hand is detected in the image to be recognized, the coordinates of the joints of the human hand are output. The hand state is determined based on the coordinates of the joints of the human hand, and the hand state includes at least one of the following: the overall hand state, the degree of finger flexion, and the state between the fingers. The classification result of the gesture in the image to be identified is determined based on the state of the hand.

[0007] In one possible implementation of the first aspect, the hand state includes the overall hand state, the degree of finger flexion, and the interdigital state; The overall state of the hand is determined based on the distance between two preset joint points and / or the positional relationship between the preset joint points and the hand contour; The degree of finger flexion is determined by the cumulative deviation angle of each joint point of the target finger. The finger position is determined based on the line connecting the fingertip joints of the two target fingers to the preset palm joints and / or the distance between the fingertip joints of the two target fingers.

[0008] In one possible implementation of the first aspect, the step of identifying the classification result of the gesture in the image to be identified based on the hand state includes: The hand state is feature-encoded; The feature encoding of the hand state is matched with the feature range constraints of each preset gesture to obtain the pre-classified gesture category that satisfies the feature range constraints. Calculate the similarity between the feature encoding of the hand state and each pre-classified gesture category; The gesture category in the image to be identified is determined based on similarity.

[0009] In one possible implementation of the first aspect, acquiring the image to be identified includes: Acquire video data; The video data is preprocessed to obtain the image to be recognized.

[0010] In one possible implementation of the first aspect, the preprocessing operations include color gamut conversion, edge cropping, and proportional scaling of the image size.

[0011] In one possible implementation of the first aspect, after identifying the classification result of the gesture in the image to be identified based on the hand state, the method further includes: determining the target gesture category of the human hand based on the classification result corresponding to the image to be identified for a preset number of frames.

[0012] In one possible implementation of the first aspect, determining the target hand gesture category based on the classification result corresponding to the image to be recognized for a preset number of frames includes: The target hand gesture category is determined based on the classification results corresponding to the image to be identified for a preset number of consecutive frames within a preset time period.

[0013] In one possible implementation of the first aspect, the target hand gesture category is determined based on the classification results corresponding to a preset number of consecutive frames of the image to be recognized within a preset time period, including: Write the classification result corresponding to each frame of a preset number of consecutive frames within a preset time period into the classification result queue; The target gesture category is determined based on the classification results in the classification result queue.

[0014] One possible implementation of the first aspect also includes: loading a gesture category description file and obtaining the feature range constraints for each preset gesture.

[0015] In one possible implementation of the first aspect, the gesture category description file includes a product-defined category description file and a user-defined category description file.

[0016] Secondly, embodiments of this application provide a gesture recognition device, including: The acquisition unit is used to acquire the image to be recognized; The detection unit is used to output the coordinates of the joints of the human hand if a human hand is detected in the image to be identified. A state determination unit is used to determine the hand state based on the joint coordinates of the human hand. The hand state includes the overall hand state, the degree of finger bending, and the state between the fingers. The recognition unit is used to identify the classification result of the gesture in the image to be recognized based on the hand state.

[0017] Thirdly, embodiments of this application provide a terminal device, including: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the method as described in any of the first aspects.

[0018] Fourthly, embodiments of this application provide a computer-readable storage medium storing a computer program that, when executed by a processor, implements the method described in any of the first aspects.

[0019] Fifthly, embodiments of this application provide a computer program product that, when run on a terminal device, causes the terminal device to execute the method described in any one of the first aspects above.

[0020] It is understood that the beneficial effects of the second to fifth aspects mentioned above can be found in the relevant descriptions in the first aspect mentioned above, and will not be repeated here.

[0021] The beneficial effects of the embodiments of this application compared with the prior art are: The gesture recognition method, device, and electronic device provided in this application determine the hand state, including the overall hand state, the degree of finger bending, and the state between the fingers, by using the coordinates of the joints to identify the gesture category. This can cope with more complex static gesture recognition scenarios, improve the accuracy of gesture recognition, and solve the problems of low recognition accuracy and limited applicable scenarios of existing gesture recognition methods. Attached Figure Description

[0022] To more clearly illustrate the technical solutions in the embodiments of this application, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0023] Figure 1 This is a schematic diagram of the architecture of an application scenario provided in an embodiment of this application; Figure 2 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application; Figure 3 This is a flowchart illustrating a gesture recognition method provided in an embodiment of this application; Figure 4 This is a schematic diagram of the network structure of the hand joint detection model provided in the embodiments of this application; Figure 5 This is a schematic diagram of 21 joints on a human hand provided in an embodiment of this application; Figure 6 This is a flowchart illustrating another gesture recognition method provided in an embodiment of this application; Figure 7 This is a schematic diagram of the structure of the gesture recognition device provided in the embodiments of this application. Detailed Implementation

[0024] In the following description, specific details such as particular system architectures and techniques are set forth for illustrative purposes and not for limitation, in order to provide a thorough understanding of the embodiments of this application. However, those skilled in the art will understand that this application may also be implemented in other embodiments without these specific details. In other instances, detailed descriptions of well-known systems, apparatuses, circuits, and methods have been omitted so as not to obscure the description of this application with unnecessary detail.

[0025] It should be understood that, when used in this application specification and the appended claims, the term "comprising" indicates the presence of the described features, integrals, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or a collection thereof.

[0026] It should also be understood that the term “and / or” as used in this application specification and the appended claims means any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.

[0027] As used in this application specification and the appended claims, the term "if" may be interpreted, depending on the context, as "when," "once," "in response to determination," or "in response to detection." Similarly, the phrase "if determined" or "if detected [the described condition or event]" may be interpreted, depending on the context, as meaning "once determined," "in response to determination," "once detected [the described condition or event]," or "in response to detection [the described condition or event]."

[0028] Furthermore, in the description of this application and the appended claims, the terms "first," "second," "third," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0029] References to "one embodiment" or "some embodiments" as described in this specification mean that one or more embodiments of this application include a specific feature, structure, or characteristic described in connection with that embodiment. Therefore, the phrases "in one embodiment," "in some embodiments," "in other embodiments," "in still other embodiments," etc., appearing in different parts of this specification do not necessarily refer to the same embodiment, but rather mean "one or more, but not all, embodiments," unless otherwise specifically emphasized. The terms "comprising," "including," "having," and variations thereof mean "including but not limited to," unless otherwise specifically emphasized.

[0030] To illustrate the specific implementation of the gesture recognition method provided in this application in more detail, the gesture recognition method provided in this application will be described below with reference to the accompanying drawings: The gesture recognition method provided in this application embodiment can be applied to, for example, Figure 1 In the scene shown. Figure 1 This is a schematic diagram illustrating an application scenario of the gesture recognition method provided in the embodiments of this application. For example... Figure 1 As shown, electronic device 10 can be connected to camera device 11. Camera device 11 can capture video or images in real time. When a user needs to control electronic device 10 using a certain type of gesture, they will enter the camera's field of view, allowing camera device 11 to capture an image of the user. The captured image is then sent to electronic device 10. Electronic device 10 can then perform gesture recognition based on the captured image, determine the gesture category contained in the image, further determine the interaction command corresponding to that gesture category, and then execute the interaction command to achieve interaction.

[0031] It should be noted that, in one embodiment of this application, the electronic device 10 may include the camera device 11, that is, the camera device 11 may be an internal component of the electronic device 10.

[0032] In the embodiments of this application, the imaging range of the camera device 11 is related to the focal length and angle of view of the lens of the camera device 11. Different camera devices have different imaging ranges, and this application does not limit them.

[0033] In this embodiment, the user entering the camera range of the camera device 11 means that the camera device 11 can capture the user's hand movements (understandably, the user can do not make any interactive gestures at this time). That is, when the user wants to control the electronic device 11 to perform a certain operation through gestures, he will make the corresponding gesture in the area that the camera device 11 can capture, so that the camera device 11 can capture an image containing the gesture to be recognized.

[0034] It should be noted that the aforementioned camera device 11 can also continuously capture the current scene by recording video, that is, the images captured by the aforementioned camera device 11 may include images where the user has not made any interactive gestures.

[0035] In one embodiment of this application, the aforementioned electronic device 10 may be a television set, mobile phone, tablet computer, wearable device, in-vehicle device, augmented reality (AR) / virtual reality (VR) device, laptop computer, ultra-mobile personal computer (UMPC), netbook, personal digital assistant (PDA), or other terminal device. This application embodiment does not impose any limitations on the specific type of electronic device.

[0036] Please see Figure 2 , Figure 2 A schematic diagram of the structure of an electronic device 10 provided in an embodiment of this application is shown. For example... Figure 2 As shown, the aforementioned electronic device 10 includes components such as a camera device 11, a communication module 12, a memory 13, and a processor 14. Those skilled in the art will understand that... Figure 2 The electronic device shown may also include more or fewer components than illustrated, or combine certain components, or have different component arrangements. The processor 13 is connected to the camera device 11, the memory 13, and the communication module 12. The memory 13 stores a computer program, which is simultaneously executed by the processor 14.

[0037] The communication module 12 can connect to external devices via a network. The communication module 12 can receive data from external devices and can also send data, instructions, and information to the external devices, which can be electronic devices such as mobile phones, televisions, laptops, desktop computers, and tablets.

[0038] The memory 13 can be used to store software programs and various data. The memory 13 may mainly include a program storage area and a data storage area. The program storage area may store the operating system, at least one application program required for a function (such as capturing the RGB values ​​and coordinates of each pixel in the image displayed on the television, and performing gesture recognition on the image captured by the camera device 11), etc.; the data storage area may store data or information created based on the use of the electronic device 10, etc. Furthermore, the memory 13 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other volatile solid-state storage device.

[0039] Processor 14 is the control center of electronic device 10. It connects various parts of the device via various interfaces and lines, and performs various functions and processes data by running or executing software programs and / or modules stored in memory 13, and by calling data stored in memory 13, thereby providing overall monitoring of the device. Processor 14 may include one or more processing units; preferably, processor 14 may integrate an application processor and a modem processor, wherein the application processor mainly handles the operating system, user interface, and applications, and the modem processor mainly handles wireless communication. It is understood that the modem processor may not be integrated into processor 14.

[0040] The aforementioned electronic device 10 may also include a power management module 15, which is used to connect to the mains power supply to realize power control and ensure the normal operation of other components.

[0041] Those skilled in the art will understand that Figure 2 The structure of the electronic device 10 shown does not constitute a limitation on the electronic device 10. The electronic device 10 may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0042] In one embodiment of this application, the camera device 11 may include a single camera, a dual camera, or a triple camera, etc., and this application embodiment does not limit this.

[0043] For example, electronic device 10 may include three cameras, one of which is a main camera, one is a wide-angle camera, and one is a telephoto camera.

[0044] Optionally, when the electronic device 10 includes multiple cameras, these multiple cameras can all be front-facing, all be rear-facing, or some can be front-facing and others rear-facing. This application embodiment does not limit this.

[0045] This application uses the example of an electronic device 10 being a television set and a camera device 11 being a single camera on a television set for the following explanation: Please see Figure 3 , Figure 3 This diagram illustrates the implementation flow of a gesture recognition method provided in an embodiment of this application. For example, the executing entity of the above gesture recognition method can be the aforementioned television set. Figure 3 As shown, the above gesture recognition method specifically includes the following steps: S11: Obtain the image to be recognized.

[0046] In this embodiment, the image to be identified can be captured by the front-facing camera on the television. That is, the camera captures video data of the current environment, and the television can then read the video frames one by one through the camera. Specifically, the front-facing camera on the aforementioned television can capture video data within its field of view, and then send the captured video data to the television. The television then performs preprocessing operations on the video data to obtain the image to be recognized.

[0047] The preprocessing operations performed on video data by the aforementioned television set may include, but are not limited to, color gamut conversion, edge cropping, and proportional scaling of image size.

[0048] The aforementioned color gamut conversion operation refers to converting the video frame image captured by the camera from its original color gamut to a target color gamut. Specifically, it can be converting the video frame image from the RGB color gamut to the grayscale color gamut.

[0049] The aforementioned edge cropping operation refers to cropping the edges of the read video frame image.

[0050] The above-mentioned proportional scaling operation reduces or enlarges the cropped image so that the size of the reduced or enlarged image matches the image size required by the television for gesture recognition.

[0051] Perform the above preprocessing operation on each frame of the video data to obtain the image to be identified.

[0052] S12: If a human hand is detected in the image to be recognized, output the coordinates of the joints of the human hand.

[0053] In this embodiment of the application, a human hand joint detection model can be used to detect whether a human hand exists in the image to be identified, and if a human hand is detected in the image to be identified, the coordinates of the joints of the human hand can be output.

[0054] Specifically, the aforementioned hand joint detection model can be constructed based on the Mobilenet network, so that its output branches include the recognition result of whether a hand exists in the image to be recognized and the recognition result of the joint coordinates of the hand.

[0055] Specifically, the network structure of the aforementioned hand joint detection model can be as follows: Figure 4 As shown, through as Figure 4 The hand joint detection model shown can identify whether a hand exists in the image to be identified, and after detecting a hand in the image, it can also output the joint coordinates of the hand.

[0056] Wherein, IPUT is the input image to be recognized. Covn0 is a standard convolutional layer with a kernel size of 3×3. Covn1-Covn13 are depthwise separable convolutional layers, also with a kernel size of 3×3.

[0057] In this embodiment of the application, the coordinates of the joints of the aforementioned human hand may include, for example: Figure 5 The coordinates of the 21 joints on the human hand are shown.

[0058] In this embodiment of the application, when the above-mentioned hand joint detection model detects that there is no hand in the image to be identified, it can also output joint coordinates. However, the output joint coordinates are messy data. At this time, the electronic device will directly ignore the output result of this frame and then continue to read the next frame image to perform the above operation.

[0059] S13: Determine the hand state based on the coordinates of the joints of the human hand.

[0060] In this embodiment of the application, after obtaining the joint coordinates of the human hand, the hand state can be determined based on the joint coordinates of the human hand.

[0061] In the embodiments of this application, the aforementioned hand state may include at least one of the following: the overall hand state, the degree of finger bending, and the interdigital state.

[0062] The overall hand state described above is determined based on the distance between two preset joints and / or the positional relationship between the preset joints and the hand contour; in other words, the overall hand state refers to multiple features used to describe the positional relationship between the palm and the electronic device in three-dimensional space. The degree of finger bending is determined based on the cumulative deviation angle of each joint of the target finger; in other words, the degree of finger bending refers to multiple features used to describe the spatial posture of each finger of the human hand. The aforementioned finger-interval state is determined based on the line connection between the fingertip joints of the two target fingers and the preset palm joints and / or the distance between the fingertip joints of the two target fingers; in other words, the finger-interval state refers to multiple features used to describe the postural relationship between different fingers.

[0063] It should be noted that electronic devices can determine the hand state based on one or more of the following: overall hand state, degree of finger bending, and interdigital position.

[0064] In one embodiment of this application, the aforementioned hand state may include the overall hand state, the degree of finger bending, and the state between the fingers.

[0065] Specifically, the overall state of the hand can include several features such as the orientation of the palm (front or side), whether the palm is facing the camera, the angle formed by the line connecting the thumbs and the vertical downward direction, and the angle formed by the line connecting the palms and the line containing the preset direction.

[0066] For details, please see Figure 5 The orientation of the palm in the overall hand position described above can be represented by the width of the palm. In this embodiment, the width of the palm can be represented by the length of the line connecting joint point 17 (the fourth joint point of the little finger) and joint point 5 (the fourth joint point of the index finger). When the width of the palm is lower than a preset width threshold, the orientation of the palm is determined to be lateral; when the width of the palm is greater than or equal to the preset threshold, the orientation of the palm is determined to be forward.

[0067] Whether the palm is facing the camera can be determined by whether the base of the hand (joint 0) is inside the captured hand. If the base of the hand (joint 0) is inside the captured hand, the palm is facing the camera; if the base of the hand (joint 0) is not inside the captured hand, the palm is not facing the camera.

[0068] The thumb line refers to the line connecting joint point 2 and joint point 4.

[0069] The palm line refers to the line connecting joint point 0 at the base of the hand and joint point 12 (the first joint point of the middle finger).

[0070] In this embodiment, the degree of finger bending may include features such as the degree of bending of the thumb, index finger, middle finger, ring finger, and little finger. Specifically, the degree of finger bending can be determined based on the cumulative deviation angle of each joint of the finger.

[0071] Please continue reading. Figure 5 The cumulative deviation angle of each joint of a finger refers to the sum of the angles formed by the lines connecting adjacent joints on a finger. For example, the angle between the line connecting joints 5 and 0 and the line connecting joints 5 and 6 on the index finger is A degrees; the angle between the line connecting joints 5 and 6 and the line connecting joints 6 and 7 is B degrees; and the angle between the line connecting joints 6 and 7 and the line connecting joints 7 and 8 is C degrees. Then, the degree of bending of the index finger is the absolute value of A degrees plus the absolute value of B degrees plus the absolute value of C degrees.

[0072] In practical applications, knuckle thresholds can be set to determine whether a finger is in a bent or straight state. For example, if the knuckle threshold is (D, F), then if the angle of all knuckles is no greater than the lower threshold D, the finger is in a bent state; if the angle of all knuckles is less than the lower threshold, the finger is in a straight state; otherwise, it is in an uncertain state. Specifically, the above-mentioned preset thresholds can be set according to the actual situation. Typically, the preset threshold for the knuckles of the palm is (60, 100), and the preset threshold for other knuckles is (30, 50).

[0073] In this embodiment, the aforementioned finger-interval state may include features such as the posture relationship between the thumb and index finger, the posture relationship between the index finger and middle finger, the posture relationship between the middle finger and ring finger, the posture relationship between the ring finger and little finger, and the posture relationship between the thumb and ring finger. Specifically, the aforementioned finger-interval state can be determined by whether the fingertips intersect with the line segments between the fingertip joints and the joints on the palm, and by whether the two fingers are together or apart by the distance between the fingertip joints.

[0074] For example, when determining whether the middle finger and index finger cross, it can be determined whether the line segments of joint point 8 and joint point 5 and joint point 12 and joint point 9 intersect; if they intersect, the middle finger and index finger cross; if they do not intersect, the middle finger and index finger do not cross.

[0075] By setting a fingertip threshold, if the distance between the knuckles of two fingers is less than the threshold, the fingers are considered to be together; if the distance is greater than the threshold and the intersection of the lines is always at the fingertip, the fingers are considered to be apart. It should be noted that the fingertip threshold is typically set to 0.08 times the palm size.

[0076] S14: Based on the hand state, identify the classification result of the gesture in the image to be identified.

[0077] In this embodiment, after determining the hand state, it can be matched with the hand states of different categories of preset gestures to find the preset gesture category that matches the hand state. This preset gesture category is then used as the pre-classified gesture category for the gesture to be recognized. It should be noted that when matching the hand state with the hand states of preset gestures, more than one gesture category may be matched. Therefore, by calculating the similarity between the hand state and the hand states of each pre-classified gesture category, the gesture category of the gesture in the image to be recognized is determined based on the similarity. Specifically, the pre-classified gesture category with the highest similarity can be identified as the gesture classification result of the gesture in the image to be recognized, thus accurately identifying the classification result of the gesture in the image to be recognized.

[0078] In one embodiment of this application, S14 may include the following steps: The hand state is feature-encoded; The feature encoding of the hand state is matched with the feature range constraints of each preset gesture to obtain the pre-classified gesture category that satisfies the feature range constraints. Calculate the similarity between the feature encoding of the hand state and each pre-classified gesture category; The gesture category in the image to be identified is determined based on similarity.

[0079] In this embodiment of the application, after the above-mentioned hand states (such as the various gesture states described in S13) are extracted, the hand states can be feature-encoded separately, and then the features of the gestures can be represented by the feature encoding.

[0080] In practical applications, the following three values ​​represent the overall hand state: palm orientation, the angle between the line connecting the thumb and the vertical downward direction, and the angle between the line connecting the palm and the line containing the preset direction. The following five values ​​represent the degree of bending of each finger. The following five values ​​represent the interfinite finger position: the distance between the thumb and index finger, the distance between the index and middle finger, the distance between the middle and ring fingers, the distance between the ring and little fingers, and the distance between the thumb and ring finger. Using these 13 values ​​to represent the hand state is the process of feature encoding the hand state.

[0081] The aforementioned feature encoding can be represented by feature vectors. In other words, the feature vector of the hand state can include the following 13 values: palm orientation, angle formed by the line connecting the thumb and the vertical downward direction, angle formed by the line connecting the palm and the line containing the preset direction, degree of bending of the thumb, degree of bending of the index finger, degree of bending of the middle finger, degree of bending of the ring finger, degree of bending of the little finger, distance between the thumb and index finger, distance between the index finger and middle finger, distance between the middle finger and ring finger, distance between the ring finger and little finger, and distance between the thumb and ring finger.

[0082] The feature range constraints for each type of preset gesture will give the range value of the feature vector of each type of preset gesture. Gestures within the range value of the feature vector of a type of preset gesture can be identified as that type of preset gesture.

[0083] By matching the feature vector of the hand state in the image to be recognized with the feature range constraints of each preset gesture, one or more preset gestures that meet the conditions can be identified.

[0084] After identifying the preset gestures within the pre-classified gesture categories, the similarity of the gestures is represented by the L1 norm distance between the feature range constraint center value and the feature vector of the preset gesture; the smaller the distance, the higher the similarity. By comparing the similarity between each preset gesture and the gesture in the image to be recognized, the pre-classified gesture category with the highest similarity is identified as the gesture classification result of the gesture in the image to be recognized. In other words, the preset gesture with the smallest L1 norm distance is identified as the gesture category of the gesture in the image to be recognized.

[0085] In one embodiment of this application, the feature range constraints of the aforementioned preset gestures can be determined by a gesture category description file. During initialization, the electronic device loads a user- or product-defined gesture category description file, thereby obtaining the feature range constraints for each preset gesture.

[0086] The gesture category description file described above is used to describe the feature range constraints of each preset gesture. The gesture category description file is a system configuration file that is loaded from the system when the electronic device is initialized. The gesture category description file contains the feature range constraints of each preset gesture.

[0087] The gesture category description file includes a product-defined category description file and a user-defined category description file.

[0088] In one embodiment of this application, the gesture category description file can be determined by the product definition category, that is, the instructions corresponding to each gesture category are predefined, and then the user's gesture habits are guided to improve the recognition accuracy.

[0089] The product definition category description file contains the feature range constraints of the gesture categories for each product, which are predefined (the gesture list is set at the factory).

[0090] In one embodiment of this application, the gesture category description file can be determined by a user-defined category. The feature range constraints of the gesture can be determined based on the user-defined gesture, and then the feature range constraints of the gesture can be added to the gesture category description file to realize the user-defined gesture.

[0091] As can be seen from the above, the gesture recognition method provided in this application determines the hand state, including the overall hand state, the degree of finger bending, and the state between the fingers, by using the coordinates of the joint points to identify the gesture category. This method can handle more complex static gesture recognition scenarios, improve the accuracy of gesture recognition, and solve the problems of low recognition accuracy and limited applicable scenarios of existing gesture recognition methods.

[0092] Please see Figure 6 , Figure 6 This illustration shows a schematic diagram of the implementation flow of a gesture recognition method according to another embodiment of this application. Unlike the previous embodiment, the gesture recognition method provided in this embodiment may further include the following steps: S21: Determine the target hand gesture category based on the classification results corresponding to the images to be recognized at a preset number of frames.

[0093] In this embodiment, after processing the image to be recognized, the classification result of the gesture in the image can be output. The electronic device can record the classification result corresponding to each frame of the image in the form of a log or other means. In order to improve the accuracy of gesture recognition, the target gesture category of the human hand can be determined by the classification results of multiple frames of images.

[0094] In this embodiment of the application, the electronic device can obtain the classification results corresponding to multiple frames of images by reading logs, and then determine the target gesture category by counting the number of times the classification results appear in the multiple frames of images. That is, the gesture that appears most often can be taken as the target gesture category.

[0095] In this embodiment of the application, S21 may specifically be: determining the target hand gesture category based on the classification results corresponding to a preset number of consecutive frames of images to be identified within a preset time period.

[0096] Specifically, the electronic device can store the classification results of a preset number of consecutive frames within a preset time period in a queue and store them in a first-in, first-out manner.

[0097] It should be noted that the preset time period and preset frame number can be determined according to the actual application, and this application will not elaborate on this.

[0098] In practical applications, the target gesture category can be determined based on the number of times the gesture appears in the classification results of a preset number of consecutive frames within a preset time period. That is, the gesture that appears most frequently can be taken as the target gesture category.

[0099] For example, given five consecutive frames within a 0.1-second timeframe, assuming the gesture recognition result for the first frame is gesture 1, the second frame is gesture 2, the third frame is gesture 1, the fourth frame is gesture 1, and the fifth frame is gesture 1, then the target gesture category corresponding to these five frames is determined as gesture 1.

[0100] In this embodiment of the application, the recognition result may also include the bounding box of the detected object (located by the human hand joint detection model), so the position of the bounding box of the detected object may be corrected based on the bounding box of the detected object for a consecutive preset number of frames.

[0101] In one embodiment of this application, determining the target hand gesture category based on the classification results corresponding to a preset number of consecutive frames of images to be identified within a preset time period may specifically include the following steps: Write the classification result corresponding to each frame of a preset number of consecutive frames within a preset time period into the classification result queue; The target gesture category is determined based on the classification results in the classification result queue.

[0102] As can be seen above, by smoothing the gesture recognition results, you can effectively prevent repeated jumps in the gesture recognition results and improve the recognition accuracy.

[0103] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0104] Corresponding to the gesture recognition method described in the above embodiments, Figure 7 A structural block diagram of the gesture device provided in the embodiments of this application is shown. For ease of explanation, only the parts related to the embodiments of this application are shown.

[0105] Reference Figure 7 In this embodiment, the gesture recognition device 70 includes an acquisition unit 71, a detection unit 72, a state determination unit 73, and a recognition unit 74. Wherein: The acquisition unit 71 is used to acquire the image to be recognized; The detection unit 72 is used to output the coordinates of the joints of the human hand if a human hand is detected in the image to be identified; The state determination unit 73 is used to determine the hand state based on the joint coordinates of the human hand. The hand state includes the overall hand state, the degree of finger bending, and the interdigital state. The recognition unit 74 is used to identify the classification result of the gesture in the image to be recognized based on the hand state.

[0106] In one embodiment of this application, the identification unit 71 may include: An encoding unit is used to encode the features of the hand state; The matching unit is used to match the feature encoding of the hand state with the feature range constraints of each preset gesture to obtain the pre-classified gesture category that satisfies the feature range constraints. A computing unit is used to calculate the similarity between the feature encoding of the hand state and each pre-classified gesture category; A determining unit is used to determine the gesture category of the gesture in the image to be identified based on similarity.

[0107] In one embodiment of this application, the acquisition unit 71 may include: The video acquisition unit is used to acquire video data; The preprocessing unit is used to perform preprocessing operations on video data to obtain the image to be recognized.

[0108] In one embodiment of this application, the preprocessing unit is specifically used to perform color gamut conversion, edge cropping, and proportional scaling of the image size on the video data.

[0109] In one embodiment of this application, the gesture recognition device further includes a smoothing unit. The smoothing unit is used to determine the target gesture category of a human hand based on the classification results corresponding to a preset number of frames of images to be recognized.

[0110] In one embodiment of this application, the smoothing unit is specifically used to determine the target gesture category based on the classification results corresponding to the image to be recognized for a preset number of consecutive frames within a preset time period.

[0111] In one embodiment of this application, the smoothing unit is specifically used to write the classification result corresponding to each frame of an image with a preset number of consecutive frames within a preset time period into a classification result queue; and to determine the target gesture category based on the classification result in the classification result queue.

[0112] In one embodiment of this application, the gesture recognition device further includes a loading unit, which is used to load a gesture category description file and obtain the feature range constraints for each preset gesture.

[0113] The gesture category description file includes a product-defined category description file and a user-defined category description file.

[0114] It should be noted that the information interaction and execution process between the above-mentioned devices / units are based on the same concept as the method embodiments of this application. For details on their specific functions and technical effects, please refer to the method embodiments section, and they will not be repeated here.

[0115] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is merely an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit. Furthermore, the specific names of the functional units and modules are only for easy differentiation and are not intended to limit the scope of protection of this application. The specific working process of the units and modules in the above system can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0116] This application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps described in the various method embodiments above.

[0117] This application provides a computer program product that, when run on a mobile terminal, enables the mobile terminal to implement the steps described in the various method embodiments above.

[0118] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include at least: any entity or device capable of carrying the computer program code to a photographing device / terminal device, a recording medium, a computer memory, a read-only memory (ROM), a random access memory (RAM), and a software distribution medium, such as a USB flash drive, a portable hard drive, a magnetic disk, or an optical disk.

[0119] In the above embodiments, the descriptions of each embodiment have different focuses. For parts that are not described in detail or recorded in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0120] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0121] In the embodiments provided in this application, it should be understood that the disclosed devices / electronic devices and methods can be implemented in other ways. For example, the device / electronic device embodiments described above are merely illustrative. For instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the displayed or discussed mutual couplings or direct couplings or communication connections may be through some interfaces; indirect couplings or communication connections between devices or units may be electrical, mechanical, or other forms.

[0122] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0123] The above-described embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be included within the protection scope of this application.

Claims

1. A gesture recognition method, characterized in that, include: Acquire the image to be recognized; If a human hand is detected in the image to be identified, the coordinates of the joints of the human hand are output. The hand state is determined based on the joint coordinates of the hand; the hand state includes at least one of the following: overall hand state, degree of finger flexion, and interdigital state. The classification result of the gesture in the image to be identified is determined based on the hand state. The classification result of the gesture in the image to be identified based on the hand state includes: The determined hand state is feature-encoded, and the feature encoding is represented by a feature vector; The feature encoding of the hand state is matched with the feature range constraints of each preset gesture to obtain the pre-classified gesture category that satisfies the feature range constraints. Calculate the similarity between the feature encoding of the hand state and each pre-classified gesture category; The pre-classified gesture category with the highest similarity is determined as the gesture classification result in the image to be identified; The overall hand position includes whether the palm faces the camera. Whether the palm faces the camera is determined by whether the joint at the base of the hand is inside the captured hand. If the joint at the base of the hand is inside the captured hand, the palm faces the camera; if the joint at the base of the hand is not inside the captured hand, the palm does not face the camera. The finger position is determined by whether the fingertips intersect with the line segments of the joints at the fingertips and the joints at the palm. The distance between the fingertips is used to determine whether the two fingers are together or apart.

2. The gesture recognition method as described in claim 1, characterized in that, The hand state includes the overall hand state, the degree of finger bending, and the state between the fingers; The overall state of the hand is determined based on the distance between two preset joint points and / or the positional relationship between the preset joint points and the hand contour; The degree of finger bending is determined based on the cumulative deviation angle of each joint point of the target finger. The finger joint state is determined based on the line connection between the fingertip joints of the two target fingers and the preset palm joints and / or the distance between the fingertip joints of the two target fingers.

3. The gesture recognition method as described in claim 1, characterized in that, The acquisition of the image to be identified includes: Acquire video data; The video data is preprocessed to obtain the image to be identified.

4. The gesture recognition method as described in claim 3, characterized in that, The preprocessing operations include color gamut conversion, edge cropping, and proportional scaling of the image size.

5. The gesture recognition method as described in claim 1, characterized in that, After identifying the classification result of the gesture in the image to be identified based on the hand state, the method further includes: The target hand gesture category is determined based on the classification results corresponding to the image to be identified at a preset number of frames.

6. The gesture recognition method as described in claim 5, characterized in that, The step of determining the target hand gesture category based on the classification results corresponding to the image to be recognized at a preset number of frames includes: The target hand gesture category is determined based on the classification results corresponding to the image to be identified for a preset number of consecutive frames within a preset time period.

7. The gesture recognition method as described in claim 6, characterized in that, The step of determining the target hand gesture category based on the classification results corresponding to a preset number of consecutive frames of images to be recognized within a preset time period includes: Write the classification result corresponding to each frame of a preset number of consecutive frames within a preset time period into the classification result queue; The target gesture category is determined based on the classification results in the classification result queue.

8. The gesture recognition method as described in claim 1, characterized in that, Also includes: Load the gesture category description file and obtain the feature range constraints for each preset gesture.

9. The gesture recognition method as described in claim 8, characterized in that, The gesture category description file includes a product-defined category description file and a user-defined category description file.

10. A gesture recognition device, characterized in that, include: The acquisition unit is used to acquire the image to be recognized; The detection unit is used to output the coordinates of the joints of the human hand if a human hand is detected in the image to be identified. A state determination unit is used to determine the hand state based on the joint coordinates of the human hand. The hand state includes the overall hand state, the degree of finger bending, and the state between the fingers. The recognition unit is used to identify the classification result of the gesture in the image to be recognized based on the hand state; The classification result of the gesture in the image to be identified based on the hand state includes: The determined hand state is feature-encoded, and the feature encoding is represented by a feature vector. The feature encoding of the hand state is matched with the feature range constraints of each preset gesture to obtain the pre-classified gesture category that satisfies the feature range constraints. Calculate the similarity between the feature encoding of the hand state and each pre-classified gesture category; The pre-classified gesture category with the highest similarity is determined as the gesture classification result in the image to be identified; The overall hand position includes whether the palm faces the camera. Whether the palm faces the camera is determined by whether the joint at the base of the hand is inside the captured hand. If the joint at the base of the hand is inside the captured hand, the palm faces the camera; if the joint at the base of the hand is not inside the captured hand, the palm does not face the camera. The finger position is determined by whether the fingertips intersect with the line segments of the joints at the fingertips and the joints at the palm. The distance between the fingertips is used to determine whether the two fingers are together or apart.

11. A terminal device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the method as described in any one of claims 1 to 9.

12. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method as described in any one of claims 1 to 9.

Images