Gesture recognition method, device, terminal device and computer readable storage medium

Abstract

This application relates to the field of machine vision technology and provides a gesture recognition method, apparatus, terminal device, and computer-readable storage medium. The method includes: acquiring video data of a target scene; if a video image in the video data is detected to meet preset detection conditions, then inputting the video image into a first neural network model for a first hand detection to obtain a first coordinate position of the hand in the video image; determining a hand image containing the hand based on the first coordinate position; and inputting the hand image into a second neural network model for a second hand detection and gesture recognition to obtain a second coordinate position of the hand in the hand image and a first gesture. This application improves the accuracy of hand detection and thus the accuracy of gesture recognition by first performing a coarse hand detection to obtain the first coordinate position of the hand in the video image, then acquiring the hand image based on the first coordinate position, and finally performing gesture recognition on the hand image.

Description

Technical Field

[0001] This application belongs to the field of machine vision technology, and in particular relates to a gesture recognition method, device, terminal equipment and computer-readable storage medium. Background Technology

[0002] Human-computer interaction (HCI) refers to the process of exchanging information between humans and computers using a certain dialogue language and interactive methods. As the most flexible part of the body, the hand is frequently used in HCI for gesture recognition, allowing users to control related devices by recognizing hand gestures.

[0003] Traditional gesture recognition typically involves detecting the presence of hands in acquired videos or images by examining skin color, and then using template matching to identify the type of gesture. However, due to environmental noise (such as lighting conditions), these methods often suffer from low accuracy. In recent years, with the rise of deep learning, deep learning-based gesture recognition has gained increasing attention. Currently, deep learning-based gesture recognition mainly uses neural networks to directly detect and recognize hands. However, because hands occupy a small area, the accuracy of gesture recognition is low when the subject's cooperation in capturing the hands is poor or when the shooting distance is far. Summary of the Invention

[0004] This application provides a gesture recognition method, apparatus, terminal device, and computer-readable storage medium, aiming to solve the problem of low accuracy in existing hand detection and gesture recognition.

[0005] In a first aspect, embodiments of this application provide a gesture recognition method, including:

[0006] Acquire video data of the target scene;

[0007] If the video image in the video data meets the preset detection conditions, the video image is input into the first neural network model to perform the first hand detection, and the first coordinate position of the human hand in the video image is obtained.

[0008] Determine the hand image containing the human hand based on the first coordinate position;

[0009] The hand image is input into the second neural network model for a second hand detection and gesture recognition, resulting in the second coordinate position of the hand in the hand image and the first gesture.

[0010] Secondly, embodiments of this application provide a gesture recognition device, including:

[0011] The acquisition module is used to acquire video data of the target scene;

[0012] The first detection module is used to input the video image into the first neural network model to perform the first hand detection if the video image in the video data meets the preset detection conditions, so as to obtain the first coordinate position of the human hand in the video image.

[0013] The determination module is used to determine an image of a hand containing a human hand based on a first coordinate position;

[0014] The second detection module is used to input the hand image into the second neural network model for a second hand detection and gesture recognition, so as to obtain the second coordinate position of the hand in the hand image and the first gesture.

[0015] Thirdly, embodiments of this application provide a terminal device, which includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the steps of the gesture recognition method described above.

[0016] Fourthly, embodiments of this application provide a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the gesture recognition method described above.

[0017] Fifthly, embodiments of this application provide a computer program product that, when run on a terminal device, causes the terminal device to execute the steps of the aforementioned gesture recognition method.

[0018] The beneficial effects of this application embodiment compared with the prior art are as follows: This application embodiment can first perform a rough detection of the hand in the video image to obtain the first coordinate position of the hand in the video image when the video image in the video data meets the preset detection conditions. Then, the hand image containing the hand is determined according to the first coordinate position. Finally, the hand image is accurately detected and the gesture is recognized by a second detection. This helps to improve the accuracy of hand detection and thus improve the accuracy of gesture recognition.

[0019] It is understood that the beneficial effects of the second to fifth aspects mentioned above can be found in the relevant descriptions in the first aspect mentioned above, and will not be repeated here. Attached Figure Description

[0020] To more clearly illustrate the technical solutions in the embodiments of this application, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0021] Figure 1This is a schematic flowchart of a gesture recognition method provided in an embodiment of this application;

[0022] Figure 2 This is a schematic diagram of a first neural network model provided in an embodiment of this application;

[0023] Figure 3 This is an example diagram of a hand image obtained according to an embodiment of this application;

[0024] Figure 4 This is a schematic diagram showing the result of the second coordinate position of a human hand in a hand image according to an embodiment of this application;

[0025] Figure 5 This is a flowchart illustrating a gesture recognition method provided in another embodiment of this application;

[0026] Figure 6 This is a schematic diagram of the structure of a gesture recognition device provided in an embodiment of this application;

[0027] Figure 7 This is a schematic diagram of the structure of a terminal device provided in an embodiment of this application. Detailed Implementation

[0028] In the following description, specific details such as particular system architectures and techniques are set forth for illustrative purposes and not for limitation, in order to provide a thorough understanding of the embodiments of this application. However, those skilled in the art will understand that this application may also be implemented in other embodiments without these specific details. In other instances, detailed descriptions of well-known systems, apparatuses, circuits, and methods have been omitted so as not to obscure the description of this application with unnecessary detail.

[0029] It should be understood that, when used in this application specification and the appended claims, the term "comprising" indicates the presence of the described features, integrals, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or a collection thereof.

[0030] It should also be understood that the term “and / or” as used in this application specification and the appended claims means any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.

[0031] As used in this application specification and the appended claims, the term "if" may be interpreted, depending on the context, as "when," "once," "in response to determination," or "in response to detection." Similarly, the phrase "if determined" or "if detected [the described condition or event]" may be interpreted, depending on the context, as meaning "once determined," "in response to determination," "once detected [the described condition or event]," or "in response to detection [the described condition or event]."

[0032] Furthermore, in the description of this application and the appended claims, the terms "first," "second," "third," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0033] References to "one embodiment" or "some embodiments" as described in this specification mean that one or more embodiments of this application include a specific feature, structure, or characteristic described in connection with that embodiment. Therefore, the phrases "in one embodiment," "in some embodiments," "in other embodiments," "in still other embodiments," etc., appearing in different parts of this specification do not necessarily refer to the same embodiment, but rather mean "one or more, but not all, embodiments," unless otherwise specifically emphasized. The terms "comprising," "including," "having," and variations thereof mean "including but not limited to," unless otherwise specifically emphasized.

[0034] The gesture recognition method provided in this application can be applied to terminal devices including cameras or those communicatively connected to cameras. These terminal devices can be televisions, robots, mobile phones, tablets, wearable devices, in-vehicle devices, augmented reality (AR) / virtual reality (VR) devices, laptops, ultra-mobile personal computers (UMPCs), netbooks, personal digital assistants (PDAs), and other similar devices. This application does not impose any limitations on the specific type of terminal device.

[0035] In one application scenario, when the gesture recognition method is applied to a television equipped with a camera or connected to a camera, the user can remotely control the television using gestures, achieving human-computer interaction. The distance between the user and the television's camera is typically 2m to 4m. Therefore, the user's gestures occupy a small portion of the video footage captured by the television's camera and are easily affected by environmental noise. In this application scenario, the gesture recognition method of this application achieves high accuracy in recognizing user gestures, effectively improving the smoothness of controlling the television based on the recognition results. Environmental noise refers to background objects, strong light, weak light, or light with a large range of intensity variations in the video footage captured by the television's camera.

[0036] To illustrate the technical solution of this application, the following embodiments will be used for explanation.

[0037] Please see Figure 1 This application provides a gesture recognition method that can be executed by a camera or by a terminal device (such as a television) connected to the camera. The gesture recognition method includes:

[0038] Step S101: Obtain video data of the target scene.

[0039] Specifically, in the target application scenario, video data of the target scene captured by the camera is acquired. The video data can be a video stream composed of multiple images in sequence. When multiple images are played at a rate exceeding a certain number per second, according to the principle of visual persistence, the human eye sees the video as having a smooth and continuous visual effect.

[0040] Step S102: If the video image in the video data meets the preset detection conditions, the video image is input into the first neural network model to perform the first hand detection, and the first coordinate position of the human hand in the video image is obtained.

[0041] Specifically, the detection process can begin with the first video image in the video data to check if it meets preset detection conditions. Meeting these conditions could mean whether the current video image is a keyframe. To satisfy visual characteristics, adjacent frames in video data are typically highly correlated; therefore, every N frames can be used as keyframes. When a video image meets the preset detection conditions, it is input into a first neural network model for the first hand detection. This first hand detection involves obtaining the first coordinate position of the hand in the video image when a hand is detected. The first neural network model is a neural network used for preliminary detection of hands in video images; it can detect the presence of a hand and output its position in the video image when it is present.

[0042] In one embodiment, before detecting that a video image in the video data meets a preset detection condition, the method further includes:

[0043] Adjust the color gamut of the video image to the target color gamut; and / or,

[0044] Adjust the size of the video image to the preset size.

[0045] Specifically, adjusting the color gamut of a video image to the target color gamut can be achieved by converting the video image to RGB format. For example, after acquiring video data, the video image read from the video data using deep learning methods is usually in BRG format, so the BRG format video image is converted to RGB format. Adjusting the size of a video image to a preset size can be achieved by scaling the video image to a preset size. This can be done using a scaling algorithm, such as using bicubic interpolation (i.e., the bicubic algorithm) to scale the image proportionally to the preset size. For instance, the length of the image can be scaled proportionally to 320 pixels or another value, and the width can be scaled proportionally to the corresponding size. Assuming the acquired video image has a length of 400 and a width of 300 pixels, and the preset size has a length of 320, scaling the video image to the preset size using bicubic interpolation can be achieved by scaling the length of the video image pixels to 320 and the width of the video image pixels to 200 = (320 / 400) * 300. It should be noted that this is only an example for understanding this application and is not intended to limit the application. In practical applications, the above image size and preset size can be other values, and there is no limitation on them.

[0046] In one embodiment, a video image in the video data satisfies a preset detection condition if the video image in the video data is located within a preset frame number range in the video data.

[0047] Specifically, whether a video image in the video data meets the preset detection conditions can be determined based on the frame number of the current video image within the video data. Meeting the preset detection conditions means that the video image's frame number falls within a preset frame number range. For example, keyframes can be pre-set based on frame numbers, such as frame 1, frame 5, frame 10, etc. If the frame number of the current video image matches a keyframe, then the current video image is determined to meet the preset detection conditions; otherwise, it is determined not to meet the preset detection conditions.

[0048] In one embodiment, if no hand is detected in the current video image, it is determined that no hand has been detected, the next frame is obtained as the current video image, and the process returns to step S102 and subsequent steps.

[0049] Specifically, before the video image is input into the first neural network model for the first hand detection, the pre-built first neural network model is trained to obtain a trained first neural network model. The output of the first neural network model is the position of the hand in the video image when a hand is detected. The first neural network model can be designed using a lightweight network, such as... Figure 2 The diagram shows the network structure of a first neural network model built based on the Mobilenet_SSD network structure. The input video image to this first neural network is a preprocessed image scaled to 320P.

[0050] Step S103: Determine the hand image containing the human hand based on the first coordinate position.

[0051] Specifically, when a human hand is detected in a video image, the resulting hand image can be an image cropped from the hand region based on a first coordinate position. Since this hand image is detected from a video image where the hand occupies a relatively large portion of the frame, it typically includes not only the hand but also other background areas of the video image.

[0052] Step S104: Input the hand image into the second neural network model to perform a second hand detection and gesture recognition, and obtain the second coordinate position of the hand and the first gesture in the hand image.

[0053] Specifically, the hand image is input into a second neural network model for a second hand detection and gesture recognition, obtaining a gesture recognition result. This result can be the precise location of the hand in the image and the type of gesture. The second neural network model is used to accurately determine the position of the hand in the hand image and identify the hand gesture. The gestures identified by the second neural network model are used as the gesture recognition result. Different gestures are pre-associated with different operations; executing the operations associated with the gesture recognition result enables human-computer interaction.

[0054] Before inputting hand images into the second neural network model for gesture recognition, the pre-built second neural network model is trained. The training process involves preparing a large number of gestures from various perspectives and other types of gestures, and generating test images in environments with varying brightness, darkness, blurriness, and noise. The gesture categories in the test images are labeled, such as: stop, previous, next, like, fist, other gestures, and background. Each category includes at least 1000 test images, each with manually labeled hand coordinates. The prepared large number of test images are used to train the second neural network model. The second neural network model can be constructed by using the Mobilenet v2 network as the backbone to extract features, followed by a gesture recognition network structure and a hand position localization structure. The gesture recognition structure can consist of three layers: a global pooling layer, a 1*1 convolutional layer, and a softmax activation function. Finally, the category with the highest probability is output as the recognition result. The hand positioning structure can be composed of two layers of network, conv1*1 and conv1*1, to obtain the coordinate position of the human hand in the hand image.

[0055] In one embodiment, when the gesture category identified by the second neural network is the background category, it is also determined that no human hand has been identified, the next frame in the video data is obtained as the current video image, and the process returns to step S102 and subsequent steps.

[0056] In one embodiment, determining an image of a hand containing a human hand based on a first coordinate position includes:

[0057] Obtain the hand detection box corresponding to the first coordinate position;

[0058] The hand detection box is enlarged according to a preset scaling factor to obtain the enlarged hand detection box;

[0059] The video image is cropped based on the magnified hand detection box to obtain the cropped video image.

[0060] Normalize and / or remove the mean from the cropped video image to obtain a hand image containing the human hand.

[0061] Based on the initial hand detection, the first coordinate position of the corresponding pixel in the video image is obtained, resulting in a hand detection bounding box. This bounding box is then enlarged by multiplying its width and height by a preset scaling factor. The preset scaling factor for both width and height can be set to 1.3, or other values ​​can be set separately, depending on the specific application; there are no restrictions on this. Next, the video image is cropped based on the position of the enlarged detection bounding box in the video image. The cropped video image is then normalized and / or mean-removed. The image processed by normalization and / or mean-removal is used as the hand image. Normalization can involve scaling the cropped video image. Mean-removal can be performed by calculating the image mean before the hand image is input into the second neural network model. Images typically exhibit a stable data distribution; mean-removal subtracts the statistical average of the corresponding dimension, eliminating common parts of the image and highlighting its features.

[0062] In one embodiment, after inputting a hand image into a second neural network model for a second hand detection and gesture recognition to obtain the second coordinate position of the hand in the hand image and the first gesture, the method further includes:

[0063] The third coordinate position of the hand in the video image is determined based on the second coordinate position of the hand in the hand image and the first coordinate position of the hand image in the video image.

[0064] Specifically, if the proportion of a person's hand in a video image is relatively small, the hand image obtained after cropping the video image will usually be a rather coarse location, such as... Figure 3 The image shown is a schematic diagram of a hand obtained after cropping a video image. By repositioning the hand within the second coordinate of this image, a more precise location can be obtained. Please refer to [link to relevant documentation]. Figure 4 The diagram illustrates the result of the second coordinate position of the hand in the hand image. Therefore, the third coordinate position of the hand in the video image can be determined based on the second coordinate position of the hand in the hand image and the first coordinate position of the hand image in the video image.

[0065] In one embodiment, after determining the third coordinate position of the hand in the video image, the method further includes:

[0066] Based on the coordinates of the hand in the first N frames of the video image, the third coordinate position of the hand in the video image is smoothed to obtain the target coordinate position of the hand in the video image, where N is an integer greater than 1.

[0067] Based on the hand gestures in the first N frames of video images, the first gesture is subjected to a second smoothing process to obtain the second gesture.

[0068] Specifically, to meet visual requirements, adjacent frames in video data are typically highly correlated. Smoothing is achieved by utilizing the coordinate positions of several consecutive past frames (e.g., the five most recent consecutive frames) of the current video image, thus preventing frequent jitter in the coordinate positions. Smoothing the current gesture recognition result using the recognition results of several consecutive past frames (e.g., the five most recent consecutive frames) prevents repeated jumps in the recognition result.

[0069] In one specific embodiment, based on the coordinate positions of the hand in the first N frames of the video image, a first smoothing process is performed on the third coordinate position of the hand in the video image to obtain the target coordinate position of the hand in the video image, including:

[0070] Obtain the coordinates of the human hand in the first N frames of the video image;

[0071] Based on the coordinates of the hand in the first N frames of video images, obtain the first center position of the hand in each of the first N frames of video images;

[0072] The second center position of the hand in the video image is determined based on the third coordinate position of the hand in the video image;

[0073] Determine the average center position of the N first center positions based on the N first center positions;

[0074] If the distance between the average center position and the second center position is within a preset transformation difference range, then the third coordinate position of the hand in the video image is determined as the target coordinate position; or,

[0075] If the distance between the average center position and the second center position is not within the preset transformation difference range, then adjust the third coordinate position of the hand in the video image to obtain the fourth coordinate position of the hand in the video image; determine the third center position of the hand in the video image based on the fourth coordinate position of the hand in the video image; if the distance between the average center position and the third center position is within the preset transformation difference range, then determine the fourth coordinate position of the hand in the video image as the target coordinate position.

[0076] Specifically, the first smoothing process can be as follows: Using the coordinates of the hand obtained from the previous N frames of video, the first center position of the hand is obtained from each of the previous N frames. An average center position is calculated from these N first center positions. The second center position of the hand is calculated based on the third coordinate position of the hand in the current video image. Since the correlation between consecutive N frames in the video data is relatively high, the difference between the calculated average center position and the first center position usually does not exceed a certain value. A preset range of variation is set and stored. If the distance between the average center position and the first center position is within this preset range, the third coordinate position in the current video image is considered the accurate hand coordinate position, and this position is used as the final coordinate position of the video image.

[0077] Alternatively, if the distance between the average center position and the second center position of the video image is not within the preset transformation difference range, the third coordinate position of the hand in the video image is adjusted to obtain the fourth coordinate position of the hand, so that the second center position of the hand in the video image is within the preset transformation difference range. The third center position of the hand in the current video image is determined based on the fourth coordinate position; if the distance between the average center position and the third center position is within the preset transformation difference range, the fourth coordinate position of the hand in the video image is determined as the final coordinate position of the video image. Adjusting the third coordinate position of the hand in the video image can be done by gradually increasing or decreasing the corresponding horizontal and vertical coordinate values ​​by preset values ​​until the distance between the average center position and the third center position is within the preset transformation difference range, at which point the adjustment stops.

[0078] In one specific embodiment, based on the hand gestures in the first N frames of video images, a second smoothing process is applied to the first gesture to obtain a second gesture, including:

[0079] The predicted gesture in the video image is determined based on the hand gestures in the previous N frames of video images and the preset smoothing relationship of the transformed gesture motion.

[0080] If the first gesture matches the predicted gesture, then the first gesture is used as the second gesture; or,

[0081] If the first gesture does not match the predicted gesture, the predicted gesture will be used as the second gesture.

[0082] Specifically, the gesture in the (N+1)th frame can be predicted based on a preset smoothing relationship of the gesture transformation and the gestures in the previous N frames. For example, assuming N is 5, if the gesture in 4 out of the first 5 frames is the first gesture, then the gesture in the 6th frame is predicted to be the first gesture as well. Similarly, if the gesture in the last two out of the first 5 frames is the first gesture, then the gesture in the 6th frame is predicted to be the first gesture, and so on. This is an example provided to illustrate the technical solution of this application and is not intended to limit the application.

[0083] In another embodiment, please refer to Figure 5 After acquiring the video data of the target scene, the method further includes steps S201 to S203:

[0084] Step S201: If the video image in the video data does not meet the preset detection conditions, then obtain the coordinate position of the hand in the previous frame of the video image.

[0085] Specifically, if the current video image does not meet the preset detection conditions, the coordinates of the hand finally determined in the previous frame of the current video image are obtained, and then step S202 is executed. If the target coordinates of the hand are not obtained in the previous frame of the current video image, it is determined that no hand has been detected and subsequent steps are not executed.

[0086] Step S202: Obtain a hand image based on the target coordinates of the hand in the previous frame of the video image.

[0087] Specifically, the hand image can be obtained based on the target coordinates of the hand in the previous frame of the video image. Since there will be some differences between the previous frame and the current video image, the hand image obtained at this time is also a roughly detected hand image.

[0088] Step S203: Input the hand image into the second neural network model to perform a second hand detection and gesture recognition, and obtain the second coordinate position of the hand in the hand image and the first gesture.

[0089] Specifically, the hand image obtained in step S202 is input into the second neural network model for a second hand detection and gesture recognition to obtain the gesture recognition result. The gesture recognition result can be the precise position of the hand in the image and the category of the gesture.

[0090] Prior to one embodiment, before inputting the hand image obtained in step S202 into the second neural network model for gesture recognition, the following steps are included:

[0091] Enlarge the hand image according to a preset scaling factor;

[0092] Normalize and / or remove the mean from the magnified hand image.

[0093] Since the first coordinate position of the hand in the video image can be obtained by first performing a rough detection when the video image in the video data meets the preset detection conditions; then the hand image containing the hand can be determined based on the first coordinate position; and then the hand image can be subjected to a second hand detection and gesture recognition, the accuracy of gesture recognition can be improved.

[0094] This application also provides a gesture recognition device for performing the steps described in the gesture recognition method embodiments above. The gesture recognition device can be a virtual appliance within a terminal device, run by the terminal device's processor, or it can be the terminal device itself.

[0095] like Figure 6 As shown, the gesture recognition device 600 provided in this application embodiment includes:

[0096] Acquisition module 601 is used to acquire video data of the target scene;

[0097] The first detection module 602 is used to input the video image into the first neural network model to perform the first hand detection if the video image in the video data meets the preset detection conditions, so as to obtain the first coordinate position of the human hand in the video image.

[0098] The determining module 603 is used to determine an image of a hand containing a human hand based on the first coordinate position;

[0099] The second detection module 604 is used to input the hand image into the second neural network model for a second hand detection and gesture recognition, so as to obtain the second coordinate position of the hand and the first gesture in the hand image.

[0100] In one embodiment, the gesture recognition device 600 further includes:

[0101] A preprocessing module is used to adjust the color gamut of the video image to the target color gamut; and / or,

[0102] Adjust the size of the video image to the preset size.

[0103] In one embodiment, the determining module 603 is specifically used for:

[0104] Obtain the hand detection box corresponding to the first coordinate position;

[0105] The hand detection box is enlarged according to a preset scaling factor to obtain the enlarged hand detection box;

[0106] The video image is cropped based on the magnified hand detection box to obtain the cropped video image.

[0107] Normalize and / or remove the mean from the cropped video image to obtain a hand image containing the human hand.

[0108] In one embodiment, a video image in the video data satisfies a preset detection condition if the video image in the video data is located within a preset frame number range in the video data.

[0109] In one embodiment, the gesture recognition device 600 further includes:

[0110] The position determination module is used to determine the third coordinate position of the hand in the video image based on the second coordinate position of the hand in the hand image and the first coordinate position of the hand image in the video image.

[0111] In one embodiment, the gesture recognition device 600 further includes:

[0112] The first smoothing module is used to perform a first smoothing process on the third coordinate position of the hand in the video image based on the coordinate position of the hand in the first N frames of the video image, so as to obtain the target coordinate position of the hand in the video image, where N is an integer greater than 1.

[0113] The second smoothing module is used to perform a second smoothing process on the first gesture based on the hand gestures in the previous N frames of video images to obtain a second gesture.

[0114] In one embodiment, the first smoothing module includes:

[0115] The first acquisition unit is used to acquire the coordinate positions of the human hand in the first N frames of the video image;

[0116] The second acquisition unit is used to acquire the first center position of the hand in the first N frames of video images based on the coordinate position of the hand in the first N frames of video images.

[0117] The first determining unit is used to determine the second center position of the hand in the video image based on the third coordinate position of the hand in the video image;

[0118] The second determining unit is used to determine the average center position of the N first center positions based on the N first center positions;

[0119] The third determining unit is used to determine the third coordinate position of the hand in the video image as the target coordinate position if the distance between the average center position and the second center position is within a preset transformation difference range.

[0120] The adjustment unit is used to adjust the third coordinate position of the hand in the video image to obtain the fourth coordinate position of the hand in the video image if the distance between the average center position and the second center position is not within the preset transformation difference range; determine the third center position of the hand in the video image based on the fourth coordinate position of the hand in the video image; and determine the fourth coordinate position of the hand in the video image as the target coordinate position if the distance between the average center position and the third center position is within the preset transformation difference range.

[0121] In one embodiment, the second smoothing module includes:

[0122] The prediction unit is used to determine the predicted gesture of the video image based on the hand gestures in the previous N frames of video images and the preset smoothing relationship of the transformed gesture motion.

[0123] The first matching determination unit is used to determine the first gesture as the second gesture if the first gesture matches the predicted gesture.

[0124] The second matching determination unit is used to take the predicted gesture as the second gesture if the first gesture does not match the predicted gesture.

[0125] In this embodiment of the application, when a video image in the video data meets the preset detection conditions, a rough hand detection is first performed to obtain the first coordinate position of the hand in the video image; then, a hand image containing the hand is determined based on the first coordinate position; and then, a precise hand detection and gesture recognition are performed on the hand image, which helps to improve the accuracy of hand detection and thus improve the accuracy of gesture recognition.

[0126] like Figure 7 As shown, one embodiment of this application also provides a terminal device 700 including: a processor 701, a memory 702, and a computer program 703 stored in the memory 702 and executable on the processor 701, such as a gesture recognition program. When the processor 701 executes the computer program 703, it implements the steps in the various gesture recognition method embodiments described above. When the processor 701 executes the computer program 703, it implements the functions of each module in the various device embodiments described above, for example... Figure 6 The functions of modules 601 to 604 are shown.

[0127] For example, computer program 703 can be divided into one or more modules, one or more of which are stored in memory 702 and executed by processor 701 to complete the present invention. One or more modules can be a series of computer program instruction segments capable of performing specific functions, which describe the execution process of computer program 703 in terminal device 700. For example, computer program 703 can be divided into an acquisition module, a first detection module, a determination detection module, and a second detection module. The specific functions of each module have been described in the above embodiments and will not be repeated here.

[0128] Terminal device 700 can be a television, robot, mobile phone, tablet computer, wearable device, in-vehicle device, augmented reality (AR) / virtual reality (VR) device, laptop computer, ultra-mobile personal computer (UMPC), netbook, personal digital assistant (PDA), and other computing devices. The terminal device may include, but is not limited to, processor 701 and memory 702. Those skilled in the art will understand that... Figure 7 This is merely an example of terminal device 700 and does not constitute a limitation on terminal device 700. It may include more or fewer components than shown, or combine certain components, or different components. For example, terminal device may also include input / output devices, network access devices, buses, etc.

[0129] The processor 701 may be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor.

[0130] The memory 702 can be an internal storage unit of the terminal device 700, such as a hard disk or RAM of the terminal device 700. The memory 702 can also be an external storage device of the terminal device 700, such as a plug-in hard disk, Smart Media Card (SMC), Secure Digital (SD) card, or FlashCard equipped on the terminal device 700. Furthermore, the memory 702 can include both internal and external storage units of the terminal device 700. The memory 702 is used to store computer programs and other programs and data required by the terminal device. The memory 702 can also be used to temporarily store data that has been output or will be output.

[0131] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is merely an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit. Furthermore, the specific names of the functional units and modules are only for easy differentiation and are not intended to limit the scope of protection of this application. The specific working process of the units and modules in the above system can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0132] In the above embodiments, the descriptions of each embodiment have different focuses. For parts that are not described in detail or recorded in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0133] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0134] In the embodiments provided by this invention, it should be understood that the disclosed apparatus / terminal devices and methods can be implemented in other ways. For example, the apparatus / terminal device embodiments described above are merely illustrative. For instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0135] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0136] Furthermore, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0137] If the integrated module is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments of the present invention can also be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include: any entity or device capable of carrying computer program code, recording media, USB flash drives, portable hard drives, magnetic disks, optical disks, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc. It should be noted that the content included in the computer-readable medium can be appropriately added or removed according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, the computer-readable medium does not include electrical carrier signals and telecommunication signals.

[0138] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included within the protection scope of the present invention.

Claims

1. A gesture recognition method, characterized in that, include: Acquire video data of the target scene; Every N frames of video image in the video data is used as a keyframe. If the video image in the video data is detected to meet the preset detection conditions, the video image is input into the first neural network model to perform the first hand detection, and the first coordinate position of the human hand in the video image is obtained. If the current video image is a keyframe, then the current video image is determined to meet the preset detection conditions; If the current video image is not a keyframe, then the current video image is determined not to meet the preset detection conditions; A hand image containing the human hand is determined based on the first coordinate position; If the current video image does not meet the preset detection conditions, the hand image is obtained based on the coordinate position of the hand finally determined in the previous frame of the current video image. The hand image is input into a second neural network model for a second hand detection and gesture recognition, thereby obtaining the second coordinate position of the human hand in the hand image and the first gesture.

2. The method according to claim 1, characterized in that, Before detecting that the video image in the video data meets the preset detection conditions, the method further includes: Adjust the color gamut of the video image to the target color gamut; and / or, Adjust the size of the video image to a preset size.

3. The method according to claim 2, characterized in that, Determining the hand image containing the human hand based on the first coordinate position includes: Obtain the hand detection box corresponding to the first coordinate position; The hand detection frame is enlarged according to a preset scaling factor to obtain an enlarged hand detection frame; The video image is cropped based on the magnified hand detection frame to obtain the cropped video image; The cropped video image is normalized and / or mean-removed to obtain a hand image containing the human hand.

4. The method according to claim 3, characterized in that, The video images in the video data meet the preset detection condition that the video images in the video data are located within a preset frame number range in the video data.

5. The method according to any one of claims 1 to 4, characterized in that, After inputting the hand image into a second neural network model for a second hand detection and gesture recognition to obtain the second coordinate position of the hand in the hand image and the first gesture, the method further includes: The third coordinate position of the human hand in the video image is determined based on the second coordinate position of the human hand in the hand image and the first coordinate position of the hand image in the video image.

6. The method according to claim 5, characterized in that, After determining the third coordinate position of the human hand in the video image, the method further includes: Based on the coordinate positions of the hand in the first N frames of the video image, a first smoothing process is performed on the third coordinate position of the hand in the video image to obtain the target coordinate position of the hand in the video image, where N is an integer greater than 1; Based on the hand gestures in the first N frames of video images, the first gesture is subjected to a second smoothing process to obtain a second gesture.

7. The method according to claim 6, characterized in that, The step of performing a first smoothing process on the third coordinate position of the hand in the video image based on the coordinate positions of the hand in the previous N frames of the video image to obtain the target coordinate position of the hand in the video image includes: Obtain the coordinates of the human hand in the first N frames of the video image; Based on the coordinates of the hand in the first N frames of video images, the first center position of the hand in the first N frames of video images is obtained respectively; The second center position of the hand in the video image is determined based on the third coordinate position of the hand in the video image; The average center position of the N first center positions is determined based on the N first center positions; If the distance between the average center position and the second center position is within a preset transformation difference range, then the third coordinate position of the hand in the video image is determined as the target coordinate position; or, If the distance between the average center position and the second center position is not within the preset transformation difference range, then the third coordinate position of the hand in the video image is adjusted to obtain the fourth coordinate position of the hand in the video image; the third center position of the hand in the video image is determined based on the fourth coordinate position of the hand; if the distance between the average center position and the third center position is within the preset transformation difference range, then the fourth coordinate position of the hand in the video image is determined as the target coordinate position.

8. The method according to claim 7, characterized in that, The step of performing a second smoothing process on the first gesture based on the hand gestures in the previous N frames of video images to obtain a second gesture includes: The predicted gesture of the video image is determined based on the hand gestures in the first N frames of video images and the preset smoothing relationship of the gesture transformation motion. If the first gesture matches the predicted gesture, then the first gesture is used as the second gesture; or, If the first gesture does not match the predicted gesture, then the predicted gesture is used as the second gesture.

9. A gesture recognition device, characterized in that, include: The acquisition module is used to acquire video data of the target scene; Every N frames of video image in the video data is used as a keyframe. The first detection module is used to input the video image into the first neural network model to perform the first hand detection if the video image in the video data meets the preset detection conditions, so as to obtain the first coordinate position of the human hand in the video image. If the current video image is a keyframe, then the current video image is determined to meet the preset detection conditions; If the current video image is not a keyframe, then the current video image is determined not to meet the preset detection conditions; The determining module is used to determine a hand image containing the human hand based on the first coordinate position; if the current video image does not meet the preset detection conditions, the hand image is obtained based on the coordinate position of the human hand finally determined in the previous frame of the current video image. The second detection module is used to input the hand image into the second neural network model for a second hand detection and gesture recognition, so as to obtain the second coordinate position of the human hand in the hand image and the first gesture.

10. A terminal device, characterized in that, The terminal device includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the method as described in any one of claims 1 to 8.

11. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the method as described in any one of claims 1 to 8.

Images