A dynamic gesture recognition method based on deep learning and related equipment

Abstract

The application discloses a dynamic gesture recognition method based on deep learning and related equipment, and belongs to the technical field of behavior action recognition. The application acquires a video stream in real time, extracts a gesture image sequence from the video stream through a hand detection model, inputs the gesture image sequence into a pre-trained gesture recognition model, and outputs a gesture recognition result. The gesture recognition model comprises a convolution unit, a prediction unit and an integration unit. The gesture image sequence is subjected to channel separation convolution operation through the convolution unit, so that image convolution sequence features are obtained. The gesture features of adjacent images before and after each frame of image in the gesture image sequence are predicted through the prediction unit, so that interframe gesture action features are obtained. The interframe gesture action features are integrated through the integration unit, and the gesture recognition result is output. The application provides a complete solution for dynamic gesture triggering collection, checking and recognition.

Description

Technical Field

[0001] This application belongs to the field of behavior and action recognition technology, specifically relating to a dynamic gesture recognition method and related equipment based on deep learning. Background Technology

[0002] Gestures, as a form of daily communication, offer advantages such as intuitiveness, naturalness, and comfort compared to facial expressions, actions, and other communication methods. Therefore, gestures are among the most commonly used communication tools besides language and are a crucial component of human-computer interaction. Gesture recognition technology has wide applications and is highly valuable in fields such as the Internet of Things (IoT), drones, autonomous driving, medical assistance, virtual reality (VR), and augmented reality (AR). In autonomous driving and drones, intelligent control systems collect and analyze gesture information, issuing commands based on the results to achieve navigation and interactive control of vehicles or drones. In VR and AR, Microsoft's HoloLens has enabled users to enjoy entertainment in virtual environments using their hands. In medical assistance, gesture recognition can assist the hearing impaired, enabling communication between deaf individuals or between deaf individuals and hearing individuals.

[0003] Gestures can be categorized into dynamic and static gestures based on their state, and further classified into two-dimensional planar hand gestures and three-dimensional gestures based on the object being recognized. By recombining these two types of gestures, we can obtain three main categories of gesture recognition: two-dimensional static hand gesture recognition, two-dimensional dynamic gesture recognition, and three-dimensional dynamic gesture recognition. Among these, three-dimensional dynamic gestures have become a research hotspot in this field in recent years due to their superior information expression capabilities, while two-dimensional dynamic gesture recognition has become the preferred choice for many practical application scenarios due to its smaller data size and ease of acquisition.

[0004] Currently, for dynamic gesture recognition, deep learning-based gesture recognition methods either focus on improving recognition accuracy, such as the paper "Contains gesture segmentation and recognition using 3DCNN and variable LSTM," which uses a combination of ResNet34 + LSTM + MobileNet to achieve an accuracy of around 95% (on the Jester dataset). However, this combined network approach results in a massive model size, making modeling very difficult, with over 60 million network parameters. Another approach consists of single-network and lightweight networks, such as C3D and MobileNet, which have only half (C3D) or even one-tenth (MobileNet2V) of the parameters of the combined networks mentioned above. However, these lightweight networks generally achieve an accuracy of around 80%, significantly lower than the combined networks. Furthermore, current gesture recognition schemes primarily focus on the gesture recognition process, paying less attention to the beginning and end stages of the gesture, leading to a large computational load and impacting recognition efficiency. Summary of the Invention

[0005] The purpose of this application is to propose a dynamic gesture recognition method, device, computer equipment, and storage medium based on deep learning, so as to solve the technical problems of large model size, high modeling difficulty, low recognition accuracy, and large model computation in existing gesture recognition schemes, which affect recognition efficiency.

[0006] To address the aforementioned technical problems, this application provides a deep learning-based dynamic gesture recognition method, employing the following technical solution:

[0007] A deep learning-based dynamic gesture recognition method includes:

[0008] Responding to gesture recognition commands, it acquires the video stream uploaded by the client in real time;

[0009] The video stream is imported into a pre-trained hand detection model, which then extracts a sequence of gesture images from the video stream.

[0010] The gesture image sequence is input into a pre-trained gesture recognition model in the form of a tensor, and the gesture recognition result is output. The gesture recognition model includes a convolutional unit, a prediction unit, and an ensemble unit. The convolutional unit is trained based on a channel-separated convolutional C3D network, the prediction unit is trained based on a Bi-LSTM network, and the ensemble unit is trained based on a Mobi LenetV2 network.

[0011] The image convolution sequence features are obtained by performing channel-separated convolution operations on the gesture image sequence using convolution units.

[0012] The gesture features between frames are obtained by predicting the gesture features of the adjacent images before and after each frame in the gesture image sequence using the prediction unit.

[0013] The gesture recognition result is output by integrating the gesture motion features between frames through the integration unit.

[0014] Furthermore, the pre-trained hand detection model is trained based on a YOLOv5 network. The video stream is imported into the pre-trained hand detection model, which extracts gesture image sequences from the video stream, specifically including:

[0015] parse the video stream and obtain keyframe images from the video stream;

[0016] Import the keyframe images into a pre-trained hand detection model to identify hand feature images in the keyframe images;

[0017] Combine hand feature images to construct a sequence of gesture images.

[0018] Furthermore, hand feature images are combined to construct a gesture image sequence, specifically including:

[0019] Identify the hand feature images in the keyframe images and store the hand feature images in memory;

[0020] When the number of identified hand feature images meets the preset first numerical threshold N1, the currently stored hand feature images are cleared, where the first numerical threshold N1 is a positive integer;

[0021] Continue to identify the remaining hand feature images until all hand feature images in the keyframe images have been identified;

[0022] Combine the hand feature images in the current memory to construct a sequence of gesture images.

[0023] Furthermore, the hand feature images currently in memory are combined to construct a gesture image sequence, specifically including:

[0024] Calculate the number of hand feature images in the current memory to obtain the number of hand images;

[0025] The number of hand images is compared with the preset second numerical threshold N2 and the preset third numerical threshold N3, respectively, where the second numerical threshold N2 and the third numerical threshold N3 are both positive integers;

[0026] When the number of hand images equals the second numerical threshold N2, combine the images in the current memory to construct a gesture image sequence;

[0027] When the number of hand images is less than the second numerical threshold N2, a preset blank image is added to the memory until the number of images in the current memory is equal to the second numerical threshold N2. The images in the current memory are then combined to construct a gesture image sequence.

[0028] When the number of hand images is greater than the second numerical threshold N2 and the number of hand images is less than or equal to the third numerical threshold N3, the images in the current memory are filtered using the preset first filtering rule, and the filtered images are combined to construct a gesture image sequence.

[0029] When the number of hand images exceeds the third numerical threshold N3, the images in the current memory are filtered using a preset second filtering rule, and the filtered images are combined to construct a gesture image sequence.

[0030] Furthermore, the relationship between the first numerical threshold N1, the second numerical threshold N2, and the third numerical threshold N3 is: N3 = 2N2, N1 = 3N2.

[0031] Furthermore, the images in the current memory are filtered using a preset first filtering rule, and the filtered images are combined to construct a gesture image sequence, specifically including:

[0032] Obtain the filtering frequency corresponding to the first filtering rule to get the first filtering frequency;

[0033] The images in the current memory are filtered at a first filtering frequency to obtain the first filtered images, and the number of the first filtered images is counted.

[0034] Compare the number of the first filtered images with the second numerical threshold N2;

[0035] When the number of images selected in the first screening is equal to the second numerical threshold N2, the images in the current memory are combined to construct a gesture image sequence.

[0036] When the number of images selected in the first screening is less than the second numerical threshold N2, blank images are added to memory until the number of images in memory is equal to the second numerical threshold N2. The images in memory are then combined to construct a gesture image sequence.

[0037] Furthermore, the images in the current memory are filtered using a preset second filtering rule, and the filtered images are combined to construct a gesture image sequence, specifically including:

[0038] Obtain the filtering frequency corresponding to the second filtering rule to get the second filtering frequency;

[0039] The images in the current memory are filtered using a second filtering frequency to obtain second-filtered images, and the number of second-filtered images is counted.

[0040] Compare the number of second-selected images with the second numerical threshold N2;

[0041] When the number of images selected in the second screening is equal to the second numerical threshold N2, the images in the current memory are combined to construct a gesture image sequence.

[0042] When the number of images in the second selection is less than the second numerical threshold N2, blank images are added to memory until the number of images in memory is equal to the second numerical threshold N2. The images in memory are then combined to construct a gesture image sequence.

[0043] To address the aforementioned technical problems, this application also provides a deep learning-based dynamic gesture recognition device, employing the following technical solution:

[0044] A deep learning-based dynamic gesture recognition device includes:

[0045] The video upload module is used to respond to gesture recognition commands and acquire the video stream uploaded by the client in real time;

[0046] The image recognition module is used to import the video stream into a pre-trained hand detection model, and extract the gesture image sequence from the video stream through the hand detection model;

[0047] The gesture recognition module is used to input the gesture image sequence into the pre-trained gesture recognition model in the form of tensors and output the gesture recognition result. The gesture recognition model includes a convolutional unit, a prediction unit, and an ensemble unit. The convolutional unit is trained based on the C3D network with channel-separated convolution, the prediction unit is trained based on the Bi-LSTM network, and the ensemble unit is trained based on the Mobi LenetV2 network.

[0048] The convolution operation module is used to perform channel-separated convolution operations on the gesture image sequence through convolution units to obtain the image convolution sequence features;

[0049] The gesture prediction module is used to predict the gesture features of the preceding and following images of each frame in the gesture image sequence through the prediction unit, so as to obtain the inter-frame gesture action features.

[0050] The gesture integration module is used to integrate the gesture action features between frames through the integration unit and output the gesture recognition result.

[0051] To address the aforementioned technical problems, this application also provides a computer device that employs the following technical solution:

[0052] A computer device includes a memory and a processor, the memory storing computer-readable instructions, the processor executing the computer-readable instructions to implement the steps of the deep learning-based dynamic gesture recognition method as described in any of the preceding claims.

[0053] To address the aforementioned technical problems, this application also provides a computer-readable storage medium, employing the technical solution described below:

[0054] A computer-readable storage medium storing computer-readable instructions, which, when executed by a processor, implement the steps of the deep learning-based dynamic gesture recognition method as described in any one of the preceding descriptions.

[0055] Compared with the prior art, the embodiments of this application have the following main advantages:

[0056] This application discloses a deep learning-based dynamic gesture recognition method and related equipment, belonging to the field of action recognition technology. The solution improves upon the comprehensive network model in traditional deep learning gesture recognition schemes by introducing a channel-separable C3D network and a MobiLenetV2 network, significantly reducing the number of network model parameters, decreasing the network model size, and lowering modeling difficulty. The introduction of a bidirectional Bi-LSTM network enhances the ability to extract information from dynamic gestures over time, thereby better extracting the feature information of dynamic gestures. Furthermore, before gesture recognition, this application uses a pre-trained hand detection model to detect hand joints in the input video stream to identify gesture images in the video stream, reducing the data processing load of the subsequent gesture recognition model and improving the efficiency and accuracy of gesture recognition. Attached Figure Description

[0057] To more clearly illustrate the solutions in this application, the accompanying drawings used in the description of the embodiments of this application will be briefly introduced below. Obviously, the accompanying drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0058] Figure 1 An exemplary system architecture diagram is shown, in which this application can be applied;

[0059] Figure 2 A flowchart of one embodiment of the deep learning-based dynamic gesture recognition method according to this application is shown;

[0060] Figure 3 A flowchart of another embodiment of the deep learning-based dynamic gesture recognition method according to this application is shown;

[0061] Figure 4 A schematic diagram of channel-wise convolution in the deep learning-based dynamic gesture recognition method according to this application is shown;

[0062] Figure 5 A schematic diagram of pointwise convolution in the deep learning-based dynamic gesture recognition method according to this application is shown;

[0063] Figure 6 A schematic diagram of the structure of one embodiment of the deep learning-based dynamic gesture recognition device according to this application is shown;

[0064] Figure 7 A schematic diagram of the structure of one embodiment of a computer device according to this application is shown. Detailed Implementation

[0065] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains; the terminology used herein in the specification of the application is for the purpose of describing particular embodiments only and is not intended to be limiting of the application; the terms "comprising" and "having," and any variations thereof, in the specification, claims, and foregoing drawings of this application, are intended to cover non-exclusive inclusion. The terms "first," "second," etc., in the specification, claims, or foregoing drawings of this application are used to distinguish different objects, not to describe a particular order.

[0066] In this document, the term "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0067] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings.

[0068] like Figure 1 As shown, system architecture 100 may include terminal devices 101, 102, and 103, a network 104, and a server 105. Network 104 serves as the medium for providing communication links between terminal devices 101, 102, and 103 and server 105. Network 104 may include various connection types, such as wired or wireless communication links, or fiber optic cables, etc.

[0069] Users can use terminal devices 101, 102, and 103 to interact with server 105 via network 104 to receive or send messages, etc. Various communication client applications can be installed on terminal devices 101, 102, and 103, such as web browser applications, shopping applications, search applications, instant messaging tools, email clients, social media platform software, etc.

[0070] Terminal devices 101, 102, and 103 can be various electronic devices with displays and support web browsing, including but not limited to smartphones, tablets, e-book readers, MP3 players (Moving Picture Experts Group Audio Layer III), MP4 players (Moving Picture Experts Group Audio Layer IV), laptops, and desktop computers, etc.

[0071] Server 105 can be a server that provides various services, such as a backend server that supports the pages displayed on terminal devices 101, 102, and 103. The server can be a standalone server or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDN), and big data and artificial intelligence platforms.

[0072] It should be noted that the deep learning-based dynamic gesture recognition method provided in this application embodiment is generally executed by a server, and correspondingly, the deep learning-based dynamic gesture recognition device is generally located in the server.

[0073] It should be understood that Figure 1 The number of terminal devices, networks, and servers shown is merely illustrative. Depending on implementation needs, any number of terminal devices, networks, and servers can be included.

[0074] Continue to refer to Figure 2 The diagram illustrates a flowchart of an embodiment of the deep learning-based dynamic gesture recognition method according to this application. This embodiment can acquire and process relevant data based on artificial intelligence technology. Artificial intelligence (AI) is a theory, method, technology, and application system that uses digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to obtain optimal results.

[0075] Foundational technologies for artificial intelligence generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, operating / interactive systems, and mechatronics. AI software technologies mainly encompass computer vision, robotics, biometrics, speech processing, natural language processing, and machine learning / deep learning.

[0076] Currently, for dynamic gesture recognition, deep learning-based gesture recognition methods typically employ a combination of ResNet34, LSTM, and Mobilnet, resulting in a massive and complex network model. Alternatively, they may use single or lightweight networks such as C3D and Mobilnet, but these lightweight networks achieve lower recognition accuracy, typically around 80%. Furthermore, current gesture recognition schemes primarily focus on the recognition process itself, paying less attention to the beginning and end phases of the gesture, leading to high computational load and impacting recognition efficiency.

[0077] To address the aforementioned issues, this application discloses a deep learning-based dynamic gesture recognition method and related equipment. This application improves upon the comprehensive network model in traditional deep learning gesture recognition schemes by introducing a channel-separable C3D network and a MobileNetV2 network, significantly reducing the number of network model parameters, decreasing the network model size, and lowering the modeling difficulty. The introduction of a bidirectional Bi-LSTM network enhances the ability to extract information from dynamic gestures along the timeline, thereby better extracting the feature information of dynamic gestures. Furthermore, before gesture recognition, this application uses a pre-trained hand detection model to detect hand joints in the input video stream to identify gesture images in the video stream, reducing the data processing load of the subsequent gesture recognition model and improving the efficiency and accuracy of gesture recognition.

[0078] Please refer to Figure 2 and Figure 3 The deep learning-based dynamic gesture recognition method includes the following steps:

[0079] S201 responds to gesture recognition commands and acquires the video stream uploaded by the client in real time.

[0080] In this embodiment, after receiving the gesture recognition command uploaded by the client, the server responds to the gesture recognition command and acquires the video stream uploaded by the client in real time. The client is connected to a camera device. After the server responds to the gesture recognition command, the client acquires the video stream of the scene where the camera device is located in real time through the camera device and uploads the video stream, which carries a dynamic gesture feature sequence.

[0081] In one specific embodiment of this application, the camera device may be a digital camera with autofocus, auto exposure, and program-controlled triggering, and the server may be any running module with GPU capabilities.

[0082] In this embodiment, the deep learning-based dynamic gesture recognition method operates on an electronic device (e.g., Figure 1 The server shown can receive gesture recognition commands uploaded by the client via wired or wireless connection. It should be noted that the aforementioned wireless connection methods may include, but are not limited to, 3G / 4G connections, Wi-Fi connections, Bluetooth connections, WiMAX connections, Zigbee connections, UWB (ultra-wireless) connections, and other currently known or future wireless connection methods.

[0083] S202, the video stream is imported into a pre-trained hand detection model, and the hand detection model extracts the gesture image sequence from the video stream.

[0084] The gesture recognition model is trained on the YoLo OV5 network. The YoLo series of network models are classic models used in object detection research, such as for liveness detection, limb detection, and gesture detection. The YoLo series uses the classic one-stage object detection algorithm and is currently the most widely used object detection network across various fields, with five relatively mature versions available. The YoLo OV5 network model is the latest version in the YoLo series. While inheriting the advantages of previous YoLo models, it boasts superior detection accuracy and faster inference speed. Furthermore, YoLo OV5 can achieve a detection frame rate of 50-60fps, which is sufficient to meet real-time requirements.

[0085] In this embodiment, a hand detection model is pre-trained based on the YOLOv5 network, and the hand detection model is used to extract gesture image sequences from the video stream.

[0086] In the above embodiments, this application trains a hand detection model using a Yolov5 network and uses a hand detection inference model to detect whether hand features appear in the video. When hand features are detected in the video, an image containing hand feature information is acquired, and one image containing hand feature information is acquired every other frame. The acquired images are stored in memory. When no hand features are detected in the video frame, image acquisition stops. This application uses a pre-trained hand detection model to recognize gesture images in the video stream, thereby reducing the data processing load of subsequent gesture recognition models and improving the efficiency and accuracy of gesture recognition.

[0087] S203, input the gesture image sequence into the pre-trained gesture recognition model in the form of tensors, and output the gesture recognition result. The gesture recognition model includes convolutional units, prediction units and integration units. The convolutional units are trained based on the C3D network with channel-separated convolution, the prediction units are trained based on the Bi-LSTM network, and the integration units are trained based on the MobilenetV2 network.

[0088] This application borrows the structures of C3D and MobileNetV2 networks, incorporating the channel separability concept from MobileNetV2 into the C3D network, thereby significantly reducing its size. The improved C3D network replaces the ResNet34 network in the combined network, and MobileNetV2 replaces the original MobileNet network. Furthermore, traditional deep learning gesture recognition combined networks use LSTMs with only single-item memory capabilities. To further enhance this memory ability, a bidirectional LSTM (Bi-LSTM) is developed. Through these optimizations, a joint gesture recognition deep recognition model combining a channel-separable C3D network, a bidirectional Bi-LSTM network, and a MobileNetV2 network is formed. This allows the network to maintain recognition accuracy while significantly reducing the number of network parameters, decreasing its size, and lowering the modeling difficulty.

[0089] In this embodiment, after extracting the gesture image sequence from the video stream, the gesture image sequence is input into the improved gesture recognition model in the form of tensors to recognize dynamic gesture actions in the video stream and obtain gesture recognition results. The gesture recognition model at this time consists of a channel-separable C3D network, a bidirectional Bi-LSTM network, and a mobile network.

[0090] S204: Perform channel-separated convolution operations on the gesture image sequence using convolution units to obtain image convolution sequence features.

[0091] This application borrows the structures of C3D and MobileNetV2 networks, incorporating the channel separability concept from MobileNetV2 into the C3D network. By training convolutional units through an improved C3D network, the number of network model parameters is reduced, thus decreasing the network model size. Channel separable convolution is performed on gesture image sequences using convolutional units. Channel separable convolution (also called depthwise separable convolution) consists of two parts: channel-wise convolution and pointwise convolution. See [link to relevant documentation]. Figure 4 and Figure 5Among them, the channel-wise convolution kernel has only one layer. One channel of the input feature is multiplied by one convolution kernel to obtain one channel of the output feature. The pointwise convolution is a 1x1 convolution, which can bring information exchange in the channel direction and change the number of channels of the output feature.

[0092] In this embodiment, the server performs channel-separated convolution operations on the gesture image sequence using convolution units to obtain image convolution sequence features.

[0093] S205, the gesture features of the preceding and following images of each frame in the gesture image sequence are predicted by the prediction unit to obtain the inter-frame gesture action features.

[0094] In this embodiment, a bidirectional Bi-LSTM network is introduced to train the prediction unit. The prediction unit predicts the gesture features of the preceding and following images in each frame of the gesture image sequence, thereby obtaining the inter-frame gesture action features. This application improves the information extraction capability of dynamic gestures on the time axis by introducing a bidirectional Bi-LSTM network, thus better extracting the feature information of dynamic gestures.

[0095] S206 integrates the inter-frame gesture motion features through the integration unit and outputs the gesture recognition result.

[0096] In this embodiment, a MobileNetV2 network is introduced to train an ensemble unit. This ensemble unit integrates and classifies inter-frame gesture motion features to obtain dynamic gesture recognition results in the video stream. This application further reduces the number of network model parameters and decreases the network model size by introducing the MobileNetV2 network. The MobileNetV2 network introduces a linear bottleneck and inverse residuals to improve the network's representational ability, making it a lightweight convolutional neural network.

[0097] In the above embodiments, this application improves the comprehensive network model in the traditional deep learning gesture recognition scheme by introducing a channel-separable C3D network and a mobile network V2 network, which greatly reduces the number of network model parameters, reduces the size of the network model, and reduces the modeling difficulty. The introduction of a bidirectional Bi-LSTM network improves the ability to extract information of dynamic gestures before and after the gestures on the time axis, thereby better extracting the feature information of dynamic gestures. At the same time, before performing gesture recognition, the scheme of this application performs hand joint detection on the input video stream through a pre-trained hand detection model to identify gesture images in the video stream, thereby reducing the data processing volume of the subsequent gesture recognition model and improving the efficiency and accuracy of gesture recognition.

[0098] Furthermore, the pre-trained hand detection model is trained based on a YOLOv5 network. The video stream is imported into the pre-trained hand detection model, which extracts gesture image sequences from the video stream, specifically including:

[0099] parse the video stream and obtain keyframe images from the video stream;

[0100] Import the keyframe images into a pre-trained hand detection model to identify hand feature images in the keyframe images;

[0101] Combine hand feature images to construct a sequence of gesture images.

[0102] In the above embodiments, the pre-trained hand detection model is trained based on the YOLOv5 network, which is currently the most widely used object detection network in various fields, with five relatively mature versions available. The server parses the video stream to obtain keyframe images, imports these keyframe images into the pre-trained hand detection model, and uses the model to detect whether the keyframe images contain hand joint node features. Keyframe images containing hand joint node features are identified as hand feature images. All hand feature images in the keyframe images are identified using the above recognition method, and a predetermined number of hand feature images are selected and combined to construct a gesture image sequence.

[0103] Furthermore, hand feature images are combined to construct a gesture image sequence, specifically including:

[0104] Identify the hand feature images in the keyframe images and store the hand feature images in memory;

[0105] When the number of identified hand feature images meets the preset first numerical threshold N1, the currently stored hand feature images are cleared, where the first numerical threshold N1 is a positive integer;

[0106] Continue to identify the remaining hand feature images until all hand feature images in the keyframe images have been identified;

[0107] Combine the hand feature images in the current memory to construct a sequence of gesture images.

[0108] In this embodiment, the server stores each hand feature image it recognizes in memory. The server also checks in real time whether the number of images in memory meets the preset first numerical threshold N1. If the number of images in memory reaches the first numerical threshold N1 and there are still recognized hand feature images that need to be stored, the server clears the currently stored hand feature images and stores the subsequently recognized hand feature images in memory until all hand feature images in the keyframe image are recognized. The hand feature images in memory are then combined to construct a gesture image sequence.

[0109] In one specific embodiment of this application, a hand may be present in the video stream without any corresponding gestures, resulting in the collection of many useless hand feature images and causing a memory overflow problem. To address this issue, an upper limit for image storage in memory is set. For example, a first numerical threshold N1 is set to 108 images. If the number of images stored in memory reaches 108 and there are other images waiting to be stored, the images currently stored in memory are cleared. This operation is repeated until all hand feature images in the keyframe images are recognized. Image recognition is then stopped, image acquisition is complete, and a gesture image sequence is constructed using the hand feature images currently in memory.

[0110] Furthermore, the hand feature images currently in memory are combined to construct a gesture image sequence, specifically including:

[0111] Calculate the number of hand feature images in the current memory to obtain the number of hand images;

[0112] The number of hand images is compared with the preset second numerical threshold N2 and the preset third numerical threshold N3, respectively, where the second numerical threshold N2 and the third numerical threshold N3 are both positive integers;

[0113] When the number of hand images equals the second numerical threshold N2, combine the images in the current memory to construct a gesture image sequence;

[0114] When the number of hand images is less than the second numerical threshold N2, a preset blank image is added to the memory until the number of images in the current memory is equal to the second numerical threshold N2. The images in the current memory are then combined to construct a gesture image sequence.

[0115] When the number of hand images is greater than the second numerical threshold N2 and the number of hand images is less than or equal to the third numerical threshold N3, the images in the current memory are filtered using the preset first filtering rule, and the filtered images are combined to construct a gesture image sequence.

[0116] When the number of hand images exceeds the third numerical threshold N3, the images in the current memory are filtered using a preset second filtering rule, and the filtered images are combined to construct a gesture image sequence.

[0117] In this embodiment, after image acquisition is completed, the number of hand feature images in the current memory is counted to obtain the number of hand images. This number is then compared with preset second threshold N2 and preset third threshold N3. When the number of hand images equals the second threshold N2, the images in the current memory are combined to construct a gesture image sequence. When the number of hand images is less than the second threshold N2, preset blank images are added to the memory until the number of images in the current memory equals the second threshold N2. The images in the current memory are then combined to construct a gesture image sequence. When the number of hand images is greater than the second threshold N2 and less than or equal to the third threshold N3, the images in the current memory are filtered using a preset first filtering rule, and the filtered images are combined to construct a gesture image sequence. When the number of hand images is greater than the third threshold N3, the images in the current memory are filtered using a preset second filtering rule, and the filtered images are combined to construct a gesture image sequence. The relationship between the first threshold N1, the second threshold N2, and the third threshold N3 is: N3 = 2N2, N1 = 3N2.

[0118] In one specific embodiment of this application, the first numerical threshold N1 is 108 images, the second numerical threshold N2 is 36 images, and the third numerical threshold N3 is 72 images. When the number of images in memory is 36, the images in memory are directly combined to construct a gesture image sequence. When the number of images in memory is less than 36, a preset blank image is added to memory so that the number of images in memory reaches 36, and the images in memory are combined to construct a gesture image sequence.

[0119] Furthermore, the images in the current memory are filtered using a preset first filtering rule, and the filtered images are combined to construct a gesture image sequence, specifically including:

[0120] Obtain the filtering frequency corresponding to the first filtering rule to get the first filtering frequency;

[0121] The images in the current memory are filtered at a first filtering frequency to obtain the first filtered images, and the number of the first filtered images is counted.

[0122] Compare the number of the first filtered images with the second numerical threshold N2;

[0123] When the number of images selected in the first screening is equal to the second numerical threshold N2, the images in the current memory are combined to construct a gesture image sequence.

[0124] When the number of images selected in the first screening is less than the second numerical threshold N2, blank images are added to memory until the number of images in memory is equal to the second numerical threshold N2. The images in memory are then combined to construct a gesture image sequence.

[0125] In the above embodiments, when the number of images in memory is greater than 36 and less than or equal to 72, the filtering frequency corresponding to the first filtering rule is obtained to obtain the first filtering frequency. The first filtering frequency is to select one hand feature image every frame and retain it. The images in the current memory are filtered using the first filtering frequency to obtain the first filtered images, and the number of the first filtered images is counted. When the number of the first filtered images is 36, the images in the current memory are combined to construct a gesture image sequence. When the number of the first filtered images is less than 36, a preset blank image is added to the memory so that the number of images in the memory reaches 36, and the images in the current memory are combined to construct a gesture image sequence.

[0126] Furthermore, the images in the current memory are filtered using a preset second filtering rule, and the filtered images are combined to construct a gesture image sequence, specifically including:

[0127] Obtain the filtering frequency corresponding to the second filtering rule to get the second filtering frequency;

[0128] The images in the current memory are filtered using a second filtering frequency to obtain second-filtered images, and the number of second-filtered images is counted.

[0129] Compare the number of second-selected images with the second numerical threshold N2;

[0130] When the number of images selected in the second screening is equal to the second numerical threshold N2, the images in the current memory are combined to construct a gesture image sequence.

[0131] When the number of images in the second selection is less than the second numerical threshold N2, blank images are added to memory until the number of images in memory is equal to the second numerical threshold N2. The images in memory are then combined to construct a gesture image sequence.

[0132] In the above embodiment, when the number of images in memory is greater than 72, the filtering frequency corresponding to the second filtering rule is obtained to obtain the second filtering frequency. The second filtering frequency is to select one hand feature image every 2 frames and retain it. The images in the current memory are filtered using the second filtering frequency to obtain the second filtered images. The number of the second filtered images is counted. When the number of the second filtered images is 36, the images in the current memory are combined to construct a gesture image sequence. When the number of the second filtered images is less than 36, a preset blank image is added to the memory so that the number of images in the memory reaches 36. The images in the current memory are then combined to construct a gesture image sequence.

[0133] In the above embodiments, before performing gesture recognition, the solution of this application performs hand joint detection on the input video stream using a pre-trained hand detection model to identify gesture images in the video stream, and filters and combines the gesture images to construct a gesture image sequence. By importing the gesture image sequence into the improved gesture recognition model, dynamic gestures in the video stream are identified, reducing the data processing volume of the gesture recognition model and improving the efficiency and accuracy of gesture recognition.

[0134] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by instructing related hardware through computer-readable instructions. These computer-readable instructions can be stored in a computer-readable storage medium. When executed, the computer-readable instructions can include the processes of the embodiments of the above methods. The aforementioned storage medium can be a non-volatile storage medium such as a magnetic disk, optical disk, or read-only memory (ROM), or random access memory (RAM).

[0135] It should be understood that although the steps in the flowcharts of the accompanying figures are shown sequentially as indicated by the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the accompanying figures may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times, and their execution order is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the sub-steps or stages of other steps.

[0136] Further reference Figure 6 As a response to the above Figure 2 The implementation of the method shown in this application provides an embodiment of a deep learning-based dynamic gesture recognition device, which is similar to... Figure 2 Corresponding to the method embodiments shown, this device can be specifically applied to various electronic devices.

[0137] like Figure 6 As shown, the deep learning-based dynamic gesture recognition device 600 described in this embodiment includes:

[0138] The video upload module 601 is used to respond to gesture recognition commands and acquire the video stream uploaded by the client in real time;

[0139] The image recognition module 602 is used to import the video stream into a pre-trained hand detection model and extract the gesture image sequence from the video stream through the hand detection model.

[0140] The gesture recognition module 603 is used to input the gesture image sequence into a pre-trained gesture recognition model in the form of a tensor and output the gesture recognition result. The gesture recognition model includes a convolutional unit, a prediction unit, and an integration unit. The convolutional unit is trained based on a channel-separated convolutional C3D network, the prediction unit is trained based on a Bi-LSTM network, and the integration unit is trained based on a Mob il enetV2 network.

[0141] The convolution operation module 604 is used to perform channel-separated convolution operations on the gesture image sequence through convolution units to obtain image convolution sequence features;

[0142] The gesture prediction module 605 is used to predict the gesture features of the adjacent images before and after each frame in the gesture image sequence through the prediction unit, so as to obtain the inter-frame gesture action features.

[0143] The gesture integration module 606 is used to integrate the gesture action features between frames through the integration unit and output the gesture recognition result.

[0144] Furthermore, the pre-trained hand detection model is trained based on a YOLOv5 network, and the image recognition module 602 specifically includes:

[0145] The video stream parsing submodule is used to parse the video stream and obtain keyframe images from the video stream;

[0146] The hand detection submodule is used to import keyframe images into a pre-trained hand detection model to identify hand feature images in the keyframe images.

[0147] The sequence construction submodule is used to combine hand feature images to construct a sequence of gesture images.

[0148] Furthermore, the sequence construction submodule specifically includes:

[0149] An image storage unit is used to identify hand feature images from keyframe images and store these hand feature images in memory.

[0150] The image clearing unit is used to clear the currently stored hand feature images when the number of recognized hand feature images meets a preset first numerical threshold N1, wherein the first numerical threshold N1 is a positive integer;

[0151] The continuous recognition unit is used to continuously recognize the remaining hand feature images until all hand feature images in the keyframe image have been recognized.

[0152] Sequence building units are used to combine hand feature images in the current memory to construct a sequence of gesture images.

[0153] Furthermore, the sequence building units specifically include:

[0154] The quantity statistics subunit is used to calculate the number of hand feature images in the current memory to obtain the number of hand images;

[0155] The quantity comparison subunit is used to compare the number of hand images with a preset second numerical threshold N2 and a preset third numerical threshold N3, respectively, wherein the second numerical threshold N2 and the third numerical threshold N3 are both positive integers.

[0156] The first result subunit is used to combine the images in the current memory to construct a gesture image sequence when the number of hand images is equal to the second numerical threshold N2.

[0157] The second result subunit is used to add a preset blank image to memory when the number of hand images is less than the second numerical threshold N2, until the number of images in the current memory is equal to the second numerical threshold N2, and combine the images in the current memory to construct a gesture image sequence.

[0158] The third result subunit is used to filter the images in the current memory using a preset first filtering rule when the number of hand images is greater than the second numerical threshold N2 and the number of hand images is less than or equal to the third numerical threshold N3, and to combine the filtered images to construct a gesture image sequence.

[0159] The fourth result subunit is used to filter the images in the current memory using a preset second filtering rule when the number of hand images is greater than the third numerical threshold N3, and to combine the filtered images to construct a gesture image sequence.

[0160] Furthermore, the relationship between the first numerical threshold N1, the second numerical threshold N2, and the third numerical threshold N3 is: N3 = 2N2, N1 = 3N2.

[0161] Furthermore, the images in the current memory are filtered using a preset first filtering rule, and the filtered images are combined to construct a gesture image sequence, specifically including:

[0162] Obtain the filtering frequency corresponding to the first filtering rule to get the first filtering frequency;

[0163] The images in the current memory are filtered at a first filtering frequency to obtain the first filtered images, and the number of the first filtered images is counted.

[0164] Compare the number of the first filtered images with the second numerical threshold N2;

[0165] When the number of images selected in the first screening is equal to the second numerical threshold N2, the images in the current memory are combined to construct a gesture image sequence.

[0166] When the number of images selected in the first screening is less than the second numerical threshold N2, blank images are added to memory until the number of images in memory is equal to the second numerical threshold N2. The images in memory are then combined to construct a gesture image sequence.

[0167] Furthermore, the images in the current memory are filtered using a preset second filtering rule, and the filtered images are combined to construct a gesture image sequence, specifically including:

[0168] Obtain the filtering frequency corresponding to the second filtering rule to get the second filtering frequency;

[0169] The images in the current memory are filtered using a second filtering frequency to obtain second-filtered images, and the number of second-filtered images is counted.

[0170] Compare the number of second-selected images with the second numerical threshold N2;

[0171] When the number of images selected in the second screening is equal to the second numerical threshold N2, the images in the current memory are combined to construct a gesture image sequence.

[0172] When the number of images in the second selection is less than the second numerical threshold N2, blank images are added to memory until the number of images in memory is equal to the second numerical threshold N2. The images in memory are then combined to construct a gesture image sequence.

[0173] This application discloses a deep learning-based dynamic gesture recognition device, belonging to the field of action recognition technology. The solution improves upon the comprehensive network model in traditional deep learning gesture recognition schemes by introducing a channel-separable C3D network and a MobileNetV2 network, significantly reducing the number of network model parameters, decreasing the network model size, and lowering modeling difficulty. The introduction of a bidirectional Bi-LSTM network enhances the ability to extract information from dynamic gestures over time, thereby better extracting the feature information of dynamic gestures. Furthermore, before gesture recognition, this application uses a pre-trained hand detection model to detect hand joints in the input video stream to identify gesture images in the video stream, reducing the data processing load of the subsequent gesture recognition model and improving the efficiency and accuracy of gesture recognition.

[0174] To address the aforementioned technical problems, embodiments of this application also provide a computer device. Please refer to [link / reference needed]. Figure 7 , Figure 7 This is a basic structural block diagram of the computer device in this embodiment.

[0175] The computer device 7 includes a memory 71, a processor 72, and a network interface 73 that are interconnected via a system bus. It should be noted that only the computer device 7 with components 71-73 is shown in the figure; however, it should be understood that it is not required to implement all the shown components, and more or fewer components can be implemented alternatively. Those skilled in the art will understand that the computer device described here is a device capable of automatically performing numerical calculations and / or information processing according to pre-set or stored instructions. Its hardware includes, but is not limited to, microprocessors, application-specific integrated circuits (ASICs), programmable gate arrays (FPGAs), digital signal processors (DSPs), embedded devices, etc.

[0176] The computer device can be a desktop computer, laptop, handheld computer, or cloud server, etc. The computer device can interact with the user via a keyboard, mouse, remote control, touchpad, or voice control.

[0177] The memory 71 includes at least one type of readable storage medium, including flash memory, hard disk, multimedia card, card-type memory (e.g., SD or DX memory), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, magnetic disk, optical disk, etc. In some embodiments, the memory 71 may be an internal storage unit of the computer device 7, such as the hard disk or memory of the computer device 7. In other embodiments, the memory 71 may also be an external storage device of the computer device 7, such as a plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, etc., equipped on the computer device 7. Of course, the memory 71 may also include both the internal storage unit and its external storage device of the computer device 7. In this embodiment, the memory 71 is typically used to store the operating system and various application software installed on the computer device 7, such as computer-readable instructions based on a deep learning-based dynamic gesture recognition method. In addition, the memory 71 can also be used to temporarily store various types of data that have been output or will be output.

[0178] In some embodiments, the processor 72 may be a central processing unit (CPU), controller, microcontroller, microprocessor, or other data processing chip. The processor 72 is typically used to control the overall operation of the computer device 7. In this embodiment, the processor 72 is used to execute computer-readable instructions stored in the memory 71 or to process data, for example, to execute computer-readable instructions for the deep learning-based dynamic gesture recognition method.

[0179] The network interface 73 may include a wireless network interface or a wired network interface, which is typically used to establish communication connections between the computer device 7 and other electronic devices.

[0180] This application discloses a computer device belonging to the field of action recognition technology. The solution of this application improves the comprehensive network model in traditional deep learning gesture recognition schemes by introducing a channel-separable C3D network and a MobileNetV2 network, which greatly reduces the number of network model parameters, decreases the network model size, and reduces modeling difficulty. The introduction of a bidirectional Bi-LSTM network enhances the ability to extract information from dynamic gestures over time, thereby better extracting the feature information of dynamic gestures. Furthermore, before performing gesture recognition, this application uses a pre-trained hand detection model to detect hand joints in the input video stream to identify gesture images in the video stream, reducing the data processing load of the subsequent gesture recognition model and improving the efficiency and accuracy of gesture recognition.

[0181] This application also provides another embodiment, namely, providing a computer-readable storage medium storing computer-readable instructions that can be executed by at least one processor to cause the at least one processor to perform the steps of the deep learning-based dynamic gesture recognition method described above.

[0182] This application discloses a storage medium belonging to the field of action recognition technology. The solution of this application improves the comprehensive network model in traditional deep learning gesture recognition schemes by introducing a channel-separable C3D network and a MOBI LENETV2 network, which greatly reduces the number of network model parameters, decreases the network model size, and reduces modeling difficulty. The introduction of a bidirectional Bi-LSTM network enhances the ability to extract information from dynamic gestures over time, thereby better extracting the feature information of dynamic gestures. Furthermore, before performing gesture recognition, this application uses a pre-trained hand detection model to detect hand joints in the input video stream to identify gesture images in the video stream, reducing the data processing load of the subsequent gesture recognition model and improving the efficiency and accuracy of gesture recognition.

[0183] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0184] This application can be used in a wide variety of general-purpose or special-purpose computer system environments or configurations. Examples include: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, and distributed computing environments including any of the above systems or devices. This application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform specific tasks or implement specific abstract data types. This application can also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.

[0185] Obviously, the embodiments described above are only some embodiments of this application, not all embodiments. The accompanying drawings show preferred embodiments of this application, but do not limit the patent scope of this application. This application can be implemented in many different forms; rather, the purpose of providing these embodiments is to provide a more thorough and comprehensive understanding of the disclosure of this application. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing specific embodiments, or make equivalent substitutions for some of the technical features. Any equivalent structures made using the content of this application's specification and drawings, directly or indirectly applied to other related technical fields, are similarly within the scope of patent protection of this application.

Claims

1. A dynamic gesture recognition method based on deep learning, characterized in that, The method comprises the steps of: in response to a gesture recognition instruction, acquiring a video stream uploaded by a client in real time; introducing the video stream into a pre-trained hand detection model, and extracting a gesture image sequence from the video stream through the hand detection model; inputting the gesture image sequence in the form of a tensor into a pre-trained gesture recognition model, and outputting a gesture recognition result, wherein the gesture recognition model comprises a convolution unit, a prediction unit and an integration unit, the convolution unit is trained based on a C3D network of channel separation convolution, the prediction unit is trained based on a Bi-LSTM network, and the integration unit is trained based on a mobilenetV2 network; performing channel separation convolution operation on the gesture image sequence through the convolution unit to obtain image convolution sequence features; predicting gesture features of adjacent images of each frame of image in the gesture image sequence through the prediction unit to obtain inter-frame gesture action features; integrating the inter-frame gesture action features through the integration unit to output the gesture recognition result; the pre-trained hand detection model is trained based on a yolov5 network, and the introduction of the video stream into the pre-trained hand detection model and the extraction of the gesture image sequence from the video stream through the hand detection model specifically comprise the steps of: analyzing the video stream to obtain key frame images in the video stream; introducing the key frame images into the pre-trained hand detection model to identify hand feature images in the key frame images; combining the hand feature images to construct the gesture image sequence.

2. A dynamic gesture recognition apparatus based on deep learning, characterized by, The method comprises the steps of: a video uploading module configured to acquire a video stream uploaded by a client in real time in response to a gesture recognition instruction; an image recognition module configured to introduce the video stream into a pre-trained hand detection model, and extract a gesture image sequence from the video stream through the hand detection model; a gesture recognition module configured to input the gesture image sequence in the form of a tensor into a pre-trained gesture recognition model, and output a gesture recognition result, wherein the gesture recognition model comprises a convolution unit, a prediction unit and an integration unit, the convolution unit is configured to perform channel separation convolution operation on the gesture image sequence to obtain image convolution sequence features, the prediction unit is configured to predict gesture features of adjacent images of each frame of image in the gesture image sequence to obtain inter-frame gesture action features, and the integration unit is configured to integrate the inter-frame gesture action features to output the gesture recognition result, the convolution unit is trained based on a C3D network of channel separation convolution, the prediction unit is trained based on a Bi-LSTM network, and the integration unit is trained based on a mobilenetV2 network; the pre-trained hand detection model is trained based on a yolov5 network, and the image recognition module specifically comprises: a video stream analysis submodule configured to analyze the video stream to obtain key frame images in the video stream; a hand detection submodule, configured to input the key frame image into a pre-trained hand detection model to identify a hand feature image in the key frame image; a sequence construction submodule, configured to combine the hand feature image to construct the hand gesture image sequence.

3. A computer device, comprising: A computer readable storage medium, having computer readable instructions stored therein, wherein the computer readable instructions, when executed by a processor, implement the steps of the dynamic hand gesture recognition method based on deep learning according to any one of claims 1-5.

4. A computer-readable storage medium, characterized in that, A computer readable storage medium, having computer readable instructions stored therein, wherein the computer readable instructions, when executed by a processor, implement the steps of the dynamic hand gesture recognition method based on deep learning according to any one of claims 1-5.

Images