Gesture recognition method and device, medium, equipment and vehicle
By constructing a dynamic gesture recognition network and combining depthwise separable convolution and long short-term memory modules, the network structure and parameters are optimized, solving the problems of high computational load and slow speed in dynamic gesture recognition technology, and achieving efficient and accurate gesture recognition.
Patent Information
- Application Number
- CN202410951386.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-16
- Publication Date
- 2024-11-22
AI Technical Summary
Existing dynamic gesture recognition technologies involve large computational workloads and are slow, making them difficult to use in real-time recognition systems.
A dynamic gesture recognition network is constructed, including a depthwise separable convolutional module, a feature pyramid module, and a long short-term memory module. The network structure and parameters are optimized by training with a variety of sample sets, initializing with pre-trained weights, and performing model compression and iterative optimization.
It improves the accuracy and efficiency of gesture recognition, is suitable for real-time recognition systems, and adapts to the needs of different scenarios and users.
Smart Images

Figure CN119007281A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of data processing, and in particular to a gesture recognition method, device, medium, equipment and vehicle. Background Art
[0002] In contemporary society, human-computer interaction has gradually become an indispensable part of our work and life. With the continuous development of artificial intelligence, especially deep learning, human-computer interaction has become more and more intelligent and efficient. In recent years, with the cooperation and research between scholars in different fields, various human-computer interaction methods have been developed to facilitate our life and work. As an important body language, gestures are an indispensable part of our lives. For example, traffic police on the road use gestures to direct traffic, and deaf-mute people use sign language to communicate with others. Therefore, gesture interaction has also received more and more attention from academia and industry, and has become one of the important interaction methods in the field of human-computer interaction.
[0003] In recent years, Convolutional Neural Network (CNN) has made great progress in static image classification. People naturally want to extend this progress to video and perform dynamic gesture recognition. However, dynamic gesture recognition technology requires tracking and recognizing gestures, which has a large computational workload and a slow speed, making it difficult to use in real-time recognition systems. Summary of the invention
[0004] The present application provides a gesture recognition method, apparatus, medium, equipment and vehicle, which can improve the accuracy and efficiency of gesture recognition.
[0005] A first aspect of an embodiment of the present application provides a gesture recognition method, the method comprising:
[0006] Acquire a data set of dynamic gesture videos, where the data set is shot or downloaded for different application scenarios;
[0007] Annotate each frame of the image in the data set, where the annotated information includes the frame number of the action from the start to the end and the category to which the action belongs, to obtain a sample set;
[0008] A dynamic gesture recognition network is constructed, wherein the dynamic gesture recognition network includes a depth-separable convolution module, a feature pyramid module and a long short-term memory module; wherein,
[0009] The depthwise separable convolution module is used to perform primary feature extraction on the image in the sample set to obtain a first feature image;
[0010] The feature pyramid module is used to downsample the first feature image to obtain feature maps of different scales, and fuse the feature maps through upsampling and lateral connection to obtain a fused feature map containing features of different depths;
[0011] The long short-term memory module is used to extract local features of the fused feature map and perform time-series processing to capture the time-series spatial features of the image, and map the processed feature map to a preset gesture category to output a dynamic gesture recognition result;
[0012] Introducing pre-trained weights as initialization weights of a dynamic gesture recognition network, inputting images in the sample set into the dynamic gesture recognition network for training, and outputting prediction results;
[0013] Continuously comparing the prediction results of the dynamic gesture recognition network with the corresponding labeled information in the sample set, calculating the loss value of the network, and feeding the loss value back to the dynamic gesture recognition network, continuously correcting the parameters of the network, and obtaining a trained dynamic gesture recognition network;
[0014] The trained dynamic gesture recognition network is used to perform gesture recognition on dynamic gesture videos.
[0015] Optionally, any of the depth-wise separable convolutions in the depth-wise separable convolution module comprises: an input layer, a depth-wise convolution layer, a point-wise convolution layer and an output layer; wherein,
[0016] The input of the input layer is a three-channel image;
[0017] The deep convolution layer is used to perform channel-by-channel convolution operation on the input three-channel image, and use three convolution kernels to convolve the three channels respectively to generate three channel feature maps;
[0018] The point-by-point convolution layer is used to use a three-dimensional convolution kernel to perform linear combination and dimensionality reduction on the three channel feature maps, synthesize the three channel feature maps into one feature map, and output it through the output layer.
[0019] Optionally, the feature pyramid module includes an input layer, a plurality of symmetrical downsampling layers and a plurality of upsampling layers, the upsampling layers and the downsampling layers are horizontally connected, and the sizes of feature maps output by the horizontally connected upsampling layers and the downsampling layers are the same; wherein:
[0020] After the input layer of the feature pyramid module obtains the first feature map, the first feature map is continuously downsampled through multiple downsampling layers, and scene context information with a global structure is extracted layer by layer to obtain a second feature map, wherein the size of the feature map output by each downsampling layer is halved successively;
[0021] The features extracted by each downsampling layer are fused into the corresponding upsampling layer through lateral connection, and the second feature map is continuously upsampled through multiple upsampling layers, and the size of the second feature map is restored layer by layer to the original size corresponding to the first feature map, so as to obtain the fused feature map.
[0022] Optionally, the method further comprises:
[0023] Adjusting any image in the sample set to a preset size to obtain a first intermediate image;
[0024] Performing grayscale processing and standardization processing on the first intermediate image to obtain a second intermediate image;
[0025] Performing random rotation, translation, scaling and mirroring processing on the second intermediate image to obtain a preprocessed image;
[0026] Divide the preprocessed images into a data set to obtain a training set, a validation set, and a test set;
[0027] Using the training set to train the dynamic gesture recognition network, adjusting the network weights and biases, and reducing the network loss value;
[0028] Using the validation set to cross-validate the dynamic gesture recognition network, adjust the network's hyperparameters, and optimize network performance;
[0029] The performance of the dynamic gesture recognition network is evaluated using the test set to obtain the trained dynamic gesture recognition network.
[0030] Optionally, the trained dynamic gesture recognition network is used to perform gesture recognition on the dynamic gesture video, including:
[0031] Deploying the trained dynamic gesture recognition network into an intelligent driving system to perform gesture recognition on gesture data generated during driving;
[0032] Obtaining user feedback, and performing an application evaluation on the dynamic gesture recognition network based on the user feedback and gesture data generated during driving to obtain an application evaluation result;
[0033] Based on the applied evaluation result, the dynamic gesture recognition network is iteratively updated and optimized.
[0034] Optionally, before deploying the trained dynamic gesture recognition network to the intelligent driving system, the method further includes:
[0035] The dynamic gesture recognition network is subjected to model pruning processing, redundant neurons and connections in the network are removed, and the network is compressed.
[0036] Based on the same inventive concept, a second aspect of an embodiment of the present application provides a gesture recognition device, the device comprising:
[0037] A data set acquisition module is used to acquire a data set of dynamic gesture videos, wherein the data set is acquired by shooting or downloading according to different application scenarios;
[0038] A sample set generation module is used to annotate each frame of the image in the data set, wherein the annotated information includes the frame number of the action from the beginning to the end and the category to which the action belongs, thereby obtaining a sample set;
[0039] A network construction module is used to construct a dynamic gesture recognition network, wherein the dynamic gesture recognition network includes a depth-separable convolution module, a feature pyramid module and a long short-term memory module; wherein,
[0040] The depthwise separable convolution module is used to perform primary feature extraction on the image in the sample set to obtain a first feature image;
[0041] The feature pyramid module is used to downsample the first feature image to obtain feature maps of different scales, and fuse the feature maps through upsampling and lateral connection to obtain a fused feature map containing features of different depths;
[0042] The long short-term memory module is used to extract local features of the fused feature map and perform time-series processing to capture the time-series spatial features of the image, and map the processed feature map to a preset gesture category to output a dynamic gesture recognition result;
[0043] A training module, used for introducing pre-trained weights as initialization weights of a dynamic gesture recognition network, inputting images in the sample set into the dynamic gesture recognition network for training, and outputting prediction results;
[0044] An adjustment and optimization module is used to continuously compare the prediction results of the dynamic gesture recognition network with the corresponding labeled information in the sample set, calculate the loss value of the network, and feed the loss value back to the dynamic gesture recognition network, continuously modify the parameters of the network, and obtain a trained dynamic gesture recognition network;
[0045] The application module is used to use the trained dynamic gesture recognition network to perform gesture recognition on dynamic gesture videos.
[0046] Based on the same inventive concept, the third aspect of the embodiment of the present application provides a storage medium, in which machine executable instructions are stored. When the machine executable instructions are executed by a processor, the gesture recognition method proposed in the first aspect of the present application is implemented.
[0047] Based on the same inventive concept, the fourth aspect of an embodiment of the present application provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the gesture recognition method proposed in the first aspect of the present application when executed.
[0048] Based on the same inventive concept, the fifth aspect of an embodiment of the present application provides a vehicle, comprising a processor and a memory; the memory stores machine executable instructions that can be executed by the processor, and the processor is used to execute the machine executable instructions to implement the gesture recognition method proposed in the first aspect of the present application.
[0049] Compared with the prior art, this application has the following advantages:
[0050] A gesture recognition method provided in an embodiment of the present application first collects and annotates a variety of gesture image data from different sources and different scenes to form a sample set with rich and diverse types. Then, based on the actual application scenarios of gesture recognition, a suitable lightweight dynamic gesture recognition network is constructed, including a deep separable convolution module, a feature pyramid module and a long short-term memory module, and the dynamic gesture recognition network is trained using a diverse sample set to reduce the complexity of the network while maintaining the feature extraction capability through deep separable convolution, and to fuse the overall structural information and local detail information of the gesture through the feature pyramid module, and then learn the spatiotemporal characteristics of the gesture through the long short-term memory module, and then identify different gesture movements. Finally, the trained dynamic gesture recognition network is deployed in actual application scenarios to perform real-time gesture recognition and improve the accuracy and efficiency of gesture recognition. BRIEF DESCRIPTION OF THE DRAWINGS
[0051] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments of the present invention will be briefly introduced below. Obviously, the accompanying drawings in the following description are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative labor.
[0052] Figure 1 is a flow chart of a gesture recognition method in one embodiment of the present application;
[0053] Figure 2 This is a schematic diagram of the application of a gesture recognition method in the field of intelligent driving in one embodiment of the present application;
[0054] Figure 3 is a schematic diagram of functional modules of a gesture recognition device in one embodiment of the present application;
[0055] Figure 4 It is a structural schematic diagram of a vehicle in one embodiment of the present application. DETAILED DESCRIPTION
[0056] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.
[0057] Please refer to Figure 1 , Figure 1 is a flow chart of a gesture recognition method proposed in one embodiment of the present application. Figure 1 As shown, the method comprises the following steps:
[0058] S101: Acquire a dataset of dynamic gesture videos, where the dataset is shot or downloaded for different application scenarios.
[0059] In this embodiment, any dynamic gesture video includes the process from the beginning to the end of the gesture action. After obtaining the dynamic gesture video to be recognized, the continuous video data can be converted into continuous gesture image data by slicing the dynamic gesture video data. The gesture image data in the dynamic gesture video includes various types of gesture actions, gesture combinations and gesture sequences, and gesture information in different scenes can be obtained from different sources to ensure the diversity of gesture image data.
[0060] Specifically, gesture image data can be obtained through a camera, a sensor or an online data set. Gesture image data can come from one or more scenarios in human-computer interaction interface, virtual reality and augmented reality applications, somatosensory interaction devices, biometric authentication, gesture control devices, and education and training software. 1. Based on the human-computer interaction interface, on devices such as computers, mobile phones, and tablets, users can use gestures to perform various operations, such as sliding, pinching, zooming in and out, rotating, etc., to control applications or operation interfaces. For example, on a touch screen device, users can browse web pages, enlarge images, switch applications, etc. through gestures such as sliding, double-clicking, and pinching. 2. In virtual reality (VR) and augmented reality (AR) applications, users can use gestures to interact with virtual or augmented environments. For example, in a virtual reality game, players can use gestures to control the character's movement, attack, jump, and other actions. 3. Based on somatosensory interaction devices, such as Kinect and Leap Motion, users can interact through natural gestures. For example, in a somatosensory game, players can control the behavior of the game character through actions such as jumping and waving their arms. 4. In terms of biometric authentication, gestures can be used as a form of biometric authentication, such as gesture passwords, finger sliding, etc., to unlock devices or applications such as mobile phones and computers. For example, some mobile phone applications support unlocking through gesture patterns, and users can protect the privacy and security of their phones through customized gesture patterns. 5. Based on gesture-controlled devices, users can operate devices or control applications through gestures. For example, smart home systems may support gesture control of lights, curtains, temperature and other functions, and users can control these functions through specific gestures. 6. In terms of education and training, gestures can be used as an interactive learning method to help students understand and learn knowledge more intuitively. For example, teaching software may support operation and control through gestures to help students participate more actively in the learning process.
[0061] Furthermore, the gesture image data from different sources also include gesture images under different lighting, background interference and / or noise backgrounds to ensure that the trained network has good generalization ability. At the same time, the gesture image data also includes gesture images under different occlusion conditions, gesture speed changes and / or gesture size conditions to improve the network's ability to handle special situations and boundary conditions and ensure the stability and robustness of the network. In addition, the gesture image data also includes gesture images for different user groups, such as user groups with different ages, genders, cultural backgrounds, etc., to take into account the characteristics and habits of different users, so that the trained network can adapt to the needs of different user groups. In addition, the gesture image data also includes gesture images for different user needs. By considering user participation and feedback, the real needs and usage of users are understood, so as to continuously improve and optimize the network to make it more representative.
[0062] S102: Label each frame of the image in the data set, where the labeled information includes the frame number of the action from the start to the end and the category to which the action belongs, to obtain a sample set.
[0063] In this embodiment, when annotating each frame of gesture image, two labels need to be assigned to each gesture image: the first label is the label of the gesture action category displayed in the gesture image. For example, if the gesture image contains the gesture actions "fist" and "open palm", the corresponding gesture category label needs to be assigned to the gesture image. The second label is the frame number label of the gesture image in the dynamic gesture video, which is used to indicate the frame number of the gesture image in the dynamic gesture video, so as to be used for the long short-term memory module in the subsequent dynamic gesture recognition network to learn the spatiotemporal characteristics of the gesture.
[0064] In this embodiment, each frame of the image in the data set is labeled with a corresponding gesture category label and frame number label to form a diversified sample set for subsequent training of the dynamic gesture recognition network to improve the network training effect. Each dynamic gesture video is a sample.
[0065] S103: Construct a dynamic gesture recognition network, which includes a depthwise separable convolution module, a feature pyramid module, and a long short-term memory module.
[0066] In this embodiment, the construction of the dynamic gesture recognition network can be obtained based on the improvement and combination of the convolutional neural network and the long short-term memory network. Specifically, the process mainly includes:
[0067] S103-1: Select multiple target models from multiple models for performing gesture recognition tasks, where the multiple target models include at least a convolutional neural network and a long short-term memory network.
[0068] Convolutional neural networks (CNNs) play a vital role in gesture recognition. They provide strong support for the development and application of gesture recognition technology through their advantages in automatic feature extraction, classification and recognition, improving accuracy, and achieving real-time and robustness. Long short-term memory networks (LSTMs) are a special type of recurrent neural network (RNN) that is particularly good at processing data with time series characteristics. Through internal memory cells, forget gates, input gates, output gates and other mechanisms, they can effectively capture and store time series information in gestures, thereby better understanding and recognizing gestures.
[0069] In this embodiment, in order to improve the accuracy and efficiency of gesture recognition, the basic network model needs to be improved to facilitate application in actual scenarios (such as human-computer interaction, smart home control, smart driving, etc.).
[0070] S103-2: Optimize multiple target models respectively to obtain multiple optimized target models.
[0071] In this implementation, the target model is optimized mainly from the perspective of network structure and training methods. Specifically, the optimization of the network structure of the convolutional neural network mainly includes: replacing the convolution in the convolutional neural network with an optimized depth-separable convolution, and adding an attention calculation module based on the attention mechanism to the replaced convolutional neural network.
[0072] In this embodiment, in the convolutional neural network, the traditional convolution layer is replaced with a depth-separable convolution layer, which can reduce the number of parameters and the amount of calculation of the model while maintaining the recognition performance of the model. At the same time, the use of deep separable convolution in the primary feature extraction layer of the network can significantly reduce the complexity of the model while maintaining the feature extraction capability. In addition, by using deep separable convolution instead of traditional convolution operations, it is also helpful to achieve lightweight models to meet the deployment requirements of mobile devices or embedded systems.
[0073] Furthermore, residual connections and / or dilated convolutions are added between depthwise separable convolution layers. Considering that depthwise separable convolutions may also reduce the performance of the model on complex tasks, other optimization techniques such as residual connections and dilated convolutions can be combined to make up for the performance deficiencies of depthwise separable convolutions, thereby increasing the network depth or width while keeping the model lightweight to improve the representation capability of the model.
[0074] Furthermore, an attention calculation module based on the attention mechanism is added to the replaced convolutional neural network. The attention mechanism allows the model to focus on more important parts when processing information, such as the SE (Squeeze-and-Excitation) module, which learns to assign weights to each input feature or position and recalibrates the dependencies between channels to increase the model's attention to key information of gesture actions, further improving the performance of the model.
[0075] The optimization of the network structure of the long short-term memory network mainly includes: based on the optimized structure of the convolutional neural network, correspondingly adjusting the parameters and components of the long short-term memory network to ensure that the long short-term memory network can be connected to the convolutional neural network.
[0076] S103-3: Combine multiple optimized target models to obtain a dynamic gesture recognition network model to be trained, where the network model includes: a depthwise separable convolution module, a feature pyramid module, and a long short-term memory module.
[0077] In this embodiment, the depthwise separable convolution module is used to perform primary feature extraction on the images in the sample set to obtain a first feature image. Specifically, the depthwise separable convolution module uses the depthwise separable convolution as a basic component and gradually extracts low-level to high-level gesture features in the gesture image by stacking multiple convolution blocks.
[0078] Each convolution block includes: input layer, depth convolution layer, point-by-point convolution layer and output layer. The input layer is a three-channel image; the depth convolution layer is used to perform channel-by-channel convolution operation on the input three-channel image, and use three convolution kernels to convolve the three channels respectively to generate three channel feature maps; the point-by-point convolution layer is used to use the three-dimensional convolution kernel to linearly combine and reduce the dimension of the three channel feature maps, synthesize the three channel feature maps into one feature map, and output it through the output layer.
[0079] In this embodiment, the gesture image is an RGB three-channel color image. After it is input into the depth separable convolution block, the depth convolution layer applies a convolution kernel to each input channel, instead of applying multiple convolution kernels to each channel as in the traditional convolution. This significantly reduces the number of parameters of the convolution layer (because each channel corresponds to only one convolution kernel). At the same time, the depth convolution can still effectively extract spatial features on each channel to retain the local information in the input gesture image. Then, the point-by-point convolution layer uses a 1x1 convolution kernel to perform a linear combination of the three channel feature maps output by the depth convolution layer, that is, introduces cross-channel feature interaction, so that information from different channels can be fused with each other, and obtain the synthesized feature map output. In addition, an activation function (such as ReLU) can be connected after the point-by-point convolution layer, which helps to introduce nonlinearity and enhance the expression ability of the network.
[0080] By processing the gesture image through the depthwise separable convolution module, key features can be quickly and effectively extracted from the gesture image, including information such as the shape, contour, and texture of the gesture, and a first feature image can be obtained. Then, the first feature image is input into the feature pyramid module. The feature pyramid module is used to downsample the first feature image to obtain feature maps of different scales, and fuse the feature maps through upsampling and lateral connection to obtain a fused feature map containing features of different depths.
[0081] In this embodiment, the feature pyramid module is constructed by depthwise separable convolution, and by applying depthwise separable convolution at different levels, multi-scale feature extraction is achieved to improve the model's recognition ability for gesture images. Specifically, the feature pyramid module includes an input layer, multiple symmetrical downsampling layers and multiple upsampling layers, the upsampling layers and downsampling layers are horizontally connected, and the size of the feature maps output by the horizontally connected upsampling layers and downsampling layers is the same; wherein: after the input layer of the feature pyramid module obtains the first feature map, the first feature map is continuously downsampled through multiple downsampling layers, and the scene context information with a global structure is extracted layer by layer to obtain a second feature map, wherein the size of the feature map output by each downsampling layer is halved in turn; the features extracted by each downsampling layer are fused to the corresponding upsampling layer through a horizontal connection, and the second feature map is continuously upsampled through multiple upsampling layers, and the size of the second feature map is restored layer by layer to the original size corresponding to the first feature map, to obtain a fused feature map.
[0082] In a feasible implementation, the number of downsampling layers and upsampling layers can be set to 4 to meet the requirements of lightweight model. Specifically, after the input layer of the feature pyramid module obtains the first feature map, the first feature map is continuously downsampled, and the scene context information with global structure is extracted layer by layer to obtain the feature set Ci = {C1, C2, C3, C4}, wherein the sizes of the feature maps C1, C2, C3, and C4 are halved successively; the feature map C4 is continuously upsampled, and the scale of the feature map C4 is restored layer by layer to the original size corresponding to the first feature map, and the feature set Pi = {P1, P2, P3, P4} is obtained, wherein P1 is the feature map obtained by upsampling the feature map C4, P2 is the feature map obtained by upsampling the feature map after superimposing the feature maps P1 and C3, P3 is the feature map obtained by upsampling the feature map after superimposing the feature maps P2 and C2, and P4 is the feature map obtained by upsampling the feature map after superimposing the feature map P3 and C1; the feature map P4 is superimposed with the first feature map to obtain a fused feature map.
[0083] In this embodiment, the first feature image is further processed through a feature pyramid structure of continuous downsampling, upsampling and lateral feature connection, which can cover multi-scale targets in the image and extract gesture feature information at different scales, including the overall structural information and local detail information of the gesture action. By fusing gesture feature information at different scales to generate a more representative feature representation, it is better to deal with challenging factors such as occlusion and deformation in gesture images, which helps to improve the accuracy and robustness of gesture recognition.
[0084] In addition, multi-scale feature extraction and fusion of gesture actions can also be performed through structures such as Gaussian pyramids and Laplacian pyramids, and this application does not make specific restrictions on this.
[0085] After the fused feature map is obtained through feature pyramid processing, it is input into the long short-term memory module. The long short-term memory module is used to extract local features of the fused feature map and perform temporal processing to capture the temporal spatial features of the image, and map the processed feature map to the preset gesture category, and output the dynamic gesture recognition result.
[0086] In this embodiment, by introducing the long short-term memory structure, the dynamic change process of the gesture can be captured using the time dimension information, so as to learn the spatiotemporal characteristics of the gesture. Then, the processed feature map is mapped to the preset gesture category, and the gesture recognition result is output. This result can be a label of a gesture category. For example, by outputting the similarity of gesture actions in each gesture image belonging to each preset category of gesture actions, and screening out the corresponding gesture actions whose similarity meets the threshold as the gesture recognition result. In addition, this result can also be a bounding box containing information such as the position and size of the gesture. Through visualization processing, the user can intuitively see the result of gesture recognition.
[0087] This embodiment combines the optimized convolutional neural network and the long short-term memory network. First, the morphology and structure of gesture actions in gesture images are captured through the depth-separable convolutional blocks and feature pyramids in the optimized convolutional neural network, and the characteristics of complex gestures are effectively described. Then, the long-term dependencies in dynamic gesture videos (gesture image sequences) are captured through the long short-term memory network, and the dynamic changes and contextual information of gestures are learned, thereby identifying the continuity and change trend of gestures. Therefore, the combined dynamic gesture recognition network simultaneously utilizes the spatial features and temporal dependencies of gestures to more accurately identify the category and intention of gestures.
[0088] S104: Introduce pre-trained weights as initialization weights of the dynamic gesture recognition network, input the images in the sample set into the dynamic gesture recognition network for training, and output prediction results.
[0089] In this embodiment, the loaded pre-trained weights are used as the initialization weights of the dynamic gesture recognition network, so that the network has a certain feature extraction capability at the beginning of training, so it can converge to a better solution more quickly, which helps to reduce training time and improve training efficiency. The prediction result output by the dynamic gesture recognition network is the category corresponding to one or more gesture actions contained in the gesture image.
[0090] S105: Continuously compare the prediction results of the dynamic gesture recognition network with the corresponding labeled information in the sample set, calculate the loss value of the network, and feed the loss value back to the dynamic gesture recognition network, continuously modify the parameters of the network, and obtain a trained dynamic gesture recognition network.
[0091] In this implementation, in order to enable the dynamic gesture recognition network to be better applied in actual scenarios, in addition to the above-mentioned structural improvements to the dynamic gesture recognition network, the training process of the dynamic gesture recognition network is also optimized, including: using batch normalization, learning rate scheduling and data enhancement technology to optimize the training process of the dynamic gesture recognition network model; and performing model compression processing on the trained dynamic gesture recognition network model.
[0092] Specifically, batch normalization can be used to effectively normalize the input distribution in the network and alleviate the gradient vanishing problem during training. In specific implementation, by inserting normalization operations between each network layer, the mean and variance of each feature map are normalized to zero mean and unit variance to accelerate the training convergence speed and improve the generalization ability of the model. Using learning rate scheduling, the learning rate can be adjusted appropriately to improve the performance of the model. For example, the learning rate can be dynamically adjusted according to the verification error or other indicators during the training process, such as using strategies such as exponential decay and cosine decay, so that the model converges better and improves the generalization ability of the model.
[0093] Model compression processing includes pruning, quantization, distillation and other processing. For example, model pruning can be used to remove redundant or less contributing neurons and connections in the model, thereby reducing the number of model parameters and the amount of calculation. In specific implementation, pruning can be based on the importance of weights or other indicators, such as the absolute value of weights or gradient information, which can reduce the model's storage space and computing resource consumption while maintaining good performance, thereby improving the adaptability of the dynamic gesture recognition network model in different scenarios.
[0094] S106: Using the trained dynamic gesture recognition network to perform gesture recognition on the dynamic gesture video.
[0095] In this embodiment, the trained dynamic gesture recognition network can be applied and deployed in a variety of scenarios, such as virtual reality and augmented reality, biometric authentication, intelligent driving and other fields, so as to achieve real-time processing and recognition of dynamic gestures.
[0096] In addition, in the actual application of the dynamic gesture recognition network, it is necessary to continuously monitor the performance and performance of the network model, and iteratively optimize the model based on user feedback and data feedback to ensure the stability and accuracy of the model.
[0097] Please refer to Figure 2 , Figure 2 Schematic diagram of the application of the gesture recognition method in the field of intelligent driving in one embodiment of the present application. Figure 2 As shown, the application process mainly includes:
[0098] S201: Collect a sample set of diverse dynamic gesture videos and pre-process each gesture image.
[0099] In this embodiment, the processing of the gesture image includes: adjusting any image in the sample set to a preset size to obtain a first intermediate image; graying and standardizing the first intermediate image to obtain a second intermediate image; and randomly rotating, translating, scaling and mirroring the second intermediate image to obtain a preprocessed image.
[0100] In this embodiment, a first intermediate image is obtained by adjusting the image size to a uniform size to ensure that the images input to the network have the same dimension. Then, the first intermediate image is converted from a color image to a grayscale image to reduce the data dimension and complexity, and then the image is standardized so that the pixel values of the image are within a smaller range, such as [0,1] or [-1,1], to obtain a second intermediate image. After that, the second intermediate image is subjected to data enhancement processing. Data enhancement refers to expanding the training set through random transformations to increase the number and diversity of different samples that the network is exposed to during training, including random rotation, translation, scaling, mirroring, adding noise to the image, etc., to improve the generalization ability and robustness of the network.
[0101] S202: Divide the preprocessed images into data sets to obtain a training set, a validation set, and a test set.
[0102] In this embodiment, the sample set can be divided into a training set, a validation set and a test set according to the time sequence of the dynamic gesture video frames, and the division ratio can be 70% training set, 15% validation set, and 15% test set. Among them, the training set is mainly used to train the dynamic gesture recognition network, adjust the network weights and bias, and reduce the loss value of the network. Then, the dynamic gesture recognition network is cross-validated using the validation set, the hyperparameters of the network are adjusted, and the network performance is optimized. Finally, the performance of the dynamic gesture recognition network is evaluated using the test set to obtain the trained dynamic gesture recognition network. During the process, the data loader can be used to load gesture image data from the sample set and perform batch processing to speed up the training process and improve computing efficiency.
[0103] S203: Based on the application requirements of the intelligent driving scenario, the dynamic gesture recognition network is adjusted accordingly, and the adjusted dynamic gesture recognition network is trained using the training set, the validation set, and the test set.
[0104] At present, gesture recognition in the field of intelligent driving mainly includes the following aspects: 1. Driver status monitoring: By monitoring the driver's gestures and postures to assess their attention, fatigue and emotional state. By recognizing gestures, the system can detect whether the driver is yawning, lowering his head or distracted, so as to provide timely warnings or intervention. 2. Gesture control interactive interface: By recognizing the driver's gestures and converting them into vehicle control commands, such as adjusting the volume, switching music, answering calls, etc. Such an interactive interface can reduce the driver's distraction from the central control system and improve driving safety. 3. Gesture recognition driving assistance system: By developing a driving assistance system, it helps the driver to perform gesture navigation, gesture control of vehicle functions or gesture activation of specific driving modes. This system can provide a more intuitive and humanized driving experience while improving the convenience of driving operations. 4. Driving behavior analysis of gesture recognition: By analyzing the driver's gesture behavior, such as frequent gesture operations, bad driving habits, etc., it helps the driver improve driving behavior and improve driving safety and efficiency. 5. In-car monitoring with gesture recognition: The in-car monitoring system can identify passengers’ gestures, such as whether the passengers have fastened their seat belts or need to adjust their seat positions, thereby providing corresponding safety tips and reminders.
[0105] In this embodiment, before applying the dynamic gesture recognition network, it is necessary to make corresponding adjustments to the dynamic gesture recognition network according to the application requirements in the intelligent driving scenario, including the limitations of the actual intelligent driving scenario on the computing and storage resources of the network model, as well as the requirements for the gesture image processing speed and gesture recognition accuracy of the network model, etc., and accordingly make detailed adjustments to the parameters and components of the dynamic gesture recognition network so that the dynamic gesture recognition network can better adapt to the intelligent driving scenario and achieve better dynamic gesture recognition effect.
[0106] S204: deploying the trained dynamic gesture recognition network into the intelligent driving system to perform gesture recognition on gesture data generated during the driving process.
[0107] In this embodiment, the trained dynamic gesture recognition network can be deployed to the vehicle computing unit, cloud server or edge computing device, and can be flexibly selected according to specific needs and technical conditions during actual deployment. For example, for application scenarios that require high real-time performance and low latency, it may be preferred to deploy the network model on the vehicle computing unit or edge computing device; while for application scenarios that require large-scale data processing and complex calculations, it may be chosen to deploy the network model on the cloud server.
[0108] In addition, considering that the performance improvement of depthwise separable convolution in the dynamic gesture recognition network may be limited by hardware, in order to improve the performance of depthwise separable convolution in actual deployment, the dynamic gesture recognition network can be optimized in the following ways: optimizing and accelerating the conversion and deployment process of the dynamic gesture recognition network.
[0109] For example, frameworks such as TensorFlow Lite and ONNX can be used to convert and deploy network models, and hardware acceleration technologies such as GPU and TPU can be used to improve the performance of dynamic gesture recognition networks in actual deployment. Among them, TensorFlowLite is a lightweight, cross-platform deep learning framework designed for machine learning model inference on mobile and embedded devices. It optimizes the size and performance of the model so that deep learning models can run efficiently on resource-constrained devices.
[0110] Furthermore, before the trained dynamic gesture recognition network is deployed to the intelligent driving system, it also includes: performing model pruning on the dynamic gesture recognition network, removing redundant neurons and connections in the network, and compressing the network to reduce the size of the network model, making the dynamic gesture recognition network more lightweight and reducing the computing cost.
[0111] S205: Obtain user feedback, and perform an application evaluation on the dynamic gesture recognition network based on the user feedback and gesture data generated during driving to obtain an application evaluation result.
[0112] In this implementation, user feedback on gesture recognition accuracy, response speed, ease of use, etc. can be collected through questionnaires, user interviews or online evaluation systems, and gesture data generated during driving can be collected, especially those gesture cases that perform poorly during the application of the network model. At the same time, the performance of the network model in actual application is monitored, including key indicators such as false recognition rate, missed recognition rate, and processing time. Then, based on the collected user feedback and data feedback, the dynamic gesture recognition network is evaluated in an applied manner to obtain the corresponding applied evaluation results.
[0113] In specific implementation, the applied evaluation includes: in-depth analysis of misrecognized and missed gestures, identifying the common features or patterns that cause these errors, and then using methods such as cross-validation to evaluate the overall performance of the dynamic gesture recognition network model, identify possible performance bottlenecks, and check the diversity and representativeness of the newly collected data set to ensure that it covers various lighting conditions, backgrounds, gesture speeds and other variables.
[0114] S206: Based on the application evaluation results, iteratively update and optimize the dynamic gesture recognition network.
[0115] In this embodiment, the iterative optimization of the dynamic gesture recognition network includes: 1. Algorithm adjustment, adjusting the algorithm parameters of the network model, such as learning rate, regularization strength, optimizer, etc., according to the application evaluation results. 2. Feature engineering, improving the feature extraction method, such as using more advanced preprocessing technology, adopting multiple feature pyramid structures cascaded to extract gesture features of more dimensions, etc. 3. Model structure adjustment, if necessary, adjusting the structure of the network model, such as increasing the number of network layers, changing the layer type (such as loop layer, attention mechanism, etc.) or using a different network model architecture. 4. Data enhancement, enhancing the original data by rotating, scaling, cropping, adding noise, etc., to improve the generalization ability of the network model.
[0116] Finally, the dynamic gesture recognition network model is retrained using the adjusted parameters, features, or structures, and the new model is tested against the old model to compare the user satisfaction and performance indicators of the new and old models to evaluate the effect of the improvement. When the improvement effect meets the requirements, the updated and optimized dynamic gesture recognition network is redeployed to the intelligent driving system and continuously monitored to prepare for the next round of iterative optimization. Thus, through continuous feedback and iteration, the performance of the gesture recognition model is continuously improved and the user experience is enhanced.
[0117] Please refer to Figure 3 Based on the same inventive concept, the second aspect of the embodiment of the present application provides a gesture recognition device, the gesture recognition device 300 includes:
[0118] The data set acquisition module 301 is used to acquire a data set of dynamic gesture videos, where the data set is acquired by shooting or downloading according to different application scenarios;
[0119] The sample set generation module 302 is used to annotate each frame of the image in the data set, and the annotated information includes the frame number from the start to the end of the action and the category to which the action belongs, so as to obtain a sample set;
[0120] The network construction module 303 is used to construct a dynamic gesture recognition network, which includes a depth-separable convolution module, a feature pyramid module and a long short-term memory module; wherein,
[0121] The depthwise separable convolution module is used to perform primary feature extraction on the images in the sample set to obtain a first feature image;
[0122] The feature pyramid module is used to downsample the first feature image to obtain feature maps of different scales, and fuse the feature maps through upsampling and lateral connection to obtain a fused feature map containing features of different depths;
[0123] The long short-term memory module is used to extract local features of the fused feature map and perform temporal processing to capture the temporal spatial features of the image, map the processed feature map to the preset gesture category, and output the dynamic gesture recognition result;
[0124] A training module 304 is used to introduce pre-trained weights as initialization weights of the dynamic gesture recognition network, input images in the sample set into the dynamic gesture recognition network for training, and output prediction results;
[0125] The adjustment and optimization module 305 is used to continuously compare the prediction results of the dynamic gesture recognition network with the corresponding labeled information in the sample set, calculate the loss value of the network, and feed the loss value back to the dynamic gesture recognition network, and continuously modify the parameters of the network to obtain a trained dynamic gesture recognition network;
[0126] The application module 306 is used to use the trained dynamic gesture recognition network to perform gesture recognition on the dynamic gesture video.
[0127] Optionally, any depth-wise separable convolution in the depth-wise separable convolution module includes: an input layer, a depth-wise convolution layer, a point-wise convolution layer and an output layer; wherein,
[0128] The input layer input is a three-channel image;
[0129] The deep convolution layer is used to perform channel-by-channel convolution operations on the input three-channel image, using three convolution kernels to convolve the three channels respectively to generate three channel feature maps;
[0130] The point-by-point convolution layer is used to use the three-dimensional convolution kernel to linearly combine and reduce the dimension of the three channel feature maps, synthesize the three channel feature maps into one feature map, and output it through the output layer.
[0131] Optionally, the feature pyramid module includes an input layer, multiple symmetrical downsampling layers and multiple upsampling layers, the upsampling layers and the downsampling layers are horizontally connected, and the sizes of the feature maps output by the horizontally connected upsampling layers and downsampling layers are the same; wherein: after the input layer of the feature pyramid module obtains the first feature map, the first feature map is continuously downsampled through multiple downsampling layers, and scene context information with a global structure is extracted layer by layer to obtain a second feature map, wherein the size of the feature map output by each downsampling layer is halved in turn; the features extracted by each downsampling layer are fused into the corresponding upsampling layer through a horizontal connection, and the second feature map is continuously upsampled through multiple upsampling layers, and the size of the second feature map is restored layer by layer to the original size corresponding to the first feature map to obtain a fused feature map.
[0132] Optionally, the device further comprises:
[0133] A first processing module, used for adjusting any image in the sample set to a preset size to obtain a first intermediate image;
[0134] A second processing module, used for performing grayscale processing and standardization processing on the first intermediate image to obtain a second intermediate image;
[0135] A third processing module is used to perform random rotation, translation, scaling and mirroring processing on the second intermediate image to obtain a pre-processed image;
[0136] The sample set division module is used to divide the preprocessed images into data sets to obtain training sets, verification sets and test sets;
[0137] The first training module is used to train the dynamic gesture recognition network using the training set, adjust the network weights and biases, and reduce the network loss value;
[0138] The second training module is used to cross-validate the dynamic gesture recognition network using the validation set, adjust the network's hyperparameters, and optimize the network performance;
[0139] The third training module is used to evaluate the performance of the dynamic gesture recognition network using the test set to obtain a trained dynamic gesture recognition network.
[0140] Optionally, the application module 306 includes:
[0141] The deployment submodule is used to deploy the trained dynamic gesture recognition network into the intelligent driving system and perform gesture recognition on the gesture data generated during driving;
[0142] A feedback submodule is used to obtain user feedback and perform an application evaluation on the dynamic gesture recognition network based on the user feedback and gesture data generated during driving to obtain an application evaluation result;
[0143] The application optimization submodule is used to iteratively update and optimize the dynamic gesture recognition network based on the application evaluation results.
[0144] Optionally, the above device further includes:
[0145] The compression module is used to perform model pruning on the dynamic gesture recognition network, remove redundant neurons and connections in the network, and compress the network.
[0146] As for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.
[0147] In a third aspect, based on the same inventive concept, an embodiment of the present application provides a storage medium, in which machine executable instructions are stored. When the machine executable instructions are executed by a processor, the gesture recognition method proposed in the first aspect of the present application is implemented.
[0148] It should be noted that the specific implementation of the storage medium in the embodiment of the present application refers to the specific implementation of the gesture recognition method proposed in the first aspect of the embodiment of the present application, and will not be repeated here.
[0149] In the fourth aspect, based on the same inventive concept, an embodiment of the present application provides an electronic device, including a processor and a memory; the memory stores machine executable instructions that can be executed by the processor, and the processor is used to execute the machine executable instructions to implement the gesture recognition method proposed in the first aspect of the present application.
[0150] It should be noted that the specific implementation of the electronic device in the embodiment of the present application refers to the specific implementation of the gesture recognition method proposed in the first aspect of the embodiment of the present application, and will not be repeated here.
[0151] In the fifth aspect, based on the same inventive concept, refer to Figure 4 An embodiment of the present application provides a vehicle 400, including a processor 401 and a memory 402; the memory 402 stores machine executable instructions that can be executed by the processor 401, and the processor 401 is used to execute the machine executable instructions to implement the gesture recognition method proposed in the first aspect of the present application.
[0152] It should be noted that the specific implementation of the vehicle 400 in the embodiment of the present application refers to the specific implementation of the gesture recognition method proposed in the first aspect of the embodiment of the present application, and will not be repeated here.
[0153] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.
[0154] Those skilled in the art will appreciate that the embodiments of the present application can be provided as methods, devices, or computer program products. Therefore, the present application can adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application can adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.
[0155] The embodiments of the present application are described with reference to the flowcharts and / or block diagrams of the methods, terminal devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing terminal device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing terminal device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.
[0156] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing terminal device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce a manufactured product including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.
[0157] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal device so that a series of operating steps are executed on the computer or other programmable terminal device to produce a computer-implemented process, thereby providing instructions for executing on the computer or other programmable terminal device to implement the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.
[0158] Although the preferred embodiments of the present application have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the embodiments of the present application.
[0159] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or terminal device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or terminal device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or terminal device including the elements.
[0160] The above is a detailed introduction to a gesture recognition method, device, medium, equipment and vehicle provided by the present application. Specific examples are used in this article to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea. At the same time, for a person skilled in the art, according to the idea of the present application, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.
Claims
1. A gesture recognition method, characterized in that: The method comprises: Acquire a data set of dynamic gesture videos, where the data set is shot or downloaded for different application scenarios; Annotate each frame of the image in the data set, where the annotated information includes the frame number of the action from the start to the end and the category to which the action belongs, to obtain a sample set; A dynamic gesture recognition network is constructed, wherein the dynamic gesture recognition network includes a depth-separable convolution module, a feature pyramid module and a long short-term memory module; wherein, The depthwise separable convolution module is used to perform primary feature extraction on the image in the sample set to obtain a first feature image; The feature pyramid module is used to downsample the first feature image to obtain feature maps of different scales, and fuse the feature maps through upsampling and lateral connection to obtain a fused feature map containing features of different depths; The long short-term memory module is used to extract local features of the fused feature map and perform time-series processing to capture the time-series spatial features of the image, and map the processed feature map to a preset gesture category to output a dynamic gesture recognition result; Introducing pre-trained weights as initialization weights of a dynamic gesture recognition network, inputting images in the sample set into the dynamic gesture recognition network for training, and outputting prediction results; Continuously comparing the prediction results of the dynamic gesture recognition network with the corresponding labeled information in the sample set, calculating the loss value of the network, and feeding the loss value back to the dynamic gesture recognition network, continuously correcting the parameters of the network, and obtaining a trained dynamic gesture recognition network; The trained dynamic gesture recognition network is used to perform gesture recognition on dynamic gesture videos.
2. The method according to claim 1, characterized in that Any of the depth-wise separable convolutions in the depth-wise separable convolution module comprises: an input layer, a depth-wise convolution layer, a point-wise convolution layer and an output layer; wherein, The input of the input layer is a three-channel image; The deep convolution layer is used to perform channel-by-channel convolution operation on the input three-channel image, and use three convolution kernels to convolve the three channels respectively to generate three channel feature maps; The point-by-point convolution layer is used to use a three-dimensional convolution kernel to perform linear combination and dimensionality reduction on the three channel feature maps, synthesize the three channel feature maps into one feature map, and output it through the output layer.
3. The method according to claim 1, characterized in that: The feature pyramid module includes an input layer, a plurality of symmetrical downsampling layers and a plurality of upsampling layers, the upsampling layers and the downsampling layers are horizontally connected, and the sizes of the feature maps output by the horizontally connected upsampling layers and the downsampling layers are the same; wherein: After the input layer of the feature pyramid module obtains the first feature map, the first feature map is continuously downsampled through multiple downsampling layers, and scene context information with a global structure is extracted layer by layer to obtain a second feature map, wherein the size of the feature map output by each downsampling layer is halved successively; The features extracted by each downsampling layer are fused into the corresponding upsampling layer through lateral connection, and the second feature map is continuously upsampled through multiple upsampling layers, and the size of the second feature map is restored layer by layer to the original size corresponding to the first feature map, so as to obtain the fused feature map.
4. The method according to claim 1, characterized in that: The method further comprises: Adjusting any image in the sample set to a preset size to obtain a first intermediate image; Performing grayscale processing and standardization processing on the first intermediate image to obtain a second intermediate image; Performing random rotation, translation, scaling and mirroring processing on the second intermediate image to obtain a preprocessed image; Divide the preprocessed images into a data set to obtain a training set, a validation set, and a test set; Using the training set to train the dynamic gesture recognition network, adjusting the network weights and biases, and reducing the network loss value; Using the validation set to cross-validate the dynamic gesture recognition network, adjust the network's hyperparameters, and optimize network performance; The performance of the dynamic gesture recognition network is evaluated using the test set to obtain the trained dynamic gesture recognition network.
5. The method according to claim 1, characterized in that The trained dynamic gesture recognition network is used to perform gesture recognition on dynamic gesture videos, including: Deploying the trained dynamic gesture recognition network into an intelligent driving system to perform gesture recognition on gesture data generated during driving; Obtaining user feedback, and performing an application evaluation on the dynamic gesture recognition network based on the user feedback and gesture data generated during driving to obtain an application evaluation result; Based on the applied evaluation result, the dynamic gesture recognition network is iteratively updated and optimized.
6. The method according to claim 5, characterized in that Before deploying the trained dynamic gesture recognition network to the intelligent driving system, the method further includes: The dynamic gesture recognition network is subjected to model pruning processing, redundant neurons and connections in the network are removed, and the network is compressed.
7. A gesture recognition device, characterized in that: The device comprises: A data set acquisition module is used to acquire a data set of dynamic gesture videos, wherein the data set is acquired by shooting or downloading according to different application scenarios; A sample set generation module is used to annotate each frame of the image in the data set, wherein the annotated information includes the frame number of the action from the beginning to the end and the category to which the action belongs, thereby obtaining a sample set; A network construction module is used to construct a dynamic gesture recognition network, wherein the dynamic gesture recognition network includes a depth-separable convolution module, a feature pyramid module and a long short-term memory module; wherein, The depthwise separable convolution module is used to perform primary feature extraction on the image in the sample set to obtain a first feature image; The feature pyramid module is used to downsample the first feature image to obtain feature maps of different scales, and fuse the feature maps through upsampling and lateral connection to obtain a fused feature map containing features of different depths; The long short-term memory module is used to extract local features of the fused feature map and perform time-series processing to capture the time-series spatial features of the image, and map the processed feature map to a preset gesture category to output a dynamic gesture recognition result; A training module, used for introducing pre-trained weights as initialization weights of a dynamic gesture recognition network, inputting images in the sample set into the dynamic gesture recognition network for training, and outputting prediction results; An adjustment and optimization module is used to continuously compare the prediction results of the dynamic gesture recognition network with the corresponding labeled information in the sample set, calculate the loss value of the network, and feed the loss value back to the dynamic gesture recognition network, continuously modify the parameters of the network, and obtain a trained dynamic gesture recognition network; The application module is used to use the trained dynamic gesture recognition network to perform gesture recognition on dynamic gesture videos.
8. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the gesture recognition method according to any one of claims 1 to 6 is implemented.
9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the gesture recognition method according to any one of claims 1 to 6 is implemented.
10. A vehicle, characterized in that: It comprises a processor and a memory; the memory stores machine executable instructions that can be executed by the processor, and the processor is used to execute the machine executable instructions to implement the gesture recognition method as described in any one of claims 1 to 6.
Citation Information
Cited By
Gesture recognition method and system based on deep learning
CN119445676A
Dynamic gesture recognition method and device, equipment and storage medium
CN119964246A
A dynamic gesture recognition method, device, equipment and storage medium
CN119964246B
Equipment control method and device, electronic equipment and computer readable storage medium
CN120491836A
Device control method and apparatus, electronic device, and computer-readable storage medium
CN120491836B
