Dynamic gesture detection network training method and device, and dynamic gesture detection method
By extracting frame differences and hand detection bounding box feature maps from gesture videos for training a dynamic gesture detection network, the problems of large training data requirements and complexity are solved, enabling faster and lighter training and detection.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- CHENGDU XGIMI TECH CO LTD
- Filing Date
- 2021-08-12
- Publication Date
- 2026-07-03
AI Technical Summary
In existing technologies, training dynamic gesture detection networks requires a large amount of training data and the training process is complex, making it difficult to perform efficiently.
By acquiring gesture videos and performing image processing, frame difference feature maps and hand detection box feature maps are extracted. Combined with convolutional layers for iterative training, the dependence on environmental factors is reduced, and a lightweight network structure is designed.
This reduces the need for training data, shortens training time, and improves the network's convergence speed and detection efficiency.
Smart Images
Figure CN115708134B_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of deep learning technology, and more specifically, to a method, apparatus, and method for training a dynamic gesture detection network. Background Technology
[0002] Before the advent of deep learning, gesture recognition was often achieved through traditional image processing methods, relying on manual feature extraction or inter-frame correlation to determine gestures. However, traditional image processing methods have many limitations, being greatly affected by environmental factors such as lighting, and are not robust enough to be applied to various scenarios. With the rise of deep learning, due to its automatic feature extraction and greater robustness compared to traditional image processing, more and more scholars have entered this field of research, and gesture recognition schemes using deep learning to replace traditional image processing have become increasingly mature.
[0003] In related technologies, most gesture recognition methods employ machine learning or deep learning to train on a large number of gesture videos or skeletal point sequences to achieve the classification and recognition of continuous video sequences. Common dynamic gesture recognition networks include spatiotemporal convolutional networks such as C3D or TSN, or SGN networks based on skeletal point sequence recognition. However, these networks are difficult to learn due to their large number of parameters, requiring more time and convergence. Furthermore, obtaining a robust model requires a large amount of training data.
[0004] There is currently no effective solution to the above problems. Summary of the Invention
[0005] This application provides a method, apparatus, and dynamic gesture detection method for training a dynamic gesture detection network, which at least solves the technical problems of requiring a large amount of training data and having a complex training process when training a dynamic gesture detection network in related technologies.
[0006] According to one aspect of the embodiments of this application, a method for training a dynamic gesture detection network is provided, comprising: acquiring a plurality of gesture videos, wherein each gesture video corresponds to a dynamic gesture; performing image processing on each gesture video to determine a first feature map and a second feature map corresponding to each gesture video; wherein the first feature map corresponding to any gesture video is used to characterize: motion information between two adjacent frames in the gesture video, or hand contour information of an image in the gesture video; the second feature map corresponding to any gesture video is used to characterize: hand region position information of an image in the gesture video; and iteratively training the dynamic gesture detection network based on the first feature map and the second feature map corresponding to each gesture video to obtain a target detection network.
[0007] Optionally, a first annotation information for a first gesture video is determined, and a first number of frame images are acquired from the first gesture video, wherein the first gesture video is any one of the plurality of gesture videos, and the first annotation information is used to annotate the dynamic gesture type corresponding to the first gesture video; the first number of frame images are converted into a first number of grayscale images; based on the first number of grayscale images, a second number of the first feature maps corresponding to the first gesture video are determined; based on the first number of grayscale images, a third number of the second feature maps corresponding to the first gesture video are determined.
[0008] Optionally, the first feature map is a frame difference feature map. Determining the second number of first feature maps corresponding to the first gesture video includes: performing frame difference processing on the first number of grayscale images to determine the second number of first feature maps corresponding to the first gesture video. The frame difference processing includes: for any two adjacent grayscale images in the first number of grayscale images, subtracting the pixels of the later frame from the pixels of the previous frame to determine the frame difference feature map corresponding to the two adjacent grayscale images. The frame difference feature map includes motion information between the two adjacent grayscale images.
[0009] Optionally, the second feature map is a hand detection bounding box feature map. Determining the third number of second feature maps corresponding to the first gesture video includes: determining the third number of single-channel feature maps, wherein the single-channel feature map is a first color and the single-channel feature map has the same size as the grayscale image; selecting the third number of target grayscale images from the first number of grayscale images and inputting them into the pre-trained hand detection model; determining the hand region location information in each frame of the target grayscale image; and drawing the hand region of the second color in the single-channel feature map of each frame based on the hand region location information to obtain the third number of hand detection bounding box feature maps.
[0010] Optionally, the dynamic gesture detection network includes a first convolutional layer, a second convolutional layer, and an array concatenation layer. The first convolutional layer performs convolution processing on a first feature map corresponding to the first gesture video to determine a first convolution result; the second convolutional layer performs convolution processing on a second feature map corresponding to the first gesture video to determine a second convolution result; the array concatenation layer connects the first and second convolution results to obtain a third convolution result; a gesture prediction result for the first gesture video is determined based on the third convolution result; a loss function value corresponding to the dynamic gesture detection network is determined according to the gesture prediction result and the first annotation information; the network parameters of the dynamic gesture detection network are updated in the direction of reducing the loss function value; and the updated dynamic gesture detection network is iteratively trained based on other gesture videos besides the first gesture video from the plurality of gesture videos.
[0011] Optionally, the second number of first feature maps are merged to obtain first data, and the first data is input into the first convolutional layer. The first convolutional layer extracts features from the first data using a convolutional kernel of a preset size to determine the first convolution result. The third number of second feature maps are merged to obtain second data, and the second data is input into the second convolutional layer. The second convolutional layer extracts features from the second data using a convolutional kernel of a preset size to determine the second convolution result.
[0012] Optionally, the dynamic gesture detection network includes a first convolutional layer and a second convolutional layer, and the iterative training includes a first training phase and a second training phase, wherein the first training phase occurs earlier than the second training phase. In the first training phase, the learning rate for the first convolutional layer is a first learning rate, and the learning rate for the second convolutional layer is 0. In the second training phase, the learning rate for the first convolutional layer is the first learning rate, and the learning rate for the second convolutional layer is a second learning rate.
[0013] Optionally, the first feature map includes: a frame difference feature map, an optical flow feature map, a HOG3D feature map, or a SIFT3D feature map; the second feature map includes: a hand detection box feature map, a region of interest feature map learned through a self-attention mechanism, or a hand region feature map extracted through image segmentation.
[0014] According to another aspect of the embodiments of this application, a dynamic gesture detection method is also provided, comprising: acquiring a video to be detected, and extracting a first number of frame images from the video to be detected; determining a second number of first feature maps and a third number of second feature maps based on the first number of frame images, wherein the first feature map is used to characterize: motion information between two adjacent frame images in the video to be detected, or hand contour information of an image in the video to be detected; the second feature map is used to characterize: hand region position information of an image in the video to be detected; inputting the second number of first feature maps and the third number of second feature maps into a pre-trained dynamic gesture detection network, and outputting a target dynamic gesture type corresponding to the video to be detected through the dynamic gesture detection network.
[0015] According to another aspect of the embodiments of this application, a dynamic gesture detection network training device is also provided, comprising: an acquisition module, configured to acquire multiple gesture videos, wherein each gesture video corresponds to a dynamic gesture; a determination module, configured to perform image processing on each gesture video to determine a first feature map and a second feature map corresponding to each gesture video; wherein the first feature map corresponding to any gesture video is used to characterize: motion information between two adjacent frames in any gesture video, or hand contour information of an image in any gesture video; the second feature map corresponding to any gesture video is used to characterize: hand region position information of an image in any gesture video; and a training module, configured to iteratively train the dynamic gesture detection network based on the first feature map and the second feature map corresponding to each gesture video to obtain a target detection network.
[0016] According to another aspect of the embodiments of this application, an electronic device is also provided, including: a memory and a processor, wherein the memory stores a computer program, and the processor is configured to execute the above-described dynamic gesture detection network training method or dynamic gesture detection method through the computer program.
[0017] In this embodiment, multiple gesture videos are first acquired, each corresponding to a dynamic gesture. Then, image processing is performed on each gesture video to determine a first feature map and a second feature map corresponding to each gesture video. The first feature map represents the motion information between two adjacent frames in the gesture video, or the hand contour information in the image of the gesture video. The second feature map represents the positional information of the hand region in the image of the gesture video. Based on the first and second feature maps corresponding to each gesture video, the dynamic gesture detection network is iteratively trained to obtain the target detection network. By pre-extracting the spatiotemporal features, morphological changes, and movement information of the hand using traditional image processing methods, the obtained features are more controllable and stable. Since the features are abstractions of real image data, the amount of training data required by the network is greatly reduced, allowing the detection network to be designed more lightweightly, with shorter training time and easier convergence. This solves the technical problem in related technologies where training dynamic gesture detection networks requires a large amount of training data and involves a complex training process. Attached Figure Description
[0018] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings:
[0019] Figure 1 This is a flowchart illustrating a dynamic gesture detection network training method according to an embodiment of this application;
[0020] Figure 2 This is a schematic diagram of the structure of a dynamic gesture detection network according to an embodiment of this application;
[0021] Figure 3 This is a schematic diagram of the structure of a data merging result according to an embodiment of this application;
[0022] Figure 4 This is a flowchart illustrating a dynamic gesture detection method according to an embodiment of this application;
[0023] Figure 5 This is a flowchart illustrating a dynamic gesture detection method in a projection scene according to an embodiment of this application;
[0024] Figure 6 This is a schematic diagram of the structure of a dynamic gesture detection network training device according to an embodiment of this application. Detailed Implementation
[0025] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort should fall within the scope of protection of the present application.
[0026] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.
[0027] Example 1
[0028] In related technologies, most gesture recognition methods employ machine learning or deep learning to train on a large number of gesture videos or skeletal point sequences to achieve the function of classifying and recognizing continuous video sequences. Common dynamic gesture recognition networks include spatiotemporal convolutional networks (C3D or TSN) or skeletal point sequence-based networks (SGN). However, these networks are difficult to learn due to their large number of parameters, requiring more time and convergence. Furthermore, obtaining a robust model requires a large amount of training data.
[0029] To address the aforementioned problems, this application provides an embodiment of a dynamic gesture detection network training method, aiming to achieve dynamic gesture detection using less data and a more lightweight deep learning network. It should be noted that the steps shown in the flowcharts of the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowcharts, in some cases, the steps shown or described may be performed in a different order than that shown here.
[0030] Figure 1 This is a flowchart illustrating the dynamic gesture detection network training method according to an embodiment of this application, as shown below. Figure 1 As shown, the method includes at least steps S102-S106, wherein:
[0031] Step S102: Acquire multiple gesture videos, where each gesture video corresponds to a dynamic gesture.
[0032] In some optional embodiments of this application, when acquiring gesture videos, 1000 sets of videos corresponding to each dynamic gesture can be collected, with each set of videos lasting 1 second, that is, the video contains 30 to 40 frames of images. The dynamic gestures include swiping the palm to the left, swiping the palm to the right, swiping the palm up, swiping the palm down, etc. The specific gesture categories can be changed according to the needs, and no specific limitation is made here.
[0033] Step S104: Perform image processing on each gesture video to determine the first feature map and the second feature map corresponding to each gesture video; wherein, the first feature map corresponding to any gesture video is used to characterize: the motion information between two adjacent frames in any gesture video, or the hand contour information of the image in any gesture video; the second feature map corresponding to any gesture video is used to characterize: the position information of the hand region in the image in any gesture video.
[0034] Specifically, the first feature map can be a frame difference feature map, an optical flow feature map, a HOG3D feature map, or a SIFT3D feature map. The frame difference feature map and the optical flow feature map are used to represent the motion information between two adjacent frames in the gesture video, while the HOG3D feature map and the SIFT3D feature map are used to represent the hand contour information in the gesture video. The second feature map can be a hand detection box feature map, a region of interest feature map learned through a self-attention mechanism, or a hand region feature map extracted through image segmentation; all of these are used to represent the hand region position information in the gesture video.
[0035] In some optional embodiments of this application, the first feature map and the second feature map can be determined by the following steps:
[0036] Step S1041: Determine the first annotation information of the first gesture video and collect a first number of frame images from the first gesture video, wherein the first gesture video is any gesture video among multiple gesture videos, and the first annotation information is used to annotate the dynamic gesture type corresponding to the first gesture video.
[0037] Specifically, for any gesture video, first label the type of dynamic gesture corresponding to the video. For example, the category labels 1-4 correspond to four dynamic gestures: swipe left, swipe right, swipe up, and swipe down. Then, capture a first number of frame images from the gesture video at equal or random intervals. The first number is usually set to 9 frames, but can be adjusted as needed. No specific limitation is made here.
[0038] Step S1042: Convert the first number of frame images into the first number of grayscale images.
[0039] Step S1043: Based on the first number of grayscale images, determine the second number of first feature maps corresponding to the first gesture video.
[0040] Specifically, taking the first feature map as the frame difference feature map as an example, firstly, the first number of grayscale images are subjected to frame difference processing to determine the second number of first feature maps corresponding to the first gesture video. The frame difference processing process includes: for any two adjacent grayscale images in the first number of grayscale images, the difference between the pixels of the later frame image and the pixels of the previous frame image is calculated to determine the frame difference feature map corresponding to the two adjacent grayscale images. The frame difference feature map includes motion information between the two adjacent grayscale images.
[0041] For example, nine frames are selected at equal intervals from the first gesture video and converted to grayscale images. The difference between adjacent frames is calculated sequentially. The difference between the grayscale images of the next frame and the previous frame is used to calculate the shifted portion between the two frames. The stationary portion is then removed, which is the diff. cur =I last -I pre , where I last I represents the number of pixels in the next frame of the image. pre Diff represents the pixels of the previous frame image. cur This indicates the obtained frame difference feature map, and a total of 8 frame difference feature maps are finally obtained.
[0042] It should be noted that when the first feature map is a frame difference feature map or an optical flow feature map, the difference between the first quantity and the second quantity is 1, while when the first feature map is a HOG3D feature map or a SIFT3D feature map, the first quantity and the second quantity are the same.
[0043] Step S1044: Based on the first number of grayscale images, determine the third number of second feature images corresponding to the first gesture video.
[0044] Specifically, taking the second feature map as the hand detection box feature map as an example, firstly, a third number of single-channel feature maps are determined, wherein the single-channel feature map is of the first color and the single-channel feature map has the same size as the grayscale image; the third number of target grayscale images are selected from the first number of grayscale images and input into the pre-trained hand detection model to determine the hand region location information in each frame of the target grayscale image, and the second color hand region is drawn in the single-channel feature map of each frame according to the hand region location information, thus obtaining the third number of hand detection box feature maps.
[0045] To facilitate data processing during subsequent training, the third quantity can be consistent with the second quantity. The first color is usually black (pixel value 0), and the second color is usually white (pixel value 255). That is, the single-channel feature map is initialized to black, and the white hand area is drawn in it to obtain a black and white hand detection box feature map. Of course, the relevant colors can also be defined according to the requirements.
[0046] For example, after obtaining 8 frames of frame difference feature maps, we can first initialize 8 single-channel black images with the same size as the grayscale images. Then, we can use a pre-trained hand detection model (such as the commonly used YOLO series or SSD series) to detect the last 8 frames of the above 9 grayscale images, locate the hand ROI (Region of Interest) position, and fill the corresponding hand ROI positions in the 8 single-channel black images with white, finally obtaining 8 frames of black and white hand detection box feature maps.
[0047] In this embodiment, by pre-extracting frame difference features, significant change regions between frames can be obtained, eliminating stationary environmental factors. Simultaneously, introducing hand detection box feature maps provides a better representation of hand movement trajectory information. Combining these two feature sets reduces the influence of environmental and lighting factors in real-world data, retaining only hand-related feature information. This allows the detection network to focus solely on learning hand-related information without needing to learn environmental information, reducing the demand for training environment data. It also makes the network more lightweight, achieving good results without requiring a deep network.
[0048] Step S106: The dynamic gesture detection network is iteratively trained based on the first and second feature maps corresponding to each gesture video to obtain the target detection network.
[0049] Because relevant feature information is extracted in advance, the dynamic gesture detection network in this embodiment can be designed to be more lightweight. Figure 2 A structural diagram of an optional dynamic gesture detection network is shown. This network can be divided into two branches, corresponding to a first convolutional layer and a second convolutional layer, respectively. It also includes an array connection layer and a fully connected layer. Based on this dynamic gesture detection network, an optional network training step according to an embodiment of this application is as follows:
[0050] Step S1061: Perform convolution processing on the first feature map corresponding to the first gesture video through the first convolution layer to determine the first convolution result.
[0051] Specifically, the first feature maps of the second quantity are merged to obtain the first data. The first data is then input into the first convolutional layer, which extracts features from the first data using a convolutional kernel of a preset size to determine the first convolution result.
[0052] For example, suppose the first feature map is a frame difference feature map, and the second number is 8, such as Figure 2 As shown, the 8 frame difference feature maps can be merged into a first data structure of M*N*8, where M and N are the width and height of the frame difference feature map. The specific data structure is as follows: Figure 3 As shown, the first data of size M*N*8 is input into the first convolutional layer. 64 convolutional kernels of size M*N*8 are used to learn the input first data, so that the convolutional kernels learn the spatial and spatiotemporal distribution of the frame difference feature map, and obtain the first convolution result of size M*N*64.
[0053] Step S1062: Perform convolution processing on the second feature map corresponding to the first gesture video through the second convolution layer to determine the second convolution result.
[0054] Specifically, the second feature maps of the third number are merged to obtain the second data. The second data is then input into the second convolutional layer, which extracts features from the second data using a convolutional kernel of a preset size to determine the second convolution result.
[0055] Taking the first feature map as the frame difference feature map and the second feature map as having 8 elements as an example, assuming the second feature map is the hand detection box feature map, and the third feature map also has 8 elements, and so on... Figure 2 The feature maps of 8 hand detection boxes can be merged into a second data of M*N*8, where M and N are the width and height of the hand detection box feature map. The second data of M*N*8 is input into the second convolutional layer, and 64 convolutional kernels of size M*N*8 are used to learn the input second data, so that the convolutional kernels learn the inter-frame movement information of the hand detection box, and obtain the second convolution result of M*N*64.
[0056] Step S1063: The first convolution result and the second convolution result are connected through an array connection layer to obtain the third convolution result.
[0057] like Figure 2 As shown, the array concatenation layer connects the M*N*64 convolution results obtained from the two convolutional layers to obtain a third convolution result of M*N*128.
[0058] Step S1064: Determine the gesture prediction result for the first gesture video based on the third convolution result.
[0059] like Figure 2 As shown, the result of the third convolution can be input into the fully connected layer to output the gesture prediction result for the first gesture video.
[0060] Step S1065: Determine the loss function value corresponding to the dynamic gesture detection network based on the gesture prediction result and the first annotation information.
[0061] Step S1066: Update the network parameters of the dynamic gesture detection network in the direction of reducing the loss function value.
[0062] Specifically, after determining the loss function value based on the gesture prediction results and gesture annotation information output by the dynamic gesture detection network, the network parameters in the dynamic gesture detection network are adjusted in the direction of reducing the loss function value, that is, the optimal fit of the network is achieved by minimizing the loss function.
[0063] Step S1067: Based on the other gesture videos (excluding the first gesture video) among the multiple gesture videos, continue to iteratively train the updated dynamic gesture detection network.
[0064] Furthermore, as a feasible approach, iterative training can be stopped when the loss function value corresponding to the dynamic gesture detection network converges, thus obtaining the object detection network. Alternatively, training can be stopped when the number of iterations reaches a pre-configured training threshold (e.g., 10,000), thus obtaining the object detection network.
[0065] In some optional embodiments of this application, the entire training process can utilize open-source frameworks such as TensorFlow, PyTorch, or Caffe. The framework's corresponding interfaces are used to build the structure of the dynamic gesture detection network and set the appropriate training parameters. Typically, the solution method is set to stochastic gradient descent with a learning rate of 0.01 and 10,000 training iterations. The specific training parameters can be adjusted according to the actual training situation. By inputting the first and second feature maps corresponding to each gesture video into the dynamic gesture detection network for iterative training, the final object detection network is obtained.
[0066] Optionally, to accelerate training and ensure more thorough learning across different feature information, the entire iterative training process can be divided into a first training phase and a second training phase, with the first training phase occurring earlier than the second. In the first training phase, the learning rate for the first convolutional layer is the first learning rate, and the learning rate for the second convolutional layer is 0. In the second training phase, the learning rate for the first convolutional layer is the first learning rate, and the learning rate for the second convolutional layer is the second learning rate. The first and second learning rates can be adjusted according to the actual training situation; they can be the same, for example, both being 0.01.
[0067] For example, the total number of training iterations is set to 10,000. The first 5,000 iterations are designated as the first training phase, and the last 5,000 iterations as the second training phase. In the first training phase, the learning rate of the first convolutional layer is set to 0.01 (i.e., the first learning rate mentioned above), and the learning rate of the second convolutional layer is set to 0. This ensures that the convolutional kernel part of the frame difference feature map is trained separately. In the second training phase, the learning rates of both the first and second convolutional layers are set to 0.01. This ensures more thorough learning of the frame difference feature map, while simultaneously learning the hand detection box feature map part, allowing for precise fine-tuning of the training results.
[0068] During training, the dynamic gesture detection network can be designed to be more lightweight, with shorter training time and easier convergence due to the pre-extraction of relevant features. The network inference speed is accelerated by learning the spatiotemporal feature distribution of multi-frame video data using non-3D convolutional kernels. At the same time, by adopting a frozen training method, the upper half of the network is trained first, and then both branches are trained simultaneously after the upper half of the network converges, which makes the training more thorough and the convergence speed faster.
[0069] In this embodiment, multiple gesture videos are first acquired, each corresponding to a dynamic gesture. Then, image processing is performed on each gesture video to determine a first feature map and a second feature map corresponding to each gesture video. The first feature map represents the motion information between two adjacent frames in the gesture video, or the hand contour information in the image of the gesture video. The second feature map represents the positional information of the hand region in the image of the gesture video. Based on the first and second feature maps corresponding to each gesture video, the dynamic gesture detection network is iteratively trained to obtain the target detection network. By pre-extracting the spatiotemporal features, morphological changes, and movement information of the hand using traditional image processing methods, the obtained features are more controllable and stable. Since the features are abstractions of real image data, the amount of training data required by the network is greatly reduced, allowing the detection network to be designed more lightweightly, with shorter training time and easier convergence. This solves the technical problem in related technologies where training dynamic gesture detection networks requires a large amount of training data and involves a complex training process.
[0070] Example 2
[0071] Based on the dynamic gesture detection network training method provided in Embodiment 1, and based on the trained dynamic gesture detection network, this application provides an embodiment of a dynamic gesture detection method. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Furthermore, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.
[0072] Figure 4 This is a flowchart illustrating the dynamic gesture detection method according to an embodiment of this application, as shown below. Figure 4 As shown, the method includes at least steps S402-S406, wherein:
[0073] Step S402: Obtain the video to be detected and extract a first number of frame images from the video to be detected.
[0074] The video to be detected can be a single gesture video or a series of frames captured by a camera. For example, when applying dynamic gesture detection in projection, the projector's front-facing camera can capture 30 consecutive frames up to the current moment, and then select 9 frames at equal intervals. Here, the initial number of 9 frames is just an example and can be adjusted as needed.
[0075] Step S404: Determine a second number of first feature maps and a third number of second feature maps based on a first number of frame images, wherein the first feature map is used to characterize: motion information between two adjacent frames in the video to be detected, or hand contour information of an image in the video to be detected; the second feature map is used to characterize: hand region position information of an image in the video to be detected.
[0076] Specifically, the process of determining the first feature map and the second feature map is basically the same as in Embodiment 1: a first number of frame images are converted into a first number of grayscale images; a second number of first feature maps are determined based on the first number of grayscale images; and a third number of second feature maps are determined based on the first number of grayscale images. Since Embodiment 1 has already described this process in detail, some details not shown in this embodiment can be referred to Embodiment 1, and will not be elaborated further here.
[0077] Step S406: Input the second number of first feature maps and the third number of second feature maps into the pre-trained dynamic gesture detection network, and output the target dynamic gesture type corresponding to the video to be detected through the dynamic gesture detection network.
[0078] In some optional embodiments of this application, after the obtained first feature map and second feature map are input into the trained dynamic gesture detection network, the dynamic gesture detection network will predict the probability of different dynamic gesture types corresponding to the video to be detected, and the dynamic gesture type with the highest probability can be taken as the target dynamic gesture type corresponding to the video to be detected.
[0079] Optionally, a probability threshold can be preset. If the highest probability among the probabilities of different dynamic gesture types predicted by the dynamic gesture detection network for the video to be detected is still less than the probability threshold, then the confidence of the current dynamic gesture detection result is considered low, and the prediction result can be abandoned and the video to be detected can be reacquired for detection. The probability threshold can be greater than 0.5 and less than 1, for example, 0.7.
[0080] Figure 5 This paper presents a complete flowchart of a dynamic gesture detection method applied to a projection scene, which combines the dynamic gesture detection network training method in Example 1 with the dynamic gesture detection method in this example. The specific flowchart is as follows:
[0081] 1. Collect 1000 sets of videos for each of the four types of dynamic gestures, with each set of videos lasting approximately 1 second;
[0082] 2. Select 9 frames from each group of videos at intervals and label them with the corresponding dynamic gesture type (1-4);
[0083] 3. Calculate the difference of changed pixels between adjacent frames to obtain the frame difference feature map;
[0084] 4. Use a hand detection model to detect the hand position and generate a hand detection bounding box feature map;
[0085] 5. Construct a dynamic gesture detection network, using frame difference feature maps and hand detection box feature maps as inputs, and fit the input and output;
[0086] 6. Iteratively train the dynamic gesture detection network to obtain the target detection network;
[0087] 7. The camera captures the projected image, accumulating 30 frames.
[0088] 8. Select 9 frames at intervals from 30 frames;
[0089] 9. Calculate the difference of changed pixels between adjacent frames to obtain the frame difference feature map;
[0090] 10. Use a hand detection model to detect the position of the hand and generate a feature map of the hand detection box;
[0091] 11. Input the frame difference feature map and the hand detection box feature map into the target detection network;
[0092] 12. The object detection network predicts the corresponding target dynamic gesture type.
[0093] In this embodiment, a video to be detected is acquired, and a first number of frame images are extracted from the video. A second number of first feature maps and a third number of second feature maps are determined based on the first number of frame images. The second number of first feature maps and the third number of second feature maps are input into a pre-trained dynamic gesture detection network, which then outputs the target dynamic gesture type corresponding to the video to be detected. Specifically, by pre-extracting the spatiotemporal features, morphological changes, and movement information of the hand using traditional image processing methods, the obtained features are more controllable and stable. Since the features are abstractions of real image data, the amount of training data required by the subsequent network is greatly reduced, allowing the detection network to be designed more lightweightly, with shorter training time and easier convergence. This solves the technical problem in related technologies where training dynamic gesture detection networks requires a large amount of training data and involves a complex training process.
[0094] Example 3
[0095] According to an embodiment of this application, a dynamic gesture detection network training device for implementing the dynamic gesture detection network training method in Embodiment 1 is also provided, such as... Figure 6 As shown, the device includes an acquisition module 60, a determination module 62, and a training module 64, wherein:
[0096] The acquisition module 60 is used to acquire multiple gesture videos, where each gesture video corresponds to a dynamic gesture.
[0097] The determining module 62 is used to perform image processing on each gesture video to determine the first feature map and the second feature map corresponding to each gesture video; wherein, the first feature map corresponding to any gesture video is used to characterize: the motion information between two adjacent frames in any gesture video, or the hand contour information of the image in any gesture video; the second feature map corresponding to any gesture video is used to characterize: the position information of the hand region in the image in any gesture video.
[0098] Training module 64 is used to iteratively train the dynamic gesture detection network based on the first and second feature maps corresponding to each gesture video to obtain the object detection network.
[0099] It should be noted that each module in the dynamic gesture detection network training device in this application embodiment corresponds one-to-one with the implementation steps of the dynamic gesture detection network training method in embodiment 1. Since embodiment 1 has been described in detail, some details not shown in this embodiment can be referred to embodiment 1, and will not be elaborated further here.
[0100] Example 4
[0101] According to an embodiment of this application, an electronic device is also provided, including: a memory and a processor, wherein the memory stores a computer program, and the processor is configured to execute the above-described dynamic gesture detection network training method or dynamic gesture detection method through the computer program.
[0102] Optionally, the processor performs the following steps via a computer program: acquiring multiple gesture videos, wherein each gesture video corresponds to a dynamic gesture; performing image processing on each gesture video to determine a first feature map and a second feature map corresponding to each gesture video; wherein the first feature map corresponding to any gesture video is used to characterize: motion information between two adjacent frames in any gesture video, or hand contour information in any gesture video; the second feature map corresponding to any gesture video is used to characterize: hand region position information in any gesture video; and iteratively training the dynamic gesture detection network based on the first and second feature maps corresponding to each gesture video to obtain an object detection network.
[0103] Optionally, the processor performs the following steps via a computer program: acquiring a video to be detected and extracting a first number of frame images from the video to be detected; determining a second number of first feature maps and a third number of second feature maps based on the first number of frame images, wherein the first feature map is used to characterize: motion information between two adjacent frame images in the video to be detected, or hand contour information of an image in the video to be detected; the second feature map is used to characterize: hand region position information of an image in the video to be detected; inputting the second number of first feature maps and the third number of second feature maps into a pre-trained dynamic gesture detection network, and outputting the target dynamic gesture type corresponding to the video to be detected through the dynamic gesture detection network.
[0104] The sequence numbers of the embodiments in this application are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.
[0105] In the above embodiments of this application, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.
[0106] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. The device embodiments described above are merely illustrative; for example, the division of units can be a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the displayed or discussed mutual couplings, direct couplings, or communication connections may be through some interfaces; indirect couplings or communication connections between units or modules may be electrical or other forms.
[0107] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.
[0108] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.
[0109] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as a USB flash drive, read-only memory (ROM), random access memory (RAM), portable hard drive, magnetic disk, or optical disk.
[0110] The above are merely preferred embodiments of this application. It should be noted that those skilled in the art can make various improvements and modifications without departing from the principles of this application, and these improvements and modifications should also be considered within the scope of protection of this application.
Claims
1. A method for training a dynamic gesture detection network, characterized in that, include: Acquire multiple gesture videos, where each gesture video corresponds to a dynamic gesture; Image processing is performed on each gesture video to determine a first feature map and a second feature map corresponding to each gesture video; wherein, the first feature map corresponding to any gesture video is used to characterize: motion information between two adjacent frames in any gesture video, or hand contour information in any gesture video; the second feature map corresponding to any gesture video is used to characterize: hand region position information in any gesture video. The dynamic gesture detection network is iteratively trained based on the first and second feature maps corresponding to each gesture video to obtain a target detection network. The dynamic gesture detection network includes: a first convolutional layer for convolutional processing of the first feature map, a second convolutional layer for convolutional processing of the second feature map, and an array concatenation layer. The iterative training includes a first training phase and a second training phase. The first training phase occurs earlier than the second training phase. In the first training phase, the learning rate for the first convolutional layer is a first learning rate, and the learning rate for the second convolutional layer is 0. In the second training phase, the learning rate for the first convolutional layer is the first learning rate, and the learning rate for the second convolutional layer is a second learning rate.
2. The method according to claim 1, characterized in that, The step of performing image processing on each gesture video to determine the first feature map and the second feature map corresponding to each gesture video includes: Determine the first annotation information of the first gesture video, and collect a first number of frame images from the first gesture video, wherein the first gesture video is any gesture video among the plurality of gesture videos, and the first annotation information is used to annotate the dynamic gesture type corresponding to the first gesture video; Convert the first number of frame images into a first number of grayscale images; Based on the first number of grayscale images, determine the second number of the first feature maps corresponding to the first gesture video; Based on the first number of grayscale images, a third number of second feature maps corresponding to the first gesture video are determined.
3. The method according to claim 2, characterized in that, The first feature map is a frame difference feature map. The step of determining a second number of the first feature maps corresponding to the first gesture video based on the first number of grayscale images includes: The first number of grayscale images are subjected to frame difference processing to determine the second number of first feature maps corresponding to the first gesture video. The frame difference processing includes: for any two adjacent grayscale images in the first number of grayscale images, the difference between the pixels of the later image and the pixels of the earlier image is calculated to determine the frame difference feature map corresponding to the two adjacent grayscale images. The frame difference feature map includes motion information between the two adjacent grayscale images.
4. The method according to claim 2, characterized in that, The second feature map is a hand detection bounding box feature map. The step of determining a third number of the second feature maps corresponding to the first gesture video based on the first number of grayscale images includes: A third number of single-channel feature maps are determined, wherein the single-channel feature map is of the first color and the single-channel feature map has the same size as the grayscale image; A third number of target grayscale images are selected from the first number of grayscale images and input into the pre-trained hand detection model. The position information of the hand region in each frame of the target grayscale image is determined, and the hand region of the second color is drawn in the single-channel feature map of each frame according to the position information of the hand region, so as to obtain the third number of hand detection box feature maps.
5. The method according to claim 2, characterized in that, The iterative training of the dynamic gesture detection network based on the first and second feature maps corresponding to each gesture video includes: The first convolutional layer is used to perform convolution processing on the first feature map corresponding to the first gesture video to determine the first convolution result. The second convolutional layer is used to perform convolution processing on the second feature map corresponding to the first gesture video to determine the second convolution result. The first convolution result and the second convolution result are channel-connected through the array connection layer to obtain the third convolution result; Based on the third convolution result, a gesture prediction result is determined for the first gesture video; Based on the gesture prediction results and the first annotation information, determine the loss function value corresponding to the dynamic gesture detection network; Update the network parameters of the dynamic gesture detection network in the direction of reducing the value of the loss function; Based on the other gesture videos among the multiple gesture videos, excluding the first gesture video, the updated dynamic gesture detection network is iteratively trained.
6. The method according to claim 5, characterized in that, The step of performing convolution processing on the first feature map corresponding to the first gesture video through the first convolution layer to determine the first convolution result includes: merging the second number of first feature maps to obtain first data, inputting the first data into the first convolution layer, and having the first convolution layer extract features from the first data through a convolution kernel of a preset size to determine the first convolution result; The step of performing convolution processing on the second feature map corresponding to the first gesture video through the second convolution layer to determine the second convolution result includes: merging the data of the third number of second feature maps to obtain second data, inputting the second data into the second convolution layer, and having the second convolution layer extract features from the second data through a convolution kernel of a preset size to determine the second convolution result.
7. The method according to claim 1, characterized in that, The first feature map includes: a frame difference feature map, an optical flow feature map, a HOG3D feature map, or a SIFT3D feature map; The second feature map includes: a hand detection box feature map, a region of interest feature map learned through a self-attention mechanism, or a hand region feature map extracted through image segmentation.
8. A dynamic gesture detection method, characterized in that, include: Acquire the video to be detected, and extract a first number of frame images from the video to be detected; Based on the first number of frame images, a second number of first feature maps and a third number of second feature maps are determined, wherein the first feature map is used to characterize: motion information between two adjacent frame images in the video to be detected, or hand contour information of an image in the video to be detected; the second feature map is used to characterize: hand region position information of an image in the video to be detected; The second number of first feature maps and the third number of second feature maps are input into a pre-trained dynamic gesture detection network. The dynamic gesture detection network outputs the target dynamic gesture type corresponding to the video to be detected. The dynamic gesture detection network includes: a first convolutional layer for convolutional processing of the first feature maps, a second convolutional layer for convolutional processing of the second feature maps, and an array concatenation layer. The dynamic gesture detection network is obtained through iterative training, and the iterative training includes a first training stage and a second training stage. The first training stage occurs earlier than the second training stage. In the first training stage, the learning rate for the first convolutional layer is a first learning rate, and the learning rate for the second convolutional layer is 0. In the second training stage, the learning rate for the first convolutional layer is the first learning rate, and the learning rate for the second convolutional layer is the second learning rate.
9. A dynamic gesture detection network training device, characterized in that, include: The acquisition module is used to acquire multiple gesture videos, where each gesture video corresponds to a dynamic gesture; The determining module is used to perform image processing on each gesture video to determine a first feature map and a second feature map corresponding to each gesture video; wherein, the first feature map corresponding to any gesture video is used to characterize: motion information between two adjacent frames in any gesture video, or hand contour information in any gesture video; the second feature map corresponding to any gesture video is used to characterize: hand region position information in any gesture video. The training module is used to iteratively train the dynamic gesture detection network based on the first feature map and the second feature map corresponding to each gesture video to obtain the target detection network. The dynamic gesture detection network includes: a first convolutional layer for convolutional processing of the first feature map, a second convolutional layer for convolutional processing of the second feature map, and an array concatenation layer. The iterative training includes a first training phase and a second training phase. The first training phase occurs earlier than the second training phase. In the first training phase, the learning rate for the first convolutional layer is a first learning rate, and the learning rate for the second convolutional layer is 0. In the second training phase, the learning rate for the first convolutional layer is the first learning rate, and the learning rate for the second convolutional layer is a second learning rate.
10. An electronic device, characterized in that, include: A memory and a processor, wherein the memory stores a computer program, and the processor is configured to execute, via the computer program, the dynamic gesture detection network training method of any one of claims 1 to 7 or the dynamic gesture detection method of claim 8.