Training method and device, gesture recognition method, electronic device, and storage medium
By enhancing the lightweight gesture detection network and using a joint training method with shared parameters, the shortcomings of lightweight gesture recognition networks in terms of training accuracy and efficiency are addressed, enabling efficient deployment and improved accuracy on resource-constrained devices.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- 伟光有限公司(CN)
- Filing Date
- 2022-12-01
- Publication Date
- 2026-05-15
AI Technical Summary
Existing lightweight gesture recognition networks are insufficient in terms of training accuracy and efficiency, and are particularly difficult to deploy effectively in resource-constrained electronic devices.
By enhancing the lightweight gesture detection network, a second gesture detection network is constructed, which shares model parameters with the first gesture detection network. The network parameters are then jointly trained using an expanded training dataset and the enhanced network as auxiliary supervision signals to optimize the network parameters.
While reducing computing resources and time costs, it significantly improves the accuracy and learning representation capabilities of lightweight gesture recognition networks, making them suitable for deployment on resource-constrained electronic devices.
Smart Images
Figure CN116189277B_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of gesture recognition, and more specifically, to a training method and apparatus, a gesture recognition method, an electronic device, and a storage medium. Background Technology
[0002] Lightweight gesture recognition networks can be applied to electronic devices such as mobile phones to recognize user gestures. The training accuracy and efficiency of lightweight gesture recognition networks have always been issues of concern. Summary of the Invention
[0003] This application provides a training method and apparatus, a gesture recognition method, an electronic device, and a storage medium. The various aspects involved in this application are described below.
[0004] In a first aspect, a training method is provided, comprising: enhancing a first gesture detection network to obtain a second gesture detection network; training the first gesture detection network and the second gesture detection network using a training dataset formed from gesture images to obtain a first loss corresponding to the first gesture detection network and a second loss corresponding to the second gesture detection network; and updating the parameters of the first gesture detection network and / or the second gesture detection network according to the first loss and the second loss.
[0005] In conjunction with the first aspect, in some implementations, the second gesture detection network shares the model parameters of the first gesture detection network with the first gesture detection network.
[0006] In conjunction with the first aspect, in some implementations, the second gesture detection network is one of a plurality of second gesture detection networks obtained by enhancing the first gesture detection network. The method further includes: for each round of training of the first gesture detection network, selecting one second gesture detection network from the plurality of second gesture detection networks to participate in each round of training.
[0007] In conjunction with the first aspect, in some implementations, enhancing the first gesture detection network includes increasing the number of channels and / or neural network layers in the first gesture detection network.
[0008] In conjunction with the first aspect, in some implementations, the training dataset includes one or more of the following types of gesture images: gesture images of different scenes; gesture images of different lighting conditions; gesture images of different distances from the camera lens; and gesture images with and without gloves.
[0009] In a second aspect, a gesture recognition method is provided, comprising: acquiring a gesture to be recognized; and recognizing the gesture using a first gesture detection network; wherein the first gesture detection network is trained based on the method described in the first aspect or any implementation thereof.
[0010] Thirdly, a training device is provided, comprising: an enhancement module for enhancing a first gesture detection network to obtain a second gesture detection network; a training module for training the first gesture detection network and the second gesture detection network using a training dataset formed from gesture images to obtain a first loss corresponding to the first gesture detection network and a second loss corresponding to the second gesture detection network; and an update module for updating the parameters of the first gesture detection network according to the first loss and the second loss.
[0011] Fourthly, a training apparatus is provided, comprising: a memory for storing code; and a processor for executing the code stored in the memory, such that the training apparatus performs the method as described in the first aspect or any implementation thereof.
[0012] Fifthly, an electronic device is provided, comprising: a memory for storing code; and a processor for executing the gesture recognition method described in the second aspect.
[0013] A sixth aspect provides a computer-readable storage medium having code stored thereon for performing the method as described in any possible implementation of the first or second aspect.
[0014] A seventh aspect provides a computer program product, including code for performing the method as described in any possible implementation of the first or second aspect.
[0015] By introducing a second gesture detection network (enhanced network) and supervising the training process of the first gesture detection network (lightweight network) based on the second gesture detection network, the accuracy of the first gesture detection network can be improved. Furthermore, since the first and second gesture detection networks are trained simultaneously, training overhead can be reduced. Attached Figure Description
[0016] To more clearly illustrate the technical solutions in the embodiments of this application or the background art, the accompanying drawings used in the embodiments of this application or the background art will be described below.
[0017] Figure 1 This is a flowchart illustrating a training method for a lightweight gesture detection network.
[0018] Figure 2This is a flowchart illustrating another training method for lightweight gesture detection networks.
[0019] Figure 3 This is a flowchart illustrating the training method provided in the embodiments of this application.
[0020] Figure 4 This is an example diagram of the network enhancement method provided in the embodiments of this application.
[0021] Figure 5 This is an example image of a gesture captured in an embodiment of this application.
[0022] Figure 6 This is a flowchart illustrating the gesture recognition method provided in the embodiments of this application.
[0023] Figure 7 This is a schematic diagram of the structure of a training device provided in one embodiment of this application.
[0024] Figure 8 This is a schematic diagram of the structure of a training device provided in another embodiment of this application.
[0025] Figure 9 This is a schematic diagram of the structure of an electronic device provided in one embodiment of this application. Detailed Implementation
[0026] With the development of neural network technology, gesture recognition technology based on gesture recognition networks is being applied more and more widely. For example, some electronic devices (such as mobile phones) have gesture recognition networks deployed inside, enabling these devices to have gesture recognition capabilities. For instance, based on gesture recognition networks, users can control their phones to answer and hang up calls, like short videos, and turn pages using gestures.
[0027] Many electronic devices, limited by their own resource configurations, often require the deployment of lightweight gesture recognition networks. Therefore, how to train lightweight gesture recognition networks has become a hot research topic.
[0028] Related Technology 1 proposes a gesture detection method based on knowledge distillation and attention mechanisms. This method first acquires gesture images to construct a training dataset and then builds a teacher network and a lightweight student network based on an attention mechanism. Next, it trains a large teacher network using the constructed dataset. Then, it expands the dataset through manual annotation, automatically generated random datasets, and prediction of unlabeled data by the teacher network. Finally, it uses the expanded dataset and the teacher network together to distill and train the student network. The resulting lightweight gesture detection network is then used for prediction. The implementation process of Related Technology 1 can be found in [link to relevant documentation]. Figure 1 .
[0029] Related technology 2 proposes a gesture detection method based on data augmentation and a lightweight backbone network. The implementation process of related technology 2 is as follows: Figure 2 As shown, firstly, a lightweight gesture detection network is constructed based on SqueezeNet as the backbone; gesture images and background images are acquired, and then data augmentation is performed to obtain a training dataset; then, the lightweight gesture detection network is trained based on the training dataset to obtain the gesture detection network for prediction.
[0030] Both related techniques 1 and 2 aim to obtain a lightweight, deployable gesture detection network. Related technique 2 directly uses a lightweight SqueezeNet as the backbone network and leverages data augmentation for network training. While data augmentation has been proven effective in improving model accuracy across numerous networks, it often proves ineffective on lightweight networks. This is because lightweight networks have weaker learning representation capabilities than larger models; data augmentation provides more and more complex data, which the lightweight network's representation capabilities are insufficient to learn the effective information within. The drawback of related technique 2 is its insufficient accuracy.
[0031] Related technique 1 introduces additional supervision through knowledge distillation, which can slightly alleviate the accuracy problem of related technique 2. However, related technique 1 requires training a large teacher network first, and then using teacher network distillation to assist in training the student network. The drawback of this approach is that it requires two training sessions, and training a large model is more time-consuming. Although related technique 1 solves the accuracy problem of related technique 2 to some extent, it is time-consuming and costly to train.
[0032] To address the aforementioned problems, this application proposes a training method, which is described below in conjunction with... Figure 3 This section provides a detailed introduction to the training method.
[0033] See Figure 3 In step S310, the first gesture detection network is enhanced to obtain the second gesture detection network.
[0034] The first gesture detection network can be a lightweight gesture detection network. For example, it can be a gesture detection network that can be deployed on handheld terminal devices such as mobile phones. The second gesture detection network can also be called an enhancement network. Enhancement can be understood as the second gesture detection network having more model parameters than the first gesture detection network. In some embodiments below, the first gesture detection network will be referred to as the base network, and the second gesture detection network as the enhancement network.
[0035] The embodiments of this application do not specifically limit the enhancement method of the first gesture detection network. The first gesture detection network can be enhanced from the width direction, or from the depth direction, or a combination of the above two enhancement methods.
[0036] Enhancing the first gesture detection network in terms of depth can include increasing the number of neural network layers in the first gesture detection network. For example, if the first gesture detection network has N neural network layers, M neural network layers can be added to the first gesture detection network to obtain a second gesture recognition network containing N+M neural network layers.
[0037] Enhancing the first gesture detection network in the width direction can include increasing the number of channels in the first gesture detection network. For example, see... Figure 4 The first gesture detection network may include one or more convolutional layers, and the network can be widened by increasing the number of convolutional channels. Compared to enhancing the first gesture detection network in the depth direction, enhancing the first gesture detection network in the width direction results in a second gesture recognition network with lower training time overhead.
[0038] In some embodiments, the second gesture detection network is one of a plurality of second gesture detection networks obtained by enhancing the first gesture detection network. That is, after enhancing the first gesture detection network in step S310, one or more second gesture detection networks can be obtained.
[0039] For example, we can first construct the largest possible augmented network. Assume the original convolution operator's width, i.e., the number of output channels, is w. We can assign an augmentation factor r to it, then the augmented convolution operator's width will be w*r. To simplify the operation, we can avoid setting the augmentation factor r layer by layer; the entire network can share the same augmentation factor r.
[0040] After constructing the largest augmentation network, other augmentation networks can be obtained by sampling the output channels from the largest augmentation network. For example, another hyperparameter s can be set to linearly divide it. For instance, if the width of the base network (i.e., the first gesture detection network mentioned earlier) is w, and the width of the largest augmentation network is w*r, we can set r=3 and s=2, then the possible augmentation network widths are [w, 2*w, 3*w]. Different layers can also have different splitting factors s, but to simplify the operation, the augmentation factor is not set layer by layer; the entire network shares the same splitting factor s.
[0041] In some embodiments, the second gesture detection network and the first gesture detection network can share the model parameters of the first gesture detection network. Sharing the model parameters of the first gesture detection network can be understood as the second gesture detection network containing the first gesture detection network. Alternatively, the first gesture detection network can be obtained by sampling the second gesture detection network. If multiple second gesture detection networks are constructed in step S310, these multiple second gesture detection networks can share model parameters with each other. For example, after constructing the largest augmentation network, other gesture detection networks can be obtained by sampling from that largest augmentation network. Since the first and second gesture detection networks share model parameters, the total storage amount of model parameters is determined by the model parameters of the largest augmentation network. Compared with storing the model parameters of each gesture detection network independently, this not only reduces computational resources and memory overhead but also reduces training time. If the shared model parameters method is used, after training is completed, the final lightweight gesture detection network can be obtained by sampling the augmentation network.
[0042] In execution Figure 1 Before step S320, a training dataset of gesture images can be obtained. This training dataset may contain one or more of the following types of gesture images: gesture images from different scenes; gesture images under different lighting conditions; and gesture images at different distances from the camera lens. Figure 5 Examples of gesture images are provided. Figure 5 The training dataset contains various gesture images from different scenes and at different distances. In some embodiments, to handle complex gesture recognition scenarios, the training dataset may also include gesture images with and without gloves. After acquiring the gesture images, they can be labeled to obtain the final training dataset.
[0043] Next, in steps S320-S330, a first gesture detection network and a second gesture detection network are trained using the training dataset formed from gesture images, resulting in a first loss for the first gesture detection network and a second loss for the second gesture detection network. Based on the first and second losses, the parameters of the first gesture detection network and / or the second gesture detection network are updated. In other words, the second gesture detection network (enhancement network) can be used as an additional supervision signal to train the first gesture detection network (base network). This supervision can be achieved by considering the loss of the enhancement network when updating the base network.
[0044] As mentioned earlier, the number of augmentation networks can be one or more. Taking multiple augmentation networks as an example, the loss function of the base network can be expressed as follows:
[0045] Laug =L(W t )+a1*L([W t ,W1])+a2*L([W t ,W2])+…+a i *L([W t W i ])
[0046] Among them, [W t W i ] represents an enhanced network that includes the basic network, W t It's the basic part, W i This is the expanded part. Among them, L(W) t ) is the basic supervision, a1*L([W t ,W1])+a2*L([W t ,W2])+…+a i *L([W t W i ]) is an auxiliary supervisory component of several enhanced networks. a i It is a hyperparameter used to control the proportion of different augmentation network supervision signals in the overall loss function.
[0047] In cases involving multiple augmentation networks, each augmentation network requires an additional forward computation and backpropagation process. Calculating the loss function of all augmentation networks at each training step would also increase computational resource overhead and time cost. Therefore, in some embodiments, only one augmentation network is randomly sampled as an auxiliary supervision signal for forward computation and backpropagation at each training step. Thus, the update of the basic network parameters can be expressed as follows:
[0048]
[0049] in, This indicates the updated basic network. This represents the base network before the update, and `lr` represents the learning rate. This parameter can be a hyperparameter, and its value can be preset to a fixed value. `a` can also be a hyperparameter to control the proportion of different augmentation network supervision signals in the overall loss function.
[0050] This application proposes a lightweight gesture detection network (i.e., the first gesture detection network mentioned above) training method. This lightweight gesture detection network features fast inference speed, small model size, and easy deployment on mobile devices. Furthermore, the network augmentation technique proposed in this application introduces an augmentation network as an auxiliary supervision signal while maintaining the network's lightweight nature, effectively improving the learning representation ability of the lightweight network and ensuring its accuracy. In this network augmentation technique, the base network (first gesture detection network) and the augmentation network (second gesture detection network) are trained together, requiring only one training iteration, saving computational resources and time costs.
[0051] This invention first enhances the base network by extending the network to obtain several enhanced networks with stronger learning representation capabilities. Next, the enhanced networks are used collaboratively for training, serving as auxiliary supervisory signals to update the parameters of the slightly weaker base network. In this way, a lightweight base model with stronger representation capabilities can be learned in a shorter training time.
[0052] Figure 6 This is a flowchart illustrating the gesture recognition method provided in the embodiments of this application. Figure 6 The gesture recognition method shown can be performed based on the first gesture detection network trained by the training method described above. Figure 6 The method includes steps S610 and S620.
[0053] In step S610, the gesture to be recognized is acquired. For example, the gesture to be recognized can be captured by the camera on a mobile phone.
[0054] In step S620, the first gesture detection network is used to recognize the gesture.
[0055] The above text combined Figures 1-6 The method embodiments of this application are described in detail below, in conjunction with... Figure 7 and Figure 9 The present application provides a detailed description of the apparatus embodiments. It should be understood that the descriptions of the apparatus embodiments correspond to the descriptions of the method embodiments; therefore, any parts not described in detail can be found in the foregoing method embodiments.
[0056] Figure 7 This is a schematic diagram of the structure of a training device provided in one embodiment of this application. Figure 7 The training device 700 shown may include an enhancement module 710, a training module 720, and an update module 730.
[0057] The enhancement module 710 can be used to enhance the first gesture detection network to obtain the second gesture detection network.
[0058] The training module 720 can be used to train the first gesture detection network and the second gesture detection network using the training dataset formed by gesture images, so as to obtain the first loss corresponding to the first gesture detection network and the second loss corresponding to the second gesture detection network.
[0059] The update module 730 can be used to update the parameters of the first gesture detection network based on the first loss and the second loss.
[0060] Figure 8 This is a schematic diagram of the structure of a training device provided in another embodiment of this application. Figure 8 The training device 800 may include a memory 810 and a processor 820.
[0061] The memory 810 can be used to store code.
[0062] The processor 820 can be used to execute code stored in memory, causing the training device 800 to perform the training method described in any of the preceding embodiments.
[0063] Figure 9 This is a schematic diagram of the structure of an electronic device provided in one embodiment of this application. Figure 9 The illustrated electronic device 900 may be, for example, a mobile phone. This electronic device may include a memory 910 and a processor 920.
[0064] Memory 910 can be used to store code.
[0065] Processor 920 can be used to execute Figure 6 The gesture recognition method shown.
[0066] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any other combination. When implemented in software, it can be implemented, in whole or in part, as a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this disclosure are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another via wired (e.g., coaxial cable, fiber optic, Digital Subscriber Line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium accessible to a computer or a data storage device such as a server or data center that integrates one or more available media. The available media can be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., digital video discs (DVDs)), or semiconductor media (e.g., solid-state drives (SSDs)).
[0067] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments of this disclosure can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this disclosure.
[0068] In the several embodiments provided in this disclosure, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.
[0069] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.
[0070] In addition, the functional units in the various embodiments of this disclosure can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.
[0071] The above description is merely a specific embodiment of this disclosure, but the scope of protection of this disclosure is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this disclosure should be included within the scope of protection of this disclosure. Therefore, the scope of protection of this disclosure should be determined by the scope of the claims.
Claims
1. A training method, characterized in that, include: The first gesture detection network is enhanced to obtain the second gesture detection network; The first gesture detection network and the second gesture detection network are trained using a training dataset formed from gesture images, and the first loss corresponding to the first gesture detection network and the second loss corresponding to the second gesture detection network are obtained. Based on the first loss and the second loss, update the parameters of the first gesture detection network and / or the second gesture detection network; The second gesture detection network shares all the model parameters of the first gesture detection network with the first gesture detection network.
2. The training method according to claim 1, characterized in that, The second gesture detection network is one of a plurality of second gesture detection networks obtained by enhancing the first gesture detection network, and the method further includes: For each round of training of the first gesture detection network, one second gesture detection network is selected from the plurality of second gesture detection networks to participate in each round of training.
3. The training method according to claim 1, characterized in that, The enhancement of the first gesture detection network includes: Increase the number of channels and / or neural network layers in the first gesture detection network.
4. The training method according to claim 1, characterized in that, The training dataset contains one or more of the following types of gesture images: Gesture images in different scenarios; Images of hand gestures under different lighting conditions; Images of hand gestures at different distances from the camera lens; and Images of hand gestures with and without gloves.
5. A gesture recognition method, characterized in that, include: Obtain the gesture to be recognized; The gesture is identified using a first gesture detection network; wherein the first gesture detection network is trained based on the method described in any one of claims 1-4.
6. A training device, characterized in that, include: An enhancement module is used to enhance the first gesture detection network to obtain a second gesture detection network; The training module is used to train the first gesture detection network and the second gesture detection network using a training dataset formed from gesture images, and to obtain the first loss corresponding to the first gesture detection network and the second loss corresponding to the second gesture detection network. An update module is used to update the parameters of the first gesture detection network based on the first loss and the second loss; The second gesture detection network shares all the model parameters of the first gesture detection network with the first gesture detection network.
7. A training device, characterized in that, include: Memory, used to store code; A processor for executing code stored in the memory, causing the training device to perform the method as described in any one of claims 1-4.
8. An electronic device, characterized in that, include: Memory, used to store code; A processor for executing the gesture recognition method of claim 5.
9. A computer-readable storage medium, characterized in that, It stores code that is used to perform the method as described in any one of claims 1-4 or claim 5.