Hand posture recognition method and device, electronic equipment and storage medium

Abstract

The present disclosure provides a hand posture recognition method, device, electronic equipment and storage medium. The hand posture recognition method comprises: acquiring a target hand image and target electromyography data corresponding to the target hand image; acquiring a time feature of the target electromyography data and a space feature of the target hand image; fusing the time feature and the space feature to obtain a fused feature; and determining posture information of a target hand in the target hand image according to the fused feature. The hand posture recognition method provided by the present disclosure combines the space feature and the time feature, and improves the accuracy of recognizing the posture information of the target hand in poor lighting conditions and when there is hand self-occlusion.

Description

Technical Field

[0001] This disclosure relates to the field of computer technology, and in particular to a hand gesture recognition method, apparatus, electronic device, and storage medium. Background Technology

[0002] Recognizing a user's hand gestures is an important aspect of human-computer interaction fields such as virtual reality, augmented reality, and extended reality. However, existing methods often suffer from reduced recognition accuracy in situations involving hand occlusion or poor lighting. Summary of the Invention

[0003] This disclosure provides a hand gesture recognition method, apparatus, electronic device, and storage medium.

[0004] The following technical solution is adopted in this disclosure.

[0005] In some embodiments, this disclosure provides a hand gesture recognition method, including:

[0006] Acquire a target hand image and the corresponding target electromyographic data;

[0007] Acquire the temporal features of the target electromyography data and the spatial features of the target hand image;

[0008] The temporal features and spatial features are fused to obtain the fused features;

[0009] The pose information of the target hand in the target hand image is determined based on the fusion features.

[0010] In some embodiments, this disclosure provides a hand gesture recognition device, including:

[0011] An acquisition unit is used to acquire a target hand image and the target electromyographic data corresponding to the target hand image;

[0012] A control unit is used to acquire the temporal characteristics of the target electromyography data and the spatial characteristics of the target hand image;

[0013] The control unit is further configured to fuse the temporal features and the spatial features to obtain fused features;

[0014] The control unit is further configured to determine the pose information of the target hand in the target hand image based on the fusion features.

[0015] In some embodiments, this disclosure provides an electronic device, including: at least one memory and at least one processor;

[0016] The memory is used to store program code, and the processor is used to call the program code stored in the memory to execute the above method.

[0017] In some embodiments, this disclosure provides a computer-readable storage medium for storing program code that, when run by a processor, causes the processor to perform the methods described above.

[0018] The hand posture recognition method provided in this disclosure combines spatial and temporal features, improving the accuracy of recognizing the posture information of the target hand under poor lighting conditions and when the hand is self-occluded. Attached Figure Description

[0019] The above and other features, advantages, and aspects of the embodiments of this disclosure will become more apparent from the accompanying drawings and the following detailed description. Throughout the drawings, the same or similar reference numerals denote the same or similar elements. It should be understood that the drawings are schematic, and elements are not necessarily drawn to scale.

[0020] Figure 1 This is a flowchart of a hand gesture recognition method according to an embodiment of the present disclosure.

[0021] Figure 2 This is a schematic diagram of hand gesture recognition according to an embodiment of the present disclosure.

[0022] Figure 3 This is a schematic diagram illustrating the generation of training hand images according to an embodiment of this disclosure.

[0023] Figure 4 This is a schematic diagram of the structure of an electronic device according to an embodiment of the present disclosure. Detailed Implementation

[0024] It is understood that before using the technical solutions disclosed in the various embodiments of this disclosure, users should be informed of the types, scope of use, and usage scenarios of the personal information involved in this disclosure in an appropriate manner in accordance with relevant laws and regulations, and user authorization should be obtained.

[0025] For example, upon receiving a user's active request, a prompt message is sent to the user to explicitly inform them that the requested operation will require the acquisition and use of the user's personal information. This allows the user to independently choose whether to provide personal information to the software or hardware, such as the electronic device, application, server, or storage medium performing the operations of this disclosed technical solution, based on the prompt message.

[0026] As an optional but non-limiting implementation, in response to a user's active request, sending a prompt message to the user can be done via a pop-up window, where the prompt message can be presented in text format. Furthermore, the pop-up window can also include a selection control allowing the user to choose "agree" or "disagree" to provide personal information to the electronic device.

[0027] It is understood that the above notification and user authorization process are merely illustrative and do not constitute a limitation on the implementation of this disclosure. Other methods that comply with relevant laws and regulations may also be applied to the implementation of this disclosure.

[0028] It is understood that the data involved in this technical solution (including but not limited to the data itself, the acquisition or use of the data) shall comply with the requirements of relevant laws, regulations and related provisions.

[0029] Embodiments of this disclosure will now be described in more detail with reference to the accompanying drawings. While some embodiments of this disclosure are shown in the drawings, it should be understood that this disclosure can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of this disclosure. It should be understood that the accompanying drawings and embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of protection of this disclosure.

[0030] It should be understood that the various steps described in the method embodiments of this disclosure can be performed in sequence and / or in parallel. Furthermore, method embodiments may include additional steps and / or omit the steps shown. The scope of this disclosure is not limited in this respect.

[0031] The term "comprising" and its variations as used herein are open-ended inclusions, meaning "including but not limited to". The term "based on" means "at least partially based on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments". Definitions of other terms will be given in the description below.

[0032] It should be noted that the concepts of "first" and "second" mentioned in this disclosure are used only to distinguish different devices, modules or units, and are not used to limit the order of functions performed by these devices, modules or units or their interdependencies.

[0033] It should be noted that the use of the word "a" in this disclosure is illustrative rather than restrictive, and those skilled in the art should understand that it should be understood as "one or more" unless otherwise expressly indicated in the context.

[0034] The names of messages or information exchanged between multiple devices in the embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of such messages or information.

[0035] In extended reality fields such as virtual reality and augmented reality, images of users' hands are typically captured, and then hand poses are identified. Under good lighting conditions, hand pose recognition is usually quite good. However, in poor lighting conditions (e.g., low light) or when the hand is occluded (part of the hand's pose is obscured by other parts of the hand, such as the index finger and thumb obscuring the little finger), accurate extraction of hand features becomes difficult, leading to a decline in hand pose recognition performance. Methods based on surface electromyography (sEMG) data can provide good temporal information by measuring muscle electrical signals under extreme lighting conditions. However, the relationship between muscle activation and sEMG is complex, resulting in a low signal-to-noise ratio. Therefore, methods using this approach to determine hand pose lack cross-user and behavioral universality, and their spatial accuracy is also unsatisfactory.

[0036] The solutions provided by the embodiments of this disclosure will be described in detail below with reference to the accompanying drawings.

[0037] like Figure 1 As shown, Figure 1 This is a flowchart of a hand gesture recognition method according to an embodiment of the present disclosure, which includes the following steps.

[0038] S11. Obtain the target hand image and the corresponding electromyographic data of the target hand image.

[0039] In some embodiments, the method proposed in this disclosure can be used in an electronic device, specifically in a target model within the electronic device. The electronic device can be an extended reality device such as virtual reality, augmented reality, or mixed reality, and the target hand image is an image of the target hand. The target hand image can be an image of the hand of a user currently using the electronic device; for example, a camera on the electronic device captures a video of the user's hand, and the hand image is extracted from the video as the target hand image. The target electromyography (EMG) data can be the EMG data of the user currently using the electronic device; specifically, it can be sEMG data. For example,... Figure 2As shown on the left, the visual stream represents a continuously captured video stream. The fourth image from the left on the right side of the visual stream (the image to the right of L) is the target hand image. In the electromyography (EMG) stream, the area covered by the arrows on both sides of L represents the target EMG data, which is the previous image of the target hand captured from the video stream. Figure 2 The electromyography (EMG) data of the user's hand during the period from the third image from the left on the right side of the visual stream to the capture of the target hand image is a time-dependent EMG data sequence. In some embodiments, the target hand image may be an image with poor lighting conditions or with self-occlusion of the hand.

[0040] S12. Acquire the temporal characteristics of the target electromyography data and the spatial characteristics of the target hand image.

[0041] In some embodiments, both temporal and spatial features can be represented by vectors. The target electromyography (EMG) data reflects the EMG data generated when the hand posture changes from the previous hand image to the target hand image. This data is time-dependent, and time-dependent feature data is extracted as temporal features. Temporal features characterize the changes in EMG data over time due to changes in hand posture. The target hand image represents the spatial posture of the hand, and spatially dependent feature data is extracted from it. The extraction of temporal features can be performed using an encoder, and the extraction of spatial features can be performed using another encoder. Both encoders are encoders in the target model, which can be derived from the Transformer model based on a self-attention mechanism proposed by Google. The encoder of the target model can be obtained by training the encoder in the Transformer model and then selecting appropriate parameters as the encoder of the target model.

[0042] S13. The temporal and spatial features are fused to obtain the fused features.

[0043] In some embodiments, fusing temporal and spatial features allows the target model implementing the method to dynamically focus on more informative features based on the input conditions (target hand image and target EMG data). For example, if the target hand image contains more information, more attention is paid to the target hand image; if the target EMG data contains more information, more attention is paid to the target EMG data.

[0044] S14. Determine the pose information of the target hand in the target hand image based on the fusion features.

[0045] In some embodiments, after obtaining the fused features, the pose information of the target hand is obtained. The pose information of the target hand can be represented by the degrees of freedom of the joints in the hand. The pose information of the target hand can be determined by decoding the fused features into the decoder of the target model.

[0046] In some embodiments of this disclosure, the pose information of the target hand is determined by combining the target hand image and the target electromyography data corresponding to the target hand image. This achieves the combination of spatial modality and EMG modality. The multimodal end-to-end target model improves the accuracy of identifying the pose information of the target hand under extreme lighting conditions and when there is hand self-occlusion, and has better versatility across users.

[0047] In some embodiments of this disclosure, obtaining the temporal features of the target electromyography data includes: segmenting the target hand image into multiple non-overlapping image blocks; projecting each image block to obtain spatial sub-features of each image block; and inputting the spatial sub-features into the encoder of a spatial converter, wherein the encoder of the spatial converter generates spatial features of the target hand image based on the spatial sub-features.

[0048] In some embodiments, such as Figure 2 As shown on the far left, the visual stream represents a continuously captured video stream, in which the target hand image I is selected. t ∈R H×W t represents the time corresponding to the image, H and W represent the length and width dimensions of the image, R represents the set of real numbers, and the target hand image I... t A matrix of real numbers with dimensions H×W. The target hand image can be a grayscale image. Processing of the target hand image is performed by the vision branch in the target model, first segmenting the input target hand image (…). Figure 2 The `Patchify` function in Python obtains multiple non-overlapping image patches, which can be achieved by applying the `Patchify` function from the Python library to these image patches as spatial sub-features V. t In some embodiments, each of the image blocks is projected ( Figure 2 Linear Projection (in the image) yields the spatial sub-features V of each image patch. t This includes: projecting the image patch, and using the projection result plus position embedding to obtain the spatial sub-feature V. t The projection method can be linear projection, and the specific process can be represented by the following equation:

[0049]

[0050] Where, N p=HW / P 2 is the number of image patches; P is the side length of the image patch, the size of the image patch is P×P, and it has 3 color channels; x p These are image patches segmented by Patchify, with a shape of N×P. 2 A 3×3 matrix, where each row represents the feature vector of an image patch. D is the feature dimension of each spatial sub-feature, and E... pos It is a learnable position embedding used to encode spatial information, while Linear is a linear projection. V t It is a spatial sub-feature with shape N p ×D; These spatial sub-features serve as input to the encoder of the spatial transformer (specifically, the encoder of the Transformer model) for further extraction of global features and spatial relationships. In this embodiment, the main purpose of using positional embedding is to provide the model with information about the spatial location of image patches within the target hand image. The Transformer model itself is based on a self-attention mechanism and lacks inherent spatial awareness; that is, by default, the Transformer model cannot understand the relative positional relationships between elements in the input sequence. Therefore, when processing data like images, positional information needs to be explicitly introduced to help the model capture these spatial relationships. By adding positional embedding, the spatial structure information of image patches can be preserved, enabling the model to understand which image patches are close to each other and which are far apart. This is very effective for capturing both local details and global context. Adding the linearly projected feature vector to the positional embedding enhances the model's understanding of image patches. Positional embedding provides additional contextual information for each image patch, which helps the model better learn the overall structure and local details of the image. In different scenarios (such as low-light scenes, occluded scenes, etc.), positional information can help the model better understand and process image content, thereby improving its generalization ability.

[0051] In some embodiments of this disclosure, the encoder of the spatial converter generates spatial features of the target hand image based on the spatial sub-features, including: extracting features from the spatial sub-features through a multi-head self-attention mechanism and a feedforward neural network, capturing local details and global spatial relationships of the spatial sub-features to generate the spatial features.

[0052] In some embodiments, an encoder using a Transformer model responsible for processing spatial features may be used. Figure 2 The Spatial Transformer Encoder (STE) in the code processes spatial sub-features, specifically through multiple Transformer layers. tspatial features F are obtained v The processing procedure can be specifically represented as follows:

[0053] h v =MHSA(V t )

[0054] F v =FFN(h v )

[0055] in:

[0056] MHSA(X)=Concat(head1,...,head h W O

[0057]

[0058]

[0059] Here, MHSA stands for Multi-Head Self-Attention Mechanism, which is used to calculate the similarity between each spatial sub-feature and other spatial sub-features to generate a new representation. h is the number of attention heads, and W... O It is the weight matrix of the output linear transformation, where X is the input, for example, V. t W i Q W i K and W i V Let XW be the learnable projection matrix corresponding to Q (query), K (key), and V (value) of the i-th head. i Q XW i K and XW i V This represents the Q (query), K (key), and V (value) of input X. k It is the dimension of K. Specifically, it can be that the spatial sub-features V are first... t The linear transformation yields Q (query), K (key), and V (value), where Q = V. t W Q K = V t W K V = V t W VThen, the similarity between Q and K is calculated using a dot product, and the attention score is obtained through scaling and translation operations. To capture information in different subspaces, multiple attention heads are used, each of which independently calculates the attention score. The results are then concatenated and subjected to another linear transformation.

[0060] Through the above process, a multi-head self-attention mechanism is used to process spatial sub-features, and the processing result is then input into a feedforward network (FFN). The feedforward network is used to further process the representation generated by the self-attention mechanism. The feedforward network can be represented as:

[0061] FFN(X)=MLP(LN(X))+X

[0062] Here, LN(X) represents the result after layer normalization of the input X, and MLP is a two-layer perceptron including the GELU (Gaussian Error Linear Units) activation function. Through this design, FFN enhances feature representation while maintaining the stability of information flow, and layer normalization (LN) ensures the scale consistency of the input at each layer, promoting training stability and efficiency. MLP introduces nonlinear transformations, increasing the model's expressive power. Residual connections help solve the gradient vanishing problem in deep networks, making training smoother.

[0063] The spatial features obtained after MHSA and FFN processing not only include local details at the image patch level (such as finger pose and position), but also capture global spatial relationships between image patches through a self-attention mechanism. This is crucial for recognizing the pose of the entire hand, as finger position often depends on the position of the wrist and other fingers. The spatial features generated by the model through MHSA and FFN processing ( Figure 2 The Visual Token representation is richer and more robust, which helps in subsequent hand pose estimation.

[0064] In some embodiments of this disclosure, obtaining the temporal features of the target electromyography (EMG) data includes: performing feature projection on the target EMG data to obtain projection data; performing convolutional downsampling on the projection data to obtain EMG features; and inputting the EMG features into a temporal convolutional encoder, which generates the temporal features of the target EMG data based on the EMG features.

[0065] In some embodiments, target electromyographic data is obtained from an EMG sequence corresponding to a video sequence, which can be represented as x. e =Extract(E t )∈RL×C , of which E t Here, L represents the length of the EMG sequence, and C represents the number of channels in the EMG sequence (typically 16). Target electromyography (EMG) data x is extracted from the EMG sequence through feature extraction. e This yields a matrix x of shape L×C. e The acquired target electromyography (EMG) data consists of EMG data between two consecutive frames of the target hand image in the video sequence. Because EMG data has a high sampling rate (e.g., 2000Hz), relative to the image frame rate (typically 60Hz to 120Hz), each target hand image corresponds to a relatively long segment of EMG data. Therefore, feature projection is performed on the EMG data, followed by convolutional downsampling (…). Figure 2 Convolutional Downsampling in the context of Et′ is represented as Et′=ConvDown(Proj(x)). e ))∈R L′×D Where L′ is the downsampled length and D is the feature dimension, convolutional downsampling reduces the temporal resolution while preserving important muscle activation patterns. The final output is a matrix E of shape L′×D. t ′ as a myoelectronic feature. Myoelectronic features are input to a temporal convolutional encoder ( Figure 2 The Temporal Conformer Encoder (TCE) in this context can be the encoder of a Transformer model responsible for processing electromyographic features. The temporal convolutional encoder captures temporal dependencies and learns temporal features. Figure 2 (EMG Tokens).

[0066] In some embodiments of this disclosure, the temporal convolutional encoder generates temporal features of the target electromyography (EMG) data based on the EMG features, including: processing the EMG features through a self-attention mechanism and convolution to capture short-term and long-term temporal features in the target EMG data to generate the temporal features.

[0067] In some embodiments, the Temporal Convolutional Encoder (TCE) combines self-attention mechanisms and convolution operations to capture both short-term and long-term temporal features, which are then fused to obtain the final temporal features. The TCE workflow formula is as follows:

[0068] h e (1) =FFN(E t ′);h e (2) =MHSA(h e (1) );h e (3)=Conv(h e (2) );F e =FFN(h e (3) ).

[0069] The formulas for FFN and MHSA are the same as those used in generating spatial features as described above. After the above processing, the myoelectronic feature E... t ' Obtain the time feature F e Conv represents the convolution operation, which can use existing convolution formulas. In the formula above, the first feedforward neural network FFN(E) t ') is used to enhance feature representation, MHSA(h e (1) Conv(h) is used to capture feature relationships between different long-term time points through a multi-head self-attention mechanism. e (2) FFN(h) is used to capture feature relationships between short time points through convolutional layers. e (3) This is used to further process the convolution results and generate the final temporal feature F. e ( Figure 2 (EMG Tokens). In this embodiment, a self-attention mechanism (MHSA) is used to capture long-term temporal features and understand the complex relationships between different time points. Convolutional operations (Conv) are used to capture short-term temporal features, suitable for detecting changes within a short period of time. This combination allows TCE to capture both short-term temporal features (such as features of EMG data from transient muscle activity caused by rapid finger movements) and long-term temporal features (such as features of EMG data from changes in overall hand posture). This is very effective for hand posture estimation because it needs to consider both short-term and long-term muscle activity features simultaneously.

[0070] In some embodiments of this disclosure, fusing the temporal features and the spatial features to obtain fused features includes: enhancing the temporal features using a multi-head self-attention mechanism; fusing the temporal features and spatial features using a multi-head cross-attention mechanism; processing the output data of the multi-head cross-attention mechanism through a feedforward neural network with residual connections; and performing average pooling on the processing result of the feedforward neural network with residual connections to obtain fused features that fuse the temporal features and spatial features.

[0071] In some embodiments, Figure 2 The right side is Figure 2A detailed diagram of the Multi-modality Adaptive Fusion (MMAF) module in the middle section. MMAF is used to fuse temporal and spatial features to obtain fused features. Before using a multi-head self-attention mechanism to enhance the temporal features, the temporal and spatial features can be normalized to obtain the normalized temporal feature Fe. ’ and spatial features F v ’ Then, as Figure 2 As shown on the right, the time feature ( Figure 2 The EMG Token is input into MHSA, and the processing result is added to the temporal features for enhancement. Spatial features can also be enhanced. The enhancement process can be represented as follows:

[0072] H e =MHSA(F′) e )+F′ e

[0073] H v =MHSA(F′) v )+F′ v

[0074] H e For the enhanced temporal features, H v To enhance the spatial features, the two are fused using a multi-head cross-attention mechanism. The fusion process of the multi-head cross-attention mechanism can be represented as follows:

[0075] A=MHCA(H e H v )+H e

[0076] MHCA(X,Y)=Concat(head1,...,head h W O

[0077]

[0078] MHCA is similar to MHSA, but differs in that, to achieve the fusion of temporal and spatial features, MHCA uses one feature as the query (Q) and the other as the key (K) and value (V). Specifically, in this embodiment, spatial features can be used as K and V, and temporal features as Q; alternatively, spatial features can be used as Q, and temporal features as K and V. This helps capture the dependencies between different modalities. The output data after MHCA fusion is then fed to a feedforward neural network with residual connections for processing. Figure 2 Norm&FFN&Add) can be represented as:

[0079] F fused =FFN(LayerNorm(A))+A

[0080] FFN is a feedforward neural network, and its processing formula has been described in the embodiments above. The A added to the right of FFN represents the residual, i.e., the normalized output of MHCA is input into FFN for calculation and added to itself. The resulting value is the processing result F of the feedforward neural network with residual connections. fused Then, average pooling is performed on the processing results of the feedforward neural network with residual connections. Figure 2 In the context of Average Pooling, it can be represented as:

[0081]

[0082] Among them, there are multiple temporal and spatial features (specifically L′), therefore the obtained F fused The number of features is also multiple, where L′ is the length of the sequence of myoelectronic features obtained by convolutional downsampling of the projected data (L′ can be equal to the number of image patches), and it is related to F. fused The number of them is the same. F i fused It is the i-th F fused F final For fusion features ( Figure 2 The Temperal Spatial Aware Feature (TSPA) is a single feature vector that encapsulates both temporal and spatial features.

[0083] In some embodiments of this disclosure, determining the pose information of a target hand in the target hand image based on the fused features includes: inputting the fused features into a multilayer perceptron to convert the fused features into pose information of the target hand. In some embodiments, a multilayer perceptron (MLP) is used to convert the fused features F final This is converted into hand pose information θ. MLPs typically consist of multiple fully connected layers, with non-linear activation functions (such as ReLU or GELU) used between each layer. The final output is a vector image of shape R. K The vector θ is given by K, where K is the number of joint angles in the target hand. θ represents the predicted hand joint angles, indicating the hand's pose information, which includes the angles of multiple hand joints.

[0084] In some embodiments of this disclosure, the method is used for a target model; the target model can be an end-to-end model that receives a target hand image and target electromyography (EMG) data and outputs pose information of the target hand. Before acquiring the target hand image and the corresponding EMG data, the method further includes: pre-training the target model using a training dataset to enable the target model, wherein the training dataset includes: training hand images with different hand shapes, different hand gestures, different lighting conditions, and / or different hand self-occlusions, corresponding training EMG data, and known hand pose parameters.

[0085] In some embodiments, the target model needs to be trained before use. During training, a training dataset is used, containing hand images (training hand images) and corresponding electromyographic (EMG) data. The hand pose information of the hand images in the training dataset is known, representing known hand pose parameters. During training, mean squared error can be used as the loss function. The Euclidean distance between the hand pose information predicted by the target model based on the training hand images and EMG data and the known hand pose parameters is calculated. The parameters of the target model are then adjusted to minimize the loss function.

[0086] In some embodiments of this disclosure, the training dataset is generated as follows: Known hand posture parameters and corresponding electromyographic (EMG) data are obtained; a hand skeleton map, a hand mask map, and a hand normal map are generated based on the known hand posture parameters; the hand skeleton map, hand mask map, and hand normal map are input into a local control adapter to generate structural features; a selected context image is obtained, and the context image is encoded into a feature vector; the feature vector is input into a global control adapter to generate features for controlling illumination conditions; the structural features and the features for controlling illumination conditions are fused to generate the training hand image; wherein the EMG data corresponding to the known hand posture parameters is used as the training EMG data corresponding to the training hand image.

[0087] In some embodiments, Figure 3 The content corresponding to the Visual Generative Model shown on the left describes the process of generating the training dataset. Figure 3 The right half shows different hand shapes. Figure 3 (Right half, upper left), different hand gestures and postures) Figure 3 Right half (upper right), different lighting conditions ( Figure 3 (Right half, lower left) and different hand self-covering ( Figure 3 In the case of the lower right half of the right half ( Figure 3In any case of the right half, from top to bottom, are: electromyography (EMG) data, hand model, and generated training hand image. In this embodiment, the known hand pose parameters and the corresponding EMG data can be from the publicly available emg2pose dataset. This embodiment can obtain the training dataset by extending the emg2pose dataset. In some embodiments, the visual generation model in this embodiment is based on the ControlNet architecture and combines a global context image as control to better handle different lighting conditions. The visual generation model includes two branches: local control and global control. In local control, a 3D hand model is first generated using the known hand pose parameters from the emg2pose dataset. Given a camera viewpoint, the 3D joints and mesh of the 3D hand model are obtained through forward kinematics and projected onto the given camera viewpoint. Hand skeleton diagram ( Figure 3 The Skeleton in the image is generated based on projected 3D joints, and the hand mask image ( Figure 3 Mask in the middle) and hand normal map ( Figure 3 The "Normal" in the model is generated based on a projected mesh. A local control adapter is set up (…). Figure 3 The Local Control Adapter in this implementation borrows from HOIDiffusion (from "Hoidiffusion: Generating realistic 3D hand-object interaction data. In Proceedings of the EEE / CVF Conference on Computer Vision and Pattern Recognition, pages 8521–8531, 2024"), extracting structural features from the hand skeleton map, hand mask map, and hand normal map. For more precise control, different fingers are rendered with different colors in the hand mask map. The Local Control Adapter is connected to UNet (from "High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE / CVF conference on computer vision and pattern recognition, pages 10684–10695, 2022") through separate feature extraction, multi-scale fusion, and zero-convolution. In global control, the context image is the image input by the user. Unlike HOIDiffusion, which uses text prompts, this implementation utilizes a context image (…). Figure 3The ContextImage is used to control lighting conditions, allowing the generated training hand images to have different lighting conditions. In this embodiment, DINOv2 (from Dinov2: Learning robust visual features without supervision, 2023) is used as the image encoder, and its class labels are passed through a trainable global control adapter containing two feedforward networks. The image encoder extracts features from the context image to obtain feature vectors, which are then integrated into the UNet. In the UNet, the structural features and the features used to control lighting conditions are fused to generate the training hand images. In this embodiment, the hand shape, finger pose, and hand self-occlusion in the generated training hand images can be adjusted using known hand pose parameters, while the lighting conditions of the training hand images can be adjusted using the context image. This allows for the generation of training hand images for four different scenes, ensuring the diversity of the training dataset.

[0088] In some embodiments of this disclosure, a training dataset is constructed, and a Transformer is used to extract visual and EMG features, forming a multimodal end-to-end structure. An attention fusion mechanism is innovatively adopted in the fusion module to achieve efficient fusion of multimodal information, enhancing the model's adaptability and accuracy. This embodiment improves the recognition of hand pose information under different lighting conditions and self-occlusion conditions, and enhances the versatility of recognizing different users and different poses.

[0089] This disclosure also proposes a hand gesture recognition device, comprising:

[0090] An acquisition unit is used to acquire a target hand image and the target electromyographic data corresponding to the target hand image;

[0091] A control unit is used to acquire the temporal characteristics of the target electromyography data and the spatial characteristics of the target hand image;

[0092] The control unit is further configured to fuse the temporal features and the spatial features to obtain fused features;

[0093] The control unit is further configured to determine the pose information of the target hand in the target hand image based on the fusion features.

[0094] In some embodiments, the temporal characteristics of acquiring the target electromyographic data include:

[0095] The target hand image is segmented into multiple non-overlapping image blocks;

[0096] The spatial sub-features of each image block are obtained by projecting each image block.

[0097] The spatial sub-features are fed into the encoder of the spatial converter, and the encoder of the spatial converter generates spatial features of the target hand image based on the spatial sub-features.

[0098] In some embodiments, projecting each of the image blocks to obtain spatial sub-features of each of the image blocks includes: projecting the image blocks and adding position embedding to the projection result to obtain the spatial sub-features;

[0099] And / or,

[0100] The encoder of the spatial converter generates spatial features of the target hand image based on the spatial sub-features, including: extracting features from the spatial sub-features through a multi-head self-attention mechanism and a feedforward neural network, capturing local details and global spatial relationships of the spatial sub-features to generate the spatial features.

[0101] In some embodiments, the temporal characteristics of acquiring the target electromyographic data include:

[0102] Projected data is obtained by performing feature projection on the target electromyography data;

[0103] The projected data is convolutionally downsampled to obtain myoelectronic features;

[0104] The electromyographic features are fed into a temporal convolutional encoder, which generates temporal features of the target electromyographic data based on the electromyographic features.

[0105] In some embodiments, the temporal convolutional encoder generates temporal features of the target electromyography (EMG) data based on the EMG features, including: processing the EMG features through a self-attention mechanism and convolution to capture short-term and long-term temporal features in the target EMG data to generate the temporal features.

[0106] In some embodiments, the temporal features and spatial features are fused to obtain fused features, including:

[0107] A multi-head self-attention mechanism is used to enhance the temporal features;

[0108] A multi-head cross-attention mechanism is employed to fuse temporal and spatial features;

[0109] The output data of the multi-head cross-attention mechanism is processed by a feedforward neural network with residual connections;

[0110] The processing results of the feedforward neural network with residual connections are subjected to average pooling to obtain a fused feature that combines temporal and spatial features.

[0111] In some embodiments, determining the pose information of the target hand in the target hand image based on the fusion features includes:

[0112] The fused features are input into a multilayer perceptron to convert the fused features into the pose information of the target hand.

[0113] In some embodiments, the hand pose recognition device is a target model. Before acquiring the target hand image and the target electromyography data corresponding to the target hand image, the control unit further includes: pre-training the target model using a training dataset to enable the target model, wherein the training dataset includes: training hand images with different hand shapes, different hand gestures, different lighting conditions and / or different hand self-occlusion, corresponding training electromyography data and known hand pose parameters.

[0114] In some embodiments, the training dataset is generated in the following manner:

[0115] Acquire known hand posture parameters and the corresponding electromyographic data;

[0116] Generate a hand skeleton diagram, a hand mask diagram, and a hand normal diagram based on the known hand posture parameters;

[0117] The hand skeleton diagram, hand mask diagram, and hand normal diagram are input into the local control adapter to generate structural features;

[0118] Obtain the selected context image and encode the context image into a feature vector;

[0119] The feature vector is input into the global control adapter to generate features for controlling lighting conditions;

[0120] The structural features and the features used to control lighting conditions are fused to generate the training hand image;

[0121] The electromyographic data corresponding to the known hand posture parameters are used as the training electromyographic data corresponding to the training hand image.

[0122] For embodiments of the apparatus, since they basically correspond to the method embodiments, relevant details can be found in the descriptions of the method embodiments. The apparatus embodiments described above are merely illustrative, and the modules described as separate modules may or may not be separate. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0123] The present invention’s computer-readable storage medium has been described above based on embodiments and application examples. The following describes these electronic devices and computer-readable storage media.

[0124] The following is for reference. Figure 4 This document illustrates a structural schematic of an electronic device (e.g., a terminal device or a server) 800 suitable for implementing embodiments of the present disclosure. It also discloses methods and apparatus for implementation. Furthermore, the present disclosure provides an electronic device and...

[0125] The terminal devices in this disclosure may include, but are not limited to, mobile terminals such as mobile phones, laptops, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), and in-vehicle terminals (e.g., in-vehicle navigation terminals), as well as fixed terminals such as digital TVs and desktop computers. The electronic devices shown in the figures are merely examples and should not be construed as limiting the functionality and scope of use of the embodiments of this disclosure.

[0126] Electronic device 800 may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 801, which can perform various appropriate actions and processes according to a program stored in read-only memory (ROM) 802 or a program loaded from storage device 808 into random access memory (RAM) 803. RAM 803 also stores various programs and data required for the operation of electronic device 800. The processing device 801, ROM 802, and RAM 803 are interconnected via bus 804. Input / output (I / O) interface 805 is also connected to bus 804.

[0127] Typically, the following devices can be connected to I / O interface 805: input devices 806 including, for example, touchscreens, touchpads, keyboards, mice, cameras, microphones, accelerometers, gyroscopes, etc.; output devices 807 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 808 including, for example, magnetic tapes, hard disks, etc.; and communication devices 809. Communication device 809 allows electronic device 800 to communicate wirelessly or wiredly with other devices to exchange data. Although an electronic device 800 with various devices is shown in the figure, it should be understood that it is not required to implement or possess all the devices shown. More or fewer devices may be implemented or possessed alternatively.

[0128] In particular, according to embodiments of this disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this disclosure include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device 809, or installed from a storage device 808, or installed from a ROM 802. When the computer program is executed by a processing device 801, it performs the functions defined in the methods of embodiments of this disclosure.

[0129] It should be noted that the computer-readable medium described in this disclosure can be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this disclosure, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in connection with an instruction execution system, apparatus, or device. In this disclosure, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium can be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wires, optical fibers, RF (radio frequency), etc., or any suitable combination thereof.

[0130] In some implementations, clients and servers can communicate using any currently known or future-developed network protocol such as HTTP (Hypertext Transfer Protocol) and can interconnect with digital data communication (e.g., communication networks) of any form or medium. Examples of communication networks include local area networks (“LANs”), wide area networks (“WANs”), the Internet (e.g., the Internet of Things), and peer-to-peer networks (e.g., ad hoc peer-to-peer networks), as well as any currently known or future-developed networks.

[0131] The aforementioned computer-readable medium may be included in the aforementioned electronic device; or it may exist independently and not assembled into the electronic device.

[0132] The aforementioned computer-readable medium carries one or more programs, which, when executed by the electronic device, cause the electronic device to perform the methods of the present disclosure.

[0133] Computer program code for performing the operations of this disclosure can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0134] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0135] The units described in the embodiments of this disclosure can be implemented in software or hardware. The names of the units are not, in some cases, intended to limit the specific unit.

[0136] The functions described above in this document can be performed, at least in part, by one or more hardware logic components. For example, exemplary types of hardware logic components that can be used, without limitation, include: Field Programmable Gate Arrays (FPGAs), Application-Specific Integrated Circuits (ASICs), Application Standard Products (ASSPs), System-on-Chip (SoCs), Complex Programmable Logic Devices (CPLDs), and so on.

[0137] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0138] According to one or more embodiments of this disclosure, a hand gesture recognition method is provided, comprising:

[0139] Acquire a target hand image and the corresponding target electromyographic data;

[0140] Acquire the temporal features of the target electromyography data and the spatial features of the target hand image;

[0141] The temporal features and spatial features are fused to obtain the fused features;

[0142] The pose information of the target hand in the target hand image is determined based on the fusion features.

[0143] According to one or more embodiments of this disclosure, a hand posture recognition method is provided to obtain the temporal characteristics of the target electromyography data, including:

[0144] The target hand image is segmented into multiple non-overlapping image blocks;

[0145] The spatial sub-features of each image block are obtained by projecting each image block.

[0146] The spatial sub-features are fed into the encoder of the spatial converter, and the encoder of the spatial converter generates spatial features of the target hand image based on the spatial sub-features.

[0147] According to one or more embodiments of this disclosure, a hand pose recognition method is provided, which projects each of the image blocks to obtain spatial sub-features of each of the image blocks, including: projecting the image blocks and using the projection result plus position embedding to obtain the spatial sub-features;

[0148] And / or,

[0149] The encoder of the spatial converter generates spatial features of the target hand image based on the spatial sub-features, including: extracting features from the spatial sub-features through a multi-head self-attention mechanism and a feedforward neural network, capturing local details and global spatial relationships of the spatial sub-features to generate the spatial features.

[0150] According to one or more embodiments of this disclosure, a hand posture recognition method is provided to obtain the temporal characteristics of the target electromyography data, including:

[0151] Projected data is obtained by performing feature projection on the target electromyography data;

[0152] The projected data is convolutionally downsampled to obtain myoelectronic features;

[0153] The electromyographic features are fed into a temporal convolutional encoder, which generates temporal features of the target electromyographic data based on the electromyographic features.

[0154] According to one or more embodiments of this disclosure, a hand pose recognition method is provided, wherein the temporal convolutional encoder generates temporal features of the target electromyography (EMG) data based on the EMG features, comprising: processing the EMG features through a self-attention mechanism and convolution to capture short-term and long-term temporal features in the target EMG data to generate the temporal features.

[0155] According to one or more embodiments of this disclosure, a hand pose recognition method is provided, which fuses the temporal features and the spatial features to obtain fused features, including:

[0156] A multi-head self-attention mechanism is used to enhance the temporal features;

[0157] A multi-head cross-attention mechanism is employed to fuse temporal and spatial features;

[0158] The output data of the multi-head cross-attention mechanism is processed by a feedforward neural network with residual connections;

[0159] The processing results of the feedforward neural network with residual connections are subjected to average pooling to obtain a fused feature that combines temporal and spatial features.

[0160] According to one or more embodiments of this disclosure, a hand pose recognition method is provided, which determines the pose information of a target hand in a target hand image based on the fused features, including:

[0161] The fused features are input into a multilayer perceptron to convert the fused features into the pose information of the target hand.

[0162] According to one or more embodiments of this disclosure, a hand gesture recognition method is provided for a target model;

[0163] Before acquiring the target hand image and the corresponding target electromyographic data, the process also includes:

[0164] The target model is pre-trained using a training dataset, wherein the training dataset includes: training hand images with different hand shapes, different hand gestures, different lighting conditions and / or different hand self-occlusion, corresponding training electromyographic data and known hand pose parameters.

[0165] According to one or more embodiments of this disclosure, a hand pose recognition method is provided, wherein the training dataset is generated in the following manner:

[0166] Acquire known hand posture parameters and the corresponding electromyographic data;

[0167] Generate a hand skeleton diagram, a hand mask diagram, and a hand normal diagram based on the known hand posture parameters;

[0168] The hand skeleton diagram, hand mask diagram, and hand normal diagram are input into the local control adapter to generate structural features;

[0169] Obtain the selected context image and encode the context image into a feature vector;

[0170] The feature vector is input into the global control adapter to generate features for controlling lighting conditions;

[0171] The structural features and the features used to control lighting conditions are fused to generate the training hand image;

[0172] The electromyographic data corresponding to the known hand posture parameters are used as the training electromyographic data corresponding to the training hand image.

[0173] According to one or more embodiments of this disclosure, a hand gesture recognition device is provided, comprising:

[0174] An acquisition unit is used to acquire a target hand image and the target electromyographic data corresponding to the target hand image;

[0175] A control unit is used to acquire the temporal characteristics of the target electromyography data and the spatial characteristics of the target hand image;

[0176] The control unit is further configured to fuse the temporal features and the spatial features to obtain fused features;

[0177] The control unit is further configured to determine the pose information of the target hand in the target hand image based on the fusion features.

[0178] According to one or more embodiments of the present disclosure, an electronic device is provided, including: at least one memory and at least one processor;

[0179] The at least one memory is used to store program code, and the at least one processor is used to call the program code stored in the at least one memory to execute the method described in any one of the above.

[0180] According to one or more embodiments of the present disclosure, a computer-readable storage medium is provided for storing program code that, when executed by a processor, causes the processor to perform the methods described above.

[0181] The above description is merely a preferred embodiment of this disclosure and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of this disclosure is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the above-described concept. For example, technical solutions formed by substituting the above features with (but not limited to) technical features disclosed in this disclosure that have similar functions.

[0182] Furthermore, while the operations are described in a specific order, this should not be construed as requiring these operations to be performed in the specific order shown or in a sequential order. In certain environments, multitasking and parallel processing may be advantageous. Similarly, while several specific implementation details are included in the above discussion, these should not be construed as limiting the scope of this disclosure. Certain features described in the context of individual embodiments may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented individually or in any suitable sub-combination in multiple embodiments.

[0183] Although the subject matter has been described using language specific to structural features and / or methodological logic, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. Rather, the specific features and actions described above are merely illustrative examples of implementing the claims.

Claims

1. A hand gesture recognition method, characterized in that, include: Acquire a target hand image and the corresponding target electromyographic data; Acquire the temporal features of the target electromyography data and the spatial features of the target hand image; The temporal features and spatial features are fused to obtain the fused features; The pose information of the target hand in the target hand image is determined based on the fusion features.

2. The method according to claim 1, characterized in that, The temporal characteristics of the target electromyographic data are obtained, including: The target hand image is segmented into multiple non-overlapping image blocks; The spatial sub-features of each image block are obtained by projecting each image block. The spatial sub-features are fed into the encoder of the spatial converter, and the encoder of the spatial converter generates spatial features of the target hand image based on the spatial sub-features.

3. The method according to claim 2, characterized in that, Projecting each of the image blocks to obtain the spatial sub-features of each of the image blocks includes: projecting the image blocks and adding position embedding to the projection result to obtain the spatial sub-features; And / or, The encoder of the spatial converter generates spatial features of the target hand image based on the spatial sub-features, including: extracting features from the spatial sub-features through a multi-head self-attention mechanism and a feedforward neural network, capturing local details and global spatial relationships of the spatial sub-features to generate the spatial features.

4. The method according to claim 1, characterized in that, The temporal characteristics of the target electromyographic data are obtained, including: Projected data is obtained by performing feature projection on the target electromyography data; The projected data is convolutionally downsampled to obtain myoelectronic features; The electromyographic features are fed into a temporal convolutional encoder, which generates temporal features of the target electromyographic data based on the electromyographic features.

5. The method according to claim 4, characterized in that, The temporal convolutional encoder generates temporal features of the target electromyography (EMG) data based on the EMG features, including: processing the EMG features through a self-attention mechanism and convolution to capture short-term and long-term temporal features in the target EMG data to generate the temporal features.

6. The method according to claim 1, characterized in that, The fused features are obtained by fusing the temporal features and the spatial features, including: A multi-head self-attention mechanism is used to enhance the temporal features; A multi-head cross-attention mechanism is employed to fuse temporal and spatial features; The output data of the multi-head cross-attention mechanism is processed by a feedforward neural network with residual connections; The processing results of the feedforward neural network with residual connections are subjected to average pooling to obtain a fused feature that combines temporal and spatial features.

7. The method according to claim 1, characterized in that, Determining the pose information of the target hand in the target hand image based on the fusion features includes: The fused features are input into a multilayer perceptron to convert the fused features into the pose information of the target hand.

8. The method according to claim 1, characterized in that, This method is used for the target model; Before acquiring the target hand image and the corresponding target electromyographic data, the process also includes: The target model is pre-trained using a training dataset, wherein the training dataset includes: training hand images with different hand shapes, different hand gestures, different lighting conditions and / or different hand self-occlusion, corresponding training electromyographic data and known hand pose parameters.

9. The method according to claim 8, characterized in that, The training dataset was generated in the following manner: Acquire known hand posture parameters and the corresponding electromyographic data; Generate a hand skeleton diagram, a hand mask diagram, and a hand normal diagram based on the known hand posture parameters; The hand skeleton diagram, hand mask diagram, and hand normal diagram are input into the local control adapter to generate structural features; Obtain the selected context image and encode the context image into a feature vector; The feature vector is input into the global control adapter to generate features for controlling lighting conditions; The structural features and the features used to control lighting conditions are fused to generate the training hand image; The electromyographic data corresponding to the known hand posture parameters are used as the training electromyographic data corresponding to the training hand image.

10. A hand gesture recognition device, characterized in that, include: An acquisition unit is used to acquire a target hand image and the target electromyographic data corresponding to the target hand image; A control unit is used to acquire the temporal characteristics of the target electromyography data and the spatial characteristics of the target hand image; The control unit is further configured to fuse the temporal features and the spatial features to obtain fused features; The control unit is further configured to determine the pose information of the target hand in the target hand image based on the fusion features.

11. An electronic device, comprising: At least one memory and at least one processor; The at least one memory is used to store program code, and the at least one processor is used to call the program code stored in the at least one memory to execute the method of any one of claims 1 to 9.

12. A computer-readable storage medium for storing program code that, when executed by a processor, causes the processor to perform the method of any one of claims 1 to 9.

Images