Unity gesture recognition method and device based on 3D hand grid reconstruction and medium
By using a neural network model based on the Transformer architecture to reconstruct 3D hand meshes, the problem of insufficient accuracy and stability in gesture recognition in existing technologies is solved, achieving high-precision virtual interaction and an immersive experience.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-12-26
- Publication Date
- 2026-03-31
AI Technical Summary
Existing gesture recognition solutions based on the Unity engine rely on sparse hand keypoint detection, resulting in insufficient accuracy and stability of keypoint coordinates, which affects the reliability and immersion of virtual interaction.
A neural network model based on the Transformer architecture is used to reconstruct 3D hand meshes. The parameterized model parameters drive the preset hand model to generate a high-precision 3D hand mesh, and the geometric features are used for gesture recognition.
It improves the accuracy and stability of gesture recognition, enhances the reliability and immersion of virtual interaction, and meets the needs of real-time interaction.
Smart Images
Figure CN121768079A_ABST
Abstract
Description
Technical Field
[0001] This application generally relates to virtual reality technology, and more particularly to Unity gesture recognition methods, devices and media based on 3D hand mesh reconstruction. Background Technology
[0002] Currently, existing gesture recognition solutions based on the Unity engine (such as the MediaPipe plugin) suffer from low reliability of virtual interactions due to insufficient accuracy and stability of keypoint coordinates caused by reliance on sparse hand keypoint detection. Summary of the Invention
[0003] This application addresses some deficiencies mentioned in the background art by providing a Unity gesture recognition method, device, and medium based on 3D hand mesh reconstruction.
[0004] In a first aspect, embodiments of this application provide a Unity gesture recognition method based on 3D hand mesh reconstruction, comprising: The system acquires RGB images captured in real-time by the Unity engine and inputs them into a neural network model based on the Transformer architecture for processing. The neural network model outputs parameterized model parameters for the hand, including posture parameters representing the movement state of the hand joints and shape parameters representing the physiological morphological features of the hand. In the Unity engine, the parameterized model parameters drive a pre-set parameterized hand model to generate a corresponding 3D hand mesh in real time. Gesture recognition is performed based on the geometric features of the 3D hand mesh, and the gesture recognition results are output.
[0005] In some embodiments, the processing procedure of the neural network model in S1 includes: The visual Transformer model is used as an encoder to extract features from RGB images, so as to output a fixed-dimensional feature vector containing global and local features of the hand. A decoder constructed using a fully connected neural network is used to decode the feature vectors and output parameterized model parameters.
[0006] In some embodiments, the parametric hand model includes a MANO model; the parametric model parameters include MANO parameters; the MANO parameters include posture parameters characterizing the motion state of the hand joints and shape parameters characterizing the physiological morphological features of the hand.
[0007] In some embodiments, the posture parameters of the hand joint motion state include global rotation parameters and finger joint rotation parameters.
[0008] In some embodiments, during the training process of the neural network model, the pose parameters output by the decoder are trained using the L1 loss function and the shape parameters output by the decoder are trained using the KL divergence loss function.
[0009] In some embodiments, the real-time generation of the corresponding 3D hand mesh in S2 is executed on the GPU, specifically including: preloading the basic deformable mesh data and joint weight matrix of the parameterized hand model into the GPU's video memory buffer, where the video memory buffer is a texture buffer or a structured buffer; calling the data in the video memory buffer through the GPU compute shader, performing vertex deformation calculation and skin weight fusion calculation on the basic deformable mesh according to the parameterized model parameters, and generating 3D hand mesh data containing vertex coordinates.
[0010] In some embodiments, S3 specifically includes: Based on a predefined vertex-key point mapping table, a weighted average is calculated on the vertex coordinates associated with the target key points in the 3D hand mesh to obtain the 3D coordinates of the hand key points; the weights for the weighted average calculation are preset vertex contribution weights. Based on the 3D coordinates of key hand points, the geometric features of the key hand points are extracted and matched with a preset gesture template to output the gesture recognition result.
[0011] In some embodiments, the geometric features of the hand keypoints include at least one of the following: relative angles between hand keypoints, Euclidean distance, or motion trajectory features.
[0012] In a second aspect, embodiments of this application also provide an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the method as described in the above embodiments.
[0013] In a third aspect, embodiments of this application also provide a computer-readable storage medium having a computer program stored thereon that, when executed by a processor, implements the method as described in the above embodiments. Attached Figure Description
[0014] The accompanying drawings are provided to further illustrate the present disclosure and form part of the specification. They are used together with the following detailed description to explain the present disclosure, but do not constitute a limitation thereof. In the drawings: Figure 1 A flowchart of a Unity gesture recognition method based on 3D hand mesh reconstruction provided in this application embodiment. Detailed Implementation
[0015] The specific embodiments of this disclosure will be described in detail below with reference to the accompanying drawings. It should be understood that the specific embodiments described herein are for illustration and explanation only and are not intended to limit this disclosure.
[0016] In this specification, for convenience, terms such as "middle," "upper," "lower," "front," "rear," "vertical," "horizontal," "top," "bottom," "inner," and "outer" are used to indicate orientation or positional relationships in conjunction with the accompanying drawings. This is solely for the purpose of describing this specification and simplifying the description, and does not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, it should not be construed as a limitation of this disclosure. The positional relationships of the constituent elements may be appropriately changed depending on the direction in which each constituent element is described. Therefore, the terminology used is not limited to those described in the specification and may be appropriately replaced as appropriate.
[0017] Unless otherwise defined, the technical or scientific terms used in the embodiments of this disclosure should be understood in their ordinary sense by one of ordinary skill in the art to which this disclosure pertains. The terms "first," "second," and similar terms used in this disclosure do not indicate any order, quantity, or importance, but are merely used to distinguish different components. Similarly, terms such as "comprising" or "including" mean that the element or object preceding the word encompasses the elements or objects listed following the word and their equivalents, without excluding other elements or objects. The terms "installed," "connected," and "joined" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can be a mechanical connection or a joint; they can be a direct connection, an indirect connection via an intermediary, or a connection within two elements. Those skilled in the art can understand the specific meaning of the above terms in this disclosure based on the specific circumstances.
[0018] In interactive applications developed using the Unity engine, gesture recognition is a core technology for achieving natural human-computer interaction. Current gesture recognition solutions rely on the MediaPipe plugin, which extracts 21 sparse hand keypoints using a convolutional neural network (CNN) and then classifies gestures based on the keypoint coordinates. However, this solution has two major drawbacks: First, CNNs have limited global feature extraction capabilities; the 3D coordinates of keypoints are easily affected by viewing angle and occlusion, resulting in low z-axis accuracy and significant jitter, leading to issues like "unable to grasp" or "abnormal slippage" during fine interactions such as virtual grasping. Second, visualization based solely on the skeleton connected by keypoints cannot reconstruct the complete hand shape, resulting in poor immersion in AR / VR scenarios. Therefore, there is an urgent need for a gesture recognition method that is compatible with Unity and balances high accuracy and real-time performance.
[0019] Figure 1A flowchart of a Unity gesture recognition method for 3D hand mesh reconstruction provided in some embodiments of this application is shown below. Figure 1 As shown in the embodiments of this application, the method includes the following steps: S1. Acquire RGB images captured in real time by the Unity engine, input the RGB images into a neural network model based on the Transformer architecture for processing, and output parameterized model parameters of the hand. The parameterized model parameters include posture parameters representing the movement state of the hand joints and shape parameters representing the physiological morphological features of the hand. S2. In the Unity engine, use the parameterized model parameters to drive the pre-set parameterized hand model to generate the corresponding 3D hand mesh in real time. S3. Perform gesture recognition based on the geometric features of the 3D hand mesh and output the gesture recognition results.
[0020] This application uses a neural network model based on the Transformer architecture to process image data, which can overcome the problems existing in the MediaPipe plugin and avoid the problem of low reliability of virtual interaction caused by insufficient accuracy and stability of key point coordinates due to reliance on sparse hand key point detection.
[0021] In some embodiments, the processing procedure of the neural network model in S1 includes: using a visual Transformer model as an encoder to extract features from an RGB image, so as to output a fixed-dimensional feature vector containing global and local features of the hand; and using a decoder composed of a fully connected neural network to decode the feature vector and output parameterized model parameters.
[0022] In some embodiments, the parametric hand model includes a MANO model; the parametric model parameters include MANO parameters; the MANO parameters include posture parameters characterizing the motion state of the hand joints and shape parameters characterizing the physiological morphological features of the hand.
[0023] Furthermore, the posture parameters of the hand joint motion state include global rotation parameters and finger joint rotation parameters.
[0024] In some embodiments, during the training process of the neural network model, the pose parameters output by the decoder are trained using the L1 loss function and the shape parameters output by the decoder are trained using the KL divergence loss function.
[0025] For example, in one embodiment, the ViT-Huge is lightweighted by annotating 2D keypoints and occlusion states based on the HInt dataset, and the last layer is set to output a 256-dimensional feature vector to adapt to the input of the decoder.
[0026] Specifically, the 256-dimensional feature vector output by the ViT-Huge model is used as the input to the decoder. This feature vector already contains global semantic information and local detail features of the hand image. The decoder's network structure employs a two-layer fully connected neural network, with 512 hidden units in the first layer and 256 in the second. This configuration allows for sufficient non-linear expressive power to fit the complex MANO parameter space while maintaining the lightweight nature of the neural network model.
[0027] In this embodiment, the output layer is fixed at 31 dimensions, including 21-dimensional pose parameters and 10-dimensional shape parameters. These parameters are directly used to drive the neural network model to generate a 3D hand mesh. During training, the pose parameter prediction is supervised using the L1 loss function. Since the L1 loss is less sensitive to outliers, this training method can better adapt to the continuous changes in hand joint angles.
[0028] During training, the shape parameter prediction uses KL divergence as the loss function to constrain the distribution of the shape parameter to be close to the predefined prior distribution of the neural network model, thereby avoiding overfitting of the neural network model during training.
[0029] The embodiments of this application employ a combination strategy of K1 and KL dual loss functions, which can effectively balance the accuracy of attitude prediction and the rationality of shape parameters.
[0030] Furthermore, during the real-time generation of the 3D hand mesh, the 31-dimensional parameters output by the trained decoder are converted into a standard format that can be recognized by the neural network model.
[0031] Among them, the posture parameters are used to characterize the three-dimensional rotation angles of 21 key joints of the hand. These angles are recursively calculated according to the skeletal hierarchy defined by the neural network model to ensure that adjacent joints can produce a natural linkage effect when the fingers are bent.
[0032] Shape parameters are used to characterize macroscopic morphological features such as palm width and finger length. These parameters affect the vertex positions of the 3D hand mesh through pre-computed PCA basis vectors.
[0033] In some embodiments, the real-time generation of the corresponding 3D hand mesh in S2 is executed on the GPU. Specifically, this includes: preloading the basic deformable mesh data and joint weight matrix of the parameterized hand model into the GPU's memory buffer, where the memory buffer is either a texture buffer or a structured buffer. Then, the GPU compute shader calls the data in the memory buffer to perform vertex deformation calculation and skin weight fusion calculation on the basic deformable mesh according to the parameterized model parameters, generating 3D hand mesh data containing vertex coordinates.
[0034] For example, in one embodiment, during the Unity initialization phase, a pre-defined base deformable mesh for the neural network model under different shape parameters, as well as a weight matrix showing the influence of joint rotation on vertex positions, are constructed. This data is stored in a texture buffer or structured buffer in the GPU memory, allowing the final mesh vertex positions to be quickly calculated at runtime through simple texture sampling and matrix multiplication, thus avoiding the performance overhead of complex skinning calculations on the CPU.
[0035] This GPU-accelerated deformation calculation method can control the generation time of a single frame of 3D hand mesh to within 2ms, thus meeting the needs of real-time interaction.
[0036] In some embodiments, gesture recognition based on the geometric features of a 3D hand mesh and outputting gesture recognition results specifically include: calculating a weighted average of the coordinates of vertices in the 3D hand mesh associated with the target key point based on a predefined vertex-key point mapping table to obtain the 3D coordinates of the hand key point; the weight of the weighted average calculation is a preset vertex contribution weight.
[0037] Furthermore, based on the 3D coordinates of key hand points, the geometric features of the key hand points are extracted and matched with a preset gesture template to output the gesture recognition result.
[0038] In this embodiment of the application, a mapping table between the vertices of the 3D hand mesh and 21 key points is established to record the main vertex index of each key point and mark the weight distribution of the surrounding influential vertices of the key point.
[0039] For example, in one instance, the fingertip keypoint is calculated by weighting the three vertices of the triangular facet region it belongs to, while the knuckle keypoint can be determined by a Gaussian weighted average of multiple vertices in a circular distribution. This multi-vertex weighting strategy effectively smooths out the discretization error caused by mesh subdivision, resulting in sub-pixel accuracy in the calculated keypoint coordinates.
[0040] In some embodiments, the geometric features of the hand keypoints include at least one of the following: relative angles between hand keypoints, Euclidean distance, or motion trajectory features.
[0041] For example, in the embodiments of this application, static gestures can be identified by calculating geometric features such as the relative angle between fingers and the distance ratio from the fingertip to the center of the palm.
[0042] For example, when judging a fist gesture, the distance ratio between each fingertip and the corresponding finger root joint can be calculated. When all ratios are below a threshold, recognition is triggered and the gesture recognition result is output.
[0043] This application also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the method as described in the above embodiments.
[0044] This application also provides a computer-readable storage medium having a computer program stored thereon that, when executed by a processor, implements the methods of the embodiments described above.
Claims
1. A Unity gesture recognition method based on 3D hand mesh reconstruction, characterized in that, The method comprises the following steps: S1, acquiring an RGB image collected in real time by a Unity engine, inputting the RGB image into a neural network model based on a Transformer architecture for processing, the neural network model outputting parameterized model parameters of a hand, the parameterized model parameters comprising pose parameters representing a joint motion state of the hand and shape parameters representing a physiological morphology feature of the hand; S2, in the Unity engine, driving a preset parameterized hand model by using the parameterized model parameters to generate a corresponding 3D hand mesh in real time; S3, performing gesture recognition based on geometric features of the 3D hand mesh and outputting a gesture recognition result.
2. The method of claim 1, wherein, The processing process of the neural network model in S1 comprises: using a visual Transformer model as an encoder to perform feature extraction on the RGB image to output a fixed-dimension feature vector containing global and local features of the hand; using a decoder composed of a fully connected neural network to decode the feature vector and output the parameterized model parameters.
3. The method of claim 2, wherein, The parameterized hand model comprises a MANO model; the parameterized model parameters comprise MANO parameters; and the MANO parameters comprise pose parameters representing a joint motion state of the hand and shape parameters representing a physiological morphology feature of the hand.
4. The method of claim 3, wherein, The pose parameters representing the joint motion state of the hand comprise global rotation parameters and finger joint rotation parameters.
5. The method of claim 3, wherein, In the training process of the neural network model, the pose parameters output by the decoder are supervised and trained by using an L1 loss function, and the shape parameters output by the decoder are supervised and trained by using a KL divergence loss function.
6. The method of claim 1, wherein, The real-time generation of the corresponding 3D hand mesh in S2 is performed on a GPU, and specifically comprises: preloading base deformation mesh data and joint weight matrices of the parameterized hand model to a video memory buffer of the GPU, the video memory buffer being a texture buffer or a structured buffer; calling data in the video memory buffer by a GPU computing shader, performing vertex deformation calculation and skin weight fusion calculation on the base deformation mesh according to the parameterized model parameters, and generating 3D hand mesh data containing vertex coordinates.
7. The method according to claim 1 or 6, characterized in that, S3 specifically comprises: based on a predefined vertex-keypoint mapping relationship table, performing weighted average calculation on vertex coordinates associated with target key points in the 3D hand mesh to obtain 3D coordinates of the hand key points; the weight of the weighted average calculation being a preset vertex contribution weight; based on the 3D coordinates of the hand key points, extracting geometric features of the hand key points, and matching the geometric features with a preset gesture template to output a gesture recognition result.
8. The method of claim 7, wherein, The geometric features of the hand key points comprise at least one of a relative angle, an Euclidean distance or a motion trajectory feature between the hand key points.
9. An electronic device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, The processor executes the program to implement the method in any one of claims 1 to 8.
10. A computer-readable storage medium having stored thereon a computer program, characterized in that, The program is executed by the processor to implement the method in any one of claims 1 to 8.