Hand pose tracking method, apparatus, storage medium, and device

Abstract

The application provides a hand posture tracking method and device, a storage medium and equipment. A current frame hand image of a user is collected through a camera of a wearable mixed reality device. The collected current frame hand image is input into a preset neural network system. A first neural network is used to extract features of the current frame hand image to obtain a feature vector of the current frame hand image. A second neural network is used to fuse the feature vector of the current frame hand image and hand posture data of a historical frame hand image to obtain feature map data of the current frame hand image. A third neural network is used to decode the feature map data of the current frame hand image to obtain hand posture data of the current frame hand image. The preset neural network system can directly output hand posture data from the input hand image, and the calculation efficiency is high. The hand posture data obtained by fusing the hand posture data of the historical frame hand image in the calculation process has good continuity.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and in particular to a hand posture tracking method, apparatus, storage medium and device. Background Technology

[0002] XR (eXtended Reality) refers to all real and virtual combined environments and human-computer interactions generated by computer technology and wearable devices. Representative forms include Augmented Reality (AR), Mixed Reality (MR), and Virtual Reality (VR), as well as interdisciplinary scenarios. The level of virtuality ranges from partially sensory input AR to fully immersive VR. A key aspect of XR is the expansion of the human experience, particularly related to the sense of presence (represented by VR) and the acquisition of cognition (represented by AR).

[0003] In the specific implementation of XR technology, gesture recognition is usually used as an interaction method. Gesture recognition technology can make virtual characters map the hand movements of real people, so that users can see the body movements of characters in the virtual environment.

[0004] Existing gesture recognition technologies typically use deep learning networks to determine key hand joint data from hand images. Then, they calculate the positions of these key points in 3D space using the key point data and the geometric relationship between the camera and the hand. Based on these key point positions, a tree-like structure of the hand skeleton is constructed to obtain hand pose data. This method requires mathematical calculations of the key points and the camera's geometric relationship, resulting in high computational cost and low efficiency. Furthermore, the resulting hand pose exhibits poor continuity, with significant hand pose jitter between consecutive frames. Summary of the Invention

[0005] In view of the above problems, embodiments of this application are proposed, which provide a hand posture tracking method, apparatus, storage medium and device to at least solve the above problems.

[0006] One or more embodiments of this application provide a hand pose tracking method, the method comprising: acquiring a current frame hand image of a user through a camera device of a wearable mixed reality device; inputting the acquired current frame hand image into a preset neural network system, the preset neural network system including at least a first neural network, a second neural network and a third neural network; extracting features from the current frame hand image through the first neural network to obtain a feature vector of the current frame hand image; fusing the feature vector of the current frame hand image and hand pose data of historical frame hand images through the second neural network to obtain feature map data of the current frame hand image, wherein the historical frame hand images are image frames preceding the current frame hand image that meet preset conditions, the preset conditions being that the time interval between the historical frame hand image and the current frame hand image is less than or equal to a preset threshold; and decoding the feature map data of the current frame hand image through the third neural network to obtain hand pose data of the current frame hand image.

[0007] Optionally, the second neural network includes at least a fusion layer and a temporal recurrent neural network. The second neural network fuses the feature vector of the current frame hand image and the hand pose data of the historical frame hand images to obtain feature map data of the current frame hand image. This includes: convolving the feature vector of the current frame hand image and the hand pose data of the historical frame hand images through the fusion layer to obtain first feature map data of the current frame hand image; processing the first feature map data through the temporal recurrent neural network to obtain second feature map data of the current frame hand image, and using the second feature map data as the feature map data of the current frame hand image.

[0008] Optionally, the second neural network further includes a normalization layer, and the method further includes: normalizing the second feature map data through the normalization layer to obtain the third feature map data of the hand image in the current frame, and using the third feature map data as the feature map data of the hand image in the current frame.

[0009] Optionally, the third neural network includes at least a feature extraction layer, a bottleneck layer, a self-attention layer, and a pooling layer. The third neural network decodes the feature map data of the current frame hand image to obtain hand pose data of the current frame hand image, including: reducing the dimensionality of the feature map data of the current frame hand image through the feature extraction layer to obtain first data to be decoded; compressing the first data to be decoded through the bottleneck layer to obtain second data to be decoded; processing the second data to be decoded through the self-attention layer to obtain third data to be decoded; and transforming the third data to be decoded through the pooling layer to obtain hand pose data of the current frame hand image.

[0010] Optionally, the method further includes: acquiring hand image training sample data; inputting the hand image training sample data into a preset neural network system, and training the preset neural network system according to a preset loss function.

[0011] Optionally, the hand pose data includes at least keypoint data, 3D position data, and rotation data. The preset loss function includes at least a first part for calculating the detection deviation of the keypoint data, a second part for calculating the detection deviation of the 3D distance information, and a third part for calculating the deviation between the rotation data of the current frame hand image and the rotation data of the historical frame hand images.

[0012] Optionally, the first neural network is a lightweight convolutional neural network.

[0013] According to another aspect of this application, a hand pose tracking device is provided, comprising: a acquisition module for acquiring a current frame hand image of a user via a camera device of a wearable mixed reality device; an input module for inputting the acquired current frame hand image into a preset neural network system, the neural network system including at least a first neural network, a second neural network, and a third neural network; a feature extraction module for extracting features from the current frame hand image through the first neural network to obtain a feature vector of the current frame hand image; a fusion module for fusing the feature vector of the current frame hand image and hand pose data of historical frame hand images through the second neural network to obtain feature map data of the current frame hand image, wherein historical frame hand images are image frames preceding the current frame hand image that meet preset conditions, the preset conditions being that the time interval between the historical frame hand image and the current frame hand image is less than or equal to a preset threshold; and an output module for decoding the feature map data of the current frame hand image through the third neural network to obtain hand pose data of the current frame hand image.

[0014] According to another aspect of this application, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to cause a computer to perform the methods described above.

[0015] According to another aspect of this application, a wearable mixed reality device is provided, comprising: a camera device for capturing images of a hand; one or more processors; and a memory for storing a program; wherein the program includes instructions that, when executed by the processor, cause the processor to perform the methods described above.

[0016] This application provides a hand pose tracking method, apparatus, storage medium, and device. It uses a wearable mixed reality device's camera to capture a user's current frame hand image. The captured current frame hand image is input into a preset neural network system, which includes at least a first neural network, a second neural network, and a third neural network. The first neural network extracts features from the current frame hand image to obtain a feature vector. The second neural network fuses the feature vector of the current frame hand image with hand pose data from historical frame hand images to obtain feature map data for the current frame hand image. The historical frame hand images are image frames preceding the current frame hand image that meet preset conditions. The third neural network decodes the feature map data of the current frame hand image to obtain the hand pose data for the current frame hand image. This application, through its preset neural network system, can directly output hand pose data from an input hand image without requiring mathematical calculations, resulting in high computational efficiency. Furthermore, by fusing hand pose data from historical frame hand images whose time interval from the current frame hand image is less than or equal to a preset threshold during the calculation process, the obtained hand pose data exhibits better continuity and reduces jitter. Attached Figure Description

[0017] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1 A flowchart illustrating a hand pose tracking method as an exemplary embodiment of this application;

[0019] Figure 2 A flowchart illustrating a hand pose tracking method as another exemplary embodiment of this application;

[0020] Figure 3 A flowchart illustrating a hand pose tracking method as another exemplary embodiment of this application;

[0021] Figure 4 This is a structural block diagram of a hand posture tracking device as an exemplary embodiment of this application;

[0022] Figure 5A A structural block diagram of a preset neural network system as an exemplary embodiment of this application;

[0023] Figure 5B A structural block diagram of the second neural network as an exemplary embodiment of this application;

[0024] Figure 5C This is a structural block diagram of a third neural network, which is an exemplary embodiment of this application. Detailed Implementation

[0025] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present application. All other embodiments obtained by those skilled in the art based on the embodiments of the present application without creative effort are within the scope of protection of the present application.

[0026] To facilitate understanding, before describing the specific embodiments of this application in detail, the application scenarios of the hand posture tracking method, apparatus, storage medium and device of this application will be illustrated by example.

[0027] The hand pose tracking method described in this application can be used in wearable mixed reality devices. XR refers to all real and virtual combined environments and human-computer interactions generated by computer technology and wearable devices. Representative forms include augmented reality, mixed reality, and virtual reality, as well as interdisciplinary scenarios. In the specific implementation of XR technology, gesture recognition is typically used as the interaction method. Gesture recognition technology allows virtual characters to map the hand movements of real people, thereby enabling users to see the body movements of characters in the virtual environment.

[0028] Existing gesture recognition technologies typically use deep learning networks to determine key hand joint data from hand images, then calculate the positions of these key points in 3D space using the key point data and the geometric relationship between the camera and the hand. Based on these key point positions, a tree-like structure of the hand skeleton is constructed to obtain the hand's position and pose. This method requires mathematical calculations of the relationships between key points and the camera, resulting in high computational cost and low efficiency. Furthermore, the continuity of the obtained hand position and pose is poor, with significant jitter between consecutive frames. Therefore, this application proposes a hand pose tracking method, apparatus, storage medium, and device that can solve the various problems existing in the prior art.

[0029] The specific embodiments of this application will be described in detail below with reference to the accompanying drawings.

[0030] Figure 1 The flowchart of a hand pose tracking method, which is an exemplary embodiment of this application, is shown in the figure. This embodiment mainly includes the following steps:

[0031] S101. Acquire the user's current frame hand image through the camera device of a wearable mixed reality device;

[0032] Exemplary examples show that the wearable mixed reality device in this application embodiment can be a head-mounted display device (hereinafter referred to as "head-mounted display"), such as an AR head-mounted display, a VR head-mounted display, a MR head-mounted display, etc., and there is no limitation in this application embodiment. The examples in the following embodiments should not be construed as limiting the scope of protection claimed in this application. The camera device can be a binocular camera, which can capture images of the user's hand through the binocular camera on the wearable mixed reality device. The captured hand images are multiple consecutive frames.

[0033] S102. Input the captured current frame hand image into the preset neural network system.

[0034] The preset neural network system includes at least a first neural network, a second neural network, and a third neural network.

[0035] For example, the preset neural network system is a pre-trained neural network system that can be deployed on a wearable mixed reality device, referencing... Figure 5A The input terminal of the preset neural network system is the input terminal of the first neural network, the output terminal of the first neural network is connected to the input terminal of the second neural network, the output terminal of the second neural network is connected to the input terminal of the third neural network, and the output terminal of the preset neural network system is the output terminal of the third neural network.

[0036] It should be noted that the size of the hand image in the current frame can be 112*112 pixels or an integer multiple of 112*112 pixels, such as 224*224 pixels. This embodiment does not impose any restrictions on this.

[0037] S103. The first neural network is used to extract features from the hand image of the current frame to obtain the feature vector of the hand image of the current frame.

[0038] For example, the first neural network can be a convolutional neural network (CNN), a type of feedforward neural network that includes convolutional computations and has a deep structure. It comprises convolutional layers and pooling layers and can be used for large-scale image processing. The first neural network can compress and extract feature information from the hand image of the current frame, converting it into a set of encoded feature vectors. This facilitates the subsequent extraction of hand pose data by the second and third neural networks.

[0039] In one specific implementation, the first neural network is a lightweight convolutional neural network.

[0040] For example, the first neural network can be a lightweight convolutional neural network such as MobileNetV3, MobileNetV2, MobileNetV1, ShuffleNet, etc. Compared with convolutional neural networks, lightweight convolutional neural networks have a smaller scale or fewer parameters, but do not affect the computational accuracy of the neural network model.

[0041] In this implementation, the first neural network uses a lightweight convolutional neural network, which can reduce the size of the preset neural network system while ensuring computational accuracy. This makes it easier to deploy the preset neural network system in a wearable mixed reality device to realize the hand posture tracking method of this application embodiment.

[0042] S104. The feature vector of the current frame hand image and the hand pose data of the historical frame hand images are fused by the second neural network to obtain the feature map data of the current frame hand image.

[0043] For example, the historical frame hand image is an image frame preceding the current frame hand image that meets a preset condition. The preset condition is that the time interval between the historical frame hand image and the current frame hand image is less than or equal to a preset threshold. For example, the historical frame hand image can be the previous frame adjacent to the current frame hand image, or it can be a frame that is not adjacent to the current frame hand image and the time interval does not exceed the preset threshold. The preset time interval can be set to 0.1 milliseconds, and this embodiment does not limit this. The feature vector of the current frame hand image output by the first neural network and the hand pose data of the historical frame hand images are input into the second neural network. If the current frame hand image is the first frame hand image, that is, there is no historical frame hand image, then the input hand pose data of the historical frame hand image is 0. The feature vector of the current frame hand image and the hand pose data of the historical frame hand images are fused in the second neural network to output the feature map data of the current frame hand image.

[0044] S105. The feature map data of the hand image in the current frame is decoded by the third neural network to obtain the hand pose data of the hand image in the current frame.

[0045] For example, the feature map data of the current frame hand image output by the second neural network is input into the third neural network. The third neural network decodes the feature map data of the current frame hand image and outputs the hand pose data of the current frame hand image. The hand pose data may include key point data, three-dimensional position data, and rotation data, wherein the key point data is two-dimensional coordinates, the three-dimensional position data is three-dimensional coordinates, and the rotation data is quaternions.

[0046] In one specific implementation, the third neural network includes at least a feature extraction layer, a bottleneck layer, a self-attention layer, and a pooling layer. The third neural network decodes the feature map data of the current frame hand image to obtain hand pose data of the current frame hand image. This includes: reducing the dimensionality of the feature map data of the current frame hand image through the feature extraction layer to obtain first data to be decoded; compressing the first data to be decoded through the bottleneck layer to obtain second data to be decoded; processing the second data to be decoded through the self-attention layer to obtain third data to be decoded; and transforming the third data to be decoded through the pooling layer to obtain the hand pose data of the current frame hand image.

[0047] For example, refer to Figure 5C The feature map data of the current frame hand image output by the second neural network is input into the feature extraction layer. The feature extraction layer reduces the dimensionality of the feature map data of the current frame hand image and outputs the first data to be decoded. The first data to be decoded is input into the bottleneck layer. The bottleneck layer compresses the first data to be decoded using a series of 3*3 channel separable convolutions and outputs the second data to be decoded. The second data to be decoded is input into the self-attention layer. The self-attention layer processes the second data to be decoded and outputs the third data to be decoded. The first, second and third data to be decoded are all feature maps. The third data to be decoded is input into the pooling layer. The pooling layer performs global average pooling on the third data to be decoded and then uses a 1*1 convolution operation to transform the data and output the hand pose data of the current frame hand image.

[0048] In this implementation, setting a bottleneck layer and a self-attention layer in the third neural network can reduce the number of computational parameters, thereby reducing the computational load and improving the computational efficiency of the preset neural network system.

[0049] This embodiment provides a hand pose tracking method that uses a wearable mixed reality device's camera to capture a user's current frame hand image. The captured current frame hand image is then input into a preset neural network system, which includes at least a first neural network, a second neural network, and a third neural network. The first neural network extracts features from the current frame hand image to obtain a feature vector. The second neural network fuses the feature vector of the current frame hand image with hand pose data from previous frames to obtain feature map data for the current frame hand image. The previous frames are image frames preceding the current frame hand image. The third neural network decodes the feature map data of the current frame hand image to obtain the hand pose data for the current frame hand image. This embodiment employs end-to-end deep learning. Through a pre-set neural network system, input hand images can directly output hand pose data without the need for mathematical calculations, resulting in low computational load. Furthermore, it can utilize GPUs, NPUs, and other technologies to accelerate performance and improve computational efficiency. During the calculation process, it incorporates hand pose data from historical frame hand images whose time interval from the current frame is less than or equal to a preset threshold, resulting in better continuity of the obtained hand pose data, improved stability, and reduced jitter.

[0050] Figure 2 This is a flowchart illustrating a hand pose tracking method as another exemplary embodiment of this application. This embodiment mainly shows a specific implementation of step S104 described above. The second neural network includes at least a fusion layer and a temporal recurrent neural network. As shown in the figure, this embodiment mainly includes the following steps:

[0051] S201. The feature vector of the current frame hand image and the hand pose data of the historical frame hand images are convolved by the fusion layer to obtain the first feature map data of the current frame hand image.

[0052] S202. The first feature map data is processed by a time recurrent neural network to obtain the second feature map data of the hand image in the current frame, and the second feature map data is used as the feature map data of the hand image in the current frame.

[0053] For example, refer to Figure 5BThe first step involves inputting the feature vector of the current frame's hand image (output from the first neural network) and the hand pose data from previous frames into a fusion layer. This fusion layer uses a 1x1 convolutional kernel to perform convolution operations on the feature vector of the current frame's hand image and the hand pose data from previous frames, fusing the data and outputting the first feature map data of the current frame's hand image. This first feature map data is then input into a temporal recurrent neural network (TRN). The TRN processes this first feature map data and outputs the second feature map data of the current frame's hand image, which is then used as the feature map data for the current frame's hand image. The TRN possesses persistence and memory; when processing the first feature map data of the current frame's hand image through the TRN, the output feature map data at the current stage incorporates the influence of feature map data from multiple previous frames of the hand image.

[0054] It should be noted that the temporal recurrent neural network can be an LSTM (Long Short-Term Memory) network. The second neural network can use one or more temporal recurrent neural networks. More iterations result in better continuity of hand pose data but also longer computation time. Preferably, a two-step temporal recurrent neural network can be used, ensuring both good continuity of hand pose data and high computational efficiency.

[0055] In this embodiment, by using a fusion layer to fuse the feature vector of the current frame hand image and the hand pose data of the historical frame hand images, a first feature map data is obtained. Then, the first feature map data is processed by a time recurrent neural network, which can fuse the features of multiple frames of hand images before the current frame hand image into the obtained feature map data, thereby making the obtained hand pose data more continuous and more accurate.

[0056] In one specific implementation, the second neural network further includes a normalization layer. The method also includes: normalizing the second feature map data through the normalization layer to obtain the third feature map data of the hand image in the current frame, and using the third feature map data as the feature map data of the hand image in the current frame.

[0057] For example, refer to Figure 5B The second feature map data of the hand image of the current frame, output by the time recurrent neural network, is input into the normalization layer. The normalization layer performs a normalization operation on the second feature map data of the hand image of the current frame and outputs the third feature map data of the hand image of the current frame. The third feature map data is used as the feature map data of the hand image of the current frame.

[0058] In this implementation, by adding a normalization layer to the second neural network, the preset neural network system can converge faster during training, thereby effectively shortening the training time of the preset neural network system.

[0059] Figure 3 This is a flowchart illustrating a hand pose tracking method as another exemplary embodiment of this application. This embodiment mainly shows optional implementations prior to step S101. As shown in the figure, this embodiment mainly includes the following steps:

[0060] S301. Obtain training sample data of hand images.

[0061] S302. Input the hand image training sample data into the preset neural network system, and train the preset neural network system according to the preset loss function.

[0062] For example, historical hand image data collected by a camera device on a wearable mixed reality device can be used as hand image training sample data; this embodiment is not limited to this. The hand image training sample data may include hand images and annotation data of the hand images, the annotation data including key points. The hand image training sample data is input into a preset neural network system, and the preset neural network system is trained according to a preset loss function.

[0063] In this embodiment, a preset neural network system is trained using hand image training sample data and a preset loss function. The trained preset neural network system is then used to calculate hand pose data, resulting in higher accuracy.

[0064] In one specific implementation, the hand pose data includes at least keypoint data, 3D position data, and rotation data. The preset loss function includes at least a first part for calculating the detection deviation of the keypoint data, a second part for calculating the detection deviation of the 3D distance information, and a third part for calculating the deviation between the rotation data of the current frame hand image and the rotation data of the historical frame hand images.

[0065] For example, the preset loss function Q can be expressed as:

[0066]

[0067] Among them, E (key) E represents the first part used to calculate the detection bias of keypoint data. (dist) E represents the second part used to calculate the detection bias of 3D distance information. (time)This represents the third part used to calculate the deviation between the rotation data of the hand image in the current frame and the rotation data of the hand images in historical frames. w1, w2, and w3 represent hyperparameters that adjust the weights of each part in the preset loss function Q. During training, w1, w2, and w3 can be set to any constant between 0.01 and 100, and this embodiment does not impose any restrictions on this.

[0068] The first part E used to calculate the detection deviation of key point data (key) This refers to the predicted value p for each keypoint. (i,j) The label value p^ for each keypoint (i,j) The sum of Euclidean distances. (key) This can be expressed as a formula:

[0069]

[0070] The second part E used to calculate the detection deviation of three-dimensional distance information (dist) This refers to the three-dimensional position prediction value m (i) With the 3D location label value m^ (i) The sum of the cosine distance and the dist distance between them, where the dist distance is determined by m. (i) Calculate the Euclidean distance of the hand skeletal tree structure in three-dimensional space. (dist) This can be expressed as a formula:

[0071]

[0072] Part E is used to calculate the deviation between the rotation data of the current frame's hand image and the rotation data of historical frame's hand images. (time) This refers to the predicted rotation data value g of the hand image in the current frame. (t) Euclidean distance between the predicted rotational data g(t-1) of the hand image in the previous frame and the predicted value of the rotational data. (time) This can be expressed as a formula:

[0073]

[0074] In this implementation, when training the preset neural network system, the minimum value of the preset loss function is found, so that the trained preset neural network system can make more accurate prediction results when predicting hand posture data.

[0075] Figure 4 This is a structural block diagram of a hand posture tracking device that is an exemplary embodiment of this application.

[0076] The hand posture tracking device 400 of this embodiment can be installed in a wearable mixed reality device, wherein the wearable mixed reality device is suitable for performing hand posture tracking tasks.

[0077] As shown in the figure, the hand posture tracking device 400 in this embodiment mainly includes: a data acquisition module 401, an input module 402, a feature extraction module 403, a fusion module 404, and an output module 405.

[0078] The acquisition module 401 is used to acquire the user's current frame hand image through the camera device of the wearable mixed reality device; the input module 402 is used to input the acquired current frame hand image into a preset neural network system, which includes at least a first neural network, a second neural network, and a third neural network; the feature extraction module 403 is used to extract features from the current frame hand image through the first neural network to obtain the feature vector of the current frame hand image; the fusion module 404 is used to fuse the feature vector of the current frame hand image and the hand posture data of historical frame hand images through the second neural network to obtain the feature map data of the current frame hand image, wherein the historical frame hand images are image frames that precede the current frame hand image and meet preset conditions, the preset conditions being that the time interval between the historical frame hand image and the current frame hand image is less than or equal to a preset threshold; and the output module 405 is used to decode the feature map data of the current frame hand image through the third neural network to obtain the hand posture data of the current frame hand image.

[0079] Optionally, the second neural network includes at least a fusion layer and a temporal recurrent neural network. The fusion module 404 is further configured to convolve the feature vector of the current frame hand image and the hand pose data of the historical frame hand images through the fusion layer to obtain the first feature map data of the current frame hand image; process the first feature map data through the temporal recurrent neural network to obtain the second feature map data of the current frame hand image, and use the second feature map data as the feature map data of the current frame hand image.

[0080] Optionally, the second neural network further includes a normalization layer, and the fusion module 404 is further used to normalize the second feature map data through the normalization layer to obtain the third feature map data of the hand image in the current frame, and use the third feature map data as the feature map data of the hand image in the current frame.

[0081] Optionally, the third neural network includes at least a feature extraction layer, a bottleneck layer, a self-attention layer, and a pooling layer. The output module 405 is further configured to reduce the dimensionality of the feature map data of the hand image in the current frame through the feature extraction layer to obtain the first data to be decoded; compress the first data to be decoded through the bottleneck layer to obtain the second data to be decoded; process the second data to be decoded through the self-attention layer to obtain the third data to be decoded; and transform the third data to be decoded through the pooling layer to obtain the hand pose data of the hand image in the current frame.

[0082] Optionally, the device further includes a training module for acquiring hand image training sample data; inputting the hand image training sample data into a preset neural network system; and training the preset neural network system according to a preset loss function.

[0083] Optionally, the hand pose data includes at least keypoint data, 3D position data, and rotation data. The preset loss function includes at least a first part for calculating the detection deviation of the keypoint data, a second part for calculating the detection deviation of the 3D distance information, and a third part for calculating the deviation between the rotation data of the current frame hand image and the rotation data of the historical frame hand images.

[0084] Optionally, the first neural network is a lightweight convolutional neural network.

[0085] Furthermore, the hand posture tracking device 400 of this application embodiment can also be used to implement other steps in the aforementioned hand posture tracking method embodiments, and has the beneficial effects of the corresponding method step embodiments, which will not be repeated here.

[0086] An exemplary embodiment of this application also provides a non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to cause a computer to perform the methods of various embodiments of this application.

[0087] An exemplary embodiment of this application also provides a wearable mixed reality device, including: a camera device for capturing images of a hand; one or more processors; and a memory for storing a program; wherein the program includes instructions that, when executed by the processor, cause the processor to perform the methods of the embodiments of this application.

[0088] It should be noted that in the description of this application, the terms "first" and "second" are used only for convenience in describing different components or names, and should not be construed as indicating or implying a sequential relationship, relative importance, or implicitly specifying the number of technical features indicated. Therefore, a feature defined with "first" or "second" may explicitly or implicitly include at least one of those features.

[0089] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the application.

[0090] It should be noted that although specific embodiments of this application have been described in detail with reference to the accompanying drawings, this should not be construed as limiting the scope of protection of this application. Various modifications and variations that can be made by those skilled in the art without inventive effort within the scope described in the claims still fall within the scope of protection of this application.

[0091] The examples of the embodiments in this application are intended to concisely illustrate the technical features of the embodiments in this application, so that those skilled in the art can intuitively understand the technical features of the embodiments in this application, and are not intended to be improper limitations on the embodiments in this application.

[0092] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.

Claims

1. A hand pose tracking method, characterized in that, The method includes: The system uses a camera device in a wearable mixed reality device to capture the user's current frame image of their hand. The acquired hand image of the current frame is input into a preset neural network system, which includes at least a first neural network, a second neural network, and a third neural network. The first neural network is used to extract features from the current frame hand image to obtain the feature vector of the current frame hand image; The feature vector of the current frame hand image output by the first neural network and the hand pose data of the historical frame hand images are input into the second neural network. If the current frame hand image is the first frame hand image, that is, there is no historical frame hand image, then the input hand pose data of the historical frame hand image is 0. The second neural network fuses the feature vector of the current frame hand image and the hand pose data of the historical frame hand images to obtain the feature map data of the current frame hand image. The historical frame hand images are image frames that precede the current frame hand image and meet preset conditions. The preset conditions are that the time interval between the historical frame hand image and the current frame hand image is less than or equal to a preset threshold. The feature map data of the current frame hand image is decoded by a third neural network to obtain the hand pose data of the current frame hand image.

2. The method according to claim 1, characterized in that, The second neural network includes at least a fusion layer and a temporal recurrent neural network. The step of fusing the feature vector of the current frame hand image and the hand pose data of historical frame hand images using the second neural network to obtain the feature map data of the current frame hand image includes: The fusion layer convolves the feature vector of the current frame hand image and the hand pose data of the historical frame hand images to obtain the first feature map data of the current frame hand image. The first feature map data is processed by the time recurrent neural network to obtain the second feature map data of the hand image in the current frame, and the second feature map data is used as the feature map data of the hand image in the current frame.

3. The method according to claim 2, characterized in that, The second neural network further includes a normalization layer, and the method further includes: The second feature map data is standardized by the standardization layer to obtain the third feature map data of the current frame hand image, and the third feature map data is used as the feature map data of the current frame hand image.

4. The method according to claim 1, characterized in that, The third neural network includes at least a feature extraction layer, a bottleneck layer, a self-attention layer, and a pooling layer. Decoding the feature map data of the current frame hand image using the third neural network to obtain the hand pose data of the current frame hand image includes: The feature map data of the current frame hand image is reduced in dimensionality by the feature extraction layer to obtain the first data to be decoded; The first data to be decoded is compressed through the bottleneck layer to obtain the second data to be decoded; The second data to be decoded is processed by the self-attention layer to obtain the third data to be decoded; The pooling layer is used to transform the third data to be decoded to obtain the hand pose data of the current frame hand image.

5. The method according to claim 1, characterized in that, The method further includes: Obtain training sample data of hand images; The hand image training sample data is input into the preset neural network system, and the preset neural network system is trained according to the preset loss function.

6. The method according to claim 5, characterized in that, The hand pose data includes at least key point data, three-dimensional position data, and rotation data. The preset loss function includes at least a first part for calculating the key point data detection deviation, a second part for calculating the three-dimensional distance information detection deviation, and a third part for calculating the deviation between the rotation data of the current frame hand image and the rotation data of the historical frame hand images.

7. The method according to claim 1, characterized in that, The first neural network is a lightweight convolutional neural network.

8. A hand posture tracking device, characterized in that, The device includes: The acquisition module is used to acquire the user's current frame hand image through the camera device of the wearable mixed reality device; An input module is used to input the acquired current frame hand image into a preset neural network system, the neural network system including at least a first neural network, a second neural network and a third neural network; The feature extraction module is used to extract features from the current frame hand image through the first neural network to obtain the feature vector of the current frame hand image; The fusion module is used to input the feature vector of the current frame hand image output by the first neural network and the hand pose data of the historical frame hand images into the second neural network. If the current frame hand image is the first frame hand image, that is, there is no historical frame hand image, then the input hand pose data of the historical frame hand image is 0. The second neural network fuses the feature vector of the current frame hand image and the hand pose data of the historical frame hand image to obtain the feature map data of the current frame hand image. The historical frame hand image is an image frame that precedes the current frame hand image and meets a preset condition. The preset condition is that the time interval between the historical frame hand image and the current frame hand image is less than or equal to a preset threshold. The output module is used to decode the feature map data of the current frame hand image through a third neural network to obtain the hand pose data of the current frame hand image.

9. A non-transitory computer-readable storage medium storing computer instructions, wherein, The computer instructions are used to cause the computer to perform the method described in any one of claims 1-7.

10. A wearable mixed reality device, characterized in that, include: A camera device used to capture images of the hand; One or more processors; And the memory for storing programs; The program includes instructions that, when executed by the processor, cause the processor to perform the method described in any one of claims 1-7.

Images