Gesture recognition method and system, head-mounted display equipment and medium

Abstract

The invention discloses a gesture recognition method and system, a head-mounted display device and a medium, and relates to the technical field of image processing, and the disclosed gesture recognition method comprises the steps: obtaining hand images collected by each camera, setting the hand images corresponding to the same collection moment as a group, and obtaining each to-be-processed combination; inputting the to-be-processed combinations into a preset prediction model to obtain hand joint point coordinates corresponding to the to-be-processed combinations; and obtaining a target hand joint point coordinate according to each hand joint point coordinate, and performing gesture recognition based on the target hand joint point coordinate to obtain a gesture recognition result. The technical scheme of the invention aims to solve the technical problem of how to improve the accuracy of gesture recognition.

Description

Gesture recognition methods, systems, head-mounted display devices and media Technical Field

[0001] This application relates to the field of image processing technology, and in particular to a gesture recognition method, system, head-mounted display device, and medium. Background Technology

[0002] Gesture recognition is a technology that analyzes and recognizes human hand movements to understand the interaction between humans and computers. It is widely used in virtual reality, augmented reality, smart homes, healthcare, gaming and other fields.

[0003] However, traditional gesture recognition methods are affected by the image used for gesture recognition, resulting in poor accuracy. For example, when there is occlusion in the image used for gesture recognition, the results obtained from that image will be inaccurate.

[0004] Therefore, how to improve the accuracy of gesture recognition is a technical problem that needs to be solved by those skilled in the art. Summary of the Invention

[0005] The main objective of this application is to provide a gesture recognition method, system, head-mounted display device, and medium, aiming to solve the technical problem of how to improve the accuracy of gesture recognition.

[0006] To achieve the above objectives, this application proposes a gesture recognition method applied to a head-mounted display device, the head-mounted display device being equipped with various cameras, the method comprising:

[0007] The hand images captured by each of the cameras are obtained, and the hand images corresponding to the same capture time are grouped together to obtain each combination to be processed;

[0008] Each of the combinations to be processed is input into a preset prediction model to obtain the hand joint coordinates corresponding to each combination to be processed.

[0009] The target hand joint coordinates are obtained based on the coordinates of each hand joint, and gesture recognition is performed based on the target hand joint coordinates to obtain the gesture recognition result.

[0010] In one embodiment, the step of obtaining the target hand joint coordinates based on the coordinates of each of the hand joints includes:

[0011] Obtain the preset weights corresponding to each of the combinations to be processed;

[0012] Based on the correspondence between the coordinates of each hand joint and the combination to be processed, and the correspondence between the combination to be processed and the weights, the coordinates of each hand joint are weighted and summed to obtain the coordinates of the target hand joint.

[0013] In one embodiment, before the step of grouping hand images corresponding to the same acquisition time to obtain various combinations to be processed, the method further includes:

[0014] Extract hand features from each of the hand images, and convert each of the hand features into a feature matrix;

[0015] The step of setting hand images corresponding to the same acquisition time as a group to obtain various combinations to be processed includes:

[0016] The feature matrices corresponding to the same acquisition time are set as a group to obtain various combinations to be processed.

[0017] In one embodiment, the prediction model includes an encoder and a decoder, wherein the encoder includes self-attention modules and the decoder includes cross-attention modules, and the step of inputting each of the combinations to be processed into a preset prediction model to obtain the hand joint coordinates corresponding to each of the combinations to be processed includes:

[0018] The latent features of each feature matrix in the combination to be processed are extracted by each of the self-attention modules, and the latent features are passed to each of the cross-attention modules;

[0019] The hand joint coordinates corresponding to the combination to be processed are obtained by predicting based on the latent features by each of the cross-attention modules, wherein the joint distribution represented by the latent features is the same as the joint distribution represented by the hand joint coordinates.

[0020] In one embodiment, the step of acquiring the hand images captured by each of the cameras includes:

[0021] The speed of hand joint movement is obtained, and a target speed range corresponding to the speed is determined in a preset speed range.

[0022] Based on the preset prediction step size corresponding to the target speed range, the hand images captured by each of the cameras are obtained.

[0023] In one embodiment, the step of performing gesture recognition based on the coordinates of the target hand joints to obtain a gesture recognition result includes:

[0024] Extract the gesture features of the target hand joint coordinates;

[0025] The gesture features are input into a preset gesture classification model to obtain gesture recognition results. The gesture classification model is trained using the gesture features of the preset gesture data as model input data and the actual gesture classification results of the gesture data as model training labels.

[0026] In one embodiment, after the step of obtaining the target hand joint coordinates based on the coordinates of each of the hand joints, the method further includes:

[0027] In the virtual page of the head-mounted display device, a virtual hand is rendered according to the coordinates of the target hand joint points, and feedback information corresponding to the virtual hand is received;

[0028] If the feedback information indicates that the virtual hand simulation is correct, then the step of performing gesture recognition based on the coordinates of the target hand joints to obtain the gesture recognition result is executed.

[0029] Furthermore, to achieve the above objectives, this application also proposes a gesture recognition system applied to a head-mounted display device, the head-mounted display device being equipped with various cameras, the gesture recognition system comprising:

[0030] The image acquisition module is used to acquire hand images captured by each of the cameras, and set the hand images corresponding to the same acquisition time as a group to obtain each combination to be processed;

[0031] The prediction module is used to input each of the combinations to be processed into a preset prediction model to obtain the hand joint coordinates corresponding to each of the combinations to be processed.

[0032] The recognition module is used to obtain the coordinates of the target hand joints based on the coordinates of each hand joint, and to perform gesture recognition based on the coordinates of the target hand joints to obtain the gesture recognition result.

[0033] In addition, to achieve the above objectives, this application also proposes a head-mounted display device, the device comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the gesture recognition method as described above.

[0034] In addition, to achieve the above objectives, this application also proposes a storage medium, which is a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, it implements the steps of the gesture recognition method described above.

[0035] In this embodiment, by acquiring hand images captured by each camera and grouping hand images corresponding to the same acquisition time into a set, various processing combinations are obtained. This allows for the combination of hand images from different angles acquired at the same time, thus providing a more comprehensive understanding of the hand's state and avoiding the problem of hand information being obscured by a single camera. Each processing combination is then input into a preset prediction model to obtain the corresponding hand joint coordinates. The pre-trained prediction model can be used to process the images in each processing combination and make a relatively complete prediction of hand postures at different times. Then, the target hand joint coordinates are obtained based on the hand joint coordinates, and gesture recognition is performed based on these coordinates to obtain the gesture recognition result. The user's hand posture can be integrated from the prediction results at different times, thereby accurately recognizing the gesture.

[0036] In this application, multiple cameras can capture the user's gestures from different angles, thereby accurately reconstructing the user's actual gestures from the captured images. It is evident that this application improves gesture recognition accuracy by using multiple cameras and integrating the coordinates of hand joints at different acquisition times. Attached Figure Description

[0037] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0038] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0039] Figure 1 is a flowchart of the gesture recognition method according to Embodiment 1 of this application;

[0040] Figure 2 is a flowchart of the gesture recognition method provided in Embodiment 5 of this application;

[0041] Figure 3 is a schematic diagram of the data processing flow of a specific embodiment of the gesture recognition method of this application;

[0042] Figure 4 is a schematic diagram of the module structure of the gesture recognition system according to an embodiment of this application;

[0043] Figure 5 is a schematic diagram of the device structure of the hardware operating environment involved in the gesture recognition method in this application embodiment.

[0044] The purpose, features, and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0045] It should be understood that the specific embodiments described herein are merely illustrative of the technical solutions of this application and are not intended to limit this application.

[0046] To better understand the technical solution of this application, a detailed description will be provided below in conjunction with the accompanying drawings and specific implementation methods.

[0047] Gesture recognition is a technology that analyzes and recognizes human hand movements to understand the interaction between humans and computers. It is widely used in virtual reality, augmented reality, smart homes, healthcare, gaming and other fields.

[0048] However, traditional gesture recognition methods are affected by the image used for gesture recognition, resulting in poor accuracy. For example, when there is occlusion in the image used for gesture recognition, the results obtained from that image will be inaccurate.

[0049] Therefore, how to improve the accuracy of gesture recognition is a technical problem that needs to be solved by those skilled in the art.

[0050] To address the aforementioned issues, this application acquires hand images captured by each camera individually and groups hand images corresponding to the same capture time to obtain various processing combinations. This allows for the combination of hand images captured at different angles at the same time, providing a more comprehensive understanding of the hand's state and avoiding the problem of hand information being obscured by a single camera. Each processing combination is then input into a pre-set prediction model to obtain the corresponding hand joint coordinates. The pre-trained prediction model can then process the images in each processing combination and make relatively complete predictions of hand postures at different times. Finally, the target hand joint coordinates are obtained based on the hand joint coordinates, and gesture recognition is performed based on these coordinates to obtain the gesture recognition result. The user's hand posture can be integrated from the prediction results at different times, thus accurately recognizing the gesture.

[0051] In this application, multiple cameras can capture the user's gestures from different angles, thereby accurately reconstructing the user's actual gestures from the captured images. It is evident that this application improves gesture recognition accuracy by using multiple cameras and integrating the coordinates of hand joints at different acquisition times.

[0052] It should be noted that the execution subject of the gesture recognition method of this application can be a head-mounted display device with data processing, network communication, and program execution functions, such as VR (Virtual Reality) glasses, VR helmets, AR (Augmented Reality) glasses, AR helmets, etc. The following uses VR glasses as an example to describe various embodiments of the gesture recognition method of this application.

[0053] This application provides a gesture recognition method. Referring to FIG1, FIG1 is a flowchart of the first embodiment of the gesture recognition method of this application.

[0054] In this embodiment, the gesture recognition method is applied to a head-mounted display device, which is equipped with various cameras. The gesture recognition method includes steps S10 to S30:

[0055] Step S10: Obtain the hand images captured by each of the cameras, and set the hand images corresponding to the same capture time as a group to obtain each combination to be processed;

[0056] It should be noted that this application does not specifically limit the number or location of the cameras. In this application, multiple cameras are used to capture hand images from different angles, thereby facilitating the prediction or reconstruction of the user's actual hand posture.

[0057] Understandably, when only one camera is present in VR glasses, hand gestures can be obscured. Therefore, this application addresses this deficiency by capturing the same hand movement from different angles. A hand image refers to an image captured by the camera that includes complete hand features.

[0058] In this embodiment, when gesture recognition is required, each camera can be controlled to simultaneously capture hand images. Then, based on the capture time, hand images corresponding to the same capture time are grouped into the same group, thereby obtaining multiple combinations to be processed.

[0059] Step S20: Input each of the combinations to be processed into the preset prediction model to obtain the hand joint coordinates corresponding to each of the combinations to be processed.

[0060] In this embodiment, the prediction model is pre-trained. Specifically, the prediction model can be a deep learning model or a machine learning model, and this application does not limit it. In this embodiment, the prediction model is used to process the hand images in each combination to be processed and to predict the user's actual hand posture at each time step. The aforementioned hand joint coordinates refer to the coordinates corresponding to each joint of the hand in a preset coordinate system.

[0061] It is understandable that for each hand image captured at the same time, that is, for each combination to be processed, the coordinates of the hand joints can be predicted by the prediction model. Thus, the hand posture corresponding to different capture times can be reconstructed based on the prediction model.

[0062] Step S30: Obtain the target hand joint coordinates based on the coordinates of each hand joint, and perform gesture recognition based on the target hand joint coordinates to obtain the gesture recognition result.

[0063] It should be noted that the target hand joint coordinates refer to the coordinates of each joint in the integrated hand pose.

[0064] It's understandable that the user's gesture is dynamic, while the image captured by the camera is static. Therefore, after predicting the hand pose at each capture moment (obtainable through hand joint coordinates), these hand poses need to be integrated to predict the actual hand pose given by the user. Then, gesture recognition can be performed based on the predicted hand pose to obtain the gesture recognition result.

[0065] In one feasible implementation, step S30 above includes:

[0066] Step S301: Extract the gesture features of the target hand joint coordinates;

[0067] It should be noted that gesture features include shape features and orientation features. Shape features characterize the shape of the hand, while orientation features characterize the direction in which the hand movement is directed. This application does not limit the method for extracting gesture features.

[0068] Step S302: Input the gesture features into a preset gesture classification model to obtain the gesture recognition result. The gesture classification model is trained by using the gesture features of the preset gesture data as the model input data and the actual gesture classification results of the gesture data as the model training labels.

[0069] It should be noted that the gesture classification model can be a machine learning model or a deep learning model, and this application does not impose any restrictions on it. Gesture data contains various static gestures; gesture data refers specifically to data used to train the gesture classification model.

[0070] After a gesture is recognized, in one feasible implementation, both the scene output in the VR glasses and the recognized gesture will affect the VR glasses' response. Therefore, after obtaining the gesture recognition result, the response to the gesture corresponding to the gesture recognition result can be based on the current scene in the VR glasses.

[0071] In this embodiment, by acquiring hand images captured by each camera and grouping hand images corresponding to the same acquisition time into a set, various processing combinations are obtained. This allows for the combination of hand images captured at different angles at the same time, thus providing a more comprehensive understanding of the hand's state and avoiding the problem of hand information being obscured by a single camera. Each processing combination is then input into a preset prediction model to obtain the corresponding hand joint coordinates. The pre-trained prediction model can be used to process the images in each processing combination and make a relatively complete prediction of hand postures at different times. Then, the target hand joint coordinates are obtained based on the hand joint coordinates, and gesture recognition is performed based on these coordinates to obtain the gesture recognition result. The user's hand posture can be integrated based on the prediction results at different times, thereby accurately recognizing the gesture.

[0072] In this application, multiple cameras can capture the user's gestures from different angles, thereby accurately reconstructing the user's actual gestures from the captured images. It is evident that this application improves gesture recognition accuracy by using multiple cameras and integrating the coordinates of hand joints at different acquisition times.

[0073] Furthermore, based on the first embodiment of the gesture recognition method of this application described above, a second embodiment of the gesture recognition method of this application is proposed.

[0074] In this embodiment, step S30 includes:

[0075] Step S303: Obtain the preset weights corresponding to each of the combinations to be processed;

[0076] Step S304: Based on the correspondence between each hand joint coordinate and each processing combination, and the correspondence between each processing combination and each weight, the hand joint coordinates are weighted and summed to obtain the target hand joint coordinates.

[0077] It is understandable that hand images captured at different times have varying degrees of influence on the predicted final gesture. For example, in the dynamic process of opening a hand, images captured later have a greater influence on the final gesture than images captured earlier. Therefore, weights can be pre-set for each combination to be processed to improve the probability of predicting the true final gesture. Of course, in a feasible implementation, the weights can also be implemented through model prediction or other methods; this application does not restrict the source of the weights.

[0078] In this embodiment, since there is a one-to-one correspondence between the hand joint coordinates and the combination to be processed, and also a one-to-one correspondence between the combination to be processed and the weights, the correspondence between the hand joint coordinates and the weights can be obtained by using the combination to be processed as a bridge. Furthermore, based on the correspondence between the hand joint coordinates and the weights, the coordinates of each hand joint can be weighted and summed to obtain the target hand joint coordinates.

[0079] For example, if there are 21 joints on a single hand, then when the user outputs a single hand movement, each hand joint coordinate includes the coordinates of all 21 joints. When there are 5 combinations to be processed, there are 5 sets of hand joint coordinates. For these 5 sets of hand joint coordinates, the coordinates of each joint can be calculated by weighted summation. Thus, the target hand joint coordinates are obtained.

[0080] In this embodiment, the method of obtaining the target hand joint coordinates through weighting can improve the accuracy of the target hand joint coordinate calculation. This, in turn, can improve the accuracy of subsequent gesture recognition.

[0081] Furthermore, based on the first and / or second embodiments of the gesture recognition method of this application described above, a third embodiment of the gesture recognition method of this application is proposed.

[0082] In this embodiment, before the step of grouping hand images corresponding to the same acquisition time to obtain various combinations to be processed, the method further includes:

[0083] Step S100: Extract hand features from each of the hand images, and convert each of the hand features into a feature matrix;

[0084] It should be noted that hand features refer to features other than environmental features.

[0085] Understandably, hand images captured by cameras often include environmental information about the user's surroundings while using VR glasses, but this environmental information is useless for gesture recognition. Therefore, after obtaining each hand image, redundant information in the hand images can be removed by extracting hand features.

[0086] Furthermore, after obtaining the hand features, in order to facilitate the subsequent prediction model to identify the hand features corresponding to various hand images belonging to the same time, the hand features can be converted into a feature matrix.

[0087] For example, after obtaining the hand image, feature extraction can be performed using a ResNet18 residual neural network to remove redundant information such as background correlation, and then the hand features corresponding to each hand image can be obtained. Then, each hand feature can be converted into a feature matrix through flattening and linear layers.

[0088] Based on this, the above step S1O includes:

[0089] The feature matrices corresponding to the same acquisition time are set as a group to obtain various combinations to be processed.

[0090] Therefore, the combination of images to be processed in the first embodiment can be converted into a combination of feature matrices to be processed.

[0091] In this embodiment, by removing redundant information from the hand image and converting the hand features into a feature matrix, this application improves the probability of the prediction model correctly recognizing image information.

[0092] Furthermore, based on the third embodiment of the gesture recognition method of this application described above, a fourth embodiment of the gesture recognition method of this application is proposed.

[0093] In this embodiment, the prediction model is a transformer model, which includes an encoder and a decoder. The encoder includes self-attention modules, and the decoder includes cross-attention modules. Step S20 includes:

[0094] Step S201: Extract the latent features of each feature matrix in the combination to be processed through each self-attention module, and pass the latent features to each cross-attention module;

[0095] It should be noted that latent features refer to features that are not directly apparent or are not easily observed intuitively. In this embodiment, the self-attention module can extract latent features based on each feature matrix.

[0096] In one feasible implementation, the latent features can be the distribution relationship of key points obtained based on multiple feature matrices.

[0097] For example, the steps for extracting latent features may be:

[0098] 1) The input feature matrix is ​​transformed into query, key, and value matrices respectively through linear transformation.

[0099] 2) For the query matrix and key matrix, the attention score is calculated using dot product, scaling and softmax operations.

[0100] 3) The value matrix is ​​weighted and summed based on the attention score to obtain the weighted feature representation.

[0101] 4) The outputs of each self-attention module are fused to obtain the final latent features.

[0102] Step S202: Based on the latent features, each of the cross-attention modules makes predictions to obtain the hand joint coordinates corresponding to the combination to be processed, wherein the joint distribution represented by the latent features is the same as the joint distribution represented by the hand joint coordinates.

[0103] It is understandable that the distribution of joints represented by the latent features is the same as the distribution of joints represented by the hand joint coordinates. This means that, for the decoder, the output hand joint coordinates match the input latent features. Therefore, it can be guaranteed that the obtained hand joint coordinates are predicted based on the latent features.

[0104] In this embodiment, this application uses a self-attention module to extract features and a cross-attention module to predict the state of each joint of the hand at the same acquisition time based on the feature matrices input at the same acquisition time. This solves the technical problem of hand information being obscured due to a single camera.

[0105] Furthermore, based on the various embodiments of the gesture recognition method of this application described above, a fifth embodiment of the gesture recognition method of this application is proposed.

[0106] Please refer to Figure 2. In this embodiment, step S10 includes:

[0107] Step S101: Obtain the speed of hand joint movement, and determine the target speed range corresponding to the speed in the preset speed ranges;

[0108] Understandably, VR glasses often include pre-installed programs to measure the speed of hand joint movements. For example, when VR glasses are detected being worn, voice prompts can be output to guide the user to perform specific gestures. Images can then be captured during the user's gesture execution, and the displacement of the joint points per unit time can be determined. This allows for the calculation of the speed of the user's hand joint movements.

[0109] After calculating the speed of the hand joint movement, the speed range corresponding to that speed can be determined from multiple preset speed ranges, that is, the target speed range can be obtained.

[0110] Step S102: Based on the preset prediction step size corresponding to the target speed range, acquire the hand images captured by each of the cameras.

[0111] It should be noted that the prediction step size refers to the number of combinations to be processed required to predict the final gesture. For example, when the hand joint movement speed is high, the number of combinations to be processed required to predict the final gesture is small. For instance, if the prediction step size is 5 corresponding to the target speed range, it means that the final gesture can be predicted from 5 frames of images simultaneously captured by each camera.

[0112] Understandably, the higher the speed, the smaller the predicted step size, and vice versa. To facilitate adjusting the predicted step size based on different speeds, various speed ranges can be pre-defined, along with their corresponding predicted step sizes. Therefore, once the target speed range is determined, the predicted step size corresponding to that range can be used as the step size for predicting the user's gesture.

[0113] For example, when the prediction step size is 5, it means that the 6th frame image needs to be predicted based on the previous 5 frames (the previous 5 frames captured by each camera). Thus, assuming that each camera starts shooting simultaneously, the previous 5 frames of hand images captured by each camera can be obtained separately.

[0114] In this embodiment, the method of adjusting the predicted step length based on the speed of the user's hand joint movement can automatically adjust the amount of data processing based on the speed characteristics of the user's hand joint movement, thereby enabling gesture recognition while taking into account both prediction efficiency and prediction accuracy.

[0115] Furthermore, based on the various embodiments of the gesture recognition method of this application described above, a sixth embodiment of the gesture recognition method of this application is proposed.

[0116] In this embodiment, after step S30, the method further includes:

[0117] Step S40: In the virtual page of the head-mounted display device, render a virtual hand according to the coordinates of the target hand joint points, and receive feedback information corresponding to the virtual hand;

[0118] Step S50: If the feedback information indicates that the virtual hand simulation is correct, then the step of performing gesture recognition based on the coordinates of the target hand joints to obtain the gesture recognition result is executed.

[0119] Understandably, after obtaining the coordinates of the target hand's joints, the positions of each joint in the hand can be determined based on these coordinates. Then, a virtual hand can be rendered based on the positions of each joint for user confirmation. Finally, if the user's feedback indicates that the virtual hand matches the user's actual gesture, subsequent gesture recognition steps can proceed.

[0120] In this embodiment, the present application avoids the technical problem of wasted computing power in gesture recognition caused by the mismatch between the calculated target hand joint coordinates and the actual gesture by rendering first and then recognizing, thereby reducing the probability of the gesture recognition result not matching the actual gesture.

[0121] Furthermore, based on the various embodiments of the gesture recognition method of this application described above, a specific embodiment of the gesture recognition method of this application is proposed.

[0122] Please refer to Figure 3, which is a schematic diagram of the data processing flow of a specific embodiment of the gesture recognition method of this application. In Figure 3, there are 4 cameras, and 4 self-attention modules and 4 cross-attention modules. After acquiring 4 hand images corresponding to the same acquisition time, hand features need to be extracted through a residual network model, and feature matrices corresponding to each of the 4 hand images are obtained through flattening and linear layers. Then, the 4 feature matrices are respectively input into 4 self-attention modules to obtain the latent features among the 4 feature matrices. Then, the hand joint coordinates corresponding to the acquisition time can be obtained by predicting based on the latent features through the 4 cross-attention modules. Then, given the known prediction step size, the target hand joint coordinates can be obtained by integrating multiple sets of hand joint coordinates for gesture recognition.

[0123] Furthermore, in this embodiment, the hand joint coordinates may include not only the two-dimensional coordinates of each joint, but also joint depth values ​​(relative depth, absolute depth, etc.) and joint categories (left hand, right hand). Before applying the cross-attention module, it needs to be trained. In one feasible implementation, the loss function of the cross-attention module may include a category loss L. cla Joint position loss L pos and depth loss L depth Probability distribution loss L reg The expressions for each loss are:

[0124]

[0125] L reg =KL(E out ||D out );

[0126] Where KL is an abbreviation for Kullback-Leibler divergence, L reg In the expression, || represents the symbol for the parameters required by the KL algorithm. The mathematical expression ‖xx‖1 represents the L1 loss of "xx", i represents the i-th joint, 42 is the total number of joints for both hands, and Cla ipred Cla represents the predicted category of the i-th joint. i gt Pos represents the true label category of the i-th joint. i pred Pos represents the predicted position of the i-th joint. i gt Dep represents the true label position of the i-th joint. i pred Dep represents the predicted depth information of the i-th joint. i gt This represents the true label depth information of the i-th keypoint, where depth information refers to the distance between the hand keypoint and the camera; E out D represents the encoder output. out This represents the output of the decoder, whose purpose is to ensure that the feature information extracted by the encoder is consistent with the distribution of keypoint positions in the final output. Based on this, the overall loss L of the cross-attention module is:

[0127] L=α1L cla +α2L pos +α3L depth +α4L reg ;

[0128] α1, α2, α3, and α4 are tuning parameters.

[0129] Therefore, based on the above loss function, the hand joint coordinates output by the cross-attention module can be limited to be relatively accurate, thus laying a good foundation for accurate gesture recognition.

[0130] It should be noted that the above examples are only for understanding this application and do not constitute a limitation on the gesture recognition method of this application. Any simple modifications based on this technical concept are within the protection scope of this application.

[0131] This application also provides a gesture recognition system applied to a head-mounted display device. The head-mounted display device is equipped with various cameras. Referring to Figure 4, the gesture recognition system includes:

[0132] The image acquisition module 10 is used to acquire the hand images captured by each of the cameras, and set the hand images corresponding to the same acquisition time as a group to obtain each combination to be processed;

[0133] The prediction module 20 is used to input each of the combinations to be processed into a preset prediction model to obtain the hand joint coordinates corresponding to each of the combinations to be processed.

[0134] The recognition module 30 is used to obtain the target hand joint coordinates based on the coordinates of each hand joint, and to perform gesture recognition based on the target hand joint coordinates to obtain the gesture recognition result.

[0135] In one embodiment, the identification module 30 is further configured to:

[0136] Obtain the preset weights corresponding to each of the combinations to be processed;

[0137] Based on the correspondence between the coordinates of each hand joint and the combination to be processed, and the correspondence between the combination to be processed and the weights, the coordinates of each hand joint are weighted and summed to obtain the coordinates of the target hand joint.

[0138] In one embodiment, the gesture recognition system includes:

[0139] An image preprocessing module is used to extract hand features from each of the hand images and convert each of the hand features into a feature matrix;

[0140] Based on this, the image acquisition module 10 described above is also used for:

[0141] The feature matrices corresponding to the same acquisition time are set as a group to obtain various combinations to be processed.

[0142] In one embodiment, the prediction model includes an encoder and a decoder, wherein the encoder includes self-attention modules, the decoder includes cross-attention modules, and the prediction module 20 is configured to:

[0143] The latent features of each feature matrix in the combination to be processed are extracted by each of the self-attention modules, and the latent features are passed to each of the cross-attention modules;

[0144] The hand joint coordinates corresponding to the combination to be processed are obtained by predicting based on the latent features by each of the cross-attention modules, wherein the joint distribution represented by the latent features is the same as the joint distribution represented by the hand joint coordinates.

[0145] In one embodiment, the image acquisition module 10 is further configured to:

[0146] The speed of hand joint movement is obtained, and a target speed range corresponding to the speed is determined in a preset speed range.

[0147] Based on the preset prediction step size corresponding to the target speed range, the hand images captured by each of the cameras are obtained.

[0148] In one embodiment, the identification module 30 is further configured to:

[0149] Extract the gesture features of the target hand joint coordinates;

[0150] The gesture features are input into a preset gesture classification model to obtain gesture recognition results. The gesture classification model is trained using the gesture features of the preset gesture data as model input data and the actual gesture classification results of the gesture data as model training labels.

[0151] In one embodiment, the gesture recognition system includes:

[0152] The hand rendering module is used to render a virtual hand in the virtual page of the head-mounted display device according to the coordinates of the target hand joint points, and to receive feedback information corresponding to the virtual hand.

[0153] The switching module is used to execute the step of performing gesture recognition based on the coordinates of the target hand joints to obtain the gesture recognition result if the feedback information indicates that the virtual hand simulation is correct.

[0154] The gesture recognition system provided in this application, employing the gesture recognition method described in the above embodiments, can solve the technical problem of how to improve the accuracy of gesture recognition. Compared with the prior art, the beneficial effects of the gesture recognition system provided in this application are the same as those of the gesture recognition method provided in the above embodiments, and other technical features of the gesture recognition system are the same as those disclosed in the methods of the above embodiments, and will not be repeated here.

[0155] This application provides a head-mounted display device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, which are executed by the at least one processor to enable the at least one processor to perform the gesture recognition method in Embodiment 1 above.

[0156] Referring now to Figure 5, a schematic diagram of a head-mounted display device suitable for implementing embodiments of this application is shown. The head-mounted display device shown in Figure 5 is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments of this application.

[0157] As shown in Figure 5, the head-mounted display device may include a processing unit 1001 (e.g., a central processing unit, a graphics processing unit, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 1002 or a program loaded from a storage device 1003 into a random access memory (RAM) 1004. The RAM 1004 also stores various programs and data required for the operation of the head-mounted display device. The processing unit 1001, ROM 1002, and RAM 1004 are interconnected via a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Typically, the following systems can be connected to I / O interface 1006: input devices 1007 including, for example, touchscreens, touchpads, keyboards, mice, image sensors, microphones, accelerometers, gyroscopes, etc.; output devices 1008 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 1003 including, for example, magnetic tapes, hard disks, etc.; and communication devices 1009. Communication device 1009 allows the head-mounted display to communicate wirelessly or wiredly with other devices to exchange data. Although head-mounted display devices with various systems are shown in the figures, it should be understood that it is not required to implement or possess all the systems shown. More or fewer systems may be implemented alternatively.

[0158] Specifically, according to the embodiments disclosed in this application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments disclosed in this application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device, or installed from storage device 1003, or installed from ROM 1002. When the computer program is executed by processing device 1001, it performs the functions defined in the methods of the embodiments disclosed in this application.

[0159] The head-mounted display device provided in this application, employing the gesture recognition method in the above embodiments, can solve the technical problem of how to improve the accuracy of gesture recognition. Compared with the prior art, the beneficial effects of the head-mounted display device provided in this application are the same as those of the gesture recognition method provided in the above embodiments, and other technical features in this head-mounted display device are the same as those disclosed in the previous embodiment method, and will not be repeated here.

[0160] It should be understood that the various parts disclosed in this application can be implemented using hardware, software, firmware, or a combination thereof. In the description of the above embodiments, specific features, structures, materials, or characteristics can be combined in any suitable manner in one or more embodiments or examples.

[0161] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

[0162] This application provides a computer-readable storage medium having computer-readable program instructions (i.e., a computer program) stored thereon, the computer-readable program instructions being used to execute the gesture recognition method in the above embodiments.

[0163] The computer-readable storage medium provided in this application may be, for example, a USB flash drive, but is not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to: electrical connections having one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, system, or device. The program code contained on the computer-readable storage medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (Radio Frequency), etc., or any suitable combination thereof.

[0164] The aforementioned computer-readable storage medium may be included in the head-mounted display device; or it may exist independently and not assembled into the head-mounted display device.

[0165] The aforementioned computer-readable storage medium carries one or more programs. When the aforementioned one or more programs are executed by the head-mounted display device, the head-mounted display device causes the following: to acquire hand images captured by each camera, and to group hand images corresponding to the same acquisition time into a group to obtain each processing combination; to input each processing combination into a preset prediction model to obtain the hand joint coordinates corresponding to each processing combination; to obtain the target hand joint coordinates based on the hand joint coordinates, and to perform gesture recognition based on the target hand joint coordinates to obtain the gesture recognition result.

[0166] Computer program code for performing the operations of this application can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a Local Area Network (LAN) or a Wide Area Network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0167] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0168] The modules described in the embodiments of this application can be implemented in software or hardware. The names of the modules do not necessarily limit the functionality of the unit itself.

[0169] The readable storage medium provided in this application is a computer-readable storage medium that stores computer-readable program instructions (i.e., a computer program) for executing the above-described gesture recognition method, thereby solving the technical problem of how to improve the accuracy of gesture recognition. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided in this application are the same as those of the gesture recognition method provided in the above embodiments, and will not be repeated here.

[0170] The above description is only a part of the embodiments of this application and does not limit the patent scope of this application. All equivalent structural transformations made under the technical concept of this application and using the contents of the specification and drawings of this application, or direct / indirect applications in other related technical fields, are included in the patent protection scope of this application.

Claims

1. A gesture recognition method, characterized in that, The gesture recognition method is applied to a head-mounted display device equipped with various cameras. The gesture recognition method includes: acquiring hand images captured by each of the cameras, and grouping hand images corresponding to the same acquisition time into a group to obtain various combinations to be processed; inputting each combination to be processed into a preset prediction model to obtain the hand joint coordinates corresponding to each combination; obtaining the target hand joint coordinates based on the hand joint coordinates, and performing gesture recognition based on the target hand joint coordinates to obtain the gesture recognition result.

2. The gesture recognition method as described in claim 1, characterized in that, The step of obtaining the target hand joint coordinates based on the coordinates of each hand joint includes: obtaining the preset weights corresponding to each of the processing combinations; and, based on the correspondence between the hand joint coordinates and the processing combinations, and the correspondence between the processing combinations and the weights, performing a weighted summation of the hand joint coordinates to obtain the target hand joint coordinates.

3. The gesture recognition method as described in claim 1, characterized in that, Before the step of grouping hand images corresponding to the same acquisition time into a group to obtain various combinations to be processed, the method further includes: extracting hand features from each of the hand images and converting each of the hand features into a feature matrix; the step of grouping hand images corresponding to the same acquisition time into a group to obtain various combinations to be processed includes: grouping feature matrices corresponding to the same acquisition time into a group to obtain various combinations to be processed.

4. The gesture recognition method as described in claim 3, characterized in that, The prediction model includes an encoder and a decoder, wherein the encoder includes self-attention modules and the decoder includes cross-attention modules. The step of inputting each of the combinations to be processed into the preset prediction model to obtain the hand joint coordinates corresponding to each combination to be processed includes: extracting latent features of each feature matrix in the combination to be processed through each self-attention module and passing the latent features to each cross-attention module; and performing prediction based on the latent features through each cross-attention module to obtain the hand joint coordinates corresponding to the combination to be processed, wherein the joint distribution represented by the latent features is the same as the joint distribution represented by the hand joint coordinates.

5. The gesture recognition method as described in claim 1, characterized in that, The step of acquiring the hand images captured by each of the cameras includes: acquiring the speed of hand joint movement, and determining a target speed range corresponding to the speed in a preset speed range; acquiring the hand images captured by each of the cameras based on a preset prediction step size corresponding to the target speed range.

6. The gesture recognition method as described in claim 1, characterized in that, The step of performing gesture recognition based on the coordinates of the target hand joints to obtain the gesture recognition result includes: extracting gesture features from the coordinates of the target hand joints; inputting the gesture features into a preset gesture classification model to obtain the gesture recognition result, wherein the gesture classification model is trained using the gesture features of preset gesture data as model input data and the actual gesture classification results of the gesture data as model training labels.

7. The gesture recognition method as described in claim 1, characterized in that, After the step of obtaining the target hand joint coordinates based on the coordinates of each hand joint, the method further includes: rendering a virtual hand in the virtual page of the head-mounted display device based on the target hand joint coordinates, and receiving feedback information corresponding to the virtual hand; if the feedback information indicates that the virtual hand simulation is correct, then performing the step of performing gesture recognition based on the target hand joint coordinates to obtain the gesture recognition result.

8. A gesture recognition system, characterized in that, The gesture recognition system is applied to a head-mounted display device, which is equipped with various cameras. The gesture recognition system includes: an image acquisition module, used to acquire hand images captured by each of the cameras, and group hand images corresponding to the same acquisition time into a group to obtain various combinations to be processed; a prediction module, used to input each combination to be processed into a preset prediction model to obtain the hand joint coordinates corresponding to each combination to be processed; and a recognition module, used to obtain the target hand joint coordinates based on the hand joint coordinates, and perform gesture recognition based on the target hand joint coordinates to obtain the gesture recognition result.

9. A head-mounted display device, characterized in that, The head-mounted display device includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the gesture recognition method as described in any one of claims 1 to 7.

10. A storage medium, characterized in that, The storage medium is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, it implements the steps of the gesture recognition method as described in any one of claims 1 to 7.

Images