Method, device, electronic device and storage medium for predicting a gesture

Abstract

The application provides a method and device for predicting a gesture, an electronic device and a storage medium. The method comprises obtaining a body feature of a target to be predicted; inputting the body feature into two MLP models obtained through training respectively to obtain two single-hand features; and obtaining a predicted gesture corresponding to the target to be predicted based on the two single-hand features. The process of training each MLP model comprises: constraining the output of the MLP model based on the single-hand true value corresponding to the MLP model, so as to obtain the single-hand features of two hands through two MLP models trained separately, and avoid the unnatural problem of the predicted gesture caused by the asymmetric motion of the two hands.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of neural network models, and in particular to a method and device for predicting gestures, electronic equipment and a storage medium. BACKGROUND

[0002] This section is intended to provide background information to facilitate a better understanding of embodiments of the present application recited in the claims. The description herein does not constitute admission that the information provided herein is prior art merely because it is included herein.

[0003] In recent years, with the rapid development of 3D gesture estimation and reconstruction technology based on deep learning, the related method of predicting 3D gestures from the upper body has been vigorously promoted. This non-verbal hand-body interaction modeling plays an important role in various virtual character scenarios, including human-computer interaction, speech gesture synthesis, holographic transmission, etc., and has a wide application prospect and huge market value. However, the method of predicting 3D gestures from the upper body movement in the related art often ignores the asymmetric movement between the two hands, but generates both hands in an overall manner, resulting in unnatural prediction results. SUMMARY

[0004] Therefore, the purpose of the present application is to provide a method and device for predicting gestures, electronic equipment and a storage medium.

[0005] In order to achieve the above purpose, the present application provides a method for predicting gestures, comprising:

[0006] obtaining the body features of a target to be predicted;

[0007] inputting the body features into two MLP models trained respectively to obtain two single-hand features;

[0008] obtaining a predicted gesture corresponding to the target to be predicted based on the two single-hand features;

[0009] wherein the process of training each MLP model comprises: constraining the output of the MLP model based on the single-hand true value corresponding to the MLP model.

[0010] Based on the same inventive concept, the present application also provides a device for predicting gestures, comprising:

[0011] an acquisition module for acquiring the body features of a target to be predicted;

[0012] a feature extraction module for inputting the body features into two MLP models trained respectively to obtain two single-hand features;

[0013] a prediction module for obtaining a predicted gesture corresponding to the target to be predicted based on the two single-hand features;

[0014] The process of training each of the MLP models comprises: constraining an output of the MLP model based on a single-hand ground truth corresponding to the MLP model.

[0015] Based on the same inventive concept, the example embodiments of the present application further provide an electronic device comprising a memory, a processor, and a computer program stored in the memory and executable by the processor, wherein the processor implements the method of predicting a gesture as described above when executing the program.

[0016] Based on the same inventive concept, the example embodiments of the present application further provide a non-transitory computer-readable storage medium storing computer instructions for causing a computer to execute the method of predicting a gesture as described above.

[0017] As can be seen from the above, the method of predicting a gesture, the device, the electronic device, and the storage medium provided by the present application obtain a body feature of a target to be predicted; input the body feature into two MLP models trained respectively to obtain two single-hand features; and obtain a predicted gesture corresponding to the target to be predicted based on the two single-hand features. The process of training each of the MLP models comprises: constraining an output of the MLP model based on a single-hand ground truth corresponding to the MLP model. Thus, two single-hand features of two hands are obtained through two MLP models trained separately, and the problem of unnatural predicted gesture caused by asymmetric motion of two hands is avoided. BRIEF DESCRIPTION OF DRAWINGS

[0018] In order to more clearly illustrate the technical solutions in the present application or related art, the following will briefly introduce the drawings needed to be used in the embodiments or related art descriptions. Obviously, the drawings in the following description are only examples of the present application, and other drawings can be obtained by those skilled in the art without creative effort.

[0019] Figure 1 A flowchart of a method of predicting a gesture according to an embodiment of the present application;

[0020] Figure 2 A flowchart of another method of predicting a gesture according to an embodiment of the present application;

[0021] Figure 3 A structural diagram of a device of predicting a gesture according to an embodiment of the present application;

[0022] Figure 4 A structural diagram of a specific electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0023] The principles and spirits of the present application will be described below with reference to several exemplary embodiments. It should be understood that these embodiments are given only so that those skilled in the art can better understand and implement the present application, and do not limit the scope of the present application in any way. On the contrary, these embodiments are provided so that the present application is more thorough and complete, and the scope of the present application is fully conveyed to those skilled in the art.

[0024] According to the embodiments of the present application, a method, system, electronic device and storage medium for predicting a gesture are provided.

[0025] In this document, it should be understood that any number of elements in the drawings are used for illustration only and not limitation, and any naming is only for differentiation and does not have any limiting meaning.

[0026] The principles and spirits of the present application will be described below with reference to several exemplary embodiments. It should be understood that these embodiments are given only so that those skilled in the art can better understand and implement the present application, and do not limit the scope of the present application in any way. On the contrary, these embodiments are provided so that the present application is more thorough and complete, and the scope of the present application is fully conveyed to those skilled in the art. SUMMARY

[0028] The method for predicting 3D gestures from upper body movements in the related art often ignores the asymmetric motion between the two hands, for example: when one hand moves, the other hand may be stationary or slowly moving. Instead, the two hands are directly generated from the overall perspective, resulting in unnatural predicted results.

[0029] To solve the above problems, the present application provides a method for predicting a gesture, specifically comprising:

[0030] obtaining a body feature of a target to be predicted; inputting the body feature into two MLP models obtained by training respectively to obtain two single-hand features; obtaining a predicted gesture corresponding to the target to be predicted based on the two single-hand features; wherein the process of training each MLP model comprises: constraining the output of the MLP model based on the single-hand ground truth corresponding to the MLP model, so as to obtain two single-hand features of two hands respectively through two separately trained MLP models, avoiding the unnatural problem of the predicted gesture caused by the asymmetric motion of the two hands.

[0031] After introducing the basic principles of the present application, various non-limiting embodiments of the present application will be specifically introduced below.

[0032] Overview of application scenarios

[0033] In some specific application scenarios, the method for predicting gestures provided in the present application can be applied to various platforms or systems designed for gesture prediction. As an example, the application scenario includes at least one server and at least one terminal. The server and the terminal can communicate with each other through a network to realize data transmission. The network can be a wired network or a wireless network, which is not limited in the present application.

[0034] The server can be a server providing various services. Specifically, the server can be used to provide background services for an application program running on the terminal. Optionally, in some implementations, the method for predicting gestures provided in the embodiments of the present application can be executed by the server. The server can be hardware or software. When the server is hardware, it can be implemented as a distributed server cluster composed of multiple servers, or as a single server. When the server is software, it can be implemented as multiple software or software modules (for example, software or software modules used to provide distributed services), or as a single software or software module. The embodiments of the present application do not make specific limitations.

[0035] Optionally, the wireless network or wired network uses standard communication technology and / or protocol. The network is usually the Internet, but can also be any network, including but not limited to local area network (LAN), metropolitan area network (MAN), wide area network (WAN), mobile, wired or wireless network, private network or any combination of virtual private network. In some embodiments, technologies and / or formats including hyper text mark-up language (HTML), extensible markup language (XML) and the like are used to represent the data exchanged through the network. In addition, all or some links can be encrypted using conventional encryption technologies such as secure socket layer (SSL), transport layer security (TLS), virtual private network (VPN), internet protocol security (IPsec) and the like. In other embodiments, custom and / or dedicated data communication technologies can be used instead of or in addition to the above data communication technologies.

[0036] The method of predicting a gesture according to the example embodiments of the present application will be described below in combination with specific application scenarios. It should be noted that the above-mentioned application scenarios are only shown for the purpose of facilitating the understanding of the spirit and principles of the present application, and the embodiments of the present application are not limited in this respect. On the contrary, the embodiments of the present application can be applied to any applicable scenario.

[0037] Exemplary method

[0038] Reference Figure 1 The embodiments of the present application provide a method of predicting a gesture, and the execution subject of the method can be, but is not limited to, a server or a terminal device. The method comprises the following steps:

[0039] S101, obtaining a body feature of a target to be predicted.

[0040] In specific implementation, the target to be predicted can be a character image or an animal image with two hands, which can be a virtual image or a photo corresponding to a real person or object, and no limitation is made in this respect. The body feature of the target to be predicted can be extracted by any one of the methods of extracting features in the related art, and no limitation is made in this respect.

[0041] In some embodiments, obtaining the body feature of the target to be predicted specifically comprises:

[0042] Inputting the body joint information of the target to be predicted into the trained body encoder, and obtaining the body feature through the encoding of the body encoder.

[0043] In specific implementation, in order to accurately obtain the body feature, the body joint information of the target to be predicted can be inputted into the trained body encoder, and the body feature can be obtained through the encoding process of the body encoder. Optionally, the body joint information can be the position information of the body skeletal joint of the target to be detected, which can be represented by coordinates or vectors, and no limitation is made in this respect.

[0044] Considering that the hand gesture to be predicted is mainly related to the upper body, in some embodiments, the body joint information of the target to be predicted mainly refers to the joint information of the upper body.

[0045] S102, inputting the body feature into two trained MLP models respectively, and obtaining two single-hand features.

[0046] In a specific implementation, in the model training stage of predicting the gesture, two independent MLP (Multi-Layer Perceptron) models are trained through the first single-hand ground truth and the second single-hand ground truth respectively. It should be noted that the first and second single hands are mainly used to distinguish the left and right hands of the target to be predicted. After the model training is completed, the extracted body features can be input into the two trained MLP models to obtain two single-hand features. Since the two MLP models are trained separately, the features of each hand can be more accurately identified, and the unnatural prediction results caused by identifying the features of both hands together are avoided.

[0047] In some embodiments, constraining the output of the MLP model based on the single-hand ground truth corresponding to the MLP model comprises:

[0048] inputting the single-hand feature output by the MLP model into the single-hand autoencoder corresponding to the MLP model obtained through training, and obtaining a single-hand gesture through decoding of the single-hand autoencoder;

[0049] inputting the single-hand ground truth corresponding to the MLP model into the encoding of the single-hand autoencoder, and obtaining a reconstructed gesture through the encoding and decoding of the single-hand autoencoder;

[0050] constraining the single-hand gesture based on the reconstructed gesture.

[0051] In a specific implementation, in order to facilitate comparison of the single-hand feature output by the MLP model during model training, the output single-hand feature is first decoded through the decoding process of the single-hand autoencoder corresponding to the single-hand feature to obtain a single-hand gesture corresponding to the single-hand feature. At this time, the single-hand ground truth can be directly used to constrain the single-hand gesture, that is, by comparing the difference between the single-hand gesture and the single-hand ground truth to optimize the parameters of the MLP model. However, further consideration is given to the fact that the gesture output by the single-hand autoencoder is generally affected by the autoencoder itself, resulting in a certain deviation between the output gesture and the true gesture. In order to eliminate the influence of this deviation when comparing the ground truth, the single-hand ground truth can be first input into the encoding of the single-hand autoencoder, and a reconstructed gesture can be obtained through the encoding and decoding of the single-hand autoencoder. The reconstructed gesture is also affected by the autoencoder, and therefore, when the reconstructed gesture is used to constrain the single-hand gesture, the influence of the autoencoder can be offset, further ensuring the accuracy of the feature output by the MLP model.

[0052] S103, obtaining a predicted gesture corresponding to the target to be predicted based on the two single-hand features.

[0053] In practice, after obtaining two single-hand features, the predicted gesture corresponding to the target to be predicted can be obtained directly from these two single-hand features. Optionally, the two single-hand features can be input into a trained autoencoder, and the predicted gesture can be obtained directly through the decoding process of the autoencoder. Optionally, the predicted gesture can be represented by the position coordinates or vectors of the joints, and there is no limitation on this.

[0054] In some embodiments, the body features are a set of sequential features from multiple time points; obtaining the predicted gesture corresponding to the target to be predicted based on two of the single-hand features specifically includes:

[0055] For each hand feature output by the MLP model, the hand feature corresponding to the initial time is obtained, the spatial residual between the hand features corresponding to two adjacent time points is determined, and the hand features corresponding to all other time points other than the initial time point are determined based on the spatial residual and the hand feature at the initial time point; the hand features corresponding to the initial time point and the hand features corresponding to all other time points are combined to form a set of hand residual features for body features corresponding to each MLP model.

[0056] Based on the two sets of single-hand residual features, the predicted gesture corresponding to the target to be predicted is obtained.

[0057] In practice, the body joint information of the target to be predicted can be a set of body joint sequences during continuous body movement. Where T is the sequence length, and the corresponding body features of the target to be predicted are a set of sequence features at multiple time points, that is, each b in the keypoint sequence set B. t Each time step corresponds to a sequence feature, and each time step's sequence feature can be used to obtain a single-hand feature through an MLP model. To fit the spatial relationship between single-hand features corresponding to adjacent time steps, the initial single-hand feature is first obtained using an MLP model. Then, when obtaining single-hand features for subsequent time steps, the output of the MLP model is no longer used. Instead, the spatial residual between single-hand features corresponding to two adjacent time steps is first determined. Then, based on the spatial residual and the initial time's single-hand feature, the single-hand features corresponding to all other time steps are determined. Finally, the initial time's single-hand feature and the single-hand features corresponding to all other time steps are combined to form a single-hand residual feature set. The predicted gesture corresponding to the target to be predicted is obtained through this single-hand residual feature set. Each time step's single-hand feature in the single-hand residual feature set can yield the predicted gesture corresponding to the target to be predicted at that time, thus obtaining a set of predicted gestures corresponding to multiple time steps. Optionally, the single-hand feature corresponding to each time step can be input into a trained autoencoder, and the predicted gesture corresponding to that time step can be obtained through the autoencoder's decoding process.

[0058] It should be noted that, since the same body feature in the embodiment of the present application is obtained by two MLP models to obtain two single-hand features, i.e., the first single-hand feature and the second single-hand feature, the process of converting the single-hand feature output by one MLP model into the single-hand residual feature set is only demonstrated in the above embodiment, and the process of converting the single-hand feature output by the other MLP model into the single-hand residual feature set is similar, and thus will not be described herein.

[0059] In some embodiments, the single-hand feature corresponding to each time point except the initial time point is determined by the following formula:

[0060]

[0061] wherein, represents the single-hand feature corresponding to the t+1 time point, represents the single-hand feature corresponding to the t time point, represents the spatial dependency between the single-hand feature corresponding to the t+1 time point and the single-hand feature corresponding to the t time point;

[0062] wherein, the process of determining the spatial dependency comprises:

[0063] determining a target spatial residual between the single-hand feature corresponding to the t+1 time point and the single-hand feature corresponding to the t time point, performing matrix multiplication on the target spatial residual and the single-hand feature corresponding to the t time point after matrix transposition of the target spatial residual, and performing a softmax operation on the result of the matrix multiplication to obtain the spatial dependency.

[0064] In specific implementation, since the single-hand feature corresponding to the initial time point is obtained initially, the single-hand feature corresponding to each time point after the initial time point can be calculated in sequence by the above formula. Optionally, the result of the matrix multiplication is subjected to a softmax (logistic regression) operation, i.e., the result of the matrix multiplication is normalized to a value between 0 and 1.

[0065] In order to accurately depict the spatial difference between the single-hand features corresponding to adjacent time points, in some embodiments, the spatial residual between the single-hand features corresponding to two adjacent time points is determined, specifically comprising:

[0066] determining the sequence feature corresponding to the current time point from the sequence feature set of the plurality of time points;

[0067] determining the first spatial residual most similar to the sequence feature corresponding to the current time point from the preset spatial residual set, and determining the first spatial residual as the spatial residual between the single-hand feature corresponding to the current time point and the single-hand feature corresponding to the next time point.

[0068] In actual implementation, the first spatial residual closest to the sequence feature corresponding to the current moment can be determined from the preset spatial residual set by using cosine similarity.

[0069] In some embodiments, the process of determining the preset spatial residual set comprises:

[0070] In the model training phase of predicting the gesture, an initial spatial residual set is constructed, an original spatial residual closest to the target sample body feature is determined from the initial spatial residual set, and the original spatial residual is constantly updated based on a preset residual update formula until the model training is completed, the updating of the original spatial residual is ended, the updating of the initial spatial residual set is completed, and the initial spatial residual set after the updating is determined as the preset spatial residual set.

[0071] The preset residual update formula is:

[0072]

[0073] wherein λ represents a first preset weight, represents a sample single-hand feature corresponding to t moment, m r represents a spatial residual before updating, m r+1 represents a spatial residual after updating.

[0074] In actual implementation, in the model training phase of predicting the gesture, an initial spatial residual set can be constructed according to needs, a plurality of spatial residuals are set in the initial spatial residual set, then an original spatial residual closest to the target sample body feature is determined from the initial spatial residual set by using cosine similarity, and the original spatial residual is constantly updated according to a preset residual update formula, so that the spatial residuals in the preset spatial residual set can more accurately represent the spatial difference between adjacent single-hand features and the spatial relationship between the body and each hand. Optionally, the first preset weight λ can be obtained by experiments according to needs, and optionally, the first preset weight λ can be set to 0.8.

[0075] In order to ensure that the motion of the two hands is consistent with the motion sequence of the body joint in time, in some embodiments, the predicted gesture corresponding to the target to be predicted is obtained based on the two single-hand residual feature sets, and the method specifically comprises:

[0076] The body joint information of the target to be predicted is input into the motion encoder obtained by training, and the body motion embedding feature of the target to be predicted is obtained by encoding of the motion encoder.

[0077] For each of the single-hand residual feature set, a target single-hand motion embedding feature most consistent with the body motion embedding feature is determined in the preset single-hand motion embedding feature set, and after the target single-hand motion embedding feature is subjected to a softmax operation, a vector dot product is performed with the single-hand residual feature set, and a result of the vector dot product is added to the single-hand residual feature set to obtain a single-hand time sequence feature set;

[0078] A predicted gesture corresponding to the to-be-predicted target is obtained based on the two single-hand time sequence feature sets.

[0079] In specific implementation, the body motion embedding feature output by the motion encoder is used to represent the time variation of the body joint in a sequence, and then the preset single-hand motion embedding feature set is used to represent the time variation of the single-hand feature in multiple sequences, and the cosine similarity is used to determine the target single-hand motion embedding feature most consistent with the body motion embedding feature from the preset single-hand motion embedding feature set, and the single-hand time sequence feature set is obtained through the target single-hand motion embedding feature, so as to ensure that the motion of the finally predicted gesture is consistent with the motion of the body joint of the to-be-predicted target in time sequence.

[0080] In some embodiments, the process of determining the preset single-hand motion embedding feature set comprises:

[0081] In the model training phase of the predicted gesture, an initial single-hand motion embedding feature set is constructed, an original single-hand motion embedding feature most similar to the target sample body motion embedding feature is determined from the initial single-hand motion embedding feature set, and the original single-hand motion embedding feature is constantly updated based on a preset motion update formula until the model training is completed, the updating of the original single-hand motion embedding feature is ended, the update of the initial single-hand motion embedding feature set is completed, and the initial single-hand motion embedding feature set after the update is determined as the preset single-hand motion embedding feature set;

[0082] The preset motion update formula is:

[0083]

[0084] wherein, a represents a second preset weight, represents a sample body motion embedding feature, m n represents a single-hand motion embedding feature before update, m n+1 represents a single-hand motion embedding feature after update.

[0085] In the model training phase of predicting the gesture, the process of obtaining the preset single-hand motion embedding feature set is similar to the process of obtaining the preset spatial residual set. Only when updating the single-hand motion embedding feature, the body motion embedding feature is used instead of the single-hand feature. Details are not repeated. Optionally, the second preset weight a can be obtained by experiment according to the need. The second preset weight a can be the same as or different from the value of the first preset weight λ. Details are not limited.

[0086] In some embodiments, the predicted gesture corresponding to the to-be-predicted target is obtained based on the two single-hand timing feature sets, specifically comprising:

[0087] The body feature is input into the encoder of the first transformer model obtained by training to obtain a query feature;

[0088] For each single-hand timing feature set, the single-hand timing feature set is input into the encoder of the transformer model corresponding to the single-hand timing feature set to obtain a key feature and a value feature. The query feature is respectively operated with the key feature and the value feature by a preset number of multi-head attention mechanisms to obtain a synchronous single-hand feature set.

[0089] The features of the two synchronous single-hand feature sets are spliced to obtain a double-hand feature set.

[0090] After the double-hand feature set is input into the encoder of the second transformer model obtained by training and operated by a preset number of multi-head attention mechanisms, the predicted gesture corresponding to the to-be-predicted target is obtained through the fully connected layer of the second transformer model.

[0091] In a specific implementation, the query feature obtained by the first transformer model represents the body feature information synchronized with the two-hand motion, and the synchronization between the single-hand feature and the body feature can be further improved by performing the multi-head attention mechanism operation on the query feature and the key feature and the value feature for a preset number of times. Optionally, the prediction number can be set as needed, and the prediction number is not limited, and the prediction number can be set as 3 times. After obtaining the synchronized single-hand feature set, in order to further combine the features of the two hands and encourage the combined two-hand feature to be more time-sequential with the body motion, the features of the two synchronized single-hand feature sets are spliced to obtain a two-hand feature set; and then the two-hand feature set is input into the encoder of the second transformer model obtained by training to perform the multi-head attention mechanism operation for a preset number of times. Optionally, when performing the multi-head attention mechanism operation in the encoder of the second transformer model, the query feature and the key feature and the value feature corresponding to the two-hand feature set are obtained first, and then the query feature is subjected to the multi-head attention mechanism operation with the key feature and the value feature for a preset number of times. Finally, the predicted gesture corresponding to the to-be-predicted target is obtained through the fully connected layer of the second transformer model. Optionally, the fully connected layer can be set as two layers. Optionally, if the obtained body feature is a sequence feature set of multiple time points, that is, the input is a sequence set of joint position points with the prediction target, the corresponding predicted gesture is a gesture sequence set including multiple predicted gestures.

[0092] It should be noted that in the model training stage of the predicted gesture, the minimum absolute error (Least Abosulote Error, LAE) between the predicted gesture and the true value can be calculated as the loss function of the model. The weight parameters of the optimization network are iteratively updated by the gradient descent algorithm, so as to obtain a natural predicted 3D gesture. In addition, in some embodiments, a motion discriminator can also be used to ensure that the predicted gesture is natural and continuous.

[0093] In order to realize the diversity of the output predicted gesture, in some embodiments, after obtaining the predicted gesture corresponding to the to-be-predicted target based on the two single-hand features, the method further comprises:

[0094] a gesture prototype feature corresponding to the predicted gesture;

[0095] obtaining a perturbation parameter of the gesture prototype feature based on gradient-based Markov chain Monte Carlo sampling;

[0096] generating another preset gesture corresponding to the predicted gesture based on the perturbation parameter and the gesture prototype feature.

[0097] In some embodiments, the gesture prototype feature corresponding to the predicted gesture specifically includes:

[0098] inputting the predicted gesture into the trained two-hand encoder, and obtaining two-hand features through the encoding process of the two-hand encoder;

[0099] determining a gesture prototype feature closest to the two-hand features from a preset gesture prototype feature set.

[0100] In implementation, the specific process of determining the gesture prototype feature corresponding to the predicted gesture can be selected as needed, for example, the two-hand features obtained through the encoding process of the two-hand encoder can be directly used as the gesture prototype feature. However, considering that the two-hand features obtained through the encoding process of the two-hand encoder may exceed the recognition range of the model itself, it is easy to cause the finally generated other preset gestures to be uncontrollable and greatly different from the originally generated predicted gesture. Therefore, a preset gesture prototype feature set is prepared in advance, and the gesture prototype feature closest to the two-hand features is determined from the preset gesture prototype feature set through cosine similarity.

[0101] In some embodiments, the process of determining the preset gesture prototype feature set includes:

[0102] In the model training phase of the predicted gesture, an initial gesture prototype feature set is constructed, an original gesture prototype feature closest to the target sample two-hand feature is determined from the initial gesture prototype feature set, and the original gesture prototype feature is constantly updated based on a preset prototype update formula until the model training is completed, the updating of the original gesture prototype feature is ended, the update of the initial gesture prototype feature set is completed, and the initial gesture prototype feature set after the update is determined as the preset single-hand motion embedding feature set;

[0103] wherein the preset prototype update formula is:

[0104]

[0105] wherein β represents a third preset weight, represents a sample two-hand feature, m p represents a single-hand motion embedding feature before update, m p+1 represents a single-hand motion embedding feature after update.

[0106] Reference Figure 2Fig. 2 shows a flowchart of another embodiment of the method for predicting a gesture, wherein the body encoder is first inputted with the predicted upper body (i.e. the coordinates of the joints of the upper body), and the body encoder outputs the body feature through the encoding process of the body encoder, and then the body feature is inputted into the first MLP model and the second MLP model respectively to obtain the first single-hand feature and the second single-hand feature respectively, and then the first single-hand feature and the body feature are inputted into the first spatial residual memory pool (the first preset spatial residual set) to obtain the first single-hand residual feature set, and the second single-hand feature and the body feature are inputted into the second spatial residual memory pool (the second preset spatial residual set) to obtain the second single-hand residual feature set, and then the body feature, the first single-hand residual feature set and the second single-hand residual feature set are inputted into the first temporal motion memory pool (the first preset single-hand motion embedding feature set) and the second temporal motion memory pool (the second preset single-hand motion embedding feature set) respectively to obtain the first single-hand temporal feature set and the second single-hand temporal feature set, and then the body feature is inputted into the body Ttansformer encoder to obtain the query feature Q, the first single-hand temporal feature set is inputted into the first single-hand Ttansformer encoder to obtain the first key feature and the first value feature, the second single-hand temporal feature set is inputted into the second single-hand Ttansformer encoder to obtain the second key feature and the second value feature, and the query feature Q is inputted into the first key feature and the first value feature, and the second key feature and the second value feature respectively to perform multiple multi-head attention mechanism operations to obtain the first synchronous single-hand feature set and the second synchronous single-hand feature set, and then the first synchronous single-hand feature set and the second synchronous single-hand feature set are spliced to obtain the double-hand feature set, and the double-hand feature set is inputted into the double-hand Ttansformer encoder, and the query feature Q is inputted into the double-hand Ttansformer encoder to perform multiple multi-head attention mechanism operations, and then the predicted gesture is outputted through the fully connected layer, and after obtaining the predicted gesture, in order to obtain other predicted gestures with diversity, the predicted gesture is inputted into the double-hand decoder to obtain the double-hand feature corresponding to the predicted gesture, and then the double-hand feature is inputted into the double-hand prototype memory pool (the preset gesture prototype feature set) to obtain the gesture prototype feature, and then the randomly initialized noise is inputted into the sampling head, and optionally, the randomly initialized noise conforms to the Gaussian distribution. Optionally, the sampling head can be an MLP model obtained through training, and the perturbation parameter of the gesture prototype feature is obtained through the gradient-based Markov chain Monte Carlo sampling (MCMC) in the sampling head, and finally the perturbation parameter and the gesture prototype feature are inputted into the double-hand decoder to output other predicted gestures.

[0107] The method for predicting a gesture provided in the application comprises the following steps: acquiring a body feature of a target to be predicted; inputting the body feature into two MLP models trained respectively to obtain two single-hand features; and obtaining a predicted gesture corresponding to the target to be predicted based on the two single-hand features.

[0108] Exemplary device

[0109] Based on the same inventive concept, the application also provides a device for predicting a gesture corresponding to any of the above-mentioned method embodiments.

[0110] Reference Figure 3 The device for predicting a gesture comprises:

[0111] An acquisition module 201 is configured to acquire a body feature of a target to be predicted.

[0112] A feature extraction module 202 is configured to input the body feature into two MLP models trained respectively to obtain two single-hand features.

[0113] A prediction module 203 is configured to obtain a predicted gesture corresponding to the target to be predicted based on the two single-hand features.

[0114] The process of training each MLP model comprises the following steps: constraining the output of the MLP model based on the single-hand true value corresponding to the MLP model.

[0115] For the convenience of description, the above system is described in various modules in terms of functions. Of course, the functions of the modules can be implemented in one or more software and / or hardware in the implementation of the application.

[0116] The system of the above-mentioned embodiments is used to implement the method for predicting a gesture in any of the above-mentioned embodiments, and has the beneficial effects of the corresponding method embodiments, which will not be described herein.

[0117] Based on the same inventive concept, the application also provides an electronic device corresponding to the method of any of the above embodiments, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the method of predicting a gesture according to any of the embodiments.

[0118] Figure 4 A more specific hardware structure of an electronic device is shown in this embodiment. The device can include a processor 1010, a memory 1020, an input / output interface 1030, a communication interface 1040, and a bus 1050. The processor 1010, the memory 1020, the input / output interface 1030, and the communication interface 1040 are connected to each other through the bus 1050 for internal communication within the device.

[0119] The processor 1010 can be implemented by a general-purpose CPU (Central Processing Unit), a microprocessor, an ASIC (Application Specific Integrated Circuit), or one or more integrated circuits, etc., for executing related programs to implement the technical solutions provided by the embodiments of the present specification.

[0120] The memory 1020 can be implemented by a ROM (Read Only Memory), a RAM (Random Access Memory), a static storage device, a dynamic storage device, etc. The memory 1020 can store an operating system and other application programs. When the technical solutions provided by the embodiments of the present specification are implemented by software or firmware, the related program codes are stored in the memory 1020 and executed by the processor 1010.

[0121] The input / output interface 1030 is used to connect input / output modules to realize information input and output. The input / output modules can be configured as components in the device (not shown in the figure) or externally connected to the device to provide corresponding functions. The input devices can include a keyboard, a mouse, a touch screen, a microphone, various sensors, etc., and the output devices can include a display, a speaker, a vibrator, an indicator light, etc.

[0122] The communication interface 1040 is used to connect a communication module (not shown in the figure) to realize the communication interaction between the device and other devices. The communication module can realize communication through wired means (such as USB, network cable, etc.) or through wireless means (such as mobile network, WIFI, Bluetooth, etc.).

[0123] The bus 1050 includes a path for transferring information between the various components (for example, the processor 1010, the memory 1020, the input / output interface 1030, and the communication interface 1040) of the device.

[0124] It should be noted that although the above device only shows the processor 1010, the memory 1020, the input / output interface 1030, the communication interface 1040 and the bus 1050, in the specific implementation process, the device can also include other components necessary for normal operation. In addition, those skilled in the art can understand that the above device can also only contain the components necessary to implement the embodiments of the present application, and does not have to contain all the components shown in the figure.

[0125] The electronic device of the above embodiment is used to implement the method of predicting the gesture corresponding to any of the above embodiments, and has the beneficial effects of the corresponding method embodiments, which are not repeated here.

[0126] Exemplary program product

[0127] Based on the same inventive concept, corresponding to the method of any of the above embodiments, the present application also provides a non-transitory computer readable storage medium, which stores computer instructions for causing the computer to execute the method of predicting the gesture according to any of the above embodiments.

[0128] The computer readable medium of the present embodiment includes permanent and non-permanent, removable and non-removable media, which can be realized by any method or technology to store information. The information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassette, magnetic tape disk storage or other magnetic storage device, or any other non-transmission medium that can be used to store information that can be accessed by a computing device.

[0129] The computer instructions stored in the storage medium of the above embodiment are used to cause the computer to execute the method of predicting the gesture according to any of the above embodiments, and have the beneficial effects of the corresponding method embodiments, which are not repeated here.

[0130] Those skilled in the art should understand that the above discussion of any embodiment is merely exemplary and is not intended to be limiting of the scope of the application (including the claims) it being recognized that the foregoing description and examples have been provided merely for the purpose of explanation and are not intended to limit the application of the principles of the present application thereto as set forth in the following claims.

[0131] In addition, to simplify the description and discussion, and so as not to make the embodiments of the present application difficult to understand, the well-known power / ground connections to integrated circuit (IC) chips and other components can or can not be shown in the provided drawings. Further, devices can be shown in block diagram form so as to avoid making the embodiments of the present application difficult to understand, and this also takes into account the fact that details regarding implementation of these block diagram devices are highly dependent on the platform to which the embodiments of the present application are to be implemented (i.e., these details should be well within the understanding of one of skill in the art). Where specific details (e.g., circuitry) are set forth in order to describe an illustrative embodiment of the present application, it should be understood that the embodiments of the present application can be practiced without these specific details or with an equivalent arrangement.

[0132] While the present application has been described with respect to a specific embodiment thereof, many alternatives, modifications and variations will be apparent to those skilled in the art in light of the foregoing description. For example, other memory architectures (e.g., dynamic RAM (DRAM)) can use the embodiments discussed.

[0133] Embodiments of the present application are intended to cover all such alternatives, modifications and variations as falling within the scope of the broadest possible interpretation of the appended claims. Accordingly, any and all such modifications, variations or equivalents that fall within the spirit and scope of the embodiments of the present application should also be included.

Claims

1. A method for predicting gestures, characterized in that, include: Obtain the physical characteristics of the target to be predicted; The body features are input into two MLP models obtained through training, resulting in two single-hand features; Based on the two single-hand features, a predicted gesture corresponding to the target to be predicted is obtained; The process of training each MLP model includes: constraining the output of the MLP model based on the one-handed ground truth value corresponding to the MLP model; The body features are a set of sequential features from multiple time points; the predicted gesture corresponding to the target to be predicted is obtained based on two of the single-hand features, specifically including: For each hand feature output by the MLP model, the hand feature corresponding to the initial time step is obtained, the spatial residual between the hand features corresponding to two adjacent time steps is determined, and the hand features corresponding to all other time steps other than the initial time step are determined based on the spatial residual and the hand feature at the initial time step. Based on the single-hand features corresponding to the initial time and the single-hand features corresponding to all other times, a set of single-hand residual features for the body features is formed for each MLP model; Based on the two sets of single-hand residual features, the predicted gesture corresponding to the target to be predicted is obtained.

2. The method according to claim 1, characterized in that, Obtaining the physical characteristics of the target to be predicted specifically includes: The body joint information of the target to be predicted is input into the trained body encoder, and the body features are obtained through the encoding of the body encoder.

3. The method according to claim 1, characterized in that, Constraining the output of the MLP model based on the one-handed truth value corresponding to the MLP model includes: The single-hand features output by the MLP model are input into the trained single-hand autoencoder corresponding to the MLP model, and the single-hand gesture is obtained by decoding through the single-hand autoencoder. The single-hand ground truth value corresponding to the MLP model is input into the encoding of the single-hand autoencoder, and the reconstructed gesture is obtained through the encoding and decoding of the single-hand autoencoder; The reconstructed gesture constrains the single-handed gesture.

4. The method according to claim 1, characterized in that, The single-handed characteristics for all times other than the initial time are determined by the following formula: in, This represents the single-handed feature at time t+1. This represents the single-handed characteristic at time t. This represents the spatial dependency between the single-handed feature at time t+1 and the single-handed feature at time t; The process of determining the spatial dependency includes: Determine the target spatial residual between the single-handed feature at time t+1 and the single-handed feature at time t. Perform matrix transpose on the target spatial residual and multiply it with the single-handed feature at time t. Then, perform a softmax operation on the result of the matrix multiplication to obtain the spatial dependency.

5. The method according to claim 1, characterized in that, Determining the spatial residual between the one-handed features corresponding to two adjacent time points specifically includes: Determine the sequence features corresponding to the current time from the set of sequence features at the multiple time points; The first spatial residual that is closest to the sequence feature corresponding to the current time is determined from the preset set of spatial residuals, and the first spatial residual is determined as the spatial residual between the single-hand feature corresponding to the current time and the single-hand feature corresponding to the next time.

6. The method according to claim 5, characterized in that, The process of determining the preset spatial residual set includes: During the model training phase for predicting gestures, an initial spatial residual set is constructed. The original spatial residual that is most similar to the body features of the target sample is determined from the initial spatial residual set. The original spatial residual is continuously updated based on a preset residual update formula until the model training is completed. Then, the update of the original spatial residual ends to complete the update of the initial spatial residual set. The updated initial spatial residual set is then determined as the preset spatial residual set. The preset residual update formula is as follows: Where λ represents the first preset weight, Let m represent the single-handed feature of the sample at time t. r m represents the spatial residual before the update. r+1 This represents the updated spatial residual.

7. The method according to claim 1, characterized in that, Based on the two sets of single-hand residual features, the predicted gesture corresponding to the target to be predicted is obtained, specifically including: The body joint information of the target to be predicted is input into the trained motion encoder, and the body motion embedding features of the target to be predicted are obtained through the encoding of the motion encoder. For each set of single-hand residual features, a target single-hand motion embedding feature that is most consistent with the body motion embedding feature is determined in the preset set of single-hand motion embedding features. After performing a softmax operation on the target single-hand motion embedding feature, a vector dot product is performed with the set of single-hand residual features. The result of the vector dot product is added to the set of single-hand residual features to obtain the single-hand temporal feature set. Based on the two sets of single-handed temporal features, the predicted gesture corresponding to the target to be predicted is obtained.

8. The method according to claim 7, characterized in that, The process of determining the preset single-handed motion embedding feature set includes: During the model training phase for predicting gestures, an initial set of single-hand motion embedding features is constructed. From the initial set of single-hand motion embedding features, the original single-hand motion embedding features that are most similar to the body motion embedding features of the target sample are determined. The original single-hand motion embedding features are continuously updated based on a preset motion update formula until the model training is completed. At this point, the update of the original single-hand motion embedding features is stopped to complete the update of the initial set of single-hand motion embedding features. The updated initial set of single-hand motion embedding features is then determined as the preset set of single-hand motion embedding features. The preset motion update formula is as follows: Where α represents the second preset weight, m represents the body motion embedding features of the sample. n This represents the single-handed motion embedding features before the update, m n+1 This represents the updated single-handed motion embedding features.

9. The method according to claim 7, characterized in that, Based on the two sets of single-handed temporal features, the predicted gesture corresponding to the target to be predicted is obtained, specifically including: The body features are input into the encoder of the first transformer model obtained through training to obtain the query features; For each of the single-hand temporal feature sets, the single-hand temporal feature set is input into the encoder of the transformer model corresponding to the single-hand temporal feature set to obtain key features and value features. The query features are then subjected to a preset number of multi-head attention mechanism operations with the key features and the value features to obtain a synchronous single-hand feature set. The features of the two synchronous single-hand feature sets are concatenated to obtain a two-hand feature set; The hand feature set is input into the encoder of the trained second transformer model for a preset number of multi-head attention operations, and then the predicted gesture corresponding to the target to be predicted is obtained through the fully connected layer of the second transformer model.

10. The method according to any one of claims 1 to 9, characterized in that, After obtaining the predicted gesture corresponding to the target to be predicted based on the two single-hand features, the method further includes: Confirm the hand gesture prototype features corresponding to the predicted hand gesture; The perturbation parameters of the gesture prototype feature are obtained by gradient-based Markov chain Monte Carlo sampling. Based on the perturbation parameters and the gesture prototype features, other preset gestures corresponding to the predicted gesture are generated.

11. The method according to claim 10, characterized in that, The hand gesture prototype features corresponding to the predicted hand gesture specifically include: The predicted gesture is input into the trained two-hand encoder, and the two-hand features are obtained through the encoding process of the two-hand encoder. The hand gesture prototype feature that is closest to the hand features is determined from the preset set of hand gesture prototype features.

12. A device for predicting gestures, characterized in that, include: The acquisition module acquires the body features of the target to be predicted; The feature extraction module inputs the body features into two MLP models acquired during training, respectively, to obtain two single-hand features; The prediction module obtains the predicted gesture corresponding to the target to be predicted based on the two single-hand features. The process of training each MLP model includes: constraining the output of the MLP model based on the one-handed ground truth value corresponding to the MLP model; The body features are a set of sequential features from multiple time points; the predicted gesture corresponding to the target to be predicted is obtained based on two of the single-hand features, specifically including: For each hand feature output by the MLP model, the hand feature corresponding to the initial time step is obtained, the spatial residual between the hand features corresponding to two adjacent time steps is determined, and the hand features corresponding to all other time steps other than the initial time step are determined based on the spatial residual and the hand feature at the initial time step. Based on the single-hand features corresponding to the initial time and the single-hand features corresponding to all other times, a set of single-hand residual features for the body features is formed for each MLP model; Based on the two sets of single-hand residual features, the predicted gesture corresponding to the target to be predicted is obtained.

13. An electronic device, characterized in that, The method includes a memory, a processor, and a computer program stored in the memory and executable by the processor, wherein the processor, when executing the program, implements the method as described in any one of claims 1 to 11.

14. A non-transitory computer-readable storage medium, characterized in that, The non-transitory computer-readable storage medium stores computer instructions for causing a computer to perform the method according to any one of claims 1 to 11.

Images