3D gesture recognition model training method and device, equipment and medium

Abstract

The application discloses a 3D gesture recognition model training method and device, equipment and medium, and relates to the technical field of image processing. The method comprises the following steps: acquiring a general hand mesh model; acquiring a plurality of simulation hand mesh models and a first real hand mesh model according to the general hand mesh model; determining the simulation skeleton 3D coordinates corresponding to any simulation hand mesh model; determining the real skeleton 3D coordinates corresponding to any first real hand mesh model; training a to-be-trained 3D gesture recognition model according to the simulation hand mesh model, the first real hand mesh model, the simulation skeleton 3D coordinates and the real skeleton 3D coordinates to obtain a target 3D gesture recognition model. The method reduces the dependence of the discriminator as the 3D gesture recognition model on the real hand mesh model, thereby reducing the learning cost of the target 3D gesture recognition model.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing, and more particularly, to a 3D gesture recognition model training method and device, equipment and medium. BACKGROUND

[0002] 3D gesture recognition technology is very important for the interaction of intelligent head-mounted devices (such as XR). At present, intelligent head-mounted devices use 3D gesture recognition algorithms to realize 3D gesture recognition. However, mainstream 3D gesture recognition algorithms mainly use supervised learning methods to achieve this. The supervised learning method needs to rely on a large number of real data sets. For example, the gesture recognition algorithm MeGaTrack of Meta needs to collect more than 300W real data to achieve a gesture recognition effect with strong robustness. This greatly increases the learning cost of the 3D gesture recognition algorithm. SUMMARY

[0003] One object of the present application is to provide a new technical solution for 3D gesture recognition model training.

[0004] According to a first aspect of the present application, a 3D gesture recognition model training method is provided, comprising:

[0005] obtaining a general hand mesh model;

[0006] obtaining a plurality of simulation hand mesh models and a first real hand mesh model according to the general hand mesh model;

[0007] for any simulation hand mesh model, determining the simulation skeleton 3D coordinates corresponding to the simulation hand mesh model;

[0008] for any first real hand mesh model, determining the real skeleton 3D coordinates corresponding to the first real hand mesh model;

[0009] training a to-be-trained 3D gesture recognition model according to the simulation hand mesh model, the first real hand mesh model, the simulation skeleton 3D coordinates and the real skeleton 3D coordinates, to obtain a target 3D gesture recognition model.

[0010] Optionally, the method further comprises:

[0011] obtaining a plurality of accurate hand mesh models;

[0012] for any accurate hand mesh model, adjusting the morphology of the general hand mesh model to be consistent with the morphology of the accurate hand 3D model to obtain a simulation hand mesh model;

[0013] and obtaining a plurality of second real hand mesh models;

[0014] For any second real hand mesh model, adjusting the shape of the general hand mesh model to be consistent with the shape of the second real hand mesh model to obtain a first real hand mesh model.

[0015] Optionally, for any of the simulation hand mesh models, the simulation hand mesh model corresponding simulation skeleton 3D coordinates include:

[0016] For any of the simulation hand mesh models, according to the simulation hand mesh model and the preset mapping relationship, the simulation hand mesh model corresponding simulation skeleton 3D coordinates are determined, wherein the preset mapping relationship is used to reflect the corresponding relationship between the hand mesh model and the skeleton 3D coordinates;

[0017] The determination of the real skeleton 3D coordinates corresponding to any of the first real hand mesh models includes:

[0018] For any of the first real hand mesh models, according to the first real hand mesh model and the preset mapping relationship, the real skeleton 3D coordinates corresponding to the first real hand mesh model are determined.

[0019] Optionally, the to-be-trained 3D gesture model is an adversarial network, wherein the adversarial network includes a generator and a discriminator, and the target 3D gesture recognition model is the discriminator when the adversarial network converges;

[0020] The training of the to-be-trained 3D gesture recognition model according to the simulation hand mesh model, the first real hand mesh model, the simulation skeleton 3D coordinates and the real skeleton 3D coordinates to obtain a target 3D gesture recognition model includes:

[0021] According to the simulation hand mesh model and the generator, image noise is generated;

[0022] According to the image noise and the simulation hand mesh model, a simulation synthetic hand mesh model is generated;

[0023] According to the simulation synthetic hand mesh model and the corresponding simulation skeleton 3D coordinates, the discriminator is trained to obtain a trained discriminator;

[0024] According to the first real hand mesh model and the corresponding real skeleton 3D coordinates, the discrimination result of the discriminator is determined;

[0025] In the case where the discrimination result represents inaccurate discrimination, the discriminator is updated according to the discrimination result;

[0026] According to the discriminator, the 3D skeleton coordinates corresponding to the simulated synthetic hand mesh model are identified, and predicted skeleton 3D coordinates are obtained;

[0027] According to the predicted skeleton 3D coordinates, the parameters of the generator are updated;

[0028] The generation of image noise is repeated according to the simulated hand mesh model and the generator until the discrimination result represents accurate discrimination.

[0029] Optionally, the determination of the discrimination result of the discriminator according to the first real hand mesh model and the real skeleton 3D coordinates comprises:

[0030] According to the first real hand mesh model and the discriminator, predicted real skeleton 3D coordinates are determined;

[0031] The predicted real skeleton 3D coordinates are projected to a 2D coordinate system to obtain predicted real skeleton 2D coordinates;

[0032] The real skeleton 3D coordinates are projected to the 2D coordinate system to obtain real skeleton 2D coordinates;

[0033] According to the predicted real skeleton 2D coordinates and the real skeleton 2D coordinates, the discrimination result of the discriminator is determined.

[0034] Optionally, the generator comprises an encoder, a reverse conditional denoising network, and a decoder, wherein the encoder is configured to decode the simulated hand mesh model into a high-dimensional latent representation, the reverse conditional denoising network is configured to remove effective features from the high-dimensional latent representation to obtain a processed low-dimensional latent representation, and the decoder is configured to decode the processed low-dimensional latent representation into image noise.

[0035] Optionally, the decoder is specifically configured to decode the processed low-dimensional latent representation into image noise according to a preset noise constraint condition.

[0036] According to a second aspect of the present application, a 3D gesture recognition model training device is provided, comprising:

[0037] A first acquisition module is configured to acquire a general hand mesh model;

[0038] A second acquisition module is configured to acquire a plurality of simulated hand mesh models and a first real hand mesh model according to the general hand mesh model;

[0039] A first determination module is configured to determine, for any simulated hand mesh model, simulated skeleton 3D coordinates corresponding to the simulated hand mesh model;

[0040] a second determining module, configured to determine, for any first real hand mesh model, real skeleton 3D coordinates corresponding to the first real hand mesh model;

[0041] a training module, configured to train a to-be-trained 3D gesture recognition model according to the simulation hand mesh model, the first real hand mesh model, the simulation skeleton 3D coordinates and the real skeleton 3D coordinates, to obtain a target 3D gesture recognition model.

[0042] According to a third aspect of the present application, an electronic device is provided, which comprises the apparatus according to the second aspect;

[0043] Alternatively, the electronic device comprises a memory and a processor, the memory is configured to store computer instructions, and the processor is configured to invoke the computer instructions from the memory to execute the method according to any one of the first aspect.

[0044] According to a fourth aspect of the present application, a computer readable storage medium is provided, which stores a computer program, the computer program, when executed by a processor, implements the method according to any one of the first aspect.

[0045] The present application provides a 3D gesture recognition model training method, comprising: acquiring a general hand mesh model; acquiring a plurality of simulation hand mesh models and a first real hand mesh model according to the general hand mesh model; determining, for any simulation hand mesh model, simulation skeleton 3D coordinates corresponding to the simulation hand mesh model; determining, for any first real hand mesh model, real skeleton 3D coordinates corresponding to the first real hand mesh model; training a to-be-trained 3D gesture recognition model according to the simulation hand mesh model, the first real hand mesh model, the simulation skeleton 3D coordinates and the real skeleton 3D coordinates, to obtain a target 3D gesture recognition model. This method can eliminate the differences between the simulation hand mesh model and the first real hand mesh model, so that the simulation hand mesh model can act as a real hand mesh model, and further use the simulation hand mesh model and the first real hand mesh model to obtain the target 3D gesture recognition model. That is, the simulation hand mesh model is migrated to the real hand mesh model, which reduces the dependence of the to-be-trained 3D gesture recognition model on the real hand mesh model, thereby reducing the learning cost of the to-be-trained 3D gesture recognition model.

[0046] Other features and advantages of the present application will become apparent from the following detailed description of exemplary embodiments thereof, taken in conjunction with the accompanying drawings. BRIEF DESCRIPTION OF DRAWINGS

[0047] The accompanying drawings incorporated in and forming a part of the specification, illustrate embodiments of the present application and, together with the description, serve to explain the principles of the application.

[0048] Figure 1 is a hardware configuration block of an electronic device for implementing the 3D gesture recognition model training method according to an embodiment of the present application Figure One ;

[0049] Figure 2 is a flowchart of the 3D gesture recognition model training method according to an embodiment of the present application

[0050] Figure 3 is a schematic diagram of a scene for implementing the second real hand mesh model according to an embodiment of the present application

[0051] Figure 4 is a schematic diagram of the generator according to an embodiment of the present application

[0052] Figure 5 is a schematic diagram of the 3D gesture recognition model training apparatus according to an embodiment of the present application

[0053] Figure 6 is a hardware configuration block of an electronic device for implementing the 3D gesture recognition model training method according to an embodiment of the present application Figure Two . DETAILED DESCRIPTION

[0054] Various exemplary embodiments of the present application will now be described in detail with reference to the accompanying drawings. It should be noted that the relative arrangement of the components and steps set forth in the embodiments, numerical expressions, and numerical values are not limiting to the scope of the present application unless specifically stated otherwise.

[0055] The following description of at least one exemplary embodiment is merely exemplary in nature and is in no way intended to limit the scope of the application, its application, or uses.

[0056] Techniques, methods, and devices known to those of ordinary skill in the relevant art can not be discussed in detail herein. However, the techniques, methods, and devices should be considered part of the specification, if appropriate.

[0057] In all of the examples shown and discussed herein, any specific values should be interpreted as merely exemplary and not limiting. Thus, other examples of the exemplary embodiments can have different values.

[0058] It should be noted that like numbers and letters refer to like elements throughout the several views of the drawings and that the exemplified embodiment should not be construed as limiting the scope of the application.

[0059] Figure 1Figure 1 is a block diagram of a hardware configuration of an electronic device for implementing a 3D gesture recognition model training method according to an embodiment of the present application Figure One .

[0060] The electronic device 1000 can be a terminal or a server. Further, the terminal can be a head-mounted device (for example, an AR device, an MR device, a VR device, and an XR device), a portable computer, a tablet computer, a palm computer, and the like. The server can be a cloud server, and the like.

[0061] The electronic device 1000 can include a processor 1100, a memory 1200, an interface device 1300, a communication device 1400, a display device 1500, an input device 1600, a speaker 1700, and a microphone 1800, and the like. The processor 1100 can be a central processing unit (CPU), a micro processing unit (MCU), and the like. The memory 1200 includes, for example, a ROM (Read Only Memory), a RAM (Random Access Memory), a nonvolatile memory such as a hard disk, and the like. The interface device 1300 includes, for example, a USB interface, a headphone interface, and the like. The communication device 1400 can perform wired or wireless communication. The display device 1500 is, for example, a liquid crystal display, a touch display, and the like. The input device 1600 can include, for example, a touch screen, a keyboard, and the like. A user can input / output voice information through the speaker 1700 and the microphone 1800.

[0062] Although a plurality of devices are shown in the electronic device 1000 in Figure 1 , the present application can only involve some of the devices, for example, the electronic device 1000 only involves the memory 1200 and the processor 1100.

[0063] In the embodiments of the present application, the memory 1200 of the electronic device 1000 is configured to store instructions for controlling the processor 1100 to perform the 3D gesture recognition model training method provided by the embodiments of the present application.

[0064] In the above description, the skilled in the art can design the instructions according to the scheme disclosed in the present application. How the instructions control the processor to operate is known in the art, and therefore will not be described in detail here.

[0065] The present application also provides a 3D gesture recognition model training method, which is applied to an electronic device as shown in Figure 1 , and as shown in Figure 2 , includes the following steps S2100 to S2500.

[0066] Step S2100, obtaining a general hand mesh model.

[0067] In the embodiment, the mesh number of the general hand mesh model is small, generally less than 5000. That is, the general hand mesh model belongs to a rough hand mesh model, which is used to roughly describe the hand surface profile. The general hand mesh model can be used for algorithm optimization calculation.

[0068] In step S2200, a plurality of simulation hand mesh models and a first real hand mesh model are obtained according to the general hand mesh model.

[0069] In an embodiment of the present application, the above step S2200 is specifically implemented through steps S2210 to S2213.

[0070] In step S2210, a plurality of accurate hand mesh models are obtained.

[0071] In an embodiment of the present application, the plurality of accurate hand mesh models can be obtained from a simulator. Compared with the general hand mesh model in step S2100, the mesh number of the accurate hand mesh model is more, generally more than 100,000. The accurate hand mesh model belongs to a fine hand mesh model, which is used to finely describe the hand surface profile. The fine hand mesh model can be used for rendering a real picture.

[0072] In step S2211, for any accurate hand mesh model, the shape of the general hand mesh model is adjusted to be consistent with the shape of the accurate hand mesh 3D model, to obtain a simulation hand mesh model.

[0073] In the embodiment, the above step S2211 is specifically implemented as follows: in step S11, for any accurate hand mesh model, the shape parameters of the accurate hand mesh model are obtained, wherein the shape parameters of the accurate hand mesh model include deformation parameters and bone bending parameters which are known; in step S12, the general hand mesh model is adjusted, so that the shape parameters of the adjusted general hand mesh model are and ; in step S13, the latest adjusted general hand mesh model is mapped to the accurate hand mesh model with the shape parameters and , to determine the error between the latest adjusted general hand mesh model and the accurate hand mesh model with the shape parameters and ; in step S14, in the case that the error is greater than a preset error, the shape of the general hand mesh model is fine-tuned, and step S13 is repeated until the latest adjusted general hand mesh model is consistent with the accurate hand mesh model with the shape parameters and .The error between the precise hand mesh models is less than or equal to the preset error; Step S15, the newly adjusted general hand mesh model is used as the simulation hand mesh model, and the morphological parameters of the simulation hand mesh model are denoted as... as well as The preset error is the maximum error between the general hand mesh model and the precise hand mesh model when their shapes are similar. In one embodiment, the aforementioned error can be determined based on the sum of the squares of the minimum mesh distances between the general hand mesh model and the precise hand mesh model.

[0074] It should be noted that, due to the differences in hand size and other parameters between the general hand mesh model and the fine hand mesh model, even if the morphological parameters are the same, the general hand mesh model and the fine hand mesh model may have different shapes. Therefore, fine-tuning is required based on the above step S13.

[0075] It is understandable that multiple simulated hand mesh models can be obtained through the above steps S2210 to S2211.

[0076] Step S2212: Obtain multiple second real hand mesh models.

[0077] In one embodiment of this application, the specific implementation of step S2212 is as follows: Step S21, construct as follows Figure 3 The scenario shown has eight Time-of-Flight (TOF) cameras. The model provides an object (a real person) making different hand gestures in this scenario. Step S22 involves acquiring hand images from the eight TOF cameras for one gesture of the model's provided object. Step S23 involves constructing a hand mesh model based on the hand images obtained in step S22 and the intrinsic and extrinsic parameters of the eight TOF cameras, resulting in a second realistic hand mesh model. The intrinsic and extrinsic parameters of the eight TOF cameras can be pre-calibrated using traditional camera calibration methods.

[0078] Step S2213: For any second real hand mesh model, adjust the shape of the general hand mesh model to match the shape of the second real hand mesh model to obtain the first real hand mesh model.

[0079] It should be noted that the specific implementation of step S2213 is the same as that of step S2211, and will not be repeated here.

[0080] It is understandable that multiple first real hand mesh models can be obtained through the above steps S2212 to S2213.

[0081] Step S2300, determining the simulation skeleton 3D coordinates corresponding to any simulation hand mesh model.

[0082] In an embodiment of the present application, the step S2300 is implemented by the following step S2310.

[0083] Step S2310, determining the simulation skeleton 3D coordinates corresponding to any simulation hand mesh model according to the simulation hand mesh model and the preset mapping relationship.

[0084] The preset mapping relationship is used to reflect the corresponding relationship between the hand mesh model and the skeleton 3D coordinates.

[0085] In the embodiment, the preset mapping relationship is prior information and is stored in the electronic device in advance. The step S2310 is implemented by multiplying the simulation hand mesh model by the preset mapping relationship to obtain the simulation skeleton 3D coordinates corresponding to the simulation hand mesh model.

[0086] Step S2400, determining the real skeleton 3D coordinates corresponding to any first real hand mesh model.

[0087] In an embodiment of the present application, the step S2400 is implemented by the following step S2410.

[0088] Step S2410, determining the real skeleton 3D coordinates corresponding to any first real hand mesh model according to the first real hand mesh model and the preset mapping relationship.

[0089] It should be noted that the step S2410 is the same as the implementation of the step S2310, which will not be described here.

[0090] In an embodiment of the present application, when the electronic device is a VR device, as shown in FIG. 2, the electronic device is provided with a grayscale camera but not a TOF camera. Figure 3 To ensure the consistency of the real skeleton 3D coordinates and the electronic device applied by the 3D gesture recognition algorithm, the 3D gesture recognition model training method provided by the present application further comprises the following steps S2411 and S2412 after the step S2410.

[0091] Step S2411, obtaining the internal and external parameters of the grayscale camera.

[0092] The internal and external parameters of the grayscale camera can be obtained in advance by a conventional camera calibration method.

[0093] Step S2412, according to the internal and external parameters of the grayscale camera, the real bone 3D coordinates corresponding to the first real hand mesh model are converted into real bone 3D coordinates in the coordinate system of the grayscale camera.

[0094] It can be known from the above step S2200 that the simulation hand mesh model and the first real hand mesh model are both obtained through the general hand mesh model, and on this basis, the perspective consistency of the labeled 3D coordinates of the simulation hand mesh model and the first real hand mesh model can be ensured based on the above step S2300. In addition, through the above steps S2300 and S2300, manual calibration of the bone 3D coordinates in the simulation hand mesh model and the first real hand mesh model can be avoided.

[0095] Step S2500, training the to-be-trained 3D gesture recognition model according to the simulation hand mesh model, the first real hand mesh model, the simulation bone 3D coordinates and the real bone 3D coordinates, to obtain a target 3D gesture recognition model.

[0096] In an embodiment of the present application, the to-be-trained 3D gesture recognition model is a MobileNet-3D model and a Recurrent 3D CNN + STT model. In this case, the simulation hand mesh model and the simulation bone 3D coordinates corresponding to the simulation hand mesh model are used as training samples, and the first real hand mesh model and the real bone 3D coordinates corresponding to the first real hand mesh model are used as training samples. The to-be-trained 3D gesture recognition model is trained by using a traditional model training method to obtain the target 3D gesture recognition model.

[0097] In another embodiment of the present application, the to-be-trained 3D gesture recognition model is a generative adversarial network, the generative adversarial network includes a generator and a discriminator, and the target 3D gesture recognition model is the discriminator when the generative adversarial network converges. In this case, the above step S2500 is specifically implemented through the following steps S2510 to S2517.

[0098] Step S2510, generating image noise according to the simulation hand mesh model and the generator.

[0099] Unlike the generator in the traditional generative adversarial network, in the present embodiment, the simulation hand mesh model is input into the generator to generate image noise as image noise to be added to the simulation hand mesh model. Based on this, in an embodiment of the present application, the generator is trained according to the simulation hand mesh model and the simulation bone 3D coordinates corresponding to the simulation hand mesh model, and the first real hand mesh model and the real bone 3D coordinates corresponding to the first real hand mesh model. Figure 4As shown, the generator comprises an encoder, a reverse conditional denoising network and a decoder, wherein the encoder is configured to encode the simulated hand mesh model into a high-dimensional latent representation, the reverse conditional denoising network is configured to remove effective features from the high-dimensional latent representation to obtain a processed low-dimensional latent representation, and the decoder is configured to decode the processed low-dimensional latent representation into image noise. Based on this, it can be known that the processing procedure of the generator in the above step S2500 is completely opposite to that of the generator in the traditional generative adversarial network.

[0100] In an embodiment of the present application, the decoder is specifically configured to decode the processed low-dimensional latent representation into image noise according to a preset noise constraint condition. The preset noise constraint condition may, for example, constrain the image noise to be between 0 and 1, or keep the image noise within a difference of two pixels. The decoder in this embodiment can be realized by adding the preset noise constraint condition to the last layer of the decoder. In this embodiment, the decoder decodes the processed low-dimensional latent representation into image noise according to the preset noise constraint condition, so that the key information in the simulated hand mesh model can be prevented from being destroyed by the image noise.

[0101] In step S2511, a simulated synthetic hand mesh model is generated according to the image noise and the simulated hand mesh model.

[0102] In this embodiment, the image noise is added to the simulated hand mesh model to generate a simulated synthetic hand mesh model as the training data of the generator.

[0103] In step S2512, the discriminator is trained according to the simulated synthetic hand mesh model and the corresponding simulated skeletal 3D coordinates to obtain a trained discriminator.

[0104] In this embodiment, the discriminator is specifically a 3D gesture recognition model to be trained, which can recognize the skeletal 3D coordinates corresponding to the gesture. The simulated synthetic hand mesh model and the simulated skeletal 3D coordinates corresponding to the simulated synthetic hand mesh model are taken as a group of training samples in a training set, and the training of the 3D gesture recognition model to be trained is completed according to the traditional neural network training method to obtain the trained discriminator.

[0105] In step S2513, the discrimination result of the discriminator is determined according to the first real hand mesh model and the corresponding real skeletal 3D coordinates.

[0106] In the embodiment, the first real hand mesh model and its corresponding real skeleton 3D coordinates are taken as a set of verification samples in the verification set, and the trained discriminator obtained based on the step S2512 is verified according to the traditional neural network verification method to obtain a discrimination result. The discrimination result can be exemplarily a discrimination accuracy. In the case that the discrimination accuracy is greater than a preset accuracy, it is determined that the discrimination result represents accurate discrimination, otherwise, it is determined that the discrimination result represents inaccurate discrimination. The preset accuracy can be exemplarily 95%.

[0107] The specific manner of the discrimination accuracy can be that, in the case that the trained discriminator obtained based on the step S2512 identifies that the deviation between the skeleton 3D coordinates corresponding to the first real hand mesh model and the real skeleton 3D coordinates obtained based on the step S2400 is greater than a preset deviation, it is determined that the discriminator discriminates incorrectly, otherwise, it is determined that the discriminator discriminates correctly, and the ratio between the number of the first real hand mesh models corresponding to the discrimination correctness and the total number of the first real hand mesh models is determined as the discrimination accuracy. The preset deviation is the maximum value of the deviation allowed by the trained discriminator to identify the skeleton 3D coordinates corresponding to the first real hand mesh model and the corresponding real skeleton 3D coordinates.

[0108] In an embodiment of the present application, in order to reduce the amount of calculation, the step S2513 can be specifically implemented through the following steps S2513-1 to S2513-4.

[0109] Step S2513-1, determining the predicted real skeleton 3D coordinates according to the first real hand mesh model and the discriminator.

[0110] The specific implementation of the step S2513-1 is that the first real hand mesh model is input into the trained discriminator obtained based on the step S2512, and the skeleton 3D coordinates corresponding to the first real hand mesh model are identified by the aforementioned discriminator, which are recorded as the predicted real skeleton 3D coordinates.

[0111] Step S2513-2, projecting the predicted real skeleton 3D coordinates to a 2D coordinate system to obtain predicted real skeleton 2D coordinates.

[0112] The specific implementation of the step S2513-2 is that the predicted real skeleton 3D coordinates are projected to a 2D coordinate system according to the conversion relationship between the 3D coordinates and the 2D coordinates to obtain the predicted real skeleton 2D coordinates.

[0113] Step S2513-3, projecting the real skeleton 3D coordinates to a 2D coordinate system to obtain real skeleton 2D coordinates.

[0114] The implementation of step S2513-3 is the same as that of step S2513-2, and thus is not described herein again.

[0115] In step S2513-4, the discrimination result of the discriminator is determined according to the predicted real skeleton 2D coordinates and the real skeleton 2D coordinates.

[0116] In step S2513-4, the discrimination result of the discriminator is determined according to the deviation between the predicted real skeleton 2D coordinates and the real skeleton 2D coordinates.

[0117] In step S2514, the discriminator is updated according to the discrimination result in a case where the discrimination result represents inaccurate discrimination.

[0118] For example, in a case where the discrimination result represents inaccurate discrimination, the parameters of the discriminator are updated according to the discrimination accuracy of the discriminator, and thus the parameters of the discriminator are changed.

[0119] Based on step S2514, the training of the discriminator in the adversarial network in step S2500 can be implemented.

[0120] In step S2515, the predicted skeleton 3D coordinates are obtained by identifying the 3D skeleton coordinates corresponding to the simulation synthetic hand mesh model according to the discriminator.

[0121] In the embodiment, the simulation synthetic hand mesh model is input into the new discriminator obtained based on step S2514, and the predicted skeleton 3D coordinates are obtained by identifying the 3D skeleton coordinates corresponding to the simulation synthetic hand mesh model according to the new discriminator.

[0122] In step S2516, the parameters of the generator are updated according to the predicted skeleton 3D coordinates.

[0123] In the embodiment, the predicted skeleton 3D coordinates are substituted into the loss function of the generator to obtain a loss value, and the generator is updated according to the loss value. The loss function of the generator is a loss function reflecting the maximization of the discrimination accuracy of the discriminator on the simulation synthetic hand mesh.

[0124] Based on step S2516, the training of the generator in the adversarial network in step S2500 can be implemented. After step S2516, the parameters of the generator are changed.

[0125] In step S2517, the image noise is repeatedly generated according to the simulation hand mesh model and the generator until the discrimination result represents accurate discrimination.

[0126] In this embodiment, the step S2510 is repeated for the new generator obtained based on the step S2516 described above until the discrimination result of the discriminator represents discrimination accuracy. In this way, the training of the adversarial network is completed. On this basis, the discriminator in the adversarial network that is completed training is taken as the target 3D gesture recognition model.

[0127] In this embodiment, through the training of the adversarial network shown in the steps S2510 to S2517 described above, the difference between the simulation hand mesh model and the first real hand mesh model can be eliminated, so that the simulation hand mesh model can serve as the real hand mesh model, and the target 3D gesture recognition model is further obtained by using the simulation hand mesh model and the first real hand mesh model. That is, the simulation hand mesh model is migrated to the real hand mesh model, which reduces the dependence of the 3D gesture recognition model to be trained on the real hand mesh model, thereby reducing the learning cost of the 3D gesture recognition model to be trained.

[0128] The present application provides a 3D gesture recognition model training method, comprising: obtaining a general hand mesh model; obtaining a plurality of simulation hand mesh models and a first real hand mesh model according to the general hand mesh model; determining a simulation skeleton 3D coordinate corresponding to any simulation hand mesh model; determining a real skeleton 3D coordinate corresponding to any first real hand mesh model; training a 3D gesture recognition model to be trained according to the simulation hand mesh model, the first real hand mesh model, the simulation skeleton 3D coordinate and the real skeleton 3D coordinate, to obtain a target 3D gesture recognition model. This method can eliminate the difference between the simulation hand mesh model and the first real hand mesh model, so that the simulation hand mesh model can serve as the real hand mesh model, and the target 3D gesture recognition model is further obtained by using the simulation hand mesh model and the first real hand mesh model. That is, the simulation hand mesh model is migrated to the real hand mesh model, which reduces the dependence of the 3D gesture recognition model to be trained on the real hand mesh model, thereby reducing the learning cost of the 3D gesture recognition model to be trained.

[0129] The present application also provides a 3D gesture recognition model training device 500, as shown in Figure 5 , comprising:

[0130] A first obtaining module 510 is configured to obtain a general hand mesh model.

[0131] A second obtaining module 520 is configured to obtain a plurality of simulation hand mesh models and a first real hand mesh model according to the general hand mesh model.

[0132] The first determination module 530 is configured to determine, for any of the simulation hand mesh model, a simulation skeleton 3D coordinate corresponding to the simulation hand mesh model.

[0133] The second determination module 540 is configured to determine, for any of the first real hand mesh model, a real skeleton 3D coordinate corresponding to the first real hand mesh model.

[0134] The training module 550 is configured to train a to-be-trained 3D gesture recognition model according to the simulation hand mesh model, the first real hand mesh model, the simulation skeleton 3D coordinate and the real skeleton 3D coordinate, to obtain a target 3D gesture recognition model.

[0135] In an embodiment of the present application, the second acquisition module 520 is specifically configured to acquire a plurality of accurate hand mesh models.

[0136] For any accurate hand mesh model, the morphology of the general hand mesh model is adjusted to be consistent with the morphology of the accurate hand 3D model, to obtain a simulation hand mesh model.

[0137] In addition, a plurality of second real hand mesh models are acquired.

[0138] For any second real hand mesh model, the morphology of the general hand mesh model is adjusted to be consistent with the morphology of the second real hand mesh model, to obtain a first real hand mesh model.

[0139] In an embodiment of the present application, the first determination module 530 is specifically configured to determine, for any of the simulation hand mesh model, a simulation skeleton 3D coordinate corresponding to the simulation hand mesh model according to the simulation hand mesh model and a preset mapping relationship, wherein the preset mapping relationship is used to reflect the corresponding relationship between the hand mesh model and the skeleton 3D coordinate.

[0140] In an embodiment of the present application, the second determination module 540 is specifically configured to determine, for any of the first real hand mesh model, a real skeleton 3D coordinate corresponding to the first real hand mesh model according to the first real hand mesh model and the preset mapping relationship.

[0141] In an embodiment of the present application, the to-be-trained 3D gesture model is an adversarial network, wherein the adversarial network includes a generator and a discriminator, the target 3D gesture recognition model is the discriminator when the adversarial network converges, and the training module 550 is specifically configured to generate image noise according to the simulation hand mesh model and the generator.

[0142] A simulation synthetic hand mesh model is generated according to the image noise and the simulation hand mesh model.

[0143] training the discriminator according to the simulation synthetic hand mesh model and the corresponding simulation skeleton 3D coordinates, to obtain a trained discriminator;

[0144] determining a discrimination result of the discriminator according to the first real hand mesh model and the corresponding real skeleton 3D coordinates;

[0145] updating the discriminator according to the discrimination result in a case where the discrimination result represents inaccurate discrimination;

[0146] identifying 3D skeleton coordinates corresponding to the simulation synthetic hand mesh model according to the discriminator, to obtain predicted skeleton 3D coordinates;

[0147] updating parameters of the generator according to the predicted skeleton 3D coordinates;

[0148] repeating the generating image noise according to the simulation hand mesh model and the generator until the discrimination result represents accurate discrimination.

[0149] In an embodiment of the present application, the training module 550 is specifically configured to determine predicted real skeleton 3D coordinates according to the first real hand mesh model and the discriminator;

[0150] projecting the predicted real skeleton 3D coordinates to a 2D coordinate system to obtain predicted real skeleton 2D coordinates;

[0151] projecting the real skeleton 3D coordinates to the 2D coordinate system to obtain real skeleton 2D coordinates;

[0152] determining a discrimination result of the discriminator according to the predicted real skeleton 2D coordinates and the real skeleton 2D coordinates.

[0153] In an embodiment of the present application, the generator comprises an encoder, a reverse conditional denoising network and a decoder, wherein the encoder is configured to decode the simulation hand mesh model into a high-dimensional latent representation, the reverse conditional denoising network is configured to remove effective features from the high-dimensional latent representation to obtain a processed low-dimensional latent representation, and the decoder is configured to decode the processed low-dimensional latent representation into image noise.

[0154] In an embodiment of the present application, the decoder is specifically configured to decode the processed low-dimensional latent representation into image noise according to a preset noise constraint condition.

[0155] The present application also provides an electronic device comprising any one of the 3D gesture recognition model training devices 500 provided in the above device embodiments.

[0156] Alternatively, as shown in Figure 6 The electronic device 1000 includes a memory 1200 for storing computer instructions and a processor 1100 for invoking the computer instructions from the memory 1200 to perform any one of the 3D gesture recognition model training methods provided by the above method embodiments.

[0157] The present application also provides a computer readable storage medium having a computer program stored thereon, which, when executed by a processor, implements any one of the 3D gesture recognition model training methods provided by the above method embodiments.

[0158] The present application can be a system, a method, and / or a computer program product. The computer program product can include a computer readable storage medium (or media) having computer readable program instructions thereon for causing a processor to carry out aspects of the present application.

[0159] The computer readable storage medium can be a tangible device that can retain and store instructions for use by an instruction execution device. The computer readable storage medium can be, for example, but is not limited to, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. More specific examples (a non-exhaustive list) of the computer readable storage medium include the following: a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), a static random access memory (SRAM), a portable compact disc read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanically encoded device such as punch-cards or punched tape, a holographic storage medium, and any suitable combination of the foregoing. A computer readable storage medium, as used herein, is not to be construed as being transitory signals per se, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through a waveguide or other transmission media (e.g., light pulses passing through a fiber-optic cable), or electrical signals transmitted through a wire.

[0160] Computer readable program instructions described herein can be downloaded to respective computing / processing devices from a computer readable storage medium or to an external computer or external storage device via a network, for example, the Internet, a local area network, a wide area network and / or a wireless network. The network can comprise copper transmission cables, optical transmission fibers, wireless transmission, routers, firewalls, switches, gateway computers and / or edge servers. A network adapter card or network interface in each computing / processing device receives computer readable program instructions from the network and forwards the computer readable program instructions for storage in a computer readable storage medium within the respective computing / processing device.

[0161] Computer readable program instructions for carrying out operations of the present application can be assembler instructions, instruction-set-architecture (ISA) instructions, machine instructions, machine dependent instructions, microcode, firmware instructions, state-setting data, or either source code or object code written in any combination of one or more programming languages, including an object oriented programming language such as Smalltalk, C++ or the like and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The computer readable program instructions can execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the latter scenario, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection can be made to an external computer (for example, through the Internet using an Internet Service Provider). In some embodiments, electronic circuitry including, for example, programmable logic circuitry, field-programmable gate array (FPGA), or programmable logic array (PLA) can execute the computer readable program instructions by utilizing state information of the computer readable program instructions to personalize the electronic circuitry, in order to perform aspects of the present application.

[0162] The computer readable program instructions can also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable apparatus or other device to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide processes for implementing the functions / acts specified in the flowchart and / or block diagram block or blocks.

[0163] These computer readable program instructions can be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions / acts specified in the flowchart and / or block diagram block or blocks. These computer readable program instructions can also be stored in a computer readable storage medium that can include random access memory (RAM), read only memory (ROM), electrically erasable programmable read only memory (EEPROM), flash memory or nonvolatile memory, or a suitable combination of the different types of computer readable storage media. The computer readable program instructions can also be downloaded to a computer, other programmable data processing apparatus, or other device from a computer readable storage medium or to an external computer or external storage device via a data signal that can be transmitted for example via a wired medium or a wireless medium such as the Internet or wireless media.

[0164] The computer readable program instructions can also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus or other device to produce a computer implemented process such that the instructions which execute on the computer, other programmable data processing apparatus, or other device implement the functions / acts specified in the flowchart and / or block diagram block or blocks.

[0165] The flow diagrams and the block diagrams in the drawings are presented to illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present application. In this regard, each block in the flow diagrams and the block diagrams can represent a module, segment, or portion of instructions, which comprises one or more executable instructions for implementing the specified logic functions. In some alternative implementations, the functions noted in the blocks can occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks can sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams and / or flow diagrams, and combinations thereof, can be implemented by special purpose hardware-based systems that perform the specified functions or acts, or combinations of special purpose hardware and

[0166] Embodiments of the application have been described above. The description is illustrative of the embodiments of the application and is not meant to be limiting. Numerous modifications and variations are possible in light of the above teachings without departing from the scope and spirit of the described embodiments of the application. No limitation is intended to the details of construction or design except as described in the claims.

Claims

1. A method for training a 3D gesture recognition model, characterized in that, include: Obtain a generic hand mesh model; Based on the general hand mesh model, multiple simulated hand mesh models and a first real hand mesh model are obtained; For any of the simulated hand mesh models, determine the 3D coordinates of the simulated skeleton corresponding to the simulated hand mesh model; For any of the first real hand mesh models, determine the real 3D coordinates of the skeleton corresponding to the first real hand mesh model; The target 3D gesture recognition model is obtained by training the simulated hand mesh model, the first real hand mesh model, the simulated skeleton 3D coordinates, and the real skeleton 3D coordinates. The step of obtaining multiple simulated hand mesh models and a first real hand mesh model based on the general hand mesh model includes: Obtain multiple accurate hand mesh models; For any precise hand mesh model, adjust the shape of the general hand mesh model to match the shape of the precise hand 3D model to obtain a simulated hand mesh model; In addition, multiple second-realistic hand mesh models were obtained; For any second realistic hand mesh model, adjust the shape of the general hand mesh model to match the shape of the second realistic hand mesh model to obtain the first realistic hand mesh model.

2. The method according to claim 1, characterized in that, For any of the simulated hand mesh models, determining the 3D coordinates of the simulated skeleton corresponding to the simulated hand mesh model includes: For any of the simulated hand mesh models, the simulated skeleton 3D coordinates corresponding to the simulated hand mesh model are determined according to the simulated hand mesh model and the preset mapping relationship, wherein the preset mapping relationship is used to reflect the correspondence between the hand mesh model and the skeleton 3D coordinates; For any of the first real hand mesh models, determining the corresponding real skeletal 3D coordinates of the first real hand mesh model includes: For any of the first real hand mesh models, the real 3D coordinates of the skeleton corresponding to the first real hand mesh model are determined according to the first real hand mesh model and the preset mapping relationship.

3. The method according to claim 1, characterized in that, The 3D gesture model to be trained is an adversarial network, wherein the adversarial network includes a generator and a discriminator, and the target 3D gesture recognition model is the discriminator when the adversarial network converges; The step of training a 3D gesture recognition model based on the simulated hand mesh model, the first real hand mesh model, the simulated skeleton 3D coordinates, and the real skeleton 3D coordinates to obtain a target 3D gesture recognition model includes: Based on the simulated hand mesh model and the generator, image noise is generated; Based on the image noise and the simulated hand mesh model, a simulated synthetic hand mesh model is generated; Based on the simulated synthetic hand mesh model and the corresponding simulated skeleton 3D coordinates, the discriminator is trained to obtain a trained discriminator. The discrimination result of the discriminator is determined based on the first real hand mesh model and the corresponding real skeleton 3D coordinates; If the discrimination result indicates that the discrimination is inaccurate, the discriminator is updated according to the discrimination result; Based on the discriminator, the 3D bone coordinates corresponding to the simulated synthetic hand mesh model are identified, and the predicted 3D bone coordinates are obtained. The generator parameters are updated based on the predicted 3D skeleton coordinates; The process of generating image noise based on the simulated hand mesh model and the generator is repeated until the discrimination result indicates accurate discrimination.

4. The method according to claim 3, characterized in that, The step of determining the discrimination result of the discriminator based on the first real hand mesh model and the real 3D coordinates of the skeleton includes: Based on the first real hand mesh model and the discriminator, the predicted real 3D coordinates of the skeleton are determined; The predicted real skeleton 3D coordinates are projected onto the 2D coordinate system to obtain the predicted real skeleton 2D coordinates; Projecting the actual 3D coordinates of the skeleton onto the 2D coordinate system yields the actual 2D coordinates of the skeleton. The discrimination result of the discriminator is determined based on the predicted real 2D skeleton coordinates and the real 2D skeleton coordinates.

5. The method according to claim 3, characterized in that, The generator includes an encoder, an inverse conditional denoising network, and a decoder. The encoder is used to decode the simulated hand mesh model into a high-dimensional latent representation. The inverse conditional denoising network is used to remove effective features from the high-dimensional latent representation to obtain a processed low-dimensional latent representation. The decoder is used to decode the processed low-dimensional latent representation into image noise.

6. The method according to claim 5, characterized in that, The decoder is specifically used to decode the processed low-dimensional latent representation into image noise according to preset noise constraints.

7. A 3D gesture recognition model training device, characterized in that, include: The first acquisition module is used to acquire a general hand mesh model; The second acquisition module is used to acquire multiple simulated hand mesh models and a first real hand mesh model based on the general hand mesh model. The first determining module is used to determine the 3D coordinates of the simulated skeleton corresponding to any of the simulated hand mesh models. The second determining module is used to determine the real 3D coordinates of the skeleton corresponding to any of the first real hand mesh models. The training module is used to train the 3D gesture recognition model to be trained based on the simulated hand mesh model, the first real hand mesh model, the simulated skeleton 3D coordinates and the real skeleton 3D coordinates, so as to obtain the target 3D gesture recognition model. Specifically, the second acquisition module is used to: acquire multiple precise hand mesh models; For any precise hand mesh model, adjust the shape of the general hand mesh model to match the shape of the precise hand 3D model to obtain a simulated hand mesh model; In addition, multiple second-realistic hand mesh models were obtained; For any second realistic hand mesh model, adjust the shape of the general hand mesh model to match the shape of the second realistic hand mesh model to obtain the first realistic hand mesh model.

8. An electronic device, characterized in that, The electronic device includes the apparatus as described in claim 7; Alternatively, the electronic device includes a memory and a processor, the memory for storing computer instructions, and the processor for retrieving the computer instructions from the memory to perform the method as described in any one of claims 1-6.

9. A computer-readable storage medium, characterized in that, It stores a computer program that, when executed by a processor, implements the method according to any one of claims 1-6.

Images