Motion capture method, device, electronic device and storage medium

Abstract

The present disclosure relates to a motion capture method, device, electronic device and storage medium, the method comprising: obtaining a whole-body posture estimation result; driving a three-dimensional model according to the whole-body posture estimation result; collecting target motion video data of detail parts in real time during the playback of the initial whole-body motion; identifying the target motion video data of detail parts to obtain the target posture estimation result of the detail parts; driving the three-dimensional model according to the target posture estimation result of the detail parts and the posture estimation results of other parts to obtain the target whole-body motion of the three-dimensional model. On the one hand, the method is based on an interactive operation process, allowing the user to split the whole-body motion capture process into multiple parts and multiple acquisition processes, and realize motion synchronization through real-time preview, so as to obtain high-quality whole-body motion information; on the other hand, based on a pure visual method for motion capture, high-precision motion acquisition is achieved at a relatively low cost.

Description

Technical Field

[0001] The present disclosure relates to the field of computer technology, and in particular to a motion capture method, device, electronic device and storage medium. Background Art

[0002] Motion capture refers to the capture of human body movements, gestures, and expressions to fully present the actor's performance. It has important applications in CG (Computer Graphics) animation film production.

[0003] Motion capture can be achieved by optical motion capture, inertial motion capture and markerless visual motion capture. Optical motion capture is an existing motion capture solution with high capture quality and is a common method in the production process of 3D (three-dimensional) animation films. Inertial motion capture is widely used in the production of semi-professional content such as virtual idols. For optical motion capture and inertial motion capture, performers are usually required to wear professional costumes and use professional acquisition equipment for motion capture, which has a high cost problem. The markerless visual motion capture method solves the problem of high motion capture cost, and performers can use ordinary mobile phones or computers with ordinary cameras to capture motion. However, in the related art, the markerless visual motion capture method cannot obtain clear full-body movements because the fingers and face occupy too few pixels on the screen when the whole body is in the picture and are too blurred. Summary of the invention

[0004] The present disclosure provides a motion capture method, device, electronic device and storage medium to at least solve the problem in the related art that motion capture cannot obtain clear full-body motion. The technical solution of the present disclosure is as follows:

[0005] According to a first aspect of an embodiment of the present disclosure, a motion capture method is provided, comprising:

[0006] Obtaining a whole-body posture estimation result, where the whole-body posture estimation result includes a detailed part posture estimation result and a part posture estimation result other than the detailed part, where the detailed part is a part of the whole body;

[0007] The three-dimensional model is driven according to the whole-body posture estimation result to obtain the initial whole-body motion of the three-dimensional model;

[0008] During the playback of the initial full-body movement, the target movement video data of the detailed parts are collected in real time;

[0009] Identify the target action video data of the detailed part and obtain the target posture estimation result of the detailed part;

[0010] The target posture estimation results of the detailed parts and the posture estimation results of other parts are used to jointly drive the three-dimensional model to obtain the target full-body motion of the three-dimensional model.

[0011] In one embodiment, the number of detail parts includes multiple; and during the playing of the initial whole-body action, the target action video data of the detail parts is collected in real time, including:

[0012] During the playback of the initial full-body action, the target action video data of each detail part is collected separately;

[0013] In this embodiment, the target action video data of the detail part is identified to obtain the target posture estimation result of the detail part, including:

[0014] The target action video data of each detail part is identified to obtain the target posture estimation result of each detail part.

[0015] In one embodiment, the three-dimensional model is driven together according to the target posture estimation results of the detail parts and the posture estimation results of other parts to obtain the target whole-body motion of the three-dimensional model, including:

[0016] In the process of obtaining the target pose estimation result of each detail part, the three-dimensional model is driven in real time according to the target pose estimation result of the detail part obtained and the pose estimation results of other parts until the target pose estimation results of all detail parts are obtained;

[0017] The target posture estimation results of all detailed parts and the posture estimation results of other parts are used to jointly drive the 3D model to obtain the target full-body motion of the 3D model.

[0018] In one embodiment, obtaining a whole body posture estimation result includes:

[0019] Obtaining pre-saved initial action video data of the whole body;

[0020] The pre-saved initial action video data of the whole body is identified to obtain the posture estimation result of the whole body.

[0021] In one embodiment, obtaining a whole body posture estimation result includes:

[0022] Collect the initial motion video data of the whole body in real time;

[0023] Identify the initial full-body motion video data collected in real time and obtain the full-body posture estimation result.

[0024] In one embodiment, after obtaining the target whole body motion of the three-dimensional model, the method further includes:

[0025] When receiving a re-collection instruction, and the part category information included in the re-collection instruction is the whole body, collecting updated action video data of the whole body, wherein the initial action video data of the whole body and the updated action video data of the whole body are obtained through the same set of performance actions of the whole body;

[0026] Identify the whole body update action video data and obtain the whole body update posture estimation result;

[0027] The three-dimensional model is driven according to the updated posture estimation result of the whole body to obtain the updated initial whole-body motion of the three-dimensional model.

[0028] In one embodiment, after obtaining the target whole body motion of the three-dimensional model, the method further includes:

[0029] When receiving a re-collection instruction, and the part category information included in the re-collection instruction is a detail part, playing the initial whole-body motion;

[0030] In the process of playing the initial whole-body action, the updated action video data of the detail parts are collected in real time, wherein the updated action video data of the detail parts and the target action video data of the detail parts are obtained through the same set of performance actions of the detail parts;

[0031] Identify the updated action video data of the detail part and obtain the updated posture estimation result of the detail part;

[0032] The three-dimensional model is driven together according to the updated posture estimation results of the detailed parts and the posture estimation results of other parts to obtain the updated whole-body motion of the three-dimensional model.

[0033] In one embodiment, the detail area includes at least one of a hand and a face.

[0034] According to a second aspect of an embodiment of the present disclosure, there is provided a motion capture device, comprising:

[0035] An acquisition module is configured to acquire a whole-body posture estimation result, wherein the whole-body posture estimation result includes a posture estimation result of a detail part and a posture estimation result of other parts except the detail part, wherein the detail part is a part of the whole body;

[0036] A driving module is configured to drive the three-dimensional model according to the whole-body posture estimation result to obtain an initial whole-body motion of the three-dimensional model;

[0037] A collection module is configured to collect target action video data of detail parts in real time during the playback of the initial whole-body action;

[0038] A detail part recognition module is configured to execute target action video data of the detail part recognition to obtain a target posture estimation result of the detail part;

[0039] The driving module is also configured to jointly drive the three-dimensional model based on the target posture estimation results of the detail parts and the posture estimation results of other parts to obtain the target whole-body movement of the three-dimensional model.

[0040] In one embodiment, the number of detail parts includes a plurality; the acquisition module is configured to execute, during the process of playing the initial whole-body action, to separately acquire the target action video data of each detail part;

[0041] The detail part recognition module is configured to execute the target action video data of each detail part recognition to obtain the target posture estimation result of each detail part.

[0042] In one embodiment, the driving module is configured to execute, in the process of obtaining a target posture estimation result of a detail part, jointly drive the three-dimensional model according to the target posture estimation result of the detail part and the posture estimation results of other parts in real time until the target posture estimation results of all detail parts are obtained; and jointly drive the three-dimensional model according to the target posture estimation results of all detail parts and the posture estimation results of other parts to obtain the target whole-body movement of the three-dimensional model.

[0043] In one of the embodiments, the acquisition module is further configured to execute acquisition of pre-saved initial action video data of the whole body;

[0044] The device also includes: a whole body posture recognition module, which is configured to perform recognition of pre-stored whole body initial action video data to obtain a whole body posture estimation result.

[0045] In one of the embodiments, the acquisition module is further configured to perform real-time acquisition of initial motion video data of the whole body;

[0046] The device also includes: a whole body posture recognition module, which is configured to perform recognition of the whole body initial action video data collected in real time to obtain a whole body posture estimation result.

[0047] In one of the embodiments, the apparatus further comprises: a receiving module configured to execute receiving a re-acquisition instruction;

[0048] The acquisition module is further configured to acquire updated action video data of the whole body when the part category information included in the re-acquisition instruction is the whole body, wherein the initial action video data of the whole body and the updated action video data of the whole body are obtained through the same set of performance actions of the whole body;

[0049] The whole body posture recognition module is further configured to perform recognition of the updated action video data of the whole body to obtain an updated posture estimation result of the whole body;

[0050] The driving module is also configured to drive the three-dimensional model according to the updated posture estimation result of the whole body to obtain the updated initial whole-body motion of the three-dimensional model.

[0051] In one of the embodiments, the apparatus further comprises: a receiving module configured to execute receiving a re-acquisition instruction;

[0052] a playing module configured to play the initial whole-body action when the part category information included in the re-collection instruction is a detail part;

[0053] The acquisition module is further configured to acquire the updated action video data of the detail parts in real time during the playing of the initial whole-body action, wherein the updated action video data of the detail parts and the target action video data of the detail parts are obtained through the same set of performance actions of the detail parts;

[0054] The detail part recognition module is further configured to execute the updated action video data of recognizing the detail part to obtain the updated posture estimation result of the detail part;

[0055] The driving module is also configured to drive the three-dimensional model together according to the updated posture estimation results of the detail parts and the posture estimation results of other parts to obtain the updated whole-body movement of the three-dimensional model.

[0056] In one embodiment, the detail area includes at least one of a hand and a face.

[0057] According to a third aspect of an embodiment of the present disclosure, there is provided an electronic device, including:

[0058] processor;

[0059] a memory for storing processor-executable instructions;

[0060] The processor is configured to execute instructions to implement the motion capture method described in any one of the embodiments of the first aspect.

[0061] According to a fourth aspect of an embodiment of the present disclosure, a storage medium is provided. When instructions in the storage medium are executed by a processor of an electronic device, the electronic device is enabled to execute the motion capture method described in any one of the embodiments of the first aspect.

[0062] According to a fifth aspect of an embodiment of the present disclosure, a computer program product is provided, the program product comprising a computer program, the computer program being stored in a readable storage medium, at least one processor of a device reading and executing the computer program from the readable storage medium, so that the device executes the motion capture method described in any one of the embodiments of the first aspect.

[0063] The technical solution provided by the embodiments of the present disclosure brings at least the following beneficial effects:

[0064] Obtain the posture estimation result of the whole body, the posture estimation result of the whole body includes the posture estimation result of the detail part and the posture estimation results of other parts except the detail part, the detail part is a part of the whole body; drive the three-dimensional model according to the posture estimation result of the whole body to obtain the initial whole-body motion of the three-dimensional model; collect the target motion video data of the detail part in real time during the playback of the initial whole-body motion; identify the target motion video data of the detail part to obtain the target posture estimation result of the detail part; drive the three-dimensional model together according to the target posture estimation result of the detail part and the posture estimation results of other parts to obtain the target whole-body motion of the three-dimensional model. According to the scheme disclosed in the present invention, on the one hand, based on the interactive operation process, the user can split the whole-body motion capture process into multiple parts and multiple acquisition processes, and realize motion synchronization through real-time preview, so as to obtain high-quality whole-body motion information; on the other hand, based on the pure visual method for motion capture, high-precision motion acquisition is achieved at a relatively low cost.

[0065] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0066] The drawings herein are incorporated into and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the description are used to explain the principles of the present disclosure, and do not constitute improper limitations on the present disclosure.

[0067] Figure 1 The figure is a diagram showing an application environment of a motion capture method according to an exemplary embodiment.

[0068] Figure 2 The figure is a flow chart of a motion capture method according to an exemplary embodiment.

[0069] Figure 3 The figure is a flowchart showing steps of re-collecting whole body movements according to an exemplary embodiment.

[0070] Figure 4 The present invention is a flowchart showing the steps of re-collecting detail parts according to an exemplary embodiment.

[0071] Figure 5 The figure is a flow chart of a motion capture method according to an exemplary embodiment.

[0072] Figure 6 The figure is a block diagram of a motion capture device according to an exemplary embodiment.

[0073] Figure 7 The figure is a diagram showing the internal structure of an electronic device according to an exemplary embodiment. DETAILED DESCRIPTION

[0074] In order to enable ordinary persons in the art to better understand the technical solutions of the present disclosure, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the accompanying drawings.

[0075] It should be noted that the terms "first", "second", etc. in the specification and claims of the present disclosure and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present disclosure described herein can be implemented in an order other than those illustrated or described herein. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present disclosure. Instead, they are merely examples of devices and methods consistent with some aspects of the present disclosure as detailed in the appended claims.

[0076] The motion capture method provided by the present disclosure can be applied to Figure 1In the application environment shown. The application environment includes a terminal 110 and a video acquisition device 120. The video acquisition device 120 can be a built-in device of the terminal 110 or a separate device, which is not limited here. A motion capture system based on a pure visual method for motion capture is deployed in the terminal 110, which is used to process the motion video data collected by the video acquisition device 120 in real time, obtain a posture estimation result corresponding to the motion video data, and drive a predefined three-dimensional model through the obtained posture estimation result. The terminal 110 also includes a display screen for playing the three-dimensional model driven by the posture estimation result in real time, so that the user can preview the motion effect of the three-dimensional model in real time. Specifically, the terminal 110 obtains the whole body posture estimation result, which includes the posture estimation result of the detail part and the posture estimation results of other parts except the detail part, and the detail part is a part of the whole body; the terminal 110 drives the three-dimensional model according to the whole body posture estimation result to obtain the initial whole body action of the three-dimensional model; in the process of playing the initial whole body action, the target action video data of the detail part is collected in real time through the video acquisition device 120; the target action video data of the detail part is identified to obtain the target posture estimation result of the detail part; the three-dimensional model is driven together according to the target posture estimation result of the detail part and the posture estimation results of other parts to obtain the target whole body action of the three-dimensional model. The terminal 110 plays the target whole body action of the three-dimensional model, so that the user can preview the action synchronization effect of other parts and the detail part. Among them, the terminal 110 can be, but not limited to, various personal computers, laptops, smart phones, tablet computers and portable wearable devices. The video acquisition device 120 can be, but not limited to, at least one of a video camera, a camera and a camera installed on an electronic device.

[0077] Figure 2 is a flow chart of a motion capture method according to an exemplary embodiment. Figure 2 As shown, the motion capture method is used in the terminal 110, and includes the following steps.

[0078] In step S210, a whole-body posture estimation result is obtained, where the whole-body posture estimation result includes a posture estimation result of a detail part and a posture estimation result of other parts except the detail part, where the detail part is a part of the whole body.

[0079] Among them, the detail parts may refer to parts of the human body that are more difficult to identify, such as hands, facial features, wrists, etc. Motion capture is based on a set of whole-body movements that need to be performed, and the whole-body movements include detail part movements and movements of other parts except detail parts. Specifically, the whole-body posture estimation result is not limited to the rotation angles of the whole-body joints. The whole-body posture estimation result can be obtained by using a trained first neural network model to identify the initial motion video data of the whole body. The initial motion video data of the whole body can be obtained by pre-collecting the video data through a video acquisition device, or it can be collected in real time when motion capture is required. After obtaining the initial motion video data of the whole body, the initial motion video data of the whole body is recognized by the trained first neural network model to obtain the whole-body posture estimation result. The first neural network model can be any model that can be used for human posture estimation, for example, DensePose (real-time human posture estimation model), OpenPose (key point detection model).

[0080] In step S220, the three-dimensional model is driven according to the whole-body posture estimation result to obtain an initial whole-body motion of the three-dimensional model.

[0081] Specifically, after obtaining the whole-body posture estimation result, the terminal uses the whole-body posture estimation result to drive the 3D model, obtains the initial whole-body motion of the 3D model, and plays the initial whole-body motion of the 3D model through the screen. The 3D model can be a pre-selected 3D virtual image, such as a virtual idol live broadcast, a virtual host, and body magic special effects in short video application scenarios. Users can select a suitable 3D model for use according to actual needs.

[0082] In step S230, target motion video data of detailed parts are collected in real time during the process of playing the initial full-body motion of the three-dimensional model.

[0083] In step S240, target action video data of the detail part is identified to obtain target posture estimation results of the detail part.

[0084] In step S250, the three-dimensional model is driven together according to the target posture estimation results of the detail parts and the posture estimation results of other parts to obtain the target whole-body motion of the three-dimensional model.

[0085] Among them, a trained second neural network model is deployed in the terminal, which is used to recognize the collected target action video data of the detail part and obtain the target posture estimation result of the detail part. The second neural network model can be determined according to the type of the detail part. For example, when the detail part is a hand, the second neural network model can be any gesture capture algorithm that can be used for gesture posture estimation, such as Hand Pose Estimation (gesture posture estimation algorithm); when the detail part is a face, the second neural network model can be any expression recognition algorithm that can be used for expression recognition, such as Deep Facial Expression Recognition (deep expression recognition algorithm).

[0086] Specifically, in the process of playing the initial full-body action of the three-dimensional model, the user performs the synchronized performance of the detailed part action as the initial full-body action is played. The terminal collects the target action video data of the detailed part in real time; and inputs the collected target action video data of the detailed part into the pre-deployed second neural network model. The terminal obtains the target posture estimation result of the detailed part currently output by the second neural network model. The three-dimensional model is driven together according to the target posture estimation result of the detailed part currently output and the posture estimation results of other parts currently played to obtain the target full-body action of the current three-dimensional model. The terminal can also play the target full-body action of the current three-dimensional model on the screen, so that the user can preview the action synchronization effects of other parts and detailed parts in real time.

[0087] In the above motion capture method, on the one hand, based on the interactive operation process, the user can split the whole-body motion capture process into multiple parts and multiple acquisition processes, and realize motion synchronization through real-time preview, so as to obtain high-quality whole-body motion information; on the other hand, motion capture based on pure visual methods realizes high-precision motion acquisition at a relatively low cost.

[0088] In an exemplary embodiment, the number of detail parts includes multiple ones; in step S230, the target action video data of the detail parts are collected in real time during the playback of the initial whole-body action, including: during the playback of the initial whole-body action, the target action video data of each detail part is collected separately.

[0089] Specifically, when the number of detail parts includes multiple, the terminal can drive the three-dimensional model according to the acquired whole-body posture estimation result, obtain the initial whole-body action of the three-dimensional model, and play the initial whole-body action. In the process of playing the initial whole-body action, the user performs the first detail part synchronously with the playback progress of the initial whole-body action. That is, the video acquisition device is used to collect the first detail part separately to obtain the target action video data of the first detail part. After the target action video data of the first detail part is collected, the terminal replays the initial whole-body action of the three-dimensional model, and in the process of replaying the initial whole-body action, the target action video data of the second detail part is collected separately according to the collection process of the target action video data of the first detail part. Repeat the above process until the target action video data of all detail parts are collected.

[0090] In this embodiment, in step S240, the target action video data of the detail part is identified to obtain the target posture estimation result of the detail part, including: identifying the target action video data of each detail part to obtain the target posture estimation result of each detail part.

[0091] Specifically, the target posture estimation result of each detail part can be obtained by real-time recognition, that is, while collecting the target action video data of each detail part, the target action video data at the current moment is recognized by the second neural network model; or, it can be obtained by offline recognition, that is, after the collection is completed, the complete target action video data of each detail part is recognized by the second neural network model, which is not limited here.

[0092] In this embodiment, by separately collecting the target motion video data of each detail part, a clear motion of each detail part can be obtained, thereby improving the quality of motion capture.

[0093] In an exemplary embodiment, in step S250, the three-dimensional model is jointly driven according to the target posture estimation results of the detail parts and the posture estimation results of other parts to obtain the target whole-body motion of the three-dimensional model, including: in the process of obtaining the target posture estimation results of each detail part, the three-dimensional model is jointly driven according to the target posture estimation results of the detail parts and the posture estimation results of other parts in real time until the target posture estimation results of all detail parts are obtained; the three-dimensional model is jointly driven according to the target posture estimation results of all detail parts and the posture estimation results of other parts to obtain the target whole-body motion of the three-dimensional model.

[0094] Specifically, when the number of detail parts includes multiple, each time a target posture estimation result of a detail part is obtained by real-time or offline identification, the terminal drives the three-dimensional model together according to the target posture estimation result of the detail part obtained at the current moment and the posture estimation results of other parts played at the current moment, so as to obtain the whole body movement of the three-dimensional model at the current moment. The terminal can also play the whole body movement of the three-dimensional model at the current moment on the screen, so that the user can preview the movement synchronization effect of other parts and detail parts in real time. Until the target posture estimation result of the last detail part is identified, the three-dimensional model is driven together according to the target posture estimation results of all detail parts and the posture estimation results of other parts, so as to obtain the target whole body movement of the three-dimensional model.

[0095] In this embodiment, by separately collecting the target action video data of each detail part, and synchronizing the action of each detail part with the actions of other parts in an interactive manner, the user can understand the action synchronization effect of different parts in real time.

[0096] In an exemplary embodiment, in step S210, obtaining a whole-body posture estimation result includes: obtaining pre-saved whole-body initial motion video data; identifying the pre-saved whole-body initial motion video data to obtain a whole-body posture estimation result.

[0097] Specifically, the whole body posture estimation result can be obtained by offline recognition. A video acquisition device is used in advance to shoot a set of performance actions of the user, obtain the whole body initial action video data, and save the whole body initial action video data to the terminal. When motion capture is required, the terminal calls the pre-deployed first neural network model to recognize the whole body initial action video data and obtain the whole body posture estimation result.

[0098] In this embodiment, by collecting and identifying the initial motion video data of the whole body in an offline manner, the operating pressure of the equipment during motion capture can be reduced; in addition, by allowing the user to collect the initial motion video data of the whole body in advance, the flexibility of use is improved.

[0099] In an exemplary embodiment, in step S210, obtaining a whole-body posture estimation result includes: collecting the whole-body initial motion video data in real time; identifying the whole-body initial motion video data collected in real time to obtain the whole-body posture estimation result.

[0100] Specifically, the whole-body posture estimation result can be obtained by real-time identification. When motion capture is required, a video acquisition device is used to shoot the user's performance action in real time to obtain initial action video data. The terminal calls the pre-deployed first neural network model to identify the whole-body initial action video data collected at the current moment, and obtains the whole-body posture estimation result at the current moment. Then, the three-dimensional model is driven according to the whole-body posture estimation result at the current moment to obtain the initial whole-body action of the three-dimensional model. In this embodiment, the whole-body initial action video data is collected and identified in real time, so that the user can perform motion capture at any time, which improves the convenience of use.

[0101] In an exemplary embodiment, if Figure 3 As shown, after obtaining the target full-body motion of the three-dimensional model, it also includes:

[0102] In step S310, when a re-capture instruction is received and the part category information included in the re-capture instruction is whole body, updated action video data of the whole body is captured, wherein the initial action video data of the whole body and the updated action video data of the whole body are obtained through the same set of whole body performance actions.

[0103] In step S320, the updated action video data of the whole body is identified to obtain the updated posture estimation result of the whole body.

[0104] In step S330, the three-dimensional model is driven according to the updated posture estimation result of the whole body to obtain an updated initial whole-body motion of the three-dimensional model.

[0105] Specifically, during the playback of the initial whole-body action or the target whole-body action, if the user judges that the final action effect does not meet the requirements by previewing the action effect of the three-dimensional model in real time, the user can select the re-collected part category through the terminal to trigger the re-collection instruction. When the part category information included in the re-collection instruction is the whole body, the user performs the same set of whole-body actions toward the video acquisition device, so that the terminal obtains the updated action video data of the whole body re-collected by the video acquisition device. The terminal can identify the updated posture estimation result of the whole body in real time or offline. While obtaining the updated posture estimation result of the whole body, the three-dimensional model can also be driven according to the updated posture estimation result of the whole body to obtain the updated initial whole-body action of the three-dimensional model.

[0106] In this embodiment, when the user determines that the obtained whole-body motion does not meet the requirements, the user can re-capture updated motion video data of the whole body by triggering a re-capture instruction, thereby improving the operability of motion capture.

[0107] In an exemplary embodiment, if Figure 4As shown, after obtaining the target full-body motion of the three-dimensional model, it also includes:

[0108] In step S410, when a re-collection instruction is received and the part category information included in the re-collection instruction is a detail part, the initial whole body motion is played.

[0109] In step S420, the updated action video data of the detail parts are collected in real time during the playing of the initial whole-body action, wherein the updated action video data of the detail parts and the target action video data of the detail parts are obtained through the same set of performance actions of the detail parts.

[0110] In step S430, the updated action video data of the detail part is identified to obtain the updated posture estimation result of the detail part.

[0111] In step S440, the three-dimensional model is driven together according to the updated posture estimation results of the detail parts and the posture estimation results of other parts to obtain an updated whole-body motion of the three-dimensional model.

[0112] Specifically, during the process of playing the target full-body action, if the user judges that the final presented action synchronization effect does not meet the requirements by previewing the action effect of the three-dimensional model in real time, the user can select the re-collected part category through the terminal to trigger the re-collection instruction. When the part category information included in the re-collection instruction is the detail part, the terminal responds to the re-collection instruction and replays the initial full-body action of the three-dimensional model. During the process of playing the initial full-body action, the user performs the same set of detail part actions as the playback progress of the initial full-body action. That is, the video acquisition device is used to collect the detail part to obtain the updated action video data of the detail part. The terminal calls the second neural network model to identify the updated action video data of the detail part to obtain the updated posture estimation result of the detail part. The three-dimensional model is driven together according to the updated posture estimation result of the detail part currently output and the posture estimation results of other parts currently played to obtain the updated full-body action of the three-dimensional model.

[0113] In this embodiment, when the user determines that the motion synchronization effect of the detail part and other parts does not meet the requirements, the user can re-capture the updated motion video data of the detail part by triggering a re-capture instruction, thereby improving the operability of motion capture.

[0114] Figure 5It is a flowchart of a specific motion capture method according to an exemplary embodiment. In the related art, when the human body is captured by a markerless visual motion capture method, the hands and face occupy too few pixels on the screen and are too blurry, and the hands are often blocked by the body, making it difficult to obtain clear full-body movements. Therefore, in this embodiment, the details include hands and faces. The hands include the left hand and the right hand. At least one neural network model capable of recognizing the whole body, hands and face is pre-deployed in the terminal. Figure 5 As shown, the motion capture method includes the following steps.

[0115] In step S501, the initial action video data of the user's whole body is collected by a video acquisition device. The video acquisition device may be a WEBCAM (WEB CAMER, network camera) camera. In order to ensure the integrity and continuity of the performance.

[0116] In step S502, a body posture capture algorithm is used to identify the initial motion video data of the whole body to obtain a whole body posture estimation result. The body posture capture algorithm can be a real-time algorithm or an offline algorithm, which is not limited here.

[0117] Among them, the whole body posture estimation result is not limited to including the rotation angles of each skeletal joint. Since the pixel area of ​​the hands and face in the whole picture is very small, the information of the hands and face may not be used by the body posture capture algorithm.

[0118] In step S503, the three-dimensional model is driven according to the whole-body posture estimation result to obtain an initial whole-body motion of the three-dimensional model.

[0119] In step S504, the target motion video data of the first hand is collected during the process of playing the initial full-body motion of the three-dimensional model. Specifically, a WEBCAM camera can be used to collect the target motion video data of the first hand of the user.

[0120] In step S505, a gesture capture algorithm is used to identify the target action video data of the first hand to obtain a target posture estimation result of the first hand. The gesture capture algorithm can be any algorithm that can be used for gesture posture estimation, for example, Hand Pose Estimation. The target posture estimation result of the first hand is not limited to including the rotation angle of each finger joint in the first hand. The gesture capture algorithm is a real-time algorithm, that is, the target posture estimation result of the first hand is obtained by real-time identification while collecting the target action video data of the first hand.

[0121] In step S506, the three-dimensional model is driven according to the target posture estimation result of the first hand and the posture estimation results of other parts to obtain the first hand movement and other parts movement of the three-dimensional model. Further, the terminal plays the first hand movement and other parts movement of the three-dimensional model in real time on the screen, so that the user can preview the movement synchronization effect of the first hand and other parts in real time, ensuring the movement synchronization of the first hand and other parts.

[0122] In step S507, during the process of playing the first hand movement and other body movements of the three-dimensional model, the target movement video data of the second hand is collected. Specifically, a WEBCAM camera can be used to collect the video data by aiming at the second hand of the user.

[0123] In step S508, a gesture capture algorithm is used to identify the target action video data of the second hand, and obtain the target posture estimation result of the second hand. The target posture estimation result of the second hand is not limited to the rotation angle of each finger joint in the second hand. The gesture capture algorithm is a real-time algorithm, that is, the target posture estimation result of the second hand is obtained by real-time recognition while collecting the target action video data of the second hand.

[0124] In step S509, the three-dimensional model is driven according to the target posture estimation result of the first hand, the target posture estimation result of the second hand, and the posture estimation results of other parts, to obtain the first hand movement, the second hand movement, and the movement of other parts of the three-dimensional model. Furthermore, the terminal plays the first hand movement, the second hand movement, and the movement of other parts of the three-dimensional model in real time on the screen, so that the user can preview the movement synchronization effect of the first hand, the second hand, and other parts in real time, and ensure the movement synchronization of the first hand, the second hand, and other parts.

[0125] Among them, in steps S504 to S509, the first hand is the left hand and the second hand is the right hand; or the first hand is the right hand and the second hand is the left hand.

[0126] In step S510, facial motion video data is collected during the process of playing the first hand motion, the second hand motion and other body motions of the three-dimensional model. Specifically, a WEBCAM camera can be used to collect the facial motion video data.

[0127] In step S511, an expression recognition algorithm is used to recognize the target action video data of the face, and obtain the target posture estimation result of the face. The hand expression recognition algorithm is a real-time algorithm, that is, the target posture estimation result of the face is obtained by real-time recognition while collecting the target action video data of the face. Among them, the expression recognition algorithm can be any algorithm that can be used for expression recognition, for example, Deep Facial Expression Recognition (deep expression recognition algorithm).

[0128] In step S512, the three-dimensional model is driven according to the target posture estimation result of the first hand, the target posture estimation result of the second hand, the target posture estimation result of the face, and the posture estimation results of other parts to obtain the target full-body motion of the three-dimensional model. Further, the terminal plays the target full-body motion of the three-dimensional model in real time on the screen, so that the user can preview the synchronization effect of the full-body motion in real time to ensure the synchronization of the full-body motion.

[0129] It should be noted that the above-mentioned order of collecting the first hand, the second hand and the face is an exemplary description of this embodiment. In actual application, the user can adjust the order of collecting the movements of each detail part at will according to the needs.

[0130] Furthermore, in this embodiment, if the user judges that the motion capture effect does not meet the requirements through the effect of the real-time preview of the action of the 3D model, the user can select the part category to be re-collected through the terminal to trigger the re-collection instruction. The terminal responds to the re-collection instruction and re-collects the updated action video data corresponding to the part category information according to the part category information carried in the re-collection instruction.

[0131] It should be understood that, although the various steps in the above-mentioned flow chart are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps is not strictly limited in order, and these steps can be executed in other orders. Moreover, at least a part of the steps in the above-mentioned flow chart may include a plurality of sub-steps or a plurality of stages, and these sub-steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these sub-steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the sub-steps or stages in other steps.

[0132] Figure 6 FIG. 6 is a block diagram of a motion capture device 600 according to an exemplary embodiment. Figure 6 The device includes an acquisition module 601, a driving module 602, a collection module 603 and a detail part recognition module 604.

[0133] An acquisition module 601 is configured to acquire a whole-body posture estimation result, wherein the whole-body posture estimation result includes a posture estimation result of a detail part and a posture estimation result of other parts except the detail part, wherein the detail part is a part of the whole body;

[0134] A driving module 602 is configured to drive the three-dimensional model according to the whole-body posture estimation result to obtain an initial whole-body motion of the three-dimensional model;

[0135] The acquisition module 603 is configured to acquire the target action video data of the detail part in real time during the playing of the initial whole body action;

[0136] The detail part recognition module 604 is configured to perform recognition of the target action video data of the detail part to obtain the target posture estimation result of the detail part;

[0137] The driving module 602 is further configured to jointly drive the three-dimensional model according to the target posture estimation results of the detail parts and the posture estimation results of other parts to obtain the target whole-body movement of the three-dimensional model.

[0138] In an exemplary embodiment, the number of detail parts includes multiple ones; the acquisition module 603 is configured to execute the acquisition of target action video data of each detail part separately during the playback of the initial whole-body action; the detail part recognition module 604 is configured to execute the recognition of the target action video data of each detail part to obtain the target posture estimation result of each detail part.

[0139] In an exemplary embodiment, the driving module 602 is configured to execute, in the process of obtaining a target posture estimation result of a detail part, jointly drive the three-dimensional model according to the target posture estimation result of the detail part and the posture estimation results of other parts in real time until the target posture estimation results of all detail parts are obtained; and jointly drive the three-dimensional model according to the target posture estimation results of all detail parts and the posture estimation results of other parts to obtain the target whole-body movement of the three-dimensional model.

[0140] In an exemplary embodiment, the acquisition module 601 is also configured to execute the acquisition of pre-saved initial motion video data of the whole body; the device 600 also includes: a whole body posture recognition module, configured to execute the recognition of pre-saved initial motion video data of the whole body to obtain a whole body posture estimation result.

[0141] In an exemplary embodiment, the acquisition module 603 is also configured to perform real-time acquisition of initial motion video data of the whole body; the device 600 also includes: a whole body posture recognition module, configured to perform recognition of the real-time acquired initial motion video data of the whole body to obtain a whole body posture estimation result.

[0142] In an exemplary embodiment, the device 600 also includes: a receiving module, configured to execute receiving a re-capture instruction; a capture module 603, further configured to execute when the part category information included in the re-capture instruction is the whole body, to capture the updated action video data of the whole body, wherein the initial action video data of the whole body and the updated action video data of the whole body are obtained through the same set of whole body performance actions; the device 600 also includes: a whole body posture recognition module, configured to execute recognition of the updated action video data of the whole body to obtain an updated posture estimation result of the whole body; a driving module 602, further configured to drive the three-dimensional model according to the updated posture estimation result of the whole body to obtain an updated initial whole-body action of the three-dimensional model.

[0143] In an exemplary embodiment, the device 600 also includes: a receiving module, configured to execute receiving a re-capture instruction; a playing module, configured to execute playing the initial whole-body action when the part category information included in the re-capture instruction is a detail part; the acquisition module 603 is further configured to execute real-time acquisition of updated action video data of the detail part during the playback of the initial whole-body action, wherein the updated action video data of the detail part and the target action video data of the detail part are obtained through the same set of performance actions of the detail part; the detail part recognition module 604 is further configured to execute recognition of the updated action video data of the detail part to obtain an updated posture estimation result of the detail part; the driving module 602 is further configured to jointly drive the three-dimensional model according to the updated posture estimation result of the detail part and the posture estimation results of other parts to obtain an updated whole-body action of the three-dimensional model.

[0144] In an exemplary embodiment, the detail part includes at least one of a hand and a face.

[0145] Regarding the device in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.

[0146] Figure 7 700 is a block diagram of a device for motion capture according to an exemplary embodiment. For example, the device 700 may be a mobile phone, a computer, a digital broadcast terminal, a messaging device, a game console, a tablet device, a medical device, a fitness device, a personal digital assistant, etc.

[0147] Reference Figure 7 , device 700 may include one or more of the following components: a processing component 702 , a memory 704 , a power component 706 , a multimedia component 708 , an audio component 710 , an input / output (I / O) interface 712 , a sensor component 714 , and a communication component 716 .

[0148] The processing component 702 generally controls the overall operation of the device 700, such as operations associated with display, phone calls, data communications, camera operations, and recording operations. The processing component 702 may include one or more processors 720 to execute instructions to complete all or part of the steps of the above-mentioned method. In addition, the processing component 702 may include one or more modules to facilitate the interaction between the processing component 702 and other components. For example, the processing component 702 may include a multimedia module to facilitate the interaction between the multimedia component 708 and the processing component 702.

[0149] The memory 704 is configured to store various types of data to support operations on the device 700. Examples of such data include instructions for any application or method operating on the device 700, contact data, phone book data, messages, pictures, videos, etc. The memory 704 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk, or optical disk.

[0150] The power supply component 706 provides power to the various components of the device 700. The power supply component 706 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to the device 700.

[0151] The multimedia component 708 includes a screen that provides an output interface between the device 700 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touch screen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touch, slide, and gestures on the touch panel. The touch sensor may not only sense the boundaries of the touch or slide action, but also detect the duration and pressure associated with the touch or slide operation. In some embodiments, the multimedia component 708 includes a front camera and / or a rear camera. When the device 700 is in an operating mode, such as a shooting mode or a video mode, the front camera and / or the rear camera may receive external multimedia data. Each front camera and rear camera may be a fixed optical lens system or have a focal length and optical zoom capability.

[0152] The audio component 710 is configured to output and / or input audio signals. For example, the audio component 710 includes a microphone (MIC), and when the device 700 is in an operating mode, such as a call mode, a recording mode, and a speech recognition mode, the microphone is configured to receive an external audio signal. The received audio signal can be further stored in the memory 704 or sent via the communication component 716. In some embodiments, the audio component 710 also includes a speaker for outputting audio signals.

[0153] I / O interface 712 provides an interface between processing component 702 and peripheral interface modules, such as keyboards, click wheels, buttons, etc. These buttons may include but are not limited to: a home button, a volume button, a start button, and a lock button.

[0154] The sensor assembly 714 includes one or more sensors for providing various aspects of status assessment for the device 700. For example, the sensor assembly 714 can detect the open / closed state of the device 700, the relative positioning of components, such as the display and keypad of the device 700, and the sensor assembly 714 can also detect the position change of the device 700 or a component of the device 700, the presence or absence of user contact with the device 700, the orientation or acceleration / deceleration of the device 700, and the temperature change of the device 700. The sensor assembly 714 may include a proximity sensor configured to detect the presence of nearby objects without any physical contact. The sensor assembly 714 may also include a light sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, the sensor assembly 714 may also include an acceleration sensor, a gyroscope sensor, a magnetic sensor, a pressure sensor, or a temperature sensor.

[0155] The communication component 716 is configured to facilitate wired or wireless communication between the device 700 and other devices. The device 700 can access a wireless network based on a communication standard, such as WiFi, a carrier network (such as 2G, 3G, 4G or 5G), or a combination thereof. In an exemplary embodiment, the communication component 716 receives a broadcast signal or broadcast-related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component 716 also includes a near field communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on radio frequency identification (RFID) technology, infrared data association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology and other technologies.

[0156] In an exemplary embodiment, the device 700 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to perform the above methods.

[0157] In an exemplary embodiment, a non-transitory computer-readable storage medium including instructions is also provided, such as a memory 704 including instructions, and the instructions can be executed by the processor 720 of the device 700 to perform the above method. For example, the non-transitory computer-readable storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc.

[0158] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. The present disclosure is intended to cover any variations, uses or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary techniques in the art that are not disclosed in the present disclosure. The description and examples are to be considered exemplary only, and the true scope and spirit of the present disclosure are indicated by the following claims.

[0159] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims

1. A motion capture method, characterized in that: include: Acquire a whole-body posture estimation result, wherein the whole-body posture estimation result includes a posture estimation result of a detail part and a posture estimation result of other parts except the detail part, wherein the detail part is a part of the whole body, and the number of the detail parts includes a plurality; driving the three-dimensional model according to the whole-body posture estimation result to obtain an initial whole-body motion of the three-dimensional model; In the process of playing the initial whole-body movement, real-time and individual target movement video data of each detail part is collected; Identify the target action video data of each detail part to obtain the target posture estimation result of each detail part; In the process of obtaining each target pose estimation result of a detail part, the three-dimensional model is jointly driven in real time according to the target pose estimation result of the detail part obtained and the pose estimation results of the other parts until the target pose estimation results of all the detail parts are obtained; The three-dimensional model is driven together according to the target posture estimation results of all the detailed parts and the posture estimation results of the other parts to obtain the target whole-body motion of the three-dimensional model.

2. The motion capture method according to claim 1, characterized in that: The step of obtaining the whole body posture estimation result includes: Obtaining pre-saved initial action video data of the whole body; The pre-saved full-body initial motion video data is identified to obtain a full-body posture estimation result.

3. The motion capture method according to claim 1, characterized in that: The step of obtaining the whole body posture estimation result includes: Collecting the initial motion video data of the whole body in real time; The real-time collected full-body initial motion video data is identified to obtain a full-body posture estimation result.

4. The motion capture method according to claim 2 or 3, characterized in that: After obtaining the target whole-body motion of the three-dimensional model, the method further includes: When a re-collection instruction is received, and the part category information included in the re-collection instruction is the whole body, collecting updated action video data of the whole body, wherein the initial action video data of the whole body and the updated action video data of the whole body are obtained through the same set of performance actions of the whole body; Identify the updated action video data of the whole body to obtain the updated posture estimation result of the whole body; The three-dimensional model is driven according to the updated whole-body posture estimation result to obtain an updated initial whole-body motion of the three-dimensional model.

5. The motion capture method according to claim 1, characterized in that: After obtaining the target whole-body motion of the three-dimensional model, the method further includes: When receiving a re-collection instruction, and the part category information included in the re-collection instruction is a detail part, playing the initial whole-body motion; In the process of playing the initial whole-body action, the updated action video data of the detail part is collected in real time, wherein the updated action video data of the detail part and the target action video data of the detail part are obtained through the same set of performance actions of the detail part; Identify the updated action video data of the detail part to obtain the updated posture estimation result of the detail part; The three-dimensional model is driven together according to the updated posture estimation results of the detail parts and the posture estimation results of the other parts to obtain an updated whole-body motion of the three-dimensional model.

6. The motion capture method according to claim 1, characterized in that: The detail parts include at least one of hands and face.

7. A motion capture device, characterized in that: include: an acquisition module, configured to execute acquisition of a whole-body posture estimation result, wherein the whole-body posture estimation result includes a posture estimation result of a detail part and a posture estimation result of other parts except the detail part, wherein the detail part is a part of the whole body, and the number of the detail parts includes a plurality; A driving module, configured to drive the three-dimensional model according to the whole-body posture estimation result to obtain an initial whole-body motion of the three-dimensional model; A collection module is configured to collect target action video data of each detail part in real time during the playing of the initial whole-body action; A detail part recognition module is configured to perform recognition of the target action video data of each detail part to obtain a target posture estimation result of each detail part; The driving module is further configured to execute, in the process of obtaining a target posture estimation result of a detail part, jointly drive the three-dimensional model according to the target posture estimation results of the detail parts and the posture estimation results of the other parts in real time until the target posture estimation results of all the detail parts are obtained, and jointly drive the three-dimensional model according to the target posture estimation results of all the detail parts and the posture estimation results of the other parts to obtain the target whole-body movement of the three-dimensional model.

8. The motion capture device according to claim 7, characterized in that: The acquisition module is further configured to acquire pre-saved initial action video data of the whole body; The device further comprises: a whole body posture recognition module configured to perform recognition of the pre-stored whole body initial motion video data to obtain the whole body posture estimation result.

9. The motion capture device according to claim 7, characterized in that: The acquisition module is further configured to perform real-time acquisition of the initial action video data of the whole body; The device further comprises: a whole body posture recognition module configured to perform recognition of the whole body initial motion video data collected in real time to obtain the whole body posture estimation result.

10. The motion capture device according to claim 8 or 9, characterized in that: The device also includes: A receiving module, configured to execute a receiving re-acquisition instruction; The acquisition module is further configured to acquire updated action video data of the whole body when the part category information included in the re-acquisition instruction is the whole body, wherein the initial action video data of the whole body and the updated action video data of the whole body are obtained through the same set of performance actions of the whole body; The whole body posture recognition module is further configured to perform recognition of the whole body update action video data to obtain the whole body update posture estimation result; The driving module is further configured to drive the three-dimensional model according to the updated posture estimation result of the whole body to obtain an updated initial whole-body motion of the three-dimensional model.

11. The motion capture device according to claim 7, characterized in that: The device also includes: A receiving module, configured to execute a receiving re-acquisition instruction; The device further comprises: a playing module configured to play the initial whole-body action when the part category information included in the re-collection instruction is a detail part; The acquisition module is further configured to acquire the updated action video data of the detail part in real time during the playing of the initial whole-body action, wherein the updated action video data of the detail part and the target action video data of the detail part are obtained through the same set of performance actions of the detail part; The detail part recognition module is further configured to execute the recognition of the update action video data of the detail part to obtain the update posture estimation result of the detail part; The driving module is further configured to jointly drive the three-dimensional model according to the updated posture estimation results of the detail parts and the posture estimation results of the other parts to obtain an updated whole-body movement of the three-dimensional model.

12. The motion capture device according to any one of claim 7, characterized in that: The detail parts include at least one of hands and face.

13. An electronic device, characterized in that: include: processor; a memory for storing instructions executable by the processor; The processor is configured to execute the instructions to implement the motion capture method as described in any one of claims 1 to 6.

14. A storage medium, characterized in that: When the instructions in the storage medium are executed by a processor of an electronic device, the electronic device is enabled to execute the motion capture method as claimed in any one of claims 1 to 6.

15. A computer program product, comprising instructions, characterized in that: When the instruction is executed by a processor of an electronic device, the electronic device is enabled to execute the motion capture method as described in any one of claims 1 to 6.

Images