Motion capture method and device, medium, equipment and computer program product

Abstract

The invention relates to a motion capture method and device, a medium, equipment and a computer program product. The method comprises the following steps: acquiring a motion video corresponding to a target object; performing image feature extraction on image frames in the motion video to obtain image features of the image frames; key point extraction is carried out on the image frame, and corresponding target key point information of the target object in the image frame is obtained; based on the image features, the target key point information and a pre-trained motion capture estimation model, three-dimensional motion parameters corresponding to the target object are obtained, and the three-dimensional motion parameters comprise attitude parameters and shape parameters. Therefore, the motion capture analysis can be performed on the target object by obtaining the motion video of the target object, so that the motion of the target object can be detected and recognized subsequently, a corresponding tracker does not need to be configured for the target object in the process, and the motion capture detection process of the target object can be effectively simplified while the cost of motion capture analysis is reduced.

Description

Technical Field

[0001] The present disclosure relates to the field of computer technology, and in particular, to a motion capture method, apparatus, medium, device and computer program product. Background Art

[0002] With the development of artificial intelligence technology, robots can simulate actions in the real world. Motion capture, also known as dynamic capture, or mocap for short, refers to the technology of recording and processing the actions of people or other objects. In related technologies, it is usually adopted to affix multiple identification points on the observer, such as pasting a tracker at the key joint position of the observer, so that the position of the key joint can be determined based on the position detected by the tracker, and then the position is transformed into a three-dimensional space coordinate and mapped to the corresponding three-dimensional virtual object, so as to realize the conversion of real action to virtual skeletal action. However, the above process needs to be combined with action equipment, the operation process is complicated and the requirements for application scenarios are strict. Summary of the invention

[0003] This summary is provided to introduce concepts in a brief form that will be described in detail in the detailed description below. This summary is not intended to identify key features or essential features of the claimed technical solution, nor is it intended to limit the scope of the claimed technical solution.

[0004] In a first aspect, the present disclosure provides a motion capture method, the method comprising: Obtaining the motion video corresponding to the target object; Extracting image features from image frames in the motion video to obtain image features of the image frames; Extracting key points from the image frame to obtain target key point information corresponding to the target object in the image frame; Based on the image features, the target key point information and a pre-trained motion capture estimation model, three-dimensional motion parameters corresponding to the target object are obtained, and the three-dimensional motion parameters include posture parameters and shape parameters.

[0005] In a second aspect, the present disclosure provides a motion capture device, the device comprising: An acquisition module, used to acquire the motion video corresponding to the target object; A first extraction module, used for extracting image features from image frames in the motion video to obtain image features of the image frames; A second extraction module is used to extract key points from the image frame to obtain target key point information corresponding to the target object in the image frame; The first processing module is used to obtain three-dimensional motion parameters corresponding to the target object based on the image features, the target key point information and a pre-trained motion capture estimation model, wherein the three-dimensional motion parameters include posture parameters and shape parameters.

[0006] In a third aspect, the present disclosure provides a computer-readable medium having a computer program stored thereon, which implements the steps of the method described in the first aspect when the computer program is executed by a processing device.

[0007] In a fourth aspect, the present disclosure provides an electronic device, including: a storage device having a computer program stored thereon; A processing device is used to execute the computer program in the storage device to implement the steps of the method described in the first aspect.

[0008] In a fifth aspect, the present disclosure provides a computer program product, including a computer program, which implements the steps of the method described in the first aspect when executed by a processor.

[0009] Therefore, through the above technical solution, the target object can be analyzed for motion capture by obtaining the motion video of the target object, so as to detect and identify the target object's action later. In this process, there is no need to configure a corresponding tracker for the target object, which reduces the cost of motion capture analysis and effectively simplifies the motion capture detection process of the target object. In addition, by combining image features and key point information in the image frame for analysis, accurate and comprehensive data support can be provided for motion capture analysis, thereby improving the accuracy of the determined three-dimensional motion parameters and providing effective data for subsequent motion processing based on the three-dimensional motion parameters.

[0010] Other features and advantages of the present disclosure will be described in detail in the following detailed description. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] The above and other features, advantages and aspects of the embodiments of the present disclosure will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. Throughout the drawings, the same or similar reference numerals represent the same or similar elements. It should be understood that the drawings are schematic and the originals and elements are not necessarily drawn to scale. In the drawings: Figure 1 It is a flowchart of a motion capture method provided according to an embodiment of the present disclosure.

[0012] Figure 2 It is a schematic diagram of the training process of a motion capture estimation model provided according to an embodiment of the present disclosure.

[0013] Figure 3 It is a schematic diagram of an image in a motion video provided according to an embodiment of the present disclosure.

[0014] Figure 4 It is a block diagram of a motion capture device provided according to an embodiment of the present disclosure.

[0015] Figure 5 A schematic diagram of the structure of an electronic device suitable for implementing the embodiments of the present disclosure is shown. DETAILED DESCRIPTION

[0016] Embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as being limited to the embodiments described herein, which are instead provided for a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are only for exemplary purposes and are not intended to limit the scope of protection of the present disclosure.

[0017] It should be understood that the various steps described in the method embodiments of the present disclosure may be performed in different orders and / or in parallel. In addition, the method embodiments may include additional steps and / or omit the steps shown. The scope of the present disclosure is not limited in this respect.

[0018] The term "including" and its variations used herein are open inclusions, i.e., "including but not limited to". The term "based on" means "based at least in part on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments". The relevant definitions of other terms will be given in the following description.

[0019] It should be noted that the concepts such as "first" and "second" mentioned in the present disclosure are only used to distinguish different devices, modules or units, and are not used to limit the order or interdependence of the functions performed by these devices, modules or units.

[0020] It should be noted that the modifications of "one" and "plurality" mentioned in the present disclosure are illustrative rather than restrictive, and those skilled in the art should understand that unless otherwise clearly indicated in the context, it should be understood as "one or more".

[0021] The names of the messages or information exchanged between multiple devices in the embodiments of the present disclosure are only used for illustrative purposes and are not used to limit the scope of these messages or information.

[0022] It is understandable that before using the technical solutions disclosed in the embodiments of the present disclosure, the types, scope of use, usage scenarios, etc. of the personal information involved in the present disclosure should be informed to the user and the user's authorization should be obtained in an appropriate manner in accordance with relevant laws and regulations.

[0023] For example, in response to receiving an active request from a user, a prompt message is sent to the user to clearly prompt the user that the operation requested to be performed will require obtaining and using the user's personal information. Thus, the user can autonomously choose whether to provide personal information to software or hardware such as an electronic device, application, server, or storage medium that performs the operation of the technical solution of the present disclosure according to the prompt message.

[0024] As an optional but non-limiting implementation, in response to receiving an active request from the user, the prompt information may be sent to the user in the form of a pop-up window, in which the prompt information may be presented in text form. In addition, the pop-up window may also carry a selection control for the user to choose "agree" or "disagree" to provide personal information to the electronic device.

[0025] It is understandable that the above notification and the process of obtaining user authorization are merely illustrative and do not constitute a limitation on the implementation of the present disclosure. Other methods that meet the relevant laws and regulations may also be applied to the implementation of the present disclosure.

[0026] At the same time, it is understandable that the data involved in this technical solution (including but not limited to the data itself, the acquisition or use of the data) shall comply with the requirements of relevant laws, regulations and relevant provisions.

[0027] Figure 1 As shown, it is a flow chart of a motion capture method provided according to an embodiment of the present disclosure, as shown in Figure 1 As shown, the method may include: In step 11, a motion video corresponding to the target object is obtained.

[0028] Among them, the sports video can be a video obtained in real time, such as the live video of the anchor in a live broadcast scene, or it can be a user's sports recording video in a fitness scene, etc., which can be obtained by real-time recording through a camera.

[0029] In step 12, image features are extracted from image frames in the motion video to obtain image features of the image frames.

[0030] Among them, feature extraction can be performed for each captured image frame, for example, feature extraction can be performed based on an image feature encoder commonly used in the art. For another example, the image feature can be obtained based on an image processing model obtained by pre-model training. The image processing model can be obtained by model training based on multiple image frames in the video as model inputs and the action type corresponding to the video as the target output, and the output of the feature layer of the image processing model is used as the image feature. Among them, the training process of the image processing model can be performed based on a common training method in the art, and the present disclosure does not limit this.

[0031] In step 13, key points are extracted from the image frame to obtain target key point information corresponding to the target object in the image frame.

[0032] Among them, the key points can be 2D key points, which are used to represent the key points corresponding to the motion characteristics of the target object. The key points of different target objects can be set in advance. For example, the key points of the human body can include the neck, left and right elbows, left and right knees, left ankle, right ankle and other key points. The key points of the quadruped motion body can include the head, left front leg, right front leg, left hind leg, right hind leg and other key points. They can be configured based on the actual application scenario, and the present disclosure does not limit this.

[0033] For example, training samples can be obtained by pre-marking key points to train the key point extraction model. In this step, the image frame can be input into the key point extraction model to obtain the key points of the identified target object and obtain the target key point information.

[0034] In step 14, based on the image features, target key point information and a pre-trained motion capture estimation model, three-dimensional motion parameters corresponding to the target object are obtained, and the three-dimensional motion parameters include posture parameters and shape parameters.

[0035] As an example, the three-dimensional motion parameters can be represented by SMPL (Skinned Multi-Person Linear) parameters, which define the pose parameter pose (β) and shape parameter shape (θ) to control the motion posture change of the target object. Among them, the motion estimation model can be implemented based on the GRU (Gated Recurrent Unit) model, which is a deep learning model that can be used to process time series data to learn the long-term dependencies in time series data.

[0036] Among them, the shape parameters can be represented by numerical values ​​of multiple dimensions to represent the shape of the target object. Taking the target object as a human body as an example, the value of each dimension can be interpreted as an indicator of the human body shape, such as height, fatness, etc. The posture parameter can be represented by a numerical value of M×N dimensions to describe the action posture of the target object at a certain moment, where M is used to represent the number of pre-defined key points of the target object, and N is used to represent the axis-angle representation of the rotation angle of the key point relative to its parent node. For example, in a human body scene, the key point can be a human joint point, and the posture parameter can be represented by a numerical value of 24×3 dimensions. Therefore, the three-dimensional posture of the target object at the current moment can be represented by the posture parameter and the shape parameter.

[0037] Therefore, through the above technical solution, the target object can be analyzed for motion capture by obtaining the motion video of the target object, so as to detect and identify the target object's action later. In this process, there is no need to configure a corresponding tracker for the target object, which reduces the cost of motion capture analysis and effectively simplifies the motion capture detection process of the target object. In addition, by combining image features and key point information in the image frame for analysis, accurate and comprehensive data support can be provided for motion capture analysis, thereby improving the accuracy of the determined three-dimensional motion parameters and providing effective data for subsequent motion processing based on the three-dimensional motion parameters.

[0038] In a possible embodiment, the motion capture estimation model is determined in the following manner: First training samples are obtained, wherein each of the first training samples includes key point information corresponding to a plurality of image frames of a first training video.

[0039] As an example, the first training sample can be obtained from an open source dataset. For example, the key point information in each image frame of the video is extracted and annotated in advance and uploaded to the open source dataset for user sharing. For another example, the first training sample can also be generated based on existing video data and a pre-trained key point extraction model.

[0040] The preset model is trained based on the first training sample to obtain a first model.

[0041] As an example, the preset model can be implemented based on the GRU model, which can include an encoder and a decoder. The implementation of this step is as follows: Figure 2 The solid line logic is shown in: The key point information corresponding to the image frame in the first training video is input into the encoder of the preset model to obtain the first training key point features corresponding to the image frame.

[0042] As an example, images may be input sequentially according to the time sequence of image frames in the first training video. Then, key point information corresponding to the first image frame may be input at the t0 time step, and feature encoding may be performed based on the encoder to obtain the first training key point feature. The encoder may be a Motion Encoder.

[0043] The decoder based on the preset model decodes the first training key point feature to obtain the predicted posture parameters, predicted shape parameters and predicted camera parameters corresponding to the image frame. The decoder may be a MotionDecoder. The camera parameters are used to project the three-dimensional motion parameters into a two-dimensional plane, and the camera parameters may include the position information of the target object in the camera coordinate system, and the projection matrix of the three-dimensional coordinate system to the two-dimensional plane.

[0044] Afterwards, plane projection is performed based on the predicted camera parameters, the predicted posture parameters, and the predicted shape parameters to obtain training key point information corresponding to the posture represented by the predicted posture parameters and the predicted shape parameters; Based on the training key point information and the key point information corresponding to the image frame, the preset model is trained to obtain the first model.

[0045] As an example, the predicted posture parameters and the predicted shape parameters can be projected into a two-dimensional plane based on the predicted camera parameters. For example, the three-dimensional key point information can be determined based on the predicted posture parameters and the predicted shape parameters. Then, the three-dimensional key point information can be projected into a two-dimensional plane based on the predicted camera parameters to obtain the training key point information. Afterwards, the training key point information and the key point information corresponding to the image frame are subjected to loss calculation. For example, the L2 distance between each key point coordinate in the training key point information and each key point coordinate in the key point information corresponding to the image frame can be used as a loss function to determine the model loss, and then the preset model is trained based on the model loss to obtain the first model. Among them, the training process can be implemented based on the common training method in the art, and the present disclosure is not limited to this.

[0046] As another example, the first training sample may also include posture parameters, shape parameters, and camera parameters corresponding to multiple image frames of the first training video, which may be obtained based on calibration data in open source data. In this embodiment, when determining the model loss, the loss corresponding to the key point information may be determined based on the method described above, and the loss of posture parameters, shape parameters, and camera parameters may be determined based on the predicted posture parameters, predicted shape parameters, and predicted camera parameters, as well as the posture parameters, shape parameters, and camera parameters corresponding to the image frames. Afterwards, the various losses may be weightedly fused to obtain a comprehensive loss, and the preset model may be trained based on the comprehensive loss.

[0047] Therefore, through the above technical solution, the first model can be first trained based on open source general data, simplifying the acquisition of training data for the first model without the need for user annotation, so that the first model can learn the relationship between two-dimensional key point information and three-dimensional posture parameters and shape parameters, thereby improving the training efficiency of the first model.

[0048] Afterwards, second training samples are obtained, wherein each of the second training samples includes a plurality of image frames of a second training video and key point information corresponding to the image frames.

[0049] In this embodiment, open source video data or video data obtained after user authorization can be obtained. Further, each image frame in the video data can be sampled, and key point extraction can be performed on the image frame to obtain key point information corresponding to the image frame, thereby obtaining the second training sample.

[0050] Furthermore, the first model is trained based on the second training sample to obtain the motion capture estimation model.

[0051] From the above, it can be seen that the first model can predict the posture parameters and shape parameters based on the key point information. In this step, in order to further improve the accuracy and stability of the model prediction, the corresponding image features can be added to the model to obtain more comprehensive features, thereby improving the accuracy of the prediction results of the first model.

[0052] As an example, the training the first model based on the second training sample to obtain the motion capture estimation model may include: The key point information corresponding to the image frame in the second training video is input into the encoder of the first model to obtain the second training key point features corresponding to the image frame.

[0053] The second training key point feature corresponding to the image frame and the image feature of the image frame are fused to obtain a fused feature.

[0054] Among them, the second training key point feature can be obtained based on the above-mentioned key point feature determination method, and the image feature of the image frame can be obtained based on the image processing model described above, which will not be repeated here. In this step, the dimension of the image feature may be different from the dimension of the second training key point feature. In this case, the image feature can be first mapped to the dimension of the second training key point feature to obtain the mapped image feature, and then the mapped image feature and the second training key point feature are added to obtain the fused feature to ensure the uniformity of the decoder input. New training modules such as Figure 2 The dotted logic is shown in .

[0055] Afterwards, the fusion features are decoded based on the decoder of the first model to obtain the predicted posture parameters, predicted shape parameters and predicted camera parameters corresponding to the image frame; plane projection is performed based on the predicted camera parameters, the predicted posture parameters and the predicted shape parameters to obtain the training key point information corresponding to the posture represented by the predicted posture parameters and the predicted shape parameters; based on the training key point information and the key point information corresponding to the image frame, the first model is trained to obtain the motion capture estimation model.

[0056] The implementation method of this step is the same as the method of training the preset model described above, and will not be repeated here. Therefore, through the above technical solution, the motion posture parameters can be further predicted based on the first model in combination with the image features, thereby improving the comprehensiveness of the fusion features while improving the accuracy and stability of the output results of the motion capture estimation model, so that the disclosed solution can be applied to the real-time motion capture process and reduce the delay of motion capture recognition.

[0057] In a possible embodiment, the three-dimensional motion parameters also include camera parameters. As an example, the image features and target key point information can be input into the motion capture estimation model together, so that the encoder based on the motion capture estimation model can encode the target key point information to obtain the target key point features. Then, the target key point features and image features are fused, and the fused features are decoded by the decoder based on the motion estimation model to obtain the camera parameters, posture parameters and shape parameters. Accordingly, the method may also include: Planar projection is performed based on the camera parameters, the posture parameters and the shape parameters to obtain predicted key point information corresponding to the predicted posture represented by the posture parameters and the shape parameters.

[0058] This step can perform plane projection based on the camera parameters to map the three-dimensional key points determined based on the posture parameters and the shape parameters to a two-dimensional plane to obtain the corresponding predicted key point information under the predicted posture.

[0059] Based on the predicted key point information and the target key point information, an adjustment amount of each parameter of the three-dimensional motion parameters is determined.

[0060] The more similar the predicted key point information is to the target key point information, the more accurate the determined posture parameters, shape parameters, and camera parameters are. Therefore, in this step, the L2 distance between each key point coordinate in the predicted key point information and each key point coordinate in the key point information corresponding to the image frame can be used as difference information, and the difference information can be input into the adjustment amount prediction model to obtain the adjustment amounts corresponding to the posture parameters, shape parameters, and camera parameters, respectively.

[0061] As an example, the adjustment amount prediction model can be trained in advance, which can be implemented by a neural network model and trained after the motion capture estimation model training is completed. For example, the image corresponding to the training video and the training key point information can be input into the motion capture estimation model to obtain the posture parameters, shape parameters and camera parameters. Then, the corresponding key point information can be obtained by the projection method described above. Then, the difference information between the key point information and the target key point information is calculated, and the difference information is used as the input of the adjustment amount prediction model to obtain the adjustment amount of each parameter. Further, the parameters can be adjusted based on the adjustment amounts corresponding to the above parameters to obtain the adjusted parameters, that is, the adjusted three-dimensional motion parameters and the adjusted camera parameters. Afterwards, the adjusted posture parameters and shape parameters are further projected based on the adjusted camera parameters to obtain the adjusted key point information, and the adjustment amount prediction model is trained based on the loss between the adjusted key point information and the target key point information to further improve the accuracy of the output result of the adjustment amount prediction model.

[0062] The parameter is adjusted based on the adjustment amount of each parameter in the three-dimensional motion parameters to obtain updated three-dimensional motion parameters.

[0063] As an example, the sum of posture parameters and the adjustment amount of posture parameters can be used as updated posture parameters, the sum of shape parameters and the adjustment amount of shape parameters can be used as updated shape parameters, and the sum of camera parameters and the adjustment amount of camera parameters can be used as updated camera parameters to obtain updated three-dimensional motion parameters.

[0064] Therefore, through the above technical scheme, the corresponding adjustment amounts can be further determined based on the predicted posture parameters, shape parameters and camera parameters, and they can be adjusted based on the adjustment amounts, thereby improving the accuracy of the updated parameters and improving the matching degree between the determined three-dimensional motion parameters and the real motion of the target object.

[0065] As an example, after determining the updated three-dimensional motion parameters, the step of performing plane projection based on the camera parameters, the posture parameters and the shape parameters to obtain the predicted key point information corresponding to the predicted posture represented by the posture parameters and the shape parameters can be re-executed; the step of determining the adjustment amount of each parameter in the three-dimensional motion parameters based on the predicted key point information and the target key point information; and the step of adjusting the parameter based on the adjustment amount of each parameter in the three-dimensional motion parameters to obtain the updated three-dimensional motion parameters. Among them, the number of loops can be set and the loop can be ended when the number is reached, or the loop can be ended when the difference between the adjusted key point information and the target key point information is less than a threshold. Thus, the three-dimensional motion parameters can be adjusted multiple times to reduce the difference between the adjusted key point information and the target key point information, thereby improving the accuracy of the updated three-dimensional motion parameters and achieving the accuracy of motion capture of the target object.

[0066] In a possible embodiment, the method may further include: Based on the posture parameter and the shape parameter, a target point of the three-dimensional virtual object corresponding to the target object is determined, wherein the target point is a rendering key point corresponding to a target special effect.

[0067] Among them, the method can be applied to the scene of adding special effects, such as adding corresponding special effects to the target object during its movement, such as adding punching special effects or turning special effects to the anchor's actions, or adding virtual pendant special effects to the target object, such as virtual wings, etc. The target special effect can be determined based on the actual selection of the user. If the target special effect is a virtual wing special effect, the rendering key points corresponding to the special effect can be pre-configured under the target special effect, that is, the key points bound between the special effect and the target object. For example, in this scenario, the rendering key points corresponding to the virtual wing special effect are the left point and the right point of the shoulder of the target object.

[0068] The three-dimensional virtual object corresponding to the target object is used to represent the virtual object generated by the target object in the virtual scene. The action of the target object can be mapped to the virtual object for display, such as Figure 3 As shown, U1 is used to represent a target object in video data, and U2 is used to represent a virtual object corresponding to the target object. Special effects are added to the target object by controlling the virtual object.

[0069] Afterwards, the target special effect is rendered based on the target point and the three-dimensional virtual object.

[0070] In this embodiment, the target point can be determined based on the posture parameters and the action parameters. As in the example described above, the target points determined are the left point of the shoulder and the right point of the shoulder. After that, the target special effect can be rendered at the target point. For example, the key point of the target special effect can be bound to the target point. Then, when the target object moves, the target point in the virtual object moves accordingly, and the key point in the target special effect also moves accordingly, so as to achieve the rendering effect that the target special effect moves with the movement of the target object, that is, by rendering the target special effect in the target point of the three-dimensional virtual object, the display effect of adding special effects to the target object in the motion video is achieved. Among them, the rendering of the target special effect can be performed based on the special effect rendering method in the field, which will not be elaborated here.

[0071] As another example, the target special effect is a special effect corresponding to the action performed by the target object. For example, multiple actions and their corresponding special effects can be pre-configured. For example, when the target object turns around, the target special effect is a turning special effect. Another example is when the target object performs an action to trigger a virtual kicking special effect. In this embodiment, the action corresponding to the continuous image frames in the motion video can be identified by the posture parameters and shape parameters of the continuous image frames. When a preset action is identified, the special effect corresponding to the preset action is used as the target special effect and rendered using the special effect rendering method described above.

[0072] Therefore, through the above technical scheme, the accuracy of posture parameters and shape parameters can be improved so as to obtain a three-dimensional virtual object that better matches the target object. By associating special effects rendering with the three-dimensional virtual object, special effects can be added to the target object, thereby improving the matching degree between the target special effects and the target object position and improving the accuracy of special effects rendering.

[0073] In a possible embodiment, the method may further include: Based on the posture parameters and the shape parameters, action information of the motion key points corresponding to the target object when performing the target action is determined.

[0074] Among them, this method can be applied to the scenario of fitness assistance, and the key points of movement can be key points pre-configured in the sports mode, which can be configured based on the actual application scenario. For example, in the high leg raising action, the left knee, right knee, left elbow and right elbow can be used as the key points of operation to monitor the positions of the multiple key points of movement and determine whether the user's movements are standard. The action information is used to represent the action corresponding to the key point of movement and its attribute information. Taking the right knee as an example, its corresponding action information can include information such as height and bending angle, which can be determined based on the general action recognition method in the field, and is not limited here. For example, the height can be determined based on the position of the right knee, and the bending angle can be determined based on the angle corresponding to the line between the right hip and the right knee and the line between the right knee and the right ankle.

[0075] Based on the action information of the motion key point and the reference action information corresponding to the target action, the action similarity is determined.

[0076] As an example, for each movement posture (such as high leg raising), the coach can perform the standard movement in advance and record a standard video in the teaching mode. The movement information of the key points of the movement in each image in the standard video can be determined in the above-mentioned manner as the benchmark movement information corresponding to the target movement.

[0077] As an example, the action similarity can be based on the similarity between the action information of the motion key point and the benchmark action information. For example, the action similarity can be obtained by converting the action information and the benchmark action information into vector representations of the same dimension and then calculating the similarity of the vectors.

[0078] Output prompt information based on the action similarity.

[0079] As an example, if the action similarity is greater than or equal to a preset threshold, a prompt message can be output to remind the user of the action standard. If the action similarity is less than the preset threshold, a prompt message can be output to remind the user that the current action is significantly different from the standard action so that the user can adjust the action in time.

[0080] Therefore, through the above technical solution, the movements of the target object during the movement process can be monitored so as to correct and guide the movements based on the determined posture parameters and shape parameters, so as to improve the standardization of the user's movements, broaden the scope of use of the disclosed method, and enhance the user experience.

[0081] Based on the same inventive concept, the present disclosure also provides a motion capture device, such as Figure 4 As shown, the device 10 comprises: An acquisition module 100 is used to acquire a motion video corresponding to a target object; A first extraction module 200 is used to extract image features from image frames in the motion video to obtain image features of the image frames; The second extraction module 300 is used to extract key points from the image frame to obtain target key point information corresponding to the target object in the image frame; The first processing module 400 is used to obtain three-dimensional motion parameters corresponding to the target object based on the image features, the target key point information and a pre-trained motion capture estimation model, where the three-dimensional motion parameters include posture parameters and shape parameters.

[0082] Optionally, the three-dimensional motion parameters further include camera parameters, and the device further includes: A second processing module is used to perform plane projection based on the camera parameters, the posture parameters and the shape parameters to obtain the predicted key point information corresponding to the predicted posture represented by the posture parameters and the shape parameters; A first determination module, configured to determine an adjustment amount of the three-dimensional motion parameter and an adjustment amount of the camera parameter based on the predicted key point information and the target key point information; An updating module is used to adjust the 3D motion parameters based on the adjustment amount of the 3D motion parameters to obtain updated 3D motion parameters, and to adjust the camera parameters based on the adjustment amount of the camera parameters to obtain updated camera parameters.

[0083] Optionally, the motion capture estimation model is determined by a second determination module, and the second determination module includes: A first acquisition submodule is used to acquire first training samples, wherein each of the first training samples contains key point information corresponding to a plurality of image frames of a first training video; A first training submodule, used to train a preset model based on the first training sample to obtain a first model; A second acquisition submodule is used to acquire second training samples, wherein each of the second training samples includes a plurality of image frames of a second training video and key point information corresponding to the image frames; The second training submodule is used to train the first model based on the second training sample to obtain the motion capture estimation model.

[0084] Optionally, the preset model is implemented based on a long short-term memory model, and the first training submodule includes: A first encoding submodule, used to input key point information corresponding to the image frame in the first training video into the encoder of the preset model to obtain a first training key point feature corresponding to the image frame; A first decoding submodule, configured to decode the first training key point feature based on a decoder of the preset model to obtain predicted posture parameters, predicted shape parameters, and predicted camera parameters corresponding to the image frame; A first processing submodule, configured to perform plane projection based on the predicted camera parameters, the predicted posture parameters, and the predicted shape parameters, to obtain training key point information corresponding to the posture represented by the predicted posture parameters and the predicted shape parameters; The third training submodule is used to train the preset model based on the training key point information and the key point information corresponding to the image frame to obtain the first model.

[0085] Optionally, the second training submodule includes: A second encoding submodule, used for inputting key point information corresponding to the image frame in the second training video into the encoder of the first model to obtain second training key point features corresponding to the image frame; A fusion submodule, used for fusing the second training key point feature corresponding to the image frame with the image feature of the image frame to obtain a fusion feature; A second decoding submodule, configured to decode the fused features based on a decoder of the first model to obtain predicted posture parameters, predicted shape parameters, and predicted camera parameters corresponding to the image frame; A second processing submodule is used to perform plane projection based on the predicted camera parameters, the predicted posture parameters and the predicted shape parameters to obtain the training key point information corresponding to the posture represented by the predicted posture parameters and the predicted shape parameters; The fourth training submodule is used to train the first model based on the training key point information and the key point information corresponding to the image frame to obtain the motion capture estimation model.

[0086] Optionally, the device further comprises: A third determination module is used to determine a target point of the three-dimensional virtual object corresponding to the target object based on the posture parameter and the shape parameter, wherein the target point is a rendering key point corresponding to a target special effect; A rendering module is used to render the target special effect based on the target point and the three-dimensional virtual object.

[0087] Optionally, the device further comprises: A fourth determination module, configured to determine, based on the posture parameter and the shape parameter, action information of a motion key point corresponding to the target object when performing a target action; A fifth determination module, configured to determine the action similarity based on the action information of the motion key point and the reference action information corresponding to the target action; An output module is used to output prompt information based on the action similarity.

[0088] Reference below Figure 5 , which shows a schematic diagram of the structure of an electronic device (such as a terminal device or a server) 600 suitable for implementing the embodiment of the present disclosure. The terminal device in the embodiment of the present disclosure may include but is not limited to mobile terminals such as mobile phones, notebook computers, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. Figure 5 The electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0089] like Figure 5 As shown, the electronic device 600 may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 601, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 602 or a program loaded from a storage device 608 into a random access memory (RAM) 603. In the RAM 603, various programs and data required for the operation of the electronic device 600 are also stored. The processing device 601, the ROM 602, and the RAM 603 are connected to each other via a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.

[0090] Typically, the following devices may be connected to the I / O interface 605: an input device 606 including, for example, a touch screen, a touch pad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; an output device 607 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 608 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 609. The communication device 609 may allow the electronic device 600 to communicate with other devices wirelessly or by wire to exchange data. Although Figure 5 The electronic device 600 is shown with various devices, but it should be understood that it is not required to implement or possess all the devices shown. More or fewer devices may be implemented or possessed instead.

[0091] In particular, according to an embodiment of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a non-transitory computer-readable medium, and the computer program contains program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication device 609, or installed from a storage device 608, or installed from a ROM 602. When the computer program is executed by the processing device 601, the above-mentioned functions defined in the method of the embodiment of the present disclosure are executed.

[0092] It should be noted that the computer-readable medium mentioned above in the present disclosure may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, device or device. In the present disclosure, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, which carries a computer-readable program code. This propagated data signal may take a variety of forms, including but not limited to an electromagnetic signal, an optical signal, or any suitable combination of the above. Computer readable signal media may also be any computer readable medium other than computer readable storage media, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer readable medium may be transmitted using any appropriate medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination of the above.

[0093] In some embodiments, the client and the server may communicate using any currently known or future developed network protocol such as HTTP (HyperText Transfer Protocol), and may be interconnected with any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network ("LAN"), a wide area network ("WAN"), an internet (e.g., the Internet), and a peer-to-peer network (e.g., an ad hoc peer-to-peer network), as well as any currently known or future developed network.

[0094] The computer-readable medium may be included in the electronic device, or may exist independently without being incorporated into the electronic device.

[0095] The above-mentioned computer-readable medium carries one or more programs. When the above-mentioned one or more programs are executed by the electronic device, the electronic device is enabled to: obtain a motion video corresponding to a target object; perform image feature extraction on image frames in the motion video to obtain image features of the image frames; perform key point extraction on the image frames to obtain target key point information corresponding to the target object in the image frames; and obtain three-dimensional motion parameters corresponding to the target object based on the image features, the target key point information and a pre-trained motion capture estimation model, wherein the three-dimensional motion parameters include posture parameters and shape parameters.

[0096] Computer program code for performing the operations of the present disclosure may be written in one or more programming languages ​​or a combination thereof, including, but not limited to, object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., through the Internet using an Internet service provider).

[0097] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some implementations as replacements, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0098] The modules involved in the embodiments described in the present disclosure may be implemented by software or hardware. The name of a module does not limit the module itself in some cases. For example, an acquisition module may also be described as a "module for acquiring a motion video corresponding to a target object."

[0099] The functions described above herein may be performed at least in part by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), and the like.

[0100] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or device, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0101] According to one or more embodiments of the present disclosure, Example 1 provides a motion capture method, the method comprising: Obtaining the motion video corresponding to the target object; Extracting image features from image frames in the motion video to obtain image features of the image frames; Extracting key points from the image frame to obtain target key point information corresponding to the target object in the image frame; Based on the image features, the target key point information and a pre-trained motion capture estimation model, three-dimensional motion parameters corresponding to the target object are obtained, and the three-dimensional motion parameters include posture parameters and shape parameters.

[0102] According to one or more embodiments of the present disclosure, Example 2 provides the method of Example 1, wherein the three-dimensional motion parameter further includes a camera parameter, and the method further includes: Performing plane projection based on the camera parameters, the posture parameters and the shape parameters to obtain prediction key point information corresponding to the predicted posture represented by the posture parameters and the shape parameters; Determining an adjustment amount of each parameter of the three-dimensional motion parameters based on the predicted key point information and the target key point information; The parameter is adjusted based on the adjustment amount of each parameter in the three-dimensional motion parameters to obtain updated three-dimensional motion parameters.

[0103] According to one or more embodiments of the present disclosure, Example 3 provides the method of Example 1, wherein the motion capture estimation model is determined in the following manner: Acquire first training samples, wherein each of the first training samples contains key point information corresponding to a plurality of image frames of a first training video; Training a preset model based on the first training sample to obtain a first model; Acquire second training samples, wherein each of the second training samples includes a plurality of image frames of a second training video and key point information corresponding to the image frames; The first model is trained based on the second training sample to obtain the motion capture estimation model.

[0104] According to one or more embodiments of the present disclosure, Example 4 provides the method of Example 3, wherein the preset model is implemented based on a long short-term memory model, and the step of training the preset model based on the first training sample to obtain the first model includes: Inputting key point information corresponding to the image frame in the first training video into the encoder of the preset model to obtain first training key point features corresponding to the image frame; Decoding the first training key point features based on the decoder of the preset model to obtain predicted posture parameters, predicted shape parameters and predicted camera parameters corresponding to the image frame; Performing plane projection based on the predicted camera parameters, the predicted posture parameters, and the predicted shape parameters to obtain training key point information corresponding to the posture represented by the predicted posture parameters and the predicted shape parameters; Based on the training key point information and the key point information corresponding to the image frame, the preset model is trained to obtain the first model.

[0105] According to one or more embodiments of the present disclosure, Example 5 provides the method of Example 3, wherein the step of training the first model based on the second training sample to obtain the motion capture estimation model includes: Inputting key point information corresponding to the image frame in the second training video into the encoder of the first model to obtain second training key point features corresponding to the image frame; Fusing the second training key point feature corresponding to the image frame with the image feature of the image frame to obtain a fused feature; Decoding the fused features based on the decoder of the first model to obtain predicted posture parameters, predicted shape parameters, and predicted camera parameters corresponding to the image frame; Performing plane projection based on the predicted camera parameters, the predicted posture parameters, and the predicted shape parameters to obtain training key point information corresponding to the posture represented by the predicted posture parameters and the predicted shape parameters; Based on the training key point information and the key point information corresponding to the image frame, the first model is trained to obtain the motion capture estimation model.

[0106] According to one or more embodiments of the present disclosure, Example 6 provides the method of Example 1, wherein the method further includes: Based on the posture parameter and the shape parameter, determining a target point of the three-dimensional virtual object corresponding to the target object, wherein the target point is a rendering key point corresponding to a target special effect; The target special effect is rendered based on the target point and the three-dimensional virtual object.

[0107] According to one or more embodiments of the present disclosure, Example 7 provides the method of Example 1, wherein the method further includes: Determining, based on the posture parameters and the shape parameters, action information of the motion key points corresponding to the target object when performing a target action; Determining action similarity based on the action information of the motion key point and the reference action information corresponding to the target action; Output prompt information based on the action similarity.

[0108] According to one or more embodiments of the present disclosure, Example 8 provides a motion capture device, the device comprising: An acquisition module, used to acquire the motion video corresponding to the target object; A first extraction module, used for extracting image features from image frames in the motion video to obtain image features of the image frames; A second extraction module is used to extract key points from the image frame to obtain target key point information corresponding to the target object in the image frame; The first processing module is used to obtain three-dimensional motion parameters corresponding to the target object based on the image features, the target key point information and a pre-trained motion capture estimation model, wherein the three-dimensional motion parameters include posture parameters and shape parameters.

[0109] According to one or more embodiments of the present disclosure, Example 9 provides a computer-readable medium having a computer program stored thereon, which implements the steps of any of the methods described in Examples 1-7 when executed by a processing device.

[0110] According to one or more embodiments of the present disclosure, Example 10 provides an electronic device, including: a storage device having a computer program stored thereon; A processing device is used to execute the computer program in the storage device to implement the steps of any one of the methods described in Examples 1-7.

[0111] According to one or more embodiments of the present disclosure, Example 11 provides a computer program product, including a computer program, which implements the steps of any one of the methods of Examples 1-7 when executed by a processor.

[0112] The above description is only a preferred embodiment of the present disclosure and an explanation of the technical principles used. Those skilled in the art should understand that the scope of disclosure involved in the present disclosure is not limited to the technical solutions formed by a specific combination of the above technical features, but should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the above disclosed concept. For example, the above features are replaced with the technical features with similar functions disclosed in the present disclosure (but not limited to) by each other to form a technical solution.

[0113] In addition, although each operation is described in a specific order, this should not be understood as requiring these operations to be performed in the specific order shown or in a sequential order. Under certain circumstances, multitasking and parallel processing may be advantageous. Similarly, although some specific implementation details are included in the above discussion, these should not be interpreted as limiting the scope of the present disclosure. Some features described in the context of a separate embodiment can also be implemented in a single embodiment in combination. On the contrary, the various features described in the context of a single embodiment can also be implemented in multiple embodiments individually or in any suitable sub-combination mode.

[0114] Although the subject matter has been described in language specific to structural features and / or method logic actions, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. On the contrary, the specific features and actions described above are merely example forms of implementing the claims. Regarding the device in the above embodiment, the specific manner in which each module performs the operation has been described in detail in the embodiment related to the method, and will not be elaborated here.

Claims

1. A motion capture method, characterized in that: The method comprises: Obtaining the motion video corresponding to the target object; Extracting image features from image frames in the motion video to obtain image features of the image frames; Extracting key points from the image frame to obtain target key point information corresponding to the target object in the image frame; Based on the image features, the target key point information and a pre-trained motion capture estimation model, three-dimensional motion parameters corresponding to the target object are obtained, and the three-dimensional motion parameters include posture parameters and shape parameters.

2. The method according to claim 1, characterized in that The three-dimensional motion parameters also include camera parameters, and the method further includes: Performing plane projection based on the camera parameters, the posture parameters and the shape parameters to obtain prediction key point information corresponding to the predicted posture represented by the posture parameters and the shape parameters; Determining an adjustment amount of each parameter of the three-dimensional motion parameters based on the predicted key point information and the target key point information; The parameter is adjusted based on the adjustment amount of each parameter in the three-dimensional motion parameters to obtain updated three-dimensional motion parameters.

3. The method according to claim 1, characterized in that The motion capture estimation model is determined by: Acquire first training samples, wherein each of the first training samples includes key point information corresponding to a plurality of image frames of a first training video; Training a preset model based on the first training sample to obtain a first model; Acquire second training samples, wherein each of the second training samples includes a plurality of image frames of a second training video and key point information corresponding to the image frames; The first model is trained based on the second training sample to obtain the motion capture estimation model.

4. The method according to claim 3, characterized in that: The preset model is implemented based on a long short-term memory model, and the preset model is trained based on the first training sample to obtain a first model, including: Inputting key point information corresponding to the image frame in the first training video into the encoder of the preset model to obtain first training key point features corresponding to the image frame; Decoding the first training key point features based on the decoder of the preset model to obtain predicted posture parameters, predicted shape parameters and predicted camera parameters corresponding to the image frame; Performing plane projection based on the predicted camera parameters, the predicted posture parameters, and the predicted shape parameters to obtain training key point information corresponding to the posture represented by the predicted posture parameters and the predicted shape parameters; Based on the training key point information and the key point information corresponding to the image frame, the preset model is trained to obtain the first model.

5. The method according to claim 3, characterized in that: The step of training the first model based on the second training sample to obtain the motion capture estimation model includes: Inputting key point information corresponding to the image frame in the second training video into the encoder of the first model to obtain second training key point features corresponding to the image frame; Fusing the second training key point feature corresponding to the image frame with the image feature of the image frame to obtain a fused feature; Decoding the fused features based on the decoder of the first model to obtain predicted posture parameters, predicted shape parameters, and predicted camera parameters corresponding to the image frame; Performing plane projection based on the predicted camera parameters, the predicted posture parameters, and the predicted shape parameters to obtain training key point information corresponding to the posture represented by the predicted posture parameters and the predicted shape parameters; Based on the training key point information and the key point information corresponding to the image frame, the first model is trained to obtain the motion capture estimation model.

6. The method according to claim 1, characterized in that The method further comprises: Based on the posture parameter and the shape parameter, determining a target point of the three-dimensional virtual object corresponding to the target object, wherein the target point is a rendering key point corresponding to a target special effect; The target special effect is rendered based on the target point and the three-dimensional virtual object.

7. The method according to claim 1, characterized in that The method further comprises: Determining, based on the posture parameters and the shape parameters, action information of the motion key points corresponding to the target object when performing a target action; Determining action similarity based on the action information of the motion key point and the reference action information corresponding to the target action; Output prompt information based on the action similarity.

8. A motion capture device, characterized in that: The device comprises: An acquisition module, used to acquire the motion video corresponding to the target object; A first extraction module, used for extracting image features from image frames in the motion video to obtain image features of the image frames; A second extraction module is used to extract key points from the image frame to obtain target key point information corresponding to the target object in the image frame; The first processing module is used to obtain three-dimensional motion parameters corresponding to the target object based on the image features, the target key point information and a pre-trained motion capture estimation model, wherein the three-dimensional motion parameters include posture parameters and shape parameters.

9. A computer readable medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processing device, the steps of the method according to any one of claims 1 to 7 are implemented.

10. An electronic device, characterized in that: include: a storage device having a computer program stored thereon; A processing device, configured to execute the computer program in the storage device to implement the steps of the method according to any one of claims 1 to 7.

11. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images