Method, device and equipment for driving digital human body movements based on video

Abstract

The present invention relates to the field of artificial intelligence technology, and in particular to a method, device and equipment for driving digital human body movements based on video. The method comprises: obtaining video stream data, decomposing the video stream data frame by frame into a plurality of picture data arranged in a preset order; inputting the plurality of picture data into a preset model in sequence in the preset order, and sequentially generating three-dimensional posture coordinates corresponding to each picture data; obtaining the standard skeleton coordinates of each picture data based on the three-dimensional posture coordinates of each picture data; obtaining the rotation quaternion of each joint of the digital human corresponding to each picture data based on the standard skeleton coordinates of each picture data; and driving the body movements of the digital human according to the rotation quaternion and the preset order. This solves the problems of many professional equipment, high cost and inaccurate skeleton capture in traditional body motion capture technology.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a method, device and equipment for driving digital human body movements based on video. Background Art

[0002] With the continuous development of science and technology, motion capture technology has received more and more attention. It collects human posture and motion information, uses appropriate data processing methods to accurately quantify and analyze these information, and finally obtains the desired motion information and corresponding subsequent processing. Motion capture has fundamental academic significance and is widely used in the fields of virtual and augmented reality, robotics, and sports biology. The three-dimensional motion capture system is a device that comprehensively records the movement of objects in three-dimensional space. According to different principles, it can be divided into mechanical motion capture, acoustic motion capture, electromagnetic motion capture, optical motion capture, and inertial motion capture.

[0003] However, optical motion capture devices are expensive and have certain limitations in accuracy, which cannot meet users' requirements for accuracy and are easily affected by external interference. Inertial motion requires a large amount of equipment, is costly and has high requirements for indoor deployment, and has certain limitations in capture accuracy. In certain motion situations, signal interference may affect signal acquisition, thereby affecting the accuracy of motion capture.

[0004] Therefore, there is an urgent need to solve the problems of high cost of professional equipment and inaccurate skeleton capture in traditional limb motion capture technology. Summary of the invention

[0005] In view of this, the present invention provides a method, device and equipment for driving digital human body movements based on video to solve the problems of multiple professional equipment, high cost and inaccurate skeleton capture in traditional body movement capture technology.

[0006] According to a first aspect of an embodiment of the present invention, a method for driving digital human body movements based on video includes:

[0007] Acquire video stream data, and decompose the video stream data frame by frame into a plurality of picture data arranged in a preset order;

[0008] The video stream data includes a series of body movements of people;

[0009] Input the plurality of image data into the preset model in sequence according to the preset order, and generate three-dimensional posture coordinates corresponding to each image data in sequence; wherein the three-dimensional posture coordinates include three-dimensional coordinates of a preset number of human body joints;

[0010] Based on the three-dimensional posture coordinates of each image data, the standard skeleton coordinates of each image data are obtained; wherein the standard skeleton coordinates include the standard coordinates of the joints of the digital human, and the joints of the digital human correspond one to one with the joints of the human body posture in the three-dimensional posture coordinates;

[0011] Based on the standard skeletal coordinates of each image data, the rotation quaternion of each joint of the digital human corresponding to each image data is obtained;

[0012] The body movements of the digital human are driven according to the rotation quaternion and the preset sequence.

[0013] Furthermore, the preset model includes:

[0014] A basic two-dimensional gesture recognition module, used for generating a two-dimensional gesture sequence based on the image data;

[0015] The dilated convolution module is used to establish a correlation between the space and time domains for the two-dimensional posture sequence to obtain the three-dimensional posture coordinates.

[0016] Furthermore, the step of sequentially inputting the plurality of image data into a preset model in the preset order and sequentially generating three-dimensional posture coordinates corresponding to each image data includes:

[0017] Obtain the plurality of image data, and sequentially generate a bias map and a heat map corresponding to each image data;

[0018] Based on the bias map and the heat map, three-dimensional posture coordinates of a preset number of human joints corresponding to each image data are generated.

[0019] Furthermore, it also includes:

[0020] Adaptive Kalman filtering and low-pass filtering are performed on the three-dimensional posture coordinates.

[0021] Furthermore, the step of obtaining the rotation quaternion of each joint of the digital human corresponding to each image data based on the standard skeleton coordinates of each image data includes:

[0022] Based on the first image data, generate the initial Y-axis orientation and the initial Z-axis orientation of each joint;

[0023] Based on the Y-axis orientation and the initial Z-axis orientation of each joint, generating an initial rotation matrix for each joint;

[0024] Based on the initial rotation matrices of the joints, generating initial alignment matrices of the joints;

[0025] Based on the newly input image data, generate the Y-axis orientation of the corresponding new output image data and the Z-axis orientation of each joint;

[0026] Generate a rotation matrix of each joint corresponding to the new input image data based on the Y-axis orientation of the new input image data and the Z-axis orientation of each joint;

[0027] The rotation matrix of the joint in the new input image data is multiplied by the initial alignment matrix to obtain the rotation quaternion of the new input image data.

[0028] Furthermore, the process of generating the Y-axis orientation and the Z-axis orientation of each joint includes:

[0029] Determine the parent-child relationship of each joint in the image data;

[0030] Determine the Z-axis orientation of the target joint based on the position difference between the target joint and its associated joints, wherein the associated joints of the target joint include joints that have parent-child joints with the target joint;

[0031] The Y-axis orientation of the image data is determined based on the normal direction of the plane formed by the spine and the left and right hip joints in the image data.

[0032] Further, the human body joint comprises:

[0033] Nose, left eye, right eye, left ear, right ear, left shoulder, right shoulder, left elbow, right elbow, left wrist, right wrist, left hip, right hip, left knee, right knee, left ankle and right ankle.

[0034] Furthermore, it also includes:

[0035] The standardization of digital human body movements is constrained by preset plug-ins.

[0036] According to a second aspect of an embodiment of the present invention, a device for driving digital human body movements based on video includes:

[0037] An acquisition module, used for acquiring video stream data, and decomposing the video stream data frame by frame into a plurality of picture data arranged in a preset order;

[0038] The video stream data includes a series of body movements of people;

[0039] A training module, used to input the plurality of image data into a preset model in sequence according to the preset order, and sequentially generate three-dimensional posture coordinates corresponding to each image data; wherein the three-dimensional posture coordinates include three-dimensional coordinates of a preset number of human body joints;

[0040] A generating module, configured to obtain standard skeletal coordinates of each image data based on the three-dimensional posture coordinates of each image data; wherein the standard skeletal coordinates include standard coordinates of joints of the digital human, and the joints of the digital human correspond one-to-one to the joints of the human body posture in the three-dimensional posture coordinates;

[0041] A rotation calculation module, used for obtaining the rotation quaternion of each joint of the digital human corresponding to each image data based on the standard bone coordinates of each image data;

[0042] A driving module is used to drive the limb movements of the digital human according to the rotation quaternion and the preset sequence.

[0043] According to a third aspect of an embodiment of the present invention, a smart device includes:

[0044] A processor, and a memory connected to the processor;

[0045] The memory is used to store a computer program, and the computer program is used to at least execute the method for driving digital human body movements based on video according to the first aspect of the embodiment of the present invention;

[0046] The processor is used to call and execute the computer program in the memory.

[0047] The present application provides a method for driving digital human body movements based on video, which decomposes a series of human body movements in video stream data frame by frame into multiple image data arranged in a preset order, and inputs the image data into a preset model in sequence in the preset order, generates three-dimensional posture coordinates corresponding to each image data, and obtains standard bone coordinates of each image data based on the three-dimensional posture coordinates of each image data. Because the joints of the digital human correspond to the joints of the human body posture in the three-dimensional posture coordinates one by one, the standard bone coordinates can be directly used as the standard coordinates of the joints of the digital human, thereby realizing the conversion of bone coordinates from the human in the video to the digital human. In addition, based on the standard bone coordinates of each image data, the rotation quaternion of each joint of the digital human corresponding to each image data is obtained, and the body movements of the digital human are driven according to the rotation quaternion and the preset order, so that the driving angle and direction of the body movements of the digital human are more closely matched to the body of the human in the video to make coordinated movements. Furthermore, this method can accurately infer the standard bone coordinates of the digital human without using a large number of professional equipment in the prior art, thereby reducing costs and driving the body movements of the digital human in a more coordinated and smoother manner.

[0048] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the invention and, together with the description, serve to explain the principles of the invention.

[0050] Figure 1is a flow chart showing a method for driving digital human body movements based on video according to an exemplary embodiment;

[0051] Figure 2 is a schematic diagram of a human joint according to an exemplary embodiment;

[0052] Figure 3 is a processing effect diagram of adaptive Kalman filtering and low-pass filtering smoothing according to an exemplary embodiment;

[0053] Figure 4 is a schematic block diagram of a device for driving digital human body movements based on video according to an exemplary embodiment;

[0054] Figure 5 is a schematic structural diagram of a smart device according to an exemplary embodiment;

[0055] Figure 6 It is a specific flow chart of a method for driving digital human body movements based on video according to another exemplary embodiment. DETAILED DESCRIPTION

[0056] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present invention. Instead, they are merely examples of devices and methods consistent with some aspects of the present invention as detailed in the appended claims.

[0057] Embodiment 1

[0058] refer to Figure 1 , Figure 1 is a flow chart of a method for driving digital human body movements based on video according to an exemplary embodiment. Figure 1 As shown, the method may specifically include the following steps:

[0059] Step S11: Obtain video stream data, and decompose the video stream data frame by frame into multiple picture data arranged in a preset order.

[0060] The video stream data includes a series of body movements of people.

[0061] Step S12: input the plurality of image data into the preset model in sequence according to the preset order, and generate the three-dimensional posture coordinates corresponding to each image data in sequence.

[0062] The three-dimensional posture coordinates include three-dimensional coordinates of a preset number of human body joints.

[0063] Step S13: obtaining the standard skeletal coordinates of each image data based on the three-dimensional posture coordinates of each image data; wherein the standard skeletal coordinates include the standard coordinates of the joints of the digital human, and the joints of the digital human correspond one-to-one to the joints of the human body posture in the three-dimensional posture coordinates.

[0064] Step S14: based on the standard skeletal coordinates of each image data, the rotation quaternion of each joint of the digital human corresponding to each image data is obtained.

[0065] Step S15: driving the body movements of the digital human according to the rotation quaternion and the preset sequence.

[0066] In step S11 "obtain video stream data, and decompose the video stream data frame by frame into multiple picture data arranged in a preset order", the video stream data is decomposed into picture data frame by frame. The preset order can be the time order in which the video plays the pictures frame by frame, that is, the multiple picture data are arranged in the order of video playback.

[0067] In step S12, "inputting the plurality of image data into the preset model in sequence according to the preset order, and generating three-dimensional posture coordinates corresponding to each image data in sequence; wherein the three-dimensional posture coordinates include three-dimensional coordinates of a preset number of human body joints":

[0068] Reference Figure 2 , Figure 2 is a schematic diagram of human body joints according to an exemplary embodiment, including 17 joints, or 17 key points of human body posture.

[0069] Preferably, the human joint comprises:

[0070] 0 nose, 1 left eye, 2 right eye, 3 left ear, 4 right ear, 5 left shoulder, 6 right shoulder, 7 left elbow, 8 right elbow, 9 left wrist, 10 right wrist, 11 left hip, 12 right hip, 13 left knee, 14 right knee, 15 left ankle and 16 right ankle.

[0071] In a specific application, the preset model is a three-dimensional posture estimation network model, and the three-dimensional posture estimation network model is an improvement based on a basic two-dimensional posture recognition module.

[0072] The basic two-dimensional posture recognition module may specifically be a VideoPose model.

[0073] In a specific embodiment, the preset model is an improvement based on the VideoPose model, and finally a three-dimensional pose estimation network model is obtained, and the three-dimensional pose estimation network model is the preset model.

[0074] Specifically, the input of the VideoPose model is a 2D image sequence, and the output is a 2D pose coordinate. The improved preset model adds a dilated convolution module and a bone orientation adjustment module to obtain 3D pose coordinates.

[0075] Specifically, the dilated convolution module is used to establish the spatial and temporal associations between two-dimensional image sequences, so as to better capture the dynamic information in the sequence. In the temporal domain, the dilated convolution can fuse the posture information between the previous and next frames, solve the problem of occlusion and reduce the jitter of action recognition. In the spatial domain, the dilated convolution can expand the receptive field and improve the ability to understand posture information.

[0076] The bone orientation adjustment module is used to fine-tune the relevant parameters of the convolution to make the three-dimensional posture coordinates and the two-dimensional posture coordinates more consistent. Specifically, this module projects the three-dimensional posture coordinates to generate two-dimensional posture coordinates and calculates the loss between the three-dimensional posture coordinates and the two-dimensional posture coordinates. If the bone position is inconsistent with the previous forward prediction, the module will perform reverse iterations and fine-tune the relevant parameters of the convolution to make the prediction results more accurate. In this way, the model can better learn the three-dimensional posture information and generate more accurate three-dimensional posture coordinates to obtain more accurate estimation results.

[0077] Furthermore, based on the above two improvements of the VideoPose model, the Human3.6M dataset (Human3.6M is one of the most commonly used datasets for 3D HPE tasks, containing 3.6 million frames of image data and corresponding 2D / 3D human poses) is used for training to obtain an improved preset model, namely the 3D pose estimation network model.

[0078] Preferably, the step of sequentially inputting the plurality of image data into a preset model in the preset order and sequentially generating the three-dimensional posture coordinates corresponding to each image data comprises:

[0079] Obtain the plurality of image data, and sequentially generate a bias map and a heat map corresponding to each image data;

[0080] Based on the bias map and the heat map, three-dimensional posture coordinates of a preset number of human joints corresponding to each image data are generated.

[0081] It can be understood that the most accurate three-dimensional coordinates of a preset number of human joints in each image data are found through the offset map and the heat map, and the three-dimensional coordinates are scaled according to a certain ratio, so that the image data is transformed into three-dimensional posture coordinates that are more in line with the laws of human posture.

[0082] Preferably, the method further includes: performing adaptive Kalman filtering and low-pass filtering smoothing processing on the three-dimensional posture coordinates.

[0083] Reference Figure 3 , Figure 3 This is the processing effect diagram of adaptive Kalman filtering and low-pass filtering smoothing, such as Figure 3 shown.

[0084] Specifically, on the first aspect, an adaptive Kalman filter algorithm is used for smoothing in order to prevent jitter.

[0085] The prediction formula of adaptive Kalman filter is:

[0086]

[0087]

[0088] \hat{x}_{k}^{-} represents the state estimate at time step k, based on the state \hat{x}_{k-1} of the previous time step k-1 and the control input u_{k-1}, predicted by the state transfer matrix A and the control input matrix B. P_{k}^{-} represents the state estimate covariance matrix at time step k, based on the state estimate covariance matrix P_{k-1} of the previous time step k-1, predicted by the state transfer matrix A and the process noise covariance matrix Q.

[0089] A represents the state transition matrix, which describes how the state evolves over time. B represents the control input matrix, which describes how the control input affects the evolution of the state. u_{k-1} represents the control input at time step k-1, and Q represents the process noise covariance matrix, which describes the uncertainty or noise in the state evolution.

[0090] The update formula is as follows:

[0091]

[0092]

[0093]

[0094] K_{k} represents the Kalman gain calculated at time step k based on the prediction covariance matrix P_{k}^{-}, the observation matrix H and the observation noise covariance matrix R. \hat{x}_{k} represents the state estimate obtained by updating the state through the observation matrix H and the observation z_{k} based on the predicted state estimate \hat{x}_{k}^{-} and the Kalman gain K_{k} at time step k. P_{k} represents the state estimate covariance matrix obtained by correcting the prediction covariance matrix P_{k}^{-} through the Kalman gain K_{k} and the observation matrix H at time step k. H represents the observation matrix, which describes the relationship between the state and the observation. R represents the observation noise covariance matrix, which describes the uncertainty or noise of the observation. z_{k} represents the observation obtained at time step k. I represents the identity matrix.

[0095] First, the five parameter values ​​of K, A, R, P and H are initialized respectively. Through parameter adjustment, it is found that the smaller the R value is, the closer the optimal estimate is to the observed data; the larger the Q value is, the closer the optimal estimate is to the observed data.

[0096] Secondly, a low-pass filtering algorithm is performed on the three-dimensional posture coordinates. The formula is as follows

[0097]

[0098] now represents the data point at the current moment, and prev represents the data point at the previous moment. smooth represents the smoothing coefficient, and its value range is [0, 1]. It represents the weight ratio of the data point at the current moment to the data point at the previous moment. The larger the smoothness, the greater the weight of the data point at the current moment, and the more obvious the smoothing effect; the smaller the smoothness, the greater the weight of the data point at the previous moment, and the weaker the smoothing effect.

[0099] Specifically, the low-pass filtering algorithm is a prior art, and the specific implementation method is not described here.

[0100] In step S13, "obtaining standard skeletal coordinates of each image data based on the three-dimensional posture coordinates of each image data; wherein the standard skeletal coordinates include standard coordinates of joints of the digital human, and the joints of the digital human correspond one-to-one to the joints of the human body posture in the three-dimensional posture coordinates";

[0101] In practical applications, it can be seen from the description of the above document that the three-dimensional posture coordinates are smoothed by adaptive Kalman filtering and low-pass filtering to obtain smooth three-dimensional posture coordinates. For example, when the obtained three-dimensional posture coordinates are the left elbow joint, the child joint relative to the left elbow joint is the left wrist joint. Here, the relationship between the left elbow joint and the left wrist joint is referred to as the parent-child relationship. If the distance between the left elbow joint and the left wrist joint is represented by D1, the distance D1 between the left elbow joint and the left wrist joint of the digital person is calculated according to the initial position coordinates of the left elbow joint and the left wrist joint in the first picture data. The distance D2 between the elbow joint and the wrist joint of the video character in the picture data is calculated according to the three-dimensional posture coordinates after smoothing of the left elbow joint and the left wrist joint, and the three-dimensional posture coordinates of the left elbow joint and the left wrist joint are subtracted, and then divided by D2 to obtain the direction angle. The distance D1 between the joints from the left elbow joint to the left wrist joint is multiplied by the direction angle, and then the initial position coordinates of the left elbow joint are added to obtain the standard skeletal coordinates of the left wrist joint.

[0102] It is understandable that the calculation of the standard bone coordinates of the left wrist joint is based on the relationship between the left elbow joint and the child node (left wrist joint) of the left elbow joint, and the standard bone coordinates of the elbow joint need to establish the relationship between the left shoulder joint and the child node (left elbow joint) of the left shoulder joint. By analogy, based on the three-dimensional posture coordinates of the joints of the human body posture in each picture, the standard coordinates of the joints of the digital human corresponding to each picture data, i.e., the standard bone coordinates, are calculated in turn.

[0103] Preferably, the step of obtaining the rotation quaternion of each joint of the digital human corresponding to each image data based on the standard skeletal coordinates of each image data comprises:

[0104] Based on the first picture data, generate the initial Y-axis orientation and each Z-axis orientation;

[0105] Based on the Y-axis orientation and each Z-axis orientation, generate an initial rotation matrix for each joint;

[0106] Based on the initial rotation matrices of the joints, generating initial alignment matrices of the joints;

[0107] Based on the newly input image data, generate the Y-axis orientation and each Z-axis orientation corresponding to the newly output image data;

[0108] Based on the Y-axis orientation and each Z-axis orientation of the new input image data, generate an initial rotation matrix for each joint corresponding to the new input image data;

[0109] The initial rotation matrix of the joint in the new input image data is multiplied by the initial alignment matrix of the same joint in the initial image data to obtain the rotation quaternion of the new input image data.

[0110] It is worth mentioning that because multiple image data are input into the preset model in a preset order to generate the rotation quaternion of each image, when the newly input image outputs the preset model, the posture of the digital human will be changed. Therefore, the rotation quaternion of the current image generated by the preset model realizes real-time driving of the digital human's limb movements.

[0111] In practical applications, the first image data is used as the initial posture image of the digital human, and the subsequent posture images of the new digital human are obtained according to the preset order for the newly input image data. Furthermore, each posture of the digital human is driven according to the rotation quaternion. How to generate the rotation quaternion? A specific example is given:

[0112] Step a: Calculate an initial rotation matrix (I) based on the Z-axis orientation and Y-axis orientation of the first image data;

[0113] That is, the Z-axis orientation and Y-axis orientation of the initial position of the digital human are calculated first: that is, the initial orientation of the digital human is calculated. For example, the Z-axis orientation of the upper arm is determined by the position difference between the upper arm and the forearm.

[0114] It is worth mentioning that the direction of the Y axis, that is, the direction of the human body, is determined by the normal direction (i.e., the vertical vector) of the plane formed by the spine and the left and right hip joints. The vertical vector is obtained by cross-producting the vector connecting the left hip joint and the spine with the vector connecting the right hip joint and the spine.

[0115] Here, the required joint coordinates used in the calculation of the Y-axis orientation of the upper arm are the three-dimensional posture coordinates. The Z-axis orientation of the upper arm is determined by the difference between the three-dimensional posture coordinates of the parent and child nodes.

[0116] Each frame of the character action in the video stream data is different, so the Z-axis direction and Y-axis direction of each picture data decomposed frame by frame of the video stream data are different.

[0117] Step b: Determine the Z-axis orientation and Y-axis orientation of the newly input image data as the Z-axis orientation and Y-axis orientation of the current image, and use the Z-axis orientation and Y-axis orientation of the current image to calculate a rotation matrix (R1) through the LookRotation method. The rotation matrix (R1) is inverted to obtain the inverse matrix (R2), and the inverse matrix (R2) is multiplied by the initial rotation matrix (I) of the joint to obtain the initial alignment matrix (A) of the joint.

[0118] For example, the inverse matrix (R2) is (-0.1, -0.6, -0.8, 0.1), corresponding to (x, y, z, w), where x, y, z are the imaginary parts, representing the coordinates of the rotation axis in three-dimensional space. w is the real part, representing the rotation angle;

[0119] The initial rotation matrix (I) is (0.3, 0.3, -0.5, -0.7);

[0120] A=R2*I=(R2.w*I.x+R2.x*I.w+R2.y*Iz-R2.z*Iy,

[0121] R2.w*I.y+R2.y*I.w+R2.z*Ix-R2.x*Iz,

[0122] R2.w*I.z+R2.z*I.w+R2.x*Iy-R2.y*Ix,

[0123] R2.w*Iw-R2.x*Ix-R2.y*Iy-R2.z*Iz)

[0124] =(-0.32,-0.14,-0.56,-0.66)

[0125] ->Assume that the initial alignment matrix of the upper arm is calculated to be A.

[0126] It is worth noting that the alignment matrix of the upper arm and the lower arm calculated here is only calculated once and is fixed, that is, the initial alignment matrix.

[0127] Step c: Get the Z-axis orientation and Y-axis orientation of the newly input image data, and calculate a rotation matrix (T1) through the LookRotation method.

[0128] Multiply this rotation matrix (T1) with the calculated initial alignment matrix (A), and the result B is the rotation quaternion that drives the digital human's limb movements in real time.

[0129] For example: T1=(0.0,0.6,0.8,0.0)A=(-0.32,-0.14,-0.56,-0.66)

[0130] The rotation quaternion of the current frame of the upper arm, that is, the current picture data, is obtained: B=T1*A=(-0.084,-0.064,0.35,-0.51)

[0131] Based on the calculation method of the rotation quaternion described in the above document, the rotation quaternion of each image data is calculated by analogy.

[0132] Preferably, the process of generating the Y-axis orientation and each Z-axis orientation includes:

[0133] Determine the parent-child relationship of each joint in the image data;

[0134] Determine the Z-axis orientation of the joint based on the position difference between the joint and its associated joints, wherein the associated joints of the joint include joints that have parent and child joints with the joint;

[0135] The Y-axis orientation of the image data is determined based on the normal direction of the plane formed by the spine and the left and right hip joints in the image data.

[0136] It is worth mentioning that the direction of the Y axis, that is, the direction of the human body, is determined by the normal direction (i.e., the vertical vector) of the plane formed by the spine and the left and right hip joints. The vertical vector is obtained by cross-producting the vector connecting the left hip joint and the spine with the vector connecting the right hip joint and the spine.

[0137] For example, in the video stream data, the Y-axis orientation of the upper arm of the character is calculated using the Spine and left and right hip joint coordinates, which are specifically the three-dimensional posture coordinates after joint smoothing. The Z-axis orientation of each upper arm is determined by the difference between the three-dimensional posture coordinates of the parent and child nodes.

[0138] Each frame of the character action in the video stream data is different, so the Z-axis direction and Y-axis direction of each picture data decomposed frame by frame of the video stream data are different.

[0139] Preferably, the method further includes: constraining the standardization of the digital human's limb movements through a preset plug-in to avoid distortion of limb joints.

[0140] Specifically, the FullBodyIK plug-in can be used to constrain the standardization of digital human model movements and avoid limb joint distortion. For example, the positions of 17 points can be used to control 12 effectors through a script.

[0141] In some embodiments, in the FullBodyIK plug-in, the literal meaning of the effector refers to a specific controller that can control certain key points of the digital character model. These controllers can be implemented through scripts to control the movement and posture of the digital character model. Specifically, these controllers include 12 key points that can be used to control the movement and posture of the hands, feet, head, torso and other parts of the digital character model. By constraining these key points, we can ensure the standardization of the movements of the digital character model and avoid the distortion of the limb joints. At the same time, these controllers can also be used to achieve some special effects, such as simulating jumping, climbing, crawling and other movements.

[0142] Understandably, effectors are a very important control mechanism that can help us achieve more natural and smooth movements of digital human models and improve the quality of games and animations.

[0143] In this embodiment, the 12 effectors include: hips, neck, left shoulder, left elbow, left wrist, right shoulder, right elbow, right wrist, left knee, left ankle, right knee and right ankle. The 12 effectors are then used to control the skeleton of the digital human and play the role of inverse dynamics constraints. The digital human will move in a coordinated manner according to the standard bone coordinates.

[0144] In some embodiments, each joint of the digital human is matched to the key point position output by the preset model, that is, the human body joint of the preset data joint, and the refresh frequency of each frame of the picture is defined, and the next frame of the dynamic capture data is read at each refresh. The rotation quaternion of each joint calculated in each frame drives the movement of the digital human's limbs.

[0145] It can be understood that by decomposing a series of human body movements in the video stream data frame by frame into a plurality of image data arranged in a preset order, the image data are sequentially input into the preset model in the preset order, and the three-dimensional posture coordinates corresponding to each image data are generated. The standard bone coordinates of each image data are obtained based on the three-dimensional posture coordinates of each image data. Because the joints of the digital human correspond one-to-one to the joints of the human body posture in the three-dimensional posture coordinates, the standard bone coordinates are the standard coordinates of the joints of the digital human, and the conversion of the bone coordinates from the human body in the video to the digital human is realized. The standard bone coordinates of the digital human converted by this method are more accurate. In addition, the rotation quaternion of each joint of the digital human corresponding to each image data is obtained based on the standard bone coordinates of each image data. The body movements of the digital human are driven according to the rotation quaternion and the preset order, so that the driving angle and direction of the body movements of the digital human are more closely matched to the body of the human body in the video to make coordinated movements. Finally, a large amount of professional equipment and high usage costs are avoided, and the standard bone coordinates of the digital human are more accurately inferred, and the body movements of the digital human are driven more coordinatedly and smoothly.

[0146] Reference Figure 4 , Figure 4 It is a schematic block diagram of a device for driving digital human body movements based on video, such as Figure 4 As shown, the device comprises:

[0147] The acquisition module 101 is used to acquire video stream data and decompose the video stream data frame by frame into a plurality of picture data arranged in a preset order; wherein the video stream data contains a series of human body movements;

[0148] The training module 102 is used to input the plurality of image data into the preset model in sequence according to the preset order, and generate three-dimensional posture coordinates corresponding to each image data in sequence; wherein the three-dimensional posture coordinates include three-dimensional coordinates of a preset number of human body joints;

[0149] A generating module 103, configured to obtain standard skeletal coordinates of each image data based on the three-dimensional posture coordinates of each image data; wherein the standard skeletal coordinates include standard coordinates of joints of a digital human, and the joints of the digital human correspond one to one with the joints of a human body posture in the three-dimensional posture coordinates;

[0150] A rotation calculation module 104, used for obtaining the rotation quaternion of each joint of the digital human corresponding to each image data based on the standard skeleton coordinates of each image data;

[0151] The driving module 105 is used to drive the limb movements of the digital human according to the rotation quaternion and the preset sequence.

[0152] Specifically, the specific implementation method of a device for driving digital human body movements based on video can refer to the specific implementation method of a method for driving digital human body movements based on video described in any of the above embodiments, which will not be repeated here.

[0153] It can be understood that by decomposing a series of human body movements in the video stream data frame by frame into a plurality of image data arranged in a preset order, the image data are sequentially input into the preset model in the preset order, and the three-dimensional posture coordinates corresponding to each image data are generated. The standard bone coordinates of each image data are obtained based on the three-dimensional posture coordinates of each image data. Since the joints of the digital human correspond one-to-one to the joints of the human body posture in the three-dimensional posture coordinates, the standard bone coordinates can be directly used as the standard coordinates of the joints of the digital human, thereby realizing the conversion of the bone coordinates from the human body in the video to the digital human. In addition, the rotation quaternion of each joint of the digital human corresponding to each image data is obtained based on the standard bone coordinates of each image data. The body movements of the digital human are driven according to the rotation quaternion and the preset order, thereby realizing the driving angle and direction of the body movements of the digital human to be more in line with the body of the human body in the video to make coordinated movements. Furthermore, this method can accurately infer the standard bone coordinates of the digital human without using a large number of professional equipment in the prior art, and drive the body movements of the digital human in a more coordinated and smoother manner.

[0154] The present invention also provides a smart device. Figure 5 , Figure 5 A schematic diagram of the structure of an intelligent device, such as Figure 5 As shown, the smart device includes:

[0155] A processor 202, and a memory 201 connected to the processor 202.

[0156] The memory 201 is used to store a computer program, and the computer program is used to at least execute the method for driving digital human body movements based on video according to any one of the embodiments of the present invention.

[0157] The processor 202 is used to call and execute the computer program in the memory 201 .

[0158] Specifically, a specific implementation method of a smart device can refer to the specific implementation method of a method for driving digital human body movements based on video described in any of the above embodiments, which will not be repeated here.

[0159] It can be understood that by decomposing a series of human body movements in the video stream data frame by frame into a plurality of image data arranged in a preset order, the image data are sequentially input into the preset model in the preset order, and the three-dimensional posture coordinates corresponding to each image data are generated. The standard bone coordinates of each image data are obtained based on the three-dimensional posture coordinates of each image data. Since the joints of the digital human correspond one-to-one to the joints of the human body posture in the three-dimensional posture coordinates, the standard bone coordinates can be directly used as the standard coordinates of the joints of the digital human, thereby realizing the conversion of the bone coordinates from the human body in the video to the digital human. In addition, the rotation quaternion of each joint of the digital human corresponding to each image data is obtained based on the standard bone coordinates of each image data. The body movements of the digital human are driven according to the rotation quaternion and the preset order, thereby realizing the driving angle and direction of the body movements of the digital human to be more in line with the body of the human body in the video to make coordinated movements. Furthermore, this method can accurately infer the standard bone coordinates of the digital human without using a large number of professional equipment in the prior art, and drive the body movements of the digital human in a more coordinated and smoother manner.

[0160] Embodiment 2

[0161] Reference Figure 6 , Figure 6 It is a specific flow chart of a method for driving digital human body movements based on video, such as Figure 6 As shown, the method includes:

[0162] Step S21: Obtain video stream data and decompose the video stream data frame by frame into multiple picture data.

[0163] Step S22: Input the image data of each frame into the preset model, and output the three-dimensional posture coordinates of 17 key points of the human body posture in real time.

[0164] Step S23: The three-dimensional posture coordinates are processed by adaptive Kalman filtering and low-pass filtering.

[0165] Step S24: Convert the smoothed three-dimensional posture coordinates into standard bone coordinates.

[0166] Step S25: Use the standard bone coordinates to calculate the Z-axis directions and Y-axis directions of the first image data respectively, convert the directions into rotation angles, generate an initial rotation matrix, and then calculate the initial alignment matrix of each joint through matrix transformation.

[0167] Step S26: Calculate the rotation matrix of each joint of each image of the subsequent multiple image data, and multiply the rotation matrix with the initial alignment matrix to calculate the rotation quaternion of each joint.

[0168] Step S27: driving the limb movements of the digital human according to the rotation quaternion of each joint in each image data.

[0169] It can be understood that by decomposing a series of human body movements in the video stream data frame by frame into a plurality of image data arranged in a preset order, the image data are sequentially input into the preset model in the preset order, and the three-dimensional posture coordinates corresponding to each image data are generated. The standard bone coordinates of each image data are obtained based on the three-dimensional posture coordinates of each image data. Since the joints of the digital human correspond one-to-one to the joints of the human body posture in the three-dimensional posture coordinates, the standard bone coordinates can be directly used as the standard coordinates of the joints of the digital human, thereby realizing the conversion of the bone coordinates from the human body in the video to the digital human. In addition, the rotation quaternion of each joint of the digital human corresponding to each image data is obtained based on the standard bone coordinates of each image data. The body movements of the digital human are driven according to the rotation quaternion and the preset order, thereby realizing the driving angle and direction of the body movements of the digital human to be more in line with the body of the human body in the video to make coordinated movements. Furthermore, this method can accurately infer the standard bone coordinates of the digital human without using a large number of professional equipment in the prior art, and drive the body movements of the digital human in a more coordinated and smoother manner.

[0170] It can be understood that the same or similar parts of the above embodiments can be referenced to each other, and the contents not described in detail in some embodiments can refer to the same or similar contents in other embodiments.

[0171] It should be noted that, in the description of the present invention, the terms "first", "second", etc. are only used for descriptive purposes and cannot be understood as indicating or implying relative importance. In addition, in the description of the present invention, unless otherwise specified, the meaning of "plurality" refers to at least two.

[0172] Any process or method description in a flowchart or otherwise described herein may be understood to represent a module, segment or portion of code that includes one or more executable instructions for implementing the steps of a specific logical function or process, and the scope of the preferred embodiments of the present invention includes alternative implementations in which functions may not be performed in the order shown or discussed, including performing functions in a substantially simultaneous manner or in the reverse order depending on the functions involved, which should be understood by those skilled in the art to which the embodiments of the present invention belong.

[0173] It should be understood that the various parts of the present invention can be implemented by hardware, software, firmware or a combination thereof. In the above-mentioned embodiments, multiple steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0174] A person skilled in the art may understand that all or part of the steps in the method for implementing the above-mentioned embodiment may be completed by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, which, when executed, includes one or a combination of the steps of the method embodiment.

[0175] In addition, each functional unit in each embodiment of the present invention may be integrated into a processing module, or each unit may exist physically separately, or two or more units may be integrated into one module. The above-mentioned integrated module may be implemented in the form of hardware or in the form of a software functional module. If the integrated module is implemented in the form of a software functional module and sold or used as an independent product, it may also be stored in a computer-readable storage medium.

[0176] The storage medium mentioned above can be a read-only memory, a magnetic disk or an optical disk, etc.

[0177] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" means that the specific features, results, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, results, materials or characteristics described can be combined in any one or more embodiments or examples in a suitable manner.

[0178] Although the embodiments of the present invention have been shown and described above, it is to be understood that the above embodiments are exemplary and are not to be construed as limitations of the present invention. A person skilled in the art may change, modify, replace and vary the above embodiments within the scope of the present invention.

Claims

1. A method for driving digital human body movements based on video, characterized in that: include: Acquire video stream data, and decompose the video stream data frame by frame into a plurality of picture data arranged in a preset order; The video stream data includes a series of body movements of people; Inputting the plurality of image data into the preset model in sequence according to the preset order, and generating three-dimensional posture coordinates corresponding to each image data in sequence; wherein the three-dimensional posture coordinates include three-dimensional coordinates of a preset number of human body joints; Based on the three-dimensional posture coordinates of each image data, the standard skeleton coordinates of each image data are obtained; wherein the standard skeleton coordinates include the standard coordinates of the joints of the digital human, and the joints of the digital human correspond one to one with the joints of the human body posture in the three-dimensional posture coordinates; Based on the standard skeletal coordinates of each image data, the rotation quaternion of each joint of the digital human corresponding to each image data is obtained; Driving the limb movements of the digital human according to the rotation quaternion and the preset sequence; The step of obtaining the rotation quaternion of each joint of the digital human corresponding to each image data based on the standard skeleton coordinates of each image data includes: Based on the first image data, generate the initial Y-axis orientation and the initial Z-axis orientation of each joint; Based on the Y-axis orientation and the initial Z-axis orientation of each joint, generating an initial rotation matrix for each joint; Based on the initial rotation matrices of the joints, generating initial alignment matrices of the joints; Based on the newly input image data, generate the Y-axis orientation of the corresponding new output image data and the Z-axis orientation of each joint; Generate a rotation matrix of each joint corresponding to the new input image data based on the Y-axis orientation of the new input image data and the Z-axis orientation of each joint; Multiplying the rotation matrix of the joint in the new input image data by the initial alignment matrix to obtain the rotation quaternion of the new input image data; The process of generating the Y-axis orientation and the Z-axis orientation of each joint includes: Determine the parent-child relationship of each joint in the image data; Determine the Z-axis orientation of the target joint based on the position difference between the target joint and its associated joints, wherein the associated joints of the target joint include joints that have parent-child joints with the target joint; The Y-axis orientation of the image data is determined based on the normal direction of the plane formed by the spine and the left and right hip joints in the image data.

2. The method according to claim 1, characterized in that The preset model includes: A basic two-dimensional gesture recognition module, used to generate two-dimensional gesture coordinates based on a two-dimensional image sequence of the picture data; A dilated convolution module, used to establish a correlation between the space and time domains for the two-dimensional image sequence to obtain the three-dimensional posture coordinates; The skeleton direction adjustment module is used to project the three-dimensional posture coordinates to generate the two-dimensional posture coordinates, and calculate the loss between the three-dimensional posture coordinates and the two-dimensional posture coordinates.

3. The method according to claim 1, characterized in that The step of sequentially inputting the plurality of image data into a preset model in the preset order, and sequentially generating three-dimensional posture coordinates corresponding to each image data comprises: Obtain the plurality of image data, and sequentially generate a bias map and a heat map corresponding to each image data; Based on the bias map and the heat map, three-dimensional posture coordinates of a preset number of human joints corresponding to each image data are generated.

4. The method according to claim 1, characterized in that: Also includes: Adaptive Kalman filtering and low-pass filtering are performed on the three-dimensional posture coordinates.

5. The method according to claim 1, characterized in that: The human body joints include: Nose, left eye, right eye, left ear, right ear, left shoulder, right shoulder, left elbow, right elbow, left wrist, right wrist, left hip, right hip, left knee, right knee, left ankle and right ankle.

6. The method according to claim 1, characterized in that Also includes: The standardization of digital human body movements is constrained by preset plug-ins.

7. A device for driving digital human body movements based on video, characterized in that: include: An acquisition module, used for acquiring video stream data, and decomposing the video stream data frame by frame into a plurality of picture data arranged in a preset order; The video stream data includes a series of body movements of people; A training module, used to input the plurality of image data into a preset model in sequence according to the preset order, and sequentially generate three-dimensional posture coordinates corresponding to each image data; wherein the three-dimensional posture coordinates include three-dimensional coordinates of a preset number of human body joints; A generating module, configured to obtain standard skeletal coordinates of each image data based on the three-dimensional posture coordinates of each image data; wherein the standard skeletal coordinates include standard coordinates of joints of the digital human, and the joints of the digital human correspond one-to-one to the joints of the human body posture in the three-dimensional posture coordinates; A rotation calculation module, used for obtaining the rotation quaternion of each joint of the digital human corresponding to each image data based on the standard bone coordinates of each image data; A driving module, used for driving the limb movements of the digital human according to the rotation quaternion and the preset sequence; The step of obtaining the rotation quaternion of each joint of the digital human corresponding to each image data based on the standard skeleton coordinates of each image data includes: Based on the first image data, generate the initial Y-axis orientation and the initial Z-axis orientation of each joint; Based on the Y-axis orientation and the initial Z-axis orientation of each joint, generating an initial rotation matrix for each joint; Based on the initial rotation matrices of the joints, generating initial alignment matrices of the joints; Based on the newly input image data, generate the Y-axis orientation of the corresponding new output image data and the Z-axis orientation of each joint; Generate a rotation matrix of each joint corresponding to the new input image data based on the Y-axis orientation of the new input image data and the Z-axis orientation of each joint; Multiplying the rotation matrix of the joint in the new input image data by the initial alignment matrix to obtain the rotation quaternion of the new input image data; The process of generating the Y-axis orientation and the Z-axis orientation of each joint includes: Determine the parent-child relationship of each joint in the image data; Determine the Z-axis orientation of the target joint based on the position difference between the target joint and its associated joints, wherein the associated joints of the target joint include joints that have parent-child joints with the target joint; The Y-axis orientation of the image data is determined based on the normal direction of the plane formed by the spine and the left and right hip joints in the image data.

8. A smart device, characterized in that: include: A processor, and a memory connected to the processor; The memory is used to store a computer program, and the computer program is used to at least execute the method for driving digital human limb movements based on video according to any one of claims 1 to 6; The processor is used to call and execute the computer program in the memory.

Images