Monocular dynamic lens motion capture method, device, electronic device and storage medium based on deep learning

Abstract

The present invention discloses a method, device, electronic device and storage medium for monocular dynamic lens motion capture based on deep learning. The method of the present invention comprises: reading video frames from an acquired video file or video stream, and obtaining the key point coordinates of a human body through neural network processing; restoring the rotation matrix of each key point of a person through inverse dynamics to obtain the posture of the person; converting the posture of the person into key point coordinates after the length of the limbs is unified through forward dynamics; and processing with a virtual universal treadmill algorithm to obtain the displacement of the root skeleton and complete motion capture. The present invention solves the problem of inaccurate root skeleton displacement prediction in the dynamic lens Ai video motion capture algorithm. The present invention uses high-precision 3D video motion capture with 3D coordinate prediction as the front end, and achieves the goal of completing stable root skeleton displacement prediction at a relatively low computing cost.

Description

Technical Field

[0001] The present invention relates to the field of lens motion capture, and in particular to a method, device, electronic device and storage medium for capturing monocular dynamic lens motion based on deep learning. Background Art

[0002] Using AI algorithms to solve motion capture problems is a popular technical direction nowadays. The reason is that traditional motion capture solutions often rely on expensive motion capture equipment, such as various optical motion capture equipment, which costs tens of thousands or even hundreds of thousands of yuan, which is unaffordable for individual developers and small studios in the animation field. Motion capture systems based on AI algorithms have an absolute advantage in cost.

[0003] In recent years, the more representative works in this direction include:

[0004] Human pose & shape estimation containing full frame location information (CLIFF: Carrying Location Information in Full Frames into Human Pose and Shape Estimation);

[0005] Reconstructing World-grounded Humans with Accurate 3D Motion (WHAM: Reconstructing World-grounded Humans with Accurate 3D Motion).

[0006] The innovation of CLIFF is to construct a rough but effective camera coordinate system, which allows the system to predict the movement of the root skeleton. However, under dynamic shots, the root skeleton displacement prediction of this scheme is basically ineffective.

[0007] WHAM integrates visual slam technology into the task of 3D pose prediction, attempting to achieve global motion prediction through accurate camera motion prediction. This solution can achieve root bone displacement prediction under dynamic shots, but once a dynamic shot scene with a solid color background is encountered, the prediction will fail. In addition, due to the need to integrate visual slam technology, the computational cost is relatively high.

[0008] It can be seen that the indicators that the industry generally pursues are stable prediction results and global motion prediction. However, the current AI motion capture solutions in the industry still cannot replace traditional solutions in high-precision motion capture scenarios. There are mainly the following reasons:

[0009] 1. The absolute accuracy is low, and it is difficult to achieve centimeter and sub-centimeter accuracy;

[0010] Second, the stability is not enough. The results predicted by the neural network are naturally uncertain, which leads to posture jitter;

[0011] Third, the root bone displacement prediction is inaccurate and has generalization issues.

[0012] The Chinese patent application with application number 202410628694.7 (publication number CN118397154A) discloses a method and system for generating Maya skeleton chain dynamic effects based on motion, including: S1, obtaining a controller chain and an animation sequence of the controller chain; S2, creating a dynamic skeleton chain through mapping according to the controller chain; S3, obtaining the world matrix of the controller chain, and processing to obtain the world matrix of the dynamic skeleton; S4, performing non-integer frame delay and simulating gravity on the dynamic skeleton chain according to the world matrix of the dynamic skeleton to obtain the dynamic matrix of the dynamic skeleton chain; S5, judging whether to loop, and aiming and calibrating the dynamic matrix of the dynamic skeleton chain to obtain the Maya skeleton chain dynamic effect; S6, copying the Maya skeleton chain dynamic effect to the controller chain to obtain a controller chain animation sequence with dynamic effects. This technical solution creates a dynamic skeleton chain through a controller chain, performs non-integer frame delay and simulates gravity operations based on the world matrix of the dynamic skeleton chain, ensures the continuity and naturalness of the skeleton chain, and improves the authenticity and fluency of the animation. However, this technical solution cannot be applied to motion capture in video data obtained by monocular dynamic lenses. Summary of the invention

[0013] The present invention solves the problem of inaccurate root skeleton displacement prediction in a dynamic lens Ai video motion capture algorithm. The present invention provides a monocular dynamic lens motion capture method, device, electronic device and storage medium based on deep learning.

[0014] The present invention uses dynamic lenses to perform motion capture, which is of great significance to animation content creators, because video materials of dynamic lenses are easier to obtain than those of fixed lenses.

[0015] In order to solve the problem of root bone displacement prediction under dynamic shots, the present invention adopts a "virtual treadmill" algorithm, which can restore the global motion trajectory of the root bone through the local motion trajectory of the two feet.

[0016] A method for capturing monocular dynamic lens motion based on deep learning, comprising the following steps:

[0017] S1: Read video frames from the video data obtained by the monocular dynamic lens, and obtain the original 3D coordinates of key parts of the human body after processing the video frames;

[0018] S2: The original 3D coordinates of each key part of the human body are restored through inverse dynamics to obtain the rotation matrix of each key part of the character and the character's posture;

[0019] S3: Convert the character's posture into key part coordinates with uniform limb lengths through forward dynamics;

[0020] S4: The coordinates of the key parts after unifying the limb length are processed using the virtual universal treadmill algorithm to obtain the displacement of the root bone and complete the motion capture;

[0021] The virtual universal treadmill algorithm includes:

[0022] S4.1: storing the key point coordinates of the footsteps in the key point coordinates after unifying the limb lengths according to the left foot trajectory and the right foot trajectory;

[0023] S4.2: extracting the motion period of the trajectory coordinate points of the left foot trajectory and the right foot trajectory;

[0024] S4.3: fitting an ellipse to points in the same motion cycle of each motion cycle extracted in step S4.2, constructing a parameter matrix equation of the ellipse and solving the ellipse parameters;

[0025] S4.4: Calculate the velocity vector of the two feet according to the motion cycle and the ellipse parameters corresponding to the motion cycle, and calculate the displacement of the root bone using the velocity vector of the two feet.

[0026] The idea of ​​the virtual treadmill algorithm is to use the trajectory of the local coordinate sequence of the two feet in space to simulate two moving treadmill tracks, and use the movement of the tracks to offset the root bone movement of the character, thereby indirectly calculating the movement of the root bone. There are two main solutions in the industry to deal with root bone movement: 1. Direct solution using neural networks (such as CLIFF), 2. Integrate visual slam into the motion capture system, and visual slam is responsible for the measurement of root bone displacement. The problem with solution 1 is that it is not robust enough, especially for dynamic shots. The problem with solution 2 is that its slam algorithm requires a lot of additional calculations, which is difficult to implement in products, and it fails for some scenes (such as solid color backgrounds). The virtual universal treadmill algorithm can avoid these problems well and is a lightweight and robust root bone prediction algorithm.

[0027] In step S1, the video data is a video file or a video stream;

[0028] The video frames are processed by a deep convolutional neural network, and the deep convolutional neural network is a deep neural convolutional network trained with 3D posture data.

[0029] In step S1, the key parts of the human body include feet and root bones.

[0030] In step S2, the inverse dynamics first calculates the rotation matrix through the coordinates of the key parts in the static posture and the moving posture, and then obtains the comprehensive rotation matrix by combining the axial rotation matrix output by the neural network.

[0031] Furthermore, the inverse dynamics adopts the following formula:

[0032] (1)

[0033] (2)

[0034] In formula (1), represents the coordinates of the kth key part of the human body of the character model in a static posture, Represents the coordinates of the parent node of the kth key part of the human body of the character model in a static posture;

[0035] represents the coordinates of the kth key part of the human body obtained under the motion posture, represents the coordinates of the parent node of the kth key part of the human body obtained under the motion posture, Represents the rotation matrix of the kth key part of the human body;

[0036] In formula (2), represents the axial rotation matrix obtained by the neural network processing in step (1), Represents the comprehensive rotation matrix of the kth key part of the human body.

[0037] In step S4.3, an ellipse is fitted to points in the same motion cycle of each motion cycle extracted in step S4.2, a parameter matrix equation of the ellipse is constructed and the ellipse parameters are solved, specifically including:

[0038] The trajectory points within the period are constructed into a coordinate matrix. Since the coordinate matrix multiplied by the parameter vector of the ellipse obtains a unit column vector, the parameter vector of the ellipse can be obtained by constructing an analytical solution.

[0039] Furthermore, solving the ellipse parameters specifically includes:

[0040] (3)

[0041] Formula (3) is the formula of a 3D ellipse in three-dimensional space, where the ellipse parameters α, β, γ, m, n, l, p, q, k are 9 independent parameters that need to fit the ellipse, X, Y, Z represent three coordinates in three-dimensional space, and the 9 independent parameters of the ellipse represent the information of the 3D ellipse in three-dimensional space. The V vector is constructed using the ellipse parameters, and the coordinate matrix Q is constructed using the spatial coordinate array of the two feet. U is the unit column vector, and formula (3) is converted to formula (4);

[0042] Q * V = U (4)

[0043] Solving formula (4), we get the expression of V vector as formula (5);

[0044] V = inverse(Q T * Q) * Q T * U (5)

[0045] In formula (5), inverse(Q T * Q) means Q T * The inverse matrix of Q, Q T represents the transpose of the coordinate matrix Q, and the ellipse parameters are obtained by solving the V vector using formula (5).

[0046] In step S4.4, the displacement of the root bone is calculated using the velocity vector of the two feet, specifically including:

[0047] When the velocity vector modulus of the left foot and the velocity vector modulus of the right foot are both greater than the threshold, the velocity of the root bone is the sum of the velocity vector of the left foot and the velocity vector of the right foot, otherwise the velocity of the root bone is 0.

[0048] Furthermore, the displacement of the root bone is calculated using the velocity vector of the two feet, including:

[0049] The calculated velocity vector of the left foot is , the velocity vector of the right foot is , for The vector modulus of for The vector modulus of and When both are greater than the threshold, the root bone speed is , otherwise the root bone's velocity is 0. The threshold range is 0.01m / s~0.05m / s.

[0050] A device for implementing the monocular dynamic lens motion capture method based on deep learning, comprising:

[0051] A video frame acquisition module is used to read video frames from video data acquired by a monocular dynamic lens;

[0052] The neural network module is used to process the video frames through a neural network to obtain the original 3D coordinates of key parts of the human body;

[0053] The inverse dynamics processing module is used to restore the rotation matrix of each key part of the human body through inverse dynamics from the original 3D coordinates of each key part of the human body to obtain the posture of the human body;

[0054] The forward dynamics processing module is used to convert the character's posture into the coordinates of key parts after unifying the limb length;

[0055] The virtual universal treadmill module is used to process the coordinates of key parts after unifying the limb length using the virtual universal treadmill algorithm to obtain the displacement of the root bone and complete motion capture.

[0056] An electronic device including a monocular dynamic lens, comprising:

[0057] Monocular dynamic lens;

[0058] one or more processors;

[0059] A memory for storing one or more programs;

[0060] When the one or more programs are executed by the one or more processors, the one or more processors implement the monocular dynamic lens motion capture method based on deep learning.

[0061] The electronic device can improve the quality of motion capture data from monocular dynamic footage.

[0062] A computer-readable storage medium stores a program, and when the program is executed by a processor, the monocular dynamic lens motion capture method based on deep learning is implemented.

[0063] The invention is of great significance to animation content creators because the video material of dynamic shots is easier to obtain than the fixed shot material. The invention uses 3D coordinate prediction as the front end of high-precision 3D video motion capture to achieve the goal of completing stable root bone displacement prediction at a low computational cost.

[0064] The solution of the present invention requires only a very small computational cost, can realize the measurement of the root bone displacement with considerable accuracy, and avoids the problem that the algorithm of the visual slam solution is prone to failure in a pure color background video. BRIEF DESCRIPTION OF THE DRAWINGS

[0065] Figure 1 The technical architecture diagram of the method of the present invention.

[0066] Figure 2 The figure is a flow chart of the algorithm of the method of the present invention.

[0067] Figure 3 Schematic diagram of the calculation results of the virtual treadmill algorithm.

[0068] Figure 4 Flowchart of the virtual treadmill algorithm.

[0069] Figure 5 Flowchart for separating motion cycles.

[0070] Figure 6 It is a schematic diagram of the structure of the device of the present invention.

[0071] Figure 7 This is a deployment architecture diagram of the present invention. DETAILED DESCRIPTION

[0072] like Figure 1 As shown, a method for capturing motion with a monocular dynamic lens based on deep learning comprises the following steps: S1: reading video frames from a video file or video stream obtained from a monocular dynamic lens, and processing the video frames with a deep convolutional neural network, which is a deep convolutional neural network trained with 3D posture data, to obtain the original 3D coordinates of each key part of the human body and the axial rotation matrix of each key part of the human body. The key parts of the human body include 24 key parts, namely: coccyx, left hip, right hip, lumbar spine, left knee, right knee, abdomen, left ankle, right ankle, thoracic spine, left foot, right foot, throat, left shoulder, right shoulder, jaw, left arm, right arm, left elbow, right elbow, left wrist, right wrist, left palm, and right palm. The method of the present invention focuses on the parts related to the feet and root bones.

[0073] S2: Performing IK (Inverse Kinematics) calculation on the original 3D coordinates of each key part of the human body and the axial rotation matrix of each key part of the human body to obtain the rotation matrix of each key part of the human body.

[0074] S3: Through FK (Foward Kinematics) calculation, the corrected 3D coordinates of key parts of the human body are obtained.

[0075] S4: Set the movement of the root bone to zero, take out the left and right foot coordinates in the corrected 3D coordinates of each key part of the human body in step S3, and store them in their respective coordinate queues to form the movement trajectories of the left and right feet.

[0076] S5: Perform ellipse fitting on the motion trajectories of the left foot and the right foot obtained in step S4, and calculate the motion of the root bone using a virtual treadmill algorithm.

[0077] The virtual treadmill algorithm includes:

[0078] S5.1: Separate the data of the two motion trajectories of the left foot and the right foot by performing motion cycle separation. The motion cycle separation method is to use two adjacent increases of the Y-axis component of the left foot / right foot coordinates as one motion cycle. Perform ellipse fitting using the coordinate sequence within the cycle as a unit.

[0079] S5.2: Construct the equation of a standard ellipse and solve the parameters of the equation of a standard ellipse.

[0080] S5.3: For the ellipses fitted for the left and right feet, calculate the velocity vectors of both feet according to their time period and ellipse parameters, and calculate the root bone motion vector using the velocity vectors of both feet.

[0081] The technical architecture of the monocular dynamic lens motion capture method based on deep learning of the present invention is as follows: Figure 1 As shown in the figure, after the input video image frames are processed by the deep convolutional neural network, the original 3D coordinates of each key part of the human body are obtained, and then the rotation matrix of each key part of the human body is restored based on IK (Inverse Kinematics). The rotation matrix of each key part is obtained through FK (Forward Kinematics) to obtain the corrected 3D coordinates of each key part. The corrected 3D coordinates of each key part are processed through the virtual treadmill algorithm to obtain the displacement of the root bone.

[0082] Figure 2 It is an algorithm flow chart. The deep convolutional neural network outputs two data: the original 3D coordinates of each key part of the human body and the axial rotation matrix of each key part of the human body. The two are calculated by IK to obtain the rotation matrix of each key part of the human body. The comprehensive rotation matrix of the kth key part is used for FK calculation to obtain the corrected 3D coordinates of each key part of the human body. The 3D coordinates of 24 key parts of the human body are calculated for each frame of the image.

[0083] Among the 3D coordinates of the 24 key parts corresponding to each frame image, the coordinates of the left foot and the right foot are taken out and stored in the left foot coordinate queue and the right foot coordinate queue respectively. The data of the left foot coordinate queue is the motion trajectory of the left foot, and the data of the right foot coordinate queue is the motion trajectory of the right foot.

[0084] The motion trajectories of the two key parts of the left foot / right foot are fitted into motion ellipses respectively. Then, the speed of the two motion ellipses of the left foot / right foot is combined to judge the motion speed of the root bone. The calculation results are as follows Figure 3 As shown, label 1 is the root bone, label 2 is the ellipse fitted to the right foot motion trajectory, and label 3 is the ellipse fitted to the left foot motion trajectory.

[0085] Inverse kinematics uses the following formula:

[0086] (6)

[0087] (7)

[0088] In formula (6) and formula (7), the subscript k represents the index of the key part. The present invention calculates 24 key parts of the human body for each frame image, and the value of k ranges from 0 to 23. The subscript pk represents the index of the parent node of the key part k. represents the coordinates of the kth key part in motion, Represents the coordinates of the parent node of the kth key part of the human body obtained under the motion posture. represents the coordinates of the kth key part in a static state, Represents the coordinates of the parent node of the kth key part of the human body of the character model in a static posture. Represents the rotation matrix of the kth key part. Represents the axial rotation matrix of k key parts, which is Figure 2 The axial rotation matrix of each key part of the human body, Represents the comprehensive rotation matrix of the kth key part.

[0089] Forward dynamics uses the following formula:

[0090] (8)

[0091] In formula (8), the subscript k represents the index of the key part, and the subscript pk represents the index of the parent node of the key part k. represents the coordinates of the kth key part in a static state, represents the coordinates of the kth key part in motion, represents the coordinates of the parent node of the kth key part in motion, Represents the comprehensive rotation matrix of the kth key part.

[0092] The detailed process of step S5.1 is as follows Figure 5 Shown:

[0093] Figure 5 In the figure, the left foot / right foot coordinate queue is the queue formed by the left foot / right foot coordinates calculated by FK. Linear filtering is performed on the left foot / right foot coordinate queue to smooth the Y-axis component of each coordinate element in the coordinate queue to obtain the filtered Y coordinate sub-queue, that is, the filtered Y coordinate sub-queue. The filtered Y coordinate sub-queue is numerically compared, and only the segments with two adjacent Y value increases need to be filtered out to obtain the separated motion cycle.

[0094] The detailed process of step S5.2 is as follows Figure 4 Shown:

[0095] Figure 4 In the example, numbers 11 and 12 correspond to Figure 5 The motion cycle after separation is expressed as the result of the separation motion cycle. The parameters of the ellipse can be obtained by fitting the data after the separation motion cycle. In mathematical calculation, the coordinate data in the same cycle needs to be constructed into a coordinate matrix Q, and the parameters of the ellipse need to be constructed into an unknown vector V to solve the equation:

[0096] Q * V = U (9)

[0097] In formula (9), U represents a column vector with a value of 1.

[0098] After obtaining the ellipse parameters of the left and right feet, combined with the time information of the ellipse's motion cycle, the movement speed of the left and right feet can be obtained, and the movement of the root bone can be obtained by combining the movement information of the left and right feet.

[0099] Figure 4 This is the calculation process of the virtual treadmill algorithm. First, based on Figure 2 The result of correcting the 3D coordinates of the key parts of the human body can get the movement trajectory of the right foot ( Figure 4 As shown in the middle label 9), the movement trajectory of the left foot ( Figure 4 The queue constituting the left foot motion trajectory data is:

[0100] [(x lt1 , y lt1 , z lt1 ), (x lt2 , y lt2 , z lt2 ), (x lt3 , y lt3 , z lt3 ) …]

[0101] where x lt1 represents the x-coordinate of the left foot at time t1, y lt1 represents the y coordinate of the left foot at time t1, z lt1 represents the z coordinate of the left foot at time t1. lt2 , y lt2 , z lt2 These are the three coordinate values ​​of the left foot at time t2, and so on.

[0102] The queue that constitutes the right foot motion trajectory data is:

[0103] [(x rt1 , yrt1 , z rt1 ), (x rt2 , y rt2 , z rt2 ), (x rt3 , y rt3 , z rt3 ) …]

[0104] where x rt1 represents the x-coordinate of the right foot at time t1, y rt1 represents the y coordinate of the right foot at time t1, z rt1 represents the z coordinate of the right foot at time t1. rt2 , y rt2 , z rt2 These are the three coordinate values ​​of the right foot at time t2, and so on.

[0105] Next, the motion trajectories of the two key parts of the left foot / right foot are separated into motion cycles. The result of the separation is to obtain a subset of the motion trajectories of the two key parts of the left foot / right foot, that is, a part of the data of each of the trajectory data queues of the two key parts of the left foot / right foot. Figure 4 The number 11 is the movement cycle data of the right foot. Figure 4 The number 12 is the motion cycle data of the left foot. The motion cycle data is generally irregular in shape and needs to be fitted into the closest ellipse.

[0106] In three-dimensional space, the formula for a 3D ellipse is:

[0107] (10)

[0108] In formula (10), α, β, γ, m, n, l, p, q, k are the nine ellipse parameters that need to be fitted, and X, Y, Z represent the three coordinates in the three-dimensional space.

[0109] First, the motion cycle data of the foot is used to construct the coordinate matrix Q. The coordinate matrix Q is a matrix with num rows and 9 columns. The value of num is equal to the number of elements contained in the motion cycle data. The elements of the i-th row of the coordinate matrix Q are shown in formula (11):

[0110] (11)

[0111] In formula (11), xi, yi, zi represent the x, y, z coordinates of the ith point. Rewrite formula 9:

[0112] Q * V = U (9)

[0113] Where V represents the column vector [α, β, γ, m, n, l, p, q, k] consisting of the 9 ellipse parameters that need to be fitted, and U represents the unit column vector with element 1.

[0114] Integrate formula (9) by multiplying both sides by inverse(Q T *Q)*Q T , and we get formula (12).

[0115] inverse(Q T *Q)*Q T *Q* V = inverse(Q T *Q)*Q T *U (12)

[0116] where inverse(Q T * Q) means Q T * The inverse matrix of Q. The analytical solution of the ellipse parameter vector V is shown in formula (13):

[0117] V = inverse(Q T * Q) * Q T * U (13)

[0118] After calculating the ellipse parameters, we can get an ellipse that approximates the motion trajectory of the left foot / right foot, such as Figure 4 As shown in the figure 13 and the figure 14. Since the two ellipses 13 and 14 already contain the time of the motion cycle, the motion speed of the two ellipses can be obtained, that is, the speed of the left foot / right foot.

[0119] Assume that the calculated velocity vector of the left foot is , the velocity vector of the right foot is , for The vector modulus of for The vector modulus of and When both are greater than the threshold of 0.01m / s, the speed of the root bone is , otherwise the root bone's velocity is 0.

[0120] like Figure 6 As shown, the device for implementing the monocular dynamic lens motion capture method based on deep learning includes:

[0121] A video frame acquisition module is used to read video frames from video data acquired by a monocular dynamic lens;

[0122] The neural network module is used to process the video frames through the neural network to obtain the original 3D coordinates of key parts of the human body;

[0123] The inverse dynamics processing module is used to restore the rotation matrix of each key part of the human body through inverse dynamics from the original 3D coordinates of each key part of the human body to obtain the posture of the human body;

[0124] The forward dynamics processing module is used to convert the character's posture into the coordinates of key parts after unifying the limb length;

[0125] The virtual universal treadmill module is used to process the coordinates of key parts after unifying the limb length using the virtual universal treadmill algorithm to obtain the displacement of the root bone and complete motion capture.

[0126] Figure 7 This is the deployment architecture diagram of the present invention. The algorithm is deployed in the cloud. The terminal user uploads the video file to the algorithm server. After the algorithm server analyzes it, it will output the corresponding fbx model file for the terminal user to download.

[0127] The biggest advantage of the present invention is that it greatly improves the motion capture quality of dynamic monocular lens video data. Two implementation and application methods are listed as follows: 1. Ordinary users can complete the collection of motion capture video data using portable electronic devices such as mobile phones. Traditional methods generally require multi-camera shooting, and have high requirements for shooting locations and shooting equipment. The system of this invention can be deployed on a personal mobile phone, and the video captured by the mobile phone can be directly called on the mobile phone to perform motion capture. 2. Under the premise of copyright permission, the present invention can use most of the public video materials on the Internet for motion capture, while traditional video motion capture solutions have specific requirements for video materials. The biggest problem is that it cannot adapt well to dynamic lens data, but most of the public video data on the Internet happens to be shot with dynamic lenses. This invention can be deployed on a personal computer to process motion capture of public video data on the Internet.

Claims

1. A method for capturing monocular dynamic lens motion based on deep learning, characterized in that: The following steps are involved: S1: Read video frames from the video data obtained by the monocular dynamic lens, and obtain the original 3D coordinates of key parts of the human body after processing the video frames; S2: The original 3D coordinates of each key part of the human body are restored through inverse dynamics to obtain the rotation matrix of each key part of the character and the character's posture; S3: Convert the character's posture into key part coordinates with uniform limb lengths through forward dynamics; S4: The coordinates of the key parts after unifying the limb length are processed using the virtual universal treadmill algorithm to obtain the displacement of the root bone and complete the motion capture; The virtual universal treadmill algorithm includes: S4.1: storing the key point coordinates of the footsteps in the key point coordinates after unifying the limb lengths according to the left foot trajectory and the right foot trajectory; S4.2: extracting the motion period of the trajectory coordinate points of the left foot trajectory and the right foot trajectory; S4.3: fitting an ellipse to points in the same motion cycle of each motion cycle extracted in step S4.2, constructing a parameter matrix equation of the ellipse and solving the ellipse parameters; S4.4: Calculate the velocity vector of the two feet according to the motion cycle and the ellipse parameters corresponding to the motion cycle, and calculate the displacement of the root bone using the velocity vector of the two feet.

2. The method for capturing monocular dynamic lens motion based on deep learning according to claim 1, characterized in that: In step S1, the video data is a video file or a video stream.

3. The method for capturing monocular dynamic lens motion based on deep learning according to claim 1, characterized in that: In step S1, the video frames are processed by a deep convolutional neural network.

4. The method for capturing monocular dynamic lens motion based on deep learning according to claim 3, characterized in that: In step S1, the deep convolutional neural network is a deep neural convolutional network trained with 3D posture data.

5. The method for capturing monocular dynamic lens motion based on deep learning according to claim 1, characterized in that: In step S1, the key parts of the human body include feet and root bones.

6. The method for capturing monocular dynamic lens motion based on deep learning according to claim 1, characterized in that: In step S4.4, the displacement of the root bone is calculated using the velocity vector of the two feet, specifically including: When the velocity vector modulus of the left foot and the velocity vector modulus of the right foot are both greater than the threshold, the velocity of the root bone is the sum of the velocity vector of the left foot and the velocity vector of the right foot, otherwise the velocity of the root bone is 0.

7. The method for capturing monocular dynamic lens motion based on deep learning according to claim 6, characterized in that: In step S4.4, the threshold value ranges from 0.01 m / s to 0.05 m / s.

8. A device for implementing the method for capturing monocular dynamic lens motion based on deep learning according to any one of claims 1 to 7, characterized in that: include: A video frame acquisition module is used to read video frames from video data acquired by a monocular dynamic lens; A neural network module is used to process video frames to obtain the original 3D coordinates of key parts of the human body; The inverse dynamics processing module is used to restore the rotation matrix of each key part of the human body through inverse dynamics from the original 3D coordinates of each key part of the human body to obtain the posture of the human body; The forward dynamics processing module is used to convert the character's posture into the coordinates of key parts after unifying the limb length; The virtual universal treadmill module is used to process the coordinates of key parts after unifying the limb length using the virtual universal treadmill algorithm to obtain the displacement of the root bone.

9. An electronic device including a monocular dynamic lens, characterized in that: include: Monocular dynamic lens; one or more processors; A memory for storing one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the monocular dynamic lens motion capture method based on deep learning as described in any one of claims 1-7.

10. A computer-readable storage medium having a program stored thereon, characterized in that: When the program is executed by a processor, the monocular dynamic lens motion capture method based on deep learning as described in any one of claims 1 to 7 is implemented.

Images