A method and system for processing a motion trajectory of a character in monocular video motion capture

Abstract

The application provides a motion trajectory processing method and system of a character in monocular video motion capture, and relates to the field of motion capture, and comprises the following steps: S1, collecting monocular video, acquiring dynamic point data of each joint of a character, and performing first jitter elimination processing on the dynamic point data to obtain processed dynamic joint point data; S2, constructing a character bone chain according to the processed dynamic joint point data to obtain three-dimensional bone chain dynamics; S3, segmenting the monocular video to obtain a video segment; S4, obtaining the motion trajectory of a root bone according to the video segment based on the processed dynamic joint point data and the three-dimensional bone chain dynamics, and correcting the motion trajectory of the root bone to obtain an initial motion trajectory of a foot root bone of the character in monocular video motion capture; and S5, performing second jitter elimination processing on the initial motion trajectory to obtain a final motion trajectory. The application determines the motion trajectory of a root bone of a character, corrects the motion trajectory to obtain an initial motion trajectory of a foot root bone, and uses inverse dynamics solving processing to obtain a final motion trajectory of the character, so that accurate capture and correction of the motion trajectory of the character are realized.

Description

Technical Field

[0001] This invention relates to the field of motion capture technology, and in particular to a method and system for processing the motion trajectory of a character in monocular video motion capture. Background Technology

[0002] In the current film, animation, and game production fields, the development of motion capture technology has become one of the key factors in producing high-quality animation and games. Optical motion capture technology, through the use of high-precision sensors and camera systems, can capture subtle changes in character movement, achieving highly accurate motion reproduction. However, optical motion capture equipment typically requires wearing a large number of sensors, has strict site requirements, and is expensive, thus limiting its application.

[0003] In contrast, inertial motion capture technology, which uses inertial sensors, requires no complex equipment and has lower site requirements, making it more convenient to use. However, inertial motion capture has relatively low accuracy and is susceptible to factors such as geomagnetic interference, which may limit its application in projects with high precision requirements.

[0004] On the other hand, video AI motion capture technology, as an emerging motion capture method, can automatically generate animation data by analyzing video footage and applying artificial intelligence algorithms. It is relatively low-cost and easy to use. However, current video AI motion capture technologies on the market have a significant drawback: a lack of accurate capture of the three-dimensional spatial motion information of the character's root skeleton. This can lead to problems such as slippage in the generated animation, affecting its quality.

[0005] Therefore, finding a video motion capture method that can effectively restore character animation and improve video motion capture quality, while also correcting problems such as sliding, is a technical problem that urgently needs to be solved by those skilled in the art. Summary of the Invention

[0006] In view of this, the present invention proposes a method and system for processing character motion trajectory in monocular video motion capture. It can determine the motion estimation of the character's root skeleton and make corrections to obtain the initial motion trajectory, and use inverse dynamics calculation to obtain the final motion trajectory of the character, thereby achieving accurate capture of the character's motion trajectory and correcting the slippage problem that occurs in video motion capture.

[0007] The technical solution of this invention is implemented as follows:

[0008] On one hand, the present invention provides a method for processing the motion trajectory of a character in monocular video motion capture, comprising the following steps:

[0009] S1. Acquire monocular video and obtain dynamic point data of each joint of the character in the monocular video. Perform a first anti-shake processing on the dynamic point data to obtain processed dynamic joint point data.

[0010] S2. Construct the character skeleton chain based on the processed dynamic joint data to obtain the three-dimensional dynamic skeleton chain.

[0011] S3. Segment the monocular video to obtain video segments;

[0012] S4. The processed dynamic joint data and three-dimensional skeleton chain dynamics obtain the motion trajectory of the root bone according to the video segment, and correct the motion trajectory of the root bone to obtain the initial motion trajectory of the root bone of the character's foot in monocular video motion capture.

[0013] S5. Perform a second anti-shaking process on the initial motion trajectory of the character's foot root skeleton in the monocular video motion capture to obtain the final motion trajectory of the character in the monocular video motion capture.

[0014] Based on the above technical solutions, preferably, the video segment includes a supporting time period M. i and the time period of jumping with both feet M j Step S4 specifically includes:

[0015] S41. Calculate M for each support time period. i The root bone matrix and root bone movement vector of the midfoot are used to obtain the movement trajectory of the root bones;

[0016] S42. Based on the processed dynamic joint point data, the time period M of the double-leg jump is... j The motion trajectory of the root bones is corrected to obtain the initial motion trajectory of the root bones of the character's feet in monocular video motion capture.

[0017] Based on the above technical solutions, preferably, step S42 specifically includes:

[0018] S421. Extract the processed dynamic joint point data and calculate the motion coordinates of the character in the xy plane in the monocular video;

[0019] S422. Based on the character's motion coordinates in the xy plane and the dynamic prediction of the three-dimensional skeletal chain, the coordinates of the root bone in the camera's depth direction are obtained, thus obtaining the three-dimensional spatial coordinates of the root bone.

[0020] S423. Calculate the distance of the character's joints in the monocular video 2D space and the projection distance in the 3D space facing the Z-axis, respectively, based on the three-dimensional spatial coordinates of the root skeleton. Obtain the spatial ratio based on the spatial distance D1 and the projection distance D2.

[0021] S424. Based on the three-dimensional spatial coordinates and spatial proportions of the root skeleton, obtain the character's motion trajectory in 3D space;

[0022] S425. Calculate the time interval M of a two-footed jump in 3D space.j The displacement difference vector of the root skeleton is obtained, and the motion trajectory of the character in 3D space is corrected to obtain the initial motion trajectory of the root skeleton of the character's feet in monocular video motion capture.

[0023] Based on the above technical solution, preferably, step S425 specifically includes:

[0024] Jumping with both feet for a period of time M j Align video frames within the frame and calculate the time interval M for a two-footed jump in 3D space. j The displacement difference vector V of the inner root skeleton is calculated using the following formula:

[0025] V = (X j *RX 3D j ,Y j *RY 3D j Z j *RZ 3D j ),

[0026] Among them, (X) 3D j ,Y 3D j Z 3D j ) represents the coordinates of the root bone in 3D space during the j-th segment of the double-foot jump, X 3D j This represents the X-axis coordinate of the root bone in 3D space during the j-th segment of the double-foot jump time interval, and the Y-axis coordinate represents the root bone's position. 3D j This represents the Y-axis coordinate of the root bone in 3D space during the j-th segment of the double-foot jump time interval, Z... 3D j R represents the Z-axis coordinate of the root bone in 3D space during the j-th segment of the double-foot jump time period, and R represents the spatial scale.

[0027] The corrected character motion trajectory is obtained based on the displacement difference vector V of the root skeleton in the 3D space and the character's motion trajectory in the 3D space.

[0028] Calculate the displacement difference vector of the root bone in the last frame of each video segment, and superimpose it into the displacement difference vector of the next video segment. Repeat this step until each video segment in the monocular video has been calculated, and obtain the initial motion trajectory of the root bone of the character's foot in the monocular video motion capture.

[0029] Based on the above technical solutions, preferably, step S41 specifically includes:

[0030] S411, Obtain support time period M iThe world coordinate matrix MR of the central root skeleton j and the world coordinate matrix MF of the feet j And calculate the offset matrix OFF of the root bones relative to the foot. j ;

[0031] S412, Supporting time period M i The displacement of the foot in the first frame is used as the support time period M i The fixed foot displacement when standing on the ground will support the time period M. i The foot's fixed-foot displacement is set into the displacement vector in the foot's world matrix, resulting in a new matrix MFP. j ;

[0032] S413, Calculate the support time period M i The trajectory of the root bones within;

[0033] S414. Repeat steps S411-S413 until the root bone is calculated for all support time periods M. i The movement trajectory within the body is obtained by analyzing the movement trajectory of the root bones.

[0034] Based on the above technical solutions, preferably, step S414 specifically includes:

[0035] Calculate the support time period M i Root skeleton matrix of the inner foot;

[0036] Calculate the support time period M i The root bone motion vector (VRM) in the last frame j ;

[0037] The root bone movement vector VRM j The root skeleton world matrix of all frames after the last frame in the i-th support time period is superimposed to obtain the support time period M. i The trajectory of the calcaneus when the foot touches the ground without slipping.

[0038] Furthermore, step S5 specifically includes:

[0039] S51. Perform Gaussian filtering on the initial motion trajectory of the root skeleton of the character's foot in the monocular video motion capture to obtain the first motion trajectory;

[0040] S52. Perform reverse dynamics calculation on the first motion trajectory to obtain the final motion trajectory of the character in monocular video motion capture.

[0041] Based on the above technical solutions, preferably, step S52 specifically includes:

[0042] S521. Create an inverse dynamics solver for the feet of the human skeleton;

[0043] S522. Calculate the support time period M based on the inverse dynamics solver. i The average position matrix of the foot;

[0044] S523, During the support period M i Video frames within the second threshold range are selected to calculate the fixed-foot weight value, and the support time period M is calculated based on the fixed-foot weight value. i The displacement of each frame of the video;

[0045] S524. Use the Slerp algorithm to calculate the rotation part of the smoothing process from the average position matrix to obtain the rotation quaternion.

[0046] S525. Convert the rotation quaternion into a matrix and merge it with the displacement of each video frame to obtain the current support time period M. i Foot position matrix;

[0047] S526. Repeat steps S522-S525 until all support time periods M are calculated. i The foot position matrix, based on all support time periods M i The position matrix of the feet yields the final motion trajectory of the character in monocular video motion capture.

[0048] Based on the above technical solution, preferably, step S52 further includes:

[0049] a. Supporting time period M i The average position matrix within the corresponding time period is set to support the time period M. i The world matrix of the foot's inverse dynamics controller is used to obtain the movement vector based on the foot's inverse dynamics controller's world matrix and the foot's initial position;

[0050] b. Based on the distance between the reverse dynamic controller and the femoral root bone, the length of the thigh and the length of the lower leg when the foot is fixed, the distance the root bone moves and the root bone movement vector are obtained.

[0051] c. Based on the world matrix position of the root bone and the root bone movement vector, obtain the offset of the waist movement.

[0052] On the other hand, the present invention provides a system for processing the motion trajectory of a character in monocular video motion capture, employing the processing method described above, including:

[0053] The video processing module is configured to acquire monocular video and perform a first anti-shake processing on the dynamic point data of each joint of the character to obtain the processed dynamic joint point data.

[0054] The character bone chain construction module is configured to construct the character bone chain based on the processed dynamic joint data, thereby obtaining a three-dimensional dynamic skeleton chain.

[0055] The video segmentation processing module is configured to segment the monocular video to obtain video segments, calculate the root bone matrix and root bone movement vector of the foot in each video segment, and obtain the motion trajectory of the root bone.

[0056] The character motion trajectory correction module is configured to extract the processed dynamic joint data, calculate the character motion coordinates and 3D skeletal chain dynamics, and correct the motion trajectory of the root bone to obtain the initial motion trajectory.

[0057] The character motion trajectory optimization module is configured to perform a second anti-jitter processing on the initial motion trajectory to obtain the first motion trajectory, and then perform inverse dynamics calculation processing to obtain the final motion trajectory.

[0058] The processing method of the present invention has the following advantages over the prior art:

[0059] (1) By performing the first anti-shaking processing on the dynamic point data of each joint, the character bone chain is constructed and the three-dimensional bone chain dynamics are obtained. The motion trajectory of the root bone of the character is determined, and the time period is corrected to obtain the initial motion trajectory of the root bone. The final motion trajectory of the character is obtained by using the second anti-shaking processing, so as to achieve accurate capture of the character's motion trajectory and correct the sliding problem that occurs in video motion capture.

[0060] (2) By segmenting the monocular video and combining video segmentation with root skeleton motion trajectory calculation, the motion characteristics of the character's feet are accurately captured. By combining 2D and 3D spatial information and calculating the spatial ratio and projection distance, the character's motion trajectory is finely corrected and controlled. The motion trajectory of the character in 3D space is effectively corrected, and the accuracy and completeness of motion capture data in monocular video are improved. (3) By aligning video frames and calculating displacement difference vectors, the motion trajectory of the character in 3D space is finely adjusted and corrected, which improves the accuracy and authenticity of the data. At the same time, by accumulating and processing root skeleton displacement difference vectors, the continuity and smoothness of the motion trajectory are ensured, making the final initial motion trajectory more coherent and natural. Attached Figure Description

[0061] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0062] Figure 1 This is a flowchart of the processing method of the present invention. Detailed Implementation

[0063] The technical solutions of the present invention will be clearly and completely described below with reference to the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present invention.

[0064] like Figure 1 As shown, this invention provides a method for processing the motion trajectory of a character in monocular video motion capture, comprising the following steps:

[0065] S1. Acquire monocular video and obtain dynamic point data of each joint of the character in the monocular video. Perform a first anti-jitter processing on the dynamic point data to obtain processed dynamic joint point data.

[0066] Specifically, step S1 will be described using a specific embodiment:

[0067] First, use a camera device to capture monocular video to ensure that the video quality and clarity are sufficient to obtain accurate human dynamic information;

[0068] Secondly, using human pose estimation solutions such as Mediapipe, the monocular video is processed to extract dynamic point data of each joint of the human body in the monocular video. This dynamic point data can include the two-dimensional coordinates (X, Y, F, Z) of each joint in the video plane space. i Y i ) and in three-dimensional space coordinates (X 3D i Y 3D i Z 3D i ), where i represents the i-th joint.

[0069] Furthermore, the acquired dynamic point data is smoothed by using a filtering algorithm to eliminate jitter and reduce data noise caused by camera shake or other factors.

[0070] Finally, the acquired dynamic point data is processed by jitter elimination to obtain processed dynamic joint point data. The processed dynamic joint point data can more accurately reflect the motion trajectory and posture changes of each joint of the human body in monocular video.

[0071] By tracking human movement in real time and extracting accurate dynamic joint point data, and using jitter elimination processing to reduce noise in the data, the accuracy and stability of the data are improved.

[0072] S2. Construct the character bone chain based on the processed dynamic joint data to obtain the three-dimensional dynamic skeleton chain.

[0073] Specifically, the character's skeletal chain is constructed using human anatomy and skeletal structure models based on processed dynamic joint data. By connecting the various joints to form the skeletal chain, the movement trajectory of the human skeletal structure can be simulated. The constructed character's skeletal chain is then simulated in three-dimensional space, and its position and posture are updated in real time based on the time-series changes in the dynamic joint data, thus obtaining the dynamic effect of the three-dimensional skeletal chain.

[0074] By constructing a character's skeletal chain and obtaining 3D skeletal chain dynamics, it is possible to more realistically simulate human movement postures and actions, providing high-quality character animation effects for fields such as virtual reality, game development, and animation production.

[0075] S3. The monocular video is segmented to obtain video segments.

[0076] Specifically, step S3 will be described using a specific embodiment:

[0077] By processing the monocular video frame by frame and extracting the information of the character's feet in contact with the ground in each frame, foot detection and tracking can be performed using computer vision technology or deep learning models.

[0078] Based on the monocular video information, identify the time periods when the character's left foot is in contact with the ground without sliding, the time periods when the character's right foot is in contact with the ground without sliding, and the time periods when the character's two feet are in contact with the ground and sliding. Record these time periods as the single-foot support time period and the double-foot support sliding time period.

[0079] Based on the recorded time periods when the left and right feet were in contact with the ground, find the time periods when neither foot touched the ground, i.e., the time periods when both feet jumped. For example, in an 85-frame video, the time periods when the left foot was in contact with the ground without sliding are 0-15, 30-47, 50-70, and 80-85, and the time periods when the right foot was in contact with the ground without sliding are 14-32, 45-51, 68-70, and 78-85. Therefore, the time period when both feet jumped is 71-78 frames.

[0080] S4. The processed dynamic joint data and three-dimensional skeleton chain dynamics obtain the motion trajectory of the root bone based on the video segment, and correct the motion trajectory of the root bone to obtain the initial motion trajectory of the root bone of the character's foot in monocular video motion capture.

[0081] Specifically, the video segment includes a supporting time period M. i and the time period of jumping with both feet M j Step S4 specifically includes:

[0082] S41. Calculate M for each support time period. i The root bone matrix and root bone movement vector of the midfoot are used to obtain the movement trajectory of the root bones;

[0083] S42. Based on the processed dynamic joint point data, the time period M of the double-leg jump is... j The motion trajectory of the root bones is corrected to obtain the initial motion trajectory of the root bones of the character's feet in monocular video motion capture.

[0084] This application utilizes a combination of video segmentation and root skeleton motion trajectory calculation to accurately capture the movement characteristics of a character's feet. By correcting the time interval of double-foot jumps, it can more accurately restore the character's movement state, improve the accuracy and completeness of motion capture data in monocular videos, and further optimize the root skeleton's motion trajectory by combining dynamic joint data, making the character's movements more natural and fluid. This provides a more reliable foundation for animation production and motion capture applications, enhancing visual effects and realism.

[0085] In one embodiment of this application, step S41 specifically includes:

[0086] S411, Obtain support time period M i The world coordinate matrix MR of the central root skeleton j and the world coordinate matrix MF of the feet j And calculate the offset matrix OFF of the root bones relative to the foot. j ;

[0087] Specifically, OFF j =MR j *MF j .inverse()

[0088] Where * represents matrix multiplication, and .inverse() represents matrix inverse.

[0089] S412, Supporting time period M i The displacement of the foot in the first frame is used as the support time period M i The fixed foot displacement when standing on the ground will support the time period M. i The foot's fixed-foot displacement is set into the displacement vector in the foot's world matrix, resulting in a new matrix MFP. j ;

[0090] S413, Calculate the support time period M i Trajectory of the calcaneus in the medial foot;

[0091] S414. Repeat steps S411-S413 until the root bone is calculated for all support time periods M. i The movement trajectory within the foot is obtained by observing the movement trajectory of the root bones.

[0092] By obtaining the support time period M i By calculating the world coordinate matrix of the root skeleton and foot and the offset matrix, accurate capture of character movement information is achieved. The fixed foot displacement on the ground is incorporated into the world matrix to establish a stable reference point, improving the accuracy of the movement trajectory. Calculating the root skeleton trajectory helps to deeply understand the character's movement patterns and action characteristics. By repeating the steps, the complete root skeleton movement trajectory is obtained, providing reliable data support for comprehensive monitoring and analysis of character movements, thereby achieving comprehensive control and analysis of the character's movement process.

[0093] Furthermore, step S414 specifically includes:

[0094] Calculate the support time period M i MRM (Meridian Root Matrix) of the Inner Foot j ;

[0095] Specifically, MRM j =OFF j *MFP j

[0096] Calculate the support time period M i The root bone motion vector VRMj in the last frame;

[0097] Specifically, VRM j =VRMD k -VRO k

[0098] Where k represents the support time period M i The last frame, VRMD k VRO represents the position of the root bone after movement in the k-th frame of the video. k This indicates the initial position of the root bone in the k-th frame of the video.

[0099] The root bone movement vector VRM j The root skeleton world matrix of all frames after the last frame in the i-th support time period is superimposed to obtain the support time period M. i The root bone trajectory.

[0100] By calculating the root bone matrix of the foot and the root bone motion vector of the last frame, and then superimposing the motion vector onto the root bone world matrix of subsequent frames, continuous tracking and trajectory reconstruction of the foot position during character movement are achieved. This method helps to accurately capture the position and posture information of the feet during character movement, providing reliable data support and making the character's movements more realistic and fluid.

[0101] Specifically, step S41 will be described using a specific embodiment:

[0102] For each support time period (e.g., 0-15, 30-47, 50-70, 80-85 for left foot support time periods), obtain the world coordinate matrix MR of the root bone and left foot in each frame. j and MF j And calculate the offset matrix of the root bone relative to the left foot;

[0103] The displacement of the left foot in the first frame of each support time period is taken as the fixed foot displacement of the left foot standing on the ground during that time period. This fixed foot displacement is applied to the last displacement vector (0, 0, 0, 1) in the left foot world matrix of that time period, replacing it with (tx, ty, tz, 1), resulting in a new matrix MFP. j ;

[0104] Within each support time period, the formula MRM is applied sequentially to frames 0, 1, 2, 3...15. j =OFF j *MFP j The matrix of the root bones is calculated when the left foot is fixed.

[0105] Calculate the root skeleton motion vector (VRM) of the last frame of each support time period. k =VRMD k -VRO k , where k is the last frame of the time period. The motion vector is superimposed on the root skeleton world matrix of all frames after this frame. For example, if the root skeleton motion vector VRM15 of frame 15 is calculated, then this vector needs to be superimposed on the root skeleton world matrix of frames 16, 17, 18...85 in sequence.

[0106] Furthermore, step S42 specifically includes:

[0107] S421. Extract the processed dynamic keypoint data and calculate the character's motion coordinates in the xy plane (X...). j Y j ); where j represents the processed dynamic joint point in the j-th double-foot jump time period;

[0108] S422. Based on the character's motion coordinates in the xy plane and the dynamic estimation of the 3D skeletal chain, the coordinates of the root bone in the camera's depth direction are obtained, thus obtaining the 3D spatial coordinates (X, Y, Z) of the root bone. j Y j Z j );

[0109] S423. Calculate the distance D1 of the character's joints in the monocular video 2D space and the projection distance D2 in the 3D space facing the Z-axis, based on the three-dimensional spatial coordinates of the root skeleton. Obtain the spatial ratio R based on the spatial distance D1 and the projection distance D2; where R = D2 / D1.

[0110] S424. Based on the three-dimensional spatial coordinates and spatial scale R of the root skeleton, obtain the character's motion trajectory in 3D space, i.e. (X j *R, Y j *R, Z j *R);

[0111] S425. Calculate the time interval M of the double-foot jump. j The displacement difference vector of the root skeleton in 3D space is used to correct the motion trajectory of the character in 3D space, thus obtaining the initial motion trajectory of the root skeleton of the character's feet in monocular video motion capture.

[0112] By combining 2D and 3D spatial information, the motion trajectory of the character in three-dimensional space is accurately calculated, improving the accuracy and completeness of motion capture data. Fine-grained correction and control of the character's motion trajectory are achieved through the calculation of spatial proportions and projection distance, enhancing the realism and lifelikeness of the character's movements. Through frame alignment and displacement difference vector calculation, the motion trajectory of the character in 3D space is effectively corrected, making the motion capture data more consistent with the actual motion state and providing a reliable foundation for subsequent animation production and motion capture.

[0113] Furthermore, step S425 specifically includes:

[0114] Jumping with both feet for a period of time M j Align video frames within the frame and calculate the two-footed jump time interval M. j The displacement difference vector V of the root skeleton in the intrinsic 3D space is calculated using the following formula:

[0115] V = (X j *RX 3D j ,Y j *RY 3D j Z j *RZ 3D j ),

[0116] Among them, (X)3D j ,Y 3D j Z 3D j ) represents the time period M of jumping with both feet. j The three-dimensional coordinates of the root skeleton in the internal 3D space, X 3D j This represents the X-axis coordinate of the root bone in 3D space during the j-th bipedal jump time interval, and the Y-axis coordinate represents the root bone's coordinate in 3D space. 3D j Z represents the Y-axis coordinate of the root bone in 3D space during the j-th bipedal jump time interval. 3D j R represents the Z-axis coordinate of the root bone in 3D space during the j-th bipedal jump time interval, and R represents the spatial scale.

[0117] The corrected character motion trajectory is obtained based on the displacement difference vector V of the root skeleton in the 3D space and the character's motion trajectory in the 3D space.

[0118] Calculate the displacement difference vector of the root bone in the last frame of each video segment, and superimpose it onto the displacement difference vector of the next video segment. Repeat this step until the time interval M of the double-foot jump is reached. j After each video segment is calculated, the initial motion trajectory of the character's foot heel bone in the monocular video motion capture is obtained.

[0119] By aligning video frames and calculating displacement difference vectors, fine-tuning and correcting the character's motion trajectory in 3D space were achieved, improving the accuracy and realism of the data. Simultaneously, by accumulating and processing root bone displacement difference vectors, the continuity and smoothness of the motion trajectory were ensured, resulting in a more coherent and natural initial motion trajectory. This provides a reliable foundation for subsequent animation production and motion capture, effectively improving the quality and accuracy of character motion data in monocular video motion capture, and providing crucial support for the realism and lifelikeness of character movements.

[0120] Specifically, step S42 will be described using a specific embodiment:

[0121] The motion coordinates of the root bone in the xy axis of the video plane within a time period of 70-78 are obtained by monocular video, and the coordinates of the root bone in the depth direction of the camera are estimated by video depth information to form a complete 3D coordinate system.

[0122] The spatial ratio R is calculated by comparing the ratio of a joint in a monocular video when it is facing the camera in 2D space (D1) to the projection distance D2 in 3D space along the Z-axis.

[0123] Multiply the root bone motion coordinates of time period 70-78 by the spatial scale R to obtain the corrected 3D spatial motion trajectory.

[0124] Align with the frame 70 preceding the first frame of the time period, and calculate the difference vector V of the 3D spatial root skeleton displacement between the 70th frame and the preceding frame.

[0125] Within the time period 71-78, the difference vector V is superimposed onto the motion trajectory in 3D space to obtain the corrected motion trajectory;

[0126] The difference vector of the last frame of the time period is calculated and superimposed on subsequent displacements to maintain motion continuity. This approach supplements and corrects missing root bone motion data, ensuring the consistency and accuracy of the character's movement trajectory.

[0127] S5. Perform a second anti-shaking process on the initial motion trajectory of the character's foot root skeleton in the monocular video motion capture to obtain the final motion trajectory of the character in the monocular video motion capture.

[0128] Specifically, step S5 includes:

[0129] S51. Perform Gaussian filtering on the initial motion trajectory of the root skeleton of the character's foot in the monocular video motion capture to obtain the first motion trajectory;

[0130] S52. Perform reverse dynamics calculation on the first motion trajectory to obtain the final motion trajectory of the character in monocular video motion capture.

[0131] By processing the initial motion trajectory with Gaussian filtering, the motion trajectory of the character's foot root bones becomes smoother and more stable, thereby reducing jitter and noise during the motion process; through inverse dynamics calculation, the actual motion trajectory of the character in monocular video motion capture can be more accurately restored, realizing the optimization and accurate restoration of the character's motion trajectory, and improving the accuracy and realism of character motion capture.

[0132] Furthermore, step S52 specifically includes:

[0133] S521. Create an inverse dynamics solver for the feet of the human skeleton;

[0134] S522. Calculate the support time period M based on the inverse dynamics solver. i The average position matrix of the foot;

[0135] S523, During the support period M i Video frames within the second threshold range are selected to calculate the fixed-foot weight value, and the support time period M is calculated based on the fixed-foot weight value. i The displacement of each frame of the video;

[0136] S524. Use the Slerp algorithm to calculate the rotation part of the smoothing process from the average position matrix to obtain the rotation quaternion.

[0137] S525. Convert the rotation quaternion into a matrix and merge it with the displacement of each video frame to obtain the current support time period M. i Foot position matrix;

[0138] S526. Repeat steps S522-S525 until all support time periods M are calculated. i The foot position matrix, based on all support time periods M i The position matrix of the feet yields the final motion trajectory of the character in monocular video motion capture.

[0139] By creating an inverse dynamics solver for the feet of the human skeleton, joint movements during character movement can be better simulated and controlled, resulting in more natural and fluid motion. Calculating the average position matrix of the feet based on the foot support time period effectively reduces trajectory deviations caused by friction and instability, improving the accuracy of the character's trajectory. Smoothing video frames within a second threshold range and calculating fixed-foot weights eliminates jitter and noise in the video, making the character's trajectory smoother and more continuous. The displacement of each video frame is calculated based on the fixed-foot weights, and the rotation component is calculated using the Slerp algorithm to obtain the position matrix of the feet during the current support time period, thus enabling precise control and updating of the character's trajectory.

[0140] Specifically, step S52 will be described using a specific embodiment:

[0141] Inverse dynamics solvers were created for the left and right feet of the human skeleton in the monocular video to enable precise control of the movement of each foot.

[0142] Based on the support time periods of the left and right feet, the foot position is locked; for example, the fixed foot time periods for the left foot are 0-15, 30-47, 50-70, 80-85. The average position matrix of the foot is calculated in each time period, and the world matrix of the IK controller is set to the average position matrix.

[0143] For the time range before and after the time period, smoothing is performed to prevent jumps; for example, for the time period 30-47, 5 frames are taken before and after, namely 25-30 and 47-52.

[0144] For example, in calculating displacement in range 25-30, the fixed foot weight value W... n The calculation formula for W is n = (i-25) / (30-25), where n is the current frame number, then the displacement T = TO n*(1-W n )+TR n *W n TO n TR represents the displacement of an undetermined foot. n This indicates the displacement of the fixed foot during the current time period;

[0145] The world matrix is ​​converted into a quaternion, and then the Slerp algorithm is used to calculate the transition values ​​of the rotated part; specifically, Q = Slerp(QO) n QR n W n ), where QO n QR is a rotation quaternion for undetermined feet. n The rotation quaternion for the current time period;

[0146] During the time period of 47-52, the fixed foot weight value W n The calculation formula for W is n = (47-i) / (52-47);

[0147] The calculated displacement and rotation results are combined, the quaternions are converted into matrices, and then combined with the previously calculated displacements to obtain the current matrix.

[0148] In one embodiment of this application, step S52 further includes:

[0149] a. Supporting time period M i The average position matrix within the time period is set as the world matrix of the inverse dynamics controller of the foot supporting the time period. The movement vector Vi is obtained based on the world matrix of the inverse dynamics controller of the foot and the initial position of the foot.

[0150] b. Based on the distance between the reverse dynamic controller and the femoral root bone, the length of the thigh and the length of the lower leg when the foot is fixed, obtain the distance Li of the root bone movement and the root bone movement vector.

[0151] c. Based on the world matrix position of the root bone and the root bone movement vector, obtain the offset of the waist movement.

[0152] By accurately calculating the motion parameters of the feet, groin bones, and waist, precise control and natural expression of the character's movement trajectory can be achieved, improving the realism and lifelikeness of character movements in monocular video motion capture. Simultaneously, by calculating the movement vector based on the world matrix of the inverse kinematics controller and the initial positions of body parts, the subtle details of the character's movements can be better captured, making the animations more vivid and fluid.

[0153] Specifically, when the distance between the inverse dynamics controller and the femoral root bone is greater than the sum of the lengths of the thigh and calf during foot positioning, a dynamic human posture algorithm is used to adjust the character's waist movement to prevent the feet from failing to touch the ground. A specific embodiment will be used to illustrate step S52:

[0154] The difference between the position TIKi of the foot after movement and the original position TFi of the foot is calculated to obtain the movement vector Vi = TIKi – TFi;

[0155] The distance to be moved is calculated as the distance between the inverse dynamics controller and the femoral root bone when the foot is fixed, minus the sum of the lengths of the thigh and lower leg, resulting in Li = LIKi - (LU + LD); where LU represents the length of the thigh and LD represents the length of the lower leg.

[0156] Normalize the movement vector Vi to a length of 1, then multiply it by the distance Li to be moved to obtain a new movement vector VMi = Vi.normalize()*Li; where .normalize() represents the normalization operation;

[0157] The obtained movement vector VMi is superimposed on the world matrix displacement part of the root bone to achieve movement adjustment of the root bone.

[0158] This application performs initial jitter elimination processing on the dynamic point data of each joint, constructs the character's skeletal chain and acquires the three-dimensional skeletal chain dynamics, accurately determines the motion trajectory of the character's foot root bones, and corrects the time interval M of the double-foot jump. j The initial motion trajectory is obtained by the movement trajectory of the central skeleton, and the final motion trajectory of the character is obtained by the second anti-shake processing, so as to achieve accurate capture of the character's motion trajectory, correct the slipping problem in video motion capture, improve the realism and smoothness of the character's movements, thereby enhancing the realism and natural expression of the character's movements in monocular video motion capture, and providing effective technical support for virtual character motion capture and performance.

[0159] This invention also provides a system for processing character motion trajectories in monocular video motion capture, employing the processing method described above, including:

[0160] The video processing module is configured to acquire monocular video and perform a first anti-shake processing on the dynamic point data of each joint of the character to obtain the processed dynamic joint point data.

[0161] The character bone chain construction module is configured to construct the character bone chain based on the processed dynamic joint data, thereby obtaining a three-dimensional dynamic skeleton chain.

[0162] The video segmentation processing module is configured to segment the monocular video to obtain video segments, calculate the root bone matrix and root bone movement vector of the foot in each video segment, and obtain the motion trajectory of the root bone.

[0163] The character motion trajectory correction module is configured to extract the processed dynamic joint data, calculate the character motion coordinates and 3D skeletal chain dynamics, and correct the motion trajectory of the root bone to obtain the initial motion trajectory.

[0164] The character motion trajectory optimization module is configured to perform a second anti-jitter processing on the initial motion trajectory to obtain the first motion trajectory, and then perform inverse dynamics calculation processing to obtain the final motion trajectory.

[0165] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for processing character motion trajectories in monocular video motion capture, characterized in that, Includes the following steps: S1. Acquire monocular video and obtain dynamic point data of each joint of the character in the monocular video. Perform a first anti-shake processing on the dynamic point data to obtain processed dynamic joint point data. S2. Construct the character skeleton chain based on the processed dynamic joint data to obtain the three-dimensional dynamic skeleton chain. S3. Segment the monocular video to obtain video segments; S4. The processed dynamic joint data and three-dimensional skeleton chain dynamics obtain the motion trajectory of the root bone according to the video segment, and correct the motion trajectory of the root bone to obtain the initial motion trajectory of the root bone of the character's foot in monocular video motion capture. S5. Perform a second anti-shaking process on the initial motion trajectory of the character's foot root bone in the monocular video motion capture to obtain the final motion trajectory of the character in the monocular video motion capture. The video segment includes a support time period Mi and a two-footed jump time period Mj. Step S4 specifically includes: S41. Calculate M for each support time period. i The root bone matrix and root bone movement vector of the midfoot are used to obtain the movement trajectory of the root bones; S42. Based on the processed dynamic joint point data, the time period M of the double-leg jump is... j The motion trajectory of the root bones is corrected to obtain the initial motion trajectory of the root bones of the character's feet in monocular video motion capture; Step S42 specifically includes: S421. Extract the processed dynamic joint point data and calculate the motion coordinates of the character in the xy plane in the monocular video; S422. Based on the character's motion coordinates in the xy plane and the dynamic prediction of the three-dimensional skeletal chain, the coordinates of the root bone in the camera's depth direction are obtained, thus obtaining the three-dimensional spatial coordinates of the root bone. S423. Calculate the distance of the character's joints in the monocular video 2D space and the projection distance in the 3D space facing the Z-axis, respectively, based on the three-dimensional spatial coordinates of the root skeleton. Obtain the spatial ratio based on the spatial distance and the projection distance. S424. Based on the three-dimensional spatial coordinates and spatial proportions of the root skeleton, obtain the character's motion trajectory in 3D space; S425. Calculate the time interval M of a two-footed jump in 3D space. j The displacement difference vector of the root skeleton is obtained, and the motion trajectory of the character in 3D space is corrected to obtain the initial motion trajectory of the root skeleton of the character's feet in monocular video motion capture. Step S425 specifically includes: Jumping with both feet for a period of time M j Align video frames within the frame and calculate the time interval M of a two-footed jump in 3D space. j The displacement difference vector V of the inner root skeleton is calculated using the following formula: V = (X j *R - X 3D j , Y j *R - Y 3D j , Z j *R - Z 3D j ), Among them, (X) 3D j , Y 3D j Z 3D j X represents the coordinates of the root bone in 3D space during the j-th segment of the double-foot jump time. 3D j This represents the X-axis coordinate of the root bone in 3D space during the j-th segment of the double-foot jump time interval, and the Y-axis coordinate represents the root bone's position. 3D j This represents the Y-axis coordinate of the root bone in 3D space during the j-th segment of the double-foot jump time interval, Z... 3D j R represents the Z-axis coordinate of the root bone in 3D space during the j-th segment of the double-footed jump time period, and R represents the spatial scale. The corrected character motion trajectory is obtained based on the displacement difference vector V of the root skeleton in the 3D space and the character's motion trajectory in the 3D space. Calculate the displacement difference vector of the root bone in the last frame of each video segment, and superimpose it into the displacement difference vector of the next video segment. Repeat this step until each video segment in the monocular video has been calculated, and obtain the initial motion trajectory of the root bone of the character's foot in the monocular video motion capture.

2. The method for processing character motion trajectories in monocular video motion capture as described in claim 1, characterized in that, Step S41 specifically includes: S411, Obtain support time period M i The world coordinate matrix MR of the central root skeleton j and the world coordinate matrix MF of the feet j And calculate the offset matrix OFF of the root bones relative to the foot. j ; S412, Supporting time period M i The displacement of the foot in the first frame is used as the support time period M i The fixed foot displacement when standing on the ground will support the time period M. i The foot's fixed-foot displacement is set into the displacement vector in the foot's world matrix, resulting in a new matrix MFP. j ; S413, Calculate the support time period M i The trajectory of the root bones within; S414. Repeat steps S411-S413 until the root bone is calculated for all support time periods M. i The movement trajectory within the body is obtained by analyzing the movement trajectory of the root bones.

3. The method for processing character motion trajectories in monocular video motion capture as described in claim 2, characterized in that, Step S414 specifically includes: Calculate the support time period M i Root skeleton matrix of the inner foot; Calculate the support time period M i The root bone motion vector (VRM) in the last frame j ; The root bone movement vector VRM j The root skeleton world matrix of all frames after the last frame in the i-th support time period is superimposed to obtain the support time period M. i The trajectory of the calcaneus when the foot touches the ground without slipping.

4. The method for processing character motion trajectories in monocular video motion capture as described in claim 1, characterized in that, Step S5 specifically includes: S51. Perform Gaussian filtering on the initial motion trajectory of the root skeleton of the character's foot in the monocular video motion capture to obtain the first motion trajectory; S52. Perform reverse dynamics calculation on the first motion trajectory to obtain the final motion trajectory of the character in monocular video motion capture.

5. The method for processing character motion trajectories in monocular video motion capture as described in claim 4, characterized in that, Step S52 specifically includes: S521. Create an inverse dynamics solver for the feet of the human skeleton; S522, Calculate the support time period M based on the inverse dynamics solver. i The average position matrix of the foot; S523, During the support period M i Video frames within the second threshold range are selected to calculate the fixed-foot weight value, and the support time period M is calculated based on the fixed-foot weight value. i The displacement of each frame of the video; S524. Use the Slerp algorithm to calculate the rotation part of the smoothing process from the average position matrix to obtain the rotation quaternion. S525. Convert the rotation quaternion into a matrix and merge it with the displacement of each video frame to obtain the current support time period M. i Foot position matrix; S526. Repeat steps S522-S525 until all support time periods M are calculated. i The foot position matrix, based on all support time periods M i The position matrix of the feet yields the final motion trajectory of the character in monocular video motion capture.

6. The method for processing character motion trajectories in monocular video motion capture as described in claim 4, characterized in that, Step S52 also includes: a. Supporting time period M i The average position matrix within the corresponding time period is set to support the time period M. i The world matrix of the foot's inverse dynamics controller is used to obtain the movement vector based on the foot's inverse dynamics controller's world matrix and the foot's initial position; b. Based on the distance between the reverse dynamic controller and the femoral root bone, the length of the thigh and the length of the lower leg when the foot is fixed, the distance the root bone moves and the root bone movement vector are obtained. c. Based on the world matrix position of the root bone and the root bone movement vector, obtain the offset of the waist movement.

7. A system for processing character motion trajectories in monocular video motion capture, characterized in that, The processing method described in any one of claims 1-6 includes: The video processing module is configured to acquire monocular video and perform a first anti-shake processing on the dynamic point data of each joint of the character to obtain the processed dynamic joint point data. The character bone chain construction module is configured to construct the character bone chain based on the processed dynamic joint data, thereby obtaining a three-dimensional dynamic skeleton chain. The video segmentation processing module is configured to segment the monocular video to obtain video segments, calculate the root bone matrix and root bone movement vector of the foot in each video segment, and obtain the motion trajectory of the root bone. The character motion trajectory correction module is configured to extract the processed dynamic joint data, calculate the character motion coordinates and 3D skeletal chain dynamics, and correct the motion trajectory of the root bone to obtain the initial motion trajectory. The character motion trajectory optimization module is configured to perform a second anti-jitter processing on the initial motion trajectory to obtain the first motion trajectory, and then perform inverse dynamics calculation processing to obtain the final motion trajectory.

Images