Motion capture system and method

Abstract

The invention discloses a motion capture system and method, and the method comprises the steps: firstly, obtaining an animation scene image containing human body motions, setting mark points at key parts of a human body, and carrying out the framing processing of an animation sequence; thirdly, pixel motion between adjacent frames is calculated through the optical flow field technology, starting and ending frames of the human body motion are accurately recognized, and pixel coordinates of the mark points in the motion frames are tracked and recorded; on the basis of the coordinate information, motion tracks of the mark points are constructed, and related actions are associated and combined, so that a data basis is provided for follow-up action analysis; and finally, solving the 3D coordinates and the spatial attitude angle of the human body in the coordinate system by applying the forward kinematics principle, calculating the accurate 3D position of the human body in the animation scene by combining the information, and adjusting the result into the coordinate system used by animation software through coordinate system conversion, thereby realizing the accurate restoration and positioning of the human body action in the animation.

Description

Technical Field

[0001] The present invention relates to the technical field of motion capture, and in particular, to a motion capture system and method. Background Art

[0002] Motion capture is a technology used to record and process the motions of humans or other objects. Through specific hardware devices, the motion trajectories of moving objects in three-dimensional space are captured, and these motion data are converted into three-dimensional models that can be understood by a computer, and then applied to multiple fields such as animation production, virtual reality, game development, gait analysis, and biomechanics research.

[0003] The Chinese invention patent with the patent application number 202110786864.0 discloses a human motion capture method: at least two optical cameras are used to accurately achieve the positioning and motion pose calculation of a human body in a ground coordinate system, and the accuracy of the motion capture result can be ensured even in the case of limited shooting scenes indoors and outdoors, greatly improving the practicability, convenience, accuracy, stability, and applicability of the motion capture technology.

[0004] However, in the field of animation production, in order to achieve accurate capture and restoration of human motions, technicians have been committed to developing efficient motion capture technologies. Traditional animation production often relies on manual drawing or key-frame animation, and these methods have limitations in representing complex or delicate human motions. With the rapid development of computer technology and image processing technology, technologies based on image processing and motion capture have gradually become a popular direction in animation production. Summary of the Invention

[0005] By providing a motion capture system and method, the present application introduces a starting motion frame to accurately identify the pixel coordinates of the motion, improving the capture accuracy.

[0006] The present application provides a motion capture method, and the method includes:

[0007] S1, obtaining a scene image containing a human motion in an animation sequence, setting marking points at key parts of the human body in the image, and performing frame division processing on the animation sequence;

[0008] S2, adding motion frames to the human motion after frame division processing, marking and tracking the starting and ending motion frames of the human motion, and tracking and recording the pixel coordinates of the marking points in the motion frames;

[0009] S3, constructing the motion trajectory of the marking points according to the collected pixel coordinates, and performing relevant motion association and combination according to the motion trajectory information; wherein, the motion trajectory is the marking point number, position, frame number, and pixel coordinates (x, y);

[0010] S4. Combine relevant actions and perform action pose calculation to obtain the 3D coordinates and spatial attitude angles of the human body in the coordinate system.

[0011] S5. Combine the 3D coordinates and spatial attitude angles to calculate the precise 3D position of the human body in the animation scene, and adjust the result to the coordinate system used by the animation software through coordinate system conversion.

[0012] Preferably, the S2 includes:

[0013] A1. Use the OpenCV computer vision library to read and preprocess the animated sequence after frame division and post - processing.

[0014] A2. Based on the optical flow field, calculate the movement of pixels between adjacent frames to identify the start and end action frames of the human body movement, thereby determining the action frames. Among them, tracking the start and end action frames of the human body movement includes the start action frame at the starting point of action analysis and the end action frame at the end point defining the complete range of the action. The animated frames between the start action frame and the end action frame are the action frames containing the human body movement.

[0015] A3. After identifying the start action frame, track and record the pixel coordinates of each marker point in each frame of the action frame.

[0016] Preferably, the S3 includes:

[0017] B1. Organize the pixel coordinates of each punctuation mark in each frame image of the action frame into a time - continuous data structure.

[0018] B2. For the data structure of each marker point, process its pixel coordinates in the action frame in sequence, and connect the pixel coordinates of the marker point in the order of the number of frames to form the movement trajectory of the marker point.

[0019] B3. Combine the movement trajectories between marker points to obtain the relevant action association combination.

[0020] Preferably, the A2 for determining the action frames further includes:

[0021] A21. Segment the identified action frames in the animated sequence and name the segmented action frames of the segmented animated sequence separately.

[0022] A22. Identify the movement trajectory of the human body in the separately named action frames, calculate the covering influence degree of the overlapping frame images in the movement frames where the marker points cannot be recognized, and judge the influence of the covering influence degree on the movement trajectory.

[0023] A23. If the covering influence degree is less than the threshold, ignore the influence of the overlapping frame images on the movement trajectory.

[0024] A24. If the occlusion influence degree is greater than the threshold value, calculate the feature point matching degree of the overlapping motion trajectories of the frame images.

[0025] Preferably, for the above-mentioned A22, the calculation of the occlusion influence degree includes:

[0026]

[0027] A cover is the area of the occluded picture in pixels, A total is the number of marked points occluded, is the ratio of the occluded picture to the human body picture area; N covered is the number of frames of the occlusion action, N total is the total number of marked points in the entire motion trajectory in the action frame, is the ratio of the occluded marked points to the marked points in the entire trajectory; F covered is the number of frames of the occluded action, F total is the total number of frames of the entire action frame, is the ratio of the number of occluded frames to the number of frames of the entire action; ΔD is the difference metric at the connection of the trajectories before and after occlusion, defined as the average displacement difference at the connection of the trajectories before and after occlusion, which directly reflects the difference at the connection of the trajectories before and after occlusion; ∝ is an adjustment coefficient to adjust the relative importance of different factors in the influence degree calculation, used to reflect the relative importance of different factors in the influence degree calculation.

[0028] Preferably, for the above-mentioned A24, the calculation of the feature point matching degree of the overlapping motion trajectories of the frame images includes:

[0029] A241. Highlight the marked points on the motion trajectory before overlap and the motion trajectory after overlap as the feature points for tracking the overlap;

[0030] A242. Extract the motion trajectory T 1 before overlap and the motion trajectory T 2 after overlap, and assume a possible motion trajectory T s through the difference method according to the occlusion influence degree;

[0031] A243. Match the feature points on T s with the feature points on the motion trajectories before and after overlap, and evaluate the matching degree of the assumed motion trajectory with the trajectories before and after overlap according to the calculated matching degree;

[0032] S244. If the matching degree of a single feature point in T s is higher than P, it is considered that the motion trajectory T s is reasonable; if T sThere are multiple feature points in it. If the number of feature points with a matching degree higher than P in the feature points exceeds 70% of the number of feature points on the overlapping motion trajectory, then the motion trajectory T s is considered reasonable; if the motion trajectory T s is unreasonable, then the motion trajectory T s is re-assumed;

[0033] A245, connect T 1 , T s and T 2 in the time order of the action frames to form a complete motion trajectory.

[0034] Preferably, for the A243, the formula for calculating the matching degree includes:

[0035] The matching degree calculation formula is:

[0036] N is the number of feature points used to calculate the matching degree, S i is the shape similarity of the i-th feature point (which can be calculated through shape descriptors such as Hough transform, edge detection, etc.) D i is the direction similarity of the i-th feature point (which can be calculated through direction vectors or angle differences), V i is the speed similarity of the i-th feature point (which can be calculated through speed vectors or speed differences); α, β, γ are weight coefficients used to adjust the relative importance of different features in the matching degree calculation.

[0037] Preferably, the A22 also includes:

[0038] A221, adjust the acquisition strategy of feature points according to body posture, clothing type, screen ratio, and occlusion relationship;

[0039] A222, dynamically adjust the occlusion influence degree threshold according to body posture, clothing type, screen ratio, and occlusion relationship;

[0040] The dynamic adjustment of the occlusion influence degree threshold includes:

[0041] I dynamic = K body * K clothing * K ratio * K occlusion * I

[0042] K body is the body posture coefficient, which is adjusted according to the body posture of the human body; K clothing is the clothing coefficient, which is adjusted according to the clothing type of the human body; K ratio is the screen ratio coefficient, which is adjusted according to the proportion of the human body in the screen; K occlusionis the occlusion relationship coefficient, which is adjusted according to the front-back occlusion relationship of the human body in the picture.

[0043] Preferably, the S4 includes:

[0044] Use forward kinematics to clarify the joint chain structure of the human body and the relative position relationship, and clarify the joint angles of the relative parent joints for each marked point in the human joint chain structure; select the human root coordinate system as the reference point for all calculations, establish a local coordinate system for each joint marked point, the origin of this coordinate system is located at the marked point of the joint, and the direction of the coordinate axis is determined according to the rotation axis of the joint; the local coordinate system will rotate with the rotation of the joint marked point, but the origin position remains unchanged.

[0045] This application also provides an action capture system, which includes:

[0046] The marked point setting and tracking module is used to set marked points at the key parts of the human body in the animation sequence and track and record the pixel coordinates of these marked points in the action frames;

[0047] The action frame recognition and segmentation module is used to identify the start and end action frames of the human body action based on the optical flow field technology, segment the animation sequence, and name the segmented action frames separately;

[0048] The motion trajectory construction and association combination module is used to construct the motion trajectory of the marked points according to the collected pixel coordinates and associate and combine the relevant actions;

[0049] The pose calculation and 3D positioning module is used to use forward kinematics to calculate the 3D coordinates and spatial pose angles of the human body in the coordinate system, and calculate the precise 3D position of the human body in the animation scene in combination with this information;

[0050] The occlusion influence degree calculation and threshold adjustment module is used to calculate the occlusion influence degree in the case of marked point overlap or occlusion, and dynamically adjust the occlusion influence degree threshold according to body build, clothing type, picture ratio and occlusion relationship.

[0051] One or more technical solutions provided in this application have at least the following technical effects or advantages:

[0052] By setting marker points at the key parts of the animated human body and using the optical flow field technology to calculate the pixel motion between adjacent frames, this solution can accurately identify the start and end frames of human actions, thus effectively defining the complete range of actions. On this basis, this solution further constructs the motion trajectory of the marker points, and provides a rich data basis for subsequent action analysis by associating and combining relevant actions. Finally, through the application of forward kinematics, this solution successfully calculates the 3D coordinates and spatial attitude angles of the human body in the coordinate system, realizing the accurate 3D position positioning of the human body in the animated scene.

[0053] Through the fine segmentation of the animation sequence and the separate naming of action frames, it provides a basis for the recognition and analysis of the human motion trajectory. For the situation where marker points cannot be recognized due to frame image overlap, a calculation method of occlusion influence degree is introduced, comprehensively considering multiple factors such as the area of the occluded picture, the proportion of occluded marker points, the proportion of occluded frame numbers, and the differences at the connection of the trajectories before and after occlusion, to comprehensively evaluate the influence of occlusion on the motion trajectory. Assume possible motion trajectories by the difference method and perform feature point matching with the actual trajectories before and after overlap. Judge the rationality of the assumed trajectory according to the matching degree. This process not only considers the similarity of the shape, direction, and speed of feature points, but also adjusts the relative importance of different features in the calculation of the matching degree through weight coefficients. Finally, connect the trajectories before and after overlap with the assumed trajectory in chronological order to form a complete motion trajectory, improving the accuracy and coherence of motion trajectory recognition.

[0054] By comprehensively considering body shape, clothing type, picture ratio, and occlusion relationship, dynamically adjusting the acquisition strategy of feature points and the occlusion influence degree threshold, this solution not only improves the accuracy of motion trajectory recognition, but also can be flexibly adjusted according to different scenarios and human characteristics, enhancing the robustness of recognition; by dynamically adjusting the occlusion influence degree threshold, it effectively reduces the influence of occlusion on the recognition effect, enabling high recognition accuracy to be maintained in complex scenarios; at the same time, this solution has high adaptability and scalability, providing a more efficient and accurate solution for fields such as animation production and motion analysis. Brief Description of the Drawings

[0055] Figure 1 It is a schematic flow chart of an action capture method according to an embodiment of the present invention;

[0056] Figure 2 It is a structural block diagram of an action capture system according to an embodiment of the present invention. Detailed Embodiments

[0057] To facilitate the understanding of the present invention, the present application will be described more comprehensively with reference to the relevant attached drawings. The preferred embodiments of the present invention are shown in the drawings. However, the present invention can be implemented in many different forms and is not limited to the embodiments described herein. On the contrary, these embodiments are provided to make the disclosure of the present invention more thorough and comprehensive.

[0058] It should be noted that the terms "vertical", "horizontal", "upper", "lower", "left", "right" and similar expressions used herein are for illustrative purposes only and do not represent the only embodiments.

[0059] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the technical field to which the present invention belongs. The terms used in the specification of the present invention herein are only for the purpose of describing specific embodiments and are not intended to limit the present invention. The term "and / or" used herein includes any and all combinations of one or more of the related listed items.

[0060] Embodiment 1: Figure 1 It is a schematic flowchart of the motion capture method of the embodiment of the present invention.

[0061] As Figure 1 shown, a motion capture method includes the following steps:

[0062] S1. Obtain the scene image containing the human body motion in the animation sequence, set marker points at the key parts of the human body in the image, and perform frame-by-frame processing on the animation sequence.

[0063] Among them, the frame-by-frame processing of the animation sequence uses the OpenCV open-source computer vision library. The positions of the marker points of the key parts of the human body in the animation frame are mainly joint points and points that will move when the human body motion occurs. Usually, the head (the top of the head and eyes), the torso (both shoulders, waist and hips), the upper limbs (elbows, wrists and finger joints) and the lower limbs (knees, ankles, heels and toes) are selected, and these marker points are saved as coordinate points on the image.

[0064] S2. Add action frames to the human body motion after frame-by-frame processing, mark and track the start and end action frames of the human body motion, and track and record the pixel coordinates of the marker points in the action frames.

[0065] In some embodiments, step S2 includes:

[0066] A1. Use the OpenCV computer vision library to read and preprocess the animation sequence after frame-by-frame processing.

[0067] Among them, the preprocessing includes grayscale conversion and denoising.

[0068] A2. Identify the start and end frames of human actions by calculating the movement of pixels between adjacent frames based on the optical flow field, thereby determining the action frames.

[0069] Among them, tracking the start and end frames of human actions includes the start action frame at the starting point of action analysis and the end action frame at the ending point defining the complete range of the action. The animation frames between the start action frame and the end action frame are the action frames containing human actions.

[0070] Specifically, identify the start and end frames of human actions by calculating the movement of pixels between adjacent frames based on the optical flow field. Perform Gaussian smoothing on the image, calculate the gradients of the smoothed image in the x and y directions, and calculate the optical flow field through the optical flow algorithm (Lucas-Kanade method); in the first frame image of the animation sequence, use the feature point detection algorithm (SIFT) to extract the landmark points of the human body, and use the optical flow algorithm to calculate the movement trajectories of each landmark point in the subsequent frames; when the optical flow velocity of a certain landmark point exceeds the preset threshold, it is considered that the human action starts, and this frame is marked as the start action frame; continue to track the movement trajectory of the landmark point in the subsequent frames. When the optical flow velocity of the landmark point gradually decreases and tends to be stable, it is considered that the human action ends, and this frame is marked as the end action frame; mark all the frames between the start action frame and the end action frame as action frames. The preset threshold of the optical flow velocity is automatically adjusted by machine learning training a large amount of animated human action data.

[0071] A3. After identifying the start action frame, track and record the pixel coordinates of each landmark point in each frame of the action frame.

[0072] S3. Construct the movement trajectory of the landmark points according to the collected pixel coordinates, and perform relevant action association and combination according to the movement trajectory information.

[0073] Among them, the movement trajectory includes the landmark point number, position (such as "left shoulder", "right knee", etc.), frame number (from the i-th frame to the n-th frame), and pixel coordinates (x, y).

[0074] In some embodiments, step S3 includes:

[0075] B1. Organize the pixel coordinates of each punctuation in each frame image of the action frame into a time-continuous data structure.

[0076] Among them, the data structure is a sequence of ordered point sets, and each point contains the frame number and the corresponding pixel coordinates.

[0077] B2. For the data structure of each landmark point, process its pixel coordinates in the action frame in turn, and connect the pixel coordinates of the landmark points in the order of frame numbers to form the movement trajectory of the landmark point.

[0078] B3. Combine the movement trajectories between the landmark points to obtain relevant action association and combination.

[0079] Among them, the movement trajectory combination is that when performing Action A, the marked point q_move will drive the marked point w_move, so Action A includes the marked point q and the marked point w. The marked point q and the marked point w are combined and associated into a group of movement trajectory combinations.

[0080] S4. Perform action pose calculation on the associated combination of relevant actions to obtain the 3D coordinates and spatial attitude angles of the human body in the coordinate system.

[0081] Specifically, using forward kinematics, clarify the joint chain structure of the human body and the relative position relationship. For the human arm, it can be simplified into a joint chain composed of the marked points between the shoulder joint, elbow joint, and wrist joint. For each marked point in the joint chain, clarify its rotation angle (joint angle) relative to the parent joint;

[0082] Select a base coordinate system (such as the coordinate system at the root of the human body) as the reference point for all calculations; establish a local coordinate system for each joint marked point. The origin of this coordinate system is located at the marked point of the joint, and the direction of the coordinate axis is determined according to the rotation axis of the joint; the local coordinate system will rotate with the rotation of the joint marked point, but the origin position remains unchanged; for calculating the transformation matrix and recursively calculating the position and attitude of the end effector, please refer to the calculation steps of forward kinematics, which will not be elaborated in this article.

[0083] S5. Combine the 3D coordinates and spatial attitude angles to calculate the accurate 3D position of the human body in the animation scene, and adjust the result to the coordinate system used by the animation software through coordinate system conversion.

[0084] Specifically, construct the skeleton model of the human body in three-dimensional space with the 3D coordinates of each joint marked point of the human body and the overall or local spatial attitude angles obtained by solving in step S4; use the obtained spatial attitude angle information to adjust the pose of the skeleton model, and calculate the accurate 3D position of the human body in the animation scene according to the skeleton model and the adjusted pose; identify the coordinate system used by the animation software, convert the coordinate system of the human 3D model to the coordinate system used by the animation software as needed, import the converted human 3D model into the animation software, and output the adjusted human 3D model in a format recognizable by the animation software.

[0085] The technical solutions in the embodiments of the present application above have at least the following technical effects or advantages:

[0086] By setting marker points at the key parts of the animated human body and using the optical flow field technology to calculate the pixel motion between adjacent frames, this solution can accurately identify the start and end frames of human actions, thus effectively defining the complete range of actions. On this basis, this solution further constructs the motion trajectories of the marker points, and provides a rich data basis for subsequent action analysis by associating and combining relevant actions. Finally, through the application of forward kinematics, this solution successfully calculates the 3D coordinates and spatial attitude angles of the human body in the coordinate system, realizing the accurate 3D position positioning of the human body in the animated scene.

[0087] Embodiment 2: In an actual animation sequence, the marker points of different human bodies overlap or the background is complex and covers the human body, resulting in some marker points of the human body being covered by the marker points of other human bodies. At this time, if only the visible marker points are relied on to construct the motion trajectory, the action details of the covered human body cannot be accurately captured. Therefore, the relationship between the marker points between adjacent frames and the overlapping changes of the marker points are extremely important for determining the motion trajectory of the start and end frames of human actions. If the influence value on the continuity of the motion trajectory after overlapping is not used to assist in the identification, it is very difficult to accurately confirm the motion trajectory of the human body.

[0088] Therefore, the embodiment of the present application is optimized to a certain extent on the basis of the above embodiment.

[0089] In some embodiments, in step A2, determining the action frames further includes:

[0090] A21, segmenting the identified action frames into an animation sequence, and separately naming the segmented action frames of the animation sequence.

[0091] A22, identifying the motion trajectory of the human body in the separately named action frames, calculating the covering influence degree of the action frames with overlapping frame images and unidentifiable marker points in the motion frames, and judging the influence of the covering influence degree on the motion trajectory.

[0092] Among them, the covering influence degree calculation formula is:

[0093]

[0094] A cover is the area of the covered picture in pixels, A total is the number of covered marker points, is the proportion of the covered picture in the human body picture area; N covered is the number of frames of the covered action frames, N total is the total number of marker points in the entire motion trajectory in the action frames, is the proportion of the covered marker points in the marker points in the entire trajectory; F covered is the number of frames of the covered action frames, F totalis the total number of frames in the entire action frame, is the ratio of the number of occluded frames to the number of frames in the entire action frame; ΔD is the difference metric at the junction of the trajectories before and after occlusion, defined as the average displacement difference of the trajectories before and after occlusion at the junction, which directly reflects the difference at the junction of the trajectories before and after occlusion; ∝ is an adjustment coefficient to adjust the relative importance of different factors in the impact degree calculation, used to reflect the relative importance of different factors in the impact degree calculation, and this coefficient is set according to the specific application scenario and data characteristics.

[0095] A23, if the occlusion impact degree is less than the threshold, the influence of the frame image overlap on the motion trajectory is ignored.

[0096] Among them, the occlusion impact degree threshold is set to define the influence size of the frame image overlap on the connection of the motion trajectories before and after overlap. The setting of the threshold should be based on the specific application scenario and data characteristics. If the occlusion impact degree is less than the threshold, the influence of the frame image overlap on the motion trajectory can be ignored, and it is considered that the occlusion has no significant impact on the recognition and analysis of the motion trajectory. In practical applications, an appropriate threshold range is determined through experiments. For example, a smaller threshold can be started with and gradually increased, and the influence on the results of the motion trajectory recognition and analysis is observed, so as to determine a threshold range that can not only meet the accuracy requirements but also take into account the processing efficiency.

[0097] A24, if the occlusion impact degree is greater than the threshold, calculate the feature point matching degree of the motion trajectory with frame image overlap.

[0098] Specifically, calculating the feature point matching degree of the motion trajectory with frame image overlap includes:

[0099] A241, mark prominent points on the motion trajectory before overlap and the motion trajectory after overlap as the feature points for tracking the overlap.

[0100] A242, extract the motion trajectory T before overlap 1 and the motion trajectory T after overlap 2 , and assume a possible motion trajectory T through the difference method according to the occlusion impact degree s .

[0101] A243, match the feature points on T s with the feature points on the motion trajectories before and after overlap, and evaluate the matching degree of the assumed motion trajectory with the motion trajectories before and after overlap according to the calculated matching degree.

[0102] Among them, the matching degree calculation formula is:

[0103]

[0104] N is the number of feature points used to calculate the matching degree, S iis the shape similarity of the i-th feature point (which can be calculated through shape descriptors such as Hough transform, edge detection, etc.), D i is the direction similarity of the i-th feature point (which can be calculated through direction vectors or angle differences), V i is the speed similarity of the i-th feature point (which can be calculated through speed vectors or speed differences); α, β, γ are weight coefficients used to adjust the relative importance of different features in the calculation of the matching degree.

[0105] S244, if T s the matching degree of a single feature point in it is higher than P, then the motion trajectory T s is considered reasonable; if T s there are multiple feature points in it, and the number of feature points with a matching degree higher than P exceeds 70% of the number of feature points on the overlapping motion trajectory, then the motion trajectory T s is considered reasonable; if the motion trajectory T s is unreasonable, then re-assume the motion trajectory T s .

[0106] Among them, P is the matching degree threshold, and in practical applications, a suitable threshold is simulated according to specific requirements and data characteristics.

[0107] A245, connect T 1 , T s and T 2 in the time order of the action frames to form a complete motion trajectory.

[0108] The technical solutions in the embodiments of the present application described above have at least the following technical effects or advantages:

[0109] Through the fine segmentation of the animation sequence and the separate naming of the action frames, a basis is provided for the recognition and analysis of the human motion trajectory. For the situation where the marker points cannot be recognized due to the overlap of frame images, a calculation method for the covering influence degree is introduced, comprehensively considering multiple factors such as the covered screen area, the proportion of covered marker points, the proportion of covered frame numbers, and the differences at the connection of the trajectories before and after covering, to comprehensively evaluate the influence of covering on the motion trajectory. Assume possible motion trajectories through the difference method and perform feature point matching with the actual trajectories before and after overlap, and judge the rationality of the assumed trajectories based on the matching degree. This process not only considers the shape, direction, and speed similarities of the feature points, but also adjusts the relative importance of different features in the calculation of the matching degree through weight coefficients. Finally, connect the trajectories before and after overlap and the assumed trajectories in time order to form a complete motion trajectory, improving the accuracy and coherence of the motion trajectory recognition.

[0110] Embodiment 3: In Embodiment 2, accurate recognition of the human body's movement trajectory is achieved by finely segmenting the animation sequence, calculating the occlusion influence degree, and hypothesizing and verifying the movement trajectory. To apply this method to different humans, different costumes, different body postures, and different proportions, it is necessary to flexibly adjust the acquisition strategy of feature points. For example, more stable marker points are selected according to the human body characteristics and costumes, and the recognition algorithm of the movement trajectory is optimized to adapt to the changes brought by different body postures and proportions, so as to ensure the accuracy and robustness of the movement trajectory recognition.

[0111] Therefore, the embodiments of the present application are optimized to a certain extent on the basis of the above embodiments.

[0112] In some embodiments, in step A22, it further includes:

[0113] A221, adjust the acquisition strategy of feature points according to the body posture, clothing type, screen ratio, and occlusion relationship.

[0114] Specifically, the body posture of the human body is recognized through image processing technology, and the body posture coefficient K is set according to the body posture type body ; among them, the body posture is slender, short and fat, etc.; the clothing type of the human body is recognized through image recognition technology, and the clothing coefficient K is set according to the clothing type clothing ; among them, the clothing type is tight clothes, loose clothes, thick coats; calculate the proportional size of the human body in the screen, and set the screen ratio coefficient K according to the screen ratio ratio ; among them, the proportional size is the ratio of the human body area to the screen area; recognize the occlusion relationship of the human body in the screen, and set the occlusion relationship coefficient K according to the occlusion relationship occlusion ; among them, the occlusion relationship is to determine which human body is in front and which is behind.

[0115] For example, for a slender human body, the acquisition density of joint points can be increased; for loose clothing, more stable feature points can be selected, etc., and the adjusted feature point acquisition strategy is used for movement trajectory recognition.

[0116] A222, dynamically adjust the occlusion influence degree threshold according to the body posture, clothing type, screen ratio, and occlusion relationship.

[0117] Among them, the formula for dynamically adjusting the occlusion influence degree threshold is:

[0118] I dynamic =K body *K clothing *K ratio *K occlusion *I

[0119] K body is the body posture coefficient, which is adjusted according to the body posture of the human body (such as slender, short and fat, etc.); Kclothing is the wearing coefficient, which is adjusted according to the wearing type of the human body (such as tight clothes, loose clothes, etc.); K ratio is the screen ratio coefficient, which is adjusted according to the proportion of the human body in the screen; K occlusion is the occlusion relationship coefficient, which is adjusted according to the front-back occlusion relationship of the human body in the screen.

[0120] For example, in an animated movie, there are multiple characters interacting in a scene. Their body shapes, postures, clothing, and positional relationships are different. It is necessary to dynamically adjust the acquisition strategy and influence degree of feature points according to this information to improve the accuracy of motion trajectory recognition;

[0121] Setting of body shape coefficient: For slender characters: Set K body = 1.2, because slender characters may have a larger range of joint movement and require a higher density of feature point acquisition; For short and stocky characters: Set K body = 1.0, because the body contour changes of short and stocky characters may be more obvious, but no additional feature point acquisition density is required; For well-proportioned characters: Set K body = 1.1, as an intermediate value;

[0122] Setting of wearing coefficient: For characters in tight clothes: Set K clothing = 1.0, because the feature points under tight clothes are easy to identify and track; For characters in loose clothes: Set K clothing = 1.2, because loose clothes may block some feature points and a higher density of feature point acquisition is required; For characters in thick coats: Set K clothing = 1.4, because thick coats block more severely and a higher density of feature point acquisition and resolution are required;

[0123] Setting of screen ratio coefficient: For small-scale characters (occupying less than 10% of the screen): Set K ratio = 1.5, because a higher density of feature point acquisition and resolution are required; For medium-scale characters (occupying 10% - 50% of the screen): Set K ratio = 1.2, as an intermediate value; For large-scale characters (occupying more than 50% of the screen): Set K ratio = 1.0, because the density of feature point acquisition and resolution are already sufficient;

[0124] Setting of occlusion relationship coefficient: For characters with no occlusion: Set K occlusion = 1.0, because the feature points are completely visible; For characters with partial occlusion (occlusion ratio less than 50%): Set K occlusion = 1.2, because it is necessary to use algorithms to predict or estimate the positions of occluded feature points; For characters with complete occlusion (occlusion ratio greater than 50%): Set K occlusion= 1.5 because a higher feature point acquisition density and a more complex algorithm are required to handle occlusion relationships;

[0125] Suppose in a certain frame of the picture, there is a slender character wearing loose clothes, occupying 20% of the picture, and being partially occluded (occlusion ratio 30%). According to the formula, its dynamically adjusted influence degree is calculated as follows:

[0126] I dynamic = 1.2×1.2×1.2×1.2×I. Through this method, the acquisition strategy and influence degree of the animation human feature points can be dynamically adjusted according to different body postures, clothing, picture proportion size, and front-back occlusion relationship, thereby improving the accuracy and robustness of motion trajectory recognition.

[0127] The technical solutions in the embodiments of the present application at least have the following technical effects or advantages:

[0128] By comprehensively considering body posture, clothing type, picture proportion, and occlusion relationship, dynamically adjusting the acquisition strategy of feature points and the occlusion influence degree threshold, this solution not only improves the accuracy of motion trajectory recognition, but also can be flexibly adjusted according to different scenarios and human characteristics, enhancing the robustness of recognition; by dynamically adjusting the occlusion influence degree threshold, the influence of occlusion on the recognition effect is effectively reduced, so that a high recognition accuracy can be maintained in complex scenarios; at the same time, this solution has high adaptability and scalability, providing a more efficient and accurate solution for fields such as animation production and motion analysis.

[0129] Furthermore, the embodiment of the present invention also provides an action capture system.

[0130] Figure 2 It is a schematic structural diagram of the action capture system according to the embodiment of the present invention.

[0131] As Figure 2 shown, an action capture system includes: a marker point setting and tracking module, an action frame recognition and segmentation module, a motion trajectory construction and association combination module, and a pose solution and 3D positioning module.

[0132] The marker point setting and tracking module: used to set marker points at the key parts of the human body in the animation sequence and track and record the pixel coordinates of these marker points in the action frame.

[0133] The action frame recognition and segmentation module: based on the optical flow field technology, identify the start and end action frames of the human body movement, segment the animation sequence, and name the segmented action frames separately.

[0134] The motion trajectory construction and association combination module: construct the motion trajectory of the marker points according to the collected pixel coordinates and associate and combine the relevant actions.

[0135] Attitude solution and 3D positioning module: Use forward kinematics to calculate the 3D coordinates and spatial attitude angles of the human body in the coordinate system, and calculate the accurate 3D position of the human body in the animation scene by combining this information.

[0136] Occlusion influence degree calculation and threshold adjustment module: Calculate the occlusion influence degree in the case of marker point overlap or occlusion, and dynamically adjust the occlusion influence degree threshold according to body build, clothing type, screen ratio and occlusion relationship.

[0137] It should be noted that other specific implementation contents of the motion capture system in the embodiments of the present invention can refer to the above-mentioned motion capture method.

[0138] The above are only the preferred embodiments of the present invention and are not used to limit the present invention. For those skilled in the art, the present invention can have various changes and modifications. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A motion capture method, characterized in that: The method comprises: S1, obtaining a scene image containing human body movements in an animation sequence, setting marker points at key parts of the human body in the image, and performing frame processing on the animation sequence; S2, adding action frames to the human body motion after frame processing, marking the start and end action frames of the human body motion, and tracking and recording the pixel coordinates of the marked points in the action frames; wherein the animation frames between the start action frame and the end action frame are the action frames containing the human body motion; S3, constructing a motion trajectory of the marker point according to the collected pixel coordinates, and performing associated combination of related actions according to the motion trajectory; wherein the motion trajectory is the marker point number, position, frame number and pixel coordinates (x, y); S4, combining related actions to solve the action posture to obtain the 3D coordinates and spatial posture angle of the human body in the coordinate system; S5, combines 3D coordinates and spatial posture angles to calculate the precise 3D position of the human body in the animation scene, and adjusts the result to the coordinate system used by the animation software through coordinate system conversion.

2. The motion capture method according to claim 1, characterized in that: The S2 comprises: A1, using OpenCV computer vision library to read and pre-process the frame-by-frame processed animation sequence; A2, based on the optical flow field, the movement of pixels between adjacent frames is calculated to identify the start and end action frames of the human action, thereby determining the action frames; wherein the start and end action frames of the human action tracking include the start action frame of the action analysis start point and the stop action frame defining the end point of the complete range of the action; A3, after identifying the action frame, track and record the pixel coordinates of each marked point in each frame of the action frame.

3. The motion capture method according to claim 1, characterized in that: The S3 includes: B1, organizing the pixel coordinates of each punctuation point of each frame image of the action frame into a time-continuous data structure; B2, for each marker point data structure, process its pixel coordinates in the action frame in turn, connect the pixel coordinates of the marker points in the order of the frame numbers, and form the motion trajectory of the marker point; B3, combine the motion trajectories between the marked points to obtain the related action association combination.

4. The motion capture method according to claim 2, characterized in that: The A2, determining the action frame further comprises: A21, segmenting the recognized action frames into animation sequences, and individually naming the segmented action frames of the segmented animation sequences; A22, identifying the motion trajectory of the human body in the individually named action frames, calculating the overlapping masking influence of the action frames in which the frame images overlap and the marked points cannot be identified, and determining the influence of the masking influence on the motion trajectory; A23, if the masking influence is less than the threshold, the influence of frame image overlap on the motion trajectory is ignored; A24, if the covering influence is greater than the threshold, the matching degree of the feature points of the motion trajectory of the overlapping frame images is calculated.

5. The motion capture method according to claim 4, characterized in that: The A22, covering influence degree calculation includes: the covering influence degree calculation formula is: A cover is the area of ​​the covered screen in pixels, A total is the number of covered markers, N is the ratio of the masked image to the human body image area; covered is the number of frames to cover the action frames, N total is the total number of marker points in the entire motion trajectory in the action frame, is the ratio of covered markers to the markers in the entire trajectory; F covered is the number of action frames covered, F total is the total number of frames of the entire action frame, is the ratio of the number of covered frames to the total number of action frames; ΔD is the difference measure of the connection between the trajectory before and after covering; ∝ is an adjustment coefficient used to adjust the relative importance of different factors in the influence calculation.

6. The motion capture method according to claim 4, characterized in that: The A24, calculating the matching degree of feature points of the motion trajectory of the overlapping frame images, comprises: A241, highlighting the marking points on the motion trajectory before and after the overlap as the feature points for tracking the overlap; A242, extract the motion trajectory T1 before overlapping and the motion trajectory T2 after overlapping, and construct a preset motion trajectory T according to the covering influence by using the difference method s ; A243, T s The feature points on the image are matched with the feature points on the motion trajectory before and after the overlap, and the matching degree between the assumed motion trajectory and the trajectory before and after the overlap is evaluated according to the calculated matching degree; S244, if T s If the matching degree of a single feature point is higher than P, then the motion trajectory T s is reasonable; if T s There are multiple feature points in the feature points. If the number of feature points with a matching degree higher than P exceeds 70% of the number of feature points on the overlapping motion trajectory, then the motion trajectory T s is reasonable; if the motion trajectory T s If it is unreasonable, then re-assume the motion trajectory T s ; A245, T1, T s It is connected with T2 in the time sequence of the action frames to form a complete motion trajectory.

7. The motion capture method according to claim 6, characterized in that: The A243, matching degree calculation formula includes: matching degree calculation formula is: N is the number of feature points used to calculate the matching degree, S i is the shape similarity of the i-th feature point, D i is the directional similarity of the i-th feature point, V i is the speed similarity of the i-th feature point; α, β, γ are weight coefficients used to adjust the relative importance of different features in the matching calculation.

8. The motion capture method according to claim 5, characterized in that: The A22 further includes: A221, adjust the feature point collection strategy according to body shape, clothing type, screen ratio and occlusion relationship; A222 dynamically adjusts the occlusion impact threshold based on body shape, clothing type, screen ratio, and occlusion relationship; The dynamically adjusting the covering influence threshold comprises: I dynamic =K body *K clothing *K ratio *K occlusion *I K body K is the body shape coefficient, which is adjusted according to the body shape of the human body; clothing K is the wearing coefficient, which is adjusted according to the type of clothing worn by the human body; ratio K is the screen ratio coefficient, which is adjusted according to the size of the human body in the screen; occlusion It is the occlusion relationship coefficient, which is adjusted according to the front and back occlusion relationship of the human body in the picture.

9. The motion capture method according to claim 1, wherein: The S4 includes: using forward kinematics to clarify the joint chain structure and relative position relationship of the human body, and clarifying the joint angle of each marker point in the human body joint chain structure relative to its parent joint; selecting the human body root coordinate system as the reference point for all calculations, and establishing a local coordinate system for each joint marker point, the origin of the coordinate system is located at the joint marker point, and the direction of the coordinate axis is determined according to the rotation axis of the joint; the local coordinate system will rotate with the rotation of the joint marker point, but the position of the origin remains unchanged.

10. A system based on a motion capture method, applied to a motion capture method according to any one of claims 1 to 9, characterized in that: The system comprises: The marker setting and tracking module is used to set markers on key parts of the human body in the animation sequence and track and record the pixel coordinates of these markers in the action frames; The action frame recognition and segmentation module is used to identify the start and end action frames of human actions based on the optical flow field technology, segment the animation sequence, and name the segmented action frames separately; The motion trajectory construction and association combination module is used to construct the motion trajectory of the marker point according to the collected pixel coordinates and associate and combine related actions; The posture calculation and 3D positioning module is used to calculate the 3D coordinates and spatial posture angles of the human body in the coordinate system using forward kinematics, and combine this information to calculate the precise 3D position of the human body in the animation scene; The occlusion influence calculation and threshold adjustment module is used to calculate the occlusion influence when there is overlap or occlusion of marker points, and dynamically adjust the occlusion influence threshold according to body shape, clothing type, picture ratio and occlusion relationship.

Images