Method for converting hand-captured 3D coordinate data into rotation data

Abstract

The invention discloses a method for converting hand-captured 3D coordinate data into rotation data, and the method comprises the following steps: S1, carrying out the motion capture of a video, and obtaining human motion data and human hand 3D coordinate data; s2, converting the 3D coordinate data of the human hand into hand capture rotation data; s3, splicing the hand capture rotation data with the human body motion data; and S4, carrying out jitter optimization on the spliced data. The method has the beneficial effects that the human hand 3D coordinate data is converted into the hand capture rotation data, then the hand capture rotation data and the human motion data are spliced, and finally the spliced data is subjected to jitter optimization, so that the method is used for converting and optimizing the human hand 3D coordinate data, repairing the defects of the hand capture data, and improving the human motion quality. Therefore, the generated data better conforms to the physical law of human motion, and a large amount of manpower cost is avoided for later repair.

Description

Technical Field

[0001] The present invention relates to the field of AI video motion capture technology, and in particular to a method for converting hand-captured 3D coordinate data into rotation data. Background Art

[0002] AI video motion capture is a technology that uses machine learning to restore human motion in two-dimensional videos into three-dimensional human motion data. Currently, due to the uncertainty of the input video source, the human motion data generated by AI video motion capture has more or less quality defects. In the capture process of hands in AI video motion capture, the 3D coordinate data of the human hand is usually first predicted and generated by the AI ​​model, and then converted into the rotation data of the human hand through an algorithm. However, in the input video to be predicted, the human hand will appear dynamic blur, cross, occlusion, etc., and the 3D coordinate data of the human hand generated by such a video will have many abnormal phenomena. For example, the 3D coordinate data of a person's hand generated from a video with dynamic hand blur will shake; the 3D coordinate data of a person's hand generated from a video with crossed hands will have inaccurate detection of the hand joints, resulting in unreasonable coordinate positions. For example, the first two nodes of a finger are a certain distance to the left or right of the finger extension direction, which means that the finger is bent to the side, while normally the finger can only bend downward, which is obviously unreasonable; the 3D coordinate data of a person's hand generated from a video with occluded hands cannot be detected. In this case, the coordinate data is usually chaotic. These defects will require manpower costs to repair later, resulting in excessive manpower costs. Summary of the invention

[0003] The purpose of the present invention is to overcome the shortcomings of the prior art and provide a method for converting hand-captured 3D coordinate data into rotation data.

[0004] The object of the present invention is achieved by the following technical solution: A method for converting hand-captured 3D coordinate data into rotation data, comprising the following steps:

[0005] S1: Capture motion on the video to obtain human motion data and 3D coordinate data of human hands;

[0006] S2: converting the 3D coordinate data of the human hand into hand capture rotation data;

[0007] S3: splicing the hand capture rotation data with the human body motion data;

[0008] S4: Perform jitter optimization on the spliced ​​data.

[0009] Preferably, step S2 further includes the following steps:

[0010] S21: Calculate the quaternion rotation value of the wrist joint point;

[0011] S22: Calculate the rotation value of each finger root joint point;

[0012] S23: Calculate the rotation value of the second joint point and the third joint point of each finger;

[0013] S24: Obtain hand capture rotation data according to the rotation values ​​obtained in step S21 to step S23, whose dimension is an array of (f, j, 4), where f is the number of frames, j is the total number of joints of the left and right hands, and 4 is the rotation value of each joint represented by a quaternion.

[0014] Preferably, step S21 further includes the following steps:

[0015] S21.1: Take the vector V in the 3D coordinate data 1 and V 2 , calculate a set of orthogonal vectors,

[0016]

[0017] Among them, V 1 =P 5 -P 17 , V 2 =P 5 -P 0 , P 5 The fifth finger node, P 17 The finger node at point 17, P 0 It is the finger node at point zero;

[0018] S21.2: From the known initial value of the human skeleton T-pose, a set of initial orthogonal vectors T corresponding to the above are obtained X and T Y , calculate the rotation axis V of the quaternion Axis ,

[0019]

[0020] S21.3: Calculate the rotation angle of the quaternion,

[0021]

[0022] Determine the rotation direction of the quaternion.

[0023] s=sgn((T X ×V X )·V Axis );

[0024] When s=1, it indicates right-hand rotation, and when s=-1, it indicates left-hand rotation;

[0025] S21.4: Calculate the rotation quaternion value R of the wrist jointwrist ,

[0026]

[0027] Preferably, step S22 further includes the following steps:

[0028] S22.1: Known E X , E Y and E Z is the basis of the global coordinate system. After the wrist joint is rotated, the value of the basis of the local coordinates of the joint in the global coordinate system is calculated.

[0029]

[0030] Among them, I 3 is the 3D identity matrix, O (a,b) is a zero matrix with a rows and b columns;

[0031] S22.2: Calculate the rotation value in the y direction,

[0032]

[0033] Among them, V 01 is the direction of the bone at the base of the finger, V 01 =P 10 -P 9 , P 9 The ninth finger node, P 10 It is the finger node at point 10;

[0034] Calculate the rotation value in the z direction,

[0035]

[0036] S22.3: Calculate the rotation value of the finger base joint point,

[0037]

[0038] Preferably, in step S23, the directions of the three bone segments of the finger are V 01 =P 10 -P 9 , V 12 =P 11 -P 10 , V 23 =P 12 -P 11 , calculate the rotation value of the second joint point of the finger:

[0039]

[0040] V Axis2 =EZ sgn(V 01 ×V 12 ·E Z1 );

[0041]

[0042] Calculate the rotation value of the third joint of the finger:

[0043]

[0044] V Axis3 =E Z sgn(V 12 ×V 23 ·E Z1 );

[0045]

[0046] Preferably, in step S3, the formula for splicing the hand capture rotation data and the human body motion data is:

[0047]

[0048] Among them, i,0≤i≤k is the subscript of each node in a path starting from the human body root node to the wrist target node, Q i is the rotation value of each node in the path, Q Ti is the initial rotation value of T-pose of each node in the path, Q wrist Rotate the wrist target node.

[0049] Preferably, in step S4, the Euler angle representation of the data of the second joint point and the third joint point of the finger has data only in one dimension. Let {e 1 ,e 2 ,Λ,e f} represents the data of a certain joint point in the changed dimension, and the optimized data {E 1 ,E 2 ,Λ,E f}for:

[0050]

[0051] E 1 =e 1 ;

[0052] E f =e f ;

[0053] Assume {m 1 ,m 2 ,Λ,m f} represents the rotation matrix data of a joint point, {d 1 ,d 2 ,Λ,d f} represents 6d rotation data, where:

[0054] d i =((100)m i (0 1 0)m i ) 1≤i≤f;

[0055] Optimized 6d rotation data {D 1 ,D 2 ,Λ,D f}for:

[0056]

[0057] Convert the optimized 6d rotation data into rotation matrix data {M 1 ,M 2 ,Λ,M f}:

[0058]

[0059] Among them, I 3 is the 3D identity matrix, O (a,b) is a zero matrix with a rows and b columns.

[0060] The present invention has the following advantages: the present invention converts the 3D coordinate data of the human hand into hand capture rotation data, then splices the hand capture rotation data with the human body motion data, and finally performs jitter optimization on the spliced ​​data, thereby being used for the conversion and optimization of the 3D coordinate data of the human hand, repairing the defects of the hand capture data, making the generated data more consistent with the physical laws of human body motion, and avoiding a large amount of manpower cost for later repair. BRIEF DESCRIPTION OF THE DRAWINGS

[0061] Figure 1 It is a structural schematic diagram of the transformation method process;

[0062] Figure 2 This is a schematic diagram of the structure of the hand node. DETAILED DESCRIPTION

[0063] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Generally, the components of the embodiments of the present invention described and shown in the drawings here can be arranged and designed in various different configurations.

[0064] Therefore, the following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the invention claimed for protection, but merely represents selected embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0065] It should be noted that, in the absence of conflict, the embodiments of the present invention and the features of the embodiments may be combined with each other.

[0066] It should be noted that similar reference numerals and letters denote similar items in the following drawings, and therefore, once an item is defined in one drawing, further definition and explanation thereof is not required in subsequent drawings.

[0067] In the description of the present invention, it should be noted that the terms "center", "upper", "lower", "left", "right", "vertical", "horizontal", "inside", "outside", etc. indicate positions or positional relationships based on the positions or positional relationships shown in the accompanying drawings, or the positions or positional relationships in which the product of the invention is usually placed when in use, or the positions or positional relationships commonly understood by those skilled in the art, which are only for the convenience of describing the present invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as limiting the present invention. In addition, the terms "first", "second", etc. are only used to distinguish the description, and cannot be understood as indicating or implying relative importance.

[0068] In the description of the present invention, it is also necessary to explain that, unless otherwise clearly specified and limited, the terms "set", "install", "connect", and "connect" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or it can be indirectly connected through an intermediate medium, or it can be the internal communication of two elements. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0069] In this embodiment, if Figure 1 and Figure 2 As shown, a method for converting hand-captured 3D coordinate data into rotation data is characterized by comprising the following steps:

[0070] S1: Perform motion capture on the video to obtain human motion data and 3D coordinate data of human hands; preferably, the human motion data includes 3D position data of human joints, which is saved in bvh format, and the 3D coordinate data of human hands is saved in json format.

[0071] S2: converting the 3D coordinate data of the human hand into hand capture rotation data;

[0072] S3: splicing the hand capture rotation data with the human body motion data;

[0073] S4: Perform jitter optimization on the spliced ​​data. The 3D coordinate data of the human hand is converted into hand capture rotation data, and then the hand capture rotation data is spliced ​​with the human motion data. Finally, the spliced ​​data is jitter optimized, which is used for the conversion and optimization of the 3D coordinate data of the human hand, repairing the defects of the hand capture data, making the generated data more consistent with the physical laws of human motion, and avoiding a lot of manpower costs for later repair. Specifically, in step S1, the 3D coordinate data of the human hand is an array of dimension (f, h, p, 3), wherein f is the number of frames, h is the left and right hands, h=0 is the left hand, h=1 is the right hand, p is the number of finger nodes, a hand has a total of 21 nodes, 1 node at the wrist and 4 nodes for each finger, 3 is the 3D coordinate value of each node, that is, the horizontal and vertical coordinates in each frame and the predicted depth coordinate; the hand capture rotation data is an array of dimension (f, j, 4), wherein f is the number of frames, j is the total number of joints of the left and right hands, a total of 32, 16 joints for each hand, including 1 joint point for each wrist and 3 joint points for each finger, compared with the 3D coordinate data of the human hand, it lacks the point at the tip of each finger, and 4 is the rotation value of each joint represented by a quaternion.

[0074] In this embodiment, step S2 further includes the following steps:

[0075] S21: Calculate the quaternion rotation value of the wrist joint point; further, step S21 also includes the following steps:

[0076] S21.1: Take the vector V in the 3D coordinate data 1 and V 2 , calculate a set of orthogonal vectors,

[0077]

[0078] Among them, V 1 =P 5 -P 17 , V 2 =P 5 -P 0 , P 5 The fifth finger node, P 17 The finger node at point 17, P 0 It is the zero point finger node;

[0079] S21.2: From the known initial value of the human skeleton T-pose, a set of initial orthogonal vectors T corresponding to the above are obtainedX and T Y , calculate the rotation axis V of the quaternion Axis ,

[0080]

[0081] S21.3: Calculate the rotation angle of the quaternion,

[0082]

[0083] Determine the rotation direction of the quaternion.

[0084] s=sgn((T X ×V X )·V Axis );

[0085] When s = 1, it indicates right rotation, and when s = -1, it indicates left rotation; that is, here the initial orthogonal vector T X As a benchmark, judge V X For the rotation axis V Axis Is it left-handed or right-handed.

[0086] S21.4: Calculate the rotation quaternion value R of the wrist joint wrist ,

[0087]

[0088] S22: Calculate the rotation value of each finger root joint point; further, taking the middle finger as an example, S22.1: Knowing E X , E Y and E Z is the basis of the global coordinate system. After the wrist joint is rotated, the value of the basis of the local coordinates of the joint in the global coordinate system is calculated.

[0089]

[0090] Among them, I 3 is the 3D identity matrix, O (a,b) is a zero matrix with a rows and b columns;

[0091] S22.2: Calculate the rotation value in the y direction,

[0092]

[0093] Among them, V 01 is the direction of the bone at the base of the finger, V 01 =P 10 -P 9 , P 9 The ninth finger node, P 10It is the finger node at point 10;

[0094] Calculate the rotation value in the z direction,

[0095]

[0096] S22.3: Calculate the rotation value of the finger base joint point,

[0097]

[0098] S23: Calculate the rotation value of the second joint point and the third joint point of each finger; Specifically, in step S23, the direction of the three bone segments of the finger is V 01 =P 10 -P 9 , V 12 =P 11 -P 10 , V 23 =P 12 -P 11 , calculate the rotation value of the second joint point of the finger:

[0099]

[0100] V Axis2 =E Z sgn(V 01 ×V 12 ·E Z1 );

[0101]

[0102] Calculate the rotation value of the third joint of the finger:

[0103]

[0104] V Axis3 =E Z sgn(V 12 ×V 23 ·E Z1 );

[0105]

[0106] S24: Obtain hand capture rotation data according to the rotation values ​​obtained in steps S21 to S23.

[0107] In this embodiment, in step S3, the formula for splicing the hand capture rotation data and the human body motion data is:

[0108]

[0109] Among them, i,0≤i≤k is the subscript of each node in a path starting from the human body root node to the wrist target node, Q i is the rotation value of each node in the path, Q Ti is the initial rotation value of T-pose of each node in the path, Q wrist is the wrist target node rotation value. Specifically, since the human body is moving, the bones on the path from the human skeleton root node to the wrist node are moving, so it is necessary to keep the expression of the hand data in the body capture front view and the hand capture front view consistent when importing the hand rotation data to the human skeleton. Taking the left hand as an example, let {J root ,J 1 ,Λ,J k ,J wrist} is the path from the root node to the wrist node. For example, the human joints Root→Chest1→Chest2→Chest3→LeftCollar→LeftShoulder→LeftElbow→LeftWrist are a path from the root node to the left wrist node. Let Q i is the rotation value of each node in the path, Q Ti is the initial rotation value of T-pose of each node in the path, 0≤i≤k, i∈N, and then the above formula is used for splicing to obtain the new wrist joint rotation data for the splicing part of the body, while the rotation data of other finger joints remain unchanged.

[0110] In this embodiment, in step S4, the Euler angle representation of the data of the second joint point and the third joint point of the finger has data only in one dimension. Let {e 1 ,e 2 ,Λ,e f} represents the data of a certain joint point in the changed dimension, and the optimized data {E 1 ,E 2 ,Λ,E f}for:

[0111]

[0112] e 1 =e 1 ;

[0113] E f =e f ;

[0114] Assume {m 1 ,m 2 ,Λ,m f} represents the rotation matrix data of a joint point, {d 1 ,d 2 ,Λ,d f} represents 6d rotation data, where:

[0115] d i =((1 0 0)m i (0 1 0)m i ) 1≤i≤f;

[0116] Optimized 6d rotation data {D 1 ,D 2 ,Λ,D f}for:

[0117]

[0118] Convert the optimized 6d rotation data into rotation matrix data {M 1 ,M 2 ,Λ,M f}:

[0119]

[0120] Among them, I 3 is the 3D identity matrix, O (a,b) is a zero matrix with a rows and b columns.

[0121] Although the present invention has been described in detail with reference to the aforementioned embodiments, it is still possible for those skilled in the art to modify the technical solutions described in the aforementioned embodiments, or to make equivalent substitutions for some of the technical features therein. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A method for converting hand-captured 3D coordinate data into rotation data, characterized in that: The following steps are involved: S1: Capture motion on the video to obtain human motion data and 3D coordinate data of human hands; S2: converting the 3D coordinate data of the human hand into hand capture rotation data; S3: splicing the hand capture rotation data with the human body motion data; S4: Perform jitter optimization on the spliced ​​data.

2. The method for converting hand-captured 3D coordinate data into rotation data according to claim 1, characterized in that: The step S2 further includes the following steps: S21: Calculate the quaternion rotation value of the wrist joint point; S22: Calculate the rotation value of each finger root joint point; S23: Calculate the rotation value of the second joint point and the third joint point of each finger; S24: Obtain hand capture rotation data according to the rotation values ​​obtained in step S21 to step S23, whose dimension is an array of (f, j, 4), where f is the number of frames, j is the total number of joints of the left and right hands, and 4 is the rotation value of each joint represented by a quaternion.

3. The method for converting hand-captured 3D coordinate data into rotation data according to claim 2, characterized in that: The step S21 further includes the following steps: S21.1: Take vectors V1 and V2 in the 3D coordinate data and calculate a set of orthogonal vectors. Where V1 = P5-P 17 , V2=P5-P0, P5 is the finger node at point 5, P 17 is the finger node at point 17, and P0 is the finger node at point 0; S21.2: From the known initial value of the human skeleton T-pose, a set of initial orthogonal vectors T corresponding to the above are obtained X and T Y , calculate the rotation axis V of the quaternion Axis , S21.3: Calculate the rotation angle of the quaternion, Determine the rotation direction of the quaternion. s=sgn((T X ×V X )·V Axis ); When s=1, it indicates right-hand rotation, and when s=-1, it indicates left-hand rotation; S21.4: Calculate the rotation quaternion value R of the wrist joint wrist , 4. The method for converting hand-captured 3D coordinate data into rotation data according to claim 3, characterized in that: The step S22 further includes the following steps: S22.1: Known E X , E Y and E Z is the basis of the global coordinate system. After the wrist joint is rotated, the value of the basis of the local coordinates of the joint in the global coordinate system is calculated. Among them, I3 is the 3D identity matrix, O (a,b) is a zero matrix with a rows and b columns; S22.2: Calculate the rotation value in the y direction, Among them, V 01 is the direction of the bone at the base of the finger, V 01 =P 10 -P9, P9 is the ninth finger node, P 10 It is the finger node at point 10; Calculate the rotation value in the z direction, S22.3: Calculate the rotation value of the finger base joint point, 5. The method for converting hand-captured 3D coordinate data into rotation data according to claim 4, characterized in that: In step S23, the directions of the three bone segments of the finger are V 01 =P 10 -P9, V 12 =P 11 -P 10 , V 23 =P 12 -P 11 , calculate the rotation value of the second joint point of the finger: V Axis2 =E Z sgn(V 01 ×V 12 ·E Z1 ); Calculate the rotation value of the third joint of the finger: V Axis3 =E Z sgn(V 12 ×V 23 ·E Z1 ); 6. The method for converting hand-captured 3D coordinate data into rotation data according to claim 1, characterized in that: In step S3, the formula for splicing the hand capture rotation data and the human body motion data is: Among them, i,0≤i≤k is the subscript of each node in a path starting from the human body root node to the wrist target node, Q i is the rotation value of each node in the path, Q Ti is the initial rotation value of T-pose of each node in the path, Q wrist Rotate the wrist target node.

7. The method for converting hand-captured 3D coordinate data into rotation data according to claim 1, characterized in that: In step S4, the Euler angles of the second and third joint points of the finger only have data in one dimension. Let {e1, e2, Λ, e f } represents the data of a certain joint point in the changed dimension, and the optimized data {E1,E2,Λ,E f }for: E1=e1; AND f =and f ; Let {m1,m2,Λ,m f } represents the rotation matrix data of a joint point, {d1,d2,Λ,d f } represents 6d rotation data, where: d i =((100)m i (010)m i ) 1≤i≤f; The optimized 6d rotation data {D1,D2,Λ,D f }for: The optimized 6d rotation data is converted into rotation matrix data {M1,M2,Λ,M f }: Among them, I3 is the 3D identity matrix, O (a,b) is a zero matrix with a rows and b columns.

Images