Conversion method, conversion program, and conversion device

Abstract

This conversion device acquires: skeleton information in which a plurality of joints included in the human body, and coordinates corresponding to the plurality of joints, are each set; and standard posture information in which the plurality of joints included in the human body, and standard coordinates corresponding to the plurality of joints, are each set. The conversion device identifies, from among the plurality of joints set in the skeleton information, a second joint corresponding to a first joint set in the standard posture information. The conversion device converts the skeleton information into hierarchical structure data by calculating the relative rotation angle from the standard coordinates of the first joint to the coordinates of the second joint and setting the relative rotation angle in the hierarchical structure data.

Description

[Technical Field]

[0001] The present invention relates to a conversion method and the like. [Background technology]

[0002] Regarding the detection of three-dimensional human movement, 3D sensing technology has been established that uses multiple 3D laser sensors to detect a person's 3D skeletal coordinates with centimeter-order accuracy. This 3D sensing technology is expected to be applied to gymnastics judging support systems and to be expanded to other sports and fields. The method using 3D laser sensors is referred to as the laser method.

[0003] The laser method emits a laser approximately 2 million times per second, and calculates the depth and information of each irradiation point, including the target person, based on the laser's time of flight (ToF). The laser method can obtain highly accurate depth data, but has the disadvantage of requiring complex and expensive hardware due to the complex configuration and processing of the laser scan and ToF measurement.

[0004] Instead of the laser method, 3D skeletal recognition can also be performed using an image method, which uses a CMOS (Complementary Metal Oxide Semiconductor) imager to acquire RGB (Red, Green, Blue) data for each pixel, allowing the use of an inexpensive RGB camera.

[0005] Here, a conventional technique for 3D skeleton recognition using multiple cameras will be described. Fig. 18 is a diagram for explaining the conventional technique for 3D skeleton recognition. In the example shown in Fig. 18, cameras 30a and 30b capture images of a user U1. The images captured by cameras 30a and 30b are referred to as images 31a and 31b, respectively.

[0006] Images 31a and 31b are input to learning models 32a and 32b, respectively, which output 2D keypoints 33a and 33b. The learning models 32a and 32b are trained deep learning models or the like. The 2D keypoints 33a and 33b are two-dimensional skeletal information or the like.

[0007] In the prior art, 2D keypoints 33a and 33b are integrated to generate 3D keypoints 34. The 3D keypoints 34 are three-dimensional skeletal information, etc. For example, the three-dimensional coordinates of each joint of a human body model are set in the 3D keypoints 34.

[0008] Fig. 19 is a diagram showing an example of a human body model. As shown in Fig. 19, the human body model is defined by 21 joints ar0 to ar20. As for the 3D key points 34, three-dimensional coordinates of x, y, and z are set for each of the joints ar0 to ar20 defined in the human body model.

[0009] The relationship between the joints ar0 to ar20 shown in FIG. 19 and the joint names is as shown in FIG. 20. FIG. 20 is a diagram showing an example of joint names. For example, the joint name of joint ar0 is "SPINE_BASE." The joint names of joints ar1 to ar20 are as shown in FIG. 20, and explanations thereof will be omitted.

[0010] To improve the performance of image-based skeleton recognition, one method is to generate training data for a deep learning model from 3D keypoints and perform additional training.

[0011] In conventional techniques, since it is difficult to directly generate training data from 3D keypoints, the 3D keypoints are first converted into hierarchical structure data, and training data is generated based on the hierarchical structure data, including BVH (Biovision Hierarchy) data.

[0012] For example, BVH data consists of two sections: "HIERARCY" and "MOTION." First, HIERARCY defines the skeletal structure of the human body using multiple nodes and holds data on the reference posture. The multiple nodes that represent the skeletal structure of the human body include "ROOT," which indicates the reference joint node of the human body, "JOINT," which indicates the joint node, and "END," which indicates the node at the tip of the limbs. HIERARCY defines the order in which the nodes are connected, from the top-level ROOT to the bottom-level END. For adjacent nodes, a "joint direction vector" is set, pointing from the higher-level node to the lower-level node.

[0013] Fig. 21 is a diagram showing an example of the reference posture. In the example shown in Fig. 21, the reference posture is indicated by nodes rn, n0 to n20. Node rn is the ROOT. Nodes n0 to n2, n4 to n12, n14 to n16, and n18 are JOINTS. Nodes n3, n13, n17, n19, and n20 are Ends. The joint direction vectors between the nodes in the reference posture are defined as "OFFSETs."

[0014] On the other hand, MOTION includes the total number of frames and the time per frame, and includes information on the three-dimensional position of the ROOT, the three-dimensional rotation angle of the ROOT, and the rotation angle of each JOINT for each frame.

[0015] 22 is a diagram illustrating an example of a three-dimensional rotation angle of the ROOT. For example, the three-dimensional rotation angle of the ROOT is an Euler angle (θx, θy, θz) that transforms the ROOT node rn-1 in the global coordinate system to the ROOT node rn-2 in the ROOT coordinate system.

[0016] 23 is a diagram illustrating an example of the three-dimensional rotation angle of a JOINT. For example, the three-dimensional rotation angle of a JOINT is the Euler angles (θx, θy, θz) that convert from a higher-level JOINT coordinate system to a JOINT coordinate system of interest. For example, if the JOINT of interest is node n4, the node above node n4 is node n2. In this case, the higher-level JOINT coordinate system is the coordinate system of node n2. The JOINT coordinate system of interest is the coordinate system of node n4.

[0017] Here, the 3D keypoints are three-dimensional coordinate data, and since there is no information about the root or joint, it is not possible to simply calculate the three-dimensional rotation angle of the root and each joint from the 3D keypoints. For example, in conventional technology, the three-dimensional rotation angle of the root and each joint is calculated from the 3D keypoints based on inverse kinematics.

[0018] Inverse kinematics is a method for calculating the joint angles required for a part such as a hand to reach a target position given as an input in a multi-joint robot such as a manipulator.

[0019] FIG. 24 is a diagram for explaining inverse kinematics. As a representative example, inverse kinematics using the gradient method will be explained. In the example shown in FIG. 24, a target position tar is given. In inverse kinematics, a joint angle q is calculated that brings the error e between the target position tar and the hand position h1 closer to 0, and the step of repeatedly updating q is repeatedly executed.

[0020] For example, the processing procedure for obtaining the three-dimensional rotation angle of the ROOT of BVH data and the three-dimensional rotation angle of each JOINT from the 3D key points is the processing procedure shown in FIG.

[0021] Fig. 25 is a flowchart showing the processing procedure of the conventional technique. In Fig. 25, an apparatus that executes the processing of the conventional technique is referred to as a conventional apparatus. As shown in Fig. 25, the conventional apparatus acquires 3D key points (step S10).

[0022] The conventional device calculates the length of the bone based on the 3D keypoints (step S11). The conventional device describes the hierarchy (step S12). The conventional device calculates the three-dimensional coordinates of the root (step S13).

[0023] The conventional device calculates the three-dimensional rotation angles of the root and joint based on inverse kinematics (step S14), and describes the motion (step S15).

[0024] Next, a description will be given of the processing procedure for calculating the three-dimensional rotation angles of the ROOT and JOINT based on inverse kinematics, which was explained in step S14 of Fig. 25. Fig. 26 is a flowchart showing the processing procedure for calculating the three-dimensional rotation angles of the ROOT and JOINT based on inverse kinematics. As shown in Fig. 26, the conventional device accepts input of the joint position p_{tar} of the 3D keypoint and the initial value q_{src} of the joint position of the BVH data (step S20).

[0025] The conventional device calculates p_{src} from q_{src} using forward kinematics FK (step S21). The conventional device calculates the error e between the joint positions based on equation (1) (step S22).

[0026] e=|p_{tar}-p_{src}|···(1)

[0027] If the error e is smaller than ε (step S23, Yes), the conventional device proceeds to step S27. On the other hand, if the error e is not smaller than ε (step S23, No), the conventional device proceeds to step S24.

[0028] The processing from step S24 onwards will be described below: Conventional devices calculate the Jacobian matrix J as the velocity of the joint angle q(q_{src}) that reduces the error e (step S24).

[0029] The conventional device calculates the displacement Δq (Δq_{src}) that reduces the error from the Jacobian matrix J of the error e based on equation (2) (step S25).

[0030] Δq=-J -1 e···(2)

[0031] The conventional device updates q based on equation (3) (step S26) and proceeds to step S21.

[0032] q=q+Δq (3)

[0033] The processing from step S27 onwards will be described. The conventional device sets the value of q_{src} to the value of q_{tar} (step S27). The conventional device executes the processing of steps S21 to S27 described above for all joints (step S28). The conventional device outputs q_{tar} (step S29).

[0034] Fig. 27 is a diagram for providing additional explanation of the processing in Fig. 26. As shown in Fig. 27, when the joint positions p_{tar} of the 3D keypoints and the initial values ​​q_{src} of the joint angles of the BVH data are input and the processing of steps S20 to S29 in Fig. 26 is performed, the motion q_{tar} of the BVH data is output. [Prior art documents] [Patent documents]

[0035] [Patent Document 1] Japanese Patent Publication No. 2022-92528 Summary of the Invention [Problem to be solved by the invention]

[0036] However, the above-mentioned conventional technology has a problem in that the amount of calculation is large because 3D key points are converted into hierarchical structure data based on inverse kinematics.

[0037] For example, in the inverse kinematics process described with reference to FIG. 26, the processes of steps S21 to S26 are repeatedly executed until the error e becomes smaller than ε.

[0038] In one aspect, the present invention aims to provide a conversion method, a conversion program, and a conversion device that can reduce the amount of calculation required to convert 3D keypoints into hierarchical structure data. [Means for solving the problem]

[0039] In the first proposal, a computer is caused to execute the following process. The computer acquires skeletal information in which a plurality of joints included in a human body and coordinates for the plurality of joints are respectively set, and reference posture information in which a plurality of joints included in a human body and reference coordinates for the plurality of joints are respectively set. The computer identifies a second joint corresponding to a first joint set in the reference posture information from the plurality of joints set in the skeletal information. The computer calculates a relative rotation angle from the reference coordinate of the first joint to the coordinate of the second joint, and sets the relative rotation angle in the hierarchical structure data, thereby converting the skeletal information into hierarchical structure data. [Effects of the Invention]

[0040] The amount of calculation required to convert 3D keypoints into hierarchical structure data can be reduced. [Brief explanation of the drawings]

[0041] [Figure 1] FIG. 1 is a diagram for explaining the process of calculating the three-dimensional rotation angle of the ROOT by rigid body registration. [Figure 2] FIG. 2 is a diagram (1) for explaining the process of calculating the three-dimensional rotation angle of a JOINT using the Rodrigues rotation formula. [Figure 3] FIG. 3 is a diagram (2) for explaining the process of calculating the three-dimensional rotation angle of a JOINT using the Rodrigues rotation formula. [Figure 4] FIG. 4 is a functional block diagram showing the configuration of the conversion device according to this embodiment. [Figure 5] FIG. 5 is a diagram illustrating an example of the data structure of the 3D keypoint table. [Figure 6]FIG. 6 is a diagram illustrating an example of the data structure of the hierarchical structure data. [Figure 7] FIG. 7 is a diagram illustrating an example of the data structure of BVH data. [Figure 8] FIG. 8 is a diagram illustrating an example of model data. [Figure 9] FIG. 9 is a diagram illustrating an example of the data structure of the correction angle dictionary. [Figure 10] FIG. 10 is a diagram for explaining the processing of the correction execution unit. [Figure 11] FIG. 11 is a diagram for explaining the processing of the generation unit. [Figure 12] FIG. 12 is a flowchart illustrating a processing procedure of the conversion device according to the present embodiment. [Figure 13] FIG. 13 is a flowchart showing the processing procedure of the first calculation processing. [Figure 14] FIG. 14 is a flowchart showing the processing procedure of the second calculation process. [Figure 15] FIG. 15 is a flowchart showing the procedure of the correction process. [Figure 16] FIG. 16 is a diagram for explaining the effect of the conversion device according to this embodiment. [Figure 17] FIG. 17 is a diagram illustrating an example of a hardware configuration of a computer that realizes the same functions as the conversion device of the embodiment. [Figure 18] FIG. 18 is a diagram for explaining a conventional technique related to 3D skeleton recognition. [Figure 19] FIG. 19 is a diagram showing an example of a human body model. [Figure 20] FIG. 20 is a diagram showing an example of joint names. [Figure 21] FIG. 21 is a diagram showing an example of the reference attitude. [Figure 22] FIG. 22 is a diagram for explaining an example of a three-dimensional rotation angle of the ROOT. [Figure 23] FIG. 23 is a diagram for explaining an example of a three-dimensional rotation angle of a JOINT. [Figure 24]FIG. 24 is a diagram for explaining inverse kinematics. [Figure 25] FIG. 25 is a flowchart showing the processing procedure of the conventional technology. [Figure 26] FIG. 26 is a flowchart showing the processing procedure for calculating the three-dimensional rotation angles of the root and joint based on inverse kinematics. [Figure 27] FIG. 27 is a diagram for providing a supplementary explanation of the processing in FIG. DETAILED DESCRIPTION OF THE INVENTION

[0042] Hereinafter, embodiments of the conversion method, conversion program, and conversion device disclosed in the present application will be described in detail with reference to the accompanying drawings. However, the present invention is not limited to these embodiments. [Example]

[0043] The processing of the conversion device according to this embodiment will be described. The conversion device focuses on the fact that the reference orientation of the hierarchical structure data is known. The explanation of the reference orientation of the hierarchical structure data is the same as that explained in FIG. 21. The conversion device calculates the relative rotation angle from the reference orientation to the 3D keypoints, thereby eliminating repeated calculations when converting the 3D keypoints into BVH data and reducing the amount of calculation.

[0044] For example, the transformation device calculates the 3D rotation angle of the ROOT by rigid body registration, and the transformation device calculates the 3D rotation angle of the JOINT by Rodrigues' rotation formula.

[0045] First, an example of the process in which the conversion device calculates the three-dimensional rotation angle of the ROOT by rigid body alignment will be described. FIG. 1 is a diagram for explaining the process of calculating the three-dimensional rotation angle of the ROOT by rigid body alignment. The conversion device divides each node of the hierarchical structure data (data of the reference posture) into three or more rigid joint nodes and other nodes according to a predefinition. As described in FIG. 21, the nodes of the reference posture are nodes rn, n0 to n20. In the example shown in FIG. 1, the conversion device selects nodes n0, n10, and n14 from the nodes included in the reference posture 40 as rigid joint nodes.

[0046] The conversion device identifies a group of joints corresponding to rigid joint nodes (nodes n0, n10, and n14) from each joint included in the 3D keypoints 41. In the example shown in FIG. 1, the group of joints corresponding to rigid joint nodes are joints ar0, ar10, and ar14. In the following description, among the joints included in the 3D keypoints, the group of joints corresponding to rigid joint nodes will be referred to as "rigid-body corresponding joints."

[0047] The conversion device calculates the relative rotation angle from the rigid joint nodes (nodes n0, n10, n14) to the rigid corresponding joints (joints ar0, ar10, ar14) by rigid body registration, and sets it as the 3D rotation angle of the ROOT.

[0048] Rigid body registration is a method for finding transformation parameters for aligning a source with a target using a combination of three or more points between the source and target using the least squares method according to equation (4). The transformation parameters include a rotation matrix R, a translation t, and a scale c. In this example, the rotation matrix R is converted into Euler angles and used as the 3D rotation angles of ROOT.

[0049]

number

[0050] In equation (4), "x" is the coordinate of three or more sources. The coordinate of the source is the three-dimensional coordinate of the rigid joint. "y" is the coordinate of three or more targets. The coordinate of the target is the three-dimensional coordinate of the rigid joint. The conversion device uses e in equation (4). 2 Find the rotation matrix R, translation t, and scale c that minimize the following.

[0051] Next, an example of a process in which the conversion device calculates the three-dimensional rotation angle of a joint using Rodrigues' rotation formula will be described. FIGS. 2 and 3 are diagrams for explaining the process of calculating the three-dimensional rotation angle of a joint using Rodrigues' rotation formula. First, FIG. 2 will be described. The conversion device converts the joint direction vector of the target joint of the 3D key point 41 into a joint direction vector in the local coordinate system. The joint direction vector of the 3D key point 41 is a vector directed from a lower joint to a higher joint among adjacent joints.

[0052] 2, the joint of interest is assumed to be joint ar0. The joint direction vector v_{tar} of joint ar0 is a vector directed from the lower joint ar0 to the higher joint ar1.

[0053] The conversion device applies the inverse matrix (R -1 ) to convert it into a joint direction vector in the local coordinate system. For example, multiplying the joint direction vector v_{tar} of the target joint by R -1 The transformation device repeatedly executes the above process for each joint of the 3D key point 41 to obtain a 3D key point 42 in the local coordinate system.

[0054] Next, the conversion device identifies the angle θ between the joint direction vector of the 3D key point 42 and the joint direction vector of the reference posture 40, with the normal to the joint direction vector of the 3D key point 42 and the joint direction vector of the reference posture 40 as the rotation axis. Here, as an example, a description will be given using the joint direction vector v_{tar_local} of the 3D key point 42 and the joint direction vector v_{src} of the reference posture 40. The joint direction vector v_{src} is a vector directed from node n0 of the reference posture to node n1.

[0055] We now move on to the explanation of Figure 3. In Figure 3, n is the normal between the joint direction vector v_{tar_local} and the joint direction vector v_{src} of the reference posture 40. The conversion device identifies the normal n by calculating the cross product of the joint direction vector v_{tar_local} and the joint direction vector v_{src}. The conversion device uses the normal n as the rotation axis to identify the angle θ between the joint direction vector v_{tar_local} and the joint direction vector v_{src}.

[0056] The conversion device uses the normal n as the rotation axis and the angle θ as the rotation angle, calculates the relative rotation angle using Rodrigues' rotation formula, and uses it as the three-dimensional rotation angle of the JOINT.

[0057] Rodrigues' rotation formula is a formula for calculating the rotation matrix R according to equation (5) from the rotation axis (normal n) and rotation angle (angle θ) specified by the source and target of the joint direction vector. The source of the joint direction vector is the joint direction vector (v_{src}) of the reference posture 40. The target of the joint direction vector is the joint direction vector (v_{tar_local}) of the 3D key point 42 in the local coordinate system.

[0058]

number

[0059] The conversion device is R calculated by Equation (5). n(θ) is converted into an Euler angle and used as the 3D rotation angle of the JOINT.

[0060] As described above, the conversion device according to this embodiment can reduce the amount of calculation by calculating the relative rotation angle from the reference posture of the hierarchical structure data to the 3D keypoint. For example, the conversion device calculates the 3D rotation angle of the ROOT by rigid body alignment. Also, the conversion device calculates the 3D rotation angle of the JOINT by the Rodrigues rotation formula.

[0061] Next, a description will be given of an example of the configuration of a conversion device that executes the processes described in Figures 1 to 3. Figure 4 is a functional block diagram showing the configuration of a conversion device according to this embodiment. As shown in Figure 4, this conversion device 100 has a communication unit 110, an input unit 120, a display unit 130, a storage unit 140, and a control unit 150.

[0062] The communication unit 110 executes data communication with an external device or the like via a network. The communication unit 110 is a NIC (Network Interface Card) or the like. The control unit 150, which will be described later, exchanges data with the external device via the communication unit 110.

[0063] The input unit 120 is an input device that inputs various pieces of information to the control unit 150 of the conversion device 100. For example, the input unit 120 corresponds to a keyboard, a mouse, a touch panel, or the like.

[0064] The display unit 130 is a display device that displays information output from the control unit 150.

[0065] The storage unit 140 includes a 3D keypoint table 141, hierarchical structure data 142, and a training data table 144. The storage unit 140 is a storage device such as a memory.

[0066] The 3D keypoint table 141 is a table that holds information about 3D keypoints. Fig. 5 is a diagram showing an example of the data structure of the 3D keypoint table. As shown in Fig. 5, the 3D keypoint table 141 holds a frame number and three-dimensional coordinates corresponding to each joint (joint identification information) in association with each other.

[0067] The frame number is a frame number that identifies the image used when generating the 3D keypoints. The identification information of each joint is information that uniquely identifies the joint. In the explanation of Figure 5, joints ar0 to ar20 are used as the identification information of the joints. For example, joint ar0 corresponds to "SPINE_BASE". joint ar1 corresponds to "SPINE_MID". joint ar20 corresponds to "HAND_TIP_RIGHT". The relationship between the other joints and the joint names is shown in Figure 20.

[0068] The three-dimensional coordinates of the joints ar0 to ar20 corresponding to a certain frame number (for example, 0001) in the 3D key point table 141 are the 3D key points corresponding to the certain frame number (for example, 0001).

[0069] The hierarchical structure data 142 is data obtained by converting 3D key points. Fig. 6 is a diagram for explaining an example of the data structure of the hierarchical structure data. As shown in Fig. 6, the hierarchical structure data 142 includes BVH data 50 and model data 60.

[0070] Fig. 7 is a diagram showing an example of the data structure of BVH data. As shown in Fig. 7, the BVH data includes a HIERARCY 51 and a MOTION 52. The HIERARCY 51 includes node definition information 51a, hierarchical structure definition information 51b, reference attitude information 51c, and channel information 51d.

[0071] The node definition information 51a is information that defines which node is a ROOT, which node is a JOINT, and which node is an END among the nodes rn and n0 to n20 shown in Fig. 21. For example, node rn is a ROOT. Nodes n0 to n2, n4 to n12, n14 to n16, and n18 are JOINT. Nodes n3, n13, n17, n19, and n20 are END.

[0072] The hierarchical structure definition information 51b is information that defines the connection order of nodes from the top-level ROOT to the bottom-level END.

[0073] The reference posture information 51c is information on the reference posture explained in Fig. 21. For example, the reference posture information 51c defines the joint direction vector of each node that is the reference posture.

[0074] The channel information 51d is information that defines data to be registered in motion data 52c of each frame, which will be described later.

[0075] MOTION 52 includes a frame count 52a, a frame time 52b, and motion data 52c. The frame count 52a indicates the total number of frames. The frame time 52b indicates the time per frame.

[0076] The motion data 52c is registered as information for one frame, including the three-dimensional position of the ROOT, the three-dimensional rotation angle of the ROOT, and the three-dimensional rotation angle of each JOINT. One piece of motion data 52c is generated from one 3D key point.

[0077] 8 is a diagram showing an example of model data 60. For example, the model data 60 includes SMPL (Skinned Multi-Person Linear Model) 61a, MakeHuman 61b, Autodesk Character Generator 61c, and the like.

[0078] SMPL61a is a 3D body CG (Computer Graphics) model trained with 3D height scan data and capable of adjusting posture and body shape. MakeHuman61b is a 3D body CG model capable of adjusting gender, height, body shape, race, etc. Autodesk Character Generator61c is a 3D body CG model capable of adjusting details such as facial expression, muscles, hair, and style.

[0079] Returning to the explanation of Fig. 4, the correction angle dictionary 143 is a dictionary that holds information for correcting the three-dimensional rotation angle of a JOINT. Fig. 9 is a diagram showing an example of the data structure of the correction angle dictionary. As shown in Fig. 9, the correction angle dictionary 143 associates frame numbers, labels, and correction angles.

[0080] The frame number is the frame number that identifies the image used to generate the 3D keypoints, and is also information that identifies the 3D keypoints. The label is set to a label such as "normal grip" or "reverse grip." If the label is "none," this means that there is no corresponding correction angle. The correction angle is set to the correction angle for each joint. The 3D rotation angle of the joint is corrected based on this correction angle.

[0081] The training data table 144 holds training data generated based on the hierarchical structure data 142. The training data is data in which an image is used as an input and joint coordinates are used as a correct label. The training data is used when training a machine learning model such as a neural network (NN).

[0082] Returning to the explanation of Fig. 4, the control unit 150 includes an acquisition unit 151, a first calculation unit 152, a second calculation unit 153, a correction execution unit 154, a setting unit 155, a generation unit 156, and a learning unit 157. The control unit 150 is a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), or the like.

[0083] The acquisition unit 151 acquires information about the 3D key point table 141, information about the correction angle dictionary 143, etc. from an external device or the like via a network. The acquisition unit 151 stores the acquired information about the 3D key point table 141 and information about the correction angle dictionary 143 in the storage unit 140.

[0084] The first calculation unit 152 acquires 3D key points from the 3D key point table 141 and acquires the reference posture from the hierarchical structure data 142. It is assumed that the 3D key points are assigned frame numbers. The first calculation unit 152 calculates the three-dimensional rotation angle of the ROOT by rigid body alignment based on the reference posture and the 3D key points. The process by which the first calculation unit 152 calculates the three-dimensional rotation angle of the ROOT by rigid body alignment corresponds to the process described in FIG. 1. The first calculation unit 152 converts the three-dimensional rotation angle of the ROOT into Euler angles using a predetermined conversion formula.

[0085] The first calculation unit 152 calculates the length of the bone between the joints based on the 3D key points, and calculates it as the three-dimensional coordinate of the ROOT.

[0086] The first calculation unit 152 outputs the 3D key points, the reference posture, and the calculation results (three-dimensional rotation angles of the ROOT <rotation matrix R, Euler angles>, three-dimensional coordinates of the ROOT) to the second calculation unit 153.

[0087] The second calculation unit 153 calculates the three-dimensional rotation angle of each JOINT using the Rodrigues rotation formula based on the 3D keypoints, the reference posture, and the three-dimensional rotation angle of the ROOT <rotation matrix R>. The process by which the second calculation unit 153 calculates the three-dimensional rotation angle of each JOINT using the Rodrigues rotation formula corresponds to the process described with reference to Figures 2 and 3. The second calculation unit 153 converts the three-dimensional rotation angle of each JOINT into Euler angles using a predetermined conversion formula.

[0088] The second calculation unit 153 outputs the 3D key points, the three-dimensional rotation angle of the ROOT (rotation matrix R, Euler angles), the calculation results (the three-dimensional rotation angle of each JOINT (rotation matrix R, Euler angles), and the three-dimensional coordinates of the ROOT to the correction execution unit 154.

[0089] The correction execution unit 154 corrects the three-dimensional rotation angle (Euler angle) of each JOINT based on the correction angle dictionary 143. FIG. 10 is a diagram for explaining the processing of the correction execution unit. The BVH data 70 in FIG. 10 is data in which the three-dimensional rotation angle (before correction) of each JOINT is applied to the reference posture. Here, information on the correction angles of each joint in the correction angle dictionary 143 that correspond to the frame number of the 3D key point is referred to as correction angle information 143-1. In the correction angle information 143-1, the correction angle of joint ar5 is "-90°" and the correction angle of joint ar8 is "+90°".

[0090] The correction execution unit 154 corrects the three-dimensional rotation angle of node n8, which corresponds to joint ar5, among the nodes (joints) of the BVH data 70, by a correction angle of "-90°." Furthermore, the correction execution unit 154 corrects the three-dimensional rotation angle of node n8, which corresponds to joint ar8, among the nodes (joints) of the BVH data 70, by a correction angle of "+90°." The correction execution unit 154 performs this correction on the BVH data 70, thereby generating BVH data 71. The rotation angles corresponding to joints ar5 and ar8 of the BVH data 70 do not match the rotation angles of the posture 75 of the actual person, but the rotation angles corresponding to joints ar5 and ar8 of the BVH data 71 do match the rotation angles of the posture 75 of the actual person.

[0091] The correction execution unit 154 corrects the three-dimensional rotation angle of the JOINT using the following procedure. The correction execution unit 154 converts the three-dimensional rotation angle of the JOINT before correction (in the form of Euler angles) into a three-dimensional rotation angle (in the form of rotation matrix R). The correction execution unit 154 calculates a correction rotation matrix R_{correct} of the correction angle from the direction of the joint of interest and the correction angle. The correction execution unit 154 calculates the corrected three-dimensional rotation angle (in the form of rotation matrix R) by multiplying the three-dimensional rotation angle before correction (in the form of rotation matrix R) by the correction rotation matrix R_{correct}.

[0092] The correction execution unit 154 restores the corrected three-dimensional rotation angle (in the format of the rotation matrix R) to the corrected three-dimensional rotation angle (in the format of the Euler angles), and updates the three-dimensional rotation angle of the JOINT.

[0093] If the correction angle of each joint corresponding to the frame number of the 3D key point is "none," the correction execution unit 154 interpolates the correction angle based on the correction angles of the previous and next frame numbers. For example, the correction execution unit 154 performs spherical linear interpolation. The correction execution unit 154 uses the interpolated correction angle to correct the three-dimensional rotation angle of the JOINT as described above.

[0094] The correction execution unit 154 outputs the 3D key points, the three-dimensional rotation angle (in Euler angle format) of the ROOT, the three-dimensional rotation angle (in Euler angle format) of each JOINT, and the three-dimensional coordinates of the ROOT to the setting unit 155.

[0095] The first calculation unit 152, the second calculation unit 153, and the correction execution unit 154 described above repeatedly execute the above process for each 3D key point.

[0096] The setting unit 155 sets various information in the hierarchical structure data 142. For example, the setting unit 155 writes information on HIERARCY 51 and MOTION 52 in the hierarchical structure data 142. The setting unit 155 writes HIERARCY 51 based on information prepared in advance. The setting unit 155 writes the number of frames 52a and frame time 52b of MOTION 52 based on information prepared in advance. The setting unit 155 writes motion data 52c based on the calculation results of the first calculation unit 152, the second calculation unit 153, and the correction execution unit 154.

[0097] The generation unit 156 generates learning data based on the hierarchical structure data 142. Fig. 11 is a diagram for explaining the processing of the generation unit. The generation unit 156 generates BVH data 80 by applying the three-dimensional rotation angle (in the form of Euler angles) of the ROOT and the three-dimensional rotation angle (in the form of Euler angles) of each JOINT to the reference posture of the hierarchical structure data 142.

[0098] The generation unit 156 generates a person model 81 based on the BVH data 80 and the model data 60. The generation unit 156 sets virtual cameras 81a and 81b. The generation unit 156 uses the virtual cameras 81a and 81b to generate images from virtual viewpoints of the model 81. For example, the generation unit 156 generates learning data 82 by associating an image 82a captured by the virtual camera 81a with joint coordinates 82b of the person model 81.

[0099] The generating unit 156 generates a plurality of pieces of training data by repeatedly executing the above process, and registers the generated training data in the training data table 144.

[0100] The learning unit 157 trains the learning model based on the learning data in the learning data table 144. For example, the learning unit 157 updates parameters based on the backpropagation algorithm so as to reduce the error between the output obtained by inputting an image into the learning model and the correct label (joint coordinate).

[0101] Next, an example of a processing procedure of the conversion device 100 according to this embodiment will be described. Fig. 12 is a flowchart showing the processing procedure of the conversion device according to this embodiment. As shown in Fig. 12, the first calculation unit 152 of the conversion device 100 acquires 3D key points from the 3D key point table 141 (step S101). The first calculation unit 152 calculates the length of the bone (step S102).

[0102] The setting unit 155 of the conversion device 100 describes the HIERARCY 51 of the hierarchical structure data 142 (step S103). The first calculation unit 152 calculates the three-dimensional coordinates of the ROOT (step S104). The first calculation unit 152 executes the first calculation process (calculation process of the ROOT three-dimensional rotation angle by rigid body alignment) (step S105).

[0103] The second calculation unit 153 of the conversion device 100 executes the second calculation process (calculation process of the three-dimensional rotation angle of the JOINT using the Rodrigues rotation formula) (step S106). The correction execution unit 154 of the conversion device 100 executes the correction process (step S107).

[0104] The setting unit 155 describes MOTION52 of the hierarchical structure data 142 based on the calculation result (step S108).

[0105] Next, a description will be given of the processing procedure of the first calculation process shown in step S105 of Fig. 12. Fig. 13 is a flowchart showing the processing procedure of the first calculation process. As shown in Fig. 13, the first calculation unit 152 of the conversion device 100 classifies each node of the BVH data into three or more rigid joints q_{src_rb} and other joints (step S201).

[0106] The first calculation unit 152 extracts the rigid joint q_{src_rb} and the corresponding joint group p_{tar_rb} of the 3D keypoint (step S202). The first calculation unit 152 calculates the relative rotation angle R_{global} by rigid body registration based on the rigid joint q_{src_rb} and the joint group p_{tar_rb} of the 3D keypoint (step S203).

[0107] The first calculation unit 152 outputs the relative rotation angle R_{global} as the three-dimensional rotation angle of the ROOT (step S204).

[0108] Next, a description will be given of the processing procedure of the second calculation process shown in step S106 of Fig. 12. Fig. 14 is a flowchart showing the processing procedure of the second calculation process. As shown in Fig. 14, the second calculation unit 153 of the transformation device 100 sets the number of joints N=21, the joint index i=1, and the local coordinate transformation matrix R=R_{global} (step S301).

[0109] The second calculation unit 153 acquires i_p as the index of the parent joint of the joint index i based on the formula (6) (step S302). parent is a data array in which joint indices are associated with the indices of their parent joints, and when a joint index i is input, the index of the parent joint is output as parent(i).

[0110] i_p=parent(i) (6)

[0111] The second calculation unit 153 calculates the joint direction vector v_{tar} of the joint index i from the joint coordinates joint(i) and joint(i_p) based on the formula (7) (step S303). "joint" is a data array in which the joint index and its joint coordinates are associated with each other, and when the joint index i is input, the joint coordinate joint(i) is output.

[0112] v_{tar}=joint(i)-joint(i_p)···(7)

[0113] The second calculation unit 153 calculates v_{tar_local}, which is the joint direction vector in the local coordinate system, from the joint direction vector v_{tar} and the local coordinate transformation matrix R based on equation (8) (step S304).

[0114] v_{tar_local}=R^-1*v_{tar}···(8)

[0115] The second calculation unit 153 calculates the joint direction vector v_{src} from the reference posture of the BVH data (step S305). The second calculation unit 153 calculates the relative rotation angle Ri from v_{src} to v_{tar_local} using Rodrigues' rotation formula, and sets this as the three-dimensional rotation angle of the JOINT with joint index i (step S306).

[0116] If the joint index i is less than N (Yes at step S307), the second calculation unit 153 outputs the three-dimensional rotation angle R_{local}=(R1, R2, . . . R21) of each JOINT (step S308).

[0117] On the other hand, if the joint index i is not less than N (step S307, No), the second calculation unit 153 adds 1 to the joint index i (step S309). The second calculation unit 153 updates the local coordinate transformation matrix R by the value obtained by multiplying the local coordinate transformation matrix R by the relative rotation angle Ri (step S310), and proceeds to step S302.

[0118] Next, the procedure for the correction process shown in step S107 in Fig. 12 will be described. Fig. 15 is a flowchart showing the procedure for the correction process. As shown in Fig. 15, the correction execution unit 154 of the conversion device 100 reads one line from the correction angle dictionary 143 (step S401). If there is no label in the read line (step S402, No), the correction execution unit 154 proceeds to step S404.

[0119] If the read line has a label (Yes at step S402), the correction execution unit 154 acquires the correction angle of each joint from the correction angle dictionary 143 (step S403).

[0120] If the correction execution unit 154 has not read all the rows of the correction angle dictionary 143 (step S404, No), the correction execution unit 154 proceeds to step S401. On the other hand, if the correction execution unit 154 has read all the rows of the correction angle dictionary 143 (step S404, Yes), the correction execution unit 154 proceeds to step S405.

[0121] The correction execution unit 154 reads one unlabeled line from the correction angle dictionary 143 (step S405). The correction execution unit 154 determines whether or not a correction angle exists in the lines immediately before and after the read line (step S406). The correction execution unit 154 interpolates a correction angle based on the correction angles in the lines immediately before and after (step S407).

[0122] If the correction execution unit 154 has not read all of the unlabeled rows in the correction angle dictionary 143 (step S408, No), the correction execution unit 154 proceeds to step S405. On the other hand, if the correction execution unit 154 has read all of the unlabeled rows in the correction angle dictionary 143 (step S408, Yes), the correction execution unit 154 proceeds to step S409.

[0123] The correction execution unit 154 reads the correction angle of each joint corresponding to the current frame number and calculates the correction rotation matrix R_{correct} (step S409). The correction execution unit 154 updates the three-dimensional rotation angle R_{local} of each joint based on equation (9) (step S410).

[0124] R_{local_corrected}=R_{correct}*R_{local}···(9)

[0125] The correction execution unit 154 sets R_{local_corrected} as the three-dimensional rotation angle of the JOINT (step S411).

[0126] Next, the effects of the conversion device 100 according to this embodiment will be described. The conversion device 100 acquires 3D key points and a reference posture of the hierarchical structure data 142, calculates a relative rotation angle from a predetermined joint in the reference posture to a corresponding joint of the 3D key point, and sets the relative rotation angle in the hierarchical structure data 142. This reduces the amount of calculation and enables generation of hierarchical structure data from 3D key points.

[0127] For example, the transformation device 100 calculates the relative rotation angles from the rigid joint nodes of the reference posture to the joints of the 3D key points corresponding to the rigid joint nodes by rigid registration, thereby enabling the 3D rotation angles of the ROOT to be calculated efficiently without repeated calculations.

[0128] The conversion device 100 converts the joint direction vectors of the 3D keypoints into joint direction vectors in the local coordinate system. The conversion device 100 identifies the angle θ between each joint direction vector, with the normal to the joint direction vector of the 3D keypoint and the joint direction vector of the reference posture as the rotation axis, calculates the relative rotation angle using Rodrigues' rotation formula, and uses it as the 3D rotation angle of the JOINT. This makes it possible to efficiently calculate the 3D rotation angle of each JOINT without performing repeated calculations.

[0129] In addition, by calculating the relative rotation angle using Rodrigues' rotation formula, the three-dimensional rotation angle of the JOINT on the shortest path can be determined, preventing twisting rotation angles that seem unnatural to humans.

[0130] 16 is a diagram for explaining the effect of the conversion device according to this embodiment. For example, the twist rotation angle is the rotation angle around bone 90a shown in image 90. Human body model 91 is a human body model generated using the result of converting 3D keypoints into hierarchical structure data using conventional technology. Human body model 92 is a human body model generated using the result of converting 3D keypoints into hierarchical structure data by the conversion device 100 according to this embodiment.

[0131] The human body model 91 has an unnatural twisting rotation angle that twists the joints, but the human body model 92 does not have an unnatural twisting rotation angle.

[0132] Furthermore, the conversion device 100 according to this embodiment corrects the three-dimensional rotation angle of the JOINT based on the information on the correction angle in the correction angle dictionary 143. This makes it possible to obtain a twist rotation angle that is consistent with the actual twist rotation, which is difficult to obtain using conventional techniques.

[0133] Next, a description will be given of an example of the hardware configuration of a computer that realizes the same functions as the above-described conversion device 100. Fig. 17 is a diagram showing an example of the hardware configuration of a computer that realizes the same functions as the conversion device of the embodiment.

[0134] 17, computer 200 has CPU 201 that executes various types of arithmetic processing, input device 202 that accepts data input from a user, and display 203. Computer 200 also has communication device 204 that exchanges data with camera 15, external devices, etc. via a wired or wireless network, and interface device 205. Computer 200 also has RAM 206 that temporarily stores various types of information, and hard disk device 207. Each of devices 201 to 207 is connected to bus 208.

[0135] The hard disk drive 207 stores an acquisition program 207a, a first calculation program 207b, a second calculation program 207c, a correction execution program 207d, a setting program 207e, a generation program 207f, and a learning program 207g. The CPU 201 reads out each of the programs 207a to 207f and loads them into the RAM 206.

[0136] The acquisition program 207a functions as an acquisition process 206a. The first calculation program 207b functions as a first calculation process 206b. The second calculation program 207c functions as a second calculation process 206c. The correction execution program 207d functions as a correction execution process 206d. The setting program 207e functions as a setting process 206e. The generation program 207f functions as a generation process 206f. The learning program 207g functions as a learning process 206g.

[0137] The processing of the acquisition process 206a corresponds to the processing of the acquisition unit 151. The processing of the first calculation process 206b corresponds to the processing of the first calculation unit 152. The processing of the second calculation process 206c corresponds to the processing of the second calculation unit 153. The processing of the correction execution process 206d corresponds to the processing of the correction execution unit 154. The processing of the setting process 206e corresponds to the processing of the setting unit 155. The processing of the generation process 206f corresponds to the processing of the generation unit 156. The processing of the learning process 206g corresponds to the processing of the learning unit 157.

[0138] It should be noted that each of the programs 207a to 207g does not necessarily have to be stored in the hard disk drive 207 from the beginning. For example, each of the programs may be stored in a "portable physical medium" such as a flexible disk (FD), CD-ROM, DVD, magneto-optical disk, or IC card that is inserted into the computer 200. Then, the computer 200 may read and execute each of the programs 207a to 207g. [Explanation of symbols]

[0139] 100 Conversion Device 110 Communications Department 120 Input section 130 Display section 140 Storage section 141 3D Keypoint Table 142 Hierarchical structure data 143 Correction Angle Dictionary 144 Training Data Table 150 control section 151 Acquisition Department 152 1st calculation section 153 2nd calculation section 154 Correction execution unit 155 Setting section 156 Generation part 157 Learning Department

Claims

1. Acquire skeletal information in which a plurality of joints included in the human body and coordinates for the plurality of joints are respectively set, and reference posture information in which a plurality of joints included in the human body and reference coordinates for the plurality of joints are respectively set, Identifying a second joint corresponding to a first joint set in the reference posture information from the plurality of joints set in the skeleton information; Calculating a relative rotation angle from a reference coordinate of the first joint to a coordinate of the second joint; The relative rotation angle is set to the hierarchical structure data, thereby converting the skeleton information into the hierarchical structure data. A conversion method characterized in that the processing is performed by a computer.

2. The transformation method according to claim 1 , wherein the calculation process calculates relative rotation angles from reference coordinates of the first joints to coordinates of the second joints by rigid body alignment.

3. 3. The method according to claim 2, wherein the calculating process further calculates the relative rotation angle by transforming the second vector of the second joint based on the calculation result of the rigid body alignment, and applying Rodriguez's rotation formula to an angle between the first vector and the second vector, with a normal to the first vector of the first joint and the transformed second vector of the second joint as a rotation axis.

4. 4. The method according to claim 3, further comprising the step of correcting the relative rotation angle based on a correction angle set in the skeleton information.

5. Acquire skeletal information in which a plurality of joints included in the human body and coordinates for the plurality of joints are respectively set, and reference posture information in which a plurality of joints included in the human body and reference coordinates for the plurality of joints are respectively set, Identifying a second joint corresponding to a first joint set in the reference posture information from the plurality of joints set in the skeleton information; Calculating a relative rotation angle from a reference coordinate of the first joint to a coordinate of the second joint; The relative rotation angle is set to the hierarchical structure data, thereby converting the skeleton information into the hierarchical structure data. A conversion program that causes a computer to execute a process.

6. Acquire skeletal information in which a plurality of joints included in the human body and coordinates for the plurality of joints are respectively set, and reference posture information in which a plurality of joints included in the human body and reference coordinates for the plurality of joints are respectively set, Identifying a second joint corresponding to a first joint set in the reference posture information from the plurality of joints set in the skeleton information; Calculating a relative rotation angle from a reference coordinate of the first joint to a coordinate of the second joint; The relative rotation angle is set to the hierarchical structure data, thereby converting the skeleton information into the hierarchical structure data. A conversion device having a control unit that executes processing.

Images