Conversion method, conversion program, and conversion device

Abstract

This conversion device acquires skeleton information in which a plurality of joints included in a human body and coordinates for the plurality of joints are set, and reference posture information in which a plurality of joints included in the human body and reference coordinates for the plurality of joints are set. The conversion device specifies, from the plurality of joints set in the bone information, a second joint corresponding to the first joint set in the reference posture information. The conversion means calculates a relative rotation angle from a reference coordinate of the first joint to a coordinate of the second joint, and sets the relative rotation angle in the hierarchical structure data, thereby converting the bone information into the hierarchical structure data.

Description

Technical Field

[0001] The present invention relates to conversion methods and the like. Background Art

[0002] Regarding the detection of three-dimensional human movements, a 3D sensing detection technology has been established that detects the 3D bone coordinates of a person with centimeter-level accuracy using multiple 3D laser sensors. This 3D sensing detection technology is expected to be applied to gymnastics scoring support systems and to expand to other sports and other fields. The method using a 3D laser sensor is denoted as a laser method.

[0003] In the laser method, the laser is irradiated about 2 million times per second, and the depth and information of each irradiation point, including the person being the subject, are obtained based on the laser's movement time (Time of Flight: ToF). The laser method can obtain high-precision depth data, but the structure and processing of laser scanning and ToF measurement are complex, so there are disadvantages that the hardware becomes complicated and expensive.

[0004] Instead of the laser method, 3D bone recognition can also be performed using an imaging method. The imaging method uses a CMOS (Complementary Metal Oxide Semiconductor) image to obtain RGB (Red Green Blue) data for each pixel, and a cheap RGB camera can be used.

[0005] Here, the prior art related to 3D skeleton recognition using multiple cameras is described. Fig.18 FIG. 1 is a diagram for explaining the prior art related to 3D skeleton recognition. Fig.18 In the example shown, the cameras 30a and 30b capture images of the user U1. The images captured by the cameras 30a and 30b are referred to as images 31a and 31b, respectively.

[0006] The images 31a and 31b are input to the learning models 32a and 32b, respectively, and the learning models 32a and 32b output 2D key points 33a and 33b, respectively. The learning models 32a and 32b are trained deep learning models, etc. The 2D key points 33a and 33b are two-dimensional skeleton information, etc.

[0007] In the prior art, 3D key points 34 are generated by integrating 2D key points 33a and 33b. The 3D key points 34 are three-dimensional skeleton information, etc. For example, the three-dimensional coordinates of each joint of the human body model are set in the 3D key points 34.

[0008] Fig.19 is a diagram showing an example of a human body model. Fig.19As shown, the human body model is defined by 21 joints ar0 to ar20. The 3D key points 34 set the three-dimensional coordinates of x, y, and z for each joint ar0 to ar20 defined by the human body model.

[0009] Fig.19 The relationship between the joints ar0~ar20 and the joint names is shown as follows Fig. 20 shown. Fig. 20 is a diagram showing an example of joint names. For example, the joint name of joint ar0 is "SPINE_BASE". The joint names of joints ar1 to a20 are Fig. 20 As shown, description is omitted.

[0010] Here, in order to improve the performance of image-based skeleton recognition, there is a method of generating learning data for a deep learning model based on 3D key points for additional learning.

[0011] In the prior art, it is difficult to directly generate learning data from 3D key points, so the 3D key points are temporarily converted into hierarchical structure data, and the learning data is generated based on the hierarchical structure data. The hierarchical structure data includes BVH (Biovision Hierarchy) data.

[0012] For example, BVH data consists of two parts: "HIERARCY" and "MOTION". First, in HIERARCY, the human body's skeletal structure is defined by multiple nodes, and has data on a reference posture. The multiple nodes representing the human body's skeletal structure include "ROOT" nodes representing reference joints of the human body, "JOINT" nodes representing joints, and "End" nodes representing the front ends of the hands and feet. In HIERARCY, the connection order of the nodes from the top ROOT to the bottom End is defined. In adjacent nodes, a "joint direction vector" is set from the upper node toward the lower node.

[0013] Fig.21 is a diagram showing an example of a reference posture. Fig.21 In the example shown, the reference posture is represented by nodes rn, n0 to n20. Node rn is ROOT. Nodes n0 to n2, n4 to n12, n14 to n16, and n18 are JOINT. Nodes n3, n13, n17, n19, and n20 are END. The joint direction vector between the nodes in the reference posture is defined as "OFFSET".

[0014] On the other hand, the total number of frames and the time per frame are set in MOTION, and each frame contains information on the three-dimensional position of the ROOT, the three-dimensional rotation angle of the ROOT, and the rotation angle of each JOINT.

[0015] Fig. 22 This is a diagram for explaining an example of the three-dimensional rotation angle of ROOT. For example, the three-dimensional rotation angle of ROOT is the Euler angle (θx, θy, θz) of converting the node rn-1 of ROOT in the global coordinate system to the node rn-2 of ROOT in the ROOT coordinate system.

[0016] Fig.23 This is a diagram for explaining an example of a 3D rotation angle of a JOINT. For example, the 3D rotation angle of a JOINT is the Euler angle (θx, θy, θz) converted from the upper JOINT coordinate system to the JOINT coordinate system of interest. For example, if the JOINT of interest is set to node n4, the upper node of node n4 is node n2. In this case, the upper JOINT coordinate system is the coordinate system of node n2. The JOINT coordinate system of interest is the coordinate system of node n4.

[0017] Here, the 3D key point is the three-dimensional coordinate data. Since there is no information about the root and joints, it is impossible to simply calculate the three-dimensional rotation angle of the root and the three-dimensional rotation angle of each joint from the 3D key point. For example, in the prior art, the three-dimensional rotation angle of the root and the three-dimensional rotation angle of each joint are calculated from the 3D key point based on inverse kinematics.

[0018] Inverse kinematics is a method of calculating the joint angles for reaching a part such as a fingertip at a target position provided as an input in a multi-jointed robot such as a robot arm.

[0019] Fig.24 This is a diagram for explaining inverse kinematics. As a representative example, inverse kinematics based on the gradient method is explained. Fig.24 In the example shown, a target position tar is provided. In inverse kinematics, a joint angle q is calculated that brings the error e between the target position tar and the position h1 of the finger tip close to 0, and the step of repeatedly updating q is repeatedly performed.

[0020] For example, the processing steps for obtaining the three-dimensional rotation angle of the ROOT and each JOINT of the BVH data from the 3D key points are as follows: Fig.25 The processing steps are shown.

[0021] Fig.25 is a flow chart showing the processing steps of the prior art. Fig.25 In the figure, the device that performs the processing of the prior art is represented as the prior art device. Fig.25 As shown, the existing device obtains 3D key points (step S10).

[0022] The existing device calculates the length of the bone according to the 3D key points (step S11). The existing device describes the HIERARCY (step S12). The existing device calculates the three-dimensional coordinates of the ROOT (step S13).

[0023] The existing device calculates the three-dimensional rotation angle of ROOT and JOINT based on inverse kinematics (step S14) and describes MOTION (step S15).

[0024] Next, Fig.25 The processing steps for calculating the three-dimensional rotation angles of ROOT and JOINT based on inverse kinematics described in step S14 are described below. Fig.26 : is a flowchart showing the processing steps for calculating the three-dimensional rotation angle of ROOT and JOINT based on inverse kinematics. Fig.26 As shown, the conventional apparatus accepts input of the joint position p_{tar} of the 3D key point and the initial value q_{src} of the joint position of the BVH data (step S20).

[0025] The conventional device calculates p_{src} based on q_{src} by forward kinematics FK (step S21). The conventional device calculates the error e between joint positions based on equation (1) (step S22).

[0026] e=|p_{tar}-p_{src}|…(1)

[0027] The conventional device moves to step S27 when the error e is smaller than ε (step S23: Yes). On the other hand, the conventional device moves to step S24 when the error e is not smaller than ε (step S23: No).

[0028] The processing after step S24 will be described. The conventional device calculates the Jacobian matrix J as the velocity of the joint angle q (q_{src}) that reduces the error e (step S24).

[0029] The existing device calculates the displacement Δq (Δq_{src}) that reduces the error according to the Jacobian matrix J of the error e according to equation (2) (step S25).

[0030] Δq=-J －1 e…(2)

[0031] The conventional device updates q according to equation (3) (step S26) and moves to step S21.

[0032] q=q+Δq…(3)

[0033] The processing after step S27 is described. The conventional device sets the value of q_{src} to the value of q_{tar} (step S27). The conventional device executes the processing of steps S21 to S27 described above for all joints (step S28). The conventional device outputs q_{tar} (step S29).

[0034] Fig. 27 For additional explanation Fig.26 The processed graph. Fig. 27 As shown, the joint position p_{tar} of the 3D key point and the initial value q_{src} of the joint angle of the BVH data are input. If the execution Fig.26 After the processing of steps S20 to S29, the activity q_{tar} of the BVH data is output.

[0035] Patent document 1: Japanese Patent Application Publication No. 2022-92528.

[0036] However, in the above-mentioned prior art, 3D key points are converted into hierarchical structure data according to inverse kinematics, so there is a problem of high amount of calculation.

[0037] For example, in Fig.26 In the inverse kinematics processing described in , the processing of steps S21 to S26 is repeatedly performed until the error e becomes smaller than ε. Summary of the invention

[0038] In one aspect, an object of the present invention is to provide a conversion method, a conversion program, and a conversion device capable of reducing the amount of calculation for converting 3D key points into hierarchical structure data.

[0039] In the first scheme, a computer is caused to perform the following processing. The computer obtains skeleton information that respectively sets multiple joints included in the human body and coordinates for the multiple joints, and reference posture information that respectively sets multiple joints included in the human body and reference coordinates for the multiple joints. The computer determines a second joint corresponding to a first joint set in the reference posture information from the multiple joints set in the skeleton information. The computer calculates a relative rotation angle from the reference coordinates of the first joint to the coordinates of the second joint, and sets the relative rotation angle in the hierarchical structure data, thereby converting the skeleton information into the hierarchical structure data.

[0040] It is possible to reduce the amount of calculation for converting 3D key points into hierarchical structure data. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Figure 1 This is a diagram for explaining the process of calculating the three-dimensional rotation angle of ROOT by rigid body positioning.

[0042] Figure 2FIG. (1) is a diagram for explaining the process of calculating the three-dimensional rotation angle of a JOINT using Rodriguez's rotation formula.

[0043] Figure 3 FIG. (2) is a diagram for explaining the process of calculating the three-dimensional rotation angle of a JOINT using Rodriguez's rotation formula.

[0044] Figure 4 It is a functional block diagram showing the structure of the conversion device involved in this embodiment.

[0045] Figure 5 This is a diagram showing an example of the data structure of a 3D key point table.

[0046] Figure 6 This is a diagram for explaining an example of the data structure of hierarchical structure data.

[0047] Figure 7 This is a diagram showing an example of the data structure of BVH data.

[0048] Figure 8 This is a diagram showing an example of model data.

[0049] Fig. 9 This is a diagram showing an example of the data structure of the correction angle dictionary.

[0050] Fig.10 This is a diagram for explaining the processing of the correction execution unit.

[0051] Fig.11 This is a diagram for explaining the processing of the generation unit.

[0052] Fig.12 It is a flowchart showing the processing steps of the conversion device involved in this embodiment.

[0053] Fig.13 : is a flowchart showing the processing steps of the first calculation process.

[0054] Fig.14 : is a flowchart showing the processing steps of the second calculation process.

[0055] Fig.15 It is a flowchart showing the processing procedure of the correction process.

[0056] Fig.16 It is a diagram for explaining the effect of the conversion device involved in this embodiment.

[0057] Fig.17 This is a diagram showing an example of the hardware configuration of a computer that realizes the same function as the conversion device of the embodiment.

[0058] Fig.18It is a diagram used to explain the prior art related to 3D skeleton recognition.

[0059] Fig.19 This is a diagram showing an example of a human body model.

[0060] Fig. 20 This is a diagram showing an example of joint names.

[0061] Fig.21 is a diagram showing an example of a reference posture.

[0062] Fig. 22 This is a diagram for explaining an example of the three-dimensional rotation angle of ROOT.

[0063] Fig.23 This is a diagram for explaining an example of the three-dimensional rotation angle of JOINT.

[0064] Fig.24 This is a diagram used to explain inverse kinematics.

[0065] Fig.25 It is a flowchart showing the processing steps of the prior art.

[0066] Fig.26 This is a flowchart showing the processing procedure for calculating the three-dimensional rotation angle of ROOT and JOINT based on inverse kinematics.

[0067] Fig. 27 For additional explanation Fig.26 Processing diagram. DETAILED DESCRIPTION

[0068] Hereinafter, embodiments of the conversion method, conversion program, and conversion device disclosed in the present application will be described in detail based on the accompanying drawings. In addition, the present invention is not limited by these embodiments.

[0069] Example

[0070] The processing of the conversion device involved in this embodiment is described. The conversion device focuses on the fact that the reference posture of the hierarchical structure data is known. Fig.21 The conversion device calculates the relative rotation angle from the reference posture to the 3D key point, thereby eliminating the need for repeated calculations when converting the 3D key point to BVH data, thereby reducing the amount of calculation.

[0071] For example, the conversion device calculates the three-dimensional rotation angle of the ROOT by rigid body position alignment. In addition, the conversion device calculates the three-dimensional rotation angle of the JOINT by Rodriguez's rotation formula.

[0072] First, an example of a process in which the conversion device calculates the three-dimensional rotation angle of ROOT by rigid body alignment will be described. Figure 1 This is a diagram for explaining the process of calculating the three-dimensional rotation angle of ROOT by rigid body position alignment. The conversion device divides each node of the hierarchical structure data (reference posture data) into three or more rigid body joint nodes and other nodes according to the predefined. Fig.21 As described in , the nodes of the reference posture are nodes rn, n0 to n20. Figure 1 In the example shown, the conversion device selects nodes n0, n10, and n14 among the nodes included in the reference posture 40 as rigid body joint nodes.

[0073] The conversion device determines the joint group corresponding to the rigid body joint node (node ​​n0, n10, n14) from each joint included in the 3D key point 41. Figure 1 In the example shown, the joint groups corresponding to the rigid body joint nodes are joints ar0, ar10, and ar14. In the following description, the joint groups corresponding to the rigid body joint nodes among the joints included in the 3D key points are referred to as "rigid body corresponding joints".

[0074] The conversion device calculates the relative rotation angle from the rigid body joint node (node ​​n0, n10, n14) to the corresponding joint of the rigid body (joint ar0, ar10, ar14) by rigid body position alignment, and sets it as the three-dimensional rotation angle of ROOT.

[0075] Rigid body alignment is a method of finding the conversion parameters that make the source and target consistent based on the source and target of a combination of more than three points by the least square method according to formula (4). The conversion parameters include the rotation matrix R, translation t, and scale c. In this embodiment, the rotation matrix R is converted into Euler angles and used as the three-dimensional rotation angle of ROOT.

[0076] [Formula 1]

[0077]

[0078] In formula (4), "x" is the coordinate of the source of more than 3 points. The coordinate of the source is the three-dimensional coordinate of the rigid body joint. "y" is the coordinate of the target of more than 3 points. The coordinate of the target is the three-dimensional coordinate of the corresponding joint of the rigid body. The conversion device calculates e of formula (4) 2 Become the smallest rotation matrix R, translation t, and scale c.

[0079] Next, an example of a process in which the conversion device calculates the three-dimensional rotation angle of the JOINT using the Rodriguez rotation formula will be described. Figure 2 as well as Figure 3 is a diagram for explaining the process of calculating the three-dimensional rotation angle of JOINT using Rodriguez's rotation formula. Figure 2 The conversion device converts the joint direction vector of the focus joint of the 3D key point 41 into the joint direction vector of the local coordinate system. The joint direction vector of the 3D key point 41 is a vector from the lower joint to the upper joint among the adjacent joints.

[0080] exist Figure 2 In the example shown, the focus joint is described as joint ar0. The joint direction vector v_{tar} of joint ar0 is a vector from the lower joint ar0 toward the upper joint ar1.

[0081] The conversion device multiplies the joint direction vector of the joint of interest by the inverse matrix (R) of the rotation matrix R obtained in the above equation (4). －1 ), and thus converted to the joint direction vector of the local coordinate system. For example, by multiplying the joint direction vector v_{tar} of the joint of interest by R －1 , thereby obtaining the joint direction vector v_{tar_local} of the local coordinate system. The conversion device repeatedly performs the above processing on each joint of the 3D key point 41, thereby obtaining the 3D key point 42 of the local coordinate system.

[0082] Next, the conversion device determines the angle θ formed by the joint direction vector of the 3D key point 42 and the joint direction vector of the reference posture 40, with the normal of the joint direction vector of the 3D key point 42 and the joint direction vector of the reference posture 40 as the rotation axis. Here, as an example, the joint direction vector v_{tar_local} of the 3D key point 42 and the joint direction vector v_{src} of the reference posture 40 are used for explanation. The joint direction vector v_{src} is a vector from the node n0 of the reference posture toward the node n1.

[0083] Move to Figure 3 Instructions. Figure 3 Where n is the normal between the joint direction vector v_{tar_local} and the joint direction vector v_{src} of the reference posture 40. The conversion device determines the normal n by the vector product of the joint direction vector v_{tar_local} and the joint direction vector v_{src}. The conversion device determines the angle θ between the joint direction vector v_{tar_local} and the joint direction vector v_{src} using the normal n as the rotation axis.

[0084] The conversion device uses the normal line n as the rotation axis and the angle θ as the rotation angle, calculates the relative rotation angle through Rodriguez's rotation formula, and uses it as the three-dimensional rotation angle of the JOINT.

[0085] Rodriguez's rotation formula is a formula for calculating the rotation matrix R according to formula (5) based on the rotation axis (normal line n) and the rotation angle (the angle θ) determined by the source and target of the joint direction vector. The source of the joint direction vector is the joint direction vector (v_{src}) of the reference pose 40. The target of the joint direction vector is the joint direction vector (v_{tar_local}) of the 3D key point 42 of the local coordinate system.

[0086] [Formula 2]

[0087] R n (θ) = cosθI + (1 - cossθ)n t n+sinθ[n] × …(5)

[0088] The conversion device converts R obtained by formula (5) n (θ) is converted to Euler angles and used as the three-dimensional rotation angle of the JOINT.

[0089] As described above, the conversion device involved in this embodiment calculates the relative rotation angle from the reference posture of the hierarchical structure data to the 3D key point, thereby reducing the amount of calculation. For example, the conversion device calculates the three-dimensional rotation angle of the ROOT by rigid body position alignment. In addition, the conversion device calculates the three-dimensional rotation angle of the JOINT by Rodriguez's rotation formula.

[0090] Next, execute Figure 1 to Figure 3 A structural example of a conversion device for the processing described in is described. Figure 4 is a functional block diagram showing the structure of the conversion device involved in this embodiment. Figure 4 As shown, the conversion device 100 includes a communication unit 110 , an input unit 120 , a display unit 130 , a storage unit 140 , and a control unit 150 .

[0091] The communication unit 110 performs data communication with external devices and the like via a network. The communication unit 110 is a NIC (Network Interface Card) and the like. The control unit 150 described later exchanges data with external devices via the communication unit 110 .

[0092] The input unit 120 is an input device for inputting various information to the control unit 150 of the conversion device 100. For example, the input unit 120 corresponds to a keyboard, a mouse, a touch panel, or the like.

[0093] The display unit 130 is a display device that displays information output from the control unit 150 .

[0094] The storage unit 140 includes a 3D key point table 141, hierarchical structure data 142, and a learning data table 144. The storage unit 140 is a storage device such as a memory.

[0095] The 3D key point table 141 is a table that holds information related to 3D key points. Figure 5 is a diagram showing an example of the data structure of a 3D key point table. Figure 5 As shown, the 3D key point table 141 stores frame numbers and three-dimensional coordinates corresponding to each joint (identification information of the joint) in association with each other.

[0096] The frame number is a frame number for identifying an image used when generating a 3D key point. The identification information of each joint is information that uniquely identifies the joint. Figure 5 In the description, joints ar0 to ar20 are used as joint identification information. For example, joint ar0 corresponds to "SPINE_BASE". Joint ar1 corresponds to "SPINE_MID". Joint ar20 corresponds to "HAND_TIP_RIGHT". The relationship between other joints and joint names is as follows: Fig. 20 shown.

[0097] Each of the three-dimensional coordinates of the joints ar0 to ar20 corresponding to a certain frame number (for example, 0001) in the 3D key point table 141 becomes a 3D key point corresponding to the certain frame number (for example, 0001).

[0098] The hierarchical structure data 142 is data obtained by converting 3D key points. Figure 6 is a diagram for explaining an example of the data structure of hierarchical structure data. Figure 6 As shown, the hierarchical structure data 142 includes BVH data 50 and model data 60 .

[0099] Figure 7 is a diagram showing an example of the data structure of BVH data. Figure 7 As shown, the BVH data includes HIERARCY 51 and MOTION 52. HIERARCY 51 includes node definition information 51a, hierarchical structure definition information 51b, reference posture information 51c, and channel information 51d.

[0100] The node definition information 51a is a definition Fig.21 The information of which node is ROOT, which node is JOINT, and which node is END among the nodes rn, n0 to n20 shown. For example, node rn is ROOT. Nodes n0 to n2, n4 to n12, n14 to n16, and n18 are JOINT. Nodes n3, n13, n17, n19, and n20 are END.

[0101] The hierarchical structure definition information 51 b is information that defines the connection order of nodes from ROOT, which is the highest level, to End, which is the lowest level.

[0102] The reference posture information 51c is Fig.21 For example, in the reference posture information 51c, the joint direction vectors of each node that becomes the reference posture are defined.

[0103] The channel information 51d is information defining data registered in activity data 52c of each frame described later.

[0104] MOTION52 includes a frame number 52a, a frame time 52b, and motion data 52c. The frame number 52a indicates the total number of frames, and the frame time 52b indicates the time per frame.

[0105] The motion data 52c registers the three-dimensional position of the ROOT, the three-dimensional rotation angle of the ROOT, and the three-dimensional rotation angle of each JOINT as information of one frame. One motion data 52c is generated based on one 3D key point.

[0106] Figure 8 61a , 61b , and 61c , respectively. For example, the model data 60 includes SMPL (Skinned Multi-Person Linear Model) 61a , MakeHuman 61b , and Autodesk Character Generator 61c .

[0107] SMPL61a is a 3D body CG (Computer Graphics) model that can adjust posture and body shape by learning from 3D body scan data. MakeHuman61b is a 3D body CG model that can adjust gender, height, body shape, race, etc. Autodesk Character Generator61c is a 3D body CG model that can adjust details such as expression, muscles, hair, posture, etc.

[0108] Return to Figure 4 The correction angle dictionary 143 is a dictionary that stores information for correcting the three-dimensional rotation angle of the JOINT. Fig. 9 is a diagram showing an example of the data structure of the correction angle dictionary. Fig. 9 As shown, the correction angle dictionary 143 establishes correspondence between frame numbers, labels, and correction angles.

[0109] The frame number is the frame number used to identify the image when generating the 3D key point, and is also the information for identifying the 3D key point. Labels are set to "forehand", "backhand" and other labels. When the label is "none", it means that there is no corresponding correction angle. The correction angle is set to the correction angle of each joint. According to the above correction angle, the three-dimensional rotation angle of the JOINT is corrected.

[0110] The learning data table 144 stores learning data generated based on the hierarchical structure data 142. The learning data is data that uses an image as input and a correct answer label as a joint coordinate. The learning data is used when training a machine learning model such as a NN (Neural Network).

[0111] Back to Figure 4 The control unit 150 includes an acquisition unit 151, a first calculation unit 152, a second calculation unit 153, a correction execution unit 154, a setting unit 155, a generation unit 156, and a learning unit 157. The control unit 150 is a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), and the like.

[0112] The acquisition unit 151 acquires information of the 3D key point table 141 and information of the correction angle dictionary 143 from an external device via a network. The acquisition unit 151 stores the acquired information of the 3D key point table 141 and correction angle dictionary 143 in the storage unit 140 .

[0113] The first calculation unit 152 obtains 3D key points from the 3D key point table 141 and obtains the reference posture from the hierarchical structure data 142. The 3D key points are assigned frame numbers. The first calculation unit 152 calculates the three-dimensional rotation angle of the ROOT by rigid body position alignment based on the reference posture and the 3D key points. Figure 1 The first calculation unit 152 converts the three-dimensional rotation angle of ROOT into Euler angles using a predetermined conversion formula.

[0114] The first calculation unit 152 calculates the length of the bones between the joints according to the 3D key points. The first calculation unit 152 calculates the three-dimensional coordinates of the ROOT.

[0115] The first calculation unit 152 outputs the 3D key point, the reference posture, and the calculation result (three-dimensional rotation angle of ROOT <rotation matrix R, Euler angle>, three-dimensional coordinates of ROOT) to the second calculation unit 153 .

[0116] The second calculation unit 153 calculates the three-dimensional rotation angle of each JOINT by using the Rodriguez rotation formula based on the 3D key point, the reference posture and the three-dimensional rotation angle of the ROOT <rotation matrix R>. Figure 2 , Figure 3 The second calculation unit 153 converts the three-dimensional rotation angle of each JOINT into the Euler angle using a predetermined conversion formula.

[0117] The second calculation unit 153 outputs the 3D key point, the three-dimensional rotation angle of ROOT <rotation matrix R, Euler angle>, the calculation result (the three-dimensional rotation angle of each JOINT <rotation matrix R, Euler angle>), and the three-dimensional coordinates of ROOT to the correction execution unit 154.

[0118] The correction execution unit 154 corrects the three-dimensional rotation angle (Euler angle) of each JOINT based on the correction angle dictionary 143 . Fig.10 This is a diagram for explaining the processing of the correction execution unit. Fig.10 The BVH data 70 is data in which the three-dimensional rotation angle (before correction) of each JOINT is applied to the reference posture. Here, the correction angle of the correction angle dictionary 143, that is, the information of the correction angle of each joint corresponding to the frame number of the 3D key point is set as the correction angle information 143-1. In the correction angle information 143-1, the correction angle of the joint ar5 is "-90°", and the correction angle of the joint ar8 is "+90°".

[0119] The correction execution unit 154 corrects the three-dimensional rotation angle of the node n8 corresponding to the joint ar5 among the nodes (joints) of the BVH data 70 with a correction angle of "-90°". In addition, the correction execution unit 154 corrects the three-dimensional rotation angle of the node n8 corresponding to the joint ar8 among the nodes (joints) of the BVH data 70 with a correction angle of "+90°". The correction execution unit 154 generates BVH data 71 by performing the above correction on the BVH data 70. The rotation angle corresponding to the joints ar5 and ar8 of the BVH data 70 does not match the rotation angle of the actual character's posture 75, but the rotation angle corresponding to the joints ar5 and ar8 of the BVH data 71 matches the rotation angle of the actual character's posture 75.

[0120] In addition, the correction execution unit 154 corrects the three-dimensional rotation angle of the JOINT through the following steps. The correction execution unit 154 converts the three-dimensional rotation angle of the JOINT before correction (in the form of Euler angles) into a three-dimensional rotation angle (in the form of rotation matrix R). The correction execution unit 154 calculates the corrected rotation matrix R_{correct} of the correction angle based on the direction of the joint of interest and the correction angle. The correction execution unit 154 multiplies the three-dimensional rotation angle before correction (in the form of rotation matrix R) by the corrected rotation matrix R_{correct} to calculate the corrected three-dimensional rotation angle (in the form of rotation matrix R).

[0121] The correction execution unit 154 returns the corrected three-dimensional rotation angle (in the form of rotation matrix R) to the corrected three-dimensional rotation angle (in the form of Euler angles), and updates the three-dimensional rotation angle of the JOINT.

[0122] In addition, when the correction angle of each joint corresponding to the frame number of the 3D key point is "none", the correction execution unit 154 interpolates the correction angle based on the correction angles of the previous and next frame numbers. For example, the correction execution unit 154 performs spherical linear interpolation. The correction execution unit 154 uses the interpolated correction angle to correct the three-dimensional rotation angle of the JOINT as described above.

[0123] The correction execution unit 154 outputs the 3D key point, the three-dimensional rotation angle of the ROOT (in the form of Euler angles), the three-dimensional rotation angle of each JOINT (in the form of Euler angles), and the three-dimensional coordinates of the ROOT to the setting unit 155 .

[0124] The first calculation unit 152 , the second calculation unit 153 , and the correction execution unit 154 repeatedly execute the above-mentioned processing for each 3D key point.

[0125] The setting unit 155 sets various information in the hierarchical structure data 142. For example, the setting unit 155 describes information of HIERARCY51 and MOTION52 in the hierarchical structure data 142. The setting unit 155 describes HIERARCY51 based on information prepared in advance. The setting unit 155 describes the number of frames 52a and frame time 52b of MOTION52 based on information prepared in advance. The setting unit 155 describes motion data 52c based on the calculation results of the first calculation unit 152, the second calculation unit 153, and the correction execution unit 154.

[0126] The generating unit 156 generates learning data based on the hierarchical structure data 142 . Fig.11 The generating unit 156 applies the three-dimensional rotation angle (in the form of Euler angles) of the ROOT and the three-dimensional rotation angle (in the form of Euler angles) of each JOINT to the reference posture of the hierarchical structure data 142 to generate the BVH data 80 .

[0127] The generation unit 156 generates a model 81 of a person based on the BVH data 80 and the model data 60. The generation unit 156 sets virtual cameras 81a and 81b. The generation unit 156 uses the virtual cameras 81a and 81b to generate images from the virtual viewpoint position of the model 81. For example, the generation unit 156 associates the image 82a captured by the virtual camera 81a with the joint coordinates 82b of the model 81 of the person, and generates the learning data 82.

[0128] The generating unit 156 generates a plurality of learning data by repeatedly executing the above-described process, and registers the generated learning data in the learning data table 144 .

[0129] The learning unit 157 trains the learning model based on the learning data in the learning data table 144. For example, the learning unit 157 updates parameters based on the error back propagation method so that the error between the output obtained by inputting the image into the learning model and the correct answer label (joint coordinates) becomes smaller.

[0130] Next, an example of the processing procedure of the conversion device 100 according to the present embodiment will be described. Fig.12 FIG. 1 is a flow chart showing the processing steps of the conversion device involved in this embodiment. Fig.12 As shown, the first calculation unit 152 of the conversion device 100 obtains 3D key points from the 3D key point table 141 (step S101 ). The first calculation unit 152 calculates the length of the bone (step S102 ).

[0131] The setting unit 155 of the conversion device 100 describes the HIERARCY51 of the hierarchical structure data 142 (step S103). The first calculation unit 152 calculates the three-dimensional coordinates of the ROOT (step S104). The first calculation unit 152 performs the first calculation process (calculation process of the three-dimensional rotation angle of the ROOT based on the rigid body position alignment) (step S105).

[0132] The second calculation unit 153 of the conversion device 100 executes the second calculation process (the calculation process of the three-dimensional rotation angle of the JOINT based on the Rodriguez rotation formula) (step S106 ). The correction execution unit 154 of the conversion device 100 executes the correction process (step S107 ).

[0133] The setting unit 155 describes the MOTION52 of the hierarchical structure data 142 based on the calculation result (step S108).

[0134] Next, Fig.12 The processing steps of the first calculation process shown in step S105 will be described. Fig.13 is a flowchart showing the processing steps of the first calculation process. Fig.13As shown, the first calculation unit 152 of the conversion device 100 classifies each node of the BVH data into three or more rigid body joints q_{src_rb} and other joints (step S201).

[0135] The first calculation unit 152 extracts the joint group p_{tar_rb} of the 3D key points corresponding to the rigid body joint q_{src_rb} (step S202 ) and calculates the relative rotation angle R_{global} by rigid body position alignment based on the rigid body joint q_{src_rb} and the joint group p_{tar_rb} of the 3D key points (step S203 ).

[0136] The first calculation unit 152 outputs the relative rotation angle R_{global} as the three-dimensional rotation angle of ROOT (step S204 ).

[0137] Next, Fig.12 The processing steps of the second calculation process shown in step S106 are described below. Fig.14 is a flowchart showing the processing steps of the second calculation process. Fig.14 As shown, the second calculation unit 153 of the conversion device 100 sets the number of joints N=21, the joint index i=1, and the local coordinate conversion matrix R=R_{global} (step S301).

[0138] The second calculation unit 153 obtains i_p as the index of the parent joint of the joint index i based on equation (6) (step S302). Parent is a data arrangement that establishes a corresponding joint index and its parent joint index. If a joint index i is input, the index of its parent joint is output as parent(i).

[0139] i_p=parent(i)…(6)

[0140] The second calculation unit 153 calculates the joint direction vector v_{tar} of the joint index i according to the joint coordinates joint(i) and joint(i_p) based on equation (7) (step S303). joint is a data arrangement that establishes a correspondence between the joint index and its joint coordinates. If the joint index i is input, its joint coordinates joint(i) are output.

[0141] v_{tar}=joint(i)-joint(i_p)…(7)

[0142] The second calculation unit 153 calculates v_{tar_local}, which is the joint direction vector of the local coordinate system, from the joint direction vector v_{tar} and the local coordinate conversion matrix R based on equation (8) (step S304 ).

[0143] v_{tar_local}=R^-1*v_{tar}…(8)

[0144] The second calculation unit 153 calculates the joint direction vector v_{src} based on the reference posture of the BVH data (step S305). The second calculation unit 153 calculates the relative rotation angle Ri from v_{src} to v_{tar_local} using Rodriguez's rotation formula and sets it as the three-dimensional rotation angle of the JOINT of the joint index i (step S306).

[0145] When the joint index i is smaller than N (step S307 : Yes), the second calculation unit 153 outputs the three-dimensional rotation angle R_{local}=(R1, R2, . . . R21) of each JOINT (step S308 ).

[0146] On the other hand, if the joint index i is not less than N (step S307: No), the second calculation unit 153 adds 1 to the joint index i (step S309). The second calculation unit 153 updates the local coordinate transformation matrix R by multiplying the local coordinate transformation matrix R by the relative rotation angle Ri (step S310), and moves to step S302.

[0147] Next, Fig.12 The processing steps of the correction processing shown in step S107 are described below. Fig.15 is a flowchart showing the processing steps of the correction process. Fig.15 As shown, the correction execution unit 154 of the conversion device 100 reads one line of the correction angle dictionary 143 (step S401). When there is no label in the read line (step S402: No), the correction execution unit 154 proceeds to step S404.

[0148] When there is a label in the read row (step S402 : Yes), the correction execution unit 154 acquires the correction angle of each joint from the correction angle dictionary 143 (step S403 ).

[0149] If all the lines of the correction angle dictionary 143 have not been read (step S404: No), the correction execution unit 154 proceeds to step S401. On the other hand, if all the lines of the correction angle dictionary 143 have been read (step S404: Yes), the correction execution unit 154 proceeds to step S405.

[0150] The correction execution unit 154 reads a line without a label in the correction angle dictionary 143 (step S405). The correction execution unit 154 determines whether there is a correction angle in the previous line or the next line of the read line (step S406). The correction execution unit 154 interpolates the correction angle based on the correction angles of the previous line and the next line (step S407).

[0151] If all the unlabeled lines in the correction angle dictionary 143 have not been read (step S408: No), the correction execution unit 154 proceeds to step S405. On the other hand, if all the unlabeled lines in the correction angle dictionary 143 have been read (step S408: Yes), the correction execution unit 154 proceeds to step S409.

[0152] The correction execution unit 154 reads the correction angle of each joint corresponding to the current frame number and calculates the correction rotation matrix R_{correct} (step S409). The correction execution unit 154 updates the three-dimensional rotation angle R_{local} of the JOINT of each joint according to equation (9) (step S410).

[0153] R_{local_corrected}=R_{correct}*R_{local}…(9)

[0154] The correction execution unit 154 sets R_{local_corrected} as the three-dimensional rotation angle of JOINT (step S411 ).

[0155] Next, the effect of the conversion device 100 involved in this embodiment is described. The conversion device 100 obtains the reference posture of the 3D key point and the hierarchical structure data 142, calculates the relative rotation angle from the prescribed joint of the reference posture to the corresponding joint of the 3D key point, and sets the relative rotation angle in the hierarchical structure data 142. In this way, the amount of calculation can be reduced and the hierarchical structure data can be generated based on the 3D key point.

[0156] For example, the conversion device 100 calculates the relative rotation angle of the joint group from the rigid body joint node of the reference posture to the 3D key point corresponding to the rigid body joint node by rigid body alignment. This makes it possible to efficiently calculate the three-dimensional rotation angle of the ROOT without repeated calculations.

[0157] The conversion device 100 converts the joint direction vector of the 3D key point into the joint direction vector of the local coordinate system. The conversion device 100 determines the angle θ formed by each joint direction vector with the normal of the joint direction vector of the 3D key point and the reference posture as the rotation axis, calculates the relative rotation angle by Rodriguez's rotation formula, and uses it as the three-dimensional rotation angle of the JOINT. In this way, the three-dimensional rotation angle of each JOINT can be efficiently calculated without repeated calculations.

[0158] Furthermore, by calculating the relative rotation angle using the Rodriguez rotation formula, the three-dimensional rotation angle of the JOINT of the shortest path can be obtained, and a twisting rotation angle that is unnatural to humans can be suppressed.

[0159] Fig.16 9 is a diagram for explaining the effect of the conversion device involved in this embodiment. For example, the twist rotation angle becomes the rotation angle with the skeleton 90a shown in the image 90 as the axis. The human body model 91 is a human body model generated by using the result of converting the 3D key points into the hierarchical structure data by the prior art. The human body model 92 is a human body model generated by the conversion device 100 involved in this embodiment using the result of converting the 3D key points into the hierarchical structure data.

[0160] The human body model 91 has an unnatural torsional rotation angle such as twisted joints, but the human body model 92 does not have an unnatural torsional rotation angle.

[0161] Furthermore, the conversion device 100 according to this embodiment corrects the three-dimensional rotation angle of the JOINT based on the correction angle information in the correction angle dictionary 143. This makes it possible to obtain a torsional rotation angle that matches the actual torsional rotation that is difficult to obtain in the conventional art.

[0162] Next, an example of the hardware configuration of a computer that realizes the same function as the above-described conversion device 100 will be described. Fig.17 This is a diagram showing an example of the hardware configuration of a computer that realizes the same function as the conversion device of the embodiment.

[0163] like Fig.17 As shown, the computer 200 has a CPU 201 that performs various calculations, an input device 202 that receives data input from a user, and a display 203. In addition, the computer 200 has a communication device 204 and an interface device 205 that transmit and receive data with the camera 15, an external device, etc. via a wired or wireless network. In addition, the computer 200 has a RAM 206 and a hard disk device 207 that temporarily store various information. Moreover, each device 201 to 207 is connected to a bus 208.

[0164] The hard disk device 207 has an acquisition program 207a, a first calculation program 207b, a second calculation program 207c, a correction execution program 207d, a setting program 207e, a generation program 207f, and a learning program 207g. The CPU 201 reads out each program 207a to 207f and expands it in the RAM 206.

[0165] The acquisition program 207a functions as the acquisition process 206a. The first calculation program 207b functions as the first calculation process 206b. The second calculation program 207c functions as the second calculation process 206c. The correction execution program 207d functions as the correction execution process 206d. The setting program 207e functions as the setting process 206e. The generation program 207f functions as the generation process 206f. The learning program 207g functions as the learning process 206g.

[0166] The processing of the acquisition process 206a corresponds to the processing of the acquisition unit 151. The processing of the first calculation process 206b corresponds to the processing of the first calculation unit 152. The processing of the second calculation process 206c corresponds to the processing of the second calculation unit 153. The processing of the correction execution process 206d corresponds to the processing of the correction execution unit 154. The processing of the setting process 206e corresponds to the processing of the setting unit 155. The processing of the generation process 206f corresponds to the processing of the generation unit 156. The processing of the learning process 206g corresponds to the processing of the learning unit 157.

[0167] In addition, each program 207a to 207g may not necessarily be initially stored in the hard disk device 207. For example, each program may be pre-stored in a "portable physical medium" such as a floppy disk (FD), CD-ROM, DVD, optical magnetic disk, IC card, etc. inserted into the computer 200. Furthermore, each program 207a to 207g may be read out by the computer 200 and executed.

[0168] Description of Reference Numerals

[0169] 100…conversion device; 110…communication unit; 120…input unit; 130…display unit; 140…storage unit; 141…3D key point table; 142…hierarchical structure data; 143…correction angle dictionary; 144…learning data table; 150…control unit; 151…acquisition unit; 152…first calculation unit; 153…second calculation unit; 154…correction execution unit; 155…setting unit; 156…generation unit; 157…learning unit.

Claims

1. A conversion method, characterized in that: The following processing is performed by the computer: Obtaining skeleton information and reference posture information, wherein the skeleton information respectively sets multiple joints included in the human body and coordinates for the multiple joints, and the reference posture information respectively sets multiple joints included in the human body and reference coordinates for the multiple joints, determining a second joint corresponding to a first joint set in the reference posture information from among a plurality of joints set in the skeleton information, Calculating the relative rotation angle from the reference coordinates of the first joint to the coordinates of the second joint, The relative rotation angle is set in the hierarchical structure data, thereby converting the skeleton information into the hierarchical structure data.

2. The conversion method according to claim 1, characterized in that: The process of performing the calculation calculates the relative rotation angle from the reference coordinates of the plurality of first joints to the coordinates of the plurality of second joints by rigid body alignment.

3. The conversion method according to claim 2, characterized in that: The calculation process converts the second vector of the second joint according to the calculation result of the rigid body position alignment, and applies the angle between the first vector and the second vector with the normal of the first vector of the first joint and the converted second vector of the second joint as the rotation axis to Rodriguez's rotation formula, so as to further calculate the relative rotation angle.

4. The conversion method according to claim 3, characterized in that: A process of correcting the relative rotation angle according to the correction angle set in the skeleton information is also executed.

5. A conversion program, characterized in that Make the computer execute the following processing: Obtaining skeleton information and reference posture information, wherein the skeleton information respectively sets multiple joints included in the human body and coordinates for the multiple joints, and the reference posture information respectively sets multiple joints included in the human body and reference coordinates for the multiple joints, determining a second joint corresponding to a first joint set in the reference posture information from among a plurality of joints set in the skeleton information, Calculating the relative rotation angle from the reference coordinates of the first joint to the coordinates of the second joint, The relative rotation angle is set in the hierarchical structure data, thereby converting the skeleton information into the hierarchical structure data.

6. The conversion program according to claim 5, characterized in that The process of performing the calculation calculates the relative rotation angle from the reference coordinates of the plurality of first joints to the coordinates of the plurality of second joints by rigid body position alignment.

7. The conversion program according to claim 6, characterized in that The calculation process converts the second vector of the second joint according to the calculation result of the rigid body position alignment, and applies the angle between the first vector and the second vector with the normal of the first vector of the first joint and the converted second vector of the second joint as the rotation axis to Rodriguez's rotation formula, so as to further calculate the relative rotation angle.

8. The conversion program according to claim 7, characterized in that A process of correcting the relative rotation angle according to the correction angle set in the skeleton information is also executed.

9. A conversion device comprising a control unit, The control unit performs the following processing: Obtaining skeleton information and reference posture information, wherein the skeleton information respectively sets multiple joints included in the human body and coordinates for the multiple joints, and the reference posture information respectively sets multiple joints included in the human body and reference posture information for reference coordinates of the multiple joints, determining a second joint corresponding to a first joint set in the reference posture information from among a plurality of joints set in the skeleton information, Calculating the relative rotation angle from the reference coordinates of the first joint to the coordinates of the second joint, The relative rotation angle is set in the hierarchical structure data, thereby converting the skeleton information into the hierarchical structure data.

10. The conversion device according to claim 9, characterized in that: The process of performing the calculation calculates the relative rotation angle from the reference coordinates of the plurality of first joints to the coordinates of the plurality of second joints by rigid body position alignment.

11. The conversion device according to claim 10, characterized in that: The calculation process converts the second vector of the second joint according to the calculation result of the rigid body position alignment, and applies the angle between the first vector and the second vector with the normal of the first vector of the first joint and the converted second vector of the second joint as the rotation axis to Rodriguez's rotation formula, so as to further calculate the relative rotation angle.

12. The conversion device according to claim 11, characterized in that: A process of correcting the relative rotation angle according to the correction angle set in the skeleton information is also executed.

Images