A method executed by a computer for causing a modeled body skeleton to assume a pose

Abstract

To make a skeleton of a modeled body of a human or an animal take a posture.SOLUTION: A method comprises the steps of: a) providing first and second skeletons each comprising rotational joints connected by bones, each rotational joint of the second skeleton being associated to a respective joint of the first skeleton ); b) determining a relative configuration of the second skeleton, mapping each joint of the first skeleton associated to the joint of the second skeleton to the joint of the second skeleton; c) making the first skeleton take a posture defined by a rotational state for each joint of the first skeleton; and d) computing transformation matrices for the joints of the second skeletons such that change of the relative configuration is minimized, where the second skeleton further comprises a prismatic joint on at least one of the bones. The step of computing comprises determining rotations of the rotational joints of the second skeleton and translation of the prismatic joint or joints of the second skeleton such that the change of the relative configuration is minimized.SELECTED DRAWING: Figure 3C

Description

Technical Field

[0001] The present invention relates to a method executed by a computer for making a modeled human or animal body skeleton take a posture. More precisely, it relates to a method of transferring or "retargeting" the posture taken by a first ("source") skeleton to another ("target") skeleton.

[0002] The present invention relates to the field of computer graphics and is itself suitable for several applications such as computer-aided design (CAD), biotechnology, video games, etc.

Background Art

[0003] Digital human models (DHMs) are widely used as part of CAD systems for design evaluation and incorporate human factors at the initial stage of design.

[0004] DHMs usually include a "skeleton" covered by "skin", even in some cases where they may consist only of a skeleton. The skeleton is composed of a set of parts called "bones", which do not necessarily correspond to actual anatomical bones, and are connected at joints by rotational joints having one, two, or three degrees of rotational freedom. Such models, also known as "avatars", are usually provided in a standard posture such as the so-called "T-pose" (with legs partially spread and arms extended horizontally) for the human body model. See, for example, Non-Patent Document 1.

[0005] To make the avatar take a posture different from the original posture, it is necessary to determine a set of rotations of the joints connecting the bones of the skeleton that lead to the desired result. From a theoretical perspective, this corresponds to the problem of inverse kinematics (IK). There are several algorithms for solving this problem.

[0006] Despite the long history of DHM applications, the lack of proper standardization has led to various application-specific models that mimic human posture and movement with varying degrees of accuracy. Some DHMs are then commercially available with diverse capabilities and performances, some emphasizing the accuracy of anthropometry and linkage structure, some being good in posture / movement modeling and biomechanics, and some emphasizing the behavioral and cognitive aspects.

[0007] The coexistence of multiple skeleton definitions is undesirable as it leads to non-digital continuity among different applications of DHM.

[0008] The present invention aims to overcome the above-mentioned drawbacks of the prior art.

Prior Art Documents

Non-Patent Documents

[0009]

Non-Patent Document 1

Summary of the Invention

[0010] According to an aspect of the present invention, a DHM having two or more skeletons, and in some cases a single skin (SN) associated with the skeletons, is provided, as different skeletons can be optimized for different applications. For example, the first skeleton may be a complex and biologically accurate skeleton that is beneficial for performing, for example, biotechnological analysis, and the second skeleton of the DHM may be much simpler and achieve a better compromise between complexity and accuracy for real-time animation.

[0011] According to another aspect of the present invention, poses are mapped from one skeleton to another in order to maintain pose synchronization between two different skeletons. This is hereinafter referred to as "retargeting". Retargeting is known per se but is typically applied to skeletons belonging to different DHMs.

[0012] According to yet another further object of the present invention, at least one of the skeletons of the DHM (intended to be used as a target for the retargeting operation) is provided with linear joints for some or all of its bones. The bones provided with linear joints have variable lengths. The additional degrees of freedom provided by the linear joints make it possible to guarantee complete or almost complete retargeting even when the target skeleton has substantially fewer rotational joints than the starting skeleton. This then makes it possible for the retargeting process to be performed accurately and at low computational cost.

[0013] The present invention is not limited to the case of multi-skeleton DHMs and may be used to transfer poses from a first skeleton belonging to (or constituting) a first DHM to a second skeleton belonging to (or constituting) a second DHM. In other words, the first aspect of the present invention is not essential as described above.

[0014] The present invention is further not limited to digital human models and applies to any modeled human or animal body that includes at least one skeleton suitable for being retargeted to another skeleton, whether or not it belongs to the model.

[0015] An object of the present invention is to enable achievement of this object, which is a method executed by a computer for causing a modeled human or animal body skeleton to assume a posture, comprising: a) providing a first skeleton and a second skeleton of at least one digital body model, each of the first skeleton and the second skeleton including a plurality of rotational joints connected by bones, the first skeleton having a greater number of rotational joints than the second skeleton, and each rotational joint of the second skeleton being associated with a respective joint of the first skeleton; b) determining a relative configuration of the second skeleton, the relative configuration mapping each joint of the first skeleton associated with a joint of the second skeleton to the joint of the second skeleton; c) causing the first skeleton to assume a posture defined by a rotational state for each joint of the first skeleton based on one or more inputs from a user; d) calculating a transformation matrix for the joints of the second skeleton such that its relative configuration remains unchanged, the second skeleton further including a linear joint (G) for at least one of its bones, and step d) including determining a rotation of the rotational joints of the second skeleton and a translation of the linear joint or joints of the second skeleton such that a change in its relative configuration is minimized.

[0016] Another object of the present invention is a computer program product stored on a non-transitory computer-readable data storage medium including computer-executable instructions, the computer-executable instructions causing a computer system to execute such a method.

[0017] A further another object of the present invention is a non-transitory computer-readable data storage medium including computer-executable instructions, the computer-executable instructions causing a computer system to execute such a method.

[0018] Yet another object of the present invention is to provide a computer system including a processor coupled to a memory and a graphical user interface, the memory storing computer-executable instructions for causing the computer system to execute such a method.

[0019] A further object of the present invention is to provide a digital body model including a first skeleton and a second skeleton, each of the first skeleton and the second skeleton including a plurality of rotational joints connected by bones, the first skeleton having a greater number of rotational joints than the second skeleton, each rotational joint of the second skeleton being associated with a respective joint of the first skeleton, and the second skeleton further including a linear joint with respect to at least one of its bones.

[0020] Additional features and advantages of the present invention will become apparent from the following description when taken in conjunction with the accompanying drawings.

Brief Description of the Drawings

[0021]

Figure 1

Figure 2A

Figure 2B

Figure 2C

Figure 3A

Figure 3B

Figure 3C

Figure 3D

Figure 3E

Figure 3F

Figure 4

Figure 5

Figure 6

MODE FOR CARRYING OUT THE INVENTION

[0022] FIG. 1 represents an exemplary DHM having two skeletons SK1 and SK2 associated with a single skin SN. In this exemplary embodiment, the skeleton SK1 is a complex and biologically accurate skeleton, including a number of rotational joints, such a skeleton being useful, for example, for performing biotechnological analysis, but too complex for real-time animation applications. The skeleton SK2 is much simpler (it includes a smaller number of joints), achieving a good compromise between complexity and accuracy for real-time animation, but too crude for biotechnological analysis. The human modeling artist is involved in the construction of the simple skeleton SK2 and the associated skin to produce a reasonable skin deformation corresponding to different poses of SK2.

[0023] Figs. 2A - 2C show the retargeting operation between SK1 and SK2.

[0024] Figure 2A shows portions of skeletons SK1 and SK2 corresponding to the upper body of the DHM in their initial or "zero" postures. The rotational joints of SK2 are emphasized by circles. Each joint of SK2 exactly or at least approximately overlaps with the corresponding joint of SK1, but the reverse is not true assuming SK1 contains a greater number of joints.

[0025] In Figure 2B, the skeleton SK1 (which serves as the "starting" skeleton) takes on a different posture while SK2 remains stationary. It can be understood that the joints of SK1 corresponding to the joints of SK2 (highlighted by rectangular boxes) do not overlap with the latter.

[0026] Figure 2C shows the result of the retargeting operation, where the "target" skeleton SK2 has taken on a new posture that matches the posture of SK1, and the corresponding joints of the two skeletons are again approximately overlapping.

[0027] Posture is defined by the configuration of the kinematic structure, i.e., the mathematical representation of the connection structure formed by the bones and joints of the skeleton. Since each skeleton has its own topology that defines its kinematic structure, retargeting is a complex problem, especially when it is required to transfer postures from a complex starting skeleton to a simple target skeleton (see Figures 2A - 2C for the cases considered above). In fact, mapping from a complex skeleton to a simple skeleton necessarily involves loss of information since a reduced number of joints implies a smaller possible posture space, and it is not always possible to achieve a possible mapping (a careful comparison of Figures 2A and 2C shows that the relative positions of some joints of the two skeletons change slightly, which is particularly visible on the left side of the figure).

[0028] The two skeletons SK1 and SK2 are defined independently, and thus, beyond having different numbers of bones and joints, they can also have different degrees of freedom (DOF) and different types of joints with different coordinate frames. Their differences are transformed into different global transformation matrices that define the position and enclosure of each joint in the global coordinate frame for a given skeleton.

[0029] The first step of the inventive method is to identify corresponding joints on the two skeletons. If it is considered that the target skeleton SK2 has fewer joints than the starting skeleton SK1, this consists of identifying a subset of the joints of SK1 that correspond to each joint of SK2. This has already been discussed with reference to FIGS. 2A - 2C. In the highly simplified example of FIGS. 3A - 3F (in particular, see FIG. 3A), the starting skeleton SK1 has five joints labeled A, A1, A2, A3, and B, and the target skeleton has two joints labeled E and F. The joints A and B of SK1 are associated with the joints E and F respectively, and the joints A1, A2, and A3 are not associated with any joint of SK2. The association may be done manually, for example, by the designer of the multi - skeleton DHM, or automatically, for example, by automatically identifying the joint of SK1 that is closest to any one of the joints of SK2 when both skeletons are in their initial or "zero" pose.

[0030] Next, the relative transformation matrix of each joint of SK1 (e.g., joint B) with respect to the corresponding joint of SK2 (e.g., joint F) is calculated. The initial transformation matrix T of joint B, which represents its position and orientation with respect to the point serving as the global origin or root B

[0031]

Number

[0032] as, R Bis a 3×3 rotation matrix, and P B is a 3×1 (column) transformation vector, 0 is a 1×3 (row) vector with all its elements being zero, and 1 is a scalar. Similarly,

[0033]

Number

[0034] Let it be so.

[0035] The configuration of each skeleton is defined by the transformation matrix of all its joints.

[0036] The (initial) relative transformation matrix of joint F with respect to joint B

[0037]

Number

[0038] is

[0039]

Number

[0040] given by, and " -1 " represents a regular matrix.

[0041] In the example of Figure 3A, joints B and F coincide and have the same orientation, that is,

[0042]

Number

[0043] where 1 specifies the 4×4 identity matrix, but this is not necessarily the case.

[0044] The relative configuration of the starting skeleton SK1 is defined by the relative transformation matrices of all its joints.

[0045] When the posture of the skeleton SK1 at the starting point changes, the joint B is specified by B1 (see Fig. 3B) and takes different configurations (positions and / or orientations along three rotation axes) described by different transformation matrices T B1 Retargeting or pose synchronization aims to find a new configuration F1 for joint F described by transformation matrix T

[0046]

Number

[0047] such that, i.e., the relative transformation between corresponding joints remains unchanged. This is applied to all joints of SK2 in a way that its relative configuration with respect to SK1 (defined by the set of relative configurations of all its joints) remains unchanged F1 Let the distance between A and B be equal to the norm of

[0048] A and B1 be equal to the norm of

[0049]

Number

[0050] d AB Similarly, let the distance between

[0051]

Number

[0052] d AB1 For the target skeleton, the distance between E and F is the length of the bone EF

[0053]

Number

[0054] equal to the norm of, d EF is assumed.

[0055] The compensatory movement that displaces the joint F to match the position of B1 can only be achieved by using the joint E. The unit vectors of EF and EB1 are P1 and P2, which can be calculated from the values T F and T B1 Let (see FIG. 3C). Let θ Eswing be the angle between P1 and P2, and let P3 be the unit vector of the normal to P1 and P2 (see FIG. 4). All of these parameters may be determined from the transformation matrix. When the joint E rotates around the axis P3 by an angle θ Eswing , the joint F reaches the position F'. However, generally, due to the distance between E and F',

[0056]

Number

[0057] and d EF’ is d AB1 is different from. In fact, a comparison of FIGS. 3A and 3C shows that B and F coincide, while B1 and F' do not coincide.

[0058] Therefore, generally, rotating the joints of the target skeleton SK2 is not sufficient to achieve complete pose synchronization between SK1 and SK2. Also, in fact, in the prior art, retargeting often cannot achieve precise pose synchronization between skeletons and can only minimize errors in relative configuration changes.

[0059] According to the present invention, this problem is solved by providing a linear joint to the bone EF that allows its length to be changed (see FIG. 3D). The rotary joint E (which undergoes rotation by an angle θ Eswing around the axis P3) and the linear joint G (with a scale of d EF -d AB1By acting on both (translation) receptions of ), as shown in FIG. 3E, it is possible to displace point F to position F1 that coincides with point B1.

[0060] From a formal perspective, the conversion of F to F1 corresponds to the conversion matrix

[0061]

Number

[0062] and

[0063]

Number

[0064] is the rotation matrix for rotation by θ around axis P3, Eswing and

[0065]

Number

[0066] is.

[0067] This matrix is added to (i.e., post-multiplied by) the global matrix

[0068]

Number

[0069] to obtain the global matrix T E .

[0070]

Number

[0071] When applied to a local transformation with respect to E (i.e., post-multiplied), the joint F is transformed to F1 (identical to B1), thus achieving position matching using only two DOFs, which are the torsional DOFs that remain unused.

[0072] However, matching the position of joint F to position B1 is not a sufficient condition for pose mapping. Joint F also needs to be oriented to ensure that the condition

[0073]

Number

[0074] is satisfied. In fact, when joint F takes its position F1 that matches the position B1 of joint B, the unit vectors f1 and b1 connecting those joints to consecutive branches of the structure may be oriented in different directions, as shown in Figure 3E. An additional rotation

[0075]

Number

[0076] and an additional transformation matrix

[0077]

Number

[0078] may be calculated, using the same approach discussed above with reference to Figure 3C, to align the unit vector f1 with b1, as shown in Figure 3F.

[0079] Moreover, in 3D, the torsional components of the transformation matrix

[0080]

Number

[0081] It is also necessary to explain the existence of

[0082] However, aligning those two unit vectors does not necessarily satisfy the constraint of the initial relative configuration (Equation 3) due to the existence of the torsion component

[0083]

Number

[0084] Therefore, it is essential to determine the torsion component required to satisfy the constraint.

[0085] T B and T B1 are the initial global transformation and the final global transformation of joint B. Similarly, T F and T F1 are the initial global transformation and the final global transformation of joint F. The initial relative configuration of joints B and F is

[0086]

Number

[0087] given by, and since all pose deformations need to be preserved, it also applies to the final configuration

[0088]

Number

[0089] applies.

[0090] T F1 has two components

[0091]

Number

[0092] and

[0093]

Number

[0094] of the combination,

[0095]

Number

[0096] is,

[0097]

Number

[0098] is easily found from the vectors f1 and b1.

[0099] Therefore,

[0100]

Number

[0101] is.

[0102] In this regard, the three transformation matrices

[0103]

Number

[0104] ,

[0105]

Number

[0106] , and

[0107]

Number

[0108] enables the calculation of a new configuration of joint F. Out of the available seven DOFs, six are used to achieve pose synchronization. One DOF (the twist parameter) is left at joint E and can be used to address problems such as the possible distortion of the movement of the range of the joint at joint F and the deformation of the skin attached to the linkage EF. In fact, in the most practical cases, the overall process of retargeting needs to be carried out without disturbing the visual appearance of the skin. In those cases, although not sufficient to obtain a precise solution for retargeting, the appearance of the skin needs to remain undamaged and undistorted. The “modular” retargeting procedure described above, which handles one bone at a time, allows separating one DOF to address additional constraints. In particular, this DOF can be used to preserve the natural appearance of the skin. Additionally, in some cases, the retargeting solution can drive some joints outside their allowed range of movement, and in this scenario, the remaining DOFs can be used to ensure that the retargeting solution complies with the constraints regarding the position and orientation of the joints.

[0109] The same procedure is applied step by step to all the joints of SK2, starting from the root node and ending at the end effector.

[0110] The inventive method may in some cases be carried out by a suitably programmed general-purpose computer or computer system that includes, among other things, a computer network, stores a suitable program in a non-transitory form on a computer-readable medium such as a hard disk, solid-state disk, or CD-ROM, and executes the program using its microprocessor(s) and memory.

[0111] Referring to FIG. 5, a computer suitable for performing a method according to an exemplary embodiment of the present invention is described. In FIG. 5, the computer includes a central processing unit (CPU) that executes the above-described processing. The processing may be stored as an executable program, i.e., a set of computer-readable instructions, in RAM M1 or ROM M2, or a hard disk drive (HDD), a solid state driver (SDD) M3, or a DVD / CD drive M4, or may be stored remotely. Additionally, one or more digital body models and / or one or more computer files defining a skeleton may also be stored in one or more of the memory devices M1 to M4, or may be stored remotely.

[0112] The claimed invention is not limited to the form of a computer-readable medium storing computer-readable instructions of the inventive process and / or digital files. For example, the instructions and files may be stored on a CD, DVD, flash memory, RAM, ROM, PROM, EPROM, EEPROM, hard disk, or any other information processing device that a computer, such as a server or another computer, communicates with. The programs and files may be stored on the same memory device or on different memory devices.

[0113] Furthermore, a computer program suitable for performing the inventive method may be provided as a utility application, a background daemon, or a component of an operating system, or a combination thereof, that executes with the CPU 800 and an operating system such as Microsoft Windows 10 (registered trademark), UNIX (registered trademark), Solaris, LINUX (registered trademark), Apple (registered trademark) MAC-OS, and other systems known to those skilled in the art.

[0114] The central processing unit (CPU) may be a Xenon processor from Intel of America or an Opteron processor from AMD of America, or other processor types such as Freescale ColdFire, IMX, or an ARM processor from Freescale Corporation of America. Alternatively, the central processing unit may be a processor such as Core2 Duo from Intel Corporation of America, as recognized by those skilled in the art, or may be implemented on an FPGA, ASIC, PLD, or using discrete logic circuits. Further, the central processing unit may be implemented as a plurality of processors that cooperate to execute computer-readable instructions for the inventive processes described above.

[0115] The computer in FIG. 5 also includes a network interface NI, such as an Intel Ethernet PRO network interface card from Intel Corporation of America, for interfacing with networks such as a local area network (LAN), a wide area network (WAN), and the Internet. The computer-aided design station further includes a display controller DC, such as an NVIDIA GeForce GTX graphics adapter from NVIDIA Corporation of America, for interfacing with a display DY, such as a Hewlett Packard HPL2445w LCD monitor. The general-purpose I / O interface IF interfaces with a keyboard KB and a pointing device PD, such as a trackball, a mouse, and a touchpad. The display, keyboard, and pointing device, together with the display controller and the I / O interface, form a graphical user interface that is used by a user to provide input commands, for example, to move to a target point, to define parameters, and that is used by the computer-aided design station to display a three-dimensional scene including an avatar.

[0116] The disk controller DKC connects the HDD M3 and the DVD / CD M4 to the communication bus CBS, which may be ISA, EISA, VESA, PCI, or the like for interconnecting all components of a computer-aided design station.

[0117] Descriptions of the general characteristics and functionality of display controllers, disk controllers, network interfaces, and I / O interfaces, along with displays, keyboards, and pointing devices, are omitted in this specification for brevity since those characteristics are known.

[0118] FIG. 6 is a block diagram of a computer system suitable for implementing a method according to different exemplary embodiments of the present invention.

[0119] In FIG. 6, the executable program EXP and computer file(s) storing the avatar(s) and optionally the three-dimensional scene are stored in a memory device connected to the server SC. The overall architecture of the memory device and the server may be the same as that discussed above with reference to FIG. 5, except that a display controller, a display, a keyboard, and / or a pointing device may not be present in the server.

[0120] The server SC is then connected to the administrator system ADS and the end-user computer EUC via the network NW.

[0121] The overall architecture of the administrator system and the end-user computer may be the same as that discussed above with reference to FIG. 6, except that the memory devices of the administrator system and the end-user computer do not store the executable program EXP and / or computer file(s) that define the body model and / or skeleton(s). However, the end-user computer does not store a client program designed to cooperate with the executable program of the server, as discussed below.

[0122] For recognition, the network NW may be a public network such as the Internet, a private network such as a LAN or WAN network, or any combination thereof, and may also include a PSTN or ISDN subnetwork. The network NW may also be wired, such as an Ethernet network, or wireless, such as a cellular network including EDGE, 3G, and 4G wireless cellular systems. The wireless network may also be Wi-Fi, Bluetooth, or any other known wireless form of communication. Thus, the network NW is merely illustrative and does not limit the scope of this improvement.

[0123] The client program stored in the memory device of the end-user computer and executed by the latter's CPU accesses, via the network NW, a database DB stored by the server SC and including files that define the body model and / or skeleton. The server executes the processes described above and transmits, to the end-user computer using the network NW again, an image file corresponding to the desired representation of the body model(s) and / or skeleton(s).

[0124] Only one administrator system ADS and one end-user system EUX are shown, but the system can support any number of administrator systems and / or end-user systems without limitation. Similarly, multiple servers may also be implemented in the system without departing from the scope of the present invention.

[0125] Any process described herein should be understood as representing a module, segment, or portion of code that includes one or more executable instructions for performing a particular logical function or step in the process.

Claims

Claim 1 A method, executed by a computer, for causing a modeled human or animal body skeleton to assume a posture, comprising: a) providing a first skeleton and a second skeleton of at least one digital body model, each of the first skeleton and the second skeleton including a plurality of rotational joints connected by bones, the first skeleton having a greater number of rotational joints than the second skeleton, and each rotational joint of the second skeleton being associated with a respective joint of the first skeleton; b) determining a relative configuration of the second skeleton, the relative configuration being defined by a transformation matrix of all rotational joints of the second skeleton with respect to at least one corresponding rotational joint of the first skeleton; c) causing the first skeleton to assume a posture defined by a rotational state for each joint of the first skeleton, based on one or more inputs from a user; d) calculating a transformation matrix for the joints of the second skeleton such that the relative configuration of the second skeleton remains unchanged, wherein the second skeleton further includes a linear joint for at least one of its bones, step d) includes determining rotation of the rotational joints of the second skeleton and translation of the linear joint or joints of the second skeleton, and the posture is further based on a twist parameter of one of the rotational joints of the second skeleton. A method characterized by the above. Claim 2 The method according to claim 1, wherein the second skeleton includes linear joints for all of its bones. Claim 3 The method according to claim 2, wherein step d) includes determining rotation of the rotational joints of the second skeleton and translation of the linear joint or joints of the second skeleton such that the relative configuration remains unchanged. Claim 4 The method according to any one of claims 1 to 3, wherein step d) includes transforming the transformation matrix for each joint of the second skeleton by successively considering the joints starting from the root joint. Claim 5 The method according to any one of claims 1 to 4, wherein step a) includes causing the first skeleton and the second skeleton to assume an initial posture respectively. Claim 6 e) further comprising the step of applying the calculated transformation matrix to the joints of the second skeleton, the second skeleton taking a pose that matches the pose of the first skeleton, The method according to any one of claims 1 to 5, characterized in that.

7. f) further comprising the step of displaying at least one of the second skeleton and the skin associated therewith on a computer screen, The method according to any one of claims 1 to 6, characterized in that.

8. The method according to any one of claims 1 to 7, characterized in that the first skeleton and the second skeleton are the same digital body model.

9. A computer program stored on a non-transitory computer-readable data storage medium containing computer-executable instructions, which, when executed by a processor, cause the computer system to perform the method according to any one of claims 1 to 8, A computer program characterized by that.

10. A non-transitory computer-readable data storage medium containing computer-executable instructions, which, when executed by a processor, cause the computer system to perform the method according to any one of claims 1 to 8, A non-transitory computer-readable data storage medium characterized by that.

11. A computer system including a processor coupled to a memory, the memory storing computer-executable instructions which, when executed by the processor, cause the computer system to perform the method according to any one of claims 1 to 8, A computer system characterized by that.

Images