Method, program, medium and system for aligning first and second virtual human body models

Abstract

The invention relates to a computer-implemented method and system for aligning a first virtual human body model and a second virtual human body model, a computer program and a computer readable storage medium. The method comprises the following steps: acquiring a first posture of a first virtual human body model matched with a first posture of a second virtual human body model; the method comprises the following steps: determining an offset between a position of a joint of a second virtual human body model and a position of a joint of a first virtual human body model associated with the joint of the second virtual human body model; the method comprises the following steps: acquiring an action of the second virtual human body model; the method includes determining, for each second gesture of the action, a second gesture of the first virtual human body model that matches a second gesture of the second virtual human body model. The determined offset between the associated joints is maintained in each determined second pose. This method forms an improved solution for aligning the first virtual human body model and the second virtual human body model.

Description

Technical Field

[0001] The present invention relates to the field of computer programs and systems, and more particularly to a method, system, program and storage medium for aligning a first virtual human model with a second virtual human model. Background Art

[0002] Numerous systems and programs are available on the market for the design, engineering, and manufacturing of objects. CAD stands for Computer-Aided Design and, for example, refers to software solutions for object design. CAE stands for Computer-Aided Engineering and, for example, refers to software solutions for simulating the physical behavior of future products. CAM stands for Computer-Aided Manufacturing and, for example, refers to software solutions for defining manufacturing processes and operations. In these computer-aided design systems, graphical user interfaces play a significant role in technical efficiency. These technologies can be embedded in product lifecycle management (PLM) systems. PLM refers to a business strategy that helps companies share product data, apply common processes, and leverage corporate knowledge throughout the product development process, from conceptualization to end-of-life, within the concept of the extended enterprise. PLM solutions offered by Dassault Systèmes (through the trademarks CATIA, ENOVIA, and DELMIA) provide an engineering center that organizes product engineering knowledge, a manufacturing center that manages manufacturing engineering knowledge, and an enterprise center that integrates and connects these two centers. All of this enables the system to provide an open object model that links products, processes, and resources to enable dynamic, knowledge-based product creation and decision support, thereby driving optimized product definition, manufacturing preparation, production, and service.

[0003] In these CAD solutions, virtual human models can be used to simulate humans in a virtual environment to design, evaluate, and assess the fit of objects in the early stages of design, long before actual real-world prototypes are implemented. For example, these virtual human models can be used to check whether the design of an object fits the human body of various shapes and sizes or to provide a realistic view of the object with the human body in place. Clearly, existing digital human modeling (DHM) solutions are able to accurately simulate the human body in various shapes and sizes, including pose modeling.

[0004] A major limitation of the use of virtual human models is their positioning. In fact, the positioning of virtual human models involves determining the optimal skeleton shape and shaping the skin (which wraps the skeleton) according to the skeleton so that the resulting posture looks natural, realistic, and suitable for the task scenario. In this regard, significant progress has been made from traditional methods to predictive technologies based on artificial intelligence (AI). The literature on geometric modeling provides a large number of techniques for creating virtual human models of different shapes based on the posture changes of the skeleton. However, the positioning of virtual human models remains complex and time-consuming, and there is no effective method to automatically achieve this task.

[0005] In particular, the lack of proper standardization and application-specific modeling have resulted in virtual human models, ranging from simple to complex, that mimic human behavior with varying degrees of accuracy. Multiple DHM applications are commercially available with varying capabilities: some emphasize anthropometry and linking structural accuracy, others excel at posture / motion modeling and biomechanics, and still others prioritize behavioral and cognitive aspects. Due to this variability, virtual human models also inherit the interoperability issues present in CAD systems. With the increasing use of multiple digital human applications within the same organization or across multiple collaborating organizations, the need for communication between these applications to facilitate data exchange is increasing.

[0006] As a result, different virtual mannequins are often involved in the design of the same product. For example, one virtual mannequin might be used for ergonomic calculations, while another might be used for virtual rendering. However, there's no effective way to replicate the poses of these different virtual mannequins relative to each other, primarily because they don't share the same joints. Consequently, each virtual mannequin used needs to be manually repositioned, even when performing the same action. Once the action of the first virtual mannequin is determined, it's helpful to automatically position the other virtual mannequins based on the same action, but there's no effective way to do this automatically and accurately.

[0007] In this context, there remains a need for an improved solution for aligning a first virtual human model and a second virtual human model. Summary of the Invention

[0008] Therefore, a computer-implemented method for aligning a first virtual human model with a second virtual human model is provided. The method includes obtaining a first pose of the first virtual human model that matches a first pose of the second virtual human model. Each virtual human model includes joints, wherein each joint represents a corresponding joint connection, and segments, wherein each segment connects a corresponding pair of joints. Each corresponding joint of at least a portion of the joints of the first virtual human model is associated with a corresponding joint of the second virtual human model. Each joint of the first virtual human model and each joint of the second virtual human model, when associated together, represent the same corresponding joint connection. The method includes, for each given joint of the first virtual human model that is associated with a given joint of the second virtual human model, determining an offset between a position of the given joint of the first virtual human model and a position of the given joint of the second virtual human model. The offset determined for at least one joint association is non-zero. The method includes obtaining a motion of the second virtual human model. The motion includes one or more second poses of the second virtual human model. The method includes, for each second pose of the motion, determining a second pose of the first virtual human model that matches the second pose of the second virtual human model. The determined offsets between the associated joints are maintained in each determined second pose.

[0009] The method may include one or more of the following:

[0010] The first virtual human model includes at least one first pair of associated joints connected by a single segment. The second virtual human model includes, for each first pair, two or more segments between joints associated with the joints of the first pair. The length of the single segment connecting each first pair in the first pose is different from the length in at least one second pose.

[0011] Each virtual human model's joints include at least four distal joints. For each virtual human model, the segments sequentially connect multiple pairs of joints up to the distal joints. Each distal joint of the first virtual human model is associated with a corresponding distal joint of the second virtual human model. An offset determined for each association of distal joints is zero.

[0012] Each distal joint has an orientation. In each determined second posture, the orientations of the associated distal joints are equal.

[0013] Obtaining the first pose of the first virtual human model that matches the first pose of the second virtual human model includes minimizing cumulative distances between joint associations.

[0014] The first virtual human model includes at least one joint connected to two associated joints. Determining a second pose of the first virtual human model includes aligning the at least one joint with the two associated joints.

[0015] Each virtual human body model represents a corresponding skin shape. The first virtual human body model includes a mesh having a plurality of vertices located on the skin shape represented by the first virtual human body model. The second virtual human body model includes surface landmarks located on the skin shape represented by the second virtual human body model. The method further includes, after determining the second posture of the first virtual human body model that matches the second posture of the second virtual human body model, deforming the mesh of the first virtual human body model to match the surface landmarks of the second virtual human body model.

[0016] Deforming the mesh includes, for each segment of the first virtual human model,

[0017] determining vertices of the mesh belonging to the segment of the first virtual human model; and,

[0018] modifying the positions of the determined vertices so that the resulting mesh includes the surface landmarks;

[0019] Deforming the mesh further includes, before modifying the position, determining a local coordinate system comprising a first axis along the segment and two second axes perpendicular to each other and to the first axis. The determined vertices have coordinates in the determined local coordinate system. Modifying the position includes, for each determined vertex, modifying the coordinates of the vertex along the two second axes.

[0020] Deforming the grid further comprises: for each pair of consecutive surface markers along the first axis,

[0021] determining a shape function between the pair of consecutive surface landmarks; and,

[0022] estimating a deformation factor to be applied to the mesh between the pair of surface landmarks based on the determined shape function and the interpolation function, and,

[0023] Wherein, modifying the coordinates of the vertex along the two second axes comprises: multiplying the coordinates of the vertex along the two second axes by the estimated deformation factor;

[0024] The number of joints of the first virtual human model is lower than the number of joints of the second virtual human model; and / or,

[0025] The first virtual human body model is a skeleton with ergonomic accuracy and / or the second virtual human body model is a skeleton with biomechanical accuracy.

[0026] Furthermore, a computer program is provided, comprising instructions for executing the method.

[0027] Furthermore, a computer-readable storage medium is provided, on which the computer program is recorded.

[0028] Further, a system is provided, comprising a processor coupled to a memory and a graphical user interface, the memory having the computer program recorded thereon.

[0029] Furthermore, a device is provided, which includes a data storage medium, on which the computer program is recorded.

[0030] The device may form or serve as a non-transitory computer-readable medium, for example, on a Software as a Service (SaaS) or other server or cloud-based platform. The device may optionally include a processor connected to the data storage medium. The device may thus form a computer system in whole or in part (e.g., the device is a subsystem of the entire system). The system may further include a graphical user interface connected to the processor. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] Non-limiting embodiments will now be described with reference to the accompanying drawings.

[0032] Figure 1 A flow chart illustrating an embodiment of the method is shown.

[0033] Figure 2 Examples of different forms of virtual human body model representations are shown.

[0034] Figure 3 Determining a pose of a first virtual human model that matches a pose of a second virtual human model is shown.

[0035] Figure 4-Figure 7 It is shown that S10 acquires a first posture of the first virtual human model that matches a first posture of the second virtual human model.

[0036] Figure 8 The joint flexibility process is shown.

[0037] Figures 9-11 It is shown that S40 determines the second posture of the first virtual human model that matches the second posture of the second virtual human model.

[0038] Figure 12-17 Deforming the mesh of a first virtual human model to match a second virtual human model is shown.

[0039] Figures 18-20An embodiment of the alignment result between the source and target virtual human models obtained by using this method is shown.

[0040] Figure 21 An embodiment of a system is shown. DETAILED DESCRIPTION

[0041] refer to Figure 1 A computer-implemented method for aligning a first virtual human model with a second virtual human model is provided, according to a flowchart. The method includes: (S10) obtaining a first posture of the first virtual human model that matches a first posture of the second virtual human model. Each virtual human model includes joints and segments, wherein each joint represents a corresponding articulation, and each segment connects a corresponding pair of joints. Each corresponding joint of at least a portion of the joints of the first virtual human model is associated with a corresponding joint of the second virtual human model. Each joint of the first virtual human model and each joint of the second virtual human model, when associated together, represent the same corresponding articulation. The method includes: (S20) determining, for each given joint of the first virtual human model that is associated with a given joint of the second virtual human model, an offset between a position of the given joint of the first virtual human model and a position of a given joint of the second virtual human model. The offset determined for at least one of the joint associations is non-zero. The method includes: (S30) obtaining a gesture of the second virtual human model. The gesture includes one or more second gestures of the second virtual human model. The method comprises: for each second posture of the action, determining, at step S40, a second posture of the first virtual human model that matches the second posture of the second virtual human model, wherein the determined offsets between the associated joints are maintained in each determined second posture.

[0042] This method forms an improved solution for aligning a first virtual human model with a second virtual human model.

[0043] Notably, this method allows for the automatic reproduction of the movements of a second virtual human model (hereinafter referred to as the "source human model") onto a first virtual human model (hereinafter referred to as the "target human model"). In effect, for each second pose of the movement, the method determines a second pose for the first ("target") virtual human model that matches the second pose of the second ("source") virtual human model. This method thus allows for the reuse of movements already determined for another virtual human model (the "source" virtual human model), thus avoiding the need to manually reposition the "target" virtual human model so that it performs the same movement as the "source" virtual human model.

[0044] In particular, the method accurately and efficiently reproduces the movements of the "source" virtual human model. In practice, the differences between the two virtual human models result in joint offsets between identical joints to ensure they are in realistic poses. By maintaining these offsets, the method allows the motion to be reproduced for the "target" virtual human model, making each determined pose of the "target" virtual human model more realistic. Furthermore, maintaining the offsets between associated joints is faster and computationally cheaper than re-determining the equivalent pose each time. Therefore, the method is particularly efficient at achieving these realistic poses.

[0045] The method is computer-implemented. This means that the steps (or substantially all of the steps) of the method are performed by at least one computer or any similar system. Thus, the steps of the method may be fully or semi-automatically performed by a computer. In an example, at least some of the steps of the method may be triggered by user-computer interaction. The level of user-computer interaction required may depend on the desired level of automation and be balanced against the need to implement the user's expectations. In an example, the level may be user-defined and / or predefined.

[0046] A typical example of a computer-implemented method is to perform the method using a system suitable for the purpose. The system may include a processor coupled to a memory and a graphical user interface (GUI), wherein the memory has recorded thereon a computer program comprising instructions for performing the method. The memory may also store a database. The memory is any hardware suitable for such storage, which may include multiple different physical parts (e.g., one part for the program and one part for the database).

[0047] The method also provides for rendering the first virtual human model. For example, the method further includes, within a rendering process, rendering the first virtual human model in each of the determined second poses after executing the method. For example, in each of the second poses of the action, the second virtual human model may be positioned relative to the object. In this case, the first virtual human model may also be positioned relative to the object in each of the determined second poses. The rendering may thus be rendering the first virtual human model positioned relative to the object.

[0048] For example, an object may include a location where a human can be positioned, such as a seat in a car or motorcycle, an aircraft cockpit, or a workplace environment similar to an assembly process layout. In this case, a second virtual human model can be positioned at the location of the human it represents within the object, and each second pose of the action can be the pose of the second virtual human model at that location. For example, the second pose of the action can be used to verify the technical constraints of the human, such as the ergonomics of the position or safety related to external events, using the second virtual human model. For example, the second virtual human model can be a biomechanically accurate skeleton. Alternatively, the first virtual human model can be used to add a human character to an image / video rendering of the object. For example, the first virtual human model can be an ergonomically accurate skeleton. The rendering process can include capturing an image for each determined pose of the first virtual human model. When multiple poses are determined, the rendering process can then include generating a video of the object using the captured images (e.g., by sequentially programming their display) in which the human represented by the first virtual human model is positioned relative to the object.

[0049] The first posture acquisition step S10 is now discussed in more detail.

[0050] The acquisition S10 may include an initial matching step for calculating a first posture of the first virtual human model that matches a first posture of the second virtual human model. The initial matching step may consider one of the two virtual human models to be in a reference posture and may include calculating a posture of the other virtual human model that matches the reference posture. For example, the initial matching step may consider the first posture of the first virtual human model to be a reference posture and may include calculating a first posture of the second virtual human model that matches the first posture of the first virtual human model. Alternatively, the initial matching step may consider the first posture of the second virtual human model to be a reference posture and may include calculating a first posture of the first virtual human model that matches the first posture of the second virtual human model.

[0051] In both cases, the reference posture may be, for example, a posture of the virtual human model standing upright. Acquiring step S10 may include downloading the reference posture from a database to obtain the reference posture. For example, the reference posture may be a standard posture of the virtual human model. Alternatively, acquiring step S10 may include determining the reference posture through user actions, such as moving a joint of the virtual human model to cause the virtual human model to assume the reference posture.

[0052] We now discuss how the initial matching step computes a pose for one of the virtual human models that matches the pose of the other virtual human model. These details apply regardless of whether the first or second virtual human model is considered to be in a reference pose.

[0053] The initial matching step may initially include obtaining joint associations between the two virtual human models. For example, the user may input the joint associations. Alternatively, the joint associations may have been determined and recorded prior to performing the method, and obtaining the joint associations may include obtaining them.

[0054] Then, the initial matching step may include determining the position of each of the joints of the virtual human model whose posture has been determined. The calculated posture matches the posture of another virtual human model. This means that in this posture, the position and orientation of the distal joints of the first and second virtual human models, such as fingers, wrists, elbows and ankles, are accurately matched, while the proximal joints, such as the pelvis, abdomen, chest, etc., have a certain offset, and the cumulative offset distance between the associated joints of the two virtual human models is minimized. The cumulative distance can be equal to the sum of the distances between the associated joints (i.e., the norm of the offsets generated). Calculating the posture may include minimizing the cumulative distance between the associated joints.

[0055] The minimization of the cumulative distance is performed while adhering to one or more other constraints, including, for example, some or all of the following constraints: A first constraint may be that the offset distance between the associated distal joints of the two virtual human models is (e.g., approximately) equal to zero. A second constraint may be that the orientations of the associated distal joints of the two virtual human models are (e.g., approximately) the same. A third constraint may be that the offset distance between the joints representing the shoulders of the two virtual human models is (e.g., approximately) equal to zero.

[0056] Thus far, the initial matching step has been described as being included in the method. However, in other examples, the initial matching step may be performed before the method and further include: after calculating the first posture of the first virtual human model that matches the first posture of the second virtual human model, recording (e.g., in a memory) the first posture of the first virtual human model that matches the first posture of the second virtual human model. In this case, S10 obtaining may include retrieving the recorded first posture of the first virtual human model that matches the first posture of the second virtual human model. The initial matching step may also include any of the features described so far in this case.

[0057] The virtual human body model is now discussed in more detail.

[0058] Each virtual human model includes joints and segments, where each joint represents a corresponding joint connection, and each segment connects a corresponding pair of joints. The combination of joints and segments is called the skeleton of the virtual human model. In an example, the virtual human model can represent the corresponding skin shape of a human body and can include other elements representing the skin shape (e.g., meshes, surface landmarks, as described later). The skeleton of the virtual human model will now be discussed in more detail.

[0059] Each virtual human body may include a corresponding joint for each major articulation of the human body. For example, each virtual human body model may include a corresponding joint for each of the shoulders, elbows, wrists, hips, knees, ankles, neck, one or more phalanges of the fingers, and / or one or more vertebral joints (e.g., two or three vertebral joints distributed along the spine).

[0060] These joints representing the major joint connections can be associated in the two virtual human models. Each of these associations associates a corresponding joint of the first virtual human model with a corresponding joint of the second virtual human model representing the same corresponding joint connection in the virtual human model to which it belongs. Parts of these associated joints, such as fingers, wrists, elbows, ankles, etc., can be distal joints (others are proximal joints). For example, each virtual human model can include at least four distal joints corresponding to the respective distal joints of the wrists and ankles. Each virtual human model can also further include corresponding distal joints corresponding to the neck and one or more phalanges of the fingers.

[0061] Each virtual human model may further include a root joint representing the pelvis (defining the proximal and distal joints relative to the pelvis) and / or one or more additional joints representing other human motions, such as motion between the clavicle and vertebrae and / or motion of the elbow. These one or more additional joints may be independent and may differ between the two virtual human models. For example, for the second virtual human model, the additional joints may further include multiple joints representing additional vertebral joints. The number of joints in the first virtual human model may be lower than the number of joints in the second virtual human model.

[0062] The offset determination step S20 is now discussed in more detail.

[0063] When the two virtual human models are in their respective first postures, the determination step S20 is performed. This means that the offset between the positions of the joints of the two virtual human models is determined when they are in their respective first postures. For each virtual human model, the position of the joint considered is the position of the joint in the first posture.

[0064] For each association of joints, the method determines a corresponding offset. The offset can be a vector between the two positions of the associated joints. For example, the vector can point from the position of a given joint of a first virtual human model to the position of a given joint of a second virtual human model, or vice versa. Its norm gives the offset distance between the joints. The offset vector can be defined in two coordinate systems: one is a local coordinate system attached to the joint, and the other is a world coordinate system.

[0065] The determination of each offset may be based on the coordinates of the positions of the associated joints. For example, determining the offset between two associated joints may include obtaining the coordinates of the two associated joints (i.e., when the two virtual human models are in their respective first poses), and subtracting these coordinates to calculate a vector between the two associated joints.

[0066] The offset magnitude determined may be zero or non-zero. The offset magnitude may be the length of the offset (i.e., the norm of the vector). In particular, at least one of the offsets determined is non-zero in magnitude. This means that, for at least one association of two joints, the two joints are not precisely superimposed. When the two virtual human models are in their respective first poses, a non-zero offset exists between the two superimposed joints.

[0067] Each offset can be determined in a local coordinate system attached to a joint (e.g., a joint of the second virtual human model) based on which the offset is determined. Each offset can also be determined in a world coordinate system (i.e., constant regardless of pose and virtual human model).

[0068] The action acquisition step S30 is now discussed in more detail.

[0069] The captured action may include only one second pose for the second virtual human model. In this case, the method determines only one corresponding second pose for the first virtual human model. This single second pose can be used to capture an image of the first virtual human model in that pose (e.g., in connection with the subject discussed previously). Alternatively, the action may include multiple poses, e.g., multiple poses that can be connected one after the other to form the action performed by the second virtual human model. In this way, the method determines multiple corresponding second poses for the first virtual human model, which can be used to capture multiple images or videos of the first virtual human model performing the action (e.g., in connection with the subject discussed previously).

[0070] Acquiring the action includes acquiring each second posture of the action. Acquiring each second posture can be performed in the same manner as acquiring the reference posture in S10. For example, the action can be pre-determined and recorded in a database, and acquiring the action can include retrieving the action from the database. For example, the action can be determined using another application, such as to verify ergonomic constraints using a second virtual human model. Alternatively, acquiring the action can include determining each second posture through the user's action, such as moving a joint of the second virtual human model so that the second virtual human model is in each second posture.

[0071] Now, let's discuss step S40 of determining a second posture for the first virtual human model. When the action includes only one second posture, the method determines only a single second posture for the first virtual human model. When the action includes multiple second postures, the method determines a corresponding second posture for the first virtual human model for each second posture. In this case, the method can determine all second postures for the first virtual human model simultaneously or sequentially.

[0072] The determination step S40 of a given second posture is now discussed in more detail. However, when the action comprises a plurality of second postures, these details apply equally to any one of the determined second postures.

[0073] The determined second posture of the first virtual human model matches the second posture of the second virtual human model. However, the second posture is not determined using a matching step like the initial matching step discussed above with respect to step S10. The second posture is determined by maintaining the determined offsets between the associated joints. In other words, the determined offsets can be constant in all postures (i.e. in the local coordinate system attached to the joint by which the offsets are determined). This means that the second posture can be inferred directly from the positions of the joints of the second virtual human model and the determined offsets, without the need to perform the aforementioned minimization. This is faster and computationally cheaper than redetermining the same posture each time.

[0074] S40 determining the given second posture may include inferring the given second posture from the second posture of the second virtual human model and the determined offset. For example, the determined offset may be embedded in the initial relative configurations. The determined offset may be expressed using a transformation matrix (e.g., from the joint position of the second virtual human model to the position of the joint of the first virtual human model) and the method includes using these transformation matrices to infer the given second posture of the first virtual human model from the second posture of the second virtual human model. For example, the method includes applying these transformation matrices to the joints of the second virtual human model in the second posture, thereby obtaining the position of the joint of the first virtual human model associated with the joint, thereby obtaining the given second posture of the first virtual human model.

[0075] Each offset can be maintained in a local coordinate system attached to the joint for which the offset is determined. The method can thereby determine the position of the associated joint of the first virtual human model in the corresponding local coordinate system, but can then transpose that position in the world coordinate system (taking into account the orientation and position of the corresponding local coordinate system relative to the world coordinate system). For distal joints, the determined position can be the same as the position of the second virtual human model (zero offset for these joints). For other associated joints with non-zero offsets, the determined position can be different from the determined offset.

[0076] In an example, the first virtual human model may include at least one first pair of associated joints connected by a single segment. This means that, for each first pair, the first virtual human model does not include any additional joints between two associated joints of the first pair (these joints being associated with corresponding joints of the second virtual human model). In this case, the second human model may include, for each first pair, two or more segments between the joints associated with the joints of this first pair. Thus, in contrast to the first virtual human model, the second virtual human model includes additional joints between the joints of the corresponding pairs. This may be the case, for example, for vertebral joints, where the second virtual human model includes more of these joints than the first virtual human model.

[0077] In that case, for the first virtual human model, the length of the individual segments connecting each first pair in the first pose can be different from the length in at least one second pose. In other words, for the first virtual human model, the segment size is variable between the initial first pose and the determined second pose. This improves the accuracy of the pose determined for the first virtual human model. In practice, maintaining a constant length for each segment would result in variations in the offset of the associated joints, which would reduce the accuracy of the rendering obtained using the virtual human model as described above.

[0078] In an example, each distal joint may have an orientation. The orientation of the distal joint may be represented by a corresponding rotation matrix. The corresponding rotation matrix may comprise columns, wherein each column represents a corresponding axis of a coordinate system attached to the distal joint. For example, the rotation matrix representing the distal joint of the wrist may determine the orientation of the hand. In that case, the orientation of the distal joint may also be the same as the orientation of the second virtual human model. The orientations between the associated distal joints in each determined second posture are equal. For example, the method may comprise: for each distal joint of the first virtual human model, obtaining the orientation of the distal joint associated therewith, and applying the orientation to the distal joint in the determined second posture. In other words, for each second posture adopted by the second skeleton, the orientations of the distal joints of the first and second virtual human models always correspond.

[0079] In this example, before executing steps S10 to S40, some of the associated joints may be ignored (i.e., they may be disassociated before executing steps S10 to S40). In the offset determination step S20, each ignored joint may be disregarded. The joints associated before executing steps S10 to S40 are referred to as anchor joints, and the number of associated joints at the time of executing steps S10 to S40 may be lower than the number of anchor joints (i.e., some of them may be ignored).

[0080] For example, before performing steps S10 to S40, at least one joint may be associated with a corresponding joint of the second virtual human model, and the method may include: removing the association of each joint in the at least one joint before determining the offset in S20. Thus, when performing step S20, the at least one joint may be unassociated and no offset is determined for the at least one joint (i.e., the at least one joint may be ignored). Then, in step S40, to determine the second pose of the first virtual human model, rather than placing the at least one joint using an offset (as previously discussed with respect to associated joints), the method may include aligning the at least one joint with two associated joints to which it is connected. For example, the method may position the at least one joint on a straight line connecting the two associated joints to which it is connected, such as at an equivalent distance between these two associated joints when in the first position. Ignoring the joints between the two associated joints may reduce the risk of over-deforming the skin mesh. It also provides significant advantages in controlling the deformation of the skin mesh as required.

[0081] In an example, each virtual human model may represent a corresponding skin shape of a human body. The skin shape may be represented differently in the two virtual human models. For example, the first virtual human model may include a mesh having a plurality of vertices located on the skin shape represented by the first virtual human model. The second virtual human model may include surface landmarks located on the skin shape represented by the second virtual human model.

[0082] In that case, for each second pose determined in step S40 (i.e., after determining the corresponding position for each joint of the first virtual human model in the second pose), the method may include: deforming the mesh of the first virtual human model to match the surface landmarks of the second virtual human model. The position of the surface landmarks is the position of the surface landmarks when the second virtual human model is in the second pose under consideration. The position of the surface landmarks may vary between different poses. The position of the surface landmarks of the second virtual human model may be predetermined (e.g., similar to the position of the second virtual human model in the second pose) or automatically calculated (e.g., taking into account the position of the joints of the second virtual human model in the second pose). For example, the position of the surface landmarks of the second virtual human model depends on the body shape of the represented human body. The deformation of the mesh of the first virtual human model may be performed so that the resulting deformed mesh passes (at least approximately) through all surface landmarks (e.g., with a tolerance of 2 to 5 mm). The deformation of the mesh makes it possible to reproduce the body shape of the represented human body for the first virtual human model, thereby improving the realism of the action reproduction.

[0083] Examples of how mesh deformation is performed for each second pose determined in step S40 are now shown in greater detail. These examples enable particularly efficient and realistic mesh deformation while reducing the risk of unrealistic deformation, thereby enabling particularly optimal subsequent rendering. However, in other examples, the method may employ any other deformation method to match the mesh of the first virtual human model to the surface landmarks of the second virtual human model.

[0084] In an example, each segment of the first virtual human model may be considered separately in deforming the mesh, and for each segment, vertices belonging to the segment may be modified. For example, the method may include, for each segment of the first virtual human model, determining mesh vertices belonging to the segment of the first virtual human model and modifying the positions of the determined vertices such that the resulting mesh includes surface landmarks. The surface landmarks may be surface landmarks of the second virtual human model belonging to the portion defined by the segment.

[0085] In an example, the positions of the determined vertices can be modified along a direction perpendicular to the segment. For example, deforming the mesh can further include, before modifying the positions, determining a local coordinate system comprising a first axis along the segment and two second axes perpendicular to each other and perpendicular to the first axis. The determined vertices can have coordinates in the determined local coordinate system. Modifying the positions can include, for each determined vertex, modifying the coordinates of the vertex along the two second axes (i.e., perpendicular to the segment). This can prevent the mesh from being distorted unrealistically.

[0086] In an example, a shape function defined along the segment can be used to control the deformation of the mesh. In particular, the method can consider a shape function for each pair of consecutive surface markers along the first axis. In that case, the method can further include: for each pair of consecutive surface markers along the first axis, determining a shape function between the pair of consecutive surface markers, and estimating a deformation factor to be applied to the mesh between the pair of surface markers based on the determined shape function and the interpolation function. Then, to modify the coordinates of each vertex along the two second axes, the method can include multiplying the coordinates of the vertex along the two second axes by the estimated deformation factor. This allows for fast and efficient deformation of the mesh and ultimately achieves an optimal mesh deformation.

[0087] refer to Figures 2 to 21 , an example of an implementation of this method is now described.

[0088] In the context of digital human modeling (DHM), pose prediction involves determining the optimal virtual human model configuration, where the configuration includes the positions of the virtual human model's joints and segments (called the virtual human model's skeleton), and deforming the skin according to the skeleton (skin wrapped around the skeleton) so that the resulting pose looks natural, realistic, and appropriate for the task scenario. Clearly, existing digital human modeling (DHM) solutions are able to accurately mimic the human body in various shapes and sizes in terms of pose modeling, such as Figure 2 shown.

[0089] In this area, significant progress has been made, moving from traditional methods to predictive techniques based on artificial intelligence (AI). The literature on geometric modeling offers numerous techniques for creating DHMs of varying shapes based on the pose of the skeleton. However, the positioning of virtual human models remains complex and time-consuming, and there is currently no effective method for automating this task.

[0090] The lack of proper standardization and application-specific modeling results in varying degrees of accuracy in simulating human behavior, from simple to complex DHMs. Multiple DHM applications are commercially available with varying capabilities: some prioritize anthropometrics and linking structural accuracy, others excel at posture / motion modeling and biomechanics, and still others focus on behavioral and cognitive aspects. Due to this variability, DHMs also inherit the interoperability issues present in CAD systems. With the increasing use of multiple digital human applications within the same organization or across collaborating organizations, the need for communication between these applications is increasing.

[0091] In the context of pose sharing between two or more human applications, it can be useful to match the shape of a target human model to that of a source human model, in addition to the skeletal configuration. In constrained environments, skeleton-to-shape matching is crucial. In such environments, aspects such as fit and adaptability cannot be predicted by skeletal configuration alone, while the shape and volume of the skin play a key role. Existing solutions do not address this aspect, especially when considering communication between two human applications with different skeletal and shape configurations.

[0092] Figure 3 The figure shows determining a posture of the first virtual human model 100 that matches the posture of the second virtual human model 200. The goal is to adapt the target virtual human model 100 (i.e., the first virtual human model) to the source virtual human model 200 (i.e., the second virtual human model) represented by the skeleton and surface landmarks by manipulating the skeleton and surface mesh of the target virtual human model 100, thereby obtaining a posture 300 that matches the posture of the source virtual human model 200.

[0093] Existing approaches to solving the pose mapping problem involve modeling the entire structure at once and treating it as an inverse kinematics problem. However, this requires modeling all joints from the root using the objective function and the initial relative configuration as constraints. Since the human structure has kinematic and dynamic redundancies across hundreds or even thousands of joints, converging to a feasible solution is a tedious task.

[0094] Furthermore, when the source skeleton assumes a new pose, some or all of its joints change their configuration, and these joints can be common joints or non-common joints (joints on the source skeleton that have no corresponding joints on the target skeleton). However, the target skeleton only has common joints, and achieving the same pose can only be obtained by manipulating the common joints while maintaining the initial relative configuration. This further complicates the pose mapping problem.

[0095] This method addresses the aforementioned limitations of existing solutions. Significantly, the method provides a comprehensive skeleton-driven joint flexible approach that allows the skeletal pose and shape of a target virtual human model to be matched to the skeletal pose and shape of a source virtual human model. In particular, the method allows the pose of a high-degree-of-freedom virtual human model (the source virtual human model) with specific anthropometric features and shape to be mapped onto a relatively low-degree-of-freedom virtual human model (the target virtual human model), particularly by matching skeletal shape, size, and pose.

[0096] The method aims to synchronize the shape and pose between two human models, which may include:

[0097] Performing an initial matching step to obtain S10 a first posture of the target virtual human model that matches the first posture of the source virtual human model;

[0098] S20 determines the offset between the associated joints;

[0099] determining, S40 , a second posture of the target virtual human model that matches the second posture of the source virtual human model in the acquired motion, for example by modifying the length of the segments of the target virtual human model;

[0100] Maintaining the determined offsets between associated joints;

[0101] Ignore one or more anchor joints; and / or,

[0102] Adjust the size of the skin shape mesh to match.

[0103] Matching the size of the skeleton bones involves determining the joints to be linked and changing the length of the bones by the offsets selected for the linked joints. Figure 4 As shown, initially, the target virtual human model 100 and the source virtual human model 200 may have different heights due to different bone lengths. Figure 4 The skeleton matching performed as shown uses the concept of the minimum anchor-offset method. The skeleton configuration mapping S10 includes matching the end effectors (hands and legs) of the two human models for all configurations without disturbing the relative configuration of the anchor joints. The method also has the flexibility to make the anchor joints active or passive (i.e., they can be ignored or not ignored in steps S10 to S40) depending on the synchronization and deformation of the skin mesh.

[0104] In this method, the pose of the human body model can be considered using the transformation matrix on the joints. Each joint has its own local coordinate system and the pose is expressed in the global / world coordinate system. Figure 4 As shown, the initial pose of the target skeleton is different from that of the source skeleton. Once the two skeletons are initially synchronized at S10 (with appropriate offsets for the spine / torso and zero offsets for the hands), for each new pose assumed by the source skeleton, the target skeleton matches its pose by maintaining the initial relative configuration of each shared joint constant.

[0105] Maintaining the initial relative configuration constant across all pose changes is difficult. This challenge stems from the lack of a one-to-one correspondence between the source and target skeletons due to the larger number of joints in the source skeleton. For example, if the configuration of the source skeleton's torso joints changes, synchronizing the target skeleton and maintaining the initial relative configuration is challenging. This method addresses this issue by modifying the bone lengths between the associated joints (the so-called joint flexibility process).

[0106] Therefore, the advantages of the proposed method include that it allows interoperability between two different virtual human models, allows for real-time pose and shape matching between two different virtual human models, and uses a manipulator arm with spherical joints. In addition, the proposed method is robust because the mathematical process does not require inverse calculations.

[0107] One requirement for pose and shape mapping between two virtual human models with different architectures is that the target virtual human model's distal pose, defined by the positions of distal segments such as hands and legs, matches that of the source virtual human model. If a hand belonging to the source virtual human model reaches a specific point, the corresponding hand of the target virtual human model can reach the same position and orientation without changing the relative configuration of the associated joints.

[0108] Existing approaches assume that all joints of the target virtual human model coincide with the corresponding joints of the source virtual human model (i.e., using zero-scale offsets). Using zero-scale offsets has limitations when the skeletons are asynchronous (i.e., the associated joints are not in the same position). In reality, the target virtual human model does not synchronize its joints with the source virtual human model, and the resulting pose of the first virtual human model is distorted (see Obtaining the Generated Pose 302 Using the Existing Approach).

[0109] To address this issue, the method determines the offsets between the associated joints of the two virtual human models and maintains these determined offsets for each calculated pose. Figure 5 An example of a pose 301 generated using this method is shown. The figure shows that the generated pose 301 is realistic and adheres to the initial proportions of the virtual human model.

[0110] Now, the acquisition step S10 of the first pose of the first virtual human model is discussed. The acquisition step S10 may include matching the initial configuration (ie, the first pose). The matching may include first determining the associated joints on the two virtual human models.

[0111] Figure 6 Examples of associated joints for two virtual human models are shown. Each virtual human model includes joints, wherein each joint represents a corresponding joint connection of the human body. Parts of these joints (for major joint connections) are associated. The first virtual human model 100 includes a corresponding joint corresponding to each of the one or more phalanges and / or three vertebral joints 103 of the shoulder 102, elbow 104, wrist 107, hip 106, knee 109, ankle 110, neck 101, and fingers 108. Each of these joints is associated with a corresponding joint of the second human model, whereby the second human model also includes a corresponding joint corresponding to each of the one or more phalanges and / or three vertebral joints 203 of the shoulder 202, elbow 204, wrist 207, hip 206, knee 209, ankle 210, neck 201, and fingers 208. Parts of these associated joints are distal joints. In particular, in this example, the joints representing the phalanges of the neck 101, 201, wrist 107, 207, fingers 108, 208, and ankles 110, 210 are distal joints. Rs 105 and Rt 205 are the root (pelvic) joints of the corresponding skeleton.

[0112] The goal is to maintain the relative configuration of these associated joints for all poses of the source human model, with or without offsets (for distal joints). Figure 6 As shown, unassociated joints are free and their rotations are captured at the associated joints.

[0113] The method determines a pose of the first virtual human model that matches the pose of the second virtual human model by minimizing the overall / cumulative offset distance on all associated joints, while constraining the shoulder offset to zero to perform pose synchronization on the end effector. When determining the pose of the first virtual human model that matches the pose of the second virtual human model, the method may include: S42 adjusting the length of the corresponding segment of the first virtual human model so as to maintain the pose as shown in FIG. Figure 7 Initially, the initial matching step may further include: after determining the compensatory angular rotation S12 redirecting the associated joints of the target virtual human model, thereby finally matching Figure 7 The initial configuration is shown.

[0114] Now refer to Figure 8This section discusses the joint flexibility process. Its goal is to map joint parameters from a source skeleton to a target skeleton so that the relative pose (skeleton and shape) is always intact. The more complex skeleton that defines pose with greater biomechanical accuracy is called the source skeleton. The simpler skeleton that defines the motion of the skin is called the target skeleton. Figure 8 Two virtual human models 401 are shown that are initially misaligned.

[0115] like Figure 8 As shown in , the joint flexibility method includes: considering two consecutive joints of the target skeleton and their shapes and modifying their bone sizes and skin shapes near the bones so that the corresponding parts of the target skeleton and the source skeleton are consistent. Figure 10 As depicted, joints At, As and Bt, Bs are not in the same position (see overlay 402 of the two virtual human models). Posture synchronization includes aligning 403 the target virtual human model with the source virtual human model by matching the segment size and skin shape to that of the source virtual human model.

[0116] The kinematic structure is the mathematical equivalent of the skeleton structure and includes joint and link definitions as well as the types of motion. It can include the following factors:

[0117] Define joint types and degrees of freedom (DOF);

[0118] The order in which the degrees of freedom are defined (the order in which the different degrees of freedom are called);

[0119] Define the coordinate system for calculating joint motion;

[0120] Choose an appropriate representation for joint motion (Euler, axis-angle, quaternion);

[0121] defining relationships between consecutive joints; and / or,

[0122] Determine relative motion using local and global transformations.

[0123] There are multiple ways to define these aspects. Depending on the requirements, the method can utilize different approaches to fully define the skeleton structure. For a given body geometry / skinning, various skeletons can be defined depending on the required fidelity. In this case, there are two skeletons, where the pose / motion is defined on the source skeleton and the skinning is defined on the target skeleton. To map the pose / motion, a relationship can be established with the target skeleton so that the relative pose remains intact while the mapping of the joint configuration is achieved.

[0124] The two skeletons are defined independently, so in addition to the number of bones and joints, the way the kinematic structure is defined (including joint types, degrees of freedom, coordinate systems, and kinematic representations) can also be different. These changes in the kinematic structure can be derived from their respective global transformation matrices, as the global transformation matrix preserves information about the kinematic structure.

[0125] The first step in deriving this relationship is to select the joints on the two skeletons for mapping (ie, to associate the joints). Since the target skeleton has a smaller number of joints, it is necessary to determine the equivalent joints on the source skeleton relative to the target skeleton.

[0126] An initial starting pose (i.e. first pose) can be chosen for both skeletons and the global configuration of the shared joints needs to be extracted to derive the relationship (i.e. offset). Typically the initial pose is the same as that obtained from a range scanner or human body rendering application. Figure 3 The 3D rigged human skinning postures 300 shown are the same. Otherwise, the zero-configuration poses on both skeletons can be selected as the initial pose.

[0127] The method may include extracting global configurations (represented as transformation matrices) from the source and target skeletons and determining initial relative configurations between the shared joints of the source and target skeletons. These configurations form a relationship between the source and target skeletons, and for any pose conceived by the source skeleton, appropriate compensations need to be derived for the target skeleton joints so that the initial relative configuration remains unchanged.

[0128] Now, we will discuss determining the second pose for the first virtual human model S40. Rather than modeling the entire structure as an inverse kinematics problem, this method considers two common joints at a time (starting from the root joint) to model a compensatory configuration on the target skeleton for the pose change of the source skeleton. The method also considers a virtual prismatic joint between the two joints on the target skeleton (unlike a hinge joint, a prismatic joint can change the relative distance of the bones), thereby making the bone length flexible.

[0129] like Figure 9 As shown, for the source skeleton, there are multiple intermediate joints between the two common joints. The displacement of joint B relative to joint A can be attributed to some or all of the intermediate joints (A1, A2, A3) until joint A.

[0130] make is the initial transformation matrix of the source skeleton joint B relative to joint A, and it is expressed as in, is the initial orientation of joint B; is the final transformation matrix of B1 and is expressed as ,in is the final rotation matrix and position of joint B. Similarly, is the initial transformation matrix of joint F of the target skeleton relative to joint E and is expressed as in, is the initial transformation matrix of joint E relative to the origin / root.

[0131] The transformation matrix of B1 relative to E can be And can be determined using the following equation (1):

[0132]

[0133] Let the relative initial transformation matrix between B and F be And it is determined using the following equation (2):

[0134]

[0135] Let the distance between AB be dAB, and Similarly, the distance between AB1 is dAB1, and On the target skeleton, the distance between EF is dEF, which is just the length of the bone segment and is obtained from The norm of is determined.

[0136] Maintenance of the determined offset is now discussed in more detail.

[0137] The compensatory motion of shifting joint F to match the position of B1 can be achieved using only joint E. Let the unit vectors of EF and EB1 be p1 and p2, and can be obtained from and Calculate in. Let θ eswing is the angle between p1 and p2, p3 is the unit vector perpendicular to p1 and p2, and these parameters can be determined using geometric relationships. eswing And p3 construct the rotation matrix LR E , and is only the rotation required to match the final position of joint F on the target skeleton to joint B. Even though dAB and dEF are the same, the rotation of the middle joint on the source skeleton causes dAB1 to be different. However, dEF is constant because it is the length of the bone between joints E and F, and there is no θ eswing The rotation amount can match the position of joint F to B1.

[0138] To solve this problem, the method can add a virtual prismatic joint between joints E and F. Manipulation with this joint can change the bone length dEF to match dAB1. When the structure returns to its initial pose, the bone length returns to its original value. The new position vector PEB1 between E and B1 can be used Calculated with dAB1.

[0139] The method may include utilizing LR E Construct a new transformation matrix with PEB1 And append it to the global matrix when When applied to the local transformation of F, joint F is transformed to B1.

[0140] Matching the position of joint F with B1 is a necessary but not sufficient condition for pose mapping. Joint F can be reoriented to respect the initial relative configuration. When joint F matches the position of B1, the unit vectors connected to consecutive branches of the structure are oriented in different directions, such as Figure 10 Using geometric relationships, the method can include calculating the rotation to match the unit vector f1 with b1, as Figure 10 The final position of 404 is shown. However, because of the presence of the torsion component, matching these two unit vectors will not necessarily satisfy the initial relative configuration constraint. Therefore, the method can include determining the torsion component required to satisfy the constraint.

[0141] Let Tbsi, Tbsc be the initial and final global transformations of joint B. Similarly, Tfsi, Tfsc be the initial and final global transformations of joint F, where only the swing component of Tfsc has been calculated.

[0142] The initial relative configuration of joints B and F is given by Equation 2. Since it is useful to maintain this across all pose deformations, the method may include assigning this relationship to the final configuration using the following equation (3):

[0143]

[0144] The method may include calculating the twist component using Tfsc and its determined swing component, thereby ensuring that a pose mapping to the target skeleton is achieved for a set of segments between two common joints on the source skeleton. The same procedure is performed for successive branches of segments until the end effector.

[0145] Now we discuss the neglect of anchor joints in more detail. One advantage of the joint flexibility process is that it can exclude anchor joints from the pose synchronization process and Figure 9 The connection is shown to be reconstructed at the distal segment. This flexibility allows for the elimination of undesirable distortions in the skin mesh.

[0146] like Figure 11 As shown, if the source joint B changes position to B1, there are multiple options for pose synchronization to match the configuration of joint F with B1. The relationship between joint F and joint B can be rebuilt by manipulating bone segment FD2 as shown in position 501 while preserving the anchor relationship, or manipulating bone FD1 to achieve the same relationship by making segment D2 passive (ignored) relative to D1. However, in the first option, the skin mesh is more distorted than in the second option. Therefore, by ignoring redundant joints, the reconstruction and synchronization process with other joints ensures that the skin mesh deforms smoothly. This flexibility provides a huge advantage in controlling the deformation of the skin mesh as required.

[0147] One example of deformation of the mesh of the first virtual human model is now discussed in more detail.

[0148] The goal of this method is to match the pose and shape of the source human model with the target human model. Initially, the size and shape of the target human model are different from the source human model, and its size is adjusted through the pose synchronization process. This method can use a skeleton-based method to match the target skin mesh with the skin of the source mesh. Figure 12 As shown, the input corresponding to the right arm is provided by surface landmarks 601 of the source human body model, and the target human body model is expected to deform the right arm shape 602 according to those surface landmarks 601. The skeleton-based method may include:

[0149] separating the mesh cloud for each bone to determine the vertices of the mesh belonging to the segment of the first virtual human model; and,

[0150] Modify the positions of the determined vertices so that the resulting mesh contains surface markers by:

[0151] Define the local coordinate system;

[0152] Determine shape function;

[0153] deforming the local mesh using shape functions; and

[0154] Converting the local mesh to a global mesh allows visualization of deformations.

[0155] Now let's discuss the separation of the mesh clouds for each bone.

[0156] Since the shape deformation of the human body mainly depends on the mass accumulated around the bones, in this procedure, the method can separate the skin mesh and assign its vertices to each bone between two consecutive joints. For example, considering Figure 13The mesh shown belongs to the right arm. This separation can include attaching the mesh between the shoulder and elbow to the upper arm, and attaching the mesh between the elbow and wrist to the forearm. Initially, all mesh vertices can be represented by the global coordinate system 'O'. This method can be done by Figure 13 The mesh is deformed around the bones by moving mesh vertices in the local coordinate system 'S' shown.

[0157] Now let's discuss the definition of the local coordinate system.

[0158] The definition of the local coordinate system may include inferring the dominant / principal axes for the selected mesh point cloud using principal component analysis (PCA). Skin mass is concentrated around bones, so the method may include selecting one of the principal axes along the joints S and E, such as Figure 14 shown.

[0159] The method can calculate the vector along the bone axis SE and consider it as one of the main axis directions (the first axis). Figure 14 Point 'C' in the grid point cloud shown, and derive vector SC. The method can calculate vector SD (second axis) from the cross product of SE and SC, which are perpendicular to these two vectors. The method can calculate SF from the cross product of SE and SD, which forms a third axis. The method can include normalizing the three main directions SE, SD, and SF and positioning them at the center S to create a coordinate axis system. The method can construct an orthogonal rotation matrix using these coordinate axes, and by locating the position vector at S, a transformation matrix Ts relative to the global transformation matrix To can be constructed. For example, the local transformation matrix Ts is calculated from the product of the inverse global matrix and Ts using the following formula.

[0160] LocTs=inv To*Ts

[0161] The method may include multiplying the global grid point by LocTs to obtain the local grid point relative to the local coordinate system established at S. Let Cs be the global grid point, and the local grid point LocCs=LocTs*Cs.

[0162] Now refer to Figure 15Discussion of shape function determination. The figure shows the lengths and surface landmarks of the bones belonging to the source human body model 701 and the same bones and mesh points on the target human body model 702. During the deformation process, the length StEt matches SoEo (see the resulting overlay 703). The method modifies the position of the mesh vertices along the bones shown in the figure by adjusting the local Z values of the mesh vertices (see the results of the mesh modification 704) without disturbing the X and Y components used to determine the shape of the skin around the bones. The method also modifies the position of the mesh vertices along the X and Y directions so that the new shape contains the surface landmarks of the source human body model (see the resulting mesh deformation 705).

[0163] By designing Figure 16 Shape matching is performed by applying an appropriate shape interpolation function between the two selected surface landmarks as shown.

[0164] Let P1 and P2 be covers Figure 16 Shape functions for the span 'a' of the bone length are shown. The value of P1 is 1 at the proximal end S of the bone and decreases to 0 as the span goes from 0 to 'a' along the length of the bone. Similarly, the shape function P2 changes from 0 to 1 as the span changes from 0 to 'a'.

[0165] The method can use the formula P1 = (ab) / a; where b is the z value of the grid point M (x, y, z) and ranges from 0 to a. Therefore, at S, the value of P1 is 1, and at T when the span is 'a', its value is 0.

[0166] Likewise, the method can utilize the formula P2=b / a; P2 is 0 at S and 1 at position T.

[0167]

[0168] By using these shape functions, the method can estimate various interpolation functions such as Figure 16 The deformation of the grid points between two consecutive surface markers is shown. Let the deformation factors be the factors by which the grid points are deformed along the X and Y directions of the established local coordinate system. The deformation factors are determined according to the following interpolation function:

[0169] Deformation factor 1 = d1*P1 + d2*P2 for linear interpolation

[0170] Deformation factor 2 = d1*P1+d2*P2*P2 for quadratic interpolation

[0171] We now discuss deforming the local mesh using shape functions.

[0172] Once the deformation factor is determined, such as Figure 17As shown, the method may include multiplying the X and Y coordinates of each mesh vertex (or point) by a deformation factor to reshape the mesh to accommodate the surface landmarks of the source virtual human body model. The method may then convert the new local mesh points into the global mesh by multiplying the global transformation matrix obtained from the product of the global coordinate system and the new local coordinate system, for example, using the following formula: Figure 17 As shown:

[0173] GCs=To*LocTs*LCs, where LCs is the local grid point and GCs is the global point.

[0174] Figures 18 to 20 An example of the result of alignment between a source virtual human model and a target virtual human model obtained by using this method is shown. Figure 18 The alignment between the source virtual human model and the target virtual human model obtained by the method is shown for different poses of the source virtual human model. Figure 19 and Figure 20 The shape alignment between the source virtual human model and the target virtual human model obtained by using the method is shown.

[0175] Figure 21 An example of a system is shown, where the system is a client computer system, such as a user's workstation.

[0176] The client computer in this example includes a central processing unit (CPU) 1010 connected to an internal communication bus 1000 and random access memory (RAM) 1070 connected to the bus. The client computer further includes a graphics processing unit (GPU) 1110 connected to video random access memory (RAM) 1100, which is also connected to the bus. In the art, video RAM 1100 is also referred to as a frame buffer. A mass storage device controller 1020 manages access to mass storage devices, such as a hard disk 1030. Mass storage devices suitable for tangibly embodying computer program instructions and data include various forms of non-volatile memory, including, for example, semiconductor memory devices such as erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), and flash memory devices; magnetic disks such as internal hard disks and removable disks; and magneto-optical disks. Any of these may be supplemented or incorporated by specially designed application-specific integrated circuits (ASICs). A network adapter 1050 manages access to a network 1060. The client computer may also include a haptic device 1090, such as a cursor control device, a keyboard, etc. A cursor control device is used in a client computer to allow a user to selectively position a cursor at any desired location on display 1080. Furthermore, the cursor control device allows the user to select various commands and input control signals. The cursor control device includes a plurality of signal generating devices for inputting control signals to the system. Typically, the cursor control device may be a mouse, wherein buttons of the mouse are used to generate signals. Alternatively or additionally, the client computer system may include a sensitive pad and / or a sensitive screen.

[0177] A computer program may include instructions for execution by a computer, the instructions including means for causing the system to perform the method. The program may be recorded on any data storage medium, including the system's memory. The program may be implemented, for example, in digital electronic circuitry or in computer hardware, firmware, software, or a combination thereof. The program may be implemented as a device, such as an article tangibly embodied in a machine-readable storage device, for execution by a programmable processor. The method steps may be performed by a programmable processor executing the instruction program to implement the functionality of the method by operating on input data and generating output. Thus, the processor may be programmable and coupled to receive data and instructions from and send data and instructions to a data storage system, at least one input device, and at least one output device. The application may be implemented in a high-level procedural programming language or an object-oriented programming language, or, if desired, in assembly language or machine language. In any case, the language may be compiled or interpreted. The program may be a fully installed program or an updater. Application of the program to the system may result in instructions for performing the method. The computer program may optionally be stored and executed on a server in a cloud computing environment, which communicates with one or more clients via a network. In this case, the processing unit executes the instructions contained in the program, thereby causing the method to be performed on the cloud computing environment.

Claims

1. A computer-implemented method for aligning a first virtual human model and a second virtual human model, the method comprising: Acquiring (S10) a first posture of the first virtual human model that matches the first posture of the second virtual human model, wherein the first virtual human model and the second virtual human model each include joints and segments, wherein each joint represents a corresponding joint connection, and each segment connects a corresponding pair of joints; each joint of at least a portion of the joints of the first virtual human model is associated with a corresponding joint of the second virtual human model; and each joint of the first virtual human model and each joint of the second virtual human model that are associated together represent the same corresponding joint connection; determining (S20) for each given joint of the first virtual human model associated with a given joint of the second virtual human model an offset between a position of the given joint of the first virtual human model and a position of the given joint of the second virtual human model; wherein the offset determined for at least one association of joints is non-zero; Acquiring (S30) an action of the second virtual human model; wherein the action includes one or more second postures of the second virtual human model; and For each second posture of the action, a second posture of the first virtual human model is determined (S40) to match the second posture of the second virtual human model; wherein the determined offsets between the associated joints are maintained in each of the determined second postures.

2. The method according to claim 1, wherein The first virtual human body model comprises at least one first pair of associated joints connected by a single segment; the second virtual human body model, for each of the first pairs, comprises two or more segments between the joints associated with the joints of the first pair; the length of the single segment connecting each of the first pairs in the first posture is different from the length in at least one second posture.

3. The method according to claim 1 or 2, wherein: The joints of each of the first virtual human model and the second virtual human model include at least four distal joints; for the first and second virtual human models, the segments sequentially connect the multiple pairs of joints until the distal joints; each distal joint of the first virtual human model is associated with a corresponding distal joint of the second virtual human model; and the offset determined for each association of the distal joints is zero.

4. The method according to claim 3, wherein: Each of the distal joints has an orientation; in each of the determined second postures, the orientations of the associated distal joints are equal.

5. The method according to any one of claims 1 to 4, wherein: Acquiring (S10) the first pose of the first virtual human model that matches the first pose of the second virtual human model includes performing minimization on a cumulative distance between associations of joints.

6. The method according to any one of claims 1 to 5, wherein: The first virtual human model includes at least one joint connected to two associated joints; Determining ( S40 ) a second pose of the first virtual human model includes aligning the at least one joint with the two associated joints.

7. The method according to any one of claims 1 to 6, wherein: The first and second virtual human models each represent a corresponding skin shape; the first virtual human model comprises a mesh having a plurality of vertices located on the skin shape represented by the first virtual human model; the second virtual human body model comprising surface landmarks located on the skin shape represented by the second virtual human body model; The method further comprises, after determining (S40) the second pose of the first virtual human model that matches the second pose of the second virtual human model, deforming the mesh of the first virtual human model to match the surface landmarks of the second virtual human model.

8. The method of claim 7, wherein deforming the mesh comprises, for each segment of the first virtual human model: determining vertices of the mesh belonging to the segment of the first virtual human body model; and modifying positions of the determined vertices such that a generated mesh contains the surface landmarks.

9. The method according to claim 8, wherein Deforming the mesh further comprises: prior to modifying the position, determining a local coordinate system comprising a first axis along the segment and two second axes perpendicular to each other and to the first axis; wherein the determined vertex has coordinates in the determined local coordinate system; Modifying the position includes: for each of the determined vertices, modifying the coordinates of the vertex along the two second axes.

10. The method according to claim 9, wherein: Deforming the grid further comprises: for each pair of consecutive surface markers along the first axis, determining a shape function between the pair of consecutive surface landmarks; and, estimating a deformation factor to be applied to the mesh between the pair of surface landmarks based on the determined shape function and the interpolation function, Wherein, modifying the coordinates of the vertex along the two second axes includes: multiplying the coordinates of the vertex along the two second axes by the estimated deformation factor.

11. The method according to any one of claims 1 to 10, wherein: The number of joints of the first virtual human body model is smaller than the number of joints of the second virtual human body model.

12. The method according to any one of claims 1 to 11, wherein: The first virtual human body model is a skeleton with ergonomic accuracy and / or the second virtual human body model is a skeleton with biomechanical accuracy.

13. A computer program comprising instructions for executing the method of any one of claims 1 to 11.

14. A computer-readable storage medium having recorded thereon the computer program according to claim 13.

15. A system for aligning a first virtual human model with a second virtual human model, comprising a processor connected to a memory and a graphical user interface, wherein the computer program according to claim 13 is recorded on the memory.

Images