Interactive motion capture method, apparatus and electronic device

Abstract

This invention provides an interactive motion capture method, apparatus, and electronic device. It acquires the rigid body pose information of a rigid body bound to an interactive object, and the rigid body pose information of a human body bound to a rigid body. The pose information of object components is determined based on the rigid body pose information, taking into account the binding relationship between the rigid body and the object, thus improving the accuracy of object pose determination. Furthermore, the pose information of human joints is determined based on the human body pose information and the object component pose information. This method considers the binding relationship between the rigid body and the human body, and the combination of the object component pose information to determine the human pose results in a more physically accurate and responsive human pose.

Description

Technical Field

[0001] This invention relates to the field of computer technology, and in particular to an interactive motion capture method, apparatus, and electronic device. Background Technology

[0002] In recent years, motion capture technology has played a crucial role in many fields. In many practical applications, the human body being motion-captured needs to interact with objects in the scene, thus requiring the simultaneous acquisition of data such as the position and posture of both the human body and the objects.

[0003] In existing technologies, parameters obtained by optical instruments attached to objects and human bodies based on optical key points are directly used as the pose information of objects and human bodies. Some optical instruments fail due to occlusion, and since the obtained parameters are the pose information of optical instruments, the gap between optical instruments and objects and human bodies is ignored, resulting in inaccurate motion (pose) capture or even non-physical laws.

[0004] Therefore, proposing an interactive motion capture method to address the shortcomings of inaccurate motion capture is an urgent problem to be solved. Summary of the Invention

[0005] This invention provides an interactive motion capture method, device, and electronic device to overcome the shortcomings of inaccurate motion capture in the prior art and achieve accurate motion capture, that is, accurate capture and determination of the pose of objects and human bodies.

[0006] This invention provides an interactive motion capture method, comprising:

[0007] Get the pose information of the rigid body of the object bound to the interactive object, and get the pose information of the rigid body of the human body bound to the interactive human body.

[0008] Determine the pose information of the object components based on the rigid body pose information of the object.

[0009] The human joint pose information is determined based on the human rigid body pose information and the object component pose information.

[0010] According to an interactive motion capture method provided by the present invention, the step of determining the pose information of object components based on the rigid body pose information of the object includes:

[0011] The object joint pose information of the interactive object is determined based on the rigid body pose information of the object.

[0012] The loss function is calculated based on the object joint pose information and the previously acquired object joint pose information from the previous moment.

[0013] The object joint pose information is optimized according to the loss function to obtain the optimized object joint pose information;

[0014] The pose information of the object components is determined based on the optimized object joint pose information.

[0015] According to an interactive motion capture method provided by the present invention, the step of determining human joint pose information based on the human rigid body pose information and the object component pose information includes:

[0016] The human rigid body pose information is optimized by inverse kinematics of the human skeleton to obtain human surface point pose information, wherein the human surface point pose information includes the pose information of multiple human surface points of the interactive human body.

[0017] Based on the pose information of the human body surface points and the pose information of the object components, the cumulative gradient of the human body joints is determined;

[0018] The human joint pose information is determined based on the human body surface point pose information and the human body joint cumulative gradient.

[0019] According to an interactive motion capture method provided by the present invention, a signed distance between the human body surface points and the interactive object is calculated based on the pose information of the human body surface points and the pose information of the object components.

[0020] The human body surface points are filtered according to the signed distance and preset conditions to obtain target human body surface points;

[0021] The first gradient is calculated based on the signed distance and pose information of the target human body surface point;

[0022] The average gradient of multiple human joints of the interactive human is determined based on the first gradient and the preset human skin weights, and the cumulative gradient of the human joints is determined based on the average gradient.

[0023] The present invention also provides an interactive motion capture device, comprising:

[0024] The acquisition unit is used to acquire the rigid body pose information of the rigid body bound to the interactive object, and to acquire the rigid body pose information of the human body bound to the rigid body of the interactive human body.

[0025] The object analysis unit is used to determine the pose information of object components based on the rigid body pose information of the object.

[0026] The human body analysis unit is used to determine the human joint pose information based on the rigid body pose information of the human body and the pose information of the object components.

[0027] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of any of the above-described interactive motion capture methods.

[0028] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of any of the above-described interactive motion capture methods.

[0029] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of any of the above-described interactive motion capture methods.

[0030] The interactive motion capture method, apparatus, and electronic device provided by this invention acquire the rigid body pose information of a rigid body bound to an interactive object, and acquire the rigid body pose information of a human body bound to a rigid body. The pose information of object components is determined based on the rigid body pose information, taking into account the binding relationship between the rigid body and the object, thus improving the accuracy of object pose determination. The pose information of human joints is determined based on the human body pose information and the object component pose information, taking into account the binding relationship between the rigid body and the human body, and combining the object component pose information to determine the human pose, resulting in a more physically accurate and responsive human pose. Attached Figure Description

[0031] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0032] Figure 1 This is a flowchart illustrating the interactive motion capture method provided by the present invention;

[0033] Figure 2 This is a schematic diagram of the interactive motion capture device provided by the present invention;

[0034] Figure 3 This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation

[0035] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0036] This invention provides an interactive motion capture method, such as... Figure 1 As shown, it includes steps S11-S13.

[0037] S11. Obtain the pose information of the rigid body of the object bound to the interactive object, and obtain the pose information of the rigid body of the human body bound to the interactive human body.

[0038] Specifically, a rigid body can be formed using optical equipment and an inertial measurement unit (IMU). One or more rigid bodies can be bound to the components of the interactive object and the limbs of the interactive human. The rigid body's pose information of the interactive object and the human body can be obtained through the rigid body. By combining optical equipment with IMU to obtain pose information, the pose information of the rigid body on the interactive object or human body can be supplemented by the IMU when the optical equipment is occluded. This facilitates the subsequent determination of the pose information of the interactive object and human body based on the pose information of the rigid body, improving the richness and accuracy of the pose information.

[0039] S12. Determine the pose information of the object components based on the rigid body pose information of the object.

[0040] S13. Determine the human joint pose information based on the human rigid body pose information and the object component pose information.

[0041] In this embodiment of the invention, the rigid body pose information of a rigid body bound to an interactive object is obtained, and the rigid body pose information of a human body bound to an interactive human body is obtained. The pose information of object components is determined based on the rigid body pose information, taking into account the binding relationship between the rigid body and the object, thus improving the accuracy of object pose determination. The pose information of human joints is determined based on the human body pose information and the object component pose information, taking into account the binding relationship between the rigid body and the human body, and combining the object component pose information to determine the human pose, resulting in a more physically accurate and erroneous human pose.

[0042] According to the interactive motion capture method provided by the present invention, step S12 may include steps S121-S124.

[0043] S121. Determine the joint pose information of the interactive object based on the rigid body pose information of the object.

[0044] Specifically, in one example, multiple reflective spheres and an inertial measurement unit are combined into a rigid body component, which is then bound to each movable part of the object. Because the rigid body component is small, and only one rigid body component needs to be bound to each movable part, if the binding position is chosen appropriately, it will not affect the interaction between the human body and the object. Furthermore, unobstructed areas can be manually selected to increase recognition accuracy.

[0045] The root joint of an articulated object is typically of the free-moving type; therefore, the pose of the component directly connected to the root joint is uniquely determined by the bound rigid body. The joint pose information of an interactive object, including the position and orientation of each joint, can be determined based on the rigid body pose information. For translational joints (prismatic), movement in one direction is allowed, with a translation vector of (x, y, z); for revolute joints (revolute), rotation about an axis is allowed, with a rotation vector of (r, y, p). The joint pose information of the interactive object is constructed from the translation and rotation vectors.

[0046] S122. Calculate the loss function based on the object joint pose information and the previously acquired object joint pose information from the previous moment.

[0047] Specifically, the joint pose information of the preceding object at the previous moment can be obtained, and the deviation between the current object joint pose information and the preceding object joint pose information can be calculated, i.e., the loss function.

[0048] In one example, the object joint pose information at the current moment includes translation vector (x1, y1, z1) and rotation vector (r1, y1, p1), and the previous object joint pose information at the previous moment includes translation vector (x2, y2, z2) and rotation vector (r2, y2, p2). The geometric distance L2 between the current moment and the previous moment can be calculated as the loss function based on (x1, y1, z1), (r1, y1, p1), (x2, y2, z2), and (r2, y2, p2).

[0049] S123. Optimize the object joint pose information according to the loss function to obtain the optimized object joint pose information.

[0050] Specifically, the loss function reveals changes in the joint pose information of an interactive object. This loss function can then be used to optimize the current joint pose information, resulting in an optimized joint pose. In one example, if the loss function exceeds a preset value, it indicates that the pose of the interactive object has changed too much between the previous and current moments. In this case, the current joint pose information can be optimized. For instance, the loss function can be increased or decreased by a preset percentage based on the translation and / or rotation vectors represented by the current joint pose information to obtain the optimized joint pose information.

[0051] S124. Determine the pose information of the object component based on the optimized object joint pose information.

[0052] Specifically, the pose information of multiple joints of an interactive object can be obtained based on the optimized object joint pose information. The joints of the interactive object are the object's components, meaning that the pose information of the object's components can be determined through the optimized object joint pose information.

[0053] In this embodiment of the invention, the rigid body pose information of the rigid body bound to the interactive object is converted into the joint pose information of each joint of the interactive object. The binding relationship between the detection device and the interactive object is considered, and the conversion is performed accordingly, providing a reliable foundation for obtaining accurate object component pose information subsequently. The current object joint pose information is optimized based on the loss function between the previous and current object joint pose information, making it more accurate. The optimized object joint pose information is then used to determine the component pose information from the joint angles of the object, achieving an accurate description of the object component pose.

[0054] According to the interactive motion capture method provided by the present invention, step S13 may include steps S131-S133.

[0055] S131. Perform inverse kinematics optimization on the human rigid body pose information to obtain the pose information of points on the human body surface.

[0056] The human body surface point pose information includes the pose information of multiple human body surface points of the interactive human body.

[0057] Specifically, due to the special nature of the human skeletal structure, the range of motion of human limbs is limited. The human skeleton inverse kinematics optimization is performed on the human rigid body pose information bound to the human body to remove the pose information of rigid bodies that do not meet the preset conditions. After removal, the point pose information of multiple surface points of the interactive human body can be determined, that is, the human body surface point pose information.

[0058] S132. Determine the cumulative gradient of the human joints based on the pose information of the human body surface points and the pose information of the object components.

[0059] Specifically, step S132 may include S1321-S1324.

[0060] S1321. Calculate the signed distance between the human body surface points and the interactive object based on the human body surface point pose information and the object component pose information.

[0061] Specifically, the pose information of multiple points on the human body surface can be determined based on the pose information of the human body surface points, and the pose information of each component of the interactive object can be determined based on the pose information of the object components. Using these two pieces of information, the signed distance between the human body surface points and the interactive object can be determined. The signed distance indicates whether the human body surface point is inside the interactive object and its distance from the interactive object. For example, if the signed distance between a human body surface point A and the interactive object is -5mm, it means that point A is inside the interactive object and its closest distance to the interactive object is 5mm. As another example, if the signed distance between a human body surface point B and the interactive object is +3mm, it means that point B is close to but not in contact with the interactive object, is outside the interactive object, and its closest distance to the interactive object is 3mm.

[0062] S1322. The human body surface points are filtered according to the signed distance and preset conditions to obtain target human body surface points.

[0063] Specifically, points on the human body surface can be filtered based on signed distance and preset conditions. In one example, the preset condition is set to points with a signed distance of less than 5mm. This allows filtering out human body surface points that are close to (5mm) but not in contact with the object, as well as human body surface points inside the object, which are then used as target human body surface points.

[0064] S1323. Calculate the first gradient based on the signed distance and pose information of the target human body surface point.

[0065] Specifically, in one example, the signed distance δ corresponding to a selected target human body surface point x can be calculated by differentiating the square of the signed distance with respect to the coordinates of the target human body surface point x. This serves as the first gradient. The corresponding first gradient can be calculated for each target human body surface point using the steps described above.

[0066] S1324. Determine the average gradient of multiple human joints of the interactive human body based on the first gradient and the preset human skin weights, and determine the cumulative gradient of the human joints based on the average gradient.

[0067] Specifically, for the first gradient corresponding to each target human body surface point, the average gradient of multiple human joints of the interactive human body can be determined according to preset human skinning weights. The average gradients of multiple human joints are summed to obtain the cumulative gradient of the human joints.

[0068] S133. Determine the human joint pose information based on the human body surface point pose information and the human body joint cumulative gradient.

[0069] Specifically, the pose of points on the human body surface can be converted into the pose of the human body joints based on the pose information of points on the human body surface and the cumulative gradient of human body joints.

[0070] In this embodiment of the invention, human body rigid body pose information is optimized using inverse kinematics of the human skeleton to obtain human body surface point pose information, making the human body surface point pose information more consistent with physical rules and more accurate. A signed distance is calculated based on the human body surface point pose information and the object component pose information. This signed distance is used to filter and obtain target human body surface points with greater interactive significance. A first gradient between the interactive human body surface and the interactive object is calculated based on the signed distance corresponding to the target human body surface point. The average gradient between the human body surface and human joints is determined based on the first gradient, and then the cumulative gradient of the human joints is calculated. Based on the cumulative gradient of the human joints and the human body surface pose information, the human joint pose information is determined from the human body surface pose, reducing errors in the human body surface point pose information. The human joint pose information of the interactive human body joints is extracted, achieving accurate capture of the interactive human body pose.

[0071] The interactive motion capture device provided by the present invention is described below. The interactive motion capture device described below can be referred to in correspondence with the interactive motion capture method described above.

[0072] The present invention also provides an interactive motion capture device, such as Figure 2 As shown, it includes:

[0073] The acquisition unit 21 is used to acquire the rigid body pose information of the rigid body bound to the interactive object and the rigid body pose information of the human body bound to the interactive human body.

[0074] The object analysis unit 22 is used to determine the pose information of the object components based on the rigid body pose information of the object.

[0075] The human body analysis unit 23 is used to determine the human body joint pose information based on the human body rigid body pose information and the object component pose information.

[0076] In this embodiment of the invention, the rigid body pose information of the rigid body bound to the interactive object is obtained, and the rigid body pose information of the human body bound to the interactive human body is obtained. The pose information of object components is determined based on the rigid body pose information, taking into account the binding relationship between the rigid body and the object, thus improving the accuracy of object pose determination. The pose information of human joints is determined based on the human body pose information and the object component pose information, taking into account the binding relationship between the rigid body and the human body, and combining the object component pose information to determine the human body pose, resulting in a more physically accurate and erroneous human body pose.

[0077] According to the interactive motion capture device provided by the present invention, the object analysis unit 22 is specifically used for:

[0078] The object joint pose information of the interactive object is determined based on the rigid body pose information of the object.

[0079] The loss function is calculated based on the object joint pose information and the previously acquired object joint pose information from the previous moment.

[0080] The object joint pose information is optimized according to the loss function to obtain the optimized object joint pose information;

[0081] The pose information of the object components is determined based on the optimized object joint pose information.

[0082] According to the interactive motion capture device provided by the present invention, the human body analysis unit 23 is specifically used for:

[0083] The human rigid body pose information is optimized by inverse kinematics of the human skeleton to obtain human surface point pose information, wherein the human surface point pose information includes the point pose information of multiple human surface points of the interactive human body.

[0084] Based on the pose information of the human body surface points and the pose information of the object components, the cumulative gradient of the human body joints is determined;

[0085] The human joint pose information is determined based on the human body surface point pose information and the human body joint cumulative gradient.

[0086] According to the interactive motion capture device provided by the present invention, the human body analysis unit 23 is specifically used for:

[0087] Based on the pose information of the human body surface points and the pose information of the object components, calculate the signed distance between the human body surface points and the interactive object;

[0088] The human body surface points are filtered according to the signed distance and preset conditions to obtain target human body surface points;

[0089] The first gradient is calculated based on the signed distance and pose information of the target human body surface point;

[0090] The average gradient of multiple human joints of the interactive human is determined based on the first gradient and the preset human skin weights, and the cumulative gradient of the human joints is determined based on the average gradient.

[0091] Figure 3 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 3 As shown, the electronic device may include a processor 310, a communications interface 320, a memory 330, and a communication bus 340. The processor 310, communications interface 320, and memory 330 communicate with each other via the communication bus 340. The processor 310 can call logical instructions in the memory 330 to execute an interactive motion capture method. This method includes: acquiring the rigid body pose information of a rigid body bound to an interactive object; acquiring the rigid body pose information of a rigid body bound to an interactive human body; determining the pose information of object components based on the rigid body pose information; and determining the pose information of human joints based on the rigid body pose information and the object component pose information.

[0092] Furthermore, the logical instructions in the aforementioned memory 330 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0093] On the other hand, the present invention also provides a computer program product, which includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the interactive motion capture method provided by the above methods. The method includes: acquiring rigid body pose information of a rigid body bound to an interactive object, acquiring rigid body pose information of a human body bound to an interactive human body; determining object component pose information based on the rigid body pose information; and determining human joint pose information based on the human body pose information and the object component pose information.

[0094] In another aspect, the present invention also provides a non-transitory computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the interactive motion capture method provided by the above methods. The method includes: acquiring rigid body pose information of a rigid body bound to an interactive object; acquiring rigid body pose information of a rigid body bound to an interactive human body; determining object component pose information based on the rigid body pose information; and determining human joint pose information based on the rigid body pose information of the human body and the object component pose information.

[0095] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0096] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0097] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. An interactive motion capture method, characterized by, include: Get the pose information of the rigid body of the object bound to the interactive object, and get the pose information of the rigid body of the human body bound to the interactive human body. Determine the pose information of the object components based on the rigid body pose information of the object. The human joint pose information is determined based on the human rigid body pose information and the object component pose information; The step of determining the human joint pose information based on the rigid body pose information of the human body and the pose information of the object components includes: The human rigid body pose information is optimized by inverse kinematics of the human skeleton to obtain human surface point pose information, wherein the human surface point pose information includes the pose information of multiple human surface points of the interactive human body. Based on the pose information of the human body surface points and the pose information of the object components, the cumulative gradient of the human body joints is determined; The human joint pose information is determined based on the human surface point pose information and the cumulative gradient of the human joints. The step of determining the cumulative gradient of human joints based on the pose information of the human body surface points and the pose information of the object components includes: Based on the pose information of the human body surface points and the pose information of the object components, calculate the signed distance between the human body surface points and the interactive object; The human body surface points are filtered according to the signed distance and preset conditions to obtain target human body surface points; The first gradient is calculated based on the signed distance and pose information of the target human body surface point; The average gradient of multiple human joints of the interactive human is determined based on the first gradient and the preset human skin weights, and the cumulative gradient of the human joints is determined based on the average gradient.

2. The interactive motion capture method according to claim 1, characterized in that, Determining the pose information of object components based on the rigid body pose information of the object includes: The object joint pose information of the interactive object is determined based on the rigid body pose information of the object. The loss function is calculated based on the object joint pose information and the previously acquired object joint pose information from the previous moment. The object joint pose information is optimized according to the loss function to obtain the optimized object joint pose information; The pose information of the object components is determined based on the optimized object joint pose information.

3. An interactive motion capture device, characterized in that, include: The acquisition unit is used to acquire the rigid body pose information of the rigid body bound to the interactive object, and to acquire the rigid body pose information of the human body bound to the rigid body of the interactive human body. The object analysis unit is used to determine the pose information of object components based on the rigid body pose information of the object. The human body analysis unit is used to determine the human joint pose information based on the human rigid body pose information and the object component pose information. The human body analysis unit is specifically used for: The human rigid body pose information is optimized by inverse kinematics of the human skeleton to obtain human surface point pose information, wherein the human surface point pose information includes the point pose information of multiple human surface points of the interactive human body. Based on the pose information of the human body surface points and the pose information of the object components, the cumulative gradient of the human body joints is determined; The human joint pose information is determined based on the human surface point pose information and the cumulative gradient of the human joints. The human body analysis unit is specifically used for: Based on the pose information of the human body surface points and the pose information of the object components, calculate the signed distance between the human body surface points and the interactive object; The human body surface points are filtered according to the signed distance and preset conditions to obtain target human body surface points; The first gradient is calculated based on the signed distance and pose information of the target human body surface point; The average gradient of multiple human joints of the interactive human is determined based on the first gradient and the preset human skin weights, and the cumulative gradient of the human joints is determined based on the average gradient.

4. The interactive motion capture device according to claim 3, characterized in that, The object analysis unit is specifically used for: The object joint pose information of the interactive object is determined based on the rigid body pose information of the object. The loss function is calculated based on the object joint pose information and the previously acquired object joint pose information from the previous moment. The object joint pose information is optimized according to the loss function to obtain the optimized object joint pose information; The pose information of the object components is determined based on the optimized object joint pose information.

5. An electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the interactive motion capture method as described in any one of claims 1 to 2.

6. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the steps of the interactive motion capture method as described in any one of claims 1 to 2.

Images