Pose optimization methods, devices, electronic equipment, and storage media

Abstract

This application relates to the field of motion capture technology, and discloses a pose optimization method, device, electronic device, and storage medium. The method includes: reconstructing a human body from a human image to obtain the human body reconstruction result corresponding to the human image; calculating the transformation matrix between the model coordinate system and the camera coordinate system based on the human 2D mesh model, the human 3D mesh model, and the depth image; transforming the human 3D mesh model to the camera coordinate system according to the transformation matrix to obtain the pose information of the human 3D mesh model in the camera coordinate system; correcting the tilt of the human 3D mesh model transformed to the camera coordinate system according to the rotation matrix in the transformation matrix; and transforming the tilt-corrected pose information of the human 3D mesh model in the camera coordinate system to the world coordinate system according to the pose information of the terminal device to obtain the motion capture information corresponding to the human image. This solution can ensure the accuracy of the obtained motion capture information.

Description

Technical Field

[0001] This application relates to the field of motion capture technology, specifically to a pose optimization method, device, electronic device, and storage medium. Background Technology

[0002] Motion capture technology is used in virtual and augmented reality, robotics, and sports biology. It's a preliminary process that drives the movement of virtual objects. Ideally, motion capture data should describe pose information in a world coordinate system. However, some technologies use motion capture data obtained from captured human images, which often doesn't accurately reflect the pose information. Summary of the Invention

[0003] In view of this, embodiments of this application propose a pose optimization method, apparatus, electronic device, and storage medium to solve the problem that the pose information reflected by motion capture data in related technologies is not accurate enough.

[0004] The embodiments of this application are implemented using the following technical solutions: In a first aspect, embodiments of this application provide a pose optimization method, comprising: acquiring a person image and a depth image captured by a terminal device facing a real person; performing human body reconstruction based on the person image to obtain a human body reconstruction result corresponding to the person image; the human body reconstruction result includes a two-dimensional mesh model and a three-dimensional mesh model of the person; calculating a transformation matrix between the model coordinate system and the camera coordinate system based on the two-dimensional coordinates of multiple human key points in the two-dimensional mesh model of the person, the three-dimensional coordinates of multiple human key points in the three-dimensional mesh model of the person, and the depth image; transforming the pose information of each human key point in the three-dimensional mesh model of the person to the camera coordinate system according to the transformation matrix to obtain the pose information of each human key point in the three-dimensional mesh model of the person in the camera coordinate system; performing tilt correction on the three-dimensional mesh model of the person transformed to the camera coordinate system according to the rotation matrix in the transformation matrix; and transforming the tilt-corrected pose information of each human key point in the three-dimensional mesh model of the person in the camera coordinate system to the world coordinate system according to the pose information of the terminal device to obtain motion capture information corresponding to the person image.

[0005] Secondly, embodiments of this application provide a pose optimization device, comprising: an acquisition module for acquiring a person image and a depth image captured by a terminal device facing a real person; a human body reconstruction module for performing human body reconstruction based on the person image to obtain a human body reconstruction result corresponding to the person image; the human body reconstruction result includes a two-dimensional mesh model of the person and a three-dimensional mesh model of the person; a transformation matrix determination module for calculating a transformation matrix between the model coordinate system and the camera coordinate system based on the two-dimensional coordinates of multiple human key points in the two-dimensional mesh model of the person, the three-dimensional coordinates of multiple human key points in the three-dimensional mesh model of the person, and the depth image; and a first transformation module. The first module is used to transform the pose information of each human key point in the 3D mesh model of the person to the camera coordinate system according to the transformation matrix, so as to obtain the pose information of each human key point in the 3D mesh model of the person in the camera coordinate system; the second module is used to perform tilt correction on the 3D mesh model of the person transformed to the camera coordinate system according to the rotation matrix in the transformation matrix; the third module is used to transform the pose information of each human key point in the 3D mesh model of the person after tilt correction to the world coordinate system according to the pose information of the terminal device, so as to obtain the motion capture information corresponding to the person image.

[0006] Thirdly, embodiments of this application provide an electronic device, including: a processor; a memory, wherein computer instructions are stored in the memory, and when the computer instructions are executed by the processor, the above-described pose optimization method is implemented.

[0007] Fourthly, embodiments of this application provide a computer-readable storage medium storing computer instructions, which, when executed by a processor, implement the above-described pose optimization method.

[0008] Fifthly, embodiments of this application provide a computer program product, including computer instructions, which, when executed by a processor, implement the above-described pose optimization method.

[0009] The inventors of this application discovered through research that using a single-view image of a person to construct a human body and then identify the pose information of a real person can result in inaccurate perspective relationships. This can lead to a forward-tilting issue in the resulting 3D mesh model of the person. Therefore, this application addresses this problem by performing human body reconstruction based on the image, obtaining the corresponding reconstruction result, and then transforming the 3D mesh model to the camera coordinate system. The transformed 3D mesh model is then tilted and corrected. This alleviates the forward-tilting problem of the 3D mesh model in the camera coordinate system and even ensures that the transformed 3D mesh model in the camera coordinate system is free from forward tilt. Finally, the tilt-corrected 3D mesh model in the camera coordinate system is transformed to the world coordinate system, ensuring the accuracy of the motion capture information corresponding to the obtained image.

[0010] These or other aspects of this application will become more apparent from the description of the following embodiments. Attached Figure Description

[0011] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0012] Figure 1 This is a flowchart illustrating a pose optimization method according to an embodiment of this application.

[0013] Figure 2 This is a flowchart illustrating step 130 in one embodiment of this application.

[0014] Figure 3 This is a schematic diagram illustrating the before and after tilt correction of a 3D mesh model of a person, as shown in one embodiment of this application.

[0015] Figure 4 This is a flowchart illustrating the steps following step 160 in one embodiment of this application.

[0016] Figure 5 This is a flowchart illustrating the steps preceding step 420 according to an embodiment of this application.

[0017] Figure 6 This is a schematic diagram illustrating the software architecture for implementing the solution of this application according to an embodiment of this application.

[0018] Figure 7 This is a schematic diagram illustrating the data flow transmission of a character pose module according to an embodiment of this application.

[0019] Figure 8 This is a schematic diagram of a motion animation display interface according to an embodiment of this application.

[0020] Figure 9 This is a block diagram of a pose optimization device according to an embodiment of this application.

[0021] Figure 10 This is a block diagram of an electronic device provided in one embodiment of this application. Detailed Implementation

[0022] The embodiments of this application are described in detail below. Examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain this application, and should not be construed as limiting this application.

[0023] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are merely some embodiments of the present application, and not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of the present application without creative effort are within the scope of protection of the present application.

[0024] In the following description, the terms "first" and "second" are used merely to distinguish similar objects and do not represent a specific ordering of objects. It is understood that "first" and "second" may be interchanged in a specific order or sequence where permitted, so that the embodiments of this application described herein can be implemented in an order other than that illustrated or described herein.

[0025] In this document, "multiple" refers to two or more. "And / or" describes the relationship between associated objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, or B alone. The character " / " generally indicates that the preceding and following associated objects are in an "or" relationship. In the following description, references to "some embodiments or some embodiment methods" describe a subset of all possible embodiments. However, it is understood that "some embodiments" can be the same subset or different subsets of all possible embodiments and can be combined with each other without conflict.

[0026] Figure 1 This is a flowchart illustrating a pose optimization method according to an embodiment of this application. The method of this application can be executed by an electronic device, such as an image data terminal device, a server, or similar device. Figure 1As shown, the method includes steps 110-160: Step 110: Obtain the image and depth image of the person captured by the terminal device.

[0027] In this application, the terminal device integrates a depth camera and a camera for acquiring texture images. The camera can be used to acquire color images or grayscale images; that is, the image of a person can be a color image or a grayscale image. The image of a person acquired at the same acquisition time point corresponds to the depth image; that is, the depth image acquired at the same acquisition time point presents the depth values ​​of each pixel in the image of the person.

[0028] The terminal device can capture images of one or more real people. Correspondingly, the person image and depth image can present one or more real people.

[0029] The terminal device can capture images of real people in real time, obtaining a sequence of person images and a sequence of depth images. The i-th person image in the person image sequence and the i-th depth image in the depth image sequence are captured at the same acquisition time point, where i is a positive integer. The person images and depth images captured at the same acquisition time point in the person image sequence and depth image sequence can be processed according to the method of this application.

[0030] Step 120: Perform human body reconstruction based on the human image to obtain the human body reconstruction result corresponding to the human image; the human body reconstruction result includes a two-dimensional mesh model and a three-dimensional mesh model of the human body.

[0031] A 3D character mesh model, also known as a character 3D mesh, is a 3D character model composed of a series of points, edges, and mesh faces used to describe a human figure. Two points (also called vertices) on a 3D character mesh model are connected to form an edge, and multiple edges form a mesh face, for example, three edges form a mesh face.

[0032] A 2D character mesh model, also known as a character 2D mesh, is a mesh model used to describe a character, formed by a series of points, edges, and mesh faces on a 2D plane. Similarly, in a 3D character mesh model, two points (also called vertices) are connected to form an edge, and multiple edges form a mesh face, for example, three edges form a mesh face.

[0033] In some embodiments, a person image can be input into a human body reconstruction model, which then performs human body reconstruction based on the image to obtain the corresponding reconstruction result. The human body reconstruction model is a neural network model used for human body reconstruction, such as the SMPL (Skinned Multi-Person Linear Model), SMPL-X, SMPLify-X, SMPLer-X, etc., without specific limitations here.

[0034] In some embodiments, depth images and human images can also be input into the human body reconstruction model. The depth images are used to provide depth information for the human body reconstruction model, and the human body reconstruction model performs human body reconstruction based on the human image and the depth images to obtain the human body reconstruction result corresponding to the human image.

[0035] In some embodiments, step 120 includes the following ①-②: ① Performing person detection on the person image to obtain person detection results; the person detection results include the position information of the detection boxes corresponding to each real person presented in the person image.

[0036] ② Based on the location information of the detection boxes corresponding to each real person and the person image, human body reconstruction is performed to obtain the human body reconstruction results of each real person; the human body reconstruction result of a real person includes the two-dimensional mesh model and the three-dimensional mesh model of the real person.

[0037] Object detection models can be used to detect people in images and output the detection results. An object detection model is a neural network model used for object detection or recognition. It can be constructed using one or more neural networks. In this application, the object detection model aims to detect real people in the image. Examples of object detection models include YOLO, Fast R-CNN, and Mask R-CNN, etc., and are not specifically limited here.

[0038] The location information of the detection bounding boxes corresponding to each real person can include the coordinates of the center point of the detection box and the size of the detection box. In a person image, the pixel area enclosed by the detection box corresponding to a real person is the pixel area that displays that real person. Of course, if only one real person is displayed in the person image, the person detection result includes the location information of one detection box; if multiple real people are displayed in the person image, the person detection result includes the location information of multiple detection boxes. Different detection boxes have different identifiers to indicate that they correspond to different real people.

[0039] In some embodiments, the position information of the detection boxes corresponding to each real person can be input into the human body reconstruction model along with the person image, and the human body reconstruction model performs human body reconstruction. In this way, the human body reconstruction model can first extract the pixel regions representing each real person from the person image based on the position information of the detection boxes corresponding to each real person, and then perform human body reconstruction based on each real person's pixel regions. This reduces the interference of background pixel regions in the person image on human body reconstruction. In some embodiments, the pixel regions representing each real person can also be pre-extracted from the person image based on the position information of the detection boxes corresponding to each real person, and then input into the human body reconstruction model for human body reconstruction.

[0040] Human body reconstruction models can extract features from input images (such as images of people (or pixel regions of people cropped from images of people) and depth images) to obtain corresponding feature maps. Then, based on the feature maps, the two-dimensional coordinates of multiple human body key points can be estimated, and the estimated two-dimensional coordinates of these key points can be used to fit and generate a two-dimensional mesh model of the person corresponding to each real person. Furthermore, based on the feature maps, the two-dimensional coordinates of the key points, and the two-dimensional mesh model, the human body reconstruction model can estimate the three-dimensional coordinates of the key points and fit and generate a three-dimensional mesh model of the person corresponding to each real person. It can be seen that multiple points on the two-dimensional mesh model of the person represent key points; similarly, multiple points on the three-dimensional mesh model of the person represent key points.

[0041] Key points in the human body refer to characteristic points in the human skeletal structure, which can be joints or limb endpoints. Examples of key points in the human body include the left shoulder joint, right shoulder joint, left elbow joint, right elbow joint, left knee joint, right knee joint, and pelvic joint.

[0042] In some embodiments, the two-dimensional mesh model of the real person can be a two-dimensional mesh model on the pixel plane (i.e., the zoy plane). Correspondingly, the two-dimensional coordinates of the human body key points on the two-dimensional mesh model of the real person can refer to the two-dimensional pixel coordinates of the human body key points on the image of the person.

[0043] Step 130: Calculate the transformation matrix between the model coordinate system and the camera coordinate system based on the two-dimensional coordinates of multiple key points in the two-dimensional mesh model of the person, the three-dimensional coordinates of multiple key points in the three-dimensional mesh model of the person, and the depth image.

[0044] The camera coordinate system refers to the camera coordinate system on the terminal device that acquires images of a person and depth images. The model coordinate system is the world coordinate system in the 3D model space, that is, the world coordinate system in the 3D model space where the 3D mesh model of the person resides. The 3D coordinates of multiple human keypoints in the 3D mesh model of the person are the coordinates of the human keypoints in the model coordinate system in the 3D model space where the 3D mesh model of the person resides. The pixel coordinates and depth values ​​of each pixel in the depth image are related to the camera coordinate system. In addition, the 3D mesh model of the person includes not only the 3D coordinates of multiple human keypoints, but also the rotation information of multiple human keypoints on it. The 2D mesh model of the person includes not only the 3D coordinates of multiple human keypoints, but also the rotation information (2D rotation information) of multiple human keypoints on it.

[0045] For the same real-world person, the corresponding 2D mesh model is essentially a projection of the 3D mesh model, such as projecting the 3D mesh model onto a pixel plane. In other words, for the same human keypoint, the 2D coordinates of that keypoint in the 2D mesh model lack depth information compared to its 3D coordinates in the 3D mesh model. Since a depth image can provide depth information for each pixel, the transformation matrix between the model coordinate system and the camera coordinate system can be calculated by combining the 2D coordinates of multiple keypoints in the 2D mesh model, the 3D coordinates of multiple keypoints in the 3D mesh model, and the depth image.

[0046] In some embodiments, such as Figure 2 As shown, step 130 includes the following steps 210-240: Step 210: Sampling of key points in the two-dimensional mesh model of the human figure to obtain the two-dimensional coordinates of K target key points. K does not exceed the total number of key points of the human body and K is an integer greater than 1.

[0047] Wherein, K can be set according to actual needs and is not specifically limited here. For example, two key points can be sampled from each mesh face in the 2D mesh model of the person. Target key points refer to the human body key points sampled in the 2D mesh model of the person. In some embodiments, to ensure the accuracy of the transformation matrix determined subsequently, in step 210, the K target key points can only be the human body key points that are not occluded by the human body itself. For example, when a real person is standing facing the camera, the human body key points on the mesh 2D of the back are the points occluded by the human body, and the human body key points on the mesh 2D of the chest are the points that are not occluded by the human body.

[0048] In some embodiments, step 210 includes the following steps A1-A4: Step A1: Sample key points in a specified body region of the character's 2D mesh model to obtain the 2D coordinates of N sampled key points; the specified body region refers to the body region other than the head region, hand region, and foot region; N is an integer greater than K.

[0049] Considering that there are many noisy human body key points in the foot, hand, and head regions of the human body in the 2D mesh model, in order to avoid inaccurate results due to the calculation of the transformation matrix using noisy human body key points, sampling can be performed in other body regions of the human body in the 2D mesh model other than the head, hand, and foot regions to obtain N sampled key points, and the corresponding 2D coordinates of the N sampled key points in the 2D mesh model of the human body can be obtained.

[0050] Step A2: Based on the two-dimensional coordinates of the N sampling key points, obtain the depth values ​​of the N sampling key points in the depth image.

[0051] Based on the two-dimensional coordinates of N sampling keypoints, the corresponding pixel representing each sampling keypoint can be located in the depth image, and the depth value of that pixel can be obtained as the depth value of the corresponding sampling keypoint. For example, if the two-dimensional coordinates of a sampling keypoint are (x1, y1), the pixel with coordinates (x1, y1) can be located in the depth image, and the pixel value z1 of that pixel in the depth image can be obtained, which is the depth value of the sampling keypoint.

[0052] Step A3: Calculate the average depth value by averaging the depth values ​​of the N sampled key points.

[0053] Step A4: Filter out the sampling key points among the N sampling key points whose distance between the depth value and the average depth value exceeds the distance threshold, and obtain K target key points.

[0054] Considering that during keypoint sampling, points may be sampled from the background area, resulting in unreliable depth values ​​and potentially leading to errors in the calculated transformation matrix, to reduce the occurrence of such situations, and given that the differences in depth values ​​between different keypoints on a real human body are relatively small, keypoints whose depth values ​​exceed a distance threshold from the average depth value among N sampled keypoints are filtered out. This distance threshold can be set according to actual needs, such as 1m, 1.1m, 1.2m, etc.

[0055] By using the above method, the number of unreliable points sampled as key sampling points for calculating the transformation matrix can be reduced, thereby ensuring the accuracy of the subsequently determined transformation matrix.

[0056] Step 220: Determine the reference three-dimensional coordinates of the K target key points based on their two-dimensional coordinates and depth values ​​in the depth image.

[0057] For a target keypoint, its two-dimensional coordinates can be combined with its depth value in a depth image to obtain its reference three-dimensional coordinates. For example, if the two-dimensional coordinates of a target keypoint are (x1, y1) and its depth value in a depth image is z1, then the reference three-dimensional coordinates of the target keypoint are (x1, y1, z1).

[0058] Step 230: Based on the two-dimensional coordinates of the K target key points, obtain the three-dimensional coordinates of the K target key points in the three-dimensional mesh model of the character.

[0059] As described above, the 3D coordinates of key points in the 3D mesh model of a person are determined based on the 2D coordinates of key points in the 2D mesh model of a person. Therefore, there is a correspondence between points in the 3D mesh model of a person and points in the 2D mesh model of a person. Thus, based on the correspondence between points in the 3D mesh model of a person and points in the 2D mesh model of a person, and the 2D coordinates of K key points, the K key points can be located in the 3D mesh model of a person, and the 3D coordinates of the K key points in the 3D mesh model of a person can be obtained.

[0060] Step 240: Based on the reference 3D coordinates of the K target key points and the 3D coordinates of the K target key points in the 3D mesh model of the person, calculate the transformation matrix between the model coordinate system and the camera coordinate system.

[0061] The transformation matrix T between the model coordinate system and the camera coordinate system can be expressed as: ;(Formula 1) Where R represents the rotation matrix, t is the translation vector, and c represents the scaling factor.

[0062] Theoretically, by transforming the i-th target key point P (whose reference 3D coordinates are (x1, y1, z1)) to the model coordinate system corresponding to the 3D mesh model of the character, the theoretical 3D coordinates of the i-th target key point P on the 3D mesh model of the character can be determined by the following formula: ;(Formula 2) in, Let P represent the theoretical 3D coordinates of the i-th target key point P on the 3D mesh model of the character; Represents the reference three-dimensional coordinates of the i-th target key point P.

[0063] However, in practice, the actual 3D coordinates of the target key point P on the character's 3D mesh model may differ from the theoretical 3D coordinates calculated using the corresponding transformation matrix T. Therefore, the transformation matrix can be determined with the objective of minimizing the difference between the theoretical 3D coordinates of the K target key points on the character's 3D mesh model and their actual 3D coordinates. For example, the objective function can be set as follows: ;(Formula 3) in, This represents the actual 3D coordinates of the i-th target key point P on the character's 3D mesh model, that is, the 3D coordinates of the target key point determined in step 240 on the character's 3D mesh model. This represents the theoretical 3D coordinates of the i-th target key point P on the 3D mesh model of the character, as determined by Formula 2. The objective can be to minimize the function value of the objective function L. Based on the initial settings of c, R, and t, c, R, and t are iteratively updated to determine the c, R, and t values ​​when L reaches their minimum. Then, the transformation matrix T is determined based on the c, R, and t values ​​when L reaches their minimum.

[0064] Step 140: Transform the pose information of each key point of the human body in the 3D mesh model of the human body to the camera coordinate system according to the transformation matrix, so as to obtain the pose information of each key point of the human body in the 3D mesh model of the human body in the camera coordinate system.

[0065] With the transformation matrix determined, we can combine it with Formula 2 above to transform the pose information of each key point of the human body in the 3D mesh model to the camera coordinate system, thereby obtaining the pose information of each key point of the human body in the 3D mesh model in the camera coordinate system.

[0066] Step 150: Based on the rotation matrix in the transformation matrix, the tilt correction is performed on the 3D mesh model of the character transformed into the camera coordinate system.

[0067] In some embodiments, pose information includes rotation information; the rotation information of the root node in the 3D mesh model of the person in the camera coordinate system can be tilted and corrected according to the rotation matrix in the transformation matrix.

[0068] In this context, the root node in the 3D human mesh model can refer to the key points of the human body that serve as the basis for positioning within the 3D human mesh model. For example, it could be a key point on the pelvis, a joint on the foot, etc., without specific limitations. If, based on the rotation information of the root node in the 3D human mesh model in the camera coordinate system, it is determined that the 3D human mesh model is tilted forward in the camera coordinate system, the rotation matrix in the transformation matrix can be used to correct this rotation information. For example, the rotation information of the root node in the camera coordinate system can be inversely transformed, thus correcting the forward tilt of the 3D human mesh model in the camera coordinate system.

[0069] Figure 3 This is a schematic diagram illustrating the before and after tilt correction of a 3D mesh model of a person, as shown in one embodiment of this application. Figure 3 The left image in the image is the 3D mesh model of the figure before tilt correction. Figure 3 The right image in the image is a 3D mesh model of the figure after tilt correction in accordance with the method described in this application.

[0070] In some embodiments, multiple human key points include pelvic key points. After step 150, the method further includes: obtaining the target depth value of the pelvic key points in the camera coordinate system in the tilt-corrected 3D mesh model of the person; determining the target reference 3D coordinates of the pelvic key points in the camera coordinate system based on the target depth value and the camera intrinsic parameters of the camera from which the person image originates; and aligning the pelvic key points of the tilt-corrected 3D mesh model of the person in the camera coordinate system to the target reference 3D coordinates.

[0071] The camera intrinsics of the camera from which the image of a person originates can be reflected by the camera intrinsics matrix. For example, suppose the camera intrinsics matrix of the camera from which the image of a person originates is: ,in,( , () indicates the camera's focal length, usually = , ( , (x, y) represents the coordinates of the camera's optical center. If the pelvic keypoint's 2D coordinates in the character's 2D mesh model are (x, y), the target reference 3D coordinates of the pelvic keypoint in the camera coordinate system can be determined using the following formula 6. , , ): ;(Formula 4) in, It is also equal to the target depth value.

[0072] In practice, the projection of the 3D network model of the person may not be highly aligned with the real person (which can be understood as the real person in the image of the person). Since the position of the person is usually located by the pelvic key points, the pelvic key points of the tilt-corrected 3D mesh model of the person in the camera coordinate system are aligned with the target reference 3D coordinates in the above way (which is equivalent to translating the tilt-corrected 3D mesh model of the person). In this way, the projection of the 3D network model of the person can be aligned with the person.

[0073] Step 160: Based on the pose information of the terminal device, transform the pose information of each human key point in the tilt-corrected 3D mesh model of the person in the camera coordinate system to the world coordinate system to obtain the motion capture information corresponding to the person image.

[0074] The motion capture information corresponding to the human image includes the pose information of multiple human key points in the world coordinate system at the corresponding moment in the human image. The pose information includes position information (3D coordinates) and rotation information. The multiple human key points can be some of the human key points in the transformed 3D human network model, or all of the human key points.

[0075] The pose information of the terminal device can be represented by the extrinsic parameter matrix of the camera on the terminal device. The extrinsic parameter matrix of the camera consists of a 3*3 identity orthogonal rotation matrix R1 and a 3*1 translation vector t1, which describes the transformation relationship between the world coordinate system and the camera coordinate system.

[0076] Based on this, the 3D coordinates of multiple human body key points in the camera coordinate system can be transformed to the world coordinate system using the camera's extrinsic parameter matrix, thus obtaining the 3D coordinates of the human body key points in the world coordinate system. This process can be represented as: ;(Formula 5) Where (Xi, Yi, Zi) are the three-dimensional coordinates of the i-th human body keypoint in the world coordinate system; , , () represents the three-dimensional coordinates of the i-th human body key point in the camera coordinate system. This represents the extrinsic parameter matrix of the camera, i.e., the pose information of the terminal device.

[0077] The rotation information of multiple human keypoints in the camera coordinate system can be transformed to the world coordinate system using the identity orthogonal rotation matrix R1 in the camera's extrinsic parameter matrix. This process can be represented as: ;(Formula 6) in,( , , ) represents the rotation information of the i-th human body keypoint in the world coordinate system, ( , , ) represents the rotation information of the i-th human body keypoint in the camera coordinate system.

[0078] In some embodiments, the pose information of the terminal device can be determined using a visual positioning method or a visual inertial odometry (VIO) method based on a person image and its corresponding depth image. Examples of visual positioning methods include Simultaneous Localization and Mapping (SLAM).

[0079] In some embodiments, an inertial measurement unit (IMU) can also be worn on the real person during the acquisition process. The pose information of the terminal device can be determined based on the person image and the IMU data acquired at the acquisition time point corresponding to the person image, according to the visual inertial odometry method.

[0080] The inventors of this application discovered through research that constructing and identifying the pose information of a real person using a single-view image of a person can result in inaccurate perspective relationships, leading to a forward-tilting issue in the resulting 3D mesh model. Therefore, this application involves reconstructing the human body from the image, obtaining the corresponding reconstruction result, and then transforming the 3D mesh model to the camera coordinate system. The transformed 3D mesh model is then tilted and corrected to alleviate the forward-tilting problem, even ensuring that the transformed 3D mesh model in the camera coordinate system is completely free of this issue. Subsequently, the tilt-corrected 3D mesh model in the camera coordinate system is transformed to the world coordinate system, ensuring the accuracy of the motion capture information corresponding to the obtained image. Using this motion capture information to drive a virtual model ensures consistency between the virtual model's movements and the movements of the real person reflected in the image, improving the driving effect and user experience.

[0081] Furthermore, the pose information of key points on the human body in the motion capture information corresponding to the image is also in the world coordinate system, which facilitates the direct use of the motion capture information corresponding to the image to drive virtual models in any coordinate system without the need for additional coordinate transformation. In addition, since the solution of this application can use a single terminal device for image data acquisition, there is no need to additionally calibrate the relative positions when the positions of multiple shooting devices change, making motion capture simpler and more convenient, effectively improving the efficiency of motion capture information acquisition. Moreover, the method of this application can be used not only for motion capture of a single real person but also for motion capture of multiple real people, making it widely applicable.

[0082] In some embodiments, considering that the real person in the image may interfere with the positioning of the terminal device, the real person in the image can be masked first to obtain a masked image; then, the terminal device is positioned based on the masked image and the target depth image to obtain the pose information of the terminal device when acquiring the target image and the target depth image.

[0083] In some embodiments, semantic segmentation models (such as Mask R-CNN models) can be used to perform semantic segmentation on person images to generate person masks corresponding to the person images. In the person mask, the pixel values ​​of the pixel region where the real person is located in the person image are different from the pixel values ​​of the pixel region where the background is located. Moreover, the pixel values ​​of different pixels in the pixel region where the real person is located in the person image are the same. For example, the pixel value of the pixel region where the real person is located in the person image is 1, and the pixel value of the pixel region where the background is located is 0.

[0084] Then, using the person mask corresponding to the person image, the real person presented in the person image can be masked to obtain a person mask image. In other words, compared to the person image, the pixel values ​​of each pixel in the pixel region representing the background in the person mask image are the same as the pixel values ​​of the corresponding pixels in the person image, while the person pixel region in the person mask image is masked to a specified pixel value, so that the actual pixel content of the person pixel region is not displayed in the person mask image.

[0085] Subsequently, the pose information of the terminal device can be determined using visual positioning methods or visual inertial odometry (VIO) based on the person mask image and the corresponding depth image. Alternatively, the pose information of the terminal device can be determined using visual inertial odometry based on the person mask image and IMU data acquired by the IMU at the acquisition time point corresponding to the person image.

[0086] By pre-masking the real person in the image and using the masked image, and then locating the terminal device based on the masked image and the target depth image, the problem of the real person in the image interfering with the terminal positioning can be effectively solved, ensuring the accuracy of the obtained position and pose information of the terminal device.

[0087] In some embodiments, the method further includes the following steps B1-B4: Step B1: Extract the face pixel region and hand pixel region from the person image.

[0088] Step B2: Perform facial expression recognition based on the facial pixel region to obtain facial expression parameters.

[0089] Step B3: Perform hand pose estimation based on the hand pixel region to obtain hand pose information.

[0090] Step B4: Add the facial expression parameters and hand pose information to the motion capture information corresponding to the person's image.

[0091] Hand pose estimation models can be used to estimate hand pose based on the hand's pixel region, yielding hand pose information. This information includes the pose information (position and rotation information) of each hand keypoint, reflecting the gesture presented by the hand's pixel region. In some embodiments, the hand pixel region includes a left-hand pixel region and a right-hand pixel region. The hand pose estimation model can estimate the hand pose of the left-hand and right-hand pixel regions separately, obtaining the left-hand pose information and the right-hand pose information, respectively. Examples of hand pose estimation models include the InterWild model, which is not specifically limited here.

[0092] In some embodiments, facial expression parameters can be obtained by using an expression parameter recognition model to perform expression recognition on facial pixel regions. Such expression parameter recognition models include, but are not limited to, MediaPipe.

[0093] Considering that in the process of reconstructing a complete human body, the 3D mesh model of the human body may contain relatively coarse areas for the hands and face, and the corresponding pose information of the hands and facial expression parameters may also be inaccurate. Therefore, in this embodiment, the facial and hand pixel regions of each real person can be extracted from the image, and hand pose estimation and expression recognition can be performed separately. This ensures that the obtained facial expression parameters and hand pose information are more accurate. Thus, subsequent use of these facial expression parameters and hand pose information to drive the target object model to present the corresponding expressions and gestures can be more realistic and produce better visual effects.

[0094] In some embodiments, the image of a person is any frame from a sequence of images of people; such as Figure 4 As shown, after step 160, the method further includes: Step 410: Arrange the motion capture information corresponding to multiple frames of human images in the human image sequence according to the position of each frame of human image in the human image sequence to obtain motion capture data.

[0095] The motion capture information corresponding to the human image is used to characterize the pose information of the real person presented in the human image at the time of image capture. According to the position of each frame of human image in the human image sequence, the motion capture information corresponding to multiple frames of human images in the human image sequence is arranged. In this way, the motion capture data can reflect the pose information of the real person at multiple consecutive time points of capture.

[0096] It is understandable that if each frame of a person image sequence only presents a real person, the corresponding motion capture data includes the pose information of a real person at multiple consecutive acquisition time points.

[0097] In some embodiments, the real people presented in the human image are multiple; the motion capture information corresponding to a frame of human image includes the motion capture information corresponding to each real person presented in the human image; the motion capture data includes the motion capture data corresponding to each real person among the multiple real people; correspondingly, step 410 includes: arranging the motion capture information corresponding to the same real person in multiple frames of human images in the human image sequence according to the position of each frame of human image in the human image sequence, to obtain the motion capture data corresponding to each real person.

[0098] The motion capture data corresponding to a real person reflects the pose information of a real person at multiple consecutive acquisition time points.

[0099] Step 420: Smooth the motion capture information that is adjacent in position in the motion capture data to obtain the target motion capture data.

[0100] In some embodiments, if the sequence of images of people is captured from multiple real people, the motion capture data of each real person can be smoothed separately.

[0101] It is understandable that the motion capture information in adjacent positions in motion capture data refers to the motion capture information at adjacent acquisition time points. Since the motion capture information corresponding to each frame of a person's image in the motion capture data includes the position information (coordinates) and rotation information of multiple human key points at the corresponding acquisition time point, the smoothing process can be smoothing the position information of the same real person at adjacent acquisition time points, as well as smoothing the rotation information of the same real person at adjacent acquisition time points.

[0102] In some embodiments, the positional information of the same real person at adjacent acquisition time points can be smoothed using the slerp function (spherical linear interpolation); the rotational information of the same real person at adjacent acquisition time points can be smoothed using the moving smoothing function.

[0103] Considering that motion capture information for each frame of a person is determined on a per-image basis, but the actions and postures of the same real person within a sequence of images are temporally continuous, determining motion capture information for each individual image fails to account for this temporal continuity. This could lead to significant variations in the motion capture data for the same real person between adjacent capture points. Consequently, directly using this motion capture data to drive virtual model movements could result in severe motion jitter. Therefore, smoothing adjacent motion capture information within the motion capture data can reduce jitter.

[0104] In some embodiments, after step 420, the method further includes: redirecting the target motion capture data to a target virtual model in a virtual scene, and driving the target virtual model to perform actions according to the target motion capture data. Thus, during the process of the target virtual model performing actions according to the target motion capture data, the posture or action presented by the target virtual model is the same as the posture or action presented by the real person in the corresponding person image.

[0105] In some embodiments, before redirecting the target motion capture data to a virtual model in a virtual scene and driving the virtual model to perform actions according to the target motion capture data, the method further includes: determining, based on the target shape features corresponding to the target motion capture data, a target virtual model in a virtual model library whose shape features have a similarity to the target shape features exceeding a shape similarity threshold; and loading the target virtual model into the virtual scene.

[0106] The target shape features corresponding to the target motion capture data refer to the shape features of the human skeleton model formed by multiple human key points in the motion capture information corresponding to a frame of human image in the target motion capture data. Shape features include, for example, height, body width, leg length, and gender. In this way, target virtual models with shape features that have a similarity exceeding the shape similarity threshold can be identified in the virtual model library.

[0107] In some embodiments, such as Figure 5 As shown, prior to step 420, the method further includes the following steps 510-530: Step 510: If it is determined from the motion capture data that there is a reference human image in the human image sequence that lacks motion capture information, determine the first adjacent human image before the reference human image and the second adjacent human image after the reference human image in the human image sequence.

[0108] A reference image refers to an image of a person in a sequence of images where motion capture information is missing. The first adjacent image refers to an image in the sequence that is adjacent to and precedes the reference image; that is, the first adjacent image is the image preceding the reference image. The second adjacent image refers to an image in the sequence that is adjacent to and follows the reference image; the second adjacent image is the image following the reference image.

[0109] Step 520: Interpolate the motion capture information corresponding to the first adjacent person image and the motion capture information corresponding to the second adjacent person image to obtain the motion capture information corresponding to the reference person image.

[0110] Since the movements of real people are continuous in a short period of time, interpolation can be performed using the motion capture information from the previous acquisition time point (i.e., the motion capture information corresponding to the first adjacent person image) and the motion capture information from the next acquisition time point (i.e., the motion capture information corresponding to the second adjacent person image) to estimate the pose information of the real person presented in the reference person image, thus obtaining the motion capture information corresponding to the reference person image.

[0111] It can be achieved by interpolating the motion capture information corresponding to the first adjacent human image and the motion capture information corresponding to the second adjacent human image, which represent the pose information of the same human key point at two different acquisition time points, to obtain the pose information of each human key point at the acquisition time point corresponding to the reference human image.

[0112] Similarly, linear interpolation can be used to interpolate the position information of the same human keypoint at two different acquisition time points. Considering that linear interpolation can cause gimbal lock when dealing with the rotation information of human keypoints, a spherical linear interpolation function can be used to interpolate the rotation information of the same human keypoint at two different acquisition time points.

[0113] If there are multiple real people in the image, reference images with missing motion capture information can be identified in the motion capture data corresponding to each real person. Then, in a similar manner, the motion capture information of the real person at the previous acquisition time point and the motion capture information of the real person at the next acquisition time point of the reference image can be interpolated in the motion capture data corresponding to the real person to obtain the motion capture information of the real person at the acquisition time point corresponding to the reference image.

[0114] Step 530: Add the motion capture information corresponding to the reference person image to the motion capture data according to the position of the reference person image in the person image sequence.

[0115] By using the above method, interpolation is performed on motion capture information from two adjacent acquisition time points to supplement the missing motion capture information at acquisition time points. This ensures that the motion capture data is relatively continuous overall, and avoids the problem of large changes in pose information between two adjacent acquisition time points, which could lead to significant motion jitter.

[0116] In some embodiments, after arranging the motion capture information of the same real person in multiple frames of the person image sequence according to the position of each frame of the person image in the person image sequence, and obtaining the motion capture data corresponding to each real person, the method further includes the following steps C1 and C2: Step C1: If the data similarity between the first action data segment corresponding to the first real person and the second action data segment corresponding to the second real person exceeds the similarity threshold, the first action data segment and the second action data segment are fused to obtain a fused action data segment; the first action data segment and the second action data segment are aligned in time.

[0117] The first motion data segment includes motion capture information of a first real person from multiple consecutively captured images, and the second motion data segment includes motion capture information of a second real person from multiple consecutively captured images. The first and second motion data segments are time-aligned, meaning that the capture time of the first motion capture information in the first motion data segment is the same as the capture time of the first motion capture information in the second motion data segment, and the capture time of the last motion capture information in the first motion data segment is the same as the capture time of the last motion capture information in the second motion data segment.

[0118] In some embodiments, fusing the first motion data segment and the second motion data segment to obtain a fused motion data segment may include: averaging the motion capture information of the first person in the first motion data segment and the motion capture information of the second person in the second motion data segment at the same acquisition time point to obtain fused motion capture information of the person at each acquisition time point; and arranging the fused motion capture information of the person at each acquisition time point in the acquisition time period corresponding to the first motion data segment in chronological order to obtain the fused motion data segment.

[0119] Step C2: Replace the first motion data segment in the motion capture data of the first real person with the fused motion data segment, and replace the second motion data segment in the motion capture data of the second real person with the fused motion data segment.

[0120] If the data similarity between the first motion data segment corresponding to the first real person and the second motion data segment corresponding to the second real person exceeds a similarity threshold, it indicates that the postures of the first and second real people are highly similar within the acquisition time period corresponding to the first (or second) motion data segment. Therefore, the first and second motion data segments are fused to obtain a fused motion data segment, which is then used to replace the first and second motion data segments. In this way, when using motion capture data from multiple real people to drive multiple virtual models, the same fused motion data segment can be bound to different virtual models with high motion similarity. This improves the process of multiple virtual models performing actions according to the target motion capture data, and enhances the visual coordination of different virtual models in terms of movement, thereby improving the user experience.

[0121] The solution of this application will be specifically described below with reference to a specific embodiment.

[0122] Figure 6 This is a schematic diagram illustrating a software architecture for implementing the solution of this application, according to an embodiment of this application. Figure 6 As shown, the software architecture includes: a recording module 610, a device positioning module 620, a person tracking module 630, a person pose module 640, a pose optimization module 650, a display module 660, and an export module 670.

[0123] The recording module 610 can receive human images and depth images captured by the terminal device, and can also receive IMU data captured by the IMU. In some embodiments, the terminal device can integrate a camera and a depth camera; in some embodiments, the IMU can also be used to capture IMU data. In other embodiments, point cloud data can also be captured by LiDAR to assist in human body reconstruction, for example, by providing depth information through point cloud data.

[0124] The device positioning module 620 is used to locate the terminal device based on the acquired image of a person (or at least one of depth image and IMU data) using VIO (Visual Inertial Odometry) or SLAM (Simultaneous Localization and Mapping) to obtain the pose information of the terminal device. Furthermore, the pixel region of the person in the image can be pre-masked before being used for terminal positioning to avoid interference from the person in the image. When the device positioning module 620 uses the image of the person for terminal device positioning via SLAM, it can also generate scene data based on the environmental scene presented in the image.

[0125] The person tracking module 630 is used to detect people in a person image and determine the detection box corresponding to each real person in the image. It can assign different detection box labels to different real people, thus enabling real person tracking across multiple person images. When multiple real people are present in a person image, multi-person tracking can also be performed, assigning different detection box labels to different detected people to distinguish different human bodies and track their trajectories. In some embodiments, the StrongSORT algorithm can be used for multi-person tracking.

[0126] The human pose module 640 has two main functions. First, it can reconstruct the human body from a human image to obtain the reconstruction result. This reconstruction result includes a 2D mesh model and a 3D mesh model of the real human figure. The 2D mesh model includes the 2D coordinates of multiple key points on the human body, and the 3D mesh model includes the pose information (position and rotation information) of these key points. Second, it can perform facial expression recognition based on the facial pixel region in the human image to obtain facial expression parameters, and perform hand pose estimation based on the hand pixel region in the human image to obtain hand pose information.

[0127] Figure 7 This is a schematic diagram illustrating the data flow transmission of a character pose module according to an embodiment of this application. For example... Figure 7 As shown, the image of the person and the position information of the detection boxes detected by the person tracking module 630 can be input into the human body reconstruction model, which then performs human body reconstruction to obtain the reconstruction result. Afterwards, tilt correction and position estimation are performed based on the reconstruction result to obtain the pose information of each key point of the person in the camera coordinate system. The person mask is then input into the device positioning module 620 for terminal device positioning. Furthermore, the hand pixel region can be extracted from the image of the person to perform gesture recognition and obtain hand pose information; the face pixel region can also be extracted from the image of the person and expression recognition can be performed to obtain facial expression parameters. Then, the hand pose information, facial expression parameters, and the pose information of each key point of the person in the camera coordinate system are input into the pose optimization model 650 for further processing.

[0128] The pose optimization module 650 transforms the 3D coordinates of multiple human keypoints in the camera coordinate system to the world coordinate system, obtaining the pose information of multiple human keypoints on a real human body in the world coordinate system (i.e., motion capture information corresponding to the human image). This results in target motion capture data that presents the movements of a real person over a continuous time period. Furthermore, the pose optimization module 650 can also fill in motion capture information for reference human images lacking motion capture information, preventing discontinuous movements of the subsequent target virtual object. Additionally, adjacent motion capture information in the motion capture data is smoothed to make the movements presented in a short time period more reasonable, reducing motion jitter and avoiding the problem of neglecting the temporal continuity of movement when performing pose estimation on a single human image basis.

[0129] The display module 660 is used to display the constructed virtual model and present animations of the process of driving the virtual model to move using target motion capture data. In addition, the display module 660 can also display the constructed virtual scene in real time, which can switch the corresponding viewpoint based on the user's selection, and the virtual scene can be zoomed in and out. The terminal device can use real-time rendering technology, that is, calculating the image while outputting the display, allowing for real-time control, real-time interaction, and high-speed image processing to achieve realistic effects.

[0130] Figure 8 This is a schematic diagram of a motion animation display interface according to an embodiment of this application, such as... Figure 8As shown, the display interface can display the driven target virtual model 810, and also displays the corresponding character image 820. Clicking the play control in the display interface can play the animation of the target virtual model moving according to the target motion capture data, and can also display the target virtual model's face and movements according to facial expression parameters. Furthermore, the position of the target virtual model 810 in the virtual scene corresponds to the position of the real person presented in the character image in the actual scene.

[0131] The motion export module 670 can save the animation of a target virtual model in a virtual scene, driven by sensor data (such as human images, depth images, IMU data, etc.) and motion capture data. In the product, the motion export module is... Figure 8 The “Export Action” control 830 shown in the display interface can be clicked to save the generated animation. In addition, the “Regenerate” control 840 shown in the display interface can be clicked to generate a new animation.

[0132] In some embodiments, before applying the target motion capture data to an animation project, i.e., before using the target motion capture data to drive the target virtual model, it is necessary to redirect the target motion capture data to avoid the problem that the target motion capture data cannot be used due to differences in size and length between the target virtual model and the character skeleton model in the target motion capture data.

[0133] Motion retargeting can be performed in 3D character animation software (such as Autodesk Motion Builder), and its main steps include: 1. Import the target character model into the 3D character animation software and convert it into a character. This step matches the character model's skeleton (i.e., the skeletal structure of the 3D mesh model reflected in the target motion capture data) with the built-in skeleton templates in the 3D character animation software.

[0134] 2. Import the target motion capture data and create an actor model for it. The actor is the bridge between the marker data exported from the target motion capture data and the skeletal animation data in 3D character animation software.

[0135] 3. Use performer model data to drive the characterization of the character model.

[0136] 4. Adjust the parameters of the driving method in step 3. Since both the performer model and the characterized model conform to the HumanIK standard built into the 3D character animation software, the matching degree can be adjusted by adjusting the weight ratio of IK (inverse dynamics) and FK (forward dynamics) in the humanIK controller.

[0137] By default, animation is driven entirely by FK (Frame-King). This inevitably leads to animation misalignment due to the body shape differences between the character model and the performer model. This can easily cause missing contact or overlapping issues on the performer model's surface during movement. In such cases, animators need to manually adjust the IK (In-K) animation based on the FK.

[0138] 5. Save the adjustment results to the target character's skeleton by baking the animation, so that the target character no longer needs to be driven by the Actor (performer model).

[0139] Furthermore, any input object model can be analyzed to obtain its external features (i.e., the human skeleton model presented by the target motion capture data, or the performer model created for the target motion capture data). These external features can be the object model's height, leg length, body proportions, etc., and this application should not limit them to these aspects. Based on the input external features, a model adjustment range is determined, which represents the adjustment range required to adjust the external features of the baseline character model to the external features of the corresponding arbitrary object model.

[0140] Alternatively, the model can be obtained by matching the shape features of the input object model with models in a model library. A model library is built based on the shape features of multiple object models, such as height, leg length, body shape, gender, and age. The model library matches the input object model's shape features with the stored object models, selecting the target object model that best matches the input. The target model's shape features are replicated 1:1 with the object model; that is, if the object model's arm is 30cm, then the corresponding target model's arm length is 30cm, and the object model's shape features are identical to the target model's. Then, the adjusted object model is rendered, and motion capture data is bound to it.

[0141] The solution proposed in this application can be applied to a single terminal device for motion capture without requiring multiple shooting devices. Therefore, it eliminates the need for additional relative position calibration when the positions of multiple shooting devices change, making it convenient and simple to use. Furthermore, the method can capture motion not only from a single real person but also from multiple real people. Moreover, by integrating gesture recognition and facial expression recognition, high-quality motion capture data can be obtained. Actions can be redirected to other virtual objects based on the user's posture information, generating virtual object animations corresponding to the acquired data stream. Actions learned from the data stream can be redirected to any character model, displaying the target virtual model in world coordinates and ensuring the continuity and accuracy of the target virtual model's movements.

[0142] The following describes an apparatus embodiment of this application, which can be used to perform the methods described in the above embodiments of this application. For details not disclosed in the apparatus embodiments of this application, please refer to the method embodiments described in the above embodiments of this application.

[0143] Figure 9 This is a block diagram of a pose optimization device according to an embodiment of this application, such as... Figure 9 As shown, the pose optimization device includes: an acquisition module 910, used to acquire a person image and depth image captured by a terminal device facing a real person; a human body reconstruction module 920, used to perform human body reconstruction based on the person image to obtain the human body reconstruction result corresponding to the person image; the human body reconstruction result includes a two-dimensional mesh model and a three-dimensional mesh model of the person; a transformation matrix determination module 930, used to calculate the transformation matrix between the model coordinate system and the camera coordinate system based on the two-dimensional coordinates of multiple human key points in the two-dimensional mesh model of the person, the three-dimensional coordinates of multiple human key points in the three-dimensional mesh model of the person, and the depth image; and a first transformation module 94. 0 is used to transform the pose information of each human key point in the 3D mesh model of the person to the camera coordinate system according to the transformation matrix, so as to obtain the pose information of each human key point in the 3D mesh model of the person in the camera coordinate system; tilt correction module 950 is used to correct the tilt of the 3D mesh model of the person transformed to the camera coordinate system according to the rotation matrix in the transformation matrix; second transformation module 960 is used to transform the pose information of each human key point in the tilt-corrected 3D mesh model of the person in the camera coordinate system to the world coordinate system according to the pose information of the terminal device, so as to obtain the motion capture information corresponding to the person image.

[0144] In some embodiments, the pose information includes rotation information; the tilt correction module 950 is further configured to: perform tilt correction on the rotation information of the root node in the 3D mesh model of the person in the camera coordinate system according to the rotation matrix in the transformation matrix.

[0145] In some embodiments, multiple human key points include pelvic key points; the pose optimization device further includes: a target depth value acquisition module, used to acquire the target depth value of the pelvic key points in the camera coordinate system in the tilt-corrected 3D mesh model of the person; a target reference 3D coordinate determination module, used to determine the target reference 3D coordinates of the pelvic key points in the camera coordinate system based on the target depth value and the camera intrinsic parameters of the camera from which the person image originates; and an alignment module, used to align the pelvic key points of the tilt-corrected 3D mesh model of the person in the camera coordinate system to the target reference 3D coordinates.

[0146] In some embodiments, the transformation matrix determination module 930 includes: a key point sampling unit, used to sample key points in a two-dimensional mesh model of a person to obtain the two-dimensional coordinates of K target key points, where K does not exceed the total number of key points on the human body and K is an integer greater than 1; a reference three-dimensional coordinate determination unit, used to determine the reference three-dimensional coordinates of the K target key points based on the two-dimensional coordinates of the K target key points and the depth values ​​of the K target key points in the depth image; a three-dimensional coordinate acquisition unit, used to acquire the three-dimensional coordinates of the K target key points in the three-dimensional mesh model of the person based on the two-dimensional coordinates of the K target key points; and a transformation matrix determination unit, used to calculate the transformation matrix between the model coordinate system and the camera coordinate system based on the reference three-dimensional coordinates of the K target key points and the three-dimensional coordinates of the K target key points in the three-dimensional mesh model of the person.

[0147] In some embodiments, the person image is any frame image from the person image sequence; the pose optimization device further includes: an arrangement module, used to arrange the motion capture information corresponding to multiple frames of person images in the person image sequence according to the position of each frame of person image in the person image sequence, to obtain motion capture data; and a smoothing processing module, used to smooth the motion capture information that is adjacent in position in the motion capture data, to obtain target motion capture data.

[0148] In some embodiments, the pose optimization device further includes: an adjacent image determination module, configured to determine a first adjacent person image preceding the reference person image and a second adjacent person image following the reference person image in the person image sequence if a reference person image with missing motion capture information is determined in the motion capture data; an interpolation processing module, configured to interpolate the motion capture information corresponding to the first adjacent person image and the motion capture information corresponding to the second adjacent person image to obtain the motion capture information corresponding to the reference person image; and a first addition module, configured to add the motion capture information corresponding to the reference person image to the motion capture data according to the position of the reference person image in the person image sequence.

[0149] In some embodiments, the real people presented in the human image are multiple; the motion capture information corresponding to a frame of human image includes the motion capture information corresponding to each real person presented in the human image; the motion capture data includes the motion capture data corresponding to each real person among the multiple real people; the arrangement module is further used to: arrange the motion capture information corresponding to the same real person in multiple frames of human images in the human image sequence according to the position of each frame of human image in the human image sequence, so as to obtain the motion capture data corresponding to each real person.

[0150] In some embodiments, the pose optimization device further includes: a fusion module, configured to fuse the first action data segment and the second action data segment to obtain a fused action data segment if the data similarity between the first action data segment corresponding to the first real person and the second action data segment corresponding to the second real person exceeds a similarity threshold; the first action data segment and the second action data segment are time-aligned; and a replacement module, configured to replace the first action data segment in the motion capture data of the person corresponding to the first real person in the target motion capture data with the fused action data segment, and to replace the second action data segment in the motion capture data of the person corresponding to the second real person with the fused action data segment.

[0151] In some embodiments, the pose optimization device further includes: a cropping module for cropping a face pixel region and a hand pixel region from a person image; an expression recognition module for performing expression recognition based on the face pixel region to obtain facial expression parameters; a hand pose estimation module for performing hand pose estimation based on the hand pixel region to obtain hand pose information; and a second adding module for adding the facial expression parameters and hand pose information to the motion capture information corresponding to the person image.

[0152] In some embodiments, the pose optimization device further includes: a masking module for masking the real person presented in the person image to obtain a person mask image; and a positioning module for positioning the terminal device based on the person mask image and the depth image to obtain the pose information of the terminal device.

[0153] In some embodiments, the pose optimization device further includes a redirection module, configured to redirect the target motion capture data to a target virtual model in a virtual scene, and drive the target virtual model to perform actions according to the target motion capture data.

[0154] Figure 10 This is a schematic diagram of an electronic device according to an embodiment of this application. This electronic device can be used to execute the pose optimization method provided in this application. Figure 10As shown, the electronic device may include: a processor 1001, such as a CPU; a network interface 1004; a user interface 1003; a memory 1005; and a communication bus 1002. The communication bus 1002 is used to enable communication between these components. The user interface 1003 may include a display screen and an input unit such as a keyboard. Optionally, the user interface 1003 may also include a standard wired interface or a wireless interface. The network interface 1004 may optionally include a standard wired interface or a wireless interface (such as a Wi-Fi interface). The memory 1005 may be high-speed RAM or non-volatile memory, such as a disk drive. The memory 1005 may also optionally be a storage device independent of the aforementioned processor 1001. Those skilled in the art will understand that... Figure 10 The structure of the electronic device shown does not constitute a limitation on the electronic device and may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0155] like Figure 10 As shown, the memory 1005, which is a computer-readable storage medium, may include an operating system, a network communication module, a user interface module, and a program for implementing the pose optimization method.

[0156] exist Figure 10 In the illustrated electronic device, the network interface 1004 is mainly used for communication connections with other devices. The user interface 1003 is mainly used for connecting to clients and communicating data with them; while the processor 1001 can be used to call the program implementing the test method stored in the memory 1005 and execute the steps of the method as described in any of the above method embodiments.

[0157] According to one aspect of the embodiments of this application, a computer-readable storage medium is provided that stores computer-readable instructions thereon, which, when executed by a processor, implement the method as described in any of the above method embodiments.

[0158] According to one aspect of the embodiments of this application, a computer program product is provided, which includes computer instructions that, when executed by a processor, implement the methods as described in any of the above method embodiments.

[0159] The units described in the embodiments of this application can be implemented in software or hardware, and the described units can also be located in a processor. The names of these units do not necessarily limit the specific unit itself.

[0160] Through the above description of the embodiments, those skilled in the art will readily understand that the exemplary embodiments described herein can be implemented by software or by combining software with necessary hardware. Therefore, the technical solutions according to the embodiments of this application can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (such as a CD-ROM, USB flash drive, external hard drive, etc.) or on a network, including several instructions to cause a computing device (such as a personal computer, server, touch terminal, or network device, etc.) to execute the methods according to the embodiments of this application.

[0161] Other embodiments of this application will readily occur to those skilled in the art upon consideration of the specification and practice of the embodiments disclosed herein. This application is intended to cover any variations, uses, or adaptations of this application that follow the general principles of this application and include common knowledge or customary techniques in the art not disclosed herein.

[0162] It should be understood that this application is not limited to the precise structure described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this application is limited only by the appended claims.

Claims

1. A pose optimization method, characterized in that, include: Acquire images and depth images of real people captured through terminal devices; Human body reconstruction is performed based on the human image to obtain the human body reconstruction result corresponding to the human image; the human body reconstruction result includes a two-dimensional mesh model of the human body and a three-dimensional mesh model of the human body. Based on the two-dimensional coordinates of multiple key human body points in the two-dimensional mesh model of the person, the three-dimensional coordinates of multiple key human body points in the three-dimensional mesh model of the person, and the depth image, calculate the transformation matrix between the model coordinate system and the camera coordinate system; The pose information of each human key point in the three-dimensional mesh model of the human figure is transformed to the camera coordinate system according to the transformation matrix, so as to obtain the pose information of each human key point in the three-dimensional mesh model of the human figure in the camera coordinate system. Based on the rotation matrix in the transformation matrix, the 3D mesh model of the person transformed into the camera coordinate system is tilted and corrected. Based on the pose information of the terminal device, the pose information of each key point of the human body in the tilt-corrected 3D mesh model of the human body in the camera coordinate system is transformed to the world coordinate system to obtain the motion capture information corresponding to the human image.

2. The method according to claim 1, characterized in that, The pose information includes rotation information; the tilt correction of the 3D mesh model of the person transformed into the camera coordinate system according to the rotation matrix in the transformation matrix includes: Based on the rotation matrix in the transformation matrix, the rotation information of the root node in the 3D mesh model of the character is tilted and corrected in the camera coordinate system.

3. The method according to claim 1 or 2, characterized in that, The aforementioned key points of the human body include key points of the pelvis. Before transforming the pose information of each human key point in the tilt-corrected 3D mesh model of the person in the camera coordinate system to the world coordinate system based on the pose information of the terminal device to obtain the motion capture data corresponding to the person image, the method further includes: Obtain the target depth value of the pelvic key points in the camera coordinate system in the tilt-corrected 3D mesh model of the person; Based on the target depth value and the camera intrinsic parameters of the camera from which the image of the person originates, determine the target reference three-dimensional coordinates of the pelvic key points in the camera coordinate system; The tilt-corrected 3D mesh model of the person is aligned with the pelvic key points in the camera coordinate system to the target reference 3D coordinates.

4. The method according to claim 1, characterized in that, The step of calculating the transformation matrix between the model coordinate system and the camera coordinate system based on the two-dimensional coordinates of multiple key points of the human body in the two-dimensional mesh model of the human body, the three-dimensional coordinates of multiple key points of the human body in the three-dimensional mesh model of the human body, and the depth image includes: Key points are sampled in the two-dimensional mesh model of the human body to obtain the two-dimensional coordinates of K target key points. K does not exceed the total number of key points of the human body and K is an integer greater than 1. Based on the two-dimensional coordinates of the K target key points and the depth values ​​of the K target key points in the depth image, the reference three-dimensional coordinates of the K target key points are determined; Based on the two-dimensional coordinates of the K target key points, obtain the three-dimensional coordinates of the K target key points in the three-dimensional mesh model of the character; Based on the reference 3D coordinates of the K target key points and the 3D coordinates of the K target key points in the 3D mesh model of the character, calculate the transformation matrix between the model coordinate system and the camera coordinate system.

5. The method according to claim 1, characterized in that, The image of a person is any frame from the sequence of images of people; After transforming the pose information of each key point of the human body in the tilt-corrected 3D mesh model of the person in the camera coordinate system to the world coordinate system based on the pose information of the terminal device to obtain the motion capture information corresponding to the person image, the method further includes: According to the position of each frame of the character image in the character image sequence, the motion capture information corresponding to multiple frames of character images in the character image sequence is arranged to obtain motion capture data; The motion capture data is smoothed by processing adjacent motion capture information in the motion capture data to obtain the target motion capture data.

6. The method according to claim 5, characterized in that, Before smoothing adjacent motion capture information in the motion capture data to obtain the target motion capture data, the method further includes: If, based on motion capture data, it is determined that there is a reference person image in the sequence of person images that lacks motion capture information, then a first adjacent person image preceding the reference person image and a second adjacent person image following the reference person image are determined in the sequence of person images. The motion capture information corresponding to the first adjacent person image and the motion capture information corresponding to the second adjacent person image are interpolated to obtain the motion capture information corresponding to the reference person image. According to the position of the reference image in the image sequence, the motion capture information corresponding to the reference image is added to the motion capture data.

7. The method according to claim 5, characterized in that, The image of a person may contain multiple real people; the motion capture information corresponding to one frame of the image of a person includes the motion capture information of each real person in the image of a person; the motion capture data includes the motion capture data of each real person among the multiple real people. The motion capture data is obtained by arranging the motion capture information corresponding to multiple frames of the character image in the character image sequence according to the position of each frame of the character image in the character image sequence, including: According to the position of each frame of the character image in the character image sequence, the motion capture information of the same real person in multiple frames of the character image sequence is arranged to obtain the motion capture data corresponding to each real person.

8. The method according to claim 7, characterized in that, After arranging the motion capture information of the same real person in multiple frames of the character image sequence according to the position of each frame of the character image in the character image sequence, and obtaining the motion capture data corresponding to each real person, the method further includes: If it is determined that the data similarity between the first action data segment corresponding to the first real person and the second action data segment corresponding to the second real person exceeds a similarity threshold, the first action data segment and the second action data segment are fused to obtain a fused action data segment; the first action data segment and the second action data segment are aligned in time. In the target motion capture data, the first motion data segment in the motion capture data corresponding to the first real person is replaced with the fused motion data segment, and the second motion data segment in the motion capture data corresponding to the second real person is replaced with the fused motion data segment.

9. The method according to claim 1, characterized in that, The method further includes: Extract the face pixel region and hand pixel region from the image of the person; Facial expression parameters are obtained by performing expression recognition based on the facial pixel regions. Hand pose estimation is performed based on the hand pixel region to obtain hand pose information; The facial expression parameters and hand pose information are added to the motion capture information corresponding to the person image.

10. The method according to claim 1, characterized in that, Before transforming the pose information of each human key point in the tilt-corrected 3D mesh model of the person in the camera coordinate system to the world coordinate system based on the pose information of the terminal device to obtain the motion capture information corresponding to the person image, the method further includes: The real person presented in the image is masked to obtain a masked image of the person; Based on the person mask image and the depth image, the terminal device is located to obtain the pose information of the terminal device.

11. The method according to any one of claims 5 to 8, characterized in that, After smoothing adjacent motion capture information in the motion capture data to obtain the target motion capture data, the method further includes: The target motion capture data is redirected to the target virtual model in the virtual scene, and the target virtual model is driven to perform actions according to the target motion capture data.

12. A pose optimization device, characterized in that, include: The acquisition module is used to acquire images of people and depth images collected from real people through terminal devices; The human body reconstruction module is used to reconstruct the human body based on the human image and obtain the human body reconstruction result corresponding to the human image; the human body reconstruction result includes a two-dimensional mesh model of the human body and a three-dimensional mesh model of the human body. The transformation matrix determination module is used to calculate the transformation matrix between the model coordinate system and the camera coordinate system based on the two-dimensional coordinates of multiple human key points in the two-dimensional mesh model of the person, the three-dimensional coordinates of multiple human key points in the three-dimensional mesh model of the person, and the depth image. The first transformation module is used to transform the pose information of each human key point in the three-dimensional mesh model of the human body to the camera coordinate system according to the transformation matrix, so as to obtain the pose information of each human key point in the three-dimensional mesh model of the human body in the camera coordinate system. The tilt correction module is used to correct the tilt of the 3D mesh model of the person transformed into the camera coordinate system according to the rotation matrix in the transformation matrix. The second transformation module is used to transform the pose information of each human key point in the tilt-corrected three-dimensional mesh model of the person in the camera coordinate system to the world coordinate system according to the pose information of the terminal device, so as to obtain the motion capture information corresponding to the person image.

13. An electronic device, characterized in that, include: processor; A memory storing computer instructions that, when executed by the processor, implement the method as described in any one of claims 1-11.

14. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions that, when executed by a processor, implement the method as described in any one of claims 1-11.

Images